CAIN-42 evidence library
VAGI Behavior Suite v1 — paid model baseline (2026-11-18)
Evidence bundle: 0 claims, 0 of 0 invariants held.
Last reviewed 2026-11-19
Verify it in your browser
Your browser downloads the bundle's SHA256SUMS manifest and the file(s) behind this page, hashes them with SHA-256 locally (WebCrypto), and compares. A match shows the record you are reading is the published one; it does not by itself prove who published it (see the signed claims registry and the bundle verifier for that).
The bundle's own README
VAGI Behavior Suite v1 — paid model baseline (2026-11-18)#
What this is. A raw frontier-model baseline on the published VAGI Behavior Suite. The model receives the task data and the same policy card the library arm codifies. It gets **no tools, no memory, no retries**, and one attempt per task. Invalid JSON counts as incorrect. This is an external baseline for the suite, not a CAIN arm and not evidence of AGI/VAGI.
Results#
| Arm | Model | Tasks | Accepted | Rate | 95% Wilson | Cost |
haiku-5-5-full | Claude Haiku 5.5 | 500 | 481 | 96.2% | [0.9414, 0.9755] | $0.0475 |
fable-5-2-stratified | Claude Fable 5.2 | 100 | 96 | 96.0% | [0.9016, 0.9843] | $0.5819 |
Published context on the same 500 held-out tasks: no-procedure floor 72/500, one fixed procedure 112/500, hand-written library 500/500, governed 500/500. Distribution shift (100 unfamiliar variants): library 5/100.
The honest finding#
The raw model scores about 96% in-distribution. The hand-written library scores 100%. Therefore the 500/500 is a procedure ceiling, not a frontier-model discriminator. The interesting number is the 5/100 distribution-shift result, where the library collapses. The model arm did not run the shift probe, so this bundle does not show whether a frontier model transfers better. It shows only that the in-distribution suite is easy for a frontier model when the policy is supplied.
Per domain#
| Domain | Haiku 5.5 | Fable 5.2 (10/domain) |
| business_decision | 35/50 | 10/10 |
| data_analysis | 48/50 | 9/10 |
| governance | 50/50 | 10/10 |
| memory_retrieval | 50/50 | 10/10 |
| planning | 50/50 | 10/10 |
| recovery | 50/50 | 10/10 |
| science_calculation | 48/50 | 10/10 |
| security_triage | 50/50 | 10/10 |
| software_repair | 50/50 | 7/10 |
| tool_orchestration | 50/50 | 10/10 |
Cost#
Total spent: $0.629442 of the $1.91 budget (remaining $1.280558). Prices: Haiku 5.5 $0.10/$0.50 per MTok; Fable 5.2 $8/$40 per MTok.
Reproduce#
python3 -I verify_offline.py .
The verifier regenerates the 500 held-out tasks, re-scores every raw answer with the published suite checker, recomputes the counts, intervals and cost, and checks that the prompt view contains no fixes, hidden, bad or reference answer. It imports the suite source bundled here, not CAIN. This is offline recomputation with the published checker, not an independent reimplementation and not a third-party audit.
Limits#
- Constructed, deterministic suite; not a public benchmark.
- Policy card supplied; this measures procedure application, not open-ended reasoning.
- No tools, no memory, no retries.
- One attempt per task; invalid JSON = incorrect.
- No independent verifier or witness.
- Not AGI, not VAGI, not production, not a certification.
Try CAIN-42 on your own agents
Create a free account and every new account starts with a 30-day trial of the full platform. Or try the sandbox first, with no account at all.
Create a free account → · Try the sandbox · See the whole ecosystem