CAIN-42 evidence library
VAGI distribution-shift probe — paid model baseline (2026-11-18)
Evidence bundle: 0 claims, 0 of 0 invariants held.
Last reviewed 2026-11-19
Verify it in your browser
Your browser downloads the bundle's SHA256SUMS manifest and the file(s) behind this page, hashes them with SHA-256 locally (WebCrypto), and compares. A match shows the record you are reading is the published one; it does not by itself prove who published it (see the signed claims registry and the bundle verifier for that).
The bundle's own README
VAGI distribution-shift probe — paid model baseline (2026-11-18)#
The result that matters#
The hand-written library scores 5/100 on the 100-task distribution-shift probe. Three raw frontier models, given the probe's data and the intended policy card, score far higher:
| Arm | Model | Scored | Accepted | Rate | 95% Wilson | Cost |
| haiku-shift | Claude Haiku 5.5 | 100 | 74 | 74.0% | [0.6463, 0.816] | $0.0149 |
| sonnet-shift | Claude Sonnet 5.5 | 100 | 81 | 81.0% | [0.7222, 0.8749] | $0.2603 |
| fable-shift | Claude Fable 5.2 | 65 | 54 | 83.1% | [0.7218, 0.9028] | $0.7511 |
| fable-shift-replication | Claude Fable 5.2 (seed 9503) | 23 | 20 | 87.0% | [0.6787, 0.9546] | $0.2859 |
The 5/100 is substantially a library-coverage gap, not a general-intelligence gap. The probe is adversarial and partly underspecified: the software hidden target (x % 3) is not inferable from the public tests, and all models scored 0 on tool_orchestration. That is a probe/prompt problem, not a general model failure. The result does not show VAGI; it shows the suite's shift probe is a coverage map, not an AGI test.
Cost and the cap#
Shift spend: $1.312174 of the $1.280558 remaining. Overshoot: $0.031616 because concurrent requests passed the budget check together. The Fable arm stopped at 65/100; the replication stopped at 23/100. Rates are on scored tasks only.
Per-domain#
| Domain | Haiku | Sonnet | Fable (scored) | Library |
| business_decision | 10/10 | 10/10 | 7/7 | 0/10 |
| data_analysis | 10/10 | 10/10 | 7/7 | 0/10 |
| governance | 10/10 | 10/10 | 6/6 | 0/10 |
| memory_retrieval | 10/10 | 10/10 | 6/6 | 0/10 |
| planning | 7/10 | 6/10 | 6/6 | 0/10 |
| recovery | 10/10 | 10/10 | 6/6 | 0/10 |
| science_calculation | 10/10 | 10/10 | 7/7 | 0/10 |
| security_triage | 2/10 | 10/10 | 7/7 | 0/10 |
| software_repair | 5/10 | 5/10 | 2/7 | 5/10 |
| tool_orchestration | 0/10 | 0/10 | 0/6 | 0/10 |
Verify#
python3 -I verify_offline.py .
Recomputes counts and cost from raw rows, regenerates the shift tasks for the two seeds, re-scores every answer with the published shift checker, and checks that the prompt view contains no expect, hidden, fixes or bad field. It imports the suite source bundled here, not CAIN.
Limits#
- Fable arms are partial (budget cap). Rates are on scored tasks, not 100.
- The probe is deliberately adversarial; it is not a fair AGI benchmark.
- Software target not inferable; tool task failed for every model.
- No tools, no memory, one attempt; no external reviewer.
- Not AGI, not VAGI, not production, not a certification.
Try CAIN-42 on your own agents
Create a free account and every new account starts with a 30-day trial of the full platform. Or try the sandbox first, with no account at all.
Create a free account → · Try the sandbox · See the whole ecosystem