CAIN-42 MCPGateAGENTIC AI 75 · #54
Deployment-driven agent benchmarks
TraceDance converted real deployment traces into targeted behaviour tests.
Last reviewed 2026-11-11
Agentic AI 75 / #54
2026 status: Reported research system · Category: Evaluation
The 2026 development
TraceDance converted real deployment traces into targeted behaviour tests.
The governance gap
A benchmark built from production traces contains production data and production failures. Using it safely requires consent, redaction, tenant separation and a way to tell whether a failure is still reproducible.
Where CAIN-42 fits: Moat 6 — Adversarial Trust Engine
Core question: Can CAIN break its own guarantees? Continuous attack, mutation and red-team testing of the controls themselves.
Convert signed production records into a redacted, frozen test suite. The suite is an adversarial asset: the agent being tested must not be able to edit it, and every result names the trace it came from.
How the wiring works
- Select traces with consent and tenant approval.
- Redact secrets and personal data before the trace becomes a test.
- Freeze the suite and hash it.
- Re-run the frozen suite after every change and record the delta.
What is real today
LIVE A public route in this deployment exercises this mechanism today, within its stated scope.
The hosted release gate evaluates fixed agent behaviour suites. It is not TraceDance and does not use customer traces.
| Component in this repository | What the module says it is |
|---|---|
keystone/agentci.py | AGENT-CI KEYSTONE: a release gate for AI agents that grades the path, not only the answer. |
keystone/evalrun.py | Signed evaluation runs: "these scores came from this code, this data, these model providers, unmodified". |
cain/runtime/incident.py | CAIN Sovereign — Autonomous Agent Incident Response (Prompt 7, Phase 21). |
keystone/heldout.py | Held-out evaluation vault + contamination certificate, run in the CAIN agent jail, signed. |
Descriptions are the first line of each module's own docstring, read without importing it. A module path is a pointer into this repository, not a hosted endpoint. Its existence is checked by the test suite; its status for this use is the label above.
Check it yourself
- Run the live agent release gate
/agent-ci-keystone - Fire a live decision through the CAIN-42 pipeline
/fabric/try— the same enforcement path every product sits behind.
Commercial package
| Offer | Deployment Trace Evaluation |
|---|---|
| Buyer | Agent-platform teams and QA organisations |
| Price (proposal) | $3,500–$7,500 per audit |
| Bundle | Code & Proof — Prove the fix, not just the diff. |
| Operator stream | Stream 10 of the 10-stream operator playbook |
Platform engineering, AI coding-tool vendors, security engineering is the bundle buyer. The entry offer for this bundle is Fabrication audit or long-run engineering attestation; the range is $1,500–$6,000 per audit; $2,500–$4,000/month retainer. A price on this page is a proposal, not a validated rate, and no customer is claimed.
Buy this as a product or service
CAIN-A75-054 — Deployment Trace Evaluation PILOT
A bounded pilot report: up to five scenarios, the relevant library engine or analysis, a written result and the stated limits. Engineering time is quoted per pilot.
Kind: Evaluation · Price (proposal): $3,500–$7,500 per audit · Lead time: 10 business days.
Open the product page · all 75 products
What this page does not claim
What this page is not. CAIN-42 is not affiliated with, endorsed by, integrated into, or shipped with the organisation named in the 2026 item. The item is public context; the page describes how CAIN-42 would govern that class of capability. No page here claims a customer, a partnership, a certification, a patent, or a production integration. Prices are proposals. Built by a three-person team.