CAIN-42 MCPGate

CAIN-42 evidence library

cain-mr-03 view-184 orphan — 2026-11-18

Evidence bundle: 0 claims, 0 of 0 invariants held.

Last reviewed 2026-11-19

The bundle's own README

cain-mr-03 view-184 orphan — 2026-11-18#

Evidence class: SELF_OBSERVED_SAME_ADMIN_DOMAIN — this is not a third-party audit or a certification. The 48-hour soak is RUNNING, not passed.

This bundle records one operational incident on the CAIN-42 four-replica PBFT cluster cain-mr-03, the fix, the rolling deployment, and the start of a fresh 48-hour soak.

What happened#

The homepage status bar (pinned to cain-mr-03) reported DEGRADED from about 00:45Z. All four replicas were reachable and held the same application state, but one replica was in view 184 while the other three committed in view 183.

Root cause chain:

1. The cluster autoheal's PBFT-alive guard only read the mr-01 and mr-02 deploy records. It could not recognise a cain-mr3 node, so pbft_alive("cain-mr3-node-1") always returned None. 2. At 00:06:27Z the autoheal restarted cain-mr3-node-1 on Docker health alone; the sentinel started the same container again four seconds later. 3. On recovery the replica installed view 184 while the other three stayed in 183. It re-broadcast a NEW_VIEW covering sequence 132613 while the cluster had committed 143k+, so the peers could never install it. It stayed orphaned and lagged. 4. Safety held: identical application_state_hash on all four, no fork, no quarantine, quorum 3/4 kept committing. The defect was liveness/view churn, not a safety breach.

What was fixed#

FixFileStatus
Autoheal reads every cain_mr*_deploy_record.json, so pbft_alive() protects mr-3 (and future clusters)scripts/ops/cain_mr_autoheal.pylive on atl, lax, mia, sjc
Sentinel never starts a container Docker reports restarting/created/paused or health startingdeploy/scripts/cain_sentinel_guardian.pylive
PBFT engine steps down from an unadopted stale view instead of re-broadcasting it foreverplatform-gateway/cain_pbft_engine_33.pylive in image mr-2478e9c
Deploy tool can roll cain-mr-03 (mr03 topology)scripts/cain42_bft/deploy_multi_region.pycommitted
Tests: autoheal record coverage/mr-3 recognition; stale-NEW_VIEW step-downtests/12/12, 46/46, 345/345

The engine fix was deployed as hotfix commit 2478e9cbf (parent e8df8998f), one file, +30/−6. The image cain-cluster-node:mr-2478e9c was derived from the running mr-e8df8998f image plus that one patched file. **The image is a commit derivative, not a clean reproducible rebuild**; a verifier can inspect /app/cain_pbft_engine_33.py but cannot rebuild the image from the commit alone.

Timeline#

See [timeline.md](timeline.md). Key points:

primary node-2 last 42 s); 4/4 HEALTHY at view 190.

certificates verified).

errors, 0 divergences**; cluster returned to HEALTHY. A later view-change window (05:09–05:10Z) produced 4 consecutive ReadTimeout check errors (max streak 4, limit 5); the cluster returned to HEALTHY at view 207. See status/post-first-fault-observation.md.

Verify this bundle#

python3 verify_bundle.py          # checks every file against MANIFEST.json

The soak evidence is also published separately and continuously:

curl -so verify_soak.py https://clawx.click/evidence/soak-multiregion-48h-2026-11-18/verify_soak.py.txt
python3 verify_soak.py https://clawx.click/evidence/soak-multiregion-48h-2026-11-18/

The published verifier prints VERDICT: RUNNING until the final 48-hour checkpoint lands. A RUNNING verdict is not a passed soak.

Files#

PathContents
incident.mdFull incident/fix/deploy/soak note
runbook.mdThe exact deploy and soak runbook used
timeline.mdDated timeline from the recovery, deploy and soak logs
tests.mdTest commands and results
deployment.jsonDeploy record (commit, image, image id, nodes)
status/Pre-recovery, post-recovery, pre-deploy, post-deploy, post-first-fault status
logs/node-1 pre-restart status/log, WAL rows, recovery actions, rolling deploy log, stabilization (renice/ionice)
soak/Snapshot of checkpoint-0000, heartbeat, index, membership, verifier, first-fault log

Honest limits#

published verifier (~2026-11-20T04:43Z).

caught a transient DEGRADED at 05:00Z, and the cluster returned to HEALTHY at view 207. Four consecutive ReadTimeout check errors at 05:09–05:10Z (max streak 4, limit 5) are disclosed, not hidden; view changes are not a pass criterion by themselves.

soak counted the five expected check errors. If strict 60 s fault windows are required, pause the autoheal during a soak or add a soak marker.

Try CAIN-42 on your own agents

Create a free account and every new account starts with a 30-day trial of the full platform. Or try the sandbox first, with no account at all.

Create a free account →  ·  Try the sandbox  ·  See the whole ecosystem