CAIN-42 evidence library
cain-mr-03 view-184 orphan — 2026-11-18
Evidence bundle: 0 claims, 0 of 0 invariants held.
Last reviewed 2026-11-19
The bundle's own README
cain-mr-03 view-184 orphan — 2026-11-18#
Evidence class: SELF_OBSERVED_SAME_ADMIN_DOMAIN — this is not a third-party audit or a certification. The 48-hour soak is RUNNING, not passed.
This bundle records one operational incident on the CAIN-42 four-replica PBFT cluster cain-mr-03, the fix, the rolling deployment, and the start of a fresh 48-hour soak.
What happened#
The homepage status bar (pinned to cain-mr-03) reported DEGRADED from about 00:45Z. All four replicas were reachable and held the same application state, but one replica was in view 184 while the other three committed in view 183.
Root cause chain:
1. The cluster autoheal's PBFT-alive guard only read the mr-01 and mr-02 deploy records. It could not recognise a cain-mr3 node, so pbft_alive("cain-mr3-node-1") always returned None. 2. At 00:06:27Z the autoheal restarted cain-mr3-node-1 on Docker health alone; the sentinel started the same container again four seconds later. 3. On recovery the replica installed view 184 while the other three stayed in 183. It re-broadcast a NEW_VIEW covering sequence 132613 while the cluster had committed 143k+, so the peers could never install it. It stayed orphaned and lagged. 4. Safety held: identical application_state_hash on all four, no fork, no quarantine, quorum 3/4 kept committing. The defect was liveness/view churn, not a safety breach.
What was fixed#
| Fix | File | Status |
Autoheal reads every cain_mr*_deploy_record.json, so pbft_alive() protects mr-3 (and future clusters) | scripts/ops/cain_mr_autoheal.py | live on atl, lax, mia, sjc |
Sentinel never starts a container Docker reports restarting/created/paused or health starting | deploy/scripts/cain_sentinel_guardian.py | live |
| PBFT engine steps down from an unadopted stale view instead of re-broadcasting it forever | platform-gateway/cain_pbft_engine_33.py | live in image mr-2478e9c |
Deploy tool can roll cain-mr-03 (mr03 topology) | scripts/cain42_bft/deploy_multi_region.py | committed |
| Tests: autoheal record coverage/mr-3 recognition; stale-NEW_VIEW step-down | tests/ | 12/12, 46/46, 345/345 |
The engine fix was deployed as hotfix commit 2478e9cbf (parent e8df8998f), one file, +30/−6. The image cain-cluster-node:mr-2478e9c was derived from the running mr-e8df8998f image plus that one patched file. **The image is a commit derivative, not a clean reproducible rebuild**; a verifier can inspect /app/cain_pbft_engine_33.py but cannot rebuild the image from the commit alone.
Timeline#
See [timeline.md](timeline.md). Key points:
- 03:23Z node-1 restarted and rejoined view 183; 03:41Z 4/4 HEALTHY.
- 04:31–04:40Z rolling deploy of
mr-2478e9c(node-1 139 s, node-3 43 s, node-4 32 s,
primary node-2 last 42 s); 4/4 HEALTHY at view 190.
- 04:44Z soak
soak-multiregion-48h-2026-11-18started; checkpoint-0000 VALID (48
certificates verified).
- 05:03:37Z first injected fault: node-1 killed, restarted 05:04:34Z; 5 expected check
errors, 0 divergences**; cluster returned to HEALTHY. A later view-change window (05:09–05:10Z) produced 4 consecutive ReadTimeout check errors (max streak 4, limit 5); the cluster returned to HEALTHY at view 207. See status/post-first-fault-observation.md.
Verify this bundle#
python3 verify_bundle.py # checks every file against MANIFEST.json
The soak evidence is also published separately and continuously:
- https://clawx.click/evidence/soak-multiregion-48h-2026-11-18/
- https://cainstudio.online/proof/bundle/soak-multiregion-48h-2026-11-18/
curl -so verify_soak.py https://clawx.click/evidence/soak-multiregion-48h-2026-11-18/verify_soak.py.txt python3 verify_soak.py https://clawx.click/evidence/soak-multiregion-48h-2026-11-18/
The published verifier prints VERDICT: RUNNING until the final 48-hour checkpoint lands. A RUNNING verdict is not a passed soak.
Files#
| Path | Contents |
incident.md | Full incident/fix/deploy/soak note |
runbook.md | The exact deploy and soak runbook used |
timeline.md | Dated timeline from the recovery, deploy and soak logs |
tests.md | Test commands and results |
deployment.json | Deploy record (commit, image, image id, nodes) |
status/ | Pre-recovery, post-recovery, pre-deploy, post-deploy, post-first-fault status |
logs/ | node-1 pre-restart status/log, WAL rows, recovery actions, rolling deploy log, stabilization (renice/ionice) |
soak/ | Snapshot of checkpoint-0000, heartbeat, index, membership, verifier, first-fault log |
Honest limits#
- No third-party audit, no certification, no independent witness.
- The previous 48-hour attempt stopped at 1.636 h on 2026-11-16; no 48-hour run has completed.
- This run is RUNNING; the pass decision comes only from the final signed checkpoint and the
published verifier (~2026-11-20T04:43Z).
- The cluster experienced view churn after the deploy (host load ~38 on 8 CPUs); the monitor
caught a transient DEGRADED at 05:00Z, and the cluster returned to HEALTHY at view 207. Four consecutive ReadTimeout check errors at 05:09–05:10Z (max streak 4, limit 5) are disclosed, not hidden; view changes are not a pass criterion by themselves.
- The autoheal can shorten a deliberately killed replica; the first fault lasted ~57 s and the
soak counted the five expected check errors. If strict 60 s fault windows are required, pause the autoheal during a soak or add a soak marker.
Try CAIN-42 on your own agents
Create a free account and every new account starts with a 30-day trial of the full platform. Or try the sandbox first, with no account at all.
Create a free account → · Try the sandbox · See the whole ecosystem