CAIN-42 multi-region PBFT cluster -- fault-injection evidence: multi-region-cluster-2026-09-26
========================================================================

What: Every quorum certificate the live 4-replica CAIN-42 PBFT cluster cain-mr-01 produced during a fault-injection run across three Vultr regions (Atlanta, Los Angeles x2, Miami) connected by a WireGuard overlay: baseline, each remote region stopped in turn, the two-replica host stopped (must fail closed), the Atlanta region with the primary killed (view change), and full recovery.
Classification: MEASURED on the live multi-region cluster cain-mr-01 (3 Vultr regions, 3 hosts, 4 replicas)
Generated: 2026-09-26T18:27:22Z
Source commitment: git e1cf69c9c394f685e16a9631f38683eef8ec1a05, image sha256:95d0ddfb2401806daac805c03f7d0b87e34be908252b65f8b4700377c9262525 (engine 33.4-production-hardened); source is not published.

Topology: n=4 f=1 quorum=3; placement {"cain-mr-node-1": "atl", "cain-mr-node-2": "lax", "cain-mr-node-3": "lax", "cain-mr-node-4": "mia"}
Tolerates: any 1 Byzantine replica; loss of the atl host; loss of the mia host
Does not tolerate: loss of the lax host (2 replicas); loss of 2 replicas anywhere

Fault schedule (real `docker stop` of replicas on their own hosts):
  A_baseline                         20/20 committed, expected COMMIT                   PASS
  B_mia_down                         6/6 committed, expected COMMIT                   PASS
  C_one_lax_replica_down             6/6 committed, expected COMMIT                   PASS
  D_lax_host_down                    0/3 committed, expected NO_COMMIT (fail closed)  PASS
  E_atl_region_and_primary_down      6/6 committed, expected COMMIT                   PASS
  F_all_back                         3/3 committed, expected COMMIT                   PASS

Baseline commit latency (client -> committed ALLOW, sequential, n=20): p50 501.74 ms, p95 796.63 ms, p99 1480.35 ms, max 1480.35 ms
Method: client on the atl host (over WireGuard), HTTP POST /api/v1/cluster/pbft/request to the entry replica, wall time until the committed ALLOW response; sequential requests, concurrency 1; atl host is a 2 vCPU / 3.4 GB VM also running other workloads

Verify in under a minute (Python 3.8+, `pip install cryptography`, no CAIN code):

  BASE=https://cainstudio.online/proof/bundle/multi-region-cluster-2026-09-26      # or https://mcpgate.online/proof/bundle/multi-region-cluster-2026-09-26
                                                            # or https://clawx.click/evidence/multi-region-cluster-2026-09-26
  curl -so verify_pbft_qc_bundle.py     "$BASE/verify_pbft_qc_bundle.py.txt"
  curl -so verify_multi_region_bundle.py "$BASE/verify_multi_region_bundle.py.txt"
  curl -so MULTI_REGION_BUNDLE.json     "$BASE/PBFT_QC_BUNDLE.json"
  sha256sum MULTI_REGION_BUNDLE.json verify_pbft_qc_bundle.py verify_multi_region_bundle.py
    # expect 393d88099d4d62686abff3bf0da841808741717a0978626b04d34e678f35fd4b  MULTI_REGION_BUNDLE.json
    # expect ba452ce42528f11f7f5235727a65cae25a34072bc52b4711fd72581615d06009  verify_pbft_qc_bundle.py
    # expect 7e5815cbf62c9cf7bbd3adfefefbbed8fb44ceb4b0cd149f8c7ac4f2cbf2b167  verify_multi_region_bundle.py
  python3 verify_multi_region_bundle.py MULTI_REGION_BUNDLE.json

Or open $BASE/index.html -- your browser runs the certificate checks and the fault-schedule checks.
The live cluster itself: /live-cluster.html on any of the three sites (read-only, verified in your browser).

Not claimed: tolerance of losing the Los Angeles host: it holds 2 of 4 replicas, so losing it halts progress (safety kept); independent providers: all three hosts are Vultr, one account, same image; region placement is operator-attested (host facts below), not cryptographically proven; hardware attestation (no TPM/TEE on these VMs); third-party review; absence of bugs.
Redacted: host public IP addresses (not needed for verification); overlay (private) addresses replaced by region labels.