CAIN-42 MCPGateCAIN-42 four-server cluster: any server can fail
Whole-server loss tests on the live CAIN-42 cluster cain-mr-02: 4 PBFT replicas on 4 servers in 4 regions (Atlanta, Los Angeles, Miami, Silicon Valley) over an encrypted WireGuard mesh, one replica per server. Each server in turn was taken completely offline, then two at once. This page loads every quorum certificate all four replicas hold afterwards, and your browser re-checks them.
What happened
Placement: cain-mr2-node-1 in atl, cain-mr2-node-2 in lax, cain-mr2-node-3 in mia, cain-mr2-node-4 in sjc. That is one replica per server, so any single server can fail.
| servers stopped | replicas stopped | committed | expected | leader was on it | result |
|---|---|---|---|---|---|
| atl | cain-mr2-node-1 | 6/6 | COMMIT | yes | PASS |
| lax | cain-mr2-node-2 | 6/6 | COMMIT | yes | PASS |
| mia | cain-mr2-node-3 | 6/6 | COMMIT | yes | PASS |
| sjc | cain-mr2-node-4 | 6/6 | COMMIT | yes | PASS |
| atl, lax | 0/3 | NO_COMMIT (fail closed) | PASS |
verify_host_loss_bundle.py proves the schedule from the signatures. No decision taken while a server was down carries a signature from its replica, and none of the requests refused with two servers down ever entered any decision chain.
Verify (about 5 seconds)
What is checked
- The membership configuration hash is recomputed from the 4 member ids and Ed25519 public keys. Every node and every certificate must carry it.
- For every sequence on every node, the COMMIT_QC and the PREPARE_QC. Each vote must be an Ed25519 signature by a distinct member over SHA-256 of the canonical signed message. It must have the right type (a COMMIT vote never counts as a PREPARE vote) and match this cluster, epoch, view, sequence and digest. Each certificate needs at least 3 distinct signers. The leader's proposal must be signed by the primary of that view, and its digest must bind the proposed operation.
- The certificate hash and signature-bundle hash are recomputed from the content.
- Evolution 3 fast path: a
FAST_COMMIT_QC(a decision taken without the COMMIT round) is accepted only if the published membership declares the fast path and all four members signed it. Three of four is never enough for a fast commit. - The decision chain is folded from genesis:
decision_hash(seq) = H(cluster, epoch, seq, digest, parent). It must be contiguous and identical on all four nodes, and the nodes must end with the same application-state hash. - View-change quorum certificates, and one consensus-to-enforcement AuthorizationCertificate per node.
- Negative controls: a certificate is tampered with in seven ways, and every tampered copy must be rejected.
The same checks, without a browser: curl -so verify_pbft_qc_bundle.py verify_pbft_qc_bundle.py.txt && python3 verify_pbft_qc_bundle.py PBFT_QC_BUNDLE.json (needs pip install cryptography, no CAIN code).