IRL Scenarios: Seven Payments Failures and Their Fixes
Seven real-world failure scenarios. Each opens on a symptom, walks the diagnosis, and lands on the Arc mechanism that prevents it from happening again.
IRL Scenarios · seven of themArchitecture documentation explains what a system does. It is much worse at explaining why anyone bothered, because the reason is always a failure, usually one somebody else already had, expensively.These are that layer: the things that actually go wrong in payments infrastructure, written as the scenarios they are. Each one opens on a symptom, walks the diagnosis the way it would really go, and ends at the specific mechanism in Arc that prevents it. They are self-contained and readable in any order.
Two of these describe things that genuinely happened while building Arc: a mutation that passed sixteen tests, and a boundary rule that silently did not apply. Those are marked. The rest are the standard failures of this domain, written as scenarios because that is how they are actually learned.
Compensation ran in the wrong order. The ledger balanced perfectly. All sixteen tests passed. The audit trail was fiction.The lesson: balance is necessary but not sufficient. This is the best argument on the site for why correctness needs more than one kind of assertion, and it is the scenario most worth retelling in an interview.
The trial balance is off by €0.03. Nobody can find it, because it was never in one place. It was 0.00995 of a cent, three hundred times, rounded away into nothing. Mechanism: residuals posted to a rounding account.
The journal that balanced and lied
A balanced ledger with a false audit trail, and a mutation test that caught what sixteen assertions did not. Mechanism: reverse-order compensation, asserted directly.
The payout that paid twice
The rail timed out. The retry succeeded. So did the original. Mechanism: idempotency keys, and retryable as a field rather than a guess.
The boundary that wasn't
The architecture rule was in CI, green on every build, and had been silently skipping half the violations it existed to catch. Mechanism: two independent enforcement layers, both probed.
The fastest chain that settled last
Routing to the chain with the shortest block time made settlement twelve times slower. Solana, Base and Polygon all behave differently, and only one of those differences is visible in a block-time column. Mechanism: finality depth, not block time, as the routing input.
The cut-off that cost a day
The payment was submitted at 16:00. It arrived Tuesday. Missing a batch window by an hour costs a day, not an hour. Mechanism: cut-off arithmetic that compounds, asserted exactly.
The quote that aged
A customer held a quote open for four minutes while the rate moved. Mechanism: a 30-second TTL, and a property test that the customer never beats mid-market.
A mechanism explained is forgettable.“Rails are idempotent on the caller’s key” is a true sentence that will not survive the week.
A mechanism that solved a specific disaster is not.“The rail timed out, we retried, and the beneficiary got paid twice” is remembered, and the mechanism comes with it.
Every scenario ends with the code that prevents it and a link to the architecture page that specifies it, so the narrative is an entry point rather than a substitute.
Prefer the specification?
The architecture tab covers the same mechanisms without the narrative framing.
