Fintech
Rebuilding a payments ledger without stopping payments
A single-table balance model was drifting under partial refunds. We moved it to double-entry behind a shadow-write, then cut over with zero downtime.
The problem
Balances were stored as a single mutable column. Every write path — disbursal, repayment, fee, refund, reversal — updated it directly. It worked until partial refunds arrived, at which point two code paths could both “correct” the same balance and the number would quietly land somewhere plausible but wrong.
Nobody noticed for four months. It surfaced during a reconciliation against the bank statement, where roughly one account in nine thousand disagreed by a small amount.
What we did
We introduced an append-only ledger_entries table with strict double-entry semantics: every transaction writes balanced debit and credit rows or the whole thing rolls back. Balances became a derived value.
The cutover ran in three stages:
- Shadow write. Every existing write path also emitted ledger entries. Nothing read from them yet. This ran for six weeks.
- Reconcile. A nightly job compared the derived balance against the stored column and reported every divergence. This is where the eleven bugs surfaced — each one a real defect that had been invisible.
- Cut over. Reads moved to the derived balance one endpoint at a time, with the old column kept as a fallback for another month before being dropped.
What we would do differently
The shadow-write period could have been shorter. We ran six weeks because that covered a full billing cycle plus a buffer, but the divergence curve flattened after eleven days. Two weeks would have been enough evidence.
The result
Reconciliation against the bank statement now runs clean. More usefully, every balance can be reconstructed from its history, so a disputed number is a query rather than an investigation.