Problem
Every backend I have operated eventually calls something that fails: a database that stalls, a payment gateway that times out, a queue that stops acknowledging. The failure itself is rarely the incident. The incident is what my code does next. Hammer a dead dependency with unbounded retries and I turn one outage into two — mine plus theirs. Give up on the first error and every transient blip becomes a user-facing failure. I built this lab because I kept explaining that tension with whiteboard arrows, and arrows never changed anyone's mind. Watching a burst of six calls burn twelve attempts into a dead dependency, then seeing the next three fail in zero attempts because the breaker opened — that changes minds.
Context
The lab is a single Go binary with SQLite, running at https://failure-lab.xmuu.my.id. One side is a fake dependency with three behaviors I control from the UI: healthy, flaky (fail the next N calls, then recover on its own), and down. Each attempt carries a configurable simulated latency. The other side is a guarded client: up to 5 attempts per call with exponential backoff (base × 2^(n−1), capped at 500 ms), fronted by a circuit breaker that opens after K consecutive failed calls, waits out a cooldown, then lets exactly one probe through in half-open state. Every attempt and every breaker transition is journaled to SQLite and rendered newest-first in the event log, so the whole story is inspectable — nothing happens off-screen. Calls in the demo run sequentially under one mutex, a deliberate simplification so the journal reads as a single deterministic narrative rather than interleaved threads.
Alternatives Considered
Retry forever with a fixed delay. This is the default shape of naive resilience code: catch the error, sleep one second, try again, forever. I rejected it because it has no theory of when to stop. Against a dependency that recovers in three calls it works, but against a real outage it converts my client into a load generator aimed at a system that is already down — and it never signals anything upstream, so no dashboard, autoscaler, or human ever learns the dependency is gone.
Fail fast on the first error, no retries at all. This is honest and cheap, and for non-idempotent operations it is sometimes the only safe choice. But it treats every failure as permanent. I measured the cost in the lab: with retries disabled, a flaky dependency set to fail 3 calls turns the first 3 calls of any burst into user-visible errors, even though the dependency heals by itself seconds later. Transients are the common case in real networks; refusing to absorb any of them pushes the failure budget onto users for no reason.
Exponential backoff with jitter, but no breaker. This is strictly better than fixed-delay retry — the growing gaps (25 ms, 50 ms, 100 ms in the lab's defaults) give a struggling dependency room to breathe, and jitter stops synchronized clients from retrying in lockstep. I use it, but it is not sufficient: backoff slows the bleeding without ever stopping it. Twelve attempts across three failed calls still hit a dead dependency in my measurements, at 0.607 s of pure waste per six-call burst. Backoff answers "how patiently should I retry", and a different mechanism has to answer "when do I stop calling entirely". That mechanism is the breaker, so the lab implements backoff inside the breaker rather than instead of it.
Solution
I built the client as two nested loops with opposite jobs. The inner loop retries: one guarded call makes up to max_attempts hits against the fake dependency, sleeping base × 2^(n−1) between them, and each attempt lands in the journal with its verdict and the backoff that followed. The outer loop is the breaker, and its gate is the whole design in miniature:
if l.breaker == BreakerOpen {
if time.Since(l.opened) < cooldown {
// Fast-fail: the dependency is never touched.
return CallResult{CallID: id, Outcome: OutcomeBreakerOpen}
}
l.breaker = BreakerHalfOpen // cooldown elapsed: one probe goes through
}A closed breaker counts consecutive failed calls and opens at the threshold. An open breaker fails calls in zero attempts — no latency sleep, no dependency hit, just a journal line. A half-open breaker allows exactly one probe with retries disabled: one question, not a campaign. Success closes it and resets the streak; failure re-opens it with a fresh cooldown. Try it in order: inject flaky with fail-count 3 and run a burst of 6 — I measured 6/6 success with attempts of [4, 1, 1, 1, 1, 1] in 0.237 s, the first call absorbing the whole fault. Then inject down and burst again: 3 failed calls at 4 attempts each, then the breaker opens and the remaining 3 calls fast-fail with 0 attempts. Then inject healthy, wait out the 2 s cooldown, and run one call: a single 5 ms probe succeeds and the breaker closes. The journal tells the same story line by line, from breaker CLOSED → OPEN: 3 consecutive failed calls down to breaker HALF-OPEN → CLOSED: probe call#13 succeeded.
Trade-offs
The sequential-execution mutex is the biggest simplification, and I would not ship it: real clients are concurrent, and under concurrency the half-open probe needs a real single-flight lock or ten threads will all "probe" at once. Here it buys a deterministic journal, which is the entire point of a teaching demo, so I kept it and documented it. The sleeps are real time.Sleep calls rather than a virtual clock, which keeps the demo honest (the 0.237 s flaky burst is genuinely 175 ms of backoff plus latencies) but means the UI genuinely blocks during a burst — inputs are clamped to keep that bounded: latency at most 2000 ms, backoff base at most 500 ms, bursts at most 20 calls. The probe performs exactly one attempt with no retry, which is the conservative choice: a healing dependency answers the first knock, and a still-dead one should re-open the breaker as fast as possible rather than burn a retry budget nobody asked for. The journal lives in SQLite capped at the 100 most recent events per view, so a long session cannot grow memory, but it also means very old history scrolls away — acceptable for disposable demo data, wrong for audit. Finally, the lab retries blindly, which is only safe because the fake dependency has no side effects; a real write path would need idempotency keys before any of this applies.
Production Notes
What ships differently in production starts with observability: in the lab the breaker transitions are journal lines, but in a real service each one must be a metric — trips counted, fast_failed rate alerted, consecutive-failure streak graphed per dependency. A breaker that never trips is either a healthy dependency or a threshold set so high it only opens after the incident is over; I tune the threshold from the streak histogram, not from intuition. The cooldown needs jitter and usually backoff of its own, because a fleet of clients with identical 2 s cooldowns will probe in lockstep and re-open in lockstep — the thundering herd the single demo client never shows. Half-open probing needs a concurrency limit of one in flight, distinct from the demo's global mutex. Backoff needs jitter added to the doubling (the lab omits it for readability of the numbers) and a total deadline per call, since attempt budgets compose badly with upstream timeouts: 5 attempts with a 500 ms cap can exceed a 1 s gateway timeout and turn a contained failure into a cascading one. The failure mode I watch most is the breaker masking a partial outage — if only some dependency hosts are down, a single global breaker flaps while per-host breakers would isolate the damage. And the oldest lesson applies unchanged: none of this retries safely unless the operation is idempotent, so the retry budget and the idempotency design get reviewed together, in the same document, or not at all.