Circuit Breaker Pattern
A retry loop makes a struggling service worse. A circuit breaker is the piece of code that decides to stop calling — and knows how to start again.
Core concept
When a downstream dependency gets slow or starts erroring, the naive caller keeps sending requests and keeps waiting on timeouts. Each waiting request holds a thread, a connection, a chunk of memory. The dependency's problem becomes your outage — that is a cascading failure, and it is how one bad database takes down five healthy services.
A circuit breaker wraps the outbound call in a small state machine that watches the failure rate. Past a threshold it trips open and every subsequent call fails immediately, without touching the network. Two things are bought at once: the caller stops burning resources on calls that were going to fail, and the dependency gets an unloaded window in which to actually recover.
The subtle part is not opening. It is closing. After a cooldown the breaker allows a few trial requests through — half-open — and lets reality decide. Success closes it, failure re-opens it. Recovery is measured, never assumed.
The three states
Requests flow through. Outcomes are recorded in a rolling window; failures and timeouts are counted.
No call is attempted. The breaker returns an error or a fallback in microseconds. The dependency sees zero load from you.
A small, capped number of trial requests are let through. Everything else still fails fast while the probe runs.
Key components
1 · Rolling failure window
A bounded, recent view of outcomes — sliding count or sliding time window. Recent-only is the point: yesterday's failures must not trip today's breaker, and a lifetime average never trips at all.
2 · Trip condition
A rate plus a minimum volume. 50% of ≥20 calls is sane; 50% of 2 calls trips on a coin flip. Without the volume floor, low-traffic endpoints flap constantly.
3 · Reset timeout & probe
How long to stay open, then how many trial calls to admit. Often exponential: each failed probe lengthens the next wait, so a dependency that is genuinely down is not hammered every few seconds.
4 · Fallback path
What the caller returns while open — cached value, sane default, degraded feature, queued write, or an honest error. A breaker with no fallback converts a slow failure into a fast one; useful, but only half the work.
How it works, step by step
- Wrap the call, not the service.One breaker per dependency and operation. A shared breaker across unrelated endpoints means a broken report query blocks healthy logins.
- Record every outcome.Success, error, and timeout all land in the rolling window. Timeouts matter most — a call that never returns is the one that eats your threads.
- Evaluate the trip condition.Once the window holds enough samples, compare the failure rate to the threshold. Over it, transition to OPEN and stamp the time.
- Fail fast while open.Every call returns instantly with an error or a fallback. Zero network work, zero held threads — this is where the cascade stops.
- Probe after the timeout.Move to HALF-OPEN and admit a handful of real requests. Do not flood: the dependency may still be fragile, and a thundering herd of retries re-breaks whatever just recovered.
- Decide from evidence.Probes succeed → close the breaker and clear the window. A probe fails → re-open and back off further. Then keep watching; the state machine never stops running.
Knobs you will actually tune
Trip percentage — commonly 50%. Lower is twitchier, higher tolerates more damage.
Samples required before any trip. Guards low-traffic endpoints from statistical noise.
Cooldown before probing — seconds, not minutes, and ideally backing off on repeat failures.
Concurrent probes allowed. Small: 1–5. This is your controlled experiment, not your traffic.
Where you meet it in the wild
- Service meshes. Envoy and Istio implement outlier detection and breaking at the sidecar, so services get the behaviour without library code.
- Payment & third-party APIs. When a provider degrades, checkout fails fast onto a backup processor instead of holding customers on a spinner.
- Database & cache clients. Connection-pool exhaustion is the classic cascade; a breaker in front of the pool keeps the app responsive while the store recovers.
- Client SDKs and mobile apps. Breakers stop a flaky network from draining battery and quota through infinite retry loops.
The mistake almost everyone makes first
Retry and breaker fight each other if you stack them carelessly
A retry policy of 3 attempts sitting inside the breaker triples the load you send to a struggling dependency and triples the time each thread is held — while reporting only one outcome to the window. Put the retry outside, or bound total attempts, and always count timeouts as failures.
Second trap: no observability. A breaker that trips silently turns a loud outage into a mysterious one. Emit state transitions as events and alert on them — "breaker opened" is one of the highest-signal alerts a distributed system can produce.
Checkpoint — answer before you move on
- Why does the HALF-OPEN state exist at all? What specifically goes wrong if a breaker jumps straight from OPEN back to CLOSED when the reset timeout expires? Hint: think about what a burst of full production traffic does to a service that has only just come back.
- A breaker on a low-traffic admin endpoint keeps flapping open and closed. The threshold is "50% failures." What is the missing parameter, and why does it fix the flapping? Hint: 50% of how many calls? One failure out of two is 50%.
- A circuit breaker prevents cascading failure — but what does it not do, and what must you add so the caller degrades gracefully rather than just failing sooner? Hint: fast failure is still failure. What does the user see?