Distributed Systems • Sensei Learning System
A System That Is Always Up and Frequently Wrong
Two different questions: did it answer at all — and was the answer right? Conflating them is why incident reviews go in circles.
Availability — did it answer?
8/8 responded · 100% up
Reliability — was it right?
correct
stale row
correct
wrong sum
2/4 correct · silent
Raise availability
Redundancy — more copies answering
Raise reliability
Correctness checks & idempotency
Core Concept
Availability and reliability are two different questions, and a system can
have one without the other in both directions. Availability asks whether the system
answered at all — uptime, the fraction of requests that got a response.
Reliability asks whether the answer was correct and the operation actually did
what it claimed. A cache that cheerfully returns stale rows is highly available and
unreliable; a payment processor that refuses ambiguous transactions rather than risk a
double charge is deliberately trading availability for reliability. Adding a replica
does not make wrong data right.
Key Components
Question 1
Availability
Did it answer at all? Measured as successful responses ÷ total requests — the uptime fraction.
Question 2
Reliability
Was it correct? Measured as mean time between failures — how often the operation fails to do what it claimed.
Remedies
Different Fixes
Redundancy raises availability. Correctness checks and idempotency raise reliability. They do not overlap.
The Trap
“Five Nines”
A 99.999% availability target says nothing about whether those responses were correct.
How It Works
- A request lands on the system. The availability counter increments on any response — correct, stale, or wrong.
- A dependency silently returns a corrupted value. The service wraps it in a 200 OK and sends it on.
- The dashboard stays green: latency is normal, error rate is zero, uptime hits its five-nines SLO.
- Only the downstream users notice — the report totals don’t add up, the checkout charged twice, the map is wrong.
- In the incident review, the team argues about uptime — the number everyone agreed on — while the real defect (correctness) was never measured.
Real-World Applications
- Stale caches: CDN or app cache serving an old price or profile — 100% up, wrong every time it’s stale.
- Payment processors: refusing an ambiguous transaction rather than risk a double charge — deliberately choosing reliability over availability.
- Health checks: a probe that only asks “does the process answer?” is an availability probe wearing a reliability badge.
- “Five nines” SLOs: a system can hit its SLO while silently serving corrupted results, dashboard green throughout.
Checkpoint — answer before you move on
- A read-replica cache serves yesterday’s data on 99.9% of requests. On which question does it score high, and on which does it fail?
Score the system on answering vs. being correct — then name the number each question uses.
- Why does adding a second replica not fix a service that returns wrong prices?
Name the remedy for each question — which one does a replica actually buy?
- Your dependency starts returning a deliberately wrong value. Which alarms should fire to prove you were measuring reliability, not just availability?
If only latency and error-rate move, what were you actually measuring?
The falsification test: return a deliberately wrong value from one dependency and watch the dashboards. If only latency and error-rate move, every “reliability” metric you had was an availability probe wearing a reliability badge.
_hero image unavailable this run (image backend outage) — text card delivered._