Hinted Handoff: Keeping Writes Available During Node Failures
Core Concept
When one of the natural replicas for a key is unreachable, a quorum store does not have to fail the write. The node accepting the write durably records a hint — target replica, key, mutation, version metadata, and expiry — then acknowledges the client under the configured consistency rule. The hint is the outstanding debt. When the target recovers, the holder replays the mutation idempotently, and the target resolves it with the database's normal version/conflict rules. Availability rises; replicas are allowed to temporarily diverge, and the system converges later.
Key Components
1. Durable hint record
Target replica, key, value/mutation, version metadata, expiry. In Cassandra it is stored on the coordinator; a Dynamo-style sloppy quorum can tag a write to another healthy node instead.
2. Acknowledgement rule
The client gets success as soon as the write-consistency rule is satisfied (e.g. 2 of 3) — before every natural replica has stored it.
3. Idempotent replay
On recovery the holder re-sends the mutation; the target applies it via normal versioning. The hint is deleted only after the target acknowledges.
4. The backstop
Hints can expire, be lost with their holder, arrive out of order, or pile up in a long outage. Anti-entropy repair and read repair are what actually guarantee convergence.
How It Works
- Key K belongs on replicas A, B, C — and C is down.
- A coordinates the write; A and B persist it. A durably records a hint for C.
- Client receives success under the configured write-consistency rule.
- Any read that reaches stale C before replay can still return the old value, unless a read quorum or reconciliation masks it.
- C returns: A replays the mutation, C applies it with normal version rules, and A deletes the hint after C's acknowledgement.
Kill C, issue timestamped writes, and watch the client's acknowledgement rule hold while A and B hold the data. Restore C and measure the replay lag. Now run the outage longer than the maximum hint window, and fail the hint holder too — the replicas that never received the mutation only converge if repair, not hints, did the work.
Real-World Applications
Note the scope: hinted handoff supports eventual convergence — it does not by itself ensure eventual consistency. Gossip is one way a system may notice recovery, not part of the definition.
Checkpoint — answer before you move on
- In the worked scenario (K on A, B, C; C down), why can the client still receive success — and exactly where is the “debt” recorded? Name the ack rule and the two things that persist before the client hears back.
- Why must the replay be idempotent, and at what moment is the hint deleted — why not at replay time? Think about what happens if the replay succeeds but the ack is lost.
- Hinted handoff does not by itself guarantee eventual consistency. Which mechanism is the backstop — and name one failure where the backstop is the only thing that converges the replicas? Outage longer than the hint window; hint holder dies; both.