Failure Handling: The Four Moves

Subtitle: Four moves that stop one slow service from sinking everything

Quadrant 1 - TIMEOUT:

Cap how long you wait for a reply No timeout means threads pile up Set it under the caller's own budget Failing fast beats hanging silently Tune from p99 latency, not a guess

Quadrant 2 - RETRY WITH BACKOFF:

Retry only safe, idempotent calls Wait longer after each attempt Add random jitter to spread load Cap it at 2-3 tries, then stop Never retry a 400-class error

Quadrant 3 - CIRCUIT BREAKER:

Counts failures, then stops calling CLOSED -> OPEN -> HALF-OPEN -> CLOSED OPEN fails instantly, no waiting HALF-OPEN sends one probe request Gives the sick service room to heal

Quadrant 4 - FALLBACK:

Answer with something useful anyway Serve cached or slightly stale data Return a safe default or empty list Degrade the feature, keep the page Say unavailable, never spin forever

Simple difference:

Timeout = stop waiting Retry = ask again, politely Breaker = stop asking for a while Fallback = still answer something

Use it when - layering them: Timeout inside Retry, Retry inside Breaker, Fallback answers last. Use it when - skipping one: No timeout makes every other move useless.

Sticky note - Common beginner mistake:

Retrying instantly with no backoff or jitter. Every client fires at once and the thundering herd finishes off the service you were trying to save.