Why a Log Is a Better Queue Than a Queue

Subtitle: One takes it away. One remembers. Left column - Queue (delete on read): - Message consumed, then deleted - Second reader gets nothing - No rewind, no replay - Broker tracks work left to do - One reader wins the record Right column - Log (append + offset): - Records append, never deleted - Each group keeps its own offset - Many readers, same record - Rewind = move offset back - Broker tracks position, not work Simple difference: - Queue = take the message away - Log = remember the position Partitions - unit of order and scale: - Order holds inside one partition - No order across partitions - Key hash picks the partition - Key by user id keeps it ordered Honest test - three steps: 1. Kill consumer after its write 2. Restart before offset commit 3. Duplicate arrives harmlessly Sticky note - Distrust exactly once: Delivery is at-least-once unless the write and the offset commit are one atomic step. So build an idempotent consumer.