Data Engineering · Sensei

The Pipe That Batches Your Events So the Warehouse Does Not Choke

A managed delivery pipe with exactly two knobs — and three things it will never do for you.

Technical poster: producers write records into a buffer with size and time triggers, optional in-flight transform to Parquet, delivery to object store, warehouse and search cluster, with warnings about no replay, duplicates and no back-pressure.

Core Concept

This is a delivery pipe, not a queue you read from — and that single distinction is the whole point. You write records in. It buffers them, and on its own schedule it writes them out to a destination: an object store, a warehouse, a search cluster. You never subscribe, never poll, never acknowledge.

Amazon Kinesis Data Firehose is the canonical example. Because it owns the flush schedule, it protects a downstream warehouse from being hammered by one-record-at-a-time writes — the thing that actually makes analytical stores choke.

Key Components

The size trigger

Flush once the buffer reaches a byte threshold — say 5 MB. Under heavy traffic this is the trigger that fires.

The time trigger

Flush once the buffer has been open for a wall-clock interval — say 300 seconds. Under light traffic this is the trigger that fires.

The in-flight transform

An optional function runs before the write — commonly converting JSON to Parquet, which is why this pipe so often sits between an app and a lake.

The error bucket

When delivery fails, records land in a separate failure destination. Nothing tells your producer. You have to go look.

How It Works

Your application writes records into the pipe. There is no partition to choose and no offset to track.
Records accumulate in a buffer that is watching two thresholds at once.
Whichever threshold fires first wins. 5 MB reached at second 40? It flushes at second 40. Only 200 KB by second 300? It flushes anyway.
If a transform is configured, the batch is converted — JSON in, columnar Parquet out — before it lands.
The batch is written to the destination as one object. On failure it is retried, then diverted to the error destination.

Those two thresholds are the entire tuning surface, and together they are really one dial: latency versus file size. Raise the time and you get fewer, fatter files that a warehouse scans efficiently. Lower it and you get fresher data spread across many small files — more per-query cost, more metadata, slower scans. There is no third setting that gets you both.

Falsification — Three Things It Does Not Do

No replay, no consumer group Unlike a Kinesis stream or a Kafka topic, you cannot rewind and re-read what you sent. Records are gone once delivered. So if your transform function has a bug, you have corrupted data at the destination and no source to re-derive it from.

No exactly-once Retries mean duplicates are normal. This is at-least-once delivery, and the destination must dedupe. If your downstream table has no idempotency key, your counts are wrong and nothing will tell you.

No back-pressure It is fire-and-forget. A struggling destination never slows your producer down; the producer keeps writing happily while failures pile into an error bucket. Your only signal is one you built yourself.

The test: revoke the destination's write permission and start a stopwatch. However long it takes you to notice is your real detection latency for every silent delivery failure you will ever have. Most teams discover it is measured in days.

Where You Actually Meet This

Clickstream to a lake — web/app events buffered into Parquet on S3 for cheap analytical scans.
Log shipping to search — service logs batched into OpenSearch instead of per-line indexing that would melt the cluster.
IoT and telemetry — thousands of tiny device writes flattened into a handful of fat objects per hour.
CDN and access logs to a warehouse — high-volume, replay-free, tolerant of duplicates: the shape this pipe was built for.
When NOT to use it — anything needing replay, ordering guarantees, or multiple independent consumers. That is a stream, not a pipe.

Checkpoint — answer before you move on

Traffic is light and only 200 KB has accumulated against a 5 MB / 300 s buffer. When does the flush happen, and which threshold decided it? Hint: the two triggers are in a race, not a sequence.
Your in-flight transform shipped a bug for six hours before anyone noticed. Explain why you cannot simply reprocess the affected window. Hint: what does the absence of a consumer group actually cost you?
Your producer's metrics look perfectly healthy while the destination has received nothing for an hour. What property of this pipe makes that possible, and what would you build to catch it? Hint: fire-and-forget cuts in one direction only.