Model serving topologies: online, batch and streaming
AI Platform & MLOps

Where the Model Actually Runs When Someone Hits Your Endpoint

Serving is not a deployment detail. It is a decision about where you are willing to keep latency, and where you are willing to keep idle cost.

Core Concept

A trained model does nothing until something asks it a question. Serving is the choice of when that question gets answered — the instant it is asked, on a schedule before it is asked, or as events flow past. Each shape puts the latency and the idle cost in a different place. There is no free arrangement; there is only the one that matches your workload.

Most teams pick the real-time shape by reflex, because it is the one that looks like software. That reflex is the expensive part.

Key Components

Shape 1

Online / real-time

Model sits behind an HTTP endpoint. One request, one inference. Latency in milliseconds — and a machine kept warm whether anyone calls it or not. You pay for readiness.

Shape 2

Batch

Predictions computed on a schedule across a whole dataset, written to a store the application reads. Serving cost is decoupled from request rate entirely. Viable only when the input is known in advance.

Shape 3

Streaming

Predictions computed as events arrive and pushed to a consumer. Sits between the other two: no idle endpoint per user, but no waiting for a nightly window either.

The real lever

Feature availability

What decides the shape is not model size — it is whether the features exist at request time. A feature that takes 200ms to compute cannot be tuned away at the endpoint.

Dynamic batching. An online server can wait a few milliseconds to group concurrent requests into one GPU pass. You trade a little latency for a large throughput gain. This is why "real-time" and "batch" are not opposites — the batching can live inside the request path.

How It Works

  1. Ask what the prediction needs as input, and ask when each of those inputs becomes known.
  2. If every input was knowable yesterday, the answer was knowable yesterday. That is a batch job writing to a store.
  3. If an input only exists at the moment of the request — this session, this cart, this click — you need real-time, and the features must be retrievable in single-digit milliseconds. That is why a feature store usually appears next to a real-time endpoint: it pre-computes the slow features so the request path only has to look them up.
  4. If inputs arrive as a continuous flow and consumers want them soon but not synchronously, stream.
  5. Once the shape is chosen, tune inside it: dynamic batching and warm pools for online, window size and freshness targets for batch and streaming.
The two failure directions. The loud, expensive one: a real-time endpoint serving a workload whose inputs were all knowable yesterday — a warm GPU answering questions a nightly batch could have answered. The quieter, worse one: batch predictions go stale. A recommendation computed at 3am is being served to a user whose session started five minutes ago.

How To Falsify It

Do not measure when your prediction was computed. Measure how old it is at the moment it is consumed. Log the timestamp of the prediction alongside the timestamp of the read, take the distribution of that gap, and look at the tail. If the gap is hours and the underlying behaviour changes in minutes, your batch shape is wrong no matter how good the model is. If the gap is milliseconds but the endpoint is idle 95% of the day, your online shape is wrong no matter how good the latency looks.

Real-World Applications

  • Fraud scoring — online, because the deciding features are the transaction that is happening right now.
  • Nightly churn or credit-risk scores — batch, because the inputs are last month's behaviour and nobody needs the answer within a millisecond.
  • Homepage recommendations — the classic trap: batch is cheap and stale, online is fresh and expensive; most teams end up hybrid, with batch candidates re-ranked online.
  • LLM chat endpoints — online with dynamic batching, because concurrent requests share a GPU pass and the throughput gain is large.
  • IoT / telemetry anomaly detection — streaming, because events arrive continuously and the consumer is another system, not a waiting human.

Checkpoint — answer before you move on

  1. Your model is fast but your endpoint is slow. Why might adding a feature store fix this when tuning the server will not? Hint: where is the 200ms actually being spent?
  2. Dynamic batching makes an individual request slightly slower. Explain why you would deliberately accept that on a GPU server. Hint: what is being amortised across the grouped requests?
  3. You inherit a batch recommendation pipeline. Describe the single measurement that tells you whether it is serving stale predictions — and say why "it runs successfully every night" does not answer that question. Hint: age at consumption, not age at computation.