Turning an Hour of Speech Into Text You Can Search
Managed speech-to-text is not interesting because it writes down words. It is interesting because of everything it attaches to each word.
Core Concept
Managed ASR (Automatic Speech Recognition) — AWS Transcribe is the reference example — takes audio you place in object storage (S3) and returns JSON, not a paragraph. Every single token in that JSON carries three things: a start time, an end time, and a confidence score.
Those timestamps are the whole product. Plain text lets you search a wall of words; timestamps let you jump to the moment a word was said. That is the difference between a transcript you read and a recording you can navigate.
Key Components
Batch
You hand over a whole file and the job runs asynchronously. You come back later for a finished, stable transcript. Best when the audio already exists.
Streaming
Returns PARTIAL results that get REVISED as more audio arrives. This is exactly why live captions visibly rewrite themselves mid-sentence — nothing is broken, the hypothesis just improved.
Diarization
Labels which speaker said which segment. Turns one undifferentiated stream into Speaker 1 / Speaker 2 lanes — which is what makes a meeting transcript usable at all.
Custom Vocabulary
Your product names, drug names, ticker symbols. A general model has never seen them and will confidently mangle them. Plus automatic language identification when you don't know the language up front.
How It Works
- Audio lands in object storage (S3) — or opens a socket if you're streaming.
- You start a job, declaring the boring-but-decisive options: diarization on/off, a custom vocabulary list, language (or auto-identify).
- The model decodes audio into tokens, each stamped with start time, end time and confidence.
- Batch: the job completes and you fetch stable JSON. Streaming: partials arrive immediately and are overwritten as context accumulates.
- You index the tokens. Now a search hit is a timecode you can seek to, not just a line you can read.
Falsification — Where This Breaks
- A benchmark WER tells you almost nothing about your audio. Word Error Rate collapses very different failures into one number, and the errors are not uniformly distributed.
- Errors concentrate exactly where you live: accents, cross-talk, background noise, domain jargon. A 5% WER can still mean every instance of your product name is wrong — because that name is in the 5%.
- Diarization cannot reliably split overlapping speakers — which is most real conversation. Two people talking at once is the normal case, not the edge case.
- Confidence is not correctness. A mis-transcription of an unfamiliar name often returns with HIGH confidence: the model is certain about a word that happens to be the wrong one.
The Test That Actually Decides It
Run it on your worst recording, not a clean one. Then count errors on the twenty terms that actually matter to you — the names, SKUs, tickers, drugs. That per-term count is your real accuracy. The aggregate WER is a number about someone else's audio.
Real-World Applications
- Searchable call archives — support or sales calls where a hit jumps straight to the second it was said.
- Live captioning — streaming partials power real-time captions; the visible rewriting is the mechanism, not a glitch.
- Meeting notes with attribution — diarization is what lets "who committed to what" survive the transcript.
- Media indexing & clipping — timestamps let editors cut to a spoken phrase without scrubbing.
- Compliance review — flagging regulated phrases, where custom vocabulary decides whether you catch them at all.
Checkpoint — answer before you move on
- Besides the words themselves, what three things does the transcript attach to every token — and why is one of them what makes the output searchable rather than merely readable? Hint: one of them is the reason you can seek instead of scroll.
- Why do live captions visibly rewrite themselves mid-sentence, and what is the trade-off you accept by choosing that mode over the other one? Hint: partials vs. an asynchronous, stable result.
- A vendor shows you 5% word error rate on their benchmark. Explain why that figure can be true while the system still gets every mention of your product name wrong — and state the test you would run instead. Hint: uniform distribution is the assumption doing the damage. Also: what does high confidence actually certify?