Searching With Small Pieces and Answering With Big Ones

Subtitle: Index one sentence, return its neighbours.

• Left column - Fixed-size chunks (the failure):
• 500 tokens per chunk, one vector
• Four unrelated ideas inside it
• Embedding is their average
• A sharp query matches nothing well
• Retrieval looks fine, answers drift

• Right column - Sentence window (the fix):
• Embed and index one sentence
• Small unit gives a sharp vector
• On a hit, return that sentence
• Plus a few sentences each side
• Model reads context, not fragments

• Simple difference:
• Fixed chunk: index 500, answer 500
• Window: index 1, answer 1 + 2k

• Decoupled units - the real idea:
• Search unit and read unit differ
• Precise unit builds the vector
• Readable unit fills the prompt
• k sets how much context returns

Taped rule strip: Search on the precise unit. Generate from the readable one.

• Honest test - three steps:
• Log every returned window
• Check if any crosses a heading
• Score the answer, not hit rate

Sticky note - Where it breaks:

A fixed window straddles boundaries. On legal or reference text it hands the model a neighbouring clause, and a confident wrong answer follows that no retrieval metric flags.

Margin note: Not sparse scoring - this is chunking and assembly.