RAG fails at retrieval, not at generation

A retrieval system gives a confidently wrong answer. The team tries a bigger model, then a better prompt, then a different provider. The actual problem is that the chunk containing the answer ranked eleventh and only ten were passed in.

Measure retrieval on its own

Before touching generation, answer one question: for a set of known queries, is the correct passage in the retrieved set at all?

retrieval_evals
  query
  expected_doc_id
  expected_chunk_hint

Score recall at whatever k you actually pass to the model. If recall@10 is 60%, no amount of prompt engineering fixes the 40% — the information is not there, and the model is doing exactly what it should with what it was given.

This single number tells you which half of the system to work on, and most teams never compute it.

Chunking is the highest-leverage knob

Fixed-size chunks split tables in half, separate headings from their content, and cut procedures mid-step. The retrieved chunk is then technically relevant and practically useless.

Chunk on document structure — headings, sections, table boundaries — and carry the heading path into the chunk text so an isolated passage still says what it is about. A chunk reading “must be completed within 30 days” is unusable; “Refunds > Timeframes: must be completed within 30 days” is not.

Hybrid beats pure vectors

Embeddings are poor at exact tokens: error codes, part numbers, policy references, surnames. A user searching for ERR_4471 gets semantically similar text and not the one document that names it.

Run keyword search alongside vector search and merge. It is the cheapest large improvement available, and it fixes the query class where users are most confident the system is broken — because they can see the document exists.

Cite, and make citations checkable

Every claim carries the chunk it came from, surfaced as a link the user can open. Two effects: users self-correct when a citation does not support the claim, and you get labelled failure data for free every time someone clicks through and the passage is wrong.

The failure to design for

Answering when nothing relevant was retrieved. If the top result is below a similarity floor, the correct behaviour is “I do not have that”, not a fluent paragraph assembled from adjacent material. That threshold is a product decision, and it should be tuned against the eval set rather than left at whatever the library defaults to.