An insurance team we spoke with had built an internal assistant over eight years of policy documents, circulars and claim precedents. Users reported it was confidently wrong about coverage limits roughly one query in five. The instinct was to change the model. The actual cause was that the limits lived in tables inside PDFs, the chunker had split those tables mid row, and the retrieved fragment showed a number with no idea which product it belonged to. The model was reasoning perfectly over garbage.
This is the standard failure. When a retrieval-augmented system gives a wrong answer, the fault is in retrieval far more often than in generation.
Chunking Is a Content Decision
The default advice, 500 tokens with 50 tokens of overlap, is a starting point that suits prose and suits almost nothing else. Contracts, policy schedules, engineering specs and financial statements all carry meaning in structure that a fixed window destroys.
What works better in practice is chunking along the document’s own boundaries. Split on headings and clause numbers. Keep a table whole even if it runs long, and prepend its caption and the section heading to the chunk text so the fragment carries its own context. For long documents, attach a short generated summary of the parent section to each child chunk. That single change tends to move retrieval accuracy more than any embedding model upgrade.
Keep the metadata too. Effective date, document version, business unit, jurisdiction. An answer drawn from a superseded circular is worse than no answer, and only metadata filtering prevents it.
The Retrieval Stack That Actually Performs
Dense vector search alone is not enough for enterprise content, which is full of product codes, policy numbers and named entities that embeddings handle poorly.
- Hybrid search. Combine BM25 keyword matching with vector similarity and fuse the result lists. Exact identifiers get found by the keyword leg, conceptual queries by the vector leg. This is usually worth 10 to 20 points of recall over vectors alone.
- A reranker over the top 50. Retrieve wide, then use a cross encoder to reorder and pass only the best 5 to 10 into context. Reranking is comparatively cheap and reliably lifts precision at the position that matters.
- Query rewriting before search. User questions are terse and full of pronouns. Expanding the query, or generating two or three variants and merging the results, closes the vocabulary gap between how people ask and how documents are written.
- Permission filtering at query time. Filter by the requesting user’s entitlements inside the search call, never after. A retrieval layer that can surface an HR file to the wrong employee is a data incident, and under India’s DPDP Act it is a reportable one.
- Citations on every claim. Each answer should name its source document and section. It makes the system checkable, and it converts user distrust into useful correction reports.
Evaluate Retrieval Separately
Judge the retriever on its own before judging the answers. Assemble 100 real questions with the documents that should have been returned, then measure recall at 10 and mean reciprocal rank. If the right chunk is not in the top 10, no amount of prompt work will save the response.
Once retrieval is solid, evaluate generation for faithfulness, meaning whether every claim is supported by the retrieved text, and for appropriate refusal when the corpus simply does not contain the answer. A system that says it does not know is worth considerably more than one that guesses well.
At Invexa, we build knowledge systems retrieval first. We spend the early weeks on document structure, metadata and access rules, because that groundwork is what determines whether the finished assistant is a reliable colleague or a plausible liability.