Reliable RAG begins where the demo ends

A retrieval-augmented generation demo can look convincing long before it is dependable. Put a few clean documents into a vector index, ask a question that has an obvious answer, and the model appears informed. The difficult work begins with real material: stale policies, conflicting notes, a question that spans several sources, or a question whose answer is simply not present.

The design goal I use is not "make the model answer." It is "make the system show what it knows, what supports it, and when it should stop." That framing moves RAG from a chat feature to an evidence system.

Retrieval is a decision, not a prelude

Generation should be downstream of a retrieval decision. Before composing an answer, inspect the evidence set:

Does it contain direct support for every important claim?
Are the best passages mutually consistent?
Is the query specific enough to retrieve useful context?
Is the evidence fresh enough for the requested decision?

Embedding similarity is useful, but it is not a confidence score. A high score can be a nearby topic rather than proof. A strong pipeline usually combines lexical retrieval for exact terms, semantic retrieval for meaning, metadata filters for scope, reranking for relevance, and a final evidence-sufficiency check.

The most valuable answer a RAG system can give is sometimes a well-explained "I could not verify that from the available sources."

Build an evidence contract

I prefer an explicit contract between retrieval and generation. Each answerable claim should map to one or more source chunks. The interface can then show citations, source titles, document dates, and a concise reason when the system abstains. That makes a bad answer reviewable instead of mysterious.

For production work, log the query, retrieval candidates, reranker result, selected context, model output, citations, and the final disposition. The trace becomes the unit of debugging. If an answer is wrong, you can separate retrieval failure from synthesis failure instead of tuning prompts blindly.

Evaluate the failure modes first

A useful RAG evaluation set is not just a collection of happy-path questions. It should include answerable questions, unanswerable questions, adversarial wording, document conflicts, time-sensitive questions, and queries that require careful scope handling. Track at least retrieval recall, citation correctness, claim support, abstention precision, and user-visible latency.

The OpenAI evaluation guidance is a helpful reminder that product quality comes from a repeatable measurement loop rather than one impressive transcript. The same applies to retrieval: change an embedding model, chunking strategy, prompt, or reranker only when you can compare the result against a fixed suite.

What changed in the PDF RAG system

The PDF RAG Chatbot does not index an upload inside the request that receives it. The API validates and stores the PDF, then a Celery worker handles extraction, page-aware chunks, embeddings, and indexing. That separation matters because parsing a long or scanned document has different failure and time characteristics from serving a question.

Each chunk retains its document and page provenance. That is what lets a generated answer return page-aware citations rather than a vague document name. The service also exposes retrieval without generation, so a bad result can be examined as a retrieval problem before changing the prompt.

Engineering decision. The released default is hybrid retrieval: dense Qdrant candidates and PostgreSQL full-text candidates are fused with reciprocal-rank fusion. Reranking remains explicit because it adds latency and did not justify becoming the default on the checked-in synthetic fixture.

PDFs are not plain text

Native text, scanned pages, two-column layouts, and tables do not fail in the same way. The ingestion path classifies pages, uses PyMuPDF for native extraction, and invokes local Tesseract OCR only when a page does not have enough native text. OCR work has page, document, image-pixel, output-size, and timeout bounds. OCR text and PDF pixels stay in the worker rather than being sent to a provider.

The limitation is important: OCR cannot truthfully restore layout that was not extracted. The system keeps page provenance and distinguishes native, OCR, and mixed material instead of fabricating table structure from weak coordinates.

A concrete abstention boundary

The RAG service returns retrieval confidence, an abstained flag, and an abstention reason. If evidence is below the configured confidence gate, it returns a grounded insufficient-evidence response with no citations and skips the provider call. This is deliberately earlier than generation: a model cannot recover support that retrieval did not find.

A practical release gate

Before I would call a RAG workflow ready for wider use, I would want the following behavior to be boring and reliable:

Answers cite the specific evidence they rely on.
Unsupported questions get an abstention or a request for a better source.
Private or out-of-scope documents cannot leak through retrieval.
A regression suite runs whenever retrieval or prompting changes.
Operators can inspect the evidence path without reproducing a user's entire session.

That is less theatrical than a fluent demo, but it is the difference between a system that sounds informed and one that can support real work.

Closing note

RAG is strongest when it makes the boundary of knowledge visible. A system that can say "this source supports the answer" and "this source does not" is easier to trust, easier to improve, and much safer to place in front of real decisions.

Reliable RAG Begins Where the Demo Ends