A general-purpose language model knows a great deal about the world and nothing about your business. It has not read your product manuals, your HR policies or last quarter’s pricing. Ask it anyway and it may answer confidently and wrongly.
Retrieval-augmented generation, usually shortened to RAG, fixes this by looking things up first. When a question comes in, the system searches your own content for the most relevant passages and gives them to the model along with the question. The model answers from what it was given, and can cite where each part came from.
How does a RAG system work?
- Ingest: documents are collected from where they live (shared drives, wikis, ticketing systems, databases) and split into passages small enough to be specific.
- Index: each passage is converted into an embedding, a numerical representation of its meaning, and stored in a search index alongside its source and access rules.
- Retrieve: a question is embedded the same way, and the closest passages are found. Most good systems combine this with traditional keyword search, because exact terms such as product codes and policy numbers matter.
- Answer: the best passages go to the model with instructions to answer only from them, to say when the answer is not there, and to cite sources.
Why do RAG assistants give wrong answers?
In our experience the model is rarely the weak link. The usual causes are upstream:
- Poor chunking: a passage split mid-table or mid-procedure loses the context that made it meaningful.
- Stale content: three versions of the same policy in the index, and the assistant finds the old one.
- Missing metadata: without dates, owners and document types, the retriever cannot prefer the current, authoritative source.
- Retrieval misses: the right passage exists but was not in the top results. A reranking step, which re-scores the candidates more carefully, often helps more than a bigger model.
Permissions belong in retrieval, not in the prompt
If a user cannot open a document, the assistant must not quote it to them. Telling the model to keep secrets is not a control. Filter results by the user’s permissions at search time, so restricted passages never reach the model in the first place. This is also the main reason to build RAG inside your own environment rather than pasting documents into a public tool.
How to know it is working
Before launch, collect real questions from the people who will use the assistant, with the answers an expert would give and the documents they would cite. Run every change against that set. Track two things separately: did retrieval find the right passages, and did the answer stay faithful to them. When quality drops, that split tells you where to look.
After launch, let users flag bad answers with one click and review them weekly. The fix is usually in the content: an outdated document to retire, or a gap nobody had written down.
