What RAG is, without the mysticism
Retrieval-augmented generation means the system searches your documents for passages relevant to the question, hands them to the model as context, and asks for an answer grounded in those passages with citations. The model stops guessing from general knowledge and starts answering from your truth.
This matters whenever being wrong is expensive: policies, contracts, product specs, compliance rules, internal procedures. It also keeps answers current, because the index updates when your documents do, without retraining anything.
Where failed projects actually failed
In our experience the model is almost never the problem. Projects fail at retrieval: documents chunked badly so answers miss context, no hybrid search so exact terms do not match, no reranking so the right passage sits at position nineteen, no metadata filters so the model sees content the user is not allowed to see.
The second failure is ingestion: PDFs with tables, scans, wikis with three versions of the same policy. If the pipeline does not handle the ugly reality of your document estate, the retrieval layer inherits garbage.
- Chunk by structure, not by character count
- Hybrid keyword plus semantic search, with reranking
- Access control enforced in retrieval, not just the interface
- Freshness: re-index on change, or answers rot
What it costs to build well
A scoped knowledge system over a bounded document set is typically a small to mid fixed-scope build: ingestion, retrieval engineering, answer interface, access control and evaluation. Running costs are modest and predictable, dominated by generation usage.
What you should refuse to buy: a bot with a document-upload box, no access model, no evaluation and no citations. That is a demo with a subscription.
Where to point it first
The highest-yield first RAG systems share a shape: a bounded, high-value corpus with people who ask the same questions weekly. Internal policy and procedure assistants, contract and compliance lookup, product knowledge for support and sales, onboarding libraries. Each turns scattered institutional memory into an answer with a citation.