RAG exists to solve one problem: a language model does not know your business, and when asked about it will produce something plausible instead of admitting that.
The mechanism is simple. Before answering, search a store of your own content for the passages most relevant to the question. Put those passages in front of the model along with the question. Ask it to answer using only what it has been given.
Why it beats the alternatives
The other options are worse for most businesses. Fine-tuning a model on your documents is slower, more expensive, has to be redone whenever the documents change, and still does not reliably prevent invention. Putting every document into each request is impossible past a certain size and costly well before that.
RAG updates instantly — change the document, the next answer changes — and it can cite what it used, which turns an unverifiable answer into a checkable one.
Where it goes wrong
Almost always in retrieval, not generation. If the search returns the wrong three passages, a perfect model gives a wrong answer confidently.
The common causes are unglamorous: documents chopped into chunks that split a table from its heading, outdated files nobody removed, and near-duplicate documents where the stale copy ranks higher. Most of the work in a RAG deployment is document hygiene, which is why an honest proposal spends more time on your content than on model choice.
The question to ask
"When the system does not know, what does it say?" A good deployment says it does not know and offers a person. A bad one guesses, and you find out from a customer.
Questions
- How current can the answers be?
- As current as the source. Because retrieval happens at question time, updating the underlying document is enough — there is no retraining step and no waiting.
- Does our data leave our systems?
- The retrieved passages are sent to whichever model is generating the answer, so where that model runs, and what it retains, are the questions that matter. Both should be named in the proposal.
Wondering whether this applies to your business? That is a thirty-minute call, not a research project.
Book a Consultation Call