The problem RAG solves

A language model knows what was in its training data. It does not know your pricing, your policies, your contracts or what you decided in last week’s meeting. Asked about any of those, a model without access to your information may produce a plausible but incorrect answer, in the same confident tone it uses when it is right.

Retrieval augmented generation fixes this by changing the order of operations. Instead of asking the model to answer from memory, the system first searches your documents for relevant passages, then hands those passages to the model with the question attached, and asks it to answer using only what it was given.

The result is an answer grounded in a source you can click through to. That single property, citation, is why RAG became the default architecture for business AI rather than an interesting option.

How it works, step by step

Five stages. The interesting engineering is in stages two and three, not in the model.

  1. Ingest and chunk

    Your documents are split into passages small enough to be precise and large enough to keep their meaning. Chunking badly is the most common cause of a RAG system that returns irrelevant material.

  2. Embed

    Each passage is converted into a vector, a numeric representation of its meaning, and stored in a vector database. Two passages about the same idea end up near each other even if they share no words.

  3. Retrieve

    The question is embedded the same way, and the system pulls the passages closest to it. Production systems combine this with keyword search, because pure vector search is weak on names, codes and exact figures.

  4. Generate

    The model receives the question plus the retrieved passages, with an instruction to answer from those passages and to say so when they do not contain the answer.

  5. Cite

    The response carries links back to the source passages. This is what makes the output checkable, and it is the part that turns a demo into something a compliance team will approve.

RAG or fine-tuning?

A frequent and consequential fork. They solve different problems and are not substitutes.

RAG Fine-tuning
Teaches the model Facts it can look up Style, format and behaviour
Updating information Add the document Retrain the model
Citations When source links are implemented No
Setup cost Lower Higher
Best for Policies, docs, product data Tone, structured output, niche tasks

If your requirement is “answer from our knowledge, accurately, with sources”, the answer is RAG. Fine-tuning is for when the model needs to behave differently, not know different things. Plenty of production systems use both.

Where RAG projects go wrong

The failures are consistent and almost all of them happen before the model is involved.

  • Bad chunking. Splitting a table across two passages, or cutting a clause in half, can remove needed context and increase the risk of incorrect answers.
  • Vector search alone. Semantic search misses exact identifiers. Hybrid retrieval, vector plus keyword, is the production standard for a reason.
  • Stale content. A knowledge base nobody maintains produces confidently outdated answers, which are worse than no answer because they look authoritative.
  • Contradictory sources. Three versions of the same policy in the index means the system picks one, effectively at random. Deduplicate before you embed.
  • No “I do not know” path. If the system is not explicitly instructed and evaluated on refusing when the retrieved passages are irrelevant, it will fill the gap.

Frequently asked questions

How much does a RAG system cost to build?
A straightforward internal knowledge assistant over well-organised documents runs $20k to $50k. Complexity comes from document variety, access-control requirements and accuracy targets, not from volume. A million clean documents can be easier than ten thousand messy ones.
Does RAG eliminate hallucinations?
No. RAG can reduce unsupported answers, but retrieval can miss relevant evidence and generated answers or citations can still be wrong. Evaluate both retrieval and answer quality. With citations, a wrong answer can be checked against its source. Without them, you have no way to tell. Well-built systems also refuse when retrieval returns nothing relevant.
Can RAG respect our existing permissions?
Yes, and it must. Retrieval is filtered by the requesting user’s access rights before anything reaches the model, so a system that indexes everything cannot leak a document to someone who could not otherwise open it. Retro-fitting this is painful, so specify it up front.
How often does the knowledge base need updating?
It should update automatically when source documents change, which is an ingestion pipeline decision made at build time. Manual re-indexing is a maintenance burden that gets abandoned within a few months.

For implementation support, explore our RAG development services or discuss your workflow in a free consultation.

For a structured approach to AI risks and evaluation, consult the NIST AI Risk Management Framework.