What Is RAG? How AI Answers from Your Own Data

Last updated: 27 شهریور 1405

You want your site's chatbot to answer the question "How many days do I have to return an item?" A language model only knows what was in its training data, and your store's policy page was never in it. So it either says it doesn't know or, worse, gives a plausible but invented number.

RAG is the most common fix, and the idea is simple: before the model answers, find the relevant text in your own data and give it to the model along with the question.

What is RAG?#

RAG (short for Retrieval-Augmented Generation) is a method in which the system, before each answer, finds relevant pieces of your documents and passes them to the language model with the question. The model no longer answers from memory; it answers from the text you put in front of it.

It isn't the only way. There are three ways to make a model aware of your data:

If all your text is a few pages, the simplest way is to put it all in the prompt. RAG becomes useful when the data doesn't fit in the context window, or when sending everything with every question is a needless cost. A common misconception is that you must fine-tune to "teach" information, but every price or policy change would require retraining. With RAG, you just update the document.

How does RAG work?#

RAG has two parts: preparing the documents, which is done once and again after each change, and answering, which repeats on every question.

Preparation#

1. Collect documents: policy pages, FAQs, product guides, technical documentation.
2. Chunking: each document is split into short sections, say one or two paragraphs, so that only the relevant part is sent later.
3. Embedding: a dedicated model turns each chunk into a list of numbers that represents its meaning. Two texts with similar meaning get numbers that are close together, even if their words differ.
4. Storing: the vectors are stored with the original text in a vector database or anywhere that supports similarity search.

Answering#

1. The user's question is turned into a vector with the same model.
2. The few chunks whose vectors are closest to the question are found.
3. These chunks are sent to the language model along with the question and a specific instruction.
4. The model writes the answer based on those chunks and, if you ask, names the source.

The instruction in step three matters a lot:

Answer only based on the texts below. If the answer is not in these texts,
say "It's not in my information" and don't guess.
At the end, write the title of the text the answer came from.

Texts:
[retrieved chunks]

User question:
[question]

Is semantic search always needed?#

No. For a small knowledge base, ordinary text search is often enough. Semantic search became popular because it links "deadline for sending goods back" to "return policy", which keyword search doesn't do.

With text in non-English languages, watch out for character variants and normalization in plain text search. For Arabic and Persian, for example, letter variants and invisible joiners can make the same word fail to match, so normalize text before indexing and searching. Many systems also work in a hybrid way, blending keyword and semantic results.

Where does RAG break?#

Almost all RAG problems come from retrieval, not from the model. If the right chunk doesn't reach the model, even the best model can't answer correctly. Six common mistakes:

• Wrong retrieval. Measure retrieval separately: for twenty real questions, are the right chunks found?
• Chunks too small or too large. A one-line chunk has no context; a multi-page chunk drags irrelevant text along with it.
• Outdated documents. If you changed the return policy but not the knowledge base, the bot will confidently repeat the old policy.
• Contradicting documents. Two pages with two different numbers force the model to choose.
• Private data in the knowledge base. Anything you index may appear in an answer to any user. Don't put customer information or internal documents into a public knowledge base.
• Hidden instructions in documents. If you index user-written text or outside pages, someone can plant instructions for the model in it; we explain this risk in Prompt injection on websites.

Where do we start?#

Start with a small set of real questions, not with choosing tools:

1. Collect twenty to thirty real customer questions from emails and tickets.
2. Mark the correct answer for each in your existing documents. If an answer isn't in any document, write the document first.
3. Start with the simplest search and see how many questions reach the right chunk.
4. Add the "answer only from the texts" instruction and read the answers one by one.
5. Whenever you fix a document, re-index and repeat the same questions.

This question set stays useful later: after every change to the system, test with it again. We build a complete chatbot on this foundation in Support chatbot for a WordPress site.

Where is RAG heading?#

RAG is moving from a specialist technique to an ordinary feature. Many tools already let you attach a few documents to a project or assistant and handle retrieval themselves. Model context windows have grown, and for small data, chunking is sometimes no longer needed. But for data that is large, changing, or has different access levels, targeted retrieval is still required. Another direction is agents that decide for themselves where to search and how many times.

Either way, answer quality is tied to document quality. To prepare:

• Write each help page about one topic, with clear headings and sections that make sense on their own.
• Record the last update date and the owner of each document, and delete old documents rather than just leaving them.
• Keep public and internal documents separate from now on.

Summary#

• RAG finds the relevant pieces of your data before answering and gives them to the model with the question.
• For changing information, RAG suits better than fine-tuning; for a few pages of text, a plain prompt is enough.
• RAG quality depends more on retrieval and documents than on the model.
• Tell the model explicitly to answer only from the texts and to name the source.
• Don't index private data, and keep documents tidy and up to date.