A language model on its own answers from what it learned in training, which may be outdated or invented. Retrieval-augmented generation, or RAG, changes this: the system first searches your documents for relevant passages, then gives those passages to the model along with the question. The answer is grounded in text you control, which makes it easier to check.
How a basic RAG system works
- Documents are split into short chunks, each small enough to fit into a prompt alongside others.
- Each chunk is converted into a numerical representation that captures its meaning, so similar text can be found.
- A question is converted the same way, and the closest chunks are retrieved for that question.
- The model receives the question plus the retrieved chunks, and is instructed to answer from them.
RAG does not remove the need to verify answers. It makes verification possible, because every answer can be traced to a source passage.