How a RAG Pipeline Actually Works: The 5 Steps, Explained Simply

Most explanations of RAG start with a definition. This one starts with the thing people actually get wrong.

A RAG pipeline is not one flow. It is two, and they run at completely different times. Miss that, and nothing else about it makes sense.

The first flow runs once, when you add your documents. The second runs every single time someone asks a question. Here is what happens in each.

Once: when you add your documents

1. Your documents go in. PDFs, docs, web pages, support tickets — whatever holds the answers.

2. They get split into chunks. Not by page, but into small pieces that make sense on their own. This step decides more about your results than anything else. A chunk that cuts a sentence in half will return half an answer forever.

3. Each chunk becomes an embedding. A list of numbers that stands for the chunk’s meaning, so that “cancel my order” and “how do I return this” land close together even though they share no words.

4. The embeddings go into a vector store. Now your documents are searchable by meaning instead of by keyword. This part is done. You do not repeat it until your documents change.

Every time: when someone asks

5. The question gets embedded, the nearest chunks come back, and the model writes the answer from them.

The question goes through the same embedding model as your chunks did — same numbers, same space — so it can be compared to them. The closest few chunks are pulled out. Then the language model is handed those chunks and asked to answer using them.

That last part is the whole point of RAG. The model is not answering from memory. It is answering from your documents, which is why it can tell you something it was never trained on, and why it can tell you where the answer came from.

What usually breaks

Almost every bad RAG result traces back to one of three things, and none of them is the model.

The chunking is wrong. Too big and the right answer gets buried in noise. Too small and the context is gone.

The retrieval is returning the wrong chunks. The model can only work with what it is handed. If step 5 pulls the wrong three paragraphs, a better model will not save you.

The two embedding models do not match. Chunks embedded with one model and questions with another means the comparison is meaningless.

When a RAG system answers badly, look at what was retrieved before you touch the prompt.

We built this live at the AI Connect India Session 03 with Khushi Yadav, AI Engineer at TruGen AI. People ended the hour with a working pipeline, not notes.

Watch the full session: https://youtu.be/-Uyx6rhWU9Q?si=cxVFNJV3kDgCdDkL

We run one of these every Saturday free and you leave with something running.