OneShopAI
AI Founders

RAG gives confident wrong answers: what actually fixes it

Ravi
Ravi
Level 5, Builder, Gold league, Helpful Voice badge

If you are new to this: RAG just means the AI looks things up in your documents before answering, instead of answering from memory. Most of what goes wrong is in the looking up, not the answering.

Why does RAG give wrong answers?

Almost never the model. If the retrieved chunks do not contain the answer, the model fills the gap, and it fills it fluently. Most of what gets called hallucination is a retrieval problem wearing a costume. Before blaming generation, log what was actually retrieved for the question that failed. Nine times out of ten the answer was not in there.

How should you chunk documents for RAG?

Fixed-size splitting is the default and it is the first thing to change. Cutting every 500 or 1000 tokens slices tables in half and separates a heading from the text beneath it, so the model answers from fragments that lost their context. Chunk on document structure instead: split at headings, keep sections whole, and carry the heading text into the chunk so the retriever knows what the section is about.

Hybrid search vs vector search: which one for RAG?

Pure vector search misses exact strings. Product names, error codes, version numbers. Embeddings match meaning, not characters, so someone searching an exact term can get back everything except the page containing it. Run keyword search alongside the vector search and merge the results. If your users type exact terms, you need both.

How do you stop RAG from hallucinating?

Two rules do most of the work. Require the model to quote the chunk it used, so an unsupported answer becomes visible instead of invisible. And set a confidence threshold: if nothing retrieved scores above it, the answer is “this is not in the docs” rather than a guess. Refusals go up, wrong answers drop much further. A refusal is a cheaper failure than a confident lie.

How do you evaluate a RAG system?

Write real questions with known answers and run them on every change. Without that you are going on “feels better”, which is not a measurement, and you will not catch the change that quietly made retrieval worse. No framework needed to start. A spreadsheet with a question column and an expected answer column is enough.

The one still unsolved: multi-document questions

Questions needing two documents at once still break it. It retrieves from one and answers as though that is the whole picture, confidently, which is the worst combination.

What is the worst wrong answer yours has given?

Paste it. The failures are more useful than the fixes, and I would rather we collect them in one place than everyone hit the same ones separately.

Add your perspective

Sign in to comment, like, and build your reputation.

Comments (0)

No comments yet. Start the first conversation.