Company BrainAI Factory

RAG Testing: The Questions Your Team Asks Are Not the Ones You Planned For

Founder & CEOOctober 16, 20263 min read

In short

Good RAG testing uses the questions real users actually ask, not the list the project team wrote. Planned questions are phrased the way the system expects, so they make it look better than it is. Real questions are short, full of internal shorthand, dependent on the previous answer and often about things the documents never covered.

Key takeaways

Test lists written by the project team make a RAG look better than it is.
Real questions are short, use internal shorthand, build on the previous answer and sometimes ask for things nobody wrote down.
Put the chat in front of real users before the design is final.
Keep every question and sort them: answered well, answered badly, not in the documents.
The second and third groups tell you exactly what to fix or add.

Every RAG project has a test list written by the project team. Real users ignore it. The gap between the questions you planned for and the questions people actually ask is where most RAG systems disappoint — and it is the most useful thing a test can reveal.

Why do planned test questions mislead?

People who build a system know how it is meant to be used, so they write questions it can answer. They use the right product names, phrase things completely and stay inside the documents they loaded. Users do not know any of that. They ask what they need to get through their day, in the words they use, and they are not polite about vocabulary.

The result is a test that measures the system on its best behavior. It passes, the project moves forward, and the problems appear in front of real users after the build.

Why it matters

A test list written by the project team tests the project team's expectations. Only real users test the system.

What do real questions look like?

Short and incomplete. Often without the product or process name — "what's the limit for that?"
Built on internal shorthand. Abbreviations, team nicknames and code names no general model has seen.
Follow-ups. Questions that only make sense after the previous answer.
Outside the documents. Questions about things that were never written down at all.

Each of these stresses a different part of a RAG system: retrieval has to find the right passage from a vague question, the system has to understand your vocabulary, it has to keep context across a conversation, and it has to admit when the answer is not there.

How do you use real questions in RAG testing?

Put the chat in front of real users early. Before you finalize the design, not after.
Keep every question. Not just the ones that worked. The failures are the valuable part.
Sort them into three groups. Answered well, answered badly, and not in the documents.
Use the second and third groups. The answered-badly group tells you what to fix in retrieval or configuration. The not-in-the-documents group tells you what knowledge to add.

A team we worked with did exactly this: their real users asked the chat their own questions on a small sample of documents, which gave them a list of real questions that no planning meeting would have produced. The format is described in try before you build, and a day-by-day version is in the two-week AI pilot plan.

How do you score what you collect?

Once the questions are sorted, score a sample of the answers — including the ones users flagged — on five points: correct, complete, sourced, honest and useful. The sheet is in how to score a RAG answer. Pay particular attention to "honest": for questions outside the documents, the right answer is "I don't know", and a system that guesses instead is a risk.

What real questions tell you about your knowledge

The questions nobody could answer are not only a test result. They are a map of the knowledge your organization runs on but never wrote down — the tacit knowledge that walks out the door when people leave. If that pattern looks familiar, what is tacit knowledge explains why it never makes it into your systems, and how a Company Brain captures it.

Frequently asked questions

Why are planned test questions misleading?

Because the people who build a system know how it is meant to be used, so they write questions it can answer. Users ask what they need to get through their day, in their own words — which is the real test.

How do we collect real user questions?

Put the chat in front of users from the roles that will rely on it, let them ask freely during their normal work, and keep a record of every question and answer.

What should a RAG evaluation framework include?

Real questions from the people who will use the system, sorted into answered well, answered badly and not in the documents, plus a scoring sheet applied to a sample of answers: correct, complete, sourced, honest and useful. Planned test lists alone measure the system on its best behavior.

What do we do with questions the documents can't answer?

Treat them as a map of missing knowledge. They show which documents need to be written, updated or added before a wider rollout.

How many users do we need?

Two to four people from each role is enough to surface the patterns that matter in a first test.

Collect your users' real questions first.

In a walkthrough we set up SphereIQ on a sample of your documents so your team can ask the questions they really ask.

Book a walkthrough See the Company Brain