AI Pilot Program: A Two-Week Plan to Test a RAG on Your Own Documents
A useful AI pilot program for RAG does not need a project team or months of work. It needs a small sample of your documents, a few real users from different roles and a plan: agree criteria first, set up the chat, let people ask freely for a week, score the answers, and decide — build, extend or stop.
Key takeaways
- A RAG pilot needs a small sample, a few real users and a plan — not a project team.
- Agree the goal, the criteria and what will be shared before day one.
- Give real users a full week to ask their own questions, and keep every one.
- Sort questions into good, poor and not covered, then score a sample of answers.
- End with a decision on a fixed date: build, extend the pilot, or stop.
A RAG pilot does not need a project team, a long procurement process or months of work. It needs a small sample of documents, a few real users and a plan. Here is a practical two-week AI pilot program — who does what, day by day — and the rules that keep the result honest.
What does the two-week plan look like?
| When | What happens | Who |
|---|---|---|
| Before day 1 | Agree the goal, the success criteria and what will be shared. Choose a small sample of documents from one area. | Champion, security |
| Days 1 to 2 | Set up the chat on the sample. Check that a source appears with every answer. | Partner |
| Days 3 to 9 | Real users ask their own questions. Keep every question and answer. | Users from two or three roles |
| Days 10 to 12 | Sort questions into good, poor and not covered. Score a sample of answers. | Champion, reviewers |
| Days 13 to 14 | Review findings and decide: build, extend the pilot, or stop. | Sponsor, all roles |
Before day 1: set the ground rules
Most pilots that go wrong go wrong here. Before anything is set up, agree three things in writing:
- The question the pilot answers. For example: "Will a RAG work on our operations documents for our service and compliance teams?"
- The success criteria. Answers are correct, the source is shown, the system says so when it does not know, and users would use it again. The reasoning behind testing this way first is in try before you build.
- The sample. Which documents, from which area — and what stays out. See what to share in a pilot.
Days 1 to 2: set up
The partner sets up the chat on the sample and checks the basics: every answer shows a source, answers only draw on documents each user is allowed to read, and the system says so when the documents do not cover a question. This is also the moment to confirm where the sample lives and what is logged.
Days 3 to 9: let real users ask
This is the heart of the pilot. Users from two or three roles use the chat for their normal work and ask whatever they need. Two rules matter:
- Do not script the questions. The value of the pilot is in the questions nobody planned.
- Keep everything. Every question and every answer, including the ones that went badly.
A week of unscripted questions from real users is the best test data you will ever collect. No planning meeting produces it.
Days 10 to 12: sort and score
Sort every question into three groups: answered well, answered badly, and not covered by the documents. Then score a sample of answers on a shared sheet — correct, complete, sourced, honest and useful, as described in how to score a RAG answer. Include some of the answers users flagged as poor.
Days 13 to 14: decide
Bring the sponsor and every role together, review the findings against the criteria agreed before day one, and decide:
- Build — the criteria were met and you know which documents and roles to start with.
- Extend — the results are promising but a gap needs another round, for example a different role or document set.
- Stop — the documents or the use case are not ready yet, and you found out early.
Rules that keep the pilot honest
- Do not script the questions.
- Include people who are skeptical, not just enthusiasts.
- Write down the success criteria before the first question is asked.
- Keep the sample small enough that everyone can review the results.
- Set the decision date in advance, so the pilot does not drift into a project.
A pilot built this way avoids the most common trap in enterprise AI — a demo that impressed everyone and then never reached production. The two-week length is a suggestion; adjust it to your organization, but keep the shape.
Frequently asked questions
How long should an AI pilot program take?
Who needs to be involved?
What if the pilot shows the documents aren't ready?
What decisions can a pilot end in?
Run this plan with us.
In a walkthrough we set up SphereIQ on a sample of your documents and map this two-week plan to your teams.