AI Vendor Evaluation: Eight Questions to Ask Any RAG Partner Before You Sign
Most RAG proposals look alike on paper, so AI vendor evaluation comes down to questions that force evidence. Ask whether you can test on your own documents, whether real users can take part, whether every answer shows its source, what happens when the system doesn't know, and how access, data location, logging and success criteria are handled.
Key takeaways
- RAG proposals look alike on paper — these eight questions show the difference.
- The first two questions — your documents, your users — usually separate partners who will prove it from those who will present it.
- Every answer should show its source, and the system should say when it doesn't know.
- Ask how access rights, data location and logging work, in specifics.
- Agree success criteria before any test starts, not after.
Most RAG proposals look alike on paper. Every vendor promises accurate answers, enterprise security and fast deployment. The difference only shows when you ask questions that force evidence rather than descriptions. These eight make a practical AI vendor evaluation — send them to every partner on your shortlist and compare the answers side by side.
The eight questions
1. Can we try it on a sample of our own documents first?
A partner confident in the product will say yes. A test on your own content shows how the system handles your formats, versions and vocabulary — something no demo can. The approach is described in try before you build.
2. Can our real users take part?
Ask for a test with people from the roles that will use the system, not only a technical review. Real users ask questions no project team would write, and those questions are the best test data you will get.
3. Does every answer show where it came from?
Source citations let users and reviewers check the answer. Ask to see a citation for every answer, and check that the cited passage actually supports what was said.
4. What does it do when it does not know?
A good system says so. A poor one guesses. Ask the vendor to show you a question the documents cannot answer, and watch what happens.
5. Who can see what?
Ask how access rights on documents are respected in the answers — and where that check happens. Enforcing permissions inside retrieval, before the model sees anything, is the hardest part of enterprise RAG.
6. Where does our data live?
Ask about deployment options: hosted by the partner, a dedicated environment, or your own cloud. Then ask what leaves your boundary in each case, including calls to model providers.
7. What is logged?
Ask about audit trails and how personal data is handled. The record should show who asked what, which sources were used, and what was answered — and it should be available to you, not only to the vendor. See what an AI audit trail should prove.
8. How will we know it worked?
Agree success criteria before the test starts, not after. Criteria written afterwards tend to describe whatever happened rather than what you needed.
The answers to the first two questions usually separate the partners who will prove it from the ones who will only present it.
How to use the list
- Send the same list to everyone. Written answers are easier to compare than meeting notes.
- Ask for specifics. "Yes, it's secure" is not an answer to question 5; "permissions are checked inside the retrieval query" is.
- Test the top two. Run the same short test with each finalist, on the same sample, with the same users — then score the answers with a shared sheet such as the one in how to score a RAG answer.
What good answers look like
| Question | A strong answer |
|---|---|
| Test on our documents? | Yes, on a sample you choose |
| Real users? | Yes, from each role |
| Sources shown? | On every answer, and they support it |
| When it doesn't know? | It says so |
| Who sees what? | Checked inside retrieval, before the model |
| Where does data live? | Your choice, including your own cloud |
| What is logged? | Every question, source and answer, available to you |
| How will we know? | Criteria agreed before the test |
A vendor who answers all eight with specifics has thought about the problems you will actually face. A vendor who answers with adjectives is asking you to find out later. If you are still deciding whether to build, buy or partner at all, start with build vs. buy AI.
Frequently asked questions
What is the most important question to ask an AI vendor?
How do we compare answers from several vendors?
What should a vendor say about data location?
When should success criteria be agreed?
Ask us the first question.
Yes, you can see SphereIQ working on your own documents. Book a walkthrough and bring the rest of the list.