Start with a claim you can check
In our illustrative procedure, the product owner approves a change after its impact on the schedule has been assessed. “The product owner” identifies the role but leaves out the condition. Linking to the entire procedure still leaves the reader to find the relevant passage.
A useful result pairs an answer with the exact passage. A user can open it and compare the wording: does it name the right role, preserve the conditions and avoid adding a new commitment?
Separate three checks
First, evaluate retrieval: was a document containing the relevant rule found? Then evaluate the answer: does every material claim follow from the passage? Finally, evaluate the next action: does the user know what to do?
These are different failure modes. A relevant document does not guarantee a correct answer. A correct answer does not guarantee enough context to act. Record the failures separately so you know what needs to change.
Ask something the sources cannot answer
A useful evaluation includes unfamiliar questions. Ask for something missing from the documents, such as the exact budget for the change. The expected behaviour is to identify missing information, not add a plausible number.
The Novera RAG example uses three illustrative documents and local passage retrieval. It lets you explore the interaction between a question, an answer, a source and missing information. It does not measure model quality on your company’s documents.
What to bring to a pilot discussion
Choose real questions, reference passages and several unanswerable questions. Include examples of outdated rules, conflicting information and documents with different access permissions.
Agree who reviews the answers and which errors are unacceptable. Then select retrieval, models and integrations against those criteria.
Try search with sources
An illustrative procedure and answers you can check against its text.