Describe the user’s action
Who receives the result, and what do they do next? In document search, an employee might open the source before making a decision. In image inspection, an operator might review the marked area.
Write down the input and expected output for one example. If people understand it differently, agree on the task first.
Collect examples that are easy to get wrong
Include incomplete documents, similar phrases or frames with reflections alongside ordinary examples. Define the expected response for each, including when the system should decline to answer.
Keep tuning examples separate from evaluation examples. Changing a model after seeing the test examples affects the independence of the next measurement; account for this in the test record.
Separate different kinds of error
In computer vision, recognising a product and counting it are different tasks. In document search, finding the right passage and answering correctly from it are also different checks.
One overall score can hide a problem in an individual operation. Define a criterion for each action that matters to the user.
Test the complete path
A good model cannot help if a frame never arrives from the camera or an employee cannot access a document. Include the data source, API and the screen that displays the answer in the test.
Record what happens after a timeout, an empty result or an unavailable source. The user needs a clear next action in each case.
Prepare for the first conversation
Bring a description of the operation, a few anonymised examples and their expected outputs. Add data-hosting requirements and identify someone who can check the result.
Then you can discuss the pilot’s scope and an estimate. Agree what would justify the next stage before development begins.