ICGichelonCONSULTING GROUP

AI · 2 min read

An evaluation checklist for an internal AI assistant.

Useful answers need more than fluent language. Test evidence, boundaries and the route out of uncertainty.

An AI agent planning a task, calling tools and APIs, and acting across a company's systems

Test the questions people actually ask

Use examples from the intended workflow and include different wording for the same need. Keep a separate set of examples for evaluating changes. A system that performs well only on the demonstration questions has not established its usefulness for everyday work.

Check the supporting evidence

Review whether cited information supports the answer and whether it was available to the user making the request. Test stale, contradictory and missing sources. The system should be able to acknowledge that it does not have enough evidence to answer confidently.

Exercise the failure paths

Include an unavailable data source, an unsuccessful tool call and a request beyond the intended task. Confirm that retries are bounded and that escalation gives a person enough information to continue. A failed dependency should not quietly become an invented answer.

Repeat the evaluation after changes

Changes to source material, instructions, retrieval or models can affect quality. Keep the evaluation process repeatable and record what changed. Review usefulness alongside response time, operating cost and the effort needed to correct errors, rather than reducing readiness to one accuracy number.

Bring it back to your business

Choose one workflow, establish the current situation and identify who needs to be involved. A small, observable improvement gives a better foundation for the next decision than an ambitious plan with no baseline.

Discuss your challenge

Continue reading

Let’s move forward

Your next chapter starts with a conversation.

Tell us what is getting in the way. We’ll work through the possibilities together.

Start the conversation