"How do you know your LLM feature is good, and that a prompt change didn't make it worse?" is the follow-up that comes after almost every LLM design answer. Evaluating a large language model (LLM) application means running it on a fixed set of realistic inputs, scoring each output against written criteria, and comparing those scores every time the prompt or model changes. It answers two questions: is this feature good enough to ship, and did the last change make anything worse?

You need this setup once output is open-ended, like a drafted reply. For structured output, such as a ticket category or a tool's arguments, plain exact-match tests are enough, and a full rubric and model grader are excessive for a prototype with no real traffic to sample yet.

This tutorial covers that offline loop before release. Online A/B test statistics and production monitoring live in other tutorials.

The running example is a support-ticket assistant for a company that sells invoicing software. It reads an incoming ticket, retrieves help-center articles to answer from (retrieval-augmented generation (RAG)), drafts a reply that a human agent reviews and sends, and can call two tools, lookup_invoice and issue_refund. It handles about 6k tickets a day. Prompt v12 is live, and the team wants to ship prompt v13.

In a mid-level round, a strong answer names a test set built from real tickets, a rubric that says what "good" means, and a habit of re-running both on every change. In a senior round, the interviewer pushes on whether you can trust a model to do the grading, how you tell a retrieval failure from a writing failure, and how you catch a change that helps on average but breaks the cases that matter most.

You should already know how to write and version a prompt from Prompt Design. Here the prompt versions are the thing being scored.