"How do you ship a prompt or model change safely, and know it's working after launch?" Running a large language model (LLM) feature in production is the answer to that question. You treat every input that changes the model's behavior as one versioned release, you roll each release out gradually, you record what happened on every request, and you watch quality, cost, and user feedback per release, with a fallback ready for when the model provider is slow or down.

This matters because LLM behavior changes without a code deploy. A support lead edits the prompt, a nightly job rebuilds the retrieval index, or the provider quietly moves the model behind a name you call. After launch there is also no answer key: nobody labels whether yesterday's 40k replies were right. The previous section built the offline eval set that gates a change before it ships.

This one covers everything that happens after it ships. Scale it to the stakes: an internal prototype with a handful of users still needs pinned models, release ids, and traces, but shadow traffic, canaries, and a fallback chain wait until real customers depend on it.

The running example is a support-ticket assistant for a B2B software company. It handles about 40k tickets a day: it drafts a reply that a human agent edits and sends, and it answers simple questions directly in the customer chat widget. Each request uses a system prompt, retrieval over a help-center index of about 3k articles, two tools (lookup_account and create_refund_request), and a hosted model called through a provider API. Release r41 is live. Release r42 changes the prompt and moves to a newer model snapshot, and we will ship it.

This topic rarely gets its own interview question, but it is the expected closing answer in AI engineering and forward-deployed design rounds (where an engineer embeds with a customer to ship their LLM project), usually in the last five to ten minutes. At mid-level, the interviewer wants to hear that you version the release and trace each request. At senior level, they also want per-release signals and alerts, a plan for provider model updates, and a fallback chain that holds up during a provider incident.