Your team ships a support-ticket assistant for a helpdesk product. For each incoming ticket it reads the thread, pulls a few help articles, and drafts a reply for the agent. It handles about 10k tickets a day. The demo looked great. A month later finance asks why the bill is $7k, and agents complain that each draft takes 9s to appear. The interviewer's version is short: this feature costs too much and feels slow, what would you change?

Both numbers come from the tokens a large language model (LLM) reads and writes. Cost is the tokens you send and receive times their prices. Latency is the wait before the first token plus the time to generate the rest. Every fix in this tutorial changes one of those terms. Prompt trimming and output limits send or generate fewer tokens. Caching reuses work you already paid for. Routing sends the work to a cheaper model. Streaming changes when the user sees the tokens, and batching changes when you need them back.

Measure before cutting. Optimizing a feature that costs a few hundred dollars a month rarely saves enough to justify the engineering time. Before making changes that can affect answers, such as routing or a semantic cache that reuses answers to similar questions, build an eval set that can detect regressions.

This is usually a five-to-ten-minute follow-up in AI engineer and forward-deployed engineer interview loops, not a whole round. Mid-level candidates are expected to price a request, split its latency, and name streaming, caching, and output caps.

Senior candidates also order the prompt so caching actually hits, price a router against a cascade, and say which cuts risk answer quality. The tutorial assumes you know what a token is, as covered in LLM Fundamentals for Engineers. All prices here are illustrative round numbers. Real prices change often, so read your provider's price sheet.