1. Self-hosting open-weight models

A hospital network wants a contract Q&A assistant for its legal team. The contracts include clauses about patient data, and the network's rule is simple: those documents never leave its own cloud account. The interviewer asks the question this section is built around: the customer's data can't leave their network, so how would you deploy the model, and when is self-hosting worth it?

Self-hosting means downloading an open-weight model, one whose trained parameter files are published, and running it on GPUs the customer controls, instead of sending prompts to a vendor's large language model (LLM) API.

This comes up in forward-deployed, enterprise, and regulated-industry AI roles, usually after the application design is settled. In product AI roles a hosted API is assumed, and the question is rare. Data control is the usual reason to self-host. Cost alone rarely justifies it unless volume is high and steady: at low or bursty traffic, a hosted API, or a provider model running inside the customer's own cloud, is cheaper and simpler.

For a mid-level candidate, the bar is ruling out the public API, estimating how much GPU memory a large model needs, and naming an inference server. Senior candidates are also expected to name the managed option that runs inside the customer's cloud, compute when self-hosting pays for itself, and gate precision and model upgrades on an eval set. Nobody expects you to explain how the inference server works inside.

The tutorial assumes you know what tokens and context windows are, from LLM Fundamentals for Engineers, and can compute a hosted API bill from tokens and price, from Cost and Latency. All prices here are illustrative round numbers, because real ones change every few months.