OpenAI System Design Example | Real Interview Question

Problem

Design a stateless proxy that forwards prompts to an LLM and streams token-by-token responses to the client

Stream tokens under 500ms latency across 50K concurrent SSE connections
Handle mid-stream breaks and JWT expiry without server-side conversation storage
Enforce per-user rate limits and token budgets to cap API costs

Key Test Points

Server-Sent Events
Grace-period auth validation
Sliding-window rate limiting

1🤖 What is an AI Chatbot App?Chapter 1

An AI chatbot app is a web application that lets users have real-time conversations with a large language model (LLM). The user types a message, the system sends it to an AI model, and the response streams back token-by-token so the user sees the answer being "typed out" progressively. Products like ChatGPT, Claude, and Gemini are well-known examples of this pattern.

2RequirementsChapter 2

3Data model designChapter 3

4API designChapter 4

5High-level designChapter 5

6Deep divesChapter 6

7Other ConsiderationsChapter 7

8Interviewer expectationsChapter 8

Your progressLogin to save progress

Spot an issue or have a suggestion?