Why ChatGPT feels broken (and how to fix it)
44sExplains a common user frustration and offers a technical solution, sparking curiosity and engagement.
▶ Play ClipAI Summary
The video provides a concise, expert-level overview of the key architectural considerations for designing a system like ChatGPT, focusing on streaming, GPU scheduling, and fairness. It emphasizes practical engineering choices that ensure a smooth user experience and efficient resource utilization.
Chapters
[00:01]
Time to First Token
Prioritize reducing the time to first token by streaming responses using server-sent events. This avoids the broken feel of waiting for the entire response before displaying anything.
[00:14]
Streaming with Redis Streams
Use server-sent events and a Redis stream to handle streaming smoothly, allowing clients to reconnect and replay messages even when servers change mid-generation.
[00:30]
GPU Scheduling and Batching
Treat GPUs as the bottleneck. Implement a queue in front of workers and batch multiple conversations into a single forward pass to maximize GPU utilization.
[00:43]
Fairness and Tiering
Ensure fairness by metering tokens rather than requests, and prioritize the queue by user tier (e.g., paid vs. free) when capacity is tight.
[00:56]
Senior-Level Answer
Getting these three things right—streaming, GPU scheduling, and fairness—forms the foundation of a senior-level system design answer.
The video succinctly outlines three critical pillars for designing a ChatGPT-like system: streaming for responsiveness, GPU scheduling for efficiency, and fairness for user satisfaction. Mastering these concepts is essential for any senior-level system design discussion.
Mentioned in this Video
Study Flashcards (5)
What is the primary reason to focus on time to first token?
easy
Click to reveal answer
What is the primary reason to focus on time to first token?
easy
Click to reveal answer
Waiting 30 seconds for the entire response feels broken; streaming the first token improves perceived performance.
00:01
What technology is suggested for streaming responses?
easy
Click to reveal answer
What technology is suggested for streaming responses?
easy
Click to reveal answer
Server-sent events from the server to the client.
00:14
How can clients handle server changes mid-generation?
medium
Click to reveal answer
How can clients handle server changes mid-generation?
medium
Click to reveal answer
Use a Redis stream so clients can reconnect and replay messages.
00:14
What is the recommended approach to GPU scheduling?
medium
Click to reveal answer
What is the recommended approach to GPU scheduling?
medium
Click to reveal answer
Put a queue in front of workers and batch many conversations together in a single forward pass.
00:30
How should fairness be implemented in a ChatGPT-like system?
medium
Click to reveal answer
How should fairness be implemented in a ChatGPT-like system?
medium
Click to reveal answer
Meter tokens, not requests, and prioritize the queue by user tier (paid vs. free).
00:43
💡 Key Takeaways
⚖️
Time to First Token is Critical
Highlights a key UX principle: streaming the first token avoids the broken feel of waiting for the full response.
00:01
🔧
Redis Streams for Resilience
Introduces a concrete technology (Redis streams) to handle reconnection and replay, a practical engineering solution.
00:14
🔧
GPU Batching for Efficiency
Emphasizes batching as a way to maximize expensive GPU utilization, a core cost-saving strategy.
00:30
💡
Fairness via Token Metering
Distinguishes between metering tokens vs. requests, a nuanced point for fair resource allocation.
00:43Full Transcript
[00:01] ChatGPT. Here's what I would focus on. The first thing would be the time to first token. See, waiting 30 seconds for the entire response to then flash onto a screen feels really broken. Instead, you need that first token or that first word
[00:14] means it'll use what's called server-sent events from the server to the stream smooth enough even when servers come and go mid-generation. That something like a Redis stream, so clients can reconnect and replay. The
[00:30] second thing I'd focus on is GPU scheduling. See, GPUs are the bottleneck expensive thing. You can put a queue in front of the workers, and the workers can then batch many conversations together in a single forward pass.
[00:43] high and not waste that expensive resource. The third thing I'd focus on is fairness. Like, paid users should feel the product more than free users when capacity is tight. That means metering tokens, not requests, and
[00:56] prioritizing the queue by tier. Get those three things right, and you're on your way to a senior-level answer. Read our complete breakdown on how to design our complete breakdown on how to design ChatGPT at hellonext.com. Link in bio.
