---
title: 'How to Design ChatGPT'
source: 'https://youtube.com/watch?v=KDLfKu1Pgag'
video_id: 'KDLfKu1Pgag'
date: 2026-08-04
duration_sec: 72
---

# How to Design ChatGPT

> Source: [How to Design ChatGPT](https://youtube.com/watch?v=KDLfKu1Pgag)

## Summary

The video provides a concise, expert-level overview of the key architectural considerations for designing a system like ChatGPT, focusing on streaming, GPU scheduling, and fairness. It emphasizes practical engineering choices that ensure a smooth user experience and efficient resource utilization.

### Key Points

- **Time to First Token** [00:01] — Prioritize reducing the time to first token by streaming responses using server-sent events. This avoids the broken feel of waiting for the entire response before displaying anything.
- **Streaming with Redis Streams** [00:14] — Use server-sent events and a Redis stream to handle streaming smoothly, allowing clients to reconnect and replay messages even when servers change mid-generation.
- **GPU Scheduling and Batching** [00:30] — Treat GPUs as the bottleneck. Implement a queue in front of workers and batch multiple conversations into a single forward pass to maximize GPU utilization.
- **Fairness and Tiering** [00:43] — Ensure fairness by metering tokens rather than requests, and prioritize the queue by user tier (e.g., paid vs. free) when capacity is tight.
- **Senior-Level Answer** [00:56] — Getting these three things right—streaming, GPU scheduling, and fairness—forms the foundation of a senior-level system design answer.

### Conclusion

The video succinctly outlines three critical pillars for designing a ChatGPT-like system: streaming for responsiveness, GPU scheduling for efficiency, and fairness for user satisfaction. Mastering these concepts is essential for any senior-level system design discussion.

## Transcript

ChatGPT. Here's what I would focus on. The first thing would be the time to first token. See, waiting 30 seconds for the entire response to then flash onto a screen feels really broken. Instead, you need that first token or that first word
means it'll use what's called server-sent events from the server to the stream smooth enough even when servers come and go mid-generation. That something like a Redis stream, so clients can reconnect and replay. The
second thing I'd focus on is GPU scheduling. See, GPUs are the bottleneck expensive thing. You can put a queue in front of the workers, and the workers can then batch many conversations together in a single forward pass.
high and not waste that expensive resource. The third thing I'd focus on is fairness. Like, paid users should feel the product more than free users when capacity is tight. That means metering tokens, not requests, and
prioritizing the queue by tier. Get those three things right, and you're on your way to a senior-level answer. Read our complete breakdown on how to design our complete breakdown on how to design ChatGPT at hellonext.com. Link in bio.