[00:01] ChatGPT. Here's what I would focus on. The first thing would be the time to first token. See, waiting 30 seconds for the entire response to then flash onto a screen feels really broken. Instead, you need that first token or that first word
[00:14] means it'll use what's called server-sent events from the server to the stream smooth enough even when servers come and go mid-generation. That something like a Redis stream, so clients can reconnect and replay. The
[00:30] second thing I'd focus on is GPU scheduling. See, GPUs are the bottleneck expensive thing. You can put a queue in front of the workers, and the workers can then batch many conversations together in a single forward pass.
[00:43] high and not waste that expensive resource. The third thing I'd focus on is fairness. Like, paid users should feel the product more than free users when capacity is tight. That means metering tokens, not requests, and
[00:56] prioritizing the queue by tier. Get those three things right, and you're on your way to a senior-level answer. Read our complete breakdown on how to design our complete breakdown on how to design ChatGPT at hellonext.com. Link in bio.