System Design: Why Kafka is Popular

[00:02] use Kafka to handle billions of messages per day? It's not just about scale. Kafka's distributed log design offers something unique, the ability to replay events, decouple services, and absorb traffic spikes. In this video, we'll

[00:16] look at how Kafka achieves this and what trade-offs you are making when you use The main reason companies use Kafka is to decouple their systems. Instead of having services talk directly to each other, they communicate through Kafka.

[00:30] This means producers and consumers can evolve independently, and Kafka absorbs traffic spikes that would otherwise overwhelm your systems. It also enables things go wrong. So, how does this distributed log

[00:45] When you send a message to Kafka, it gets written to a partition, which is basically append-only log files sitting on disk. These partitions live on servers called brokers, and when you put multiple brokers together, you get a

[00:59] Kafka cluster. Partitions are organized into topics, which are categories for your messages. You might have a topic for payments, another for user clicks, and another for video uploads. Producers write messages into topics, and

[01:13] consumers read them. Every message contains a key, a value, a timestamp, and sometimes headers for metadata. The key determines which partition your message lands in. If you send multiple messages with the same key, they will

[01:27] always go to the same partition and stay in order. When you don't provide a key, Kafka spreads messages around to balance the load across partitions. A single broker on modern hardware can handle hundreds of thousands of messages

[01:40] per second and store as much data as your disk can hold. In practice, though, the broker will usually hit network bandwidth limits before CPU or disk bandwidth limits before CPU or disk becomes the bottleneck.

[01:55] best way to code with AI agents. Too often, agents write code that's almost right, leaving developers stuck debugging instead of shipping. Warp is different. Ranked top of terminal bench and sweet bench verified, Warp's agent

[02:09] understands your context and writes production-ready code out of the box. Prompt, review, and refine, all in one interface. No context switching, no wasted time. You stay in control. And it pays off. On average, users are saving

[02:23] over an hour a day with Warp. Download Warp by clicking the link in the Warp by clicking the link in the description. whether your system scales gracefully or falls apart under load. Pick the wrong

[02:37] partition key, and you'll end up with hot partitions, where one partition gets hammered while the others sit idle. Imagine you're building a streaming service and you partition by movie ID. Everything works fine until Friday night

[02:51] when a blockbuster drops, and suddenly millions of users are streaming the same movie. All these events hit the same partition, and your system starts choking. The solution is to use compound keys. Combine the movie ID with a hash

[03:05] of the user ID, and now events for that blockbuster get spread across multiple stay in order. There are other partitioning schemes, too, each with its own trade-offs. For example, time-based partitions work

[03:20] great for log data because they make retention policies simple, but they complicate real-time aggregation. Consumers track their progress through partitions using offsets, which are basically bookmarks to tell you which

[03:32] message you last processed. They save these offsets back to Kafka periodically, so if they crash, they know exactly where to pick up again. The timing of these commits matters. Commit too early, and you might lose messages

[03:45] if you crash. Commit too late, and you might process the same message twice. Consumer groups let multiple consumers work together, with Kafka making sure each message gets processed by exactly one consumer in the group. If a consumer

[03:59] fails, Kafka reassigns its partition to the surviving consumers through rebalancing. It handles most failure scenarios without any manual intervention. Kafka offers three delivery guarantees.

[04:12] At most once is fast, but might lose messages. At least once ensures no loss, but might produce duplicates. Exactly once is possible, but is complicated to set up and runs slower. Durability comes with replication. Every

[04:27] partition has one leader that handles all reads and writes, plus several followers that copy everything the leader does. If the leader fails, one of the followers takes over. Most production systems run with three

[04:39] replicas, which means you can lose a broker and still have backup. You can configure Kafka to wait for all active replicas to acknowledge writes before considering them successful. This gives you maximum safety, but slow things

[04:52] down. With three replicas, you can typically survive one broker failure without losing data. These mechanics enable powerful patterns in production. At Uber, location updates of millions of drivers reportedly flow

[05:06] through Kafka to calculate search pricing in real time. They partition geographically, so each region can scale independently. Some companies use Kafka as their source of truth for data. Instead of updating database records

[05:19] directly, they append every state change as an event to Kafka. Want the current state? Replay the events. This pattern, called event sourcing, gives you a complete audit trail of everything that happened in your system.

[05:33] But Kafka isn't the right choice for every use case. It optimizes for throughput, not latency. The batching and buffering that enables high throughput adds some delay, making it unsuitable for request-response

[05:45] Kafka only guarantees order within a single partition, not across an entire topic. If you absolutely need global ordering, you're stuck with a single partition, which kills your ability to parallelize. Most systems work around

[05:59] this by accepting partial ordering. Exactly once processing requires careful setup on both producer and consumer sides, but when you need it for financial transactions or critical data pipelines, the complexity is worth it.

[06:14] producers from consumers, letting them evolve independently without breaking each other. Traffic spikes that would overwhelm a direct connection get absorbed by the log. When something goes wrong in production, you can replay

[06:27] events to see exactly what happened. This power comes with a cost. Kafka adds significant operational complexity to your stack. Ready to ace your next technical interview? Join our community where we

[06:40] offer comprehensive courses on system design, coding, behavioral questions, machine learning, and object-oriented design. Learn more at bytebytego.com.