---
title: 'Kafka Simply Explained'
source: 'https://youtube.com/watch?v=d-NJTcTvyhw'
video_id: 'd-NJTcTvyhw'
date: 2026-07-31
duration_sec: 1194
---

# Kafka Simply Explained

> Source: [Kafka Simply Explained](https://youtube.com/watch?v=d-NJTcTvyhw)

## Summary

This video explains Apache Kafka in simple, analogy-driven terms, starting with the problems of tightly coupled synchronous microservices in a food delivery app. It then shows how event-driven architecture with a message buffer solves those problems, and covers Kafka's core concepts, real-world use cases, and the key differences between Kafka and traditional message queues like RabbitMQ.

### Key Points

- **Order processing in a food delivery app** [00:01] — When a customer places an order, the backend order service takes payment, stores the order in a database, and then makes HTTP calls to restaurant, delivery, analytics, and notification services.
- **Synchronous blocking calls** [01:30] — The Python code example shows the order service making HTTP calls and waiting for each response before proceeding — classic synchronous, blocking processing, which creates tightly coupled architecture.
- **Failure under heavy load** [02:15] — During a spike like a World Cup final, the tightly coupled synchronous architecture breaks down because one slow service creates congestion and blocks every other call, similar to a traffic junction.
- **Two key issues with tight coupling** [03:34] — First, one slow or down service can bring down the entire system. Second, adding a new service (e.g., loyalty points) requires a code change in the main codebase and the new service still has to wait for all other calls.
- **Event-driven alternative** [04:32] — Instead of calling dependent services directly, the order service writes an event to a message buffer. Each consumer picks up the event when free, like a postman dropping mail in a postbox.
- **Offsets enable independent consumption** [05:38] — Each consumer maintains its own offset marker in the message stream, so different services can read the same events independently and at their own pace.
- **Kafka is a distributed event streaming platform** [07:13] — Apache Kafka is not just a message queue; it is an open-source distributed event streaming platform where producers write events to topics and consumers pull messages at their convenience.
- **Decoupling benefits** [09:58] — The order service simply writes to one topic and doesn't need to know who consumes it. If analytics is down, other services continue unaffected, and analytics can catch up later.
- **Replay events for recovery** [10:12] — Kafka persists events to disk with configurable retention (days or years). At Bloomberg, teams replayed events after production bugs to reprocess and backfill data.
- **What is event streaming?** [11:42] — Event streaming is the practice of capturing data in real time from sources like databases, sensors, mobile devices, and clickstreams in the form of streams of events.
- **Real-world use cases** [12:08] — Netflix uses Kafka for recommendations, billing, and analytics; Tesla uses it for IoT sensor data and real-time monitoring; credit card companies use it for fraud detection and notifications.
- **Industry adoption** [14:17] — Kafka was invented at LinkedIn, which processes 7 trillion messages per day; Uber processes 1 trillion messages per day; 80% of Fortune 100 companies use Kafka.
- **Kafka vs RabbitMQ** [14:54] — RabbitMQ deletes a message once consumed and uses a push-based model. Kafka retains messages for a configured period and uses pull-based consumption, allowing multiple consumers to read the same message via offsets.
- **When to use Kafka (and when not)** [18:31] — Use Kafka for real-time data pipelines and event-driven processing with huge volumes. Avoid it for complex SQL queries, small data volumes (less than a thousand messages a day), or simple request-response patterns.

### Conclusion

Kafka solves the pain points of synchronous microservice communication by providing a durable, replayable event log that decouples producers from consumers, but it should only be used when the scale and use case justify it.

## Transcript

is Kafka in a super intuitive language. We all use Uber Eats, Zomemetto, etc. for food delivery. Have you ever wondered how things work behind the scene? Let's say you are a back-end engineer working at Zomemetto. Now,
somebody is placing an order of paneeta masala through their app. The order will go to the back end. Let's call it order processing service. It will take a payment and it will insert a record in a database for your order. But does it
stop there? No. The system is much more complex than what you think. It will uh send an order to a given restaurant. Let's say you order through this gulati spice market. It needs to send the order to that restaurant. And let's assume
that restaurant has some kind of API or HTTP call. So it will make a call. It HTTP call. So it will make a call. It will also uh call delivery service because it needs to schedule the driver etc. so that they can deliver the food
once it is prepared at the restaurant. It will also make another call to analytic service, another call to notification service so that you can get notification service so that you can get an SMS on your phone regarding your
order. All of these are microservices. Okay. So assume that these are microservices and this is the back end for your app which is calling all these microservices in a code which may look something like this. So this is a Python
code and it is making bunch of HTTP calls to all these services. Now watch this. What it is doing is it is making a call to restaurant service and it is waiting for the response code. Then it's making a call to a delivery service and
waiting for a response. So this is a blocking call. It is a synchronous processing. Okay. So this is called tightly coupled synchronous tightly coupled synchronous architecture. Now let's say Zomemetto is
in its early days. The programmers created this kind of architecture. It is working fine. But all of a sudden now Zomemetto has become more popular. There are more and more users using their service. And there is a world cup final
match and everybody every single user of Zomemetto is now ordering food through their app and their load has increased. Now they are getting millions of request from those users. Okay. And this architecture which was tightly coupled
synchronous architecture was not designed by keeping in mind this kind of designed by keeping in mind this kind of a huge workload. Okay. So what happens is basically since this is a blocking call remember that R2 has to wait for
R1. What will happen is similar to a situation at a traffic junction. You might have seen this kind of traffic situation where there is a huge traffic coming in and let's say there is outgoing traffic to these three
different roads and there is some kind of congestion. Let's say there is some kind of road work going on and there is a congestion here. So due to this congestion the traffic will get blocked here and these people will also get
impacted. Okay. So this is number one issue in a tightly coupled synchronous architecture. If something goes wrong with one service, right? One call is slow or let's say one service is down everybody else will be impacted. The
second issue with tightly coupled synchronous architecture is that when you want to add a new service, let's say you want to add a new service for loyalty point service. If you're using Zomemet or Uber Eats, you will get this
loyalty points, right? Let's say they decided to add that feature later on. decided to add that feature later on. Now you have to add one more call. Okay? So it requires a code change into your main code base and also this R5 has to
wait for all these services. All right. So there are these two key issues with tightly coupled synchronous architecture. Number one, one slow service can bring down the entire system. Number two, there is little
flexibility because you are making a core change every time you are adding new service whether it's loyalty service, notification service and so on. So how about we design an alternative architecture where this part remains
same. Okay, a customer places an order, order will go to database, you take a payment and that's it. But instead of calling all those four dependent services, you write an event to a message buffer. Okay, it's like a
message cue where you will write an event saying that order is placed, customer ID is this, order amount is this. And then your restaurant service will simply monitor this. It's like your postman delivers a mail to your post
box, right? And you will get it whenever you are free. So your postman is not blocked right. They are not waiting. They will put the mail in your post box and then you will get it whenever you are free. Same thing happens in email.
Somebody sends you an email, it is in your inbox and you consume it whenever you are free. So here this one is a producer. Okay. So this is producing the events and this one is a consumer. It is consuming those events. And once this
event is consumed right see orders are coming continuously second order will come third order they're coming continuously they will pile up and this one once it is consumed they will have some kind of offset right so right now
the current marker is here once it is consumed they will move the marker to here okay and same thing can be done by other services they will have their own other services they will have their own different markers okay so let's say this
delivery service when restaurant service is at this delivery service will be at this and once that is consumed they will shift this pointer here and so on. Okay, tomorrow if loyalty service needs to be added, it's easy. Nothing changes here,
right? This order processing back end is putting a message in this buffer and you can have any number of consumer consuming the messages from this buffer.
It is like I want to play a volleyball. I love playing volleyball by the way and I have my friends who come to play volleyball with me. Now one option I have is let's say tomorrow weather is nice at 4:00 I want to play volleyball I
can call all of these friends individually but that's not what we do nowadays what we do instead yes you guessed it right you create a volleyball guessed it right you create a volleyball group you post a message and all these
people will consume the message whenever they are free so this message buffer that we referred to initially is nothing but Kafka now you'll think Okay, so
basically it's a message Q. Well, not quite right. We will look into the difference between message Q and Kafka later on. But Kafka is actually an later on. But Kafka is actually an open-source distributed event streaming
platform. Okay, you have to remember this. It is a distributed event streaming platform. Apache Kafka, it's an open-source platform. So in terms of code, you will have the producer. So our producer is that order service. Correct?
So after the order is saved into the database, it will simply send an event. Okay, it will send an event. This is an event and this is the topic. We will see what exactly is the topic. But topic is an entity related to your business
scenario. Here is an order, right? There can be another topic called payments. So can be another topic called payments. So you are pushing an event in uh Kafka using this producer. Okay. And once you do this, see this is asynchronous. You
are not waiting for people to consume. You push it and then you are done. So this way this code is pretty safe. It is not going to be impacted by what what is going on in consumer side. On the
consumer side, see this is a restaurant service. You will create Kafka consumer and this is Python code by the way. You will create this Kafka consumer and you
will pull the messages. So it's like an event loop, right? For messaging consumer infinite loop. So you're constantly checking. It's like I have a post box. I'm constantly checking every five minute. Okay. Do I have a postal
mail? Do I have a postal mail? I'm checking every half an hour, let's say. So this is that for loop. I'm checking. Okay. Do we have a message? Do we have a message? If you have a message, you will get this message here. And same thing
for other consumers, right? This is delivery service. Similar code. So there delivery service. Similar code. So there is producer, there is consumer. Now just is producer, there is consumer. Now just to highlight the differences uh between
both the approaches without Kafka when you're doing tightly coupled synchronous you're doing tightly coupled synchronous architecture sync HTTP calls your order architecture sync HTTP calls your order service calls four services directly. If
one is down the whole flow will break. Adding a new consumer means changing the order service code. Okay. and latency increases with each downstream call because you're waiting right, R2, R3, you're waiting. So the R5 will have to
wait until all the four calls are done. Whereas if you're using Kafka, order service will just write to one topic. It's like you have a message buffer, you are putting your event and that's it.
And then any number of services can consume without order service knowing. Order service doesn't need to know who is consuming my messages. You're just is consuming my messages. You're just pushing the messages to Kafka. And if
analytics is down, others still work. Analytics can catch up later and you can replay the event. See, this is very important. In Kafka, you can replay the old events. So events are persisted to disk and you can configure when you are
configuring your Kafka cluster. You can set up a retention policy. You can say last 5 days, last 1 year, last 2 years. I want to maintain all the messages. Okay? And you will be able to replay those messages. When I was at Bloomberg,
we used to have uh this kind of event streaming platform and let's say if some uh something fails in production, what we used to do is we will go back and we will replay all those events. So let's say there is a production bug due to
say there is a production bug due to which all the processing in last half an hour failed but we still have those events in Kafka. So you deploy new software and then whatever events occurred in last half an hour you can
replay them because events are persisted to disk and you can use it for reprocessing backfilling and so on. Okay. So Kafka is a distributed event streaming platform. If somebody asks you this question in an
interview, this should be your answer. It's a distributed event streaming platform where applications write events and other applications read them with events stored durable and in order. Okay, in case if you're wondering what
exactly is event streaming, well, it's a practice of capturing data in real time practice of capturing data in real time from sources like databases, sensors. Okay, this is used a lot in IoT devices by the way, mobile devices, click
stream, software applications in the form of streams of events. So here are the real world use cases. Number one is Netflix. So I love K drama on Netflix
and this is the series I recently watched Born Appetite Your Majesty. So when I click on it or when I say thumbs up etc. It will push a message on Kafka and there will be a recommendation engine. There will be a IML algorithm
which will listen to that event stream and it will get updated. There is a billing system which will track the usage of this. There is analytics services you know which will track okay how many people are watching Kdrama, how
many people have given thumbs up to born appetite etc and so on. Second use case appetite etc and so on. Second use case is autonomous driving by Tesla. So Tesla cars can drive on their own and they have lot of sensors installed. So every
moment when they're driving these sensors are generating the IoT data. sensors are generating the IoT data. It's a stream of IoT data which can be It's a stream of IoT data which can be GPS speed battery all kind of metrics on
GPS speed battery all kind of metrics on on this and that gets pushed to Kafka and then there can be a real-time monitoring dashboard which will consume these events. predictive maintenance alerts, long-term analytic storage.
Okay. The third use case is credit card transaction. See most of the credit card transaction companies have this Kafka setup where when you make a transaction the message will will be pushed to Kafka and there will be fraud detection model
and there will be fraud detection model which will run and if the transaction is fraud within few seconds or few milliseconds you will get a message that oh did you do this transaction there can be user notification service this is a
separate service okay micros service and it will send you a SMS saying that you have made this transaction in this amount at this place and there is a which can be capturing all these transactions and the this service will
be used by some kind of analytics dashboard to view overall analytics on credit card usage. If you talk about industry adoption, industry adoption, Kafka was invented at LinkedIn. Okay,
LinkedIn is a company that invented it and they process 7 trillion messages every day. Uber processes 1 trillion messages every day. You see trillion, do you even realize trillion is a huge number folks? Trillion means 1,000
number folks? Trillion means 1,000 billion. 1 billion means 1,000 million. billion. 1 billion means 1,000 million. 1 million means 10 lakh. Okay? So it's 1 million means 10 lakh. Okay? So it's humongous volume of data that all these
humongous volume of data that all these companies are processing and 80% of companies are processing and 80% of Fortune 100 companies use Kafka today. If you have worked in software engineering, you will have a natural
question. What is the difference between Kafka and traditional message cues like Rabbit MQ? See in the architecture that we saw before let's say here if we are using rabbit Q what will happen is when restaurant service reads this particular
restaurant service reads this particular message this message will be deleted other consumers won't be able to see this message. Okay. And even restaurant service let's say after some time if it wants to replay it it won't be able to
replay it. Okay. It's like you're going to a restaurant and the chef has served to a restaurant and the chef has served you samosa, you eat samosa, samosa is gone. It's like message is consumed once it's gone. There are ways by the way in
it's gone. There are ways by the way in Rabbit MQ where you can enable multiple uh consumers. But the fundamental design is this. It's oneonone message is consumed. It's deleted. You can't replay it. There can be five different
consumers consuming it. Okay. So message Q is more like I want to send a letter to my brother. I will send it via postal mail. Okay. So there is a letter. I send
it via postal mail. My brother will read it and then he will throw this in a garbage bin. Okay. So let's say he throws this in a garbage bin. So this later is gone. So that is Rabbit MQ. Kafka on the other hand which is a
distributed event streaming platform is like me uploading a video on YouTube. Now if my brother wants to learn data bras he can watch the video he can replay it when he watches the video doesn't get deleted right video just
stays there for a long time and if he wants to replay it rewind it he can do all of that not just my brother but millions of other people on internet can see that video and they can replay and rewind. Okay, so here are the key
differences between message Q and event streaming platform. Message retention. In message Q, messages are deleted once consumed. Q will act as a buffer whereas in event stream processing, messages are retained for a configurable period
regardless of consumption. Okay, so this acts as a durable log. Okay, some people acts as a durable log. Okay, some people call Kafka a durable log as well. The consumption model in message Q is pushbased. Okay. So broker will push the
message to consumer. We'll look into this broker concept later on. And messages are consumed once. There is a little star because there is a way in behavior. But this is a fundamental behavior. And then in Kafka it is pull
based. Consumers pull messages. multiple consumers can independently read the same message by maintaining their own offset. So every consumer will maintain the offset. Okay. So I have read let's say first three messages so my pointer
will be on a third message. Somebody else has read only first message so because first one is read so it will be on a second. Somebody else has read 10 messages then their own pointer will be at 11th position. So there is a specific
offset. We will go over uh this concept in detail as well later on. Now typical use cases for Rabbit MQ is background jobs, email sending queue, task
distribution. Whereas Kafka you use it for realtime data injection, IoT sensor, click stream and in all these cases the volume of events is humongous. It's like
millions and trillions of events. Whereas if you have like thousands of Whereas if you have like thousands of events, you better use rabbit MQ. Okay. So when to use Kafka? Well, realtime data pipelines, event-driven
processing. See, you will understand some of these concepts as we do some hands-on practice. Okay. So if you're not getting all these jargon for now, don't worry. When not to use Kafka. Well, when you need complex SQL queries,
don't use Kafka as your database. Okay? If you have a database thousands and thousands of records, you want to perform complex SQL queries, don't use Kafka there. If you have small data volumes, less than thousand messages a
day, you're better off using Rabbit MQ and your request response patterns, right? So there are cases where having tightly coupled architecture actually makes sense. Okay, so direct
calling REST API can be better. If you're a software engineer, you will understand my point. In your coding, right, you would have done this so many times where you are calling a rest API. You wait till you get the response and
then you do further processing. It is perfectly fine. You should not throw that away because now you know this new concept behind Kafka. Okay? Use it only concept behind Kafka. Okay? Use it only when it makes sense.