Kafka's Secret: Zero Copy Explained
45sClear, step-by-step breakdown of a complex concept that appeals to developers and tech enthusiasts.
▶ Play ClipAI Summary
This video explains two fundamental design choices that give Apache Kafka its high performance: sequential I/O and the zero-copy principle. It contrasts the traditional data transfer path with Kafka's optimized approach, detailing how system calls and DMA reduce CPU involvement and copying overhead.
[00:16]
Zero copy principle
Modern Unix OS optimizes data transfer from disk to network, avoiding excess copying.
[00:32]
Without zero copy
Data goes: disk to OS cache → Kafka application → socket buffer → NIC buffer → network. Four copies and two system calls, inefficient.
[01:14]
With zero copy (sendfile)
Sends data directly from OS cache to network card buffer using sendfile system call, with only one copy.
[01:14]
DMA efficiency
Modern NICs use Direct Memory Access (DMA), copying without CPU involvement, boosting performance further.
[01:30]
Cornerstones
Sequential I/O and zero copy are the most important techniques, though Kafka uses others too.
Study Flashcards (4)
What are the two cornerstones of Kafka's high performance?
easy
Click to reveal answer
What are the two cornerstones of Kafka's high performance?
easy
Click to reveal answer
Sequential I/O and zero copy principle.
01:30
How many copies and system calls does a traditional path (without zero copy) involve?
medium
Click to reveal answer
How many copies and system calls does a traditional path (without zero copy) involve?
medium
Click to reveal answer
Four copies and two system calls.
00:44
What system call does Kafka use to enable zero copy?
medium
Click to reveal answer
What system call does Kafka use to enable zero copy?
medium
Click to reveal answer
sendfile
01:14
What does DMA stand for and why does it improve efficiency?
hard
Click to reveal answer
What does DMA stand for and why does it improve efficiency?
hard
Click to reveal answer
Direct Memory Access; it allows copying without CPU involvement.
01:14
💡 Key Takeaways
⚖️
Sequential I/O and zero copy as key principles
Clearly identifies the two main technical pillars that give Kafka its performance edge.
01:30Full Transcript
[00:00] The second design choice that gives Kafka its performance advantage is its focus on efficiency. Kafka moves a lot of data from network to disk and then from disk to network. It is critically important to eliminate excess copy when moving pages and pages of data between the disk and the
[00:16] network. This is where zero copy principle comes into the picture. Modern Unix operating systems are highly optimized for transferring data from disk to network without copying data excessively. Let's dive deeper into see how this is done. First, we look at how Kafka sends a page of data on disk
[00:32] to the consumer when zero copy is not used at all. First, the data is loaded from disk to the OS cache. Second, the data is copied from the OS cache into the Kafka application. Third, the data is copied
[00:44] from Kafka to the socket buffer. And fourth, the data is copied from the socket buffer to the network interface card buffer. And finally, the data is sent over the network to the consumer. Now this is clearly inefficient. There are four copies and two system calls.
[00:58] Now let's compare this to zero copy. The first step is the same. The data page is loaded from the disk to the OS cache. With zero copy, the Kafka application uses a system call called sendfile to tell the operating system to directly copy the data from the OS cache to the network interface card buffer.
[01:14] In this optimized path, the only copy is from the OS cache into the network card buffer. With a modern network card, this copying is done with DMA. DMA is Direct Memory Access. When DMA is used, the CPU is not involved, making it even more
[01:30] efficient. Now to recap, sequential I.O. and zero copy principle are the cornerstones to Kafka's high performance. Kafka uses other techniques to squeeze every ounce of performance out of the modern hardware, but these two are the most important in our view.
