[00:00] The second design choice that gives Kafka its performance advantage is its focus on efficiency. Kafka moves a lot of data from network to disk and then from disk to network. It is critically important to eliminate excess copy when moving pages and pages of data between the disk and the
[00:16] network. This is where zero copy principle comes into the picture. Modern Unix operating systems are highly optimized for transferring data from disk to network without copying data excessively. Let's dive deeper into see how this is done. First, we look at how Kafka sends a page of data on disk
[00:32] to the consumer when zero copy is not used at all. First, the data is loaded from disk to the OS cache. Second, the data is copied from the OS cache into the Kafka application. Third, the data is copied
[00:44] from Kafka to the socket buffer. And fourth, the data is copied from the socket buffer to the network interface card buffer. And finally, the data is sent over the network to the consumer. Now this is clearly inefficient. There are four copies and two system calls.
[00:58] Now let's compare this to zero copy. The first step is the same. The data page is loaded from the disk to the OS cache. With zero copy, the Kafka application uses a system call called sendfile to tell the operating system to directly copy the data from the OS cache to the network interface card buffer.
[01:14] In this optimized path, the only copy is from the OS cache into the network card buffer. With a modern network card, this copying is done with DMA. DMA is Direct Memory Access. When DMA is used, the CPU is not involved, making it even more
[01:30] efficient. Now to recap, sequential I.O. and zero copy principle are the cornerstones to Kafka's high performance. Kafka uses other techniques to squeeze every ounce of performance out of the modern hardware, but these two are the most important in our view.