---
title: 'What is Apache Spark?'
source: 'https://youtube.com/watch?v=KAuIvccwbPY'
video_id: 'KAuIvccwbPY'
date: 2026-07-31
duration_sec: 441
---

# What is Apache Spark?

> Source: [What is Apache Spark?](https://youtube.com/watch?v=KAuIvccwbPY)

## Summary

This video provides a beginner-friendly introduction to Apache Spark, a distributed computing engine. It explains the fundamentals of big data, the MapReduce model, cluster architecture, and compares Spark with Hadoop, using an engaging Indian marriage analogy.

### Key Points

- **The Big Data Explosion** [00:02] — With the rise of the internet and social media, data generated by digital devices exploded. Examples include social media content, IoT sensor data, and web logs. Amazon alone has 300+ million subscribers, and Twitter produces more than 500 million tweets per day.
- **Why Single Computers Fail** [00:31] — Data analysis or AI model training on big data cannot be handled by a single computer. Multiple computers and distributed computing are required.
- **MapReduce Explained** [00:46] — To compute price-to-earnings ratio for 4 million stock records, data can be split into 4 batches. Each batch is processed on a separate computer, then results are aggregated. Splitting is called 'map' and aggregating is called 'reduce'.
- **Clusters, Drivers, and Workers** [01:58] — A collection of computers is a cluster. The main computer that coordinates work is the driver, and individual computers are workers. If a worker crashes, the driver reassigns the task to another computer—this is fault tolerance.
- **Marriage Puri Analogy** [02:54] — Preparing 2000 puris with one chef takes 4 hours; hiring 4 chefs in parallel takes 1 hour. If one chef is absent, others cover the work. A caterer acts as a single point of contact, managing all chefs—similar to a driver managing workers.
- **Hadoop's Disk Bottleneck** [04:17] — Hadoop is a popular distributed computing framework that uses MapReduce. However, it is disk-heavy: reading and writing to disk is slow, making Hadoop slow for huge data processing.
- **Spark's In-Memory Advantage** [04:48] — Matei Zaharia at UC Berkeley AMPL lab invented Apache Spark. It is similar to Hadoop but performs processing in memory, making it extremely fast—10 to 100 times faster than Hadoop.
- **Programming with Spark** [05:14] — In PySpark, you create a SparkSession, read a CSV into a DataFrame, and add a computed column. Spark handles the distribution of tasks automatically, allowing developers to focus on business logic.

### Conclusion

Apache Spark is a distributed computing engine that abstracts the complexity of parallel processing, making big data analysis fast and developer-friendly. It is significantly faster than Hadoop due to in-memory processing and offers simple APIs for Python, Java, and Scala.

## Transcript

intuitive language. Before the explosion of internet, the data generated by digital devices was small enough that it can be handled on a single computer. But then came internet and social media. We saw explosion of data in form of social
media content, IoT census data, web logs and so on. Amazon alone has 300 million plus subscribers. Twitter produces more than 500 million tweets every single day. Imagine you are an engineer working at one of these companies and you want
to do data analysis on this big data or let's say you want to train an AI model. Do you think a single computer will be enough? The answer is no. You need multiple computers and a distributed computing. Let us take this simple case
where you have stocks data 4 million records containing price and EPS and you want to calculate price to earning ratio. Now you can use a single computer. You can write a program in Python, pandas or let's say Java, C++
and you can produce this price to earning ratio for all 4 million records. Now of course you need a powerful computer but what if this 4 million becomes 400 million then single computer won't be able to process it. For
processing 4 million records even for a single computer is going to take a long time. So you can use a simple technique of dividing these records into four of dividing these records into four batches. Then give each batch to each
computer. Okay. So you have total four computers which are producing this price to earning ratio column and then you aggregate the results back. So this approach of splitting a big task into small task is called map. And then after
computation when you aggregate the results that process is called reduce and map and reduce is an essential technique in a distributed computing. If
you have to put this picture in a generic fashion, there is a big task which you are giving it to this computer and then it is dividing that task into n and then it is dividing that task into n number of small task giving that to n
number of computers and then you aggregate the results. Now this collection of computers is called cluster. This main computer which is sort of like a team lead for all these workers is called driver. And these
individual computers are called workers. The beauty of this system is that if let's say computer 2 is processing the data and let's say it crashes, okay, something happens, right? In that case,
this driver program will allocate that task to a different computer. Just like if you're working in a team and let's say one of your team member is sick, your team manager will allocate that work to a different member in your
group. And this is called fault tolerance. Now let me use a very interesting analogy to explain this. In my own marriage, of course, if you look at any Indian marriage, food is a major highlight. Okay. So this is a food
counter and we had to prepare 2,000 puris. Now we initially hired a single chef and he said it will take them 4 hours to prepare this 2,000 puris. And we were like okay that's too much time. We want to prepare these puris in 1
hour. So we hired four different chefs. Okay. And then each chef was preparing 500 puris. So they worked in parallel. So in 1 hour all 2,000 puris were made.
The beauty of this concept is that let's say due to whatever reason if one chef is out let's say they get sick or they're stuck in traffic, they are late in marriage. In that case other chefs can handle this work. So you also get a
fall tolerance here. Nowadays marriages have become advanced and you don't even talk to these four individual chefs. You hire a caterer. So you have a single point of contact and this person will manage those four different chefs. Let's
say if this person gets sick or something, caterer is responsible to arrange a new chef or distributed that work to the remaining chefs. Okay, you're just giving it an order and as a result you get your end product. So this
is distributed computing. It is the process of splitting a large computational task into smaller subtask and executing them in parallel. Okay. Now, Hadoop is a popular framework that is known for distributed computing. It
is essentially doing map and reduce. But one disadvantage with Hadoop is that it is very disk heavy. Okay, it will do everything on disk and reading writing from disk is you know it is very slow. So when you're doing huge data
processing, Hadoop will be slow. So while people were dealing with this while people were dealing with this issue, this person mate Zahara at UC issue, this person mate Zahara at UC Berkeley AMPL lab invented Spark which
is similar to Hadoop. It is a distributed comput engine but the beauty is that it is in memory. It will do the processing in memory due to which it is extremely fast. Okay. Okay, so it's a distributed computing engine designed to
process large scale data quickly and efficiently. If you go back to our previous example to process this particular file, you can write a program in pi spark. So spark is a compute engine. It is available in python, java,
scala, etc. And if you're using spark, you will create a spark session object you will create a spark session object and then you'll simply say stocks csv. Okay, read that in a in a data frame and then create this new column P ratio
which is division of these two. You don't worry about okay sending this batch to this computer and that batch to this different computer. You don't have to worry about it. Okay, Spark will take care of it. So you give this program to
Spark and then while setting up the Spark you will set up the cluster also. Okay, so you'll say okay I need four nodes, five nodes. Nowadays there is serverless compute also where you don't have to worry about any nodes. Okay. But
let's say you are setting up spark cluster with four nodes and there is one driver program. You write this code you give it to spark. Spark will take care of distributing the work and producing the result. Now while processing the
down you as a programmer don't have to worry. Spark will take care of it. Okay. It is like having that caterer which is a single point of contact for you. If take care of it. Okay. So it's a distributed comput engine that abstracts
the complexity of parallelism letting developers focus on business logic. See as a developer here you are only focusing on business logic and that is the beauty of Spark. It is 10 to 100 times faster compared to Hadoop. It is
times faster compared to Hadoop. It is in memory and it has simple APIs. Now Spark is something you can use in uh data bricks Azure data bricks etc. Or you can have your own setup. I will have more videos coming up on data
engineering topics on this channel. So please watch it out. If you have any questions, there is a comment box below. Thank you for watching this video. Thank you for watching this video. [Music]