[00:02] intuitive language. Before the explosion of internet, the data generated by digital devices was small enough that it can be handled on a single computer. But then came internet and social media. We saw explosion of data in form of social
[00:16] media content, IoT census data, web logs and so on. Amazon alone has 300 million plus subscribers. Twitter produces more than 500 million tweets every single day. Imagine you are an engineer working at one of these companies and you want
[00:31] to do data analysis on this big data or let's say you want to train an AI model. Do you think a single computer will be enough? The answer is no. You need multiple computers and a distributed computing. Let us take this simple case
[00:46] where you have stocks data 4 million records containing price and EPS and you want to calculate price to earning ratio. Now you can use a single computer. You can write a program in Python, pandas or let's say Java, C++
[01:01] and you can produce this price to earning ratio for all 4 million records. Now of course you need a powerful computer but what if this 4 million becomes 400 million then single computer won't be able to process it. For
[01:16] processing 4 million records even for a single computer is going to take a long time. So you can use a simple technique of dividing these records into four of dividing these records into four batches. Then give each batch to each
[01:30] computer. Okay. So you have total four computers which are producing this price to earning ratio column and then you aggregate the results back. So this approach of splitting a big task into small task is called map. And then after
[01:45] computation when you aggregate the results that process is called reduce and map and reduce is an essential technique in a distributed computing. If
[01:58] you have to put this picture in a generic fashion, there is a big task which you are giving it to this computer and then it is dividing that task into n and then it is dividing that task into n number of small task giving that to n
[02:13] number of computers and then you aggregate the results. Now this collection of computers is called cluster. This main computer which is sort of like a team lead for all these workers is called driver. And these
[02:26] individual computers are called workers. The beauty of this system is that if let's say computer 2 is processing the data and let's say it crashes, okay, something happens, right? In that case,
[02:39] this driver program will allocate that task to a different computer. Just like if you're working in a team and let's say one of your team member is sick, your team manager will allocate that work to a different member in your
[02:54] group. And this is called fault tolerance. Now let me use a very interesting analogy to explain this. In my own marriage, of course, if you look at any Indian marriage, food is a major highlight. Okay. So this is a food
[03:07] counter and we had to prepare 2,000 puris. Now we initially hired a single chef and he said it will take them 4 hours to prepare this 2,000 puris. And we were like okay that's too much time. We want to prepare these puris in 1
[03:23] hour. So we hired four different chefs. Okay. And then each chef was preparing 500 puris. So they worked in parallel. So in 1 hour all 2,000 puris were made.
[03:36] The beauty of this concept is that let's say due to whatever reason if one chef is out let's say they get sick or they're stuck in traffic, they are late in marriage. In that case other chefs can handle this work. So you also get a
[03:50] fall tolerance here. Nowadays marriages have become advanced and you don't even talk to these four individual chefs. You hire a caterer. So you have a single point of contact and this person will manage those four different chefs. Let's
[04:04] say if this person gets sick or something, caterer is responsible to arrange a new chef or distributed that work to the remaining chefs. Okay, you're just giving it an order and as a result you get your end product. So this
[04:17] is distributed computing. It is the process of splitting a large computational task into smaller subtask and executing them in parallel. Okay. Now, Hadoop is a popular framework that is known for distributed computing. It
[04:34] is essentially doing map and reduce. But one disadvantage with Hadoop is that it is very disk heavy. Okay, it will do everything on disk and reading writing from disk is you know it is very slow. So when you're doing huge data
[04:48] processing, Hadoop will be slow. So while people were dealing with this while people were dealing with this issue, this person mate Zahara at UC issue, this person mate Zahara at UC Berkeley AMPL lab invented Spark which
[05:00] is similar to Hadoop. It is a distributed comput engine but the beauty is that it is in memory. It will do the processing in memory due to which it is extremely fast. Okay. Okay, so it's a distributed computing engine designed to
[05:14] process large scale data quickly and efficiently. If you go back to our previous example to process this particular file, you can write a program in pi spark. So spark is a compute engine. It is available in python, java,
[05:30] scala, etc. And if you're using spark, you will create a spark session object you will create a spark session object and then you'll simply say stocks csv. Okay, read that in a in a data frame and then create this new column P ratio
[05:44] which is division of these two. You don't worry about okay sending this batch to this computer and that batch to this different computer. You don't have to worry about it. Okay, Spark will take care of it. So you give this program to
[05:57] Spark and then while setting up the Spark you will set up the cluster also. Okay, so you'll say okay I need four nodes, five nodes. Nowadays there is serverless compute also where you don't have to worry about any nodes. Okay. But
[06:10] let's say you are setting up spark cluster with four nodes and there is one driver program. You write this code you give it to spark. Spark will take care of distributing the work and producing the result. Now while processing the
[06:23] down you as a programmer don't have to worry. Spark will take care of it. Okay. It is like having that caterer which is a single point of contact for you. If take care of it. Okay. So it's a distributed comput engine that abstracts
[06:39] the complexity of parallelism letting developers focus on business logic. See as a developer here you are only focusing on business logic and that is the beauty of Spark. It is 10 to 100 times faster compared to Hadoop. It is
[06:55] times faster compared to Hadoop. It is in memory and it has simple APIs. Now Spark is something you can use in uh data bricks Azure data bricks etc. Or you can have your own setup. I will have more videos coming up on data
[07:08] engineering topics on this channel. So please watch it out. If you have any questions, there is a comment box below. Thank you for watching this video. Thank you for watching this video. [Music]