Data Engineer Roadmap 2026: How I'd Learn Data Engineering in 2026

[00:02] data. Good quality data is enabled by solid data engineering infrastructure in any company. Data engineer is a person who helps companies build this infrastructure. In this video, we are going to go over a practical road map

[00:16] that you can use to build this data engineering skills. This road map can be used by both freshers as well as uh experienced professionals. Now when you go to YouTube, you will find tons of road map videos. Many of these videos

[00:30] are created by people who themselves don't have any industry experience. When I started creating this road map, I wanted it to be based on absolute reality. So what we did is we scrapped thousands of latest data engineering

[00:44] jobs from various job portals. We extracted uh some of the hot skills which are mentioned in all these job posting. Then we combined that analysis

[00:56] with my own experience working at Bloomberg as a data engineer. We also consulted many senior data engineers who are working in the industry right now. And then we added some findings from our own experience working on data

[01:09] engineering projects in my AI and data consultancy company Atlake Technologies. The result is this road map which will have weekbye study plan with free learning resources, checklist etc. We have covered both technical skills and

[01:25] core skills. This road map will require anywhere between 6 to 8 months if you anywhere between 6 to 8 months if you study 4 hours every day, 6 days a week. So you can see that there is lot of hard work that you have to put in. So if

[01:38] you're looking for a shortcut, please leave this video right now. This road map requires a lot of patience and hard work. Before we start, we should find out if there are enough jobs in the market or not. You can go to any job

[01:51] portal. I went to no.com and searched for data engineer. I found thousands of jobs. They post jobs using different titles as well such as big data engineer, data architect etc. And those titles are also showing many jobs.

[02:05] Talking about salaries, we took salary data from these job portals and built this histogram. In India, you can see that many of these jobs are in range 6 to 15 lakh peranom. Some of the jobs also offer higher salaries such as 1 K.

[02:21] also offer higher salaries such as 1 K. In US the median salary is $150,000 a year. While data engineering career looks lucrative, you should figure out if it fits your natural abilities and skills. For this, we have designed a

[02:35] free test where you can answer some of the questions and it will tell you if this is a right career for you or not. So, I suggest you take this test before you start your study. Let's look at our job analysis report. As you can see,

[02:50] these are the top five skills in all those data engineering jobs which are posted on various job portals, Python, SQL, data pipelines, ETL and so on. We are going to provide the code of this analysis tool to you. It's a free

[03:05] repository on GitHub. You can run it at any time. Now, here are tech skills and core skills. First thought you will have in your mind is it is too much to learn.

[03:17] But the fact is that data engineering is very tool heavy. So you will be learning many different tools in this upskilling journey. And these are some of the good skills to have. Before we jump into the actual road map, we need to clearly

[03:33] understand the category of jobs. Some people make this mistake which they start learning data engineering without actually understanding. It's like you sure if you want to become heart doctor or a lung doctor. You need to understand

[03:46] the categories and what is involved in each of them. There are three main categories. The first one is platform data engineer. Then analytics data data engineer. Then analytics data engineer and allrounder. For platform

[03:59] data engineer, there are two further categories batch and streaming. So these are the engineers who have expertise in the infrastructure. They are not associated with domains such as retail, healthcare and so on. They have specific

[04:14] skills in a platform. For batch platform uh data engineer category, you will find uh data engineer category, you will find jobs under different titles such as data engineer 1 2 AWS as your GCP data engineer. Okay. So if you have expertise

[04:29] in let's say AWS data engineering then you will see a job post called AWS data engineer which means you are an expert in AWS ecosystem. You are an expert in

[04:41] Azure ecosystem. Okay. So these are the platform data engineers and this is one of the job post on no.com. This is data engineer 2 at Amazon. And when you read

[04:53] the description uh you will see the description talks about being an expert on a platform and the data infrastructure. See here you can see clearly develop and maintain end to end

[05:08] scalable data infrastructure and data pipeline. You will also see this kind of job Azure data engineer which means you know Azure data lake, Azure data factory, Azure data bricks you know you have expertise in Azure ecosystem then

[05:24] there this is another job for a data engineer role. Talking about streaming you will require skills such as Kafka, Flink and so on. The job post are posted

[05:36] under different titles such as realtime data engineer, streaming platform data engineer, Kafka developer and so on. See here it says data streaming uh experience. And here you will see all the technologies related to uh

[05:53] streaming. The second one is analytics data engineer. Analytics data engineer sits closer to analytics and business teams rather than the core infrastructure. So these are little bit on business side. They are closer to

[06:08] analytics and business team. Whereas the platform data engineer are hardcore techy people. They don't care that much about business. They care more about the platform. In analytics data engineer you will see further categories such as BI

[06:22] engineer, data modeling engineer and so on. Here is one job post which is posted under senior analytics engineer title and if you read through the description you will find what are the skills that companies are looking for. So folks, I

[06:37] highly recommend that you go to any job portal and search for all these jobs and try to read the job description. That will give you a lot of valuable insights

[06:49] on what kind of skills you should be preparing for, what kind of category you should be aiming based on your personal inclination. Let's say I am more inclination. Let's say I am more interested in analytics in business etc.

[07:02] then I should go for analytics data engineer. If you are a very techy guy who is interested in the platform, you you know Azure very well, GCP very well, infrastructure, in that case you should go for this particular role. And the

[07:18] go for this particular role. And the third role is allrounder, you will see job post under titles such as staff data engineer, fullstack data engineer. See here this one says fullstack data engineer. Similar to fullstack software

[07:31] engineer. Similar to fullstack software engineer here a person knows everything end to end. Before I open the week one plan let's hear from experts. Let's plan let's hear from experts. Let's check the ground reality.

[07:47] can they treat data like a code. So we just don't want someone you know who can write a very complex SQL query. We would probably want somebody who writes modular testable version in control pipeline and they understand CI/CD. Do

[08:01] their transformations what they're writing? So they can treat data like a code. That's one of the code skills which we look for. I think the second thing will be can they structure chaos right because there are so many tools

[08:15] available and so many platforms available when it comes to the you know data engineering. What we look for that a person actually understand how data should be stored for the retrieval right think you know think from the business

[08:29] perspective and then you know it doesn't really matter whether it's a dimensional modeling it's a data vault or it's an you know one big giant table but if you know they can explain the trade-offs and they can think in the direction how they

[08:43] would be structuring the code for the retrieval purpose that's something you know which is very important skill to have when when it comes to data engineering, right? I think the the third point would be do they understand

[08:56] the cost of the decision, right? And we do ask this question to see, you know, if they understand the ecosystem, do they know when to use stream, when to use badge, do they understand the item potency, can they explain what happens

[09:09] to a downstream dependency if the schema changes. So these are the three points I would say we look for. I would say 80% of the time any data engineer would kind of know spend in terms of debugging their pipelines why exactly it is

[09:22] their pipelines why exactly it is failing. So the ability to break down your thinking identify the root cause and isolate it and kind of know provide the solution that hey these are the reasons or these are the series of

[09:35] reasons because of which the things are failing. with really great engineers and uh one of the things which I you know observed of the things which I you know observed about them is like the top 1% focuses on

[09:50] you know about construction but it's more about simplification so for example right if you think about an average engineer average data engineer whenever you know they face a complex data

[10:03] problem their instinct is to add more right uh they spin up a new spark Python script can solve the problem. They introduce new trendy tools. they build a fragile pipeline that might be working perfectly today but it will be

[10:17] nightmare to you know uh debug it for the next 6 months from now on what's right and uh a lot of time uh data engineers or I would say engineers they measure success by the complexity of the code what they have written in my

[10:31] opinion that's not the uh right metrics to see that the top 1% they do have that you know quality that they look at the problem and ask you know what is the simplest and boring ing solution for this particular problem, right? They

[10:45] understand that every new tool or line of code will add the technical debt and maintenance overhead. So instead of writing, you know, 500 line complex transformation, they find a way to do it with a very clean SQL, you know, window

[10:58] maintainability, they'll prioritize simplification. So the 1% engineer software, it's it's not about writing it, but it's keeping it running for the it, but it's keeping it running for the next 5 years. As you saw from all the

[11:13] experts who are also the interviewers, okay, all these experts uh whom you just okay, all these experts uh whom you just listened to, they take interviews and the kind of person that they look for is like this skilled handyman. Whenever you

[11:25] have a skilled handyman, they come with different tools, hammer, wrench, uh a screwdriver and so on. They are jack of all trades, master of none. So they have

[11:37] a judgment to pick right tool at a right time. Okay. So they know Kafka, airflow, data bricks, snowflake but they know if they want to put a nail in the wall they will use hammer. If they want to put a screw in a hard surface they will use

[11:51] screwdriver and so on. So similarly you have knowledge of all these tools but you also have a judgment. See this judgment is very crucial. you have a judgment to pick right tool at the right time. Now based on the category that you

[12:08] time. Now based on the category that you pick, you can focus on uh some skills. Okay. So for example, if you want to become platform data engineer in streaming, you need to know Kafka really well. But if you're doing batch

[12:22] processing, Kafka is not required. That's why there is a red arrow. You can take a screenshot of this and you can focus on the areas which are green for the role that you select. Okay. So here are some of the cloud and manage

[12:37] are some of the cloud and manage platform skills for all these category platform skills for all these category of roles. Okay. And this is uh another uh page for the core skills. Core skills are very important. Your communication,

[12:49] stakeholder management, system design, all of that. Here is the road map PDF. Let's go to weekby- week study plan and it all starts with week zero where you will do proper research and prepare your mind and body because it's going to be a

[13:05] long journey. So make sure you have checked all these uh items and then we have total four phases. Phase one is foundation, phase two is cloud and scale, phase three is advance and specialization and phase four is

[13:22] portfolio and job hunt. Now let's say if you're already working as an experienced software developer then phase one is going to be very very fast for you. I'll tell you why. Uh here we are learning programming basics. Okay. So week one is

[13:38] Python basics plus essential data structures. And in Python you need to learn just the basic concepts which are listed here. And in data structures you don't need to worry too much about trees and graphs and all that arrays hashmaps

[13:53] stack Q hashmap is very very important by the way bigo notation so just a basic understanding is good enough and we have mentioned some free learning resources here there are some YouTube playlist I personally like this playlist by Corey

[14:08] he explains things really well on my channel also you will find a playlist also for data structure ES I have this playlist in Python basically we have

[14:20] implemented different data structures in Python and in the road map file I have mentioned okay how many videos you need to watch you don't need to watch the mentioned here just use that you can also use chat GPT and claude as your

[14:34] tutor for example you learned dictionaries and list and now you want to take a quiz and you want to test your knowledge you can go to chat GPT and say okay chd take my test and you can just give the answer like this and it will

[14:50] evaluate it and it will say see incorrect. See, so this way it can ask you question uh and it can also evaluate your answers. During week one, you should also create your professional LinkedIn

[15:05] profile. Many people start creating LinkedIn once they are done with all the 6 to 8 month study. But you should do this thing in parallel because building online credibility through LinkedIn is going to take time. So it's not like I'm

[15:19] ready to prepare my job and then I create a LinkedIn profile. Okay? It's going to take time. So you should create your LinkedIn start becoming active on it from day one. We have provided a LinkedIn checklist. When you click on

[15:33] that link, you will find a PDF. You just go through the PDF and make sure you have checked all the items and at the end you will have a good-looking LinkedIn profile and these are the assignments in that YouTube playlist

[15:45] especially the one on code basics channel has the exercises. So we have exercises and solutions. So make sure you finish all of that. Now why LinkedIn is important? Just imagine you are having a samosa stall in a village. You

[15:59] make very good samosa but you don't make too much income. Now you move the same samosa store to a busy market in Delhi in a crowded street and you will make

[16:11] huge business. Your quality of samosa your professionalism is same but you change the location and your business is now growing. LinkedIn is like that busy street where you are showcasing your skills and the recruiters will notice

[16:26] you. In week two and three you will learn Python specifically for data engineering. In data engineering, we use for example request package to make API calls. We deal with dates and times and time zones. We write unit test using

[16:42] Pyest. We use pandas for data transformation. So all these Python topics are very very crucial for data engineering job. And you can use the same uh YouTube tutorial playlist and as a mini project you can work on this

[16:58] project called ops pulse where you will pick a data set let's say NYC taxi or any public API weather finance etc. You will fetch data from public API using this request module which you learned in

[17:13] this particular week and then you will save that data to either JSON or CSV file and transform it using pandas. Okay, remember you learned pandas in this week. So this assignments are prepared based on whatever you have

[17:27] prepared based on whatever you have learned so far. Then week four is all about data engineering foundations and git. As a data engineer, you need to know what is OLTP versus OLAP, ELT versus ETL, batch streaming, etc. And

[17:43] you also need to know the fundamentals of Git version control system. As a data engineer, you will be using Git almost daily to manage your code. Okay. So, you need to do at least basics Git clone, add, commit, push, branch, merge,

[17:57] conflicts and so on. And we have provided some of the useful resources. You just click on this link and these are all free resources folks in order to learn these skills. All you need is a laptop, internet and a strong willpower.

[18:11] Now when you appear for the interview, your communication is going to matter. you are interacting with your colleagues, clients, once again communication is going to matter. So there is this club called Toastmaster.

[18:25] So if you go to their website uh you can just say find and visit a club and you just say find and visit a club and you can find a club near you. It is a it is a free community. They conduct sessions uh so that your communication can be

[18:39] improved in a practical way. Folks whoever have attended Toastmaster sessions and prepared with discipline have received amazing results. So I highly highly recommend attending Toastmasters. And in terms of offline

[18:55] communities, there's this thing called Indian data club. So if you go to their discord server which is this, you will find all the city chapters. Okay. So they conduct meetups, they conduct learning sessions, challenges. For

[19:07] example, they have 21day SQL challenge. So when you are learning with community, you will be inspired, you will be motivated, you will learn from each other. So this idea of learning in group learning by community is very effective.

[19:23] As an assignment, whatever code you have prepared in week one and two, move that into a clean repo structure. Create a repository on GitHub. Add a readme file explaining all these things. And you definitely want to show your work on

[19:40] LinkedIn. So share your GitHub repo on LinkedIn. Mention what problem does it solve? mention one thing you learn about de fundamentals. When you start posting on LinkedIn, it will not immediately work. It will take time. But once you

[19:54] start gaining uh traction, once you improve yourself, it will work wonders. improve yourself, it will work wonders. You will learn SQL in week five and six. SQL is heavily used by data engineers and you need to know even the advanced

[20:09] concepts. So you'll start with basics. What is relational database? basic queries, select, group by, order by and so on. And you will then move into advanced stuff such as window functions,

[20:22] CTE and so on. And there are tons of free uh resources that you can find on YouTube. So YouTube we have this complete free course with uh practical

[20:34] explanation, exercises and so on. You also have this Khan Academyy's free course uh that you can take and then there is a this interactive platform to there is a this interactive platform to practice SQL. So here let's say it is

[20:48] showing all the movies by year and you want to let's say practice where clause. want to let's say practice where clause. So you will say where year equal to 2003 and see it is dynamically updating the results. When it comes to LinkedIn, you

[21:02] results. When it comes to LinkedIn, you will start following prominent data engineering influencers. You might have heard about Zach Wilson who was a data heard about Zach Wilson who was a data engineer at Airbnb. He he writes a lot

[21:15] of good content on data engineering. Then there is Bar, there is Gotham. So you can follow all these people and learn from them. You will also increase engagement. So you are not just reading post by these influencers. You can ask

[21:31] meaningful question in their post. You can engage, you can share your thoughts. This way you are spreading your online presence. Other people will start noticing you. You will build relationships with other folks and these

[21:45] relations will be very helpful in your career growth. You will also utilize Discord servers to ask questions in a right way. We have code basics discord

[21:57] server with more than 50,000 members. You will find channels such as data engineering. It's a thriving community folks. People discuss uh data engineering. They uh ask questions. They share jobs and so on. We have uh

[22:12] programming language or tool specific channels as well. For example, this one is SQL. This one is Python. Folks, go ahead. There are there are tons of resources available for free. You need to make good use of it. You will also

[22:27] build business understanding. Now why is it needed? When I was talking with a it needed? When I was talking with a data engineering head working for a pharma company here in New York, uh he's a friend of mine. He told me that when

[22:41] they hire data engineers, they want to specifically hire data engineers who have domain expertise in pharma domain. Okay. So if you have expertise in Okay. So if you have expertise in certain domain during hiring that domain

[22:55] understanding or domain skills will be valued. The way you can build this is by gaining basic understanding on metrics KPIs functions. Okay. So let's say I want to know what are different domains and

[23:10] functions. Then you can click on this link. There is a LinkedIn post that will teach you that. Okay. domains are these many and these are the functions within

[23:22] the domain and so on. You can learn from uh channels such as think school and you can look at the case study videos and you can take note and pen. You can

[23:34] understand digest the information from that case study. Maybe create a LinkedIn post with your analysis with your thoughts. This way you can build a good domain understanding. We have linked one good video on how hotar application

[23:49] good video on how hotar application scaled to 25 million concurrent users. See all this information is available for free on YouTube. In terms of assignment, remember we started this ops pulse project in very first week. Now

[24:05] you will continue this project and you will load process data into a relational DB MySQL or Postgress. then you will write 15 to 20 SQL queries covering joins, aggregations, filters and so on. Now if you don't get good quality data

[24:20] from public APIs etc. However, it's going to be almost impossible because there are so many good data sets which are available on internet. Uh but just in case if you want to customize your data set then you can use your friend.

[24:35] Who is your friend? Chat GPT Claude Gemini. You can say okay generate 10,000 records for this this domain this this use case and it will generate a nice CSV file for you. You can dump that data into your MySQL database and run all

[24:49] these queries. So that way you are practicing your SQL skills. Then you will solve some medium level SQL problems on platforms like data leour.

[25:01] So data leour is a platform that allows you to practice SQL questions especially the interview questions. For example, these are the interview questions from LinkedIn. Okay. So here the problem statement is given. You write a query,

[25:17] you run code and it will show you the uh output and there is a solution discussion forum and so on. This is created by my friend Nick Singh. Amazing created by my friend Nick Singh. Amazing platform. You can also show your work on

[25:31] LinkedIn once again by writing meaningful comments on at least 10 data engineering uh related post. Okay folks, so for this two weeks let's have the aim so for this two weeks let's have the aim that you will comment on 10 post. Okay,

[25:48] you'll also create your own post where you will talk about one SQL problem you struggle with and why your final query is safer than your first attempt. If you want to gain some visibility, feel free to tag codebasics taval or hamman with a

[26:05] hashtag dromap206. Using the hashtag uh we can locate you and we can engage. My team can engage of course based on our availability. But when we engage the people in our network can also see your post. So that way the

[26:20] visibility of your post increases. Week seven and eight folks it's all about seven and eight folks it's all about data modeling, warehousing and Linux fundamentals. As a data engineer you need to know data modeling and

[26:33] need to know data modeling and warehousing. For data ops you will be using Linux. Okay. So for Linux you should know the basics like basic should know the basics like basic command CD move graph and so on process

[26:45] management cell scripting basics you know bash scripting basics and so on. And when it comes to the data modeling and warehousing concepts, you need to learn all these concepts. Now from where do you learn all these concepts? Well,

[26:58] once again, we have a section called learning resources. So here there are all free learning resources. Uh if you want to pay a little fee and pursue a

[27:10] structured course with practical and industry relevant content, then we have industry relevant content, then we have provided links of some courses as well. In terms of assignment, you will uh design a star schema. Star schema is

[27:24] design a star schema. Star schema is both fact and dimension table for your ops pulse project. Remember we started ops pulse project in week one for NYXC weather etc. data you can select whatever data set you want to select and

[27:37] and every week as an assignment you are enhancing that project. You are making that project bigger and bigger. Okay. So these are the exercises and once again you will do some LinkedIn activity. So that will conclude your phase one. So if

[27:54] I have to summarize phase one, you started with Python basics. Then you further advanced in Python by learning topics which are specific to data engineering. Then you touched based on data engineering foundation. You learned

[28:09] data engineering foundation. You learned Git. you gain SQL mastery and then data modeling and warehousing basics plus Linux fundamentals. Now someone may Linux fundamentals. Now someone may argue that okay this is too less time or

[28:22] maybe this is too much of time. Well it depends on your current skill level. If you're a fresher starting out this might take you maybe 10 weeks. If you already know SQL if you already know coding then you can fast track this into four weeks

[28:39] instead of 8 weeks. So it all depends on your skill, your learning speed, how much time you're spending every day. This weekly structure is just a rough general approximation. Here is a rough statistics. Around 60 out of 100

[28:54] aspirants who started will continue further into phase 2. You want to be one among those 60 aspirants who will continue further because people don't

[29:06] have patience. They leave in between. You know people have this give up attitude. There is lot to learn. It's sometime boring. It is requiring lot of hard work and that is the reason people will drop out. Make sure that you

[29:20] continue with persistence into phase two. Phase two is all about cloud and two. Phase two is all about cloud and scale. In week 9 and 10 you will learn scale. In week 9 and 10 you will learn ETL on cloud. Okay. So you will pick one

[29:34] ETL on cloud. Okay. So you will pick one of the clouds either AWS, Azure, GCP. AWS is very popular. Azure is also very popular. We saw in the hot skills that Azure was at the top. GCP is also used by some of the companies. But if you

[29:48] look at the market share, it's mostly AWS and Azure. So you pick one cloud. You don't need to learn all the cloud tools. Okay? If you learn one cloud tool, then you have transferable skills. You can learn other cloud tools easily.

[30:04] So here in terms of topics you will learn about park a JSON. These are popular file formats which are used in data engineering. You will learn data injection into data lake. Okay. What is data lake? What is median architecture?

[30:21] Uh incremental versus full load. Add important batch processing. Very important concept. data partitioning and so on. Okay. So you'll choose one cloud

[30:34] uh provider and you will uh continue your learning journey and we have your learning journey and we have provided some of the YouTube video links etc. Now at this stage you need to start thinking about ownership and

[30:47] reliability. As a data engineer it's not like you're writing a data pipeline it like you're writing a data pipeline it worked okay once or twice and that's it. The pipelines must run consistently without manual interventions. It should

[31:03] include handling failures, ensuring data correctness, managing reruns, and being accountable for cost and reliability. It shouldn't be like your AWS bill goes

[31:15] through the roof after you have run your pipeline for a month. You should be very much aware and vigilant about cost and reliability as if you are paying that AWS bill out of your own pocket. These are some of the good traits of a quality

[31:31] data engineer. You are not just worried about okay I ran it once and it is working. You are worried about age cases. You cover all the age cases failure scenarios which

[31:44] happens once in a while but it can cause a lot of damage. Okay. And most of the engineering career, they will not think about all this. And if you want to stand about all this. And if you want to stand out in competition, you need to learn

[31:59] strong ownership and reliability skills. Okay. So these are the topics that you need to master and we have provided some of the great uh YouTube video links. For

[32:11] example, for pipeline failures and recovery, there is this video from Pyon USA. Okay, so this is Python conference which happened in US and this is a talk

[32:23] by Mihir Kawatkar where he talked about building resilient data pipelines. Okay, you should listen to all these Pyon videos etc. the

[32:35] engineers who are working for big tech companies who are doing amazing work will be representing uh these stocks. So you can learn a lot of good skills which once again can stand you apart uh from the competition. In terms of assignment,

[32:52] we'll take our ops pulse project. We will move raw and process data into cloud storage. Let's say you move it to AWS. You will then build a Python ETL job to ingest, transform and load data. So let's say I have my raw data in S3.

[33:08] So let's say I have my raw data in S3. Okay, Amazon S3 uh object store. Then I Okay, Amazon S3 uh object store. Then I have an ETL job which can take that data and by the way S3 is often used as a data lake. So you're taking data from

[33:21] data lake transforming it and maybe you load that into red shift data warehouse and you will summarize your learning via LinkedIn you will also upload updated

[33:33] code and readme to your GitHub repository. You are probably noticing this theme that as we study across weeks one thing is constant which is posting on LinkedIn updating GitHub with the latest code updates and so on. Week 11

[33:49] latest code updates and so on. Week 11 is about Docker and ATS rum. Docker is a you get a package at your home, right? There are three or four items and they are packed in a nice box or container. Similarly, when you ship your code to

[34:04] production, you have code reference files, some data, etc. You put everything in a nice box or a container which is your docker container along with all the dependencies and [snorts] then you ship it to production. That way

[34:19] you can get predictable reproducible behavior. You don't want to do a PhD in Docker. Okay, just learn the basics. Skip the Docker internals. [snorts] Kubernetes is is good if you learn it but at this stage it's not required.

[34:34] Okay, so skip it. And here is a YouTube video. Just single video is good enough. And then you will do some research on what are the projects your professional connections are working on. So now you have become active on LinkedIn. You have

[34:50] built some connections. Maybe you have built some connections with data engineers. So once in a while send them a message. Okay. Can we have a call? I learning data engineering. You are doing

[35:02] nuances of communication. You need to respect their time. Okay? If you push them then nobody will respond. So there is an art on how to communicate on LinkedIn. And for all these also you can use chat GPT and AI tools. You can talk

[35:17] to chat GPT and say that I want to initiate conversation with this particular data engineer. How do I write the message? And it will guide you on writing message uh with good manners. Then what are the strategies people use

[35:31] to land jobs? Okay, you need to do all this research and maybe write down all this research on notion or in your notebook for presentation. This video if you have watched my

[35:45] channel I have mentioned this video multiple times. It doesn't get old. The video title is how to avoid death by PowerPoint. It is a classic talk folks. If you follow the principles outlined in

[35:58] this video, you can become the master of presentation. Okay, very simple, very powerful tips. Now, why do you need presentation? As a data engineer, you will be presenting to the stakeholders. So, you need to master presentation

[36:12] skills. Okay, data engineers unfortunately focus on learning Python unfortunately focus on learning Python and uh Snowflake and uh data bricks but they don't focus on improving their presentation skills. It doesn't take

[36:24] much time but the value you get out of it is going to be tremendous. it is going to be tremendous. Now ATS ré means application tracking system compliant ré. When there is a job posting usually a recruiter will get

[36:39] multiple rums. They don't have time to scan each and every résumé. So they let it go through the ATS system. And if you want your résumé to pass through the principles. So we have created this réumé template which you can go to our

[36:57] website. This is all free by the way. So let's check the ré of this person Peter Pande. So maybe you can take download this ré and you can make changes uh so that it fits your profile and you also have this checklist. So

[37:13] this particular checklist let me click on this. Once you go through all these on this. Once you go through all these points your resume will look much better and professional and of course you can take help of chat GPT. Now in terms of

[37:28] assignment you will containerize whatever Python code you have written. so far. So we have been building ops pulse project across many weeks. We have been checking in that code on GitHub. Now take that code and build a container

[37:43] docker container with all the dependencies and everything and deploy it and then update your resume with this particular project. Week 12 and 13 is

[37:55] about spark and datab bricks. Datab bricks is gaining lot of popularity. It is a unified analytics platform which means you can do data engineering, data analytics, AI all in one place. They are investing billions of dollars to make

[38:12] this platform perfect for variety of needs organizations are facing. So it is important that we learn data bricks. Datab bricks is built on top of spark. It's a company that is founded by the inventor of spark distributed framework.

[38:27] Spark distributed framework is used by more than 80% companies. Wherever you go and if you are doing distributed compute, you will definitely see Spark. It is that popular and powerful. Okay, these are the list of topics that you

[38:42] need to cover in this week 12 and 13. And we have an excellent crash course. We published a video on our channel. You can see in one month it got 100 uh K views and you can read all the comments. By the way, this is a quality course

[38:57] which covers theory uh practical fundamentals and then at the end we have built one end to end project. So this is sort of like a mini course. Okay, you will pay a price to

[39:10] buy this kind of course but here it is available on YouTube for free. Now during this same time period you will also work on system design thinking. See

[39:22] normal data engineers, average data engineers will build pipelines in a casual way. But great data engineers will always think about system design. They will think about scale. They will think about horizontal scaling, single

[39:36] node versus distributed system, data movements, bottleneck, choosing spark versus simpler solutions. When we were consulting with some of the senior data engineers, you have seen their video clips at the beginning. uh they

[39:49] Now these are the engineers who have worked for big tech companies like Eston Young uh some of these engineers have their own consultancy firms. Okay. So these are seasoned professionals with more than 10 years of experience and

[40:04] these topics are added based on their inputs based on what they use in their day-to-day life. Okay. So folks we have not created this by okay going to chat GPT and doing it. This road map is created based on real experience by

[40:20] senior data engineers or data engineering managers. Okay. And for them these things are super important because when they get into the interview the data bricks and snowflake question but

[40:34] when you touch base on these kind of advanced topics they don't answer. And if you are able to answer those questions then you will definitely get a job. Here we have mentioned some uh playlist. Okay. So you can go through it

[40:48] and these are the assignments. So folks you can download this PDF from video assignments and see working on these assignments is very important. See it is one thing to watch a video and learn the concept. But when you actually work on

[41:02] the assignment that is when you are doing the real learning. So make sure you are clicking all these check boxes. Okay take a print out. So make sure you are checking these boxes when you are studying. Okay? And in terms of showing

[41:16] your work, see there won't be a week where you're not posting on LinkedIn. So make sure you continue your engagement. Week 14 and 15 is to build two end to-end data engineering projects. See, so far we have learned all the skills.

[41:31] Okay. Now it is the time to build some serious projects. In the previous week, by the way, while going through this tutorial, you have already built one project. Now, in this two weeks, if you build two more projects, you will have

[41:46] three solid end toend projects in your portfolio. And that's going to make a huge difference when you start applying for the jobs. Now, these projects should be such that they cover median processing through bronze, silver, and

[42:00] gold layers. They cover spark declarative pipelines. they cover full and incremental load. Okay. So when you look at any industry grade big project they will have all these concepts covered. Okay. And fortunately on

[42:16] YouTube you will find many free videos. So on our channel we recently published these two projects. And if you look at these projects uh they are truly end to

[42:28] end. And what I mean by that is we start with the problem statement in a comic style. Okay. So you will uh feel like you're watching a movie. Then there is a stakeholder discussion. So stakeholder discussion is basically this person is a

[42:44] data engineer. This person is his manager and they will talk about the will get a feel as if you're working in a company and you are sitting in a meeting room. You'll get that kind of simulated environment and then we go

[42:59] over technical architecture. Okay. So look at this technical architecture. This is the kind of project you build in the industry and then create catalogs. You go through all the coding and everything. Okay. Access management

[43:13] using Unity catalog. Then the stakeholder review of the project and so stakeholder review of the project and so on. Okay. So these projects are created through lot of hard work through lot of experiences. By the way, Hamanan whom

[43:26] you see here, he worked as a data analytics manager in Europe for more than 10 years. I worked for Bloomberg for like 12 years and I have my own company. We both have serious industry experience and we are trying to share

[43:41] this experience via these YouTube videos. So folks, I hope you can make videos. So folks, I hope you can make best use out of this knowledge that is available to you for free. Week 16 and 17 is about snowflake and airflow.

[43:56] Snowflake is a distributed warehouse basically. Okay, cloud data warehouse. It's a cloud data warehouse and these are the topics. Okay, zero copy cloning, copy into is a popular command that you will use in snowflake and for airflow

[44:12] you will cover all these topics. Okay, so we have provided some resources as well and in this same week you will learn project management. See when you will get job as a data engineer uh you will be working with a team and there

[44:27] will be task breakdown there will be project management there will be scrum scrum and kanban are the project management methodologies okay when I was at Bloomberg uh I remember there was a coach who came to give us a training on

[44:42] coach who came to give us a training on scrum and he pointed me to this amazing website this is a free website these are 5 10 minutes short videos. So if you 5 10 minutes short videos. So if you click on it, you will see that see short

[44:56] click on it, you will see that see short video and these videos folks are more video and these videos folks are more than amazing. Okay, these videos are so good. You will get a very comprehensive understanding of this uh frameworks uh

[45:10] frameworks such as scrum, kban and so on. Okay. And for motivation I have included this video where this girl got job as a data engineer at Grubhub after job as a data engineer at Grubhub after 6 years of career break. So here is the

[45:24] video you can watch it. Especially people who are having career break they will find this very motivating uh working moms and so on. In terms of assignment you will pick one manage platform. Okay. I I will say choose data

[45:39] bricks. It's very popular. It's good even if you learn snowflake or fabric and then work on all these assignments. Okay. In terms of showing your work, you will push your code and config to GitHub, update your readme, post on

[45:53] LinkedIn, all that usual stuff. Now, at this stage, only 50 people will survive. There will be remaining 50 who will not have enough patience, who will be distracted by various, you know, shiny objects that exist in this world. I hope

[46:10] that you the person who are watching this video be among those 50 who continue into phase three. Please take a sip of water, sip of coffee. We are getting into advanced stuff now. In phase three, we will cover advanced and

[46:25] phase three, we will cover advanced and spatial topics. So week 18 and 19, you will learn about streaming data system. These are realtime systems where you use streaming platforms such as Kafka and Flink for doing a realtime processing.

[46:41] fundamentals. For Kafka, you need to know Kafka architecture, what is producer, consumer, message key, partitions and so on. And then for Fling there are essential concepts such as watermarks, checkpoints, Flink

[46:55] architecture and so on. And fling SQL is something that you use to run the query on the streaming data. So when you run a fling query, the query results will be updating real time as if you are in a stock market monitor, you know, where

[47:10] stock prices update. So fling SQL is very different than your traditional very different than your traditional SQL. Fling SQL is used for querying the realtime data. Okay. And we have provided some free learning resources

[47:24] for Flink. And in terms of system design thinking, you need to think about latency versus complexity. Exactly once versus at least once. Okay. So these are versus at least once. Okay. So these are the core concepts, fundamental concepts.

[47:38] Designing for failure in real time system. Choosing streaming only when necessary. Now these are the learning resources. So these are blogs mostly but resources. So these are blogs mostly but these are very very uh insightful blogs

[47:53] that you can go through. This one is on latency and complexity and so on. Okay. latency and complexity and so on. Okay. Now for assignment you will set up Kafka locally. By the way you can use Kafka in Confluent also. So Confluent is a

[48:07] managed service for Kafka. It depends on your preference. If you do it through Confluent it will be easy. Setting up Kafka locally might take you some time. Okay. You will create some producer consumer um and process streaming data.

[48:22] Now folks, when you're writing code, make sure you are using AI to boost your productivity. When you are learning concept uh for the first time, I suggest that you write code with hand. That way you are gaining understanding on the

[48:37] fundamentals. And once you are a little bit familiar with that code, try to boost your productivity with AI tools. You can use tools such as cursor or You can use tools such as cursor or anti-gravity to write code faster. Then

[48:51] for showing your work uh you can work on all these three assignments. Week 20 and all these three assignments. Week 20 and 21 is going to be spent in learning 21 is going to be spent in learning analytics engineering with DBT. So DBT

[49:06] is this amazing tool. Uh and what exactly is DBT by the way? So DBT is used to run a transformation. So DBT is basically T and E. Okay. So extract load

[49:20] transform and dbt is used to transform the data in data warehouse. So let's say your data is coming from OLTP to your data warehouse. So let's say you have raw data. When you want to transform data, let's say you want to clean data,

[49:35] enrich it for your BI or AI needs, you can use DBT. It will run your SQL queries fundamentally. But it offers certain benefits. And to learn what benefits it offers,

[49:50] learn.getdbt.com is a website. They have many free courses like for example dbt fundamentals. So you can just learn by enrolling into this. I think these are all free learning resources. Okay. And I

[50:04] have mentioned the topics for analytics engineering. Once again, analytics engineering. Once again, analytics engineer is a category of data engineer who works closely with business, who works closely with data analyst. So they

[50:17] come into play at a later part of your pipeline and they're not very much focused on the infrastructure. They are more business and outcome oriented. Okay. And these are the topics you need to learn for cloud scaling. I have

[50:31] provided the links of all the learning resources. Now at this stage you should create your project portfolio website because we created few projects already. We created three datab bricks project one that ops pulse project and couple of

[50:46] more. So all these projects you can uh add to your project portfolio website. Now you can use GitHub to create project portfolio website or this platform data science portfolio or if you're part of code basics data engineering boot camp

[51:02] then we provide that feature builtin. Okay. So these are all the portfolio websites of all these projects. Okay. So if you want to look at let's say a sample portfolio here let me just open someone's portfolio. Rajna's portfolio.

[51:17] Okay. So project portfolio website is nothing but your online resume. So here if I play this video uh Rajna will be talking okay and as a recruiter I can get hints on her verbal communication her personality then she has mentioned

[51:33] her skills okay Excel PowerBI and so on and she has mentioned uh her projects so let's say I want to know about her uh sales analytics projects so I can click sometimes you can embed your live project also I can go to GitHub and look

[51:51] at the code you see it's very live and dynamic as opposed to a dead resume. Okay, you're mentioning your email GitHub folks trust me if you have project portfolio website your chances of getting interview call will increase

[52:06] because you are giving the potential recruiter a lot of useful data points. Okay, and you're showing your sincerity as well. Then comes data governance. Data governance is a very crucial topic in data engineering. Here you talk about

[52:23] your data access. Okay. Your PII, personally identifiable data. You want personally identifiable data. You want to mask that data. There are various regulations. For example, in Europe, you have GDPR. If you're working on a

[52:38] project for European client, you want to be GDPR compliant. Or even if you have consumers, right? If you have any presence in Europe, you need to be GDPR compliant. Similar to GDPR, every country have their own data protection,

[52:51] data access policies. You need to adhere to that. And data governance is not about adhering to some government policy. It is more than that. It is about ownership and responsibility, schema consistency and evolution. Okay.

[53:06] schema consistency and evolution. Okay. Basic governance practices and so on. We Basic governance practices and so on. We have some good resources. So this one is have some good resources. So this one is from IBM. Okay. So there is a video on

[53:18] data governance that you can watch and you have couple of blogs as well on data governance. Okay. And then in the assignment you will set up DBT. You will

[53:30] work on all these assignment items. Folks make sure I know you are sincere student. So you will not only learn you need to promise that you will finish all need to promise that you will finish all these assignments. Then week 22 and 23

[53:43] is about data quality and reliability. See now you'll be like okay we are not learning tools but don't focus on collecting thousand tools in your resume folks. Build real skills. Real skills are do you care about data quality and

[53:58] reliability when you are working on a data engineering project. Do you uh care about schema checks, null checks etc. Okay, you need to implement data quality checks in analytics pipeline, monitoring data freshness. All these topics once

[54:13] again folks, these are very important topics. We got list of these topics by talking to industry experts multiple times. So we are talking with data engineering managers, people who run data engineering companies. We had

[54:27] conversation with them multiple times and we have created uh this particular list. Okay. Okay. And these are the resources that you can learn. Uh then there is ownership and reliability. Okay. So you can follow all of this. And

[54:43] here we have an interview with uh Zach Wilson. I interviewed him I think some Wilson. I interviewed him I think some time back and we have given lot of tips. So Zach is a popular name in data engineering world and you will find some

[54:58] useful tips here. Okay. And you can also see some engineering blogs on data incident and reliability. Okay, these are important ones. All right. And here are the assignments. At this stage, you will finish phase three and only 30

[55:13] among 100 will persist. Remaining people will probably give up. I want you the person who are watching this video. I have trust in you. You need to be one among these 30 people who will continue into phase four. So far in first three

[55:28] phases we have learned almost all the skills. Now in phase four you will work skills. Now in phase four you will work on building additional business projects you will work on contributing to opensource so that your online

[55:41] credibility increases. Okay. So in terms of topics of course you will uh take a business problem statement and you will design an end toend data engineering project. You will integrate batch streaming analytics pipelines. You will

[55:56] use GitHub for collaboration and version control. You will use CI/CD for data pipelines. And you will also contribute to uh some of the open-source

[56:08] repositories. Okay. And here are the learning resources. If you want to learn Git, here is a playlist. If you want to uh look at some of the sample business projects, then we have an entire playlist. See, right now there are three

[56:22] projects. In this playlist, we are going to add more videos. So, it is likely that if you're watching this video in the future, sometimes you will see more projects here. And these are end to-end projects with stakeholder discussions

[56:35] and simulated environment as if you are working in a company and witnessing that particular project. Contributing to open-source is extremely important. It will help you stand apart from the competition. And this video talks about

[56:50] some of the basics of opensource contribution. And by the way, you can contribution. And by the way, you can use your AI friend Jet GPT, Claude, etc. to take help uh when it comes to open-source contribution. Okay. Uh now

[57:05] where do you find the project ideas? So the first point is your own pain points. So let's say you are working in a company or you're doing uh even uh at home you're doing some repetitive work manual CSV handling delayed dashboards

[57:21] all of that. So you can use data engineering to fix this pain points you can use public APIs and public data sets. So just take any public data set and talk to your AI friend okay GPT and say okay give me a project idea. It will

[57:36] give you an idea which will not be perfect. So you will refine, you will talk to it further. You will say okay give me a project that I can build using snowflake or build using data bricks and that way it will not only give you uh

[57:51] the project problem statement and the architecture etc. it will also generate data set for you. So if you take juice out of all these AI tools, you will be

[58:03] able to build amazing projects. Okay. So these are the points that you need to these are the points that you need to read in order to find and refine project read in order to find and refine project ideas. Now why does this matter? Well,

[58:15] ideas. Now why does this matter? Well, contributing to uh opensource why does these which means opensource contribution matters? It shows that you can build a solution, you can uh communicate with the community, you can

[58:30] build something real that matters to the world. Okay. And many opensource contributions begin as self-defined projects. Let's say you are facing some problem in your work. You create a a repository. You create a Python package

[58:45] and that starts as a open-source project and other people contribute. You can also contribute to some of the existing repositories. So you can go to GitHub and search for good first issue. Okay. So uh GitHub good first issue

[59:03] So uh GitHub good first issue that's a tag and using that tag you can that's a tag and using that tag you can find some first issue. So let's say if I am in pandas so let me just show you GitHub pandas repository and I'm a new

[59:17] person. Okay I am looking uh to contribute in GitHub. So here you can go contribute in GitHub. So here you can go to issues tab and here you can search to issues tab and here you can search for label. Okay. So good first issue

[59:30] which means people who are beginner they can work on these issues. See look at this 55 open issues folks. So you can definitely contribute okay you don't need an excuse. You don't need to be an expert. And then uh after that you have

[59:44] some of the good resources that you can look at it and these are the look at it and these are the assignments. Okay. Now in week 27 to 30 assignments. Okay. Now in week 27 to 30 okay so 27 8 29 34 weeks as if you want

[59:58] to make it 8 month road map 8 month is 32 weeks then you get two additional weeks during this time you will prepare on your personal branding interviews long-term career growth okay so when it

[1:00:11] long-term career growth okay so when it comes to interview preparation for SQL you know you these are the topics you will focus on Python and so on and you can use chatgp PT for mock interview. Let's say I want to practice data

[1:00:25] engineering system design. Then I can type this in. CHP is my interviewer right now. So it is asking you a question. Okay, here's the scenario. You need to build realtime analytics system live ops dashboard. Then I will type in

[1:00:38] sometimes you know I use this audio. I will click on audio button. I will talk instead of typing. So that feels like a real interview that is being conducted. And then you can say hgd can you evaluate my answer out of five? and it

[1:00:50] will say okay you got three out of five four out of five here is the place where you can make improvements folks as I said you need to take juice out of all said you need to take juice out of all these AI tools uh and you can gain

[1:01:04] significant career advantage if you know how to use AI tools in a right way and communication of course is is very very important now for ré you need to use

[1:01:17] star framework during the interview also and the interviewer asks you to explain the project you need to use star framework okay situation task uh and so on so you can just just Google it and we have a couple of videos on our channel

[1:01:31] have a couple of videos on our channel too and then personal branding you have to uh read through all these points okay so star method I think this is probably the video I think >> I'm not getting

[1:01:45] >> okay this is one of the short video we have a long video as well And uh for job applications we have mentioned different techniques like one of the techniques is cold emailing. So I have a video with a person who used cold emailing technique

[1:01:59] to get the job success. I think this video let me see which video is this. I think this video is with Nick Singh. Okay. So Nick Singh talks about Okay. So Nick Singh talks about interview tips and cold emailing is one

[1:02:13] of the tips for interview calls. I I think on my channel I have another video where I interviewed one person and what he used to do is send cold emails to so many people and he got an interview call and he got a success as well. So this

[1:02:26] thing works folks. All right, job refers. So for job referrals uh here is a tip that you need to use. So this is a LinkedIn post Hammond talked about how do you exactly approach people for job referrals. Okay. So there

[1:02:42] is this structure contextual intro clear subject whatever empathetic outro. Okay. So you can use that template and for assignment solve 30 to 40 SQL question. We already saw data leour folks. Don't you think it's amazing? We are living in

[1:02:59] a world where you get all these resources for free. Previously you would have spent thousands of dollars in university to go through this. Now if you have willpower, laptop and good internet you can just crack you know you

[1:03:16] can crack even Google or Amazon interview. All right. So go through that and that's it folks. That's the end of phase four and that's the end of your phase four and that's the end of your road map. Only 10 out of 100 uh

[1:03:30] aspirants will continue further which means they will get a job. Okay. And I want that the person you the person who are watching this video be 10 among this 100. And I have my prayers with you. I have my good wishes with you. If you

[1:03:47] below. Some of you might have this preference that I don't want to go through this free videos with ads and scattered resources. Our instructors are scattered resources. Our instructors are changing and it's just too much. In that

[1:04:00] case we have this data engineering boot camp on codebasics.io website. Okay. So camp then you don't need to follow this road map because the boot camp contains

[1:04:12] everything in sequence and the videos are even a better quality. These are taught by mostly me Hmonan. We also have live session by the way. So it's not just the videos. We have live practical

[1:04:27] sessions where the industry experts come and they conduct a live session. We have live monthly webinars. You can also get this portfolio website, LinkedIn optimizer and so on. If you want to know all the features then just go through

[1:04:43] the first module. See all these lectures are free. So you don't need to pay any money. Uh and you can go through this lectures. Uh and if you feel it's worth that by spending some money, it is going to save lot of my time and it is going

[1:04:59] to give me even higher quality learning and more practical relevant experience. Then explore this boot camp. Uh if not, we have already talked about free resources. There is supplementary learning by the way. So DP 6700

[1:05:15] certifications are useful. These are Microsoft certification. So click on these links to get more information on it. So after finishing the road map, first finish the road map then worry about these certificates. Okay. And then

[1:05:29] there are cloud certifications AWS, Azure and so on. All right. Datab bricks also have official certifications. Data bricks has this free edition that you

[1:05:41] can use to practice the concepts and build projects and so on. Folks, we have provided all the links. At the end, only one thing matters. You should not spend

[1:05:54] too much time in consuming. Let's say if I'm learning, let's say I'm on data I'm learning, let's say I'm on data engineering upskilling journey. If I'm engineering upskilling journey. If I'm spending 1 hour studying, maybe spend 30

[1:06:06] minutes to watch tutorials and 30 minutes to digest what you learned to implement to write code and share it with your friends. Okay? You can use this group finder. So in our discord server, we have partner and group finder

[1:06:21] channel. So find your buddies. It's like you are going for running and if you go but if you go in a group, you'll be motivated. So make a group study group and that way you will make faster progress. And this is the final

[1:06:35] checklist. Make sure at the end of it you have checked most of these check you have checked most of these check boxes and if you have done that you will most likely be already working in a company as a data engineer or you'll be

[1:06:48] building your own project folks. I wish you all the best. Thank you very much for watching. I know it was lot right? You must be tired. You'll be like oh my god am I doing PhD or what? But see it's it's needed folks if you want to build a

[1:07:02] career as a data engineer it's definitely going to be very tool heavy. So you are learning a lot of tools you are learning a lot of soft skills. Therefore this sounded intense but if you take one step at a time folks with

[1:07:15] the help of AI with the help of community with the help of YouTube you'll be able to crack it. It's not a rocket science. I wish you all the best. Thank you very much for watching this video.