AI & Data Careers in 2026: Insights from Databricks Leaders

[00:02] career and constantly wondering which skills, roles and platforms matter the most, this podcast is for you. I'm joined by Archika Dogra, a product manager at Databicks and Danny Lee, PM director at Datab Bricks. Danny is also

[00:18] a longtime contributor to Apache Spark and MLflow. Both of them have been closely watching the evolution of technology, working hands-on with customers across many industries. I'm confident you will find insights from

[00:32] this conversation valuable for your own career journey. All right, let's get started. All right. Uh before we talk about tools or platforms, let's zoom out. What has fundamentally changed in data and AI careers over the last 5

[00:48] years? And what do you see changing again by 2026? Denny, you have worked in data bricks. you have been a contributor to MLflow and Apache Spark both you have

[01:00] been watching the evolution of technology very closely. So I would like to uh know your views on this. >> Oh yeah thanks for the question. Um the thing that people have to understand is even before

[01:15] we started talking about data and AI that we we actually had a problem in didn't follow software engineering practices in the first place. Right? And so we had that minor technicality like even five five definitely a decade ago

[01:28] where you know oh sure let's go ahead have test scripts that was actually a secondary thought right and so we just started the process of basically properly putting software development practices directly into data engineering

[01:43] right and then we introduced AI which threw everything for a loop right uh we're done and we're not going to think about anything right of course be very clear, Sophie. Anybody decides to clip this video, right? Let's be very

[01:58] do it. But so I bring this up because basically even within the context of AI, standard software development engineering principles again, right? We still have to have highquality documentation or requirements documents.

[02:14] And and often those things are forgotten, right? So the number one reason when enterprises or even just startups are trying to build AI projects to actually understand what business problem they were trying to solve in the

[02:28] first place, right? They didn't have very clear OKRs or metrics that stated, okay, we're trying to improve uh the number of clicks on my website. not explain that. They just said, oh, let me just go ahead and vibe code a

[02:42] website. Like cool, but what's the information architecture? what are what's your baseline? Did you do AB testing? Um what are you trying to testing? Um what are you trying to compare? These things never are thought

[02:54] about properly and so fundamentally it's about even in the age of AI can we bring back software development engineering practices so that way we can actually measure what we're building and why we're building it. Right? that I would

[03:09] we're building it. Right? that I would say that's the fundamental right uh that has to be applied on everything before we get into any of the details. >> Okay. And is there any trend you're seeing specifically in 2026? Uh because

[03:25] w coding was a thing that people use for prototyping mostly. Now it's getting serious and we are just mentioning that uh people use it for large code business and it's really working well. So, how do you see things changing in 2026? I mean,

[03:40] any of you can answer this. Yeah. So, I I'll set it up because got some really great details around this, but when it comes to things like vibe coding, it it's not like you're not supposed to vibe code. Quite the opposite. Please,

[03:53] for the love of dear God, go do it. I I think the key aspect is but how do you make that into a mature process? How do you actually allow like the vast majority of time when people vibe codes like if I'm doing like SQL coding or if

[04:06] Honestly, Claude or any of these other systems are just going to go ahead and generate some pretty high quality code for me. All right. But did you think about what is it you're trying to go do? Are have you thought about how you're

[04:20] going to shift it. I'm going to do an audible Dartica just because I think an import super important aspect is that what I've implied is about what your requirements and so she's an amazing product manager, right? And that's what

[04:33] she does. It's all about those requirements. So I' I'd love you Ar to explain like for example the shift that we're already seeing right now. >> Yeah. Yeah. So I can kind of speak to two angles. So first the shift that

[04:46] like how vibe coding is impacting how we ship and iterate through the customer and then also a huge part of my job is talking to a bunch of people in data scientists, analysts and how I've kind of seen them using data bricks change a

[05:02] lot um in the past few months. But kind of starting with the product manager angle of that. Um one of the main things that we do at data bricks is we write a lot. So we write requirements. We try to tell stories to our engineers um about

[05:15] solve and like what are these key requirements that we need to achieve to be able to successfully make the lives of our users easier or unblock things do on the platform. These cycles will take a long time cuz writing takes a

[05:29] long time. Um, and so, and also people just have to sit and like read 20page what these are called, about the problem, the solution, the P zeros, the

[05:41] P1's, the P2s, all that stuff. And so, it was really hard to like communicate your vision, especially the more complicated it got in writing. And so, what vibe coding has enabled us to do is to storytell better to our stakeholders.

[05:54] So, what we do now is we don't even start with writing. I open up lovable. I have this thing called whisper. Whisper flow. It's like it texts to text. I use >> My interaction with computer is through whisper nowadays. All of it.

[06:08] >> Yes. Same here. So that makes it 10 times faster rather than me having to type everything. And we kind of like build out our vision or our northstar like what we think this should look like, what the user journeys that need

[06:21] to be possible have to look like etc. um in a more interactive way so that someone can go and like get that story not by reading like three paragraphs but actually seeing it alive before we have to like actually write any code. Um and

[06:34] to like actually write any code. Um and so that has increased the iteration faster now because we're able to align easier to see something in front of you than to like read and then kind of map

[06:46] it to like an image or a visual in your head. So that's like a huge change that's been on the PM front. on the customers I've talked to. So like I've on intelligent document processing here at data brick. So like how do we enable

[06:59] people to turn 80% of their data so 80% of enterprise data and unstructured data like contracts invoices reports etc into structured insights that they can query downstream make business decisions off of etc. And um a lot of PM is like

[07:15] like in some products there's a very clear persona like a software engineer, a data analyst, a data scientist. But what I've noticed over the last four months is like I don't even know who I'm talking to anymore. It's like I'll I'll

[07:28] have customers come and like build IDP pipelines, intelligent document look them up on LinkedIn afterwards and they're like a business analyst and I'm I've learned is, you know, we have all these like amazing tools in data bricks

[07:42] like the data science agent um that kind of help you build out a lot of things that were previously impossible because everyone has the same job to be done. make a decision off of it. And it's really democratizing like who can build

[07:54] these insights. You don't have to wait or file a ticket with your data contracts into like a set of structured extracted insights. And so that's been a seeing where now that like kind of solidifies how we think about it from a

[08:09] PM perspective. It's not about like necessarily concrete buckets of who of what's the job to be done. Um and that often is shared across a ton of more people can do it because things like cloud code because we have agents

[08:23] embedded to assist workflows in data bricks. Um and all you need is like natural language for doing any of that. So once you build a code in lovable, do like say say that okay go ahead and enhance it?

[08:37] enhance it? >> Um I not yet. So they don't like use the lovable code to actually create production code from it yet. Maybe one Yeah. >> That would be

[08:49] let me into the codebase and I'll just start ruining everything. I mean ju just interject like one one of the things that I I've definitely seen is that like to build a prototype and explain how it's going to go work. Yeah, absolutely

[09:04] 100%. But then for example if you crack into the code for example like just simpler simple ideas like moduleization of code right all of them are guilty of this right? Uh so this this is not specific to claude or to chatb's version

[09:18] of or gemini doesn't doesn't matter right like they just won't modulize the code. So for example, I create a new method that only needs uh sorry a new API that only needs one more additional method. What does it do? It'll literally

[09:31] replicate the entire like uh API and add in that method. It's like no dude, you could have just like there are 20 other ways you could have done this, but this is not the way to do it, right? And so and it's so like I

[09:45] said it's it's great for prototyping and great for even providing initial great for even providing initial guidance on the like which way you're supposed to go right but like I mean this is going to be different especially

[09:57] like I said before like right now today in 2026 like SQL and Python it's actually does a pretty good job these days on if I was asked it to do Rust

[10:09] right like it's just not producing good crates to to put it bluntly. Right. So that that's just the quick call out like that's why I mean just like Arch said maybe at some point it'll get really good. Maybe maybe they'll let us play

[10:21] can probably vouch for me that none of the engineers want you or me to actually play with the production code right now right like just just calling spade spade >> I have seen this trend with marketing manager that we have in our team.

[10:34] Previously she would have design for her landing pages her ideas she will put it on paper show it to her engineers. Nowadays she's doing the same thing Archa she will just create it in lovable and she will just give it to her

[10:47] engineers for integration. All right. So that kind of explains that there is some kind of merging happening in terms of roles. So the next question is earlier

[10:59] the data roles were siloed. So there is data analyst, data engineer, data scientist. from what you see in the data bricks ecosystem today, how are these >> Yeah, I think um I guess I kind of alluded to that also in the last last

[11:14] thing that I just said, but I think they're blending immensely in terms of things that you didn't even think were possible like 3 years ago are like wildly easy today. So for example, if I have like a set of customer reviews and

[11:27] I wanted to classify sentiment, I would have to like go to like hugging face download like a classifier model like you know use emoflow try to figure out how well it does maybe serve that if I want that to actually be something in

[11:39] production and that like had a very high technical barrier for doing like a relatively simple task like classifying natural language or maybe today it feels simple because of the tools we have. Um, but today like it's very easy for anyone

[11:53] to like call the open AI API or something. Um, so like I see a ton of people using the foundation model APIs in data bricks. Um, that aren't just like ML engineers, they're like data scientists, data engineers again, like

[12:05] analysts. Um, they're using like our AI functions. So we have these like task specific AI functions like AI parse document, AI classify, AI extract that like is a very friendly interface for people to do things that they've never

[12:17] need to do that like handoff of like oh I can't get this insight from this data or I don't know how to like train a model to do this thing. And so I think

[12:29] um there's a lot of blurred lines between like who's doing what and how people are deriving analytics from what kinds of data. Um and especially as I mentioned kind of in the IDP IDP space we're seeing that a lot. Um because

[12:42] been untapped for years. It's like people have millions of contracts and reports and invoices and it's been a very manual process to like access any insight from it. So people are very hungry to to turn that into something.

[12:55] Um and the barrier of entry in doing that is like super low with the democratization of like APIs, the interfaces that we're building on top of those APIs to help accomplish some of these tasks like as I mentioned extract,

[13:09] these tasks like as I mentioned extract, classify, etc. So yeah, it's it's a wild mean, obviously getting to production is like one story, but anyone can, you easier. >> Actually, that's a good segue. So, let

[13:22] me add in exactly that point. Well, our go is absolutely right. The barrier to entry is so disgustingly low. It's awesome. Like, seriously, it is amazing. And as you can tell from her, from the way I'm speaking, it's a lot of fun.

[13:35] It's exciting. But, and there always is a butt, right? But productionization, the bar may actually go higher now, right? It depends. Okay, the with the

[13:47] written, it actually does simplify the flow and in fact productionization get me wrong, I'm not this isn't like me going I'm trying to scare you guys. No, quite the opposite. But then for example, let's just take the

[14:00] unstructured data of like processing in IDP intelligent document processing and you're processing all of these PDFs. All right, how do you prevent duplicates? How do you prevent uh dirty data, corrupt data, things of that nature,

[14:12] right? uh the volume that you're accessing. Uh are you going to do add can index it or whatever else, right? There's all these additional steps to place this into production because when you're playing with it, you're okay to

[14:25] wait a couple minutes for you to take a couple documents and chuck it in there, right? But if you're processing thousands of voices, you know, every hour, right? You obviously can't do that, right? need to have this built-in

[14:38] pipeline that actually can handle all the different issues including data corruption, duplication, and whatever else, right? And so that's the key thing I I'd like to remind everybody. So it's like, yeah, yeah, the heck yeah, we're

[14:51] and we should, but everybody that is about to make this into production, the thing they have to remember is the complexity of sometimes we have to still go back to that old-fashioned software development life cycle to actually make

[15:05] Otherwise, what ends up happening is that you're putting a lot of gobblegook into your pipelines and then, you know, it's the old garbage in, garbage out. So >> yes, I totally agree. I think the AI tools and vibe coding is making it

[15:20] easier. At the same time, uh platforms like data bricks, they are also evolving. Now it's a unified analytics platform. You can do everything your data engineering, data analytics, AI at one place. Previously, we didn't have

[15:33] that. So we had lot of siloed tools which you are kind of connecting through like hard coding and that process was very very uh laborious and uh timeconuming. I have this AI and data consultancy

[15:48] company called ATL technologies where we had a client in Washington DC. They are had a client in Washington DC. They are into medical data uh extraction process and this is I think we are having conversation few years back. He was like

[16:01] okay I have terabytes and terabytes worth of data. What do I do? People say data is gold but he was not sure what to do. And at that time chat GPD was new and we were thinking about using some some of the statistical models to

[16:16] extract data and do classification and it was so timeconuming. Now we are living in a world where he can use IDP and data bricks platform and can use IDP and data bricks platform and do the entire thing at one place. So in

[16:30] terms of career roles u you know I often like people ask me okay how do you see this career roles evolving and I'm like okay these career roles are merging and okay these career roles are merging and and few years down the road there will

[16:43] be a job opening called a builder a person who can build things right who has fundamentals of course as Denny you mentioned you need to have the knowledge of scale fall tolerance you know all those

[16:56] nuances which are required to build enterprisegrade production systems. So you need to have that knowledge but as far as actual work is concerned it's going to be uh done by these AI tools. So we are heading for a very interesting

[17:10] times. All right. So now let's talk about skills because many people who are watching this video will have questions on how do they evolve their career in on how do they evolve their career in changing times. So many learners today

[17:24] focus uh heavily on tools and certifications. I have seen people who are busy collecting different uh tools in their resumes. In reality, what non-obvious skills separate average data professionals from high impact ones?

[17:41] >> Uh I I'll definitely start and Archer definitely chime in. Um but the first thing I try to remind everybody is that you can either go into this idea of breath or depth and this is not even specific to technical right and you you

[17:54] can do this almost in any industry whatsoever. But specifically in the tech industry, like especially during like the '90s Microsoft years, like we're talking three decades ago, right? We were all about breath because the

[18:07] premise or the context unequivocally was that, you know, we can mold as I'm a former Microsoft employee so I can say this like we can mold people into what this like we can mold people into what they're going to go do, right? But right

[18:20] now 2026 unequivocally even with the age of AI, right? depth is even more important, right? Because it's the only way to showcase that you have a skill

[18:32] set that other people don't have. If if you all you do is show like I can do you all you do is show like I can do React all the way to like fine-tuning a model, that's great. That's amazing, by the way. But the problem is you're not

[18:46] an expert on anything, right? And so that usually is problematic when it comes to the career cycle. So, at least that's my little two cents. Yeah. >> All right. Okay. Now, datab bricks has a lot of money to spend. It raised another

[19:00] lot of money to spend. It raised another $4 billion recently. Uh, so where are you guys spending that money? And what new things can the world expect? I'm >> That's all going to Denny's salary.

[19:15] >> Dang it. We were supposed to pubize this already. No. Uh, not even close. I wish. already. No. Uh, not even close. I wish. Um, no. No, the the reality is you know data bricks is really well known for analytics right and also known for AI

[19:30] but the the reality is that we have also things we're we're creating a bricks document processing which is so important to the community uh there's also the areas of olp uh you hear talk about lakebase or neon right the

[19:45] acquisition where we really need to go ahead and target the world of online actually really funny because I came from the SQL Server team at Microsoft. So it's like it's all it's coming right back around again, you know, for for me,

[20:00] which is really funny, right? And the reality is there's so much investment we perspective and we're doing it from a global perspective, right? So we're global perspective, right? So we're we're expanding pretty much every office

[20:13] like globally, right? You name a country we're in, we're expanding the offices again. Why? Because we're hiring more engineers. We're hiring, of course, more sales, too. But the point is that we're hiring engineers to actually help us

[20:25] achieve those goals, right? And so ultimately, if I was to just summarize ultimately, if I was to just summarize it this way, it's all about agents, it's all about IDP, and it's all about lakebase. Like those are the three big

[20:39] things for us right now. And so, you know, and don't forget flow we toss in that, right? But like the reality is that everything's around the idea of these new agent systems now. Yeah. And I think what's like really powerful about

[20:54] mentioned like we are this unified platform. So we already have our customers already have so much data in data bricks like structured data now more and more unstructured data. And so

[21:06] powerful layer on top of it that will use agentic reasoning to help them get to the insights faster uh than them having to go and like you know hodge podge everything themselves like start with a table and be like oh like what do

[21:19] these questions like things like agent bricks, genie, etc. Um, we're really investing in like making those state-of-the-art quality, really enterprisegrade outcomes for our customers. Um, and I guess, you know,

[21:33] you need a lot of very smart people working on those problems and a lot of resources going into it. So, >> I have used Genie by the way. I love it. Uh, I'm not writing any SQL queries by hand now. Just just talk to it through.

[21:48] to a dashboard now, if I have a question that's not answered by graph, I'm like maybe Genie like has the right data sets or can create the right query around it and it just saves so much time. >> Yeah. It feels like Genie in a true

[22:02] >> Yeah. >> All right. So among many features that data bricks have shipped, which one do you think has impacted the business a lot and why? Archie, I'm pretty sure you are dealing with you're talking to a lot

[22:16] of data bricks customers. So, I would I would let you answer this question. >> Yeah. Yeah. Um, so obviously like I'm biased. I'm like an AI PM, so I'd love to say my own product, but I think one that's kind of sneaky and isn't as, you

[22:29] know, uh, flashy as a lot of the things going on um, in terms of what people are talking about is Unity Catalog. I think that is like the strongest foundation for anything you're trying to build on top of it. So, we basically, for

[22:42] context, Unity Catalog lets you govern your data. So the moment that it enters data bricks, it's like super secure. You have full lineage of how it's used downstream, where it came from upstream. And for IDP for example, that's like

[22:55] super important. It's like how do I understand which which data sources this came from, who had access to that? How do I transform that into like silver or gold layers of insights? Um how do I maintain the lineage of that? And so

[23:07] when people buy into data bricks, even our AI products, a big reason as to why they're buying into it is Unity catalog. They're like this is a really safe uh framework to innovate on top of because like agent sprawl is like a huge thing.

[23:20] my god people are building agents. I don't know what data is sitting behind to get access to the data that this agent is built on top of. And so UC has to make sure that everything we build from like an innovation perspective is

[23:34] don't have to worry about that and they can move fast because they're not worried about all these like security, compliance, governance concerns that inevitably come up in every single enterprise. Um, so I think in a way like

[23:47] UC has really helped make AI safer and um, that's been like a huge value ad to an enterprise because as a person I don't like as an individual I'm like not the moment that you're trying to build this in a company or an enterprise like

[24:02] >> Yeah. When you are running agents at scale, right, you don't know, you don't have time to monitor them. And when your comput company's reputation is at stake, you want to make sure that there are uh proper boundaries and guardrails set up.

[24:18] catalog plays an important role. >> And by the way, we had a couple of data bricks tutorial on our channel where where we have covered Unity catalog. So if any anyone is watching this video and you want to learn technical details, u

[24:32] it's all there on YouTube. you can watch it for free. Uh so we conducted a little survey on our YouTube community and we collected the questions from people. So I would like to mention one of the questions that we received which is in

[24:47] the future will data analysts need to learn data bricks and will it become an important tool in a data analyst skill set? So traditionally data analysts

[24:59] focus on BI tools, Tableau or PowerBI, SQL, Excel and Python. These are like four main pillars for data analyst skill set. Uh how do you see data bricks

[25:11] uh you know joining uh this this >> I will absolutely take this one. Okay. And the reason why is because uh you know I mentioned I was former SQL server right? Well, uh, so I was SQL Server as

[25:24] in the database, but I was also former SQL server analysis services and I also helped build Power now is known as PowerBI. Okay. So, we were always talking about the, you know, that person that dude or that dude that that knew

[25:38] everything and they binded everything into Excel, right? And so the will the data ass to learn data bricks? Yes, 100% they're going to learn, but not I don't mean it in a way where all a sudden they have to know Spark or they have to know

[25:54] already talking about when we talk about Genie, we're there's another there's the other half the AIBI, right? Which is all about the ability for us to use AI

[26:06] because we're using Uni catalog. We understand the metadata of the tables, the volumes, the functions, the models that we're using and we're actually able to automatically generate the code underneath the scene. So you're typing

[26:21] natural language or you're speaking n natural language to data bricks. Data bicks is then able to say oh you've asked for gross profits. Okay, what's the definition of gross profit? Which tables am I supposed to query? the user

[26:35] that analyst doesn't actually need to know all this information. It's already in uni catalog. We've already defined it for you. So now when the query is run in thinking? What's the SQL query? The reality is it's backed by UC. So that

[26:51] way it actually generates the the correct query against the correct tables. Right. Right. And genai is all about hallucination and we love the fact that it can be creative but it can only work when you can trust the underlying

[27:06] numbers and so the fact that you have UC allows you to do exactly that trust the numbers that it's generating. So for example a common approach especially in the BI world when they're using power pivot excels they they keep on dragging

[27:19] and dropping different dimensions to look for outliers. Wouldn't it be just better if I just simply asked Genie, asked AIBI to say, "Hey, what's odd about my data? What are the outliers?" And then have it generate the whole

[27:32] thing, graph everything out so you can see it right away and deliver a report to you right from the get- go. Right? That's the power of what we're seeing, the new version of BI. A lot of people want to stay in the world of Excel. And

[27:46] don't get me wrong, Excel's powerful. I I'm not against Excel, but do I want to have my analysts spend hours upon hours dragging and dropping dimensions or would I rather have data bricks just generate the uh anomalies or the issues

[28:02] right away and so that way the the analyst can actually make use of the that's how I look at like that's why I'm like super happy to talk about Genie, like it fundamentally changes the way we do analysts

[28:16] and lot of time that data analysts spend is uh many times see they spend time building dashboards which executive team will use but they also time spending writing a lot of ad hoc queries. So now with Genie and you didn't catalog if you

[28:32] have understanding of your uh data ecosystem you just use Genie to generate the answers for your management. So that way I think I I agree that having level where you need to know the fundamentals of Spark and so on but

[28:48] overall tooling and platform uh literacy is going to definitely be helpful. All right. So let's say if you have a best friend or someone whom you know wants to study data bricks now uh what

[29:03] advice would you give to them? >> Well, we've got the amazing data bricks free edition that I think everyone should start off like I personally learn something rather than like watching videos or uh reading documentation. And

[29:19] >> of course these videos from coexistas are amazing. >> True. True. Um, but yeah, I mean, if they really want to get their hands dirty, we've really I think we've spent a lot of time

[29:32] revamping the or I think we kind of newly released the free edition in the community edition. Um, and I think it has like a really awesome set of features that help people get dirty in data bricks. like maybe go in with like

[29:45] a sort of task or data set that you want to do like Denny mentioned like what's your OKR basically and then kind of work your way through data bricks to figure out how to do that. Um but yeah I think learning by doing is really great but

[29:58] Denny I'm sure you have a lot of other resources that would make the doing part >> No just add when you go data free well. So there's a whole bunch of free training database training as well

[30:11] for data free edition, we give you access to a bunch of data sets. We give Uh in fact our friends here at code basics, they are doing some amazing free training as well. So so the that's unequivocally 100% data bricks free

[30:27] Yes, I'm hawking for these guys too because I love these guys, right? That's why we're here, right? That's the whole point. This is the best way for you to jump start yourself and because I'm with Arctica 100%. it. Everybody can talk

[30:40] Yeah. >> And all it takes is a internet, uh, laptop and a willpower, right? You don't You have free edition. You have YouTube tutorials. Uh, you just need motivation.

[30:56] >> All right. Thank you very much, folks. This was truly insightful. Uh, any any last comments or thoughts that you want to share with our audience? So we have a huge community of AI and data learners. Uh so go ahead if if you have anything

[31:11] Uh so go ahead if if you have anything uh any last comments to share. >> Uh I guess the one little tidbit that I would add especially Allah the vibe coding since we started there remember to have fun. A lot of people forget that

[31:23] like you start with something fun. Play use data tree edition but play like because that's actually how you're going to best learn how to do the stuff. Amazing. Thank you very much. I hope you found some useful insights from this

[31:39] conversation. If you have any questions, there is a comment box below. Thank you there is a comment box below. Thank you very much for watching.