Databricks Head of Applied AI on Industry Trends, Career Growth, and Staying Sane

[00:01] often give to to my mentees who are like when they describe to their situation hey how do I how do I level up and when I hear the rest of their context it almost immediately jumps out as no I don't think you have a hope here for

[00:15] today's hello interview I'm really excited to introduce rendra uh rendra is the head of Applied AI at data bricks he's also worked at meta as a director responsible for video recommendations and then as a senior staff engineer at

[00:31] Google so really excited to have you on the channel rendra thank you for having the channel rendra thank you for having me I guess um let's jump into some some high level discussion you've been in the industry for for quite a while and and

[00:43] seen a lot of different companies I I think over the last say decade there's think over the last say decade there's been this very big shift I think it it started with covid and and it leads up to now in really The Vibes of the

[00:55] industry I think some of this started with a series of layoffs the RTO fights that are happening in companies and just trying to figure out what is the new order for Tech I'd love to get your your perspectives on this what's happening

[01:09] behind the scenes especially amongst leaders in these big companies that are driving all these cultural changes in general there is this the big shift has been driven by Ai and how AI has changed the way people work we seeing a huge

[01:26] amount of quote being written by Ai and at the same time we also seeing seeing at the same time we also seeing seeing some amount of like postco sort of drop in engagement among new employees so if you take those two like okay on one hand

[01:41] you have a bunch of AI assistant that everyone has access to which makes their coding faster cleaner better and all of that on the other hand employees feel less engaged because they're not meeting people face to face specifically postco

[01:57] so I think like those two the biggest forces like where are you going to be working out of and what kind of technological help are you going to get when you're building software so in in some sense what's happening is the

[02:10] easier stuff is going to keep getting served by Ai and the harder stuff that requires inperson collaborations and all of that might be the main reason why we have human software Engineers rather than machines doing everything because

[02:25] there's a lot of incredible amount of subjectivity in what we build how we subjectivity in what we build how we build it and so AI is not at that level like if in some sort of hierarchy of increasingly complex AI problems the AI

[02:39] of today is still very much following instructions tell me what you need and I'll do it it's not actually creating those instructions on its own sufficient amount of time to really replace humans human actions that are sort of at that

[02:53] higher level of like what should we build how should we you know prioritize a versus B what is our next strategy for the next 6 12 18 months so if you just take that all together we are in a world

[03:07] where there's this opposing forces of use more AI but also have more human interaction so it's a somewhat polarizing right like on one hand you're saying okay you have like an assistant that does everything for you but then

[03:22] more and more of that like your value proposition as an engineer in Tech is to like figure out what is it that you want to do and for that being in isolation is

[03:34] harder and I'm not saying in I'm not I'm not sure about this inperson versus um you know physically in person versus virtually in person I'm not particularly I don't have a strong opinion on that okay but I do think that

[03:48] Engineers that communicate better and collaborate better with others are going to be able to ask those right questions better and that's something that you can't do by sitting in isolation and doing things because value that you

[04:00] bring by just figuring out the difficult coding aspects of what you do is no longer such a big value prop because you can always like get a

[04:12] machine to do that for you does that make sense yeah I think it does you feel like a lot of this is actually driven by really the response to gbd3 in in some respects because I I guess another interpretation that I've heard floating

[04:26] around is that all of these companies were just waiting for this moment where they'd be able to push on efficiency and kind of pull back some of the benefits that you know software Engineers have enjoyed over the last decade and then

[04:38] you know this uh economic uh weakness they just given them an opportunity but you're saying actually no the job of software Engineers is changing in part from technology and you know this is this this shift that we're seeing inside

[04:52] bigger companies is largely a response to that is that correct I I I mean my all going to level up I mean that's the expectation I mean up it means that you know the very lowlevel things that you Ed to get paid

[05:08] for is going to be done by machines and so you you have to level up at the same you have to effectively level up in that sort of in one semantic or two semantic layers above where you used to sit at the same level of job like job level

[05:24] like you know um that's kind of what I'm trying to say like I think is it reduc ring employability I'm not sure I I I think it's definitely reducing it's going to eventually reduce the number of jobs at the low level but I think there

[05:39] are many competing forces that will decide the total market right so it's like the efficiency with which people can like software Engineers can build software competes with the increasing like number of different problems that

[05:53] need to be solved with some software sure and it's very difficult to imagine that this is not going to have an impact on this profession you you said level up interpretations and you sound like you're in a really good position to at

[06:07] least forecast for us about what this means I think one interpretation of leveling up is like along the traditional career axes and in a lot of architect and being responsible for bigger decisions or aligning teams I

[06:22] think this could also mean you know taking on more like product style responsibilities where you're instead more respons responsible for shaping the direction of the business or the product when you say level up in this context

[06:36] what did you have in mind I'm actually talking about the latter where you are going to be much more responsible for the higher level purpose of the software you're building I don't know actually the economics of like basically at the

[06:49] the economics of like basically at the same level professional level or salary level whatever you call that people will expect your your the leadership in these companies would expect more from you as it pertains to understanding the

[07:02] software that you're building and being much more aware of like the implications of changing the software a certain way or building new features because the just writing simple code that would just make something happen is just not going

[07:18] at your level for so what are you going to be valuable for you're going to be valuable for deciding the next level of um at the next level of whatever you call it complexity or SE like next SE next higher semantic layer of how you

[07:33] see the software right okay what is this for who is the user like people will like engineers at the same level as they were like five years back are now going to have to understand the life cycle of that software the critical user journeys

[07:48] of the software much more than they used to because why else would they hire a human for that role sure software Engineers will have to be exploit their

[08:01] humanlike characteristics more on their job interesting like things that are more difficult to replicate so I'm sure your various traits and your your your your various traits and your your your various qualities uh fall in you know in

[08:16] different buckets of like how how replaceable is this skill so like if you look at the distribution of like easily replaceable to much much more difficult

[08:28] to replace your distribution has to shift to the right your skill set has to shift to the right in in in that sense right so you should spend less time doing things that machines can easily do in the future and more time more time

[08:42] honing the skills that you know machines have very little chance of or or or or skills that other people don't trust machines to be able to do like it's not even clear if humans are better at

[08:56] judging Aesthetics of photograph it's just the other humans around them don't trust machines with certain tasks definitely um yeah I think there that's a subtle that's a subtle difference a ton of I I I like the the emphasis here

[09:09] but I guess in my mind this also kind of aligns with classical career advice that many Engineers have received which is to say you know your hands on keyboard sort of code writing um will certainly get you pretty far in your career but at a

[09:24] figure out how to influence be a part of the business decide on larger Direction design larger systems so I think in some sense most Engineers are kind of primed for graduating on these semantic layers that you've talked about but many

[09:38] struggle with that and I I love some of the writing that you have on LinkedIn you've got a lot of hot takes about management and about leadership what advice might you give our audience who is sitting in a position where they feel

[09:52] like they could be doing those things but either their leadership isn't giving them the opportunity or they feel somehow hobbled by the organization this somehow hobbled by the organization this is a very tactical advice but the one

[10:06] that really resonates with me is communication skills and uh like how well you communicate gives your leadership the communicate gives your leadership the impression that you can think even

[10:21] though between all else being equal someone who communicates better is usually given those higher level tasks is what I've seen people judge other people B literally and they don't have enough time to evaluate others so then

[10:34] sentences that exchange that they exchange with them oh this guy seems really good or this girl seems really like 10 sentences that they exchange like immediately people pass judgment

[10:48] and especially leadership part of leadership's job is to judge whether we like it or not that's the job like or constantly evaluating who's good who's who can do this task who can lead this area who can sort of communicate to the

[11:01] area who can sort of communicate to the CEO who can you know uh you know part build Partnerships between teams who can resolve conflicts like constantly that's those are all higher level tasks than sitting in a corner writing code for a

[11:15] well-defined problem right so I would say like working on your communication skills not just this is not just pure English or or whatever language you speak but it's about like what do you speak about and with how much clarity

[11:29] and Conference can you speak about that is a very tactical way to sort of level up in my opinion again this level up is more around the level up that we just promoted although that could be an outcome of this but the main thing is

[11:42] like instead of telling you can you can you write this code or can you fix this bug to can you take charge of this area can you tell us what we should do here that's the level up I'm talking about like going from less like Concrete

[11:56] technical tasks to like more abstract higher level uh missions right so if yeah and the emphasis on communication is is really uh kind of tractable at

[12:08] least for most the communication is necessary but has to be preceded by deep thinking around products because you can't communicate meaningfully especially especially the highly perceptive leaders around you they'll

[12:22] judge the M the content that you're talking about and like what what kind of surprising elements are you bringing to the table that means that you can bring a fresh perspective that even they don't that leader doesn't have so so not only

[12:36] do you want to communicate well but you should precede that by studying the problems that you're going to talk about in a level of depth that you sort of haven't planned on before or haven't done before like I love to think in

[12:50] done before like I love to think in terms of like okay here is a problem I'll think deeply about it and when I think when I feel like I have thought deeply enough enough about it I'll I'll go deeper it's like going down a mine

[13:03] and then seeing if there's another hole that you can go down further that's I like that analogy it sounds like becoming an expert and communicating well are are kind of two of the the Tactical pieces some of what you've been

[13:16] saying is has been really about this shift in the style of work but I think for a lot of Engineers they originally get into software engineering in particular because they're fascinated by the technical challenges and there's no

[13:30] doubt going to be technical challenges in the Horizon of increasing difficulty what would you say for an engineer who really wants to stay in that zone how do they continue to level up and be more effective especially in light of the

[13:43] advances in AI is it that they've got to find a company that's working on these kind of Frontier problems or is it that they need to change their approach where they're at I mean honestly let's be really honest there are going to be

[13:56] companies where the only way to level up is to like do a few things like manage people or you know do all this kind of like presenting to senior leaders so there's a narrow way of like leveling up in in terms of levels and all that so

[14:11] definitely in some cases if you're in a company that's somewhat like traditional with how they grow people you do have to find a new job in a in a company I mean that's the pragmatic advice I often give to to my mentees who are like when they

[14:24] describe to their situation hey how do I how do I level up and when I hear the rest of their context it almost immediately jumps out as no I don't think you have a hope here the kind of work you want to do the kind of deep

[14:37] technical work you you want to do and continue to do is not going to be appreciated at this company that you're at so your only bet is the only pragmatic thing to do at that point is to like find another employer who would

[14:50] to like find another employer who would respect that the depth in put another appreciated there are jobs where that depth is like you you FL flatten out some level of depth in in terms of technical uh tech technical Excellence

[15:04] or technical depth or technical complexity but if you are at a company where increasingly complex problems exist and you are not really solving those problems you have to find a way to get there to take on bigger and bigger

[15:18] challenges and how you do it is again I'll go back to my old friend communicate to some extent that you can lead increasingly complex problems so that people can trust you with those problems and you you have to assume good

[15:34] good faith effort on the behalf of your leadership to find the right problem for you based on what they know so you have to make them know more about you by capable of doing more than what you're doing now through various I mean again

[15:50] communication you can also write increasingly complex design documents about the work you're doing that shows your level of depth in your thinking so that somebody can bet on you for the next level of comp complex work so Step

[16:06] One is make sure you're at a company where the dep depth complexity all of that is appreciated and second is once you're there or once you've made sure that you're there make sure that you're also like you can be UST you're giving

[16:21] enough signals that you can be entrusted to solve increasingly more complex not always going to be purely technical problems some of this is also going to be um people some combination of Technical and people problems because

[16:35] almost I've never seen a case of increasingly more complex problems where increasingly more complex problems where people people problems did not emerge it you're solving increasingly more interesting problems that your top

[16:48] leadership cares about there will be hungry like hungry wolves there'll be folks like you know looking to have a piece of it right and so there you'll deal with that complexity you to find know smart ways of sort of dividing and

[17:03] conquering or finding your n or something like that but there's definitely it's there's almost at least in software maybe in certain very in software maybe in certain very technical areas of software or or or

[17:17] Hardware that you can just do increasingly harder things in complete isolation of the people around you but I I fail to believe that complexity that is necess to to level up in software does not involve people it's if

[17:35] you're if you're constantly uh asking help from a manager or a colleague or something to to help you solve you'll you'll you'll sort of Plato at some level because they'll be like okay this person needs uh babysitting around

[17:51] that they'll they'll be this is a this is a blocker for your career growth the people aspect you can't you can't can't remove entirely from the picture and it it seems like the the advice for you is if you're in a company where you're just

[18:05] not getting that technical opportunity that it's probably an imperative that you go find that position because the the the C is rising and then for those that are in a reasonable position but maybe not getting that work then you

[18:18] come back to this idea about communication kind of finding sponsorship and figuring out how you can start to work on those problems yes yes and you and you sort of of and you people are like the thing I find people

[18:32] problem they have to deal with as they go up a lot of time people just say I'm because I don't want to deal with that so that's your choice if you have to like always rely on just increasingly more technical complex problems to in

[18:47] order to grow in your career there will be a time when you'll be at a Crossroads about whether you want to take on that like because you know it's a zero sum game like one overly simplistic thing I like to think about is when you're

[18:59] solving infinitely interesting problems you'll get zero pay and if you're if you're solving in extremely terrible problems you'll get like the like infinite amount of money so like we all stradling that sort of that those two

[19:14] extremes like we're moving like left and right of it because because that interesting I'm assuming is correlated with like demand uh sorry Supply supply of talent

[19:26] who be willing to do this like for example AI jobs like the number one thing people tell me is like oh I want to get an AI job but I'm training a model that's the number one thing I love to write the pie torch code I must be

[19:38] writing the pie torch code I love to do that yes you have to be if you don't do that it's not real machine learning so the demand for that is so high that they can easily find someone who will just do it for cheap but and you you almost

[19:52] think like it's the opposite but like people who can train a model following some instruction versus people who can fundamentally innovate modeling through deep expertise is very different so the pay pay band there would be very wide

[20:07] but just wanting to train a model people will work for Less because they get to do it so it's it's a I I think it's quite flawed what I just said but it's an interesting yeah it's a it's a model I I guess I think I've always seen it

[20:21] most acutely in the games industry where you know everybody wants to become a game developer a lot of people do and as a result the working conditions game industry veterans would would agree with that not good yeah you've you've

[20:35] kind of U painted a picture though it sounds like there's some urgency for people to be just gradually elevating you know the the level that they're operating at and in some respects that people might be mispositioned that where

[20:49] they are now is not going to take them there that means that a lot of people are theoretically looking for a shift they're either looking to move into more technical roles or maybe up that that semantic pyramid that you talked about

[21:01] what in your mind is kind of the key for people in making that transition a lot of education is going to be involved and that I'm not I don't mean formal education like learning learning to do new things that You' have not done

[21:15] before being in a very uncomfortable position for some time till you become comfortable using that skill to interview well at these companies where interesting work is happening in that space or even like showing in your

[21:30] things that the company is not working on because this is a very sort of we are all learning like especially in this AI space we are all learning every day there a very sort of learning oriented those who are just finding new insights

[21:45] and are willing to experiment with whatever they get their hands on are uh being sought out because most people there's not a lot of theoretical basis for what we are seeing out there so it's not like there's a very welldefined book

[22:01] that can be written about like all the things we are learning but lots of anecdotes are emerging and we are sort of figuring out like how much of these anecdotes can be generalized into something theoretical or is is it that

[22:15] just going to keep trying different things and those who are able to try more things are more likely to chance upon something great and so then you start seeking out people who have have the tenacity to learn something and then

[22:30] apply it over and over again till you see something good and so you don't need any formal education for that you just need perseverance you need a willingness you just keep doing it over and over again one of the higher level things you

[22:43] can do in AI today is quality which means no matter what kind of AI problem you're solving the initial version of this is going to be shipped so you're going to your job is going to be to make it better right so and this is a kind of

[22:56] want to do that I don't want to keep trying and figuring out but definitely one way to like level up in a somewhat narrow way is to be an expert at just trying failing trying succeeding that Loop of like trying succeeding or

[23:11] failing and based on that deciding to sort of merge that change into your and like going through that Loop over and over again till over some period of time you've made a product that was kind

[23:24] of me to something that's really useful and so the difference for any given AI and so the difference for any given AI product is between how many is is like how many different like like people were asked to do that

[23:37] for a single product over what period of time and you've seen that kind of success at Google at at at facebook/ meta we seeing that at like companies like Netflix over the years you know where you just keep getting better and

[23:51] where you just keep getting better and better at like you know quality of AI systems right and that's somewhat narrow but like it just came to me because it's like something that neither requires too much specialization nor requires like

[24:05] being in the right place at the right time you can just be the you can just be in the minority of people who would want to do that I like the emphasis on kind of grit and tenacity as well I think those are two qualities that over the

[24:18] long run end up really separating out people who are successful from those who are I think that grit and tenacity is like very applicable to scientific discovery in general and this is a we are in that phase where we don't really

[24:31] understand what we are obser we like all literally observing emergent behavior in in in neural networks and we're trying to like there there's there's not you're not going to have an aha moment here in most cases you're going to just keep

[24:43] iterating and finding new new local Optima it's a very interesting idea to say we're kind of entering into a newly scientific age for engineering where we have to be more Curious yeah because it's too complex for any kind of

[24:56] theoretical basis uh Rend I've got one last question I know we're almost out of time you you have this very contrarian take that I actually love a lot that really emphasizes Having side projects and hobbies and not being overly focused

[25:10] on work and you feel that this is actually a way to kind of in some sense get ahead even in the work itself I'd love to hear a little bit about why you a lot of really interesting hobbies that we could get into and probably won't

[25:23] have time but tell me why you think this is so powerful I think like the big big big thing is like me mental health I mean to me that's the number one reason I have so much so many hobbies because

[25:37] if I don't have a good day at work I can go back and do something else with my art if I don't have a good day in art I can come back to work and find some value because ultimately we are all pursuing feeling valued all the time

[25:50] I've distilled that it down to just that like after talking to hundreds of people by the way I have lots of men mentees and that's a completely new thing that happened this year when I like started talking to like 170 different St

[26:02] strangers about their problems and I found out that this is the number one thing that at this is a bias set of people who decide to you know choose me that those conversations is that

[26:14] that's it basically if you then take a step back feeling valued can should need not just come from your work it need not come from your family it could also come from something else in fact with with work and family it feels somewhat like

[26:30] uh you're expected to do a certain things whereas with with Hobbies one of the things that happens with even little effort you you get a lot of validation for what you do because people don't expect that to be your profession I I

[26:43] like that I guess the the key here is that for people to have any sort of intensity at work they need to have in some cases an outlet where if they fail it's not the end of the world you've got other things going on and I would say

[26:57] like two two more things the outlet a lot of people say oh I I also play the guitar but the problem with that is yes you frustrated you play the guitar at depressed at work you'll stop playing your guitar so it has to be more than

[27:10] that it has to be something where you are accountable for delivering something and you are evaluated on it in some sense without too much pressure right in my case like I'm I'm I'm I'm running a theater production right I have to

[27:24] deliver it on by a certain date and people will come and watch it and give me their feedback and if it works well you you you sort of like get a great feeling it's a warm fuzzy feeling when that happens but if it doesn't work well

[27:40] this is not your day job so you can sort of go back you can improve you can learn from those mistakes and you can go back and to the drawing board and do it again better this time and you keep getting better and better render this is a great

[27:53] way to round out our discussion I love the the ideas that you brought to the table from kind of how to be effective in in kind of this new world the newly scientific economy how to progress your career but also how to round out as an

[28:05] individual and make sure that you don't burn out this has been super informative burn out this has been super informative really appreciate the time