[00:03] Good morning, everyone. I'm Kriti Dangay from India. Watashi wa Kriti desu. Uh a little bit about me. I'm a senior software engineer at Uber. I've worked at Uber for almost 4 years now, and I work in app quality teams as well as [00:16] production observability space. And I'll be sharing some of that today with you. So, this talk is about Debug Assist, an AI agent for debugging that we built at Uber. So, today we'll talk about why debugging [00:31] is actually hard in reality. How does Debug Assist help with it? We'll also have a demo, and then we'll see a quick look at the internal architecture. And lastly, I'll share a couple of interesting numbers with you. [00:49] All right. So, what you have in front of you is Uber's in-app bug reporter. So, all the Uber internal employees, as well as beta testers, have access to report bugs on the Uber app itself via the beta version. [01:02] And this is what gets captured in the system. So, let's say I'm using my app, the screen goes blank, or maybe I'm experiencing hangs, or maybe there are battery issues. I can simply just click and report a bug. And for me, it looks [01:16] like saying, "Oh, my screen is blank." That's it. The system captures all the information alongside it. Wisdom internally. [01:29] So, along with the natural language bug description, it captures the metadata around the issue. For example, what was the app that was affected? What was the app version that was affected? What was the OS that was affected? Which city is [01:42] the user experiencing the issue in? Along with this, we also capture a multitude of logs, like network logs, analytics logs, console logs, GraphQL analytics logs, console logs, GraphQL logs, UI state logs, and so many more. [01:56] And as we can see, screenshots are also captured from the device. Um, there are no security concerns here. We only capture the Uber screenshots to debug capture the Uber screenshots to debug the issue later on. [02:09] It's called Healthline. So, Healthline is essentially a app quality analytics tool. Um, similar to Wisdom, it also captures issue. However, it does that automatically. So, every time your app [02:23] crashes, you experience a hang, a jitter, or any kind of performance degradation, Healthline is a system that kicks in the backend. So, similar to Wisdom, it also captures the metadata. For example, what was the [02:37] session ID? What was the analytics ID? What was the user doing before this? listed and many more. However, unlike Wisdom, it captures the [02:49] stack trace all the whole whole crash dump as well. So, we can say which code dump as well. So, we can say which code was actually causing this problem now. almost 50,000 issues are assigned to engineers every year. 90% of these are [03:06] customer impacting and as a result, we end up sending 12,000 pagers to our engineering teams everywhere. So, someone has to debug these, right? Someone has to fix these. [03:21] Before I move on to the next slide, a quick show of hands, how many of us do quick show of hands, how many of us do not like fixing production issues? [03:33] pager at 2:00 a.m. at night and you have to sit all night and debug it and you don't even know where to start because your brain is so foggy. [03:46] The errors are often unclear. Um, we all have received that message, to figure out what went wrong. So, we logic. The context is also scattered. Um you [04:00] have to go through code, you have to go through logs, you have to go through configs, and you have to go through data. And to access that data, there are so many tools internally. Speaking of code bases, [04:12] Speaking of code bases, code bases at Uber at Uber are vast. So, we have multiple mono repos. We operate in a mono repo architecture style, which means that many, many teams are contributing to a single repo. [04:25] are contributing to a single repo. So, to figure out who caused the problem is another problem in itself. And even when you do, there are still cross-team dependencies. The code might be owned by someone else. Even if it is [04:37] owned in your team, you don't know all the code written by your team, right? engineer at Uber takes almost 28 days to resolve an issue on an average. [04:52] And by uh and as seen in a public report done by Stripe, developers spend 40% of time debugging an issue. And this is why we developed Debug Assist, which aims to improve the app quality and enhance the debugging [05:07] experience of developers so that we can do what we do best, develop. generally follow. It is a end-to-end agentic resolution journey. It starts by [05:21] discovery. Uh discovery happens through Healthline and Wisdom, as we just saw. Next, we move to the auto triad stage. This is the stage which figures out what is the priority of this issue, how severe it is, and who actually owns it. [05:37] here? Then starts the Debug Assist actual pipeline. It does a deep root cause analysis in under 30 minutes. We'll We'll see how it does that. We'll take a deep dive here. [05:51] And lastly, we aim towards a resolution. So, we don't just want to provide a big dump of data to the developers. We want to make it actionable. We want to to make it actionable. We want to resolve the issue end to end. [06:07] part. I hope all of you are attentive. So, this is the Wisdom screen that we just saw earlier. And this is a real-life example, by the way. One day one of our Uber developers woke up and he booked a cab to office. [06:20] While waiting for the Uber, he realized that his phone was burning up in his pocket. So, he filed a Wisdom bug report. All he said was "Uber app is responsible for nearly 30% [06:32] of my battery usage. My phone is heating up. That's it." Then he happily sat in the Uber, came to office, opened his laptop to look at the issue. In those 20 minutes, [06:44] created a fix without him clicking a single button. All the developer had to do was review it and deploy it. So, how does this magic happen in the background? [06:58] As you can see here, we Wisdom has captured the natural language description. It has captured the screenshot of what the developer was seeing while booking his cab right as well as a screenshot uploaded by [07:11] developer which says that "Oh, my battery usage is being consumed by battery usage is being consumed by Uber." And we can clearly see it's 29%. There are metadata and logs associated it with it as well. [07:30] Debug Assist. Um on the left side, we can see that Debug Assist has root caused the issue to a hot looping code path, which means that there was certain code path that was supposed to run in the background, [07:45] but it was actually being treated as a foreground loop path, which led to high CPU cycle usage and high battery regression. Now, along with this RCA, it also tells us that, "Oh, by the way, the app was [07:58] backgrounded for 17 minutes. It was not in the foreground." And all of this rooted is rooted in evidence. So, on the right side, we can see there is a evidence timeline. So, we know that the agent is not hallucinating and we [08:12] are somewhat confident in our fixes. And lastly, we can see the developer was pretty happy. [08:24] crash that was occurring on the Uber app due to a bad rollout. Essentially, notification very, very quickly, the app was crashing. This was a high severity crash, that is why it is marked as a P1, which is priority one. [08:39] And DebugAssist created a whole deep deep RCA for it. It pinpointed the code which was causing this and it also created a fix for it. The fix was sent to developer via a Slack message later on. And in this case, the fix was merged [08:55] and deployed the same day. So, we ended up catching the crash at a 5% rollout up catching the crash at a 5% rollout and fixing it at the very same day. [09:09] DebugAssist actually does. How does it actually accomplish this? So, as you can see, the agent is built on a LangGraph pipeline. It is a mix of deterministic and LLM nodes. Why LangChain, you might ask? [09:24] So, agents generally have a planning phase and an execution phase. What LangGraph or LangChain allows us to do is bypass that planning phase. We give the plan to the agent that these are the steps you have to follow every [09:39] are the steps you have to follow every time. So, it removes certain levels of hallucination from the agent. We're not totally dependent on the LLM to come up with a good plan to fix the issue. We control the plan. [09:52] So, it starts with a context collector node. This node essentially captures all the data that we just talked about from health line, wisdom, all the logs, etc. not an LLM node. [10:05] Why not LLM in this world of AI? Because everything doesn't have to be via everything doesn't have to be via agents. It's simply fetching data. Um a lot of context bloating. We don't want to add latencies to the agent of the [10:19] APIs that we already know of. And of course, LLM cost is a huge problem these days and we don't want to bloat that as well. Um also, this allows us to parse and pre-process and prune those logs. So, [10:33] these logs that I'm talking about are huge. They are megabytes and megabytes and gigabytes of logs. You cannot simply dump them to an agent. You have to pre-process them in some ways. Next, we move to the classification or [10:46] the RCA node. This is a LLM node. As you can see, it's highlighted in blue. Now, this node looks at all the data we just fed it. It first tries to categorize the crash. Is this even happening because of an [11:00] Uber issue? Or is this a third-party library code? Is this an infrastructure issue? Is this a network issue? Do I need to create a PR to fix this? [11:12] Once the categorization is done, it comes up with a confidence score as we can see. And finally, if it feels I need to get more data, it goes and queries Uber ecosystem. So, it does that via MCP connections like Jaeger logging, [11:27] um Jaeger distributed tracing, logging, uh Uh, incident data. Is there any ongoing incident that's generally impacting the users, etc. Once that is done, it performs the RCA via sub agents. So, we fan out multiple [11:43] sub agents in parallel and let them do RCA and once we have that, we consolidate the results. To give you example of some of the sub agents, um, one sub agent is breadcrumb analysis. So, we give all the session [11:56] data to the agent and we ask it to recreate the timeline of the user. This not only helps with the good RCA, but it also helps us reproduce the issue. Another agent, um, I can share with you is crash correlation agent. So, whenever [12:12] a crash is occurring, can I correlate it to a particular version release? Can I uh, confidently say that this version is experiencing the crash? [12:24] the next step is a fixed node. Now, before I move to the fixing part, um, you might see there are certain configs given in every node and RCA node actually runs on a lower model, Sonnet, with limited number of turns. You might [12:38] it's really not. If you think about it, it's mostly looking at the data and gathering information and summarizing it. it runs on Sonnet with limited number of turns because, uh, with trial and error [12:52] we figured out that if it's taking more than 20 turns, it's generally going in hole. And that's how you also put guardrails around your agent. the fixing node. Now, this is the node which, which needs [13:07] the big brains. So, it goes through the code base of Uber and Uber code base is billions and billions of lines of code. Now, it tries to figure out which module might be affected, which code path would be affected, and all the states are [13:22] state of the previous node as as well. It has not forgotten anything. Before fixing, we first try to mitigate the issue, too. So, as you can see, most of the Uber issues are behind a feature flag, and [13:36] this node connects to a feature flag MCP. It tries to correlate, is this feature flag causing this issue? And um mind you, in all of your phones that you're using Uber app, there are at least tens of thousands of feature [13:52] flag that are active in the background. So, just correlating that one crash to one of those feature flags is already a very complicated task. simply goes and rolls the feature flag back, and the issue is mitigated. And [14:08] then you're um free to fix it on your own time. So, we have the fixing and the mitigation step here, which runs on Opus with a higher number of max turns. Next, we go to the validation stage. [14:21] legacy reasons, um but it's actually a validation stage. So, in this case, the agent tries to reproduce the issue either via a unit test or a mobile automated end-to-end test. [14:34] Some flows at Uber have a automation test that runs on simulators or emulators directly, especially for the core flows like booking a ride. validate its own fix. [14:47] So, you not only just give the PR to a developer and say test it, you also validate it and increase the confidence of your own fix. And finally, when you are happy with the fix, we create a diff or a PR in this [15:00] case. The PR is attached to the Jira, to the issues on healthline, wisdom. Um it is uh sent to the developer via Slack, and all the information is updated internally as metadata. [15:14] here is this is the basic skeleton of the agent. It never changes. What changes is the skills and plugins. So, each of these LLM nodes have their own skill. How to create a good PR at [15:30] Uber? What are some good practices we follow for test planning? How to fix an issue in Android? Do I want to do lazy loading always? Do I want to defer something to background? etc. Now, these skills and plugins are, as [15:44] the name suggests, pluggable. So, the skeleton remains the same and all of this core business logic is outside the skeleton, which is fetched Which means tomorrow if I want to add a new agent, it's a matter of crowd [15:58] new agent, it's a matter of crowd sourcing the skills. did we manage to productionize an agent at Uber scale? Now, running something on your local machine is one story, but running it at Uber scale is another [16:12] story altogether. So, as I said, we have provided the harness to the developers. We have provided the steps to follow. We have provided the runtime environments, the dependencies, etc. All the teams have to [16:25] do is bring in their own skills, own specialized skills on top of what we already provide. Which also reduces the load on us because now we don't have to maintain a multitude of skills and plugins and [16:37] whatnot for everyone. You bring your own specialization and we provide the harness. So, right now we have almost eight agents running at scale across five mono repos and all of them share a single [16:51] All of the code is inside a Python mono repo and it is packaged as a Python binary called pex. The pex is uploaded to a common artifactory and every time an agent wants to run, uh it it knows its type. So, according to the agent [17:05] type, it fetches the correct binary. It fetches the correct Docker runtime and Docker images, which provides all the dependencies and runtime environments. And finally, it fetches all of its pipelines, which is uh scattered across [17:17] different mono repos. Again, um I would like to highlight that this is a Harness, and even the skills and plugins for every agent is fetched and plugins for every agent is fetched at runtime. So, we don't do a dump of [17:31] all the marketplace plugins. Uber has a skills marketplace, which is huge. There are almost 3,000 plugins right now, and we can't dump the context of all the plugins to an agent. It goes crazy. So, you need to limit the agent um in some [17:45] way, and this is how we do it using the agent type parameter. forget about it. We provide a interactive environment to [17:58] the developers, where they can talk to the agent and actually refine and validate the output themselves. Firstly, there is a small diff fixer that we have at Uber, and what it essentially allows us to do is, let's [18:13] say someone wants to add um more unit tests, they want to refactor, they want to change a variable name. So, all of these small and incremental fixes can be developer doesn't need to do anything. It's just a click of a button and a [18:27] single line prompt. Next up, we have the Ask AI chat. So, the developer is free to talk to Debug Assist itself. Ask about the RCA, the issue, and even provide corrections on top of it. [18:41] Now, as soon as you provide corrections, you also build a feedback loop to the agent, which helps improve its own prompts and skills. And lastly, we have open in your machine. Even after all of this, some [18:55] people do like to open the issue in their own machines and debug it by hand. Um Now, that's the same feature that I'm talking about here. So, you can open the issue in your own machine or in a remote [19:09] um um, remote machine that we have uh from a pool of machines and the environment and everything regarding that issue is already set up for you. So, there is no installation required. It's a single click operation [19:23] required. It's a single click operation again. Uh, currently Debug Assist runs on almost 9,000 issues every month. It [19:35] almost 9,000 issues every month. It provides 9,000 RCAs, deep RCAs every month. The RCA accuracy right now is 50% as compared to exact developer fixes and out of the remaining 50% [19:49] almost 40% of those issues are still in the right direction. So, Debug Assist maybe is going in the right module but not picking the right file or going to the right file or not picking the right function. But, still it's in the same [20:01] direction which still helps the developers fix the issue much, much faster. We have also noticed that the Jira resolution time has increased. It has almost, um, [20:13] increased by twofold, which is a huge improvement in in itself to clear the improvement in in itself to clear the backlog of out of SLA issues. And lastly, the PR merge rate. Um, back when we started this 1 year ago, the PRs [20:27] that we were creating, um, were, let's just say, not getting merged as frequently as you would like them to. So, the merge rate was only 5% and now that number has increased to almost 48%. [20:41] here. The AI space is evolving continuously and we aim to push these numbers further. So, we want to improve the coverage, we want to improve, add more and more use cases to it. We want [20:53] to improve the accuracy and push it to ideally 100% and as a result, we hope to increase the Jira resolution further as well as the mean time to resolve. Thank you, everyone. And, uh, if you have any questions, please [21:08] uh, if you have any questions, please the floor is open. [21:24] maybe please come to the front? There's a mic here. [21:56] Yeah. Well, a wonderful presentation by the way. I want to incorporate a lot of this into my my own workflow. So, fantastic. Thank you. Um I had a few questions. One was for the testing. [22:11] Um You were mentioning the the Basel test that you do, stage three. Is that done via a like a a model? [22:23] And does it deploy, for example, multiple versions of Android and multiple versions of Android and iOS as well to actually test like the new version of the app? Um that was that was one question. [22:38] Um I had another one, but anyway, yeah. >> Okay, thank you for that wonderful question. I'll just state the question for everyone again. So, we have a question that that that's [22:52] basically asking how do we exactly test it? Every time is a new version of the app built deployed for Android and iOS where we want to test. Um the answer is yes as well as no. [23:07] So, in cases where you are able to repro an issue using a unit test, use that. Use the shortest and the simplest path possible, similar to how a developer would do. If you can repro the issue via a [23:21] integration test or maybe a component test, And finally, if you are not able to repro an issue via code at all, you use the automation test. So, [23:35] there was a particular issue that was occurring near an airport where the network strength was quite low and the issue was only occurring when the network strength was low. So, in this case, you can't really mock [23:51] a network via code. It's really a question of the environment that the app is running in. So, we basically spawned a simulator and an emulator and that is a separate infrastructure at Uber. We have a [24:05] separate team, similar to if you know about BrowserStack or Kobiton, similar What they do is they create that environment. We already have all the logs. So, we know what was the network strength. We know what was the user [24:20] battery. We know what the parameter flags or the feature flags the user was behind. We simply mock that information in that simulator or emulator and then we try to build the APK or the IPA file and upload it. And that goes in a [24:36] iterative feedback loop and we keep doing that till it passes. However, we again like have guardrails for it. There are fixed number of retries and then finally the fix is pushed to the developer. [24:49] fix is pushed to the developer. I hope that answers your question. [25:09] RCA tracking. So, you mentioned driving sub-agents to check for cards. And I suppose there are like 10 or 20 sub-agents running to get to the truth of the RCA. So, how do you control that? Do you [25:25] So, you mentioned about the um some agent tracing the session to identify the trace of user path and other agent trying to check the which version this error is having a statistical analysis. [25:39] how many strategies have you defined? And how many agents are running in parallel to go to go to the to track down the RCA? >> Awesome. Uh thank you for that lovely [25:53] question. I'll just state the question for everyone. The question was around We have these sub-agents running in the RCA node. So, how do we keep a track of those agents? How we How do we see how many agents are running? And how do we [26:06] Um Now, um we do have built-in tracing um in this agent. I didn't talk about the observability of the agent itself, even though I work in observability space. I [26:19] didn't have enough time. But, we do have built-in tracing. Um if you are aware, it's called Arise Tracing, and that comes in really, really handy. So, we have plugged it in with LangChain, as well as each of the [26:31] LLM calls that we make. Number one, that helps us track not just sub-agents, but every tool call that we make. So, all the MCPs that we call, every bash command that the agent runs, everything an agent runs, we have visibility into [26:45] that. Number one. Um number two, how to control what each each itself does. Again, I would say the biggest contributor there is the number of max turns. So, if the agent is able to perform a [27:00] certain action in maybe let's say 20 or 30 turns, it's good. If it's taking more wrong. specialized sub-agents. Um I'll just give you one more example. [27:12] There is a sub-agent that tries to figure out who which was the offending commit that led to this crash. Now, that agent, while it can access all the Git history, uh do we want to do that? Probably not. So, we supply that [27:29] agent sub-agent with some additional information. For example, this was the this is the bad version. Something must have happened in between this. So, you look between this. Then, I want to look in maybe rider module, maybe driver [27:42] module, maybe Uber Eats module. So, let's just filter the history by that. And you keep pruning and pruning and pruning. So, you also control the input Does that answer your question? >> Oh, yes. Thank you. And [27:55] that's one more. So, how many strategies do you have you defined for each agent for one strategy? So, how many strategies have you defined? Or is it defined dynamically by user agent or is it deterministic? [28:10] >> Right. Um so, how many strategies have we defined for sub-agents? Essentially, it boils down to how many sub-agents or unique sub-agents do we and do we have at Uber, right? Right. Um right now, the number of [28:23] sub-agents is pretty limited. I would say we have like um all in all, 30 sub-agents, 30 unique sub-agents. But, we are building out something called um domain extensions, so that all the different domain owners can plug in [28:39] their own sub-agents here. And they have to manage their own knowledge base and Because we are not the owners of domain, we are the providers of the harness. So, by our projections, is bound to shoot up to roughly 10,000 by the end of [28:55] >> Great. Thank you. >> Thank you. had a good time. >> [applause]