---
title: 'Building a Debugging Agent Harness at Uber - Kriti Dangi'
source: 'https://www.youtube.com/watch?v=iVCDIOf7vXw'
video_id: 'iVCDIOf7vXw'
date: 2026-11-05
duration_sec: 1752
channel: 'Agentic AI Foundation'
---

# Building a Debugging Agent Harness at Uber - Kriti Dangi

> Source: [Building a Debugging Agent Harness at Uber - Kriti Dangi](https://www.youtube.com/watch?v=iVCDIOf7vXw)

## Summary

Kriti Dangi, a senior software engineer at Uber, presents Debug Assist, an AI agent for debugging production issues at scale. The talk covers why debugging is hard, how Debug Assist works end-to-end, its internal architecture built on LangGraph, and key production metrics.

### Key Points

- **Introduction and Background** [00:03] — Kriti Dangi, senior software engineer at Uber, works in app quality and production observability. She introduces Debug Assist, an AI agent for debugging.
- **Uber's Bug Reporting Systems** [00:49] — Uber uses two systems: Wisdom (in-app bug reporter capturing natural language descriptions, metadata, logs, and screenshots) and Healthline (automatic app quality analytics capturing crashes, hangs, and performance issues with stack traces).
- **Scale of the Problem** [03:06] — Almost 50,000 issues are assigned to engineers yearly, 90% customer-impacting, leading to 12,000 pagers. Debugging is hard due to unclear errors, scattered context, vast codebases, and cross-team dependencies.
- **Debugging Time and Cost** [04:52] — Engineers take ~28 days to resolve an issue on average. Stripe reports developers spend 40% of their time debugging.
- **Debug Assist Overview** [05:07] — Debug Assist is an end-to-end agentic resolution journey: discovery (via Wisdom/Healthline), auto-triage (priority, severity, ownership), deep RCA in under 30 minutes, and actionable resolution.
- **Real-Life Example: Battery Issue** [06:07] — A developer reported 'Uber app is responsible for nearly 30% of my battery usage.' Debug Assist root-caused it to a hot looping code path, provided evidence timeline, and created a fix within 20 minutes without the developer clicking a button.
- **Real-Life Example: Crash on Rollout** [08:24] — A high-severity crash (P1) due to a bad rollout was caught at 5% rollout. Debug Assist pinpointed the code, created a fix, and it was merged and deployed the same day.
- **Architecture: LangGraph Pipeline** [09:09] — Debug Assist is built on LangGraph, a mix of deterministic and LLM nodes. It bypasses the planning phase by providing a fixed plan, reducing hallucination. Nodes: context collector (deterministic), RCA node (LLM, Sonnet), fixing node (LLM, Opus), validation, and PR creation.
- **Context Collector Node** [10:05] — Deterministic node that fetches data from Wisdom, Healthline, logs, etc. Avoids LLM for simple data fetching to reduce latency, cost, and context bloat. Pre-processes and prunes logs (which can be gigabytes).
- **RCA Node** [10:46] — LLM node that categorizes the crash (Uber issue, third-party, infrastructure, network), computes confidence score, queries Uber ecosystem via MCP (Jaeger tracing, logging, incident data), and fans out sub-agents for parallel RCA.
- **Fixing Node** [12:24] — Runs on Opus with higher max turns. Goes through Uber's billions of lines of code, identifies affected modules, mitigates via feature flags (tens of thousands active), and creates fixes.
- **Validation Stage** [14:21] — Agent tries to reproduce the issue via unit tests, integration tests, or mobile automated end-to-end tests (simulators/emulators). Validates its own fix to increase confidence.
- **Skills and Plugins** [15:14] — The skeleton is fixed; skills and plugins are pluggable. Each LLM node has its own skill (e.g., how to create a good PR, test planning). Uber has a skills marketplace with ~3,000 plugins, fetched at runtime based on agent type.
- **Productionization at Scale** [16:11] — Eight agents run at scale across five mono repos. Code is in a Python mono repo, packaged as a pex binary, uploaded to artifactory. Agents fetch correct binary, Docker runtime, and pipelines at runtime.
- **Developer Interaction** [17:58] — Interactive environment: small diff fixer (one-click fixes), Ask AI chat (feedback loop), and 'open in your machine' (single-click setup with environment).
- **Metrics and Results** [19:23] — Debug Assist runs on ~9,000 issues/month, provides 9,000 deep RCAs. RCA accuracy is 50% exact, 40% in the right direction. Jira resolution time increased twofold. PR merge rate increased from 5% to 48%.
- **Future Goals** [20:41] — Aim to improve coverage, accuracy (ideally 100%), and further increase Jira resolution and mean time to resolve.

### Conclusion

Debug Assist demonstrates a production-grade AI agent that significantly improves debugging efficiency at Uber, reducing resolution time and increasing PR merge rates. The key is a fixed, deterministic harness with pluggable skills, enabling scalability and developer trust.

## Transcript

Good morning, everyone. I'm Kriti Dangay from India. Watashi wa Kriti desu. Uh a little bit about me. I'm a senior software engineer at Uber. I've worked at Uber for almost 4 years now, and I work in app quality teams as well as
production observability space. And I'll be sharing some of that today with you. So, this talk is about Debug Assist, an AI agent for debugging that we built at Uber. So, today we'll talk about why debugging
is actually hard in reality. How does Debug Assist help with it? We'll also have a demo, and then we'll see a quick look at the internal architecture. And lastly, I'll share a couple of interesting numbers with you.
All right. So, what you have in front of you is Uber's in-app bug reporter. So, all the Uber internal employees, as well as beta testers, have access to report bugs on the Uber app itself via the beta version.
And this is what gets captured in the system. So, let's say I'm using my app, the screen goes blank, or maybe I'm experiencing hangs, or maybe there are battery issues. I can simply just click and report a bug. And for me, it looks
like saying, "Oh, my screen is blank." That's it. The system captures all the information alongside it. Wisdom internally.
So, along with the natural language bug description, it captures the metadata around the issue. For example, what was the app that was affected? What was the app version that was affected? What was the OS that was affected? Which city is
the user experiencing the issue in? Along with this, we also capture a multitude of logs, like network logs, analytics logs, console logs, GraphQL analytics logs, console logs, GraphQL logs, UI state logs, and so many more.
And as we can see, screenshots are also captured from the device. Um, there are no security concerns here. We only capture the Uber screenshots to debug capture the Uber screenshots to debug the issue later on.
It's called Healthline. So, Healthline is essentially a app quality analytics tool. Um, similar to Wisdom, it also captures issue. However, it does that automatically. So, every time your app
crashes, you experience a hang, a jitter, or any kind of performance degradation, Healthline is a system that kicks in the backend. So, similar to Wisdom, it also captures the metadata. For example, what was the
session ID? What was the analytics ID? What was the user doing before this? listed and many more. However, unlike Wisdom, it captures the
stack trace all the whole whole crash dump as well. So, we can say which code dump as well. So, we can say which code was actually causing this problem now. almost 50,000 issues are assigned to engineers every year. 90% of these are
customer impacting and as a result, we end up sending 12,000 pagers to our engineering teams everywhere. So, someone has to debug these, right? Someone has to fix these.
Before I move on to the next slide, a quick show of hands, how many of us do quick show of hands, how many of us do not like fixing production issues?
pager at 2:00 a.m. at night and you have to sit all night and debug it and you don't even know where to start because your brain is so foggy.
The errors are often unclear. Um, we all have received that message, to figure out what went wrong. So, we logic. The context is also scattered. Um you
have to go through code, you have to go through logs, you have to go through configs, and you have to go through data. And to access that data, there are so many tools internally. Speaking of code bases,
Speaking of code bases, code bases at Uber at Uber are vast. So, we have multiple mono repos. We operate in a mono repo architecture style, which means that many, many teams are contributing to a single repo.
are contributing to a single repo. So, to figure out who caused the problem is another problem in itself. And even when you do, there are still cross-team dependencies. The code might be owned by someone else. Even if it is
owned in your team, you don't know all the code written by your team, right? engineer at Uber takes almost 28 days to resolve an issue on an average.
And by uh and as seen in a public report done by Stripe, developers spend 40% of time debugging an issue. And this is why we developed Debug Assist, which aims to improve the app quality and enhance the debugging
experience of developers so that we can do what we do best, develop. generally follow. It is a end-to-end agentic resolution journey. It starts by
discovery. Uh discovery happens through Healthline and Wisdom, as we just saw. Next, we move to the auto triad stage. This is the stage which figures out what is the priority of this issue, how severe it is, and who actually owns it.
here? Then starts the Debug Assist actual pipeline. It does a deep root cause analysis in under 30 minutes. We'll We'll see how it does that. We'll take a deep dive here.
And lastly, we aim towards a resolution. So, we don't just want to provide a big dump of data to the developers. We want to make it actionable. We want to to make it actionable. We want to resolve the issue end to end.
part. I hope all of you are attentive. So, this is the Wisdom screen that we just saw earlier. And this is a real-life example, by the way. One day one of our Uber developers woke up and he booked a cab to office.
While waiting for the Uber, he realized that his phone was burning up in his pocket. So, he filed a Wisdom bug report. All he said was "Uber app is responsible for nearly 30%
of my battery usage. My phone is heating up. That's it." Then he happily sat in the Uber, came to office, opened his laptop to look at the issue. In those 20 minutes,
created a fix without him clicking a single button. All the developer had to do was review it and deploy it. So, how does this magic happen in the background?
As you can see here, we Wisdom has captured the natural language description. It has captured the screenshot of what the developer was seeing while booking his cab right as well as a screenshot uploaded by
developer which says that "Oh, my battery usage is being consumed by battery usage is being consumed by Uber." And we can clearly see it's 29%. There are metadata and logs associated it with it as well.
Debug Assist. Um on the left side, we can see that Debug Assist has root caused the issue to a hot looping code path, which means that there was certain code path that was supposed to run in the background,
but it was actually being treated as a foreground loop path, which led to high CPU cycle usage and high battery regression. Now, along with this RCA, it also tells us that, "Oh, by the way, the app was
backgrounded for 17 minutes. It was not in the foreground." And all of this rooted is rooted in evidence. So, on the right side, we can see there is a evidence timeline. So, we know that the agent is not hallucinating and we
are somewhat confident in our fixes. And lastly, we can see the developer was pretty happy.
crash that was occurring on the Uber app due to a bad rollout. Essentially, notification very, very quickly, the app was crashing. This was a high severity crash, that is why it is marked as a P1, which is priority one.
And DebugAssist created a whole deep deep RCA for it. It pinpointed the code which was causing this and it also created a fix for it. The fix was sent to developer via a Slack message later on. And in this case, the fix was merged
and deployed the same day. So, we ended up catching the crash at a 5% rollout up catching the crash at a 5% rollout and fixing it at the very same day.
DebugAssist actually does. How does it actually accomplish this? So, as you can see, the agent is built on a LangGraph pipeline. It is a mix of deterministic and LLM nodes. Why LangChain, you might ask?
So, agents generally have a planning phase and an execution phase. What LangGraph or LangChain allows us to do is bypass that planning phase. We give the plan to the agent that these are the steps you have to follow every
are the steps you have to follow every time. So, it removes certain levels of hallucination from the agent. We're not totally dependent on the LLM to come up with a good plan to fix the issue. We control the plan.
So, it starts with a context collector node. This node essentially captures all the data that we just talked about from health line, wisdom, all the logs, etc. not an LLM node.
Why not LLM in this world of AI? Because everything doesn't have to be via everything doesn't have to be via agents. It's simply fetching data. Um a lot of context bloating. We don't want to add latencies to the agent of the
APIs that we already know of. And of course, LLM cost is a huge problem these days and we don't want to bloat that as well. Um also, this allows us to parse and pre-process and prune those logs. So,
these logs that I'm talking about are huge. They are megabytes and megabytes and gigabytes of logs. You cannot simply dump them to an agent. You have to pre-process them in some ways. Next, we move to the classification or
the RCA node. This is a LLM node. As you can see, it's highlighted in blue. Now, this node looks at all the data we just fed it. It first tries to categorize the crash. Is this even happening because of an
Uber issue? Or is this a third-party library code? Is this an infrastructure issue? Is this a network issue? Do I need to create a PR to fix this?
Once the categorization is done, it comes up with a confidence score as we can see. And finally, if it feels I need to get more data, it goes and queries Uber ecosystem. So, it does that via MCP connections like Jaeger logging,
um Jaeger distributed tracing, logging, uh Uh, incident data. Is there any ongoing incident that's generally impacting the users, etc. Once that is done, it performs the RCA via sub agents. So, we fan out multiple
sub agents in parallel and let them do RCA and once we have that, we consolidate the results. To give you example of some of the sub agents, um, one sub agent is breadcrumb analysis. So, we give all the session
data to the agent and we ask it to recreate the timeline of the user. This not only helps with the good RCA, but it also helps us reproduce the issue. Another agent, um, I can share with you is crash correlation agent. So, whenever
a crash is occurring, can I correlate it to a particular version release? Can I uh, confidently say that this version is experiencing the crash?
the next step is a fixed node. Now, before I move to the fixing part, um, you might see there are certain configs given in every node and RCA node actually runs on a lower model, Sonnet, with limited number of turns. You might
it's really not. If you think about it, it's mostly looking at the data and gathering information and summarizing it. it runs on Sonnet with limited number of turns because, uh, with trial and error
we figured out that if it's taking more than 20 turns, it's generally going in hole. And that's how you also put guardrails around your agent. the fixing node. Now, this is the node which, which needs
the big brains. So, it goes through the code base of Uber and Uber code base is billions and billions of lines of code. Now, it tries to figure out which module might be affected, which code path would be affected, and all the states are
state of the previous node as as well. It has not forgotten anything. Before fixing, we first try to mitigate the issue, too. So, as you can see, most of the Uber issues are behind a feature flag, and
this node connects to a feature flag MCP. It tries to correlate, is this feature flag causing this issue? And um mind you, in all of your phones that you're using Uber app, there are at least tens of thousands of feature
flag that are active in the background. So, just correlating that one crash to one of those feature flags is already a very complicated task. simply goes and rolls the feature flag back, and the issue is mitigated. And
then you're um free to fix it on your own time. So, we have the fixing and the mitigation step here, which runs on Opus with a higher number of max turns. Next, we go to the validation stage.
legacy reasons, um but it's actually a validation stage. So, in this case, the agent tries to reproduce the issue either via a unit test or a mobile automated end-to-end test.
Some flows at Uber have a automation test that runs on simulators or emulators directly, especially for the core flows like booking a ride. validate its own fix.
So, you not only just give the PR to a developer and say test it, you also validate it and increase the confidence of your own fix. And finally, when you are happy with the fix, we create a diff or a PR in this
case. The PR is attached to the Jira, to the issues on healthline, wisdom. Um it is uh sent to the developer via Slack, and all the information is updated internally as metadata.
here is this is the basic skeleton of the agent. It never changes. What changes is the skills and plugins. So, each of these LLM nodes have their own skill. How to create a good PR at
Uber? What are some good practices we follow for test planning? How to fix an issue in Android? Do I want to do lazy loading always? Do I want to defer something to background? etc. Now, these skills and plugins are, as
the name suggests, pluggable. So, the skeleton remains the same and all of this core business logic is outside the skeleton, which is fetched Which means tomorrow if I want to add a new agent, it's a matter of crowd
new agent, it's a matter of crowd sourcing the skills. did we manage to productionize an agent at Uber scale? Now, running something on your local machine is one story, but running it at Uber scale is another
story altogether. So, as I said, we have provided the harness to the developers. We have provided the steps to follow. We have provided the runtime environments, the dependencies, etc. All the teams have to
do is bring in their own skills, own specialized skills on top of what we already provide. Which also reduces the load on us because now we don't have to maintain a multitude of skills and plugins and
whatnot for everyone. You bring your own specialization and we provide the harness. So, right now we have almost eight agents running at scale across five mono repos and all of them share a single
All of the code is inside a Python mono repo and it is packaged as a Python binary called pex. The pex is uploaded to a common artifactory and every time an agent wants to run, uh it it knows its type. So, according to the agent
type, it fetches the correct binary. It fetches the correct Docker runtime and Docker images, which provides all the dependencies and runtime environments. And finally, it fetches all of its pipelines, which is uh scattered across
different mono repos. Again, um I would like to highlight that this is a Harness, and even the skills and plugins for every agent is fetched and plugins for every agent is fetched at runtime. So, we don't do a dump of
all the marketplace plugins. Uber has a skills marketplace, which is huge. There are almost 3,000 plugins right now, and we can't dump the context of all the plugins to an agent. It goes crazy. So, you need to limit the agent um in some
way, and this is how we do it using the agent type parameter. forget about it. We provide a interactive environment to
the developers, where they can talk to the agent and actually refine and validate the output themselves. Firstly, there is a small diff fixer that we have at Uber, and what it essentially allows us to do is, let's
say someone wants to add um more unit tests, they want to refactor, they want to change a variable name. So, all of these small and incremental fixes can be developer doesn't need to do anything. It's just a click of a button and a
single line prompt. Next up, we have the Ask AI chat. So, the developer is free to talk to Debug Assist itself. Ask about the RCA, the issue, and even provide corrections on top of it.
Now, as soon as you provide corrections, you also build a feedback loop to the agent, which helps improve its own prompts and skills. And lastly, we have open in your machine. Even after all of this, some
people do like to open the issue in their own machines and debug it by hand. Um Now, that's the same feature that I'm talking about here. So, you can open the issue in your own machine or in a remote
um um, remote machine that we have uh from a pool of machines and the environment and everything regarding that issue is already set up for you. So, there is no installation required. It's a single click operation
required. It's a single click operation again. Uh, currently Debug Assist runs on almost 9,000 issues every month. It
almost 9,000 issues every month. It provides 9,000 RCAs, deep RCAs every month. The RCA accuracy right now is 50% as compared to exact developer fixes and out of the remaining 50%
almost 40% of those issues are still in the right direction. So, Debug Assist maybe is going in the right module but not picking the right file or going to the right file or not picking the right function. But, still it's in the same
direction which still helps the developers fix the issue much, much faster. We have also noticed that the Jira resolution time has increased. It has almost, um,
increased by twofold, which is a huge improvement in in itself to clear the improvement in in itself to clear the backlog of out of SLA issues. And lastly, the PR merge rate. Um, back when we started this 1 year ago, the PRs
that we were creating, um, were, let's just say, not getting merged as frequently as you would like them to. So, the merge rate was only 5% and now that number has increased to almost 48%.
here. The AI space is evolving continuously and we aim to push these numbers further. So, we want to improve the coverage, we want to improve, add more and more use cases to it. We want
to improve the accuracy and push it to ideally 100% and as a result, we hope to increase the Jira resolution further as well as the mean time to resolve. Thank you, everyone. And, uh, if you have any questions, please
uh, if you have any questions, please the floor is open.
maybe please come to the front? There's a mic here.
Yeah. Well, a wonderful presentation by the way. I want to incorporate a lot of this into my my own workflow. So, fantastic. Thank you. Um I had a few questions. One was for the testing.
Um You were mentioning the the Basel test that you do, stage three. Is that done via a like a a model?
And does it deploy, for example, multiple versions of Android and multiple versions of Android and iOS as well to actually test like the new version of the app? Um that was that was one question.
Um I had another one, but anyway, yeah. &gt;&gt; Okay, thank you for that wonderful question. I'll just state the question for everyone again. So, we have a question that that that's
basically asking how do we exactly test it? Every time is a new version of the app built deployed for Android and iOS where we want to test. Um the answer is yes as well as no.
So, in cases where you are able to repro an issue using a unit test, use that. Use the shortest and the simplest path possible, similar to how a developer would do. If you can repro the issue via a
integration test or maybe a component test, And finally, if you are not able to repro an issue via code at all, you use the automation test. So,
there was a particular issue that was occurring near an airport where the network strength was quite low and the issue was only occurring when the network strength was low. So, in this case, you can't really mock
a network via code. It's really a question of the environment that the app is running in. So, we basically spawned a simulator and an emulator and that is a separate infrastructure at Uber. We have a
separate team, similar to if you know about BrowserStack or Kobiton, similar What they do is they create that environment. We already have all the logs. So, we know what was the network strength. We know what was the user
battery. We know what the parameter flags or the feature flags the user was behind. We simply mock that information in that simulator or emulator and then we try to build the APK or the IPA file and upload it. And that goes in a
iterative feedback loop and we keep doing that till it passes. However, we again like have guardrails for it. There are fixed number of retries and then finally the fix is pushed to the developer.
fix is pushed to the developer. I hope that answers your question.
RCA tracking. So, you mentioned driving sub-agents to check for cards. And I suppose there are like 10 or 20 sub-agents running to get to the truth of the RCA. So, how do you control that? Do you
So, you mentioned about the um some agent tracing the session to identify the trace of user path and other agent trying to check the which version this error is having a statistical analysis.
how many strategies have you defined? And how many agents are running in parallel to go to go to the to track down the RCA? &gt;&gt; Awesome. Uh thank you for that lovely
question. I'll just state the question for everyone. The question was around We have these sub-agents running in the RCA node. So, how do we keep a track of those agents? How we How do we see how many agents are running? And how do we
Um Now, um we do have built-in tracing um in this agent. I didn't talk about the observability of the agent itself, even though I work in observability space. I
didn't have enough time. But, we do have built-in tracing. Um if you are aware, it's called Arise Tracing, and that comes in really, really handy. So, we have plugged it in with LangChain, as well as each of the
LLM calls that we make. Number one, that helps us track not just sub-agents, but every tool call that we make. So, all the MCPs that we call, every bash command that the agent runs, everything an agent runs, we have visibility into
that. Number one. Um number two, how to control what each each itself does. Again, I would say the biggest contributor there is the number of max turns. So, if the agent is able to perform a
certain action in maybe let's say 20 or 30 turns, it's good. If it's taking more wrong. specialized sub-agents. Um I'll just give you one more example.
There is a sub-agent that tries to figure out who which was the offending commit that led to this crash. Now, that agent, while it can access all the Git history, uh do we want to do that? Probably not. So, we supply that
agent sub-agent with some additional information. For example, this was the this is the bad version. Something must have happened in between this. So, you look between this. Then, I want to look in maybe rider module, maybe driver
module, maybe Uber Eats module. So, let's just filter the history by that. And you keep pruning and pruning and pruning. So, you also control the input Does that answer your question? &gt;&gt; Oh, yes. Thank you. And
that's one more. So, how many strategies do you have you defined for each agent for one strategy? So, how many strategies have you defined? Or is it defined dynamically by user agent or is it deterministic?
&gt;&gt; Right. Um so, how many strategies have we defined for sub-agents? Essentially, it boils down to how many sub-agents or unique sub-agents do we and do we have at Uber, right? Right. Um right now, the number of
sub-agents is pretty limited. I would say we have like um all in all, 30 sub-agents, 30 unique sub-agents. But, we are building out something called um domain extensions, so that all the different domain owners can plug in
their own sub-agents here. And they have to manage their own knowledge base and Because we are not the owners of domain, we are the providers of the harness. So, by our projections, is bound to shoot up to roughly 10,000 by the end of
&gt;&gt; Great. Thank you. &gt;&gt; Thank you. had a good time. &gt;&gt; [applause]
