[00:00] application, transactions are easy. You have a database, and when a customer places an order, you wrap the whole thing in a transaction. Charge their card, reserve the inventory, record a ledger entry for accounting. If any of
[00:12] those rights fails, the database rolls everything back automatically, and it's don't even think about it much. Your database gives you what are called acid guarantees, which basically means two things that matter here. First,
[00:25] atomicity. Either all three of those rights happen together, or none of them do. There's no world where the card gets charged, but the inventory doesn't get reserved. Second is isolation. While that transaction is in progress, no
[00:38] other part of your system can see the half-finished state. Another query see a charge for an order that hasn't fully processed yet. The database handles all of this behind the scenes, and you just write your SQL and move on.
[00:51] start to get more traffic, more data, more rights. And eventually, that single database starts hitting its limits. So, you do what everyone does at this point. You split things up. Maybe you shard the database to spread write load across
[01:05] multiple machines, or maybe you break up your monolith into microservices, where each service now owns its own database. The specifics can vary, but the result is the same. Your data now lives on multiple independent machines instead of
[01:18] one. And at this point, everything changes. The payment flow that used to be one transaction against one database is now three completely separate operations against three separate databases on three separate machines.
[01:31] The card gets charged to the payment database, inventory gets reserved in the inventory database, and that ledger entry gets recorded in the accounting database. You can't wrap a transaction across
[01:43] they don't know about each other. So, if the card charge commits, but then the inventory reservation fails because the item is out of stock, there's no database-level rollback that can undo that charge. It's already committed in a
[01:57] completely different machine. When you're processing thousands of transactions a second across distributed infrastructure, partial failures like become pretty routine. Now, this whole class of problem is what's called a
[02:10] distributed transaction. A single logical operation that needs to span multiple independent databases or services, where all the steps need to either succeed together or be cleaned up when something goes wrong. The textbooks
[02:23] give us two approaches to distributed transactions. Two-phase commit and the has overwhelmingly chosen one over the you a lot of pain. The two-phase commit is a classic academic solution to
[02:36] distributed transactions. The idea is to introduce a new component called a sure that all participants in a transaction agree on the outcome before works in two phases, which is where the name comes from, of course. In the first
[02:51] phase, called the prepare phase, the coordinator sends a message to every participant, asking, "Can you commit this transaction?" the actual work. It processes the request, durably records the changes so
[03:04] that nothing is lost if it crashes, and locks the affected rows so that no other meantime. Then it responds to the coordinator with either yes, I'm ready to commit, or no, something went wrong.
[03:17] If any single participant votes no, the coordinator tells everyone to abort and release their locks. If every participant votes yes, on the other hand, then the coordinator moves to phase two. It sends a commit message to
[03:29] changes permanent and releases those locks, and the transaction is now consistency, the same guarantee you had participant agrees on the outcome before anything is finalized. So, there's no
[03:44] window where the system is in a partial or inconsistent state. On paper, it's exactly what you want. But, the problems show up when you try to run this in production. The fundamental problem with two-phase commit or 2PC is that it's a
[03:56] blocking protocol and blocking in a distributed system is dangerous because machines all staying healthy at the same time. I want you to picture this. The coordinator collects all three yes votes from the participants. But then it
[04:09] crashes. Right there, after collecting the votes but before it gets a chance to send the commit decision. Now the participants are all stuck. held on the rows they prepared and they have no idea what to do next. They can't
[04:22] maybe the coordinator was about to tell them to abort. They can't abort on their own either because maybe the coordinator was about to tell them to commit and the other participants already went through with it. So they just wait. And every
[04:35] needs to touch any of those locked rows is now blocked, too, waiting for the locks that nobody can release. Crashes aren't even the only problem with 2PC. A single slow participant holds up the entire transaction. So if the ledger
[04:49] seconds to respond to the prepared message, the card service and the there with their locks held for those full 10 seconds doing nothing. That means the entire system moves at the speed of the slowest participant. And if
[05:03] coordinator can't reach the participant at all, there's no safe default. It through or not. This is why almost nobody uses two-phase commit across services in productions. Pat Helland wrote a really influential paper called
[05:18] "Life Beyond Distributed Transactions". In that paper he argues exactly this point. Distributed transactions across autonomous services don't work at lesson to heart. 2PC does exist in production, but only
[05:32] inside distributed databases like Google Spanner or Yugabyte DB, where the coordinator and the participants are tightly coupled within the same system. complexity internally so that you as the caller don't have to. But across
[05:45] deployment schedules and different failure characteristics, that's where it instead? Well, when companies need to coordinate work across multiple services, the saga pattern is what they reach for. Uber,
[05:58] Netflix, Amazon, DoorDash, they all use this pattern in production. Sagas start 2PC. You don't actually need all or nothing atomicity spanning multiple services. You just need a way to eventually get to
[06:10] a consistent state, even when things go wrong along the way. Instead of transaction with locks held across services, you break the work into a chain of independent local transactions. So, each service does its piece of work
[06:25] and commits to its own database on its own terms. When something fails further down the chain, there's no way to roll back to earlier steps since they've database. So, instead, you run what is called a
[06:37] compensating action. These are business level undos that reverse the effects of what already happened. So, a refund instead of a abort. Something needs to detect that failure
[06:51] and trigger those compensations. And how that works is a key design decision the trade-off is that instead of getting the strong consistency you get with 2PC, saga gives you what's called eventual consistency. The system might be
[07:05] temporarily in an inconsistent state while compensations are running. their card before the refund goes through, but it always converges to a correct state and nothing is blocked while that convergence is happening.
[07:18] Other transactions can keep flowing normally that entire time. Now, there choice between them determines who's responsible for detecting failures and The first approach is called choreography and it's the decentralized
[07:32] option. It uses a publish subscribe pattern where each service broadcasts an event when it finishes its work and any interested service can pick it up and react. So, the card service charges the
[07:44] card and then publishes a card charged event. The inventory service is arrives, it reserves the stock and publishes an inventory reserved event. The ledger service picks that up and then records the entry. If something
[07:58] fails, the failing service publishes a failure event and the upstream services react by running their own compensations. This works well when you three steps. But, once you get to five or six services all publishing and
[08:11] out the current state of any given transaction becomes really difficult. Where exactly did it fail? Which compensating actions have already run? Without a central place tracking all of this, you end up digging through logs
[08:26] across a dozen different services trying to piece together what happened. The and it's what most teams end up using once they reach any serious scale. Instead of services reacting to each others events, you have a dedicated
[08:39] entire flow. It tells each service what to do one step at a time. Card service, charge the card. It waits for confirmation. Inventory service, reserve the stock. Wait for confirmation. If something
[08:52] fails, the orchestrator knows exactly what steps failed and can run the right compensating action in the right order. Tools like Temporal, which was created by the engineer behind Uber's Cadence workflow engine, or AWS Step Functions,
[09:05] are purpose-built for exactly this kind of orchestration. It's what we use here payment and fulfillment flows and it's the pattern we'd recommend that most teams use. The important difference between Saga orchestration and 2PC
[09:18] coordinators is what happens when it crashes. The orchestrator doesn't leave locks dangling across your system. It's durable. When it restarts, it reads its own state from a database and it picks up exactly where it left off. So, no
[09:31] other transactions are blocked during that recovery period. Sagas solve the blocking problem that makes 2PC impractical, but they introduce a different kind of complexity, the compensating actions themselves.
[09:44] The idea of just undo the previous step sounds clean, but in practice it gets Say the card charge went through and committed, and then the inventory out of stock. The compensation is to issue a refund on
[09:59] rollback, that refund is visible to the customer. They see an actual charge show up on their card, and then a few seconds later they see a refund. Their bank might even send them a push notification for each one. It works correctly, but
[10:12] it's not the invisible cleanup that a database rollback gives you. And some actions are genuinely hard to undo at all. If one of the steps in your flow is to send a confirmation email, you can't unsend that email. If you fired a
[10:25] webhook to a third-party payment system, you can send a follow-up cancellation, but you can't guarantee that they'll process it in time or at all. well-defined compensating action, and some of those compensations are
[10:38] inherently imperfect. On top of that, compensating actions can themselves fail. What if the refund API is down when you need to issue that refund? Now you need retry logic for your compensations. And if you're retrying a
[10:50] item potent, meaning it produces the same result whether you run it once or 10 times, so that a retry doesn't accidentally refund the customer twice. You end up needing the same level of reliability engineering for your failure
[11:03] handling as you do for your happy paths. Even with solid compensation logic in that catches teams off guard. When your card service finishes charging the card, it needs to do two things: save the result to its own database, and publish
[11:16] an event to a message broker so that the next service in the chain knows it's those are two completely separate rights to two completely separate systems. This is called the dual write problem. If the database write succeeds but the
[11:30] your saga never gets triggered and the whole flow stalls. If the event publish is successfully, but the database write fails, now downstream services are actually happen in your database. You can fix this with something called a
[11:45] Instead of writing to your database and publishing an event as two separate operations, you write both your data and the outgoing event into the same database via a single local transaction. The event goes into a special outbox
[11:59] table right alongside your regular data write. So that either both commit or Then, a separate background process watches the outbox table and publishes those events to your message broker. That background process can use change
[12:14] data capture, which means it tails the database's own transaction logs to pick up new entries, or it can simply pull the outbox table on a regular interval. patterns, the first question ask yourself is whether you actually need a
[12:26] distributed transaction at all. If you can design your service transacts together lives in the same database, do that. This is easier to get later. So it's worth thinking about early. For example, move the inventory
[12:42] and ledger table into the same database that has the payments table if they way a local database transaction is simpler, faster, and more reliable than any distributed alternative. And this is always the best answer when you can make
[12:55] distributing the transaction across services, you're going to use a saga. industry anymore. The question is which flavor of saga makes sense for your situation. Choreography is usually where teams start, and for simple flows it
[13:10] steps, the services are truly independent, and you don't need centralized visibility into where each transaction stands, choreography keeps Something like an e-commerce notification system where an order is
[13:24] placed event triggers an email or a notification independently, this is a outgrow it as their flows get more complex, but there's no reason to over-engineer from day one. For anything more complex than that, orchestration is
[13:37] the way to go. Complex flows with branching logic, flows where you need to see exactly where a transaction is stuck, flows where the compensation logic is tricky and you want it defined in one clear place rather than scattered
[13:49] across half a dozen services, most teams end up here and tools like Temporal or AWS Step Functions make it very practical to implement. One last note, if eventual consistency truly is not acceptable for a particular piece of
[14:01] your system, consider whether that data can live in a single distributed database like Spanner or Yugabyte DB that handle that strong consistency different thing from trying to build 2PC yourself across independent services. At
[14:16] you'll see at most companies operating at scale is Saga with orchestration, independent operations at every step so the retries are always safe, and a transactional outbox to make sure events are as reliable as database writes. It
[14:30] but that's a trade-off the industry has made deliberately. And it's the architecture that Uber, Netflix, and Amazon actually run in production today.