AWS Outages Explained — Full Breakdown & Transcript

[00:03] hospitals across the United States try to pull up patient records. Nothing loads. Airline gate agents try to issue boarding pass. The system is frozen. Coinbase goes down. Slack goes down. Snapchat goes down. Dualingo goes down.

[00:19] Fortnite goes down. People try to ask Alexa what's happening. Alexa doesn't answer. For the next 14 hours, a significant portion of the modern >> Amazon, >> Amazon, Amazon.com suffered a major

[00:35] outage today. The US, the major cloud computing services went offline. >> Roughly a third of all websites on the internet use it. It's difficult to believe how many use it. Uh so this outage immediately sent shock waves

[00:48] around the world. >> There's no cyber attack, no natural disaster, no ransomware. The cause is a timing issue between two software processes inside a data center in Ashurn, Virginia. One data center, one

[01:04] region, one company's infrastructure, and the world holds its breath. Here's the thing, though. This wasn't new. This was the sixth time. Before we get into it, what even is US- East1? AWS divides its infrastructure into regions,

[01:20] geographical areas where they've built data centers. US East one is their Northern Virginia region. And it's not just another region. It was the first region. Every foundational AWS service like S3, EC2, Dynamo DB, and Lambda,

[01:37] they were built and tested there first, which means it became the default. developers building in the early 2010s picked US- East1 because it was where everything was. And once enough people were on it, it became the region you had

[01:54] to use. Today, estimates put somewhere between 30 and 50% of all internet between 30 and 50% of all internet traffic running through that one region, one data center cluster, one state, Virginia. Keep that in mind.

[02:15] old. The cloud is still a new idea. Most companies still run their own servers, but an early generation of startups like Reddit, Forsquare, Quora, and Hootuite have gone allin on this new thing called the cloud. That morning, the AWS team

[02:31] begins a routine network upgrade in US East1. The goal is to move traffic to higher capacity network connections, a standard procedure. The engineer standard procedure. The engineer executing the change makes a mistake.

[02:45] Instead of routing traffic to the primary network and shifting the old connections to backup, they do it in reverse. The backup network gets the primary traffic, but the backup wasn't built for that load. It starts dropping

[03:00] packets. And here's where things get interesting. EBS elastic block store the virtual hard drives that EC2 instances run on begins trying to remirror itself.

[03:13] Thousands of volumes all detecting that their data might be at risk. All simultaneously trying to create backup copies of themselves. Normally this is fine. Normally the reiring takes spare network capacity. There was no spare

[03:28] capacity. The network was already overwhelmed. So the re mirroring clogs the network further which causes more EBS volumes to think they're losing data which causes more reiring which clogs the network more a feedback loop a

[03:45] spiral. Reddit goes down forsquare goes down. Quora goes down. >> AWS the major cloud computing services went offline early this morning which brought down major websites such as Reddit forsquare and giant bomb. AWS

[04:00] engineers work around the clock. Full recovery takes four days. The cloud was three years old and it had already proven something important. When it proven something important. When it breaks in ways nobody predicted.

[04:20] thunderstorms tears through Northern Virginia. The kind of storm with 80 mph winds and rapid fire lightning. >> Racho. It left a big mark on our region. The damage impacting thousands of people in our area. It

[04:34] >> hits the Ashberry data centers. Emergency generators kick in. Then the transfer switches start malfunctioning. There are the hardware that manages the transition from grid power to generator power. Some of them fail on the way up.

[04:49] Power fluctuate. Servers go down. Netflix, Pinterest, Instagram, Heroku, all affected. But here's the interesting part. Netflix stayed up. Their competitors were scrambling. Netflix engineers spent two minutes watching

[05:04] their dashboards, saw that AWS was degraded, and their systems automatically shifted traffic away from the affected zone. They had spent the previous two years building something called chaos monkey software that

[05:19] randomly terminated their own instances in production on purpose to force their systems to become resilient to exactly this kind of failure. They had literally

[05:31] rehearsed for this disaster. So, while Instagram went dark and Pinterest went down, Netflix kept streaming and engineers across the internet learned a lesson about what it means to build for failure. The storm lasted a few hours.

[05:47] failure. The storm lasted a few hours. The outage lasted about the same. says it's experiencing issues with its cloud-based computing service, which is

[06:03] used by nearly a million customers. >> No storm this time. No network upgrade >> No storm this time. No network upgrade gone wrong. This time, it's a typo. An S3 engineer is debugging a billing issue. S3 or simple service is the

[06:17] backbone of the internet. Websites, apps, databases, enormous amounts of the web either live on S3 or depend on it. The engineer follows an established playbook. They run a command to remove a small number of servers from one of S3's

[06:34] internal subsystems, but they type in the wrong number. Instead of removing a small number of servers, they remove a large one. Specifically, they removed the index subsystem, the service that tracks the location of every single

[06:50] object in S3. Every photo, every file, every database backup. Without the index, S3 can't find anything. Requests start failing. Trello goes down. Slack goes down. Thousands of apps relying on

[07:04] goes down. Thousands of apps relying on S3 go down. The AWS team tries to restart the index subsystem. And here's the problem. S3 has grown enormously the problem. S3 has grown enormously since it launched in 2006. The index

[07:17] subsystem hasn't needed a full restart in years. Nobody knows how long it's in years. Nobody knows how long it's going to take. It takes hours. The crawl bar moves slowly across the screen as engineers wait. At some point, someone

[07:31] tries to check the AWS status dashboard to see if there's an update. The status dashboard is hosted on S3. It won't load. Amazon is using Amazon to check if

[07:43] Amazon is down. The dashboard shows all green, everything is fine, services are operational because the dashboard itself can't update its own status. 4 hours

[07:55] later, S3 recovers. The billing issue gets fixed. The engineer presumably has gets fixed. The engineer presumably has a very long walk home. Amazon quietly adds a note to their postmortem that they're going to move the status

[08:09] they're going to move the status dashboard off of S3. Thanksgiving, AWS makes a change to Kinesis data streams. Kinesis is a

[08:26] service for processing real-time data. It's not as well known as S3 or EC2, but it turns out it's quietly woven into the foundation of how AWS works internally. The change causes Kinesis to start consuming far more resources than

[08:41] expected on the servers running the front end of the service. Kinesis starts to struggle and then things get strained. IM starts having problems. IM is AWS's authentication service. It's the thing that verifies who you are and

[08:57] whether you're allowed to do something. Cognto starts having problems. That's the service that lets applications handle user signin. Cloudatch starts

[09:09] having problems. That's the monitoring service, the one engineers use to figure out what's wrong. Route 53 health checks start failing. Autocaling stops working.

[09:21] All of them had dependencies on Kinesis that nobody had fully mapped out. When Kinesis slowed down, it dragged everything connected to it. Engineers try to log into the AWS console to diagnose the situation. They can't log

[09:37] diagnose the situation. They can't log in. AM is down. The tool you need to fix the problem is broken by the problem. It takes hours to untangle the cascade. services recover slowly, one by one through the night. On Thanksgiving,

[09:53] people notice things are working again. But at the time, almost no one has any But at the time, almost no one has any idea what happened.

[10:06] >> If you're having some trouble this morning using Amazon's web services, you >> Amazon Web Services suffered a major outage today, disrupting access to many popular websites. It took down services including Prime Music and video Alexa

[10:20] and its Ring Smartome systems. >> Something is wrong with AWS Lambda and >> Something is wrong with AWS Lambda and several other services in US East1. Apps start throwing errors. >> People trying to use Instacart, Venmo,

[10:33] Kindle, Roku, and Disney Plus have reported issues. >> Nothing catastrophic at first. Then it gets worse. But this outage is different from the others because of who goes down. Alexa stops responding. Tens of

[10:48] millions of people say Alexa and get silence. Ring cameras go offline. Doorbell cameras, security cameras, all offline. Iroot Roomba vacuums scheduled

[11:00] to clean while their owners are at work stop midfloor and refuse to move. And stop midfloor and refuse to move. And then it gets worse. Amazon itself starts to break. Amazon Flex drivers, who are the gig workers who deliver Amazon

[11:13] packages in their own cars, open their apps to start their shifts. The apps won't load. They can't scan packages. They can't start routes. Inside Amazon fulfillment centers, the handheld scanners that workers use to track

[11:29] inventory stop connecting. The company that owns the cloud is being taken down by its own cloud. For a few hours, Amazon's ability to fulfill orders, which you know is its entire reason for existing, is being impacted by AWS going

[11:45] down. The same infrastructure Amazon built to sell to the world is the same infrastructure Amazon depends on to function. The outage eventually resolves, but Amazon doesn't go into great detail about the root cause. The

[12:02] great detail about the root cause. The silence is its own kind of answer.

[12:14] an automated system is doing what it always does, managing DNS records for >> This outage so massive, in fact, it's really hard to pinpoint an industry that wasn't affected by this. >> Dynamo DB is Amazon's flagship database

[12:29] service. But it's not just a database people use. Internally, dozens of AWS services use Dynamob as their coordination layer, storing state, tracking configurations, managing metadata. It's the connective tissue of

[12:45] the platform. The automation manages hundreds of thousands of DNS records, constantly updating them to point to the endpoint. Somewhere in that automation, two redundant components encounter a rare timing condition, erased. They

[13:02] collide in a way that's never happened before. The automation deletes the DNS record for the Dynamo DB regional endpoint. Not the data, not the servers,

[13:14] just the address that every AWS service uses to find Dynamo DB. In seconds, uses to find Dynamo DB. In seconds, Dynamob becomes unreachable. Not broken, just unfindable. And then the cascade begins. EC2 which are the virtual

[13:29] machines that power the entire cloud stores its operational data in Dynamob. With Dynamo DB unreachable EC2's orchestration system stalled new instances can't launch. Existing

[13:43] instances can't be managed. Other services that depend on those services start failing too. Netflix goes down. Slack goes down. Coinbase goes down. Expedia goes down. Hospitals can't access records. Airlines can't issue

[13:58] boarding passes. Banks go dark. Engineering teams around the world open their AWS management consoles to diagnose the problem. The consoles are either unreachable or showing stale data. The fixed tools are broken by the

[14:14] things being fixed. Again, AWS engineers work to recreate the missing DNS record, but the systems that should automatically recover are themselves dependent on Dynamo DB. They can't recover. Restoring everything carefully

[14:32] in the right order, validating each step. This takes 14 hours. step. This takes 14 hours. 14 hours. Six outages. 15 years. One region. A typo, a storm, a mistyped command, a holiday eve configuration

[14:49] command, a holiday eve configuration change, a dependency nobody mapped, a race condition in automation, six completely different causes, six completely different failure modes and yet same region, same impact, same

[15:05] yet same region, same impact, same story. The uncomfortable truth about US East one is this. We have known for 15 years that it is too important to fail. years that it is too important to fail. And we have watched it fail anyway

[15:19] repeatedly for reasons ranging from bad luck to human error to hidden complexity that nobody und understood until it broke. Every time Amazon publishes a post-mortem. Every time they promise new safeguards. Every time the next outage

[15:36] finds a different crack to follow through. Because systems this complex don't fail in ways you predict. They fail in the ways you didn't think to protect against. And because 30 to 50% of the internet chose the same region,

[15:52] the same data center cluster, the same slice of land in Northern Virginia. Every time a single thing goes wrong there, the whole world finds out about

[16:04] there, the whole world finds out about it.