How Slack scales MySQL to 1.5B messages/day
45sReveals the surprising scale of Slack's database and the hotspot problem with enterprise customers, creating curiosity about the solution.
▶ Play ClipAI Summary
Slack's MySQL database handles over 1.5 billion messages daily, initially sharded by workspace. As large enterprises caused hotspots, Slack migrated to Vitess, an open-source system that enables horizontal scalability without changing application code, achieving 2.3 million queries per second at peak.
Chapters
[00:03]
Initial Architecture
Slack used MySQL sharded by workspace, with each company's data on its own cluster and routing logic in the application code.
[00:18]
Hotspot Problem
Large enterprises created hotspots because a single company's data hammered one cluster, and routing logic baked into the app prevented further sharding.
[00:31]
Migration to Vitess
Slack migrated to Vitess, an open-source system originally built at YouTube, which makes MySQL horizontally scalable by allowing sharding without app awareness.
[00:56]
Flexible Sharding
Vitess enabled splitting data by users, channels, or workspaces independently, with the app treating Vitess as a single database.
[01:10]
Post-Migration Performance
After 3+ years, 99% of MySQL traffic runs through Vitess, handling 2.3 million queries per second at peak, median latency of 2 ms, and 40 million daily active users.
Slack's successful migration to Vitess demonstrates that relational databases can scale massively with the right sharding strategy, countering the myth that they don't scale.
Mentioned in this Video
Study Flashcards (5)
How many messages does Slack's MySQL database handle daily?
easy
Click to reveal answer
How many messages does Slack's MySQL database handle daily?
easy
Click to reveal answer
Over 1.5 billion messages every day.
00:03
What was Slack's initial sharding strategy?
easy
Click to reveal answer
What was Slack's initial sharding strategy?
easy
Click to reveal answer
Sharded by workspace, with each company's data on its own database cluster.
00:03
Why did the initial sharding strategy become problematic?
medium
Click to reveal answer
Why did the initial sharding strategy become problematic?
medium
Click to reveal answer
Large enterprises created hotspots because a single company's data hammered one cluster, and routing logic baked into the app prevented further sharding.
00:18
What is Vitess and where was it originally built?
medium
Click to reveal answer
What is Vitess and where was it originally built?
medium
Click to reveal answer
Vitess is an open-source system originally built at YouTube that makes MySQL horizontally scalable.
00:31
What are the peak performance metrics after migration?
medium
Click to reveal answer
What are the peak performance metrics after migration?
medium
Click to reveal answer
2.3 million queries per second at peak, median latency of 2 milliseconds, and over 40 million daily active users.
01:10
💡 Key Takeaways
📊
Scale of Slack's MySQL
Provides a concrete, impressive number that sets the stage for the technical discussion.
00:03
💡
Hotspot Problem
Illustrates a common scaling pitfall with real-world consequences.
00:18
🔧
Vitess as a Solution
Highlights a proven open-source tool for horizontal scaling of MySQL.
00:31
📊
Post-Migration Success
Demonstrates the effectiveness of the migration with impressive performance metrics.
01:10Full Transcript
[00:03] Wrong. Slack's MySQL database handles over 1.5 billion messages every single day. They started with MySQL sharded by workspace. Each company's data lived on its own database cluster, and the application code handled the routing.
[00:18] When most teams were small, this worked great. The load spread out naturally. But then big enterprise showed up. Now you've got a single company with tens of thousands of users, and all their messages, channels, and DMs are
[00:31] hammering one cluster. So it became a hotspot. And because the routing logic is baked into the app, there's no way to break it up further. So they migrated to Vitess, an open-source system originally built at
[00:43] YouTube that makes MySQL horizontally scalable. Basically, it lets you shard your database without your apps having to know about it. So instead of only splitting data by workspace, Vitess let them split by users, channels, or
[00:56] workspaces independently. The app just talks to Vitess like it's one giant database, and Vitess figures out where to send each query behind the scenes. Now it's been over 3 years since the migration, and 99% of their MySQL
[01:10] traffic runs through this system. That's 2.3 million queries per second at peak, a median latency of 2 milliseconds, and over 40 million daily active users, all So next time someone tells you your relational database doesn't scale, send
[01:25] relational database doesn't scale, send them this.
