---
title: 'Rate Limiter System Design: Token Bucket, Leaky Bucket, Scaling'
source: 'https://youtube.com/watch?v=YXkOdWBwqaA'
video_id: 'YXkOdWBwqaA'
date: 2026-08-06
duration_sec: 466
---

# Rate Limiter System Design: Token Bucket, Leaky Bucket, Scaling

> Source: [Rate Limiter System Design: Token Bucket, Leaky Bucket, Scaling](https://youtube.com/watch?v=YXkOdWBwqaA)

## Summary

This video provides a comprehensive system design walkthrough for building a rate limiter, covering core algorithms like fixed window, token bucket, and leaky bucket, along with architectural decisions for scaling and avoiding race conditions. It explains how rate limiters protect APIs from overload while ensuring fair access, and discusses implementation strategies including client-side, server-side, and middleware approaches.

### Key Points

- **What is a Rate Limiter?** [00:03] — Rate limiters control how many requests a client can make to an API in a given time window, protecting systems from overload while maintaining fair access for legitimate users.
- **Core Requirements** [00:17] — The system should limit requests based on configurable rules (e.g., 100 requests per minute per user), reject with HTTP 429 when exceeded, include headers for remaining and reset time, introduce minimal latency (under 3ms P95), and be highly available across multiple servers.
- **Fixed Window Counting** [00:58] — Simplest approach: divide time into fixed windows (e.g., 1 minute). Each user has a counter that resets at window start. When the counter hits the limit, additional requests are rejected until the next window.
- **Storage Considerations** [01:27] — Database is too slow for counters. In-memory storage on a single server is fast but doesn't work across multiple servers, leading to separate counters and bypassing limits. Redis solves this as a shared in-memory data store.
- **Fixed Window Flaw** [02:37] — A user can make 100 requests in the last 10 seconds of a minute and 100 more in the first 10 seconds of the next minute, totaling 200 requests in 20 seconds, violating the intended limit. This edge case occurs at every window boundary.
- **Token Bucket Algorithm** [03:06] — Industry standard (used by AWS, Stripe). A bucket holds tokens added at a steady rate. Each request consumes one token; if none left, reject. Tokens accumulate during quiet periods, allowing legitimate bursts while maintaining overall rate.
- **Token Bucket Parameters** [04:03] — Bucket capacity determines burst size; refill rate determines sustained throughput. Example: capacity 100, refill 100/min allows up to 100 requests instantly, maintaining 100/min long-term.
- **Implementation Placement** [04:33] — Options: client-side (untrusted), server-side (mixes with business logic), middleware (dedicated service like API gateway). Middleware offers best balance of control and simplicity for most systems.
- **System Architecture** [05:44] — Rate limiting rules stored in a configuration service. Middleware fetches rules, stores token bucket state in Redis. On request, identifies user, checks tokens, decrements if available, else returns 429 with retry header.
- **Scaling and Race Conditions** [06:26] — Multiple servers can read the same counter value, leading to lost updates. Solved with atomic operations in Redis using Lua scripts that bundle read, check, and increment into a single unit.

### Conclusion

The video provides a solid foundation for designing a rate limiter, covering key algorithms, storage, and scaling considerations. It emphasizes the token bucket as the industry standard and highlights the importance of atomic operations for consistency in distributed systems.

## Transcript

many requests to GitHub, Stripe or AWS and your requests get rejected. How do this system work? Ray limiters control how many requests a client can make to an API in a given time window. They protect systems from overload while
maintaining fair access for legitimate users. Let's tackle the core design challenges. Let's go over the requirements first. The system should limit incoming requests based on configurable rules like 100 API requests
per minute per user. When limits are exceeded, the system should reject requests with HTTP 429 and include helpful headers showing rate limit remaining and reset time. The system should introduce minimal latency
overhead, say under 3 millisecond P95 per check. The system should be highly available and accessible by multiple servers. Now that we understand what we need to build, let's start with the simplest
approach, fixed window counting. We divide time into fixed windows like one minute intervals. Each user gets a counter that resets at the start of each window. Here's how it works. A user is allowed 100 requests per minute. At the
start of each minute, their counter resets to zero. Each request increments to counter. When they hit 100 requests, we reject additional requests until the next minutes begins. We need to store these counters somewhere fast. The
database is too slow for this. Today's video is sponsored by Warped, the best way to code with AI agents. Too often, agents write code that's almost right, leaving developers stuck debugging instead of shipping. Warp is different.
Rank top of terminal bench and bench verified. Warps agent understands your context and writes production ready code out of the box. Prom, review, and refine all in one interface. No context switching, no wasted time. You stay in
control and it pays off. On average, users are saving over an hour a day with Warp. Download Warp by clicking the link in the description. We're adding a database query to every request, which could overload the very
system we're trying to protect. What about in-memory storage and this server? This would be very fast, but it only works for a single server. When we scale to multiple servers, each server would have its own separate counters. A user
could make 100 requests to server A and 100 requests to server B, effectively getting 200 requests per minute instead of 100. Reddit solves both problems. Is an in-memory data store that's shared across all our servers. Reddis provides
primitives to increment counters and reset them automatically. But fix window have a critical flaw. Consider this scenario. A user makes 100 requests in the last 10 seconds of a minute, then 100 more requests in the first 10
seconds of the next minute. Both bursts are within the 100 requests per minute limit individually, but they've made 200 requests in just 20 seconds, which clearly violates the intended ray limit. This edge case happens at every window
algorithm. The token bucket algorithm solves the fixed window problem. It's the industry standard used by companies like AWS and Stripe. Think of it like this. Imagine a bucket that hosts tokens. New tokens are
added at a steady rate. Each request consume one token. When there are no tokens left, we reject the request. Let's see how this fixes our window boundary problem. We set a bucket capacity of 100 tokens that refills at
100 tokens per minute. During quiet periods, tokens accumulate. When user makes burst requests across window boundaries, they consume accumulated tokens but can't exceed the refill rate over time. The key insight is that
tokens accumulate during quiet period. This allow legitimate traffic bursts while maintaining the overall rate limit. A user can game the system by timing the request to window boundaries. The algorithm uses two parameters.
Bucket capacity determines burst size and refill rate determines sustained throughput. A capacity of 100 with a refill rate of 100 per minute allows up to 100 requests instantly that maintains exactly 100 requests per minute
long-term. There are other algorithms like sliding window logs, sliding window counters, and leaking buckets that solve the fixed window problem differently. balance between simplicity and effectiveness for most use cases. Now
that we have our algorithm, we need to decide where to implement it in our options for where to place our ray limiter. Client side ray limiting puts the logic in client applications, but we can't trust clients to enforce their own
limits. Malicious users can modify the code or bypass restrictions entirely. Serverside ray limiting embeds the logic in our application code. This gives us complete control over the algorithm and keeps everything in one place. The
downsides is that ray limiting gets mixed with business logic and each service needs its own implementation. Middleware ray limiting uses a dedicated service between clients and the APIs. This could be an API gateway, a reverse
proxy or a custom service. This keeps ray limiting separate from business logic and provides a single place to manage policies. The trade-off is an increase in system complexity. Which should we choose? It depends on our
situation. If we already have an API gateway handling authentication, adding rate limiting there makes sense. If we need a custom algorithm, server side gives us flexibility. For most systems, middleware offers the best balance of
control and operational simplicity. Our system architecture looks like this. We store ray limiting rules in a configuration service. These rules define limits like premium users get a th00and requests per hour or free users
get 100 requests per hour. The middleware fetches these rules and stores token bucket state in Reddus. When a request arrives, the middleware identifies the user, fetches their token bucket from Reddus and checks if tokens
are available. If yes, it decrements the token count and forwards the request to API servers. If no token remains, it returns HTTP 429 with header showing when the user can retry. This works great for a single rail limited server.
But what happens when we need to scale to multiple servers, we run into race conditions. Here's what happens. Server A reads a counter value of three from radius. At the same time, server B also reads three. Both servers check their
limits, decide the request is okay, increment the value to four, and write it back to Reddus. The counter now shows four instead of the correct value of five. We've lost a count. We can solve this with atomic operations. Reddus
supports these two lure scripts that bundle the read, check, and increment into a single indivisible unit. This prevents race condition entirely. We've covered the essential building blocks for ray limiters in this video. There
are other interesting topics we didn't cover. How do we handle geographic latency with multi-reion deployment? How do we handle hot keys when a few users generate most traffic? How do we upload rate limiting rules without restarting
servers? Should the system fail open or fail closed when key components go down? These are all interesting topics worth exploring. Ready to ace your next technical interview? Join our community where we
offer comprehensive courses on system design, coding, behavioral questions, machine learning, and object-oriented design. Learn more at byitebico.com.