Imagine this, you run a SaaS platform, and POST /v1/messages is your primary revenue generator. But every time the clock hits 00:00 seconds on a new minute, customer support gets a flurry of tickets.

"Why are we getting hit with HTTP 429 Too Many Requests when we haven't hit our rate limit for the hour?"

If your system uses a naive rate-limiting algorithm like Fixed Window, you’re suffering from the classic thundering herd at minute boundaries.

The Problem

In a typical Fixed Window setup, traffic limits reset at exact clock boundaries (e.g., 12:00:00, 12:01:00). This leads to two critical failure modes:

Traffic Bursts at :00: Automated client scripts and cron jobs wait for the next minute boundary to retry or fire requests simultaneously. Your Redis instance and rate limiter are hit with a massive surge at XX:XX:00.

The Double Capacity Spike: If a customer gets a quota of 100 requests/minute and sends 100 requests between 12:00:50 and 12:01:00, then another 100 requests between 12:01:00 and 12:01:10, they just sent 200 requests in a 20-second span without breaching their fixed-window limit—crashing downstream services.

The Token Bucket Solution

The token bucket rate limiter works by giving every API key a bucket with a maximum capacity (e.g., N tokens) that allows short bursts of requests. Tokens are continuously added to the bucket at a fixed, steady rate (e.g., 10 tokens per second) until the bucket reaches its maximum capacity. Whenever a request arrives, the rate limiter checks whether the bucket contains at least one token. If a token is available, it removes one token from the bucket and forwards the request to the Message Service. If no tokens remain, the request is immediately rejected with a 429 Too Many Requests response.

The token bucket algorithm avoids clock alignment issues because tokens are replenished continuously over time instead of resetting simultaneously at fixed intervals (e.g., at every :00), which smooths traffic and prevents sudden request spikes at time boundaries. It also supports controlled bursts by allowing clients to temporarily exceed the average request rate, up to the bucket's maximum capacity, without overwhelming downstream services or databases. In Redis, the bucket can be implemented using lazy evaluation, where tokens are only recalculated when a request arrives by computing the elapsed time since the last update (tokens=min(capacity, current+elapsed×refill_rate)) and atomically updating both the token count and timestamp within a single Redis script, eliminating the need for background refill timers.

Token Bucket Example

import time


class TokenBucket:

    def __init__(self, capacity: float, refill_rate: float):
        """
        :param capacity: Maximum number of tokens the bucket can hold (burst limit).
        :param refill_rate: Number of tokens added per second.
        """
        self.capacity = capacity
        self.refill_rate = refill_rate
        self.tokens = capacity  # Start with a full bucket
        self.last_refill_time = time.monotonic()

    def _refill(self):
        """Calculates elapsed time and adds tokens back to the bucket."""
        now = time.monotonic()
        elapsed = now - self.last_refill_time

        # Add new tokens generated over elapsed time
        self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_rate)
        self.last_refill_time = now

    def allow_request(self, tokens_requested: float = 1.0) -> bool:
        """
        Checks if enough tokens are available to allow the request.
        Deducts tokens if successful.
        """
        self._refill()

        if self.tokens >= tokens_requested:
            self.tokens -= tokens_requested
            return True

        return False


# --- Example Usage ---
if __name__ == "__main__":
    # Capacity = 5 tokens max burst limit
    # Refill rate = 2 tokens per second
    bucket = TokenBucket(capacity=5, refill_rate=2)

    print("--- 1. Firing initial burst of 6 rapid requests ---")
    for i in range(1, 7):
        allowed = bucket.allow_request()
        status = "200 OK" if allowed else "429 Too Many Requests"
        print(f"Request {i}: {status} (Tokens remaining: {bucket.tokens:.2f})")

    print("\n--- 2. Waiting 1.5 seconds to refill tokens ---")
    time.sleep(1.5)  # Should regenerate ~3 tokens (1.5s * 2 tokens/s)

    print("\n--- 3. Trying requests again after waiting ---")
    for i in range(7, 10):
        allowed = bucket.allow_request()
        status = "200 OK" if allowed else "429 Too Many Requests"
        print(f"Request {i}: {status} (Tokens remaining: {bucket.tokens:.2f})")
                                        

If your customers are hitting artificial walls every time the minute clock resets, ditch fixed counters. Implementing a Token Bucket (or Sliding Window Log/Counter) keeps your API predictable, protects downstream primary databases from stampedes, and keeps your revenue endpoint flowing.