Rate Limiting and Throttling in C++: How to Protect Your API from Overloads

When Your API Starts to Choke: Introduction to Rate Limiting

Imagine this: you've written a perfect server in C++. It's fast, efficient, uses all CPU cores. You deploy it to production, and... five minutes later it crashes. What happened? It turns out some rogue script decided to "test" your API by sending thousands of requests per second.

Sound familiar? If not, you're lucky. But sooner or later, every developer faces the need to control load. This is where rate limiting and throttling come into play — two powerful mechanisms that save servers from overloads and ensure even resource distribution.

In this article, we'll break down how to implement these mechanisms in C++. You'll not only learn the theory but also get working code you can use immediately in your projects. Let's go!

Rate Limiting vs Throttling: What's the Difference?

Before diving into code, let's clearly distinguish these two concepts. Many people confuse them, but the difference is critical.

Rate Limiting

This is a "barrier." You say: "No more than 10 requests per second. Anything beyond that gets banned (or queued)." Rate limiting protects your API from avalanche requests, DDoS attacks, and random traffic spikes.

Throttling

This is a "volume control." You don't block requests entirely; you slow down their processing. For example, if a client sends 100 requests per second, you process them but with a 10ms delay between each. Throttling is a more "polite" way to manage load — it doesn't discard data, it just smooths out peaks.

In real projects, these techniques are often combined: first throttling for smoothing, and when a critical threshold is exceeded — rate limiting with a 429 (Too Many Requests) error.

Rate Limiting Algorithms in C++

There are several classic algorithms. We'll cover the two most popular and effective ones: Token Bucket and Sliding Window. Both are easy to implement in C++ and deliver excellent results.

1. Token Bucket

Imagine a bucket into which N tokens drip every second. Each request takes one token. If the bucket is empty — the request is rejected. Simple and brilliant.

Here's an implementation in modern C++ using std::chrono:

#include <chrono>#include <mutex>#include <atomic>

class TokenBucket {public:    TokenBucket(size_t max_tokens, double refill_rate)        : max_tokens_(max_tokens)        , refill_rate_(refill_rate)        , current_tokens_(max_tokens)        , last_refill_(std::chrono::steady_clock::now()) {}

    bool try_consume() {        std::lock_guard<std::mutex> lock(mutex_);                // Calculate how many tokens have accumulated since last time        auto now = std::chrono::steady_clock::now();        auto elapsed = std::chrono::duration_cast<std::chrono::milliseconds>(now - last_refill_);                // Refill the bucket        double tokens_to_add = elapsed.count() * refill_rate_ / 1000.0;        current_tokens_ = std::min(max_tokens_, current_tokens_ + tokens_to_add);        last_refill_ = now;                // Try to consume a token        if (current_tokens_ > 0) {            --current_tokens_;            return true;        }        return false;    }

private:    std::mutex mutex_;    size_t max_tokens_;           // Maximum bucket capacity    double refill_rate_;          // Tokens per second    double current_tokens_;       // Current number of tokens    std::chrono::steady_clock::time_point last_refill_;};

Usage:

TokenBucket limiter(10, 1.0); // 10 tokens, refill 1 token/sec

// In the request handler:if (limiter.tr