Rate Limiting and Load Shedding: Staying Up Under Overload
Overload design: token bucket and sliding window algorithms, distributed rate limiting, prioritized load shedding, graceful degradation and fair quota strategies.
What should happen when traffic exceeds capacity?
In undesigned systems the answer is predictable: every request slows a little more, queues fill, a timeout storm begins, and everything collapses for everyone. In a designed system the answer was chosen in advance: excess traffic is politely rejected at the gate, critical functions are protected, and the system survives by shrinking. Rate limiting and load shedding are the two instruments of that choice — one works at the door, the other inside.
Rate limiting algorithms: choosing the right tool
Algorithm | Behavior | Strength | Watch out |
|---|---|---|---|
Token bucket | Fixed refill rate + burst capacity | Tolerates short bursts; the industry default | Bucket size = permitted burst; choose deliberately |
Leaky bucket | Smooths output to a fixed rate | Protects downstream with even flow | Queues bursts; adds latency |
Fixed window | Counter per window | Simplest implementation | 2x burst gap at window boundaries |
Sliding window | Weighted count over a sliding window | Closes the boundary burst; precise | Slightly costlier counting |
The practical choice is usually token bucket (burst tolerance that respects user experience) or sliding window (precise quotas). Policy matters more than algorithm: the limit applies to whom (IP, user, API key, tenant), on which resource (per endpoint), and with which response (429 + Retry-After header)?
Distributed limiting: a single node's counter is not enough
In an N-replica service, node-local limits permit N times the intended rate; a consistent boundary centralizes the counter — in practice, atomic counting on Redis (Lua-scripted token bucket) is the industry standard. Two engineering notes: the check sits on the hot path, so counter access must stay sub-millisecond; and if Redis is unreachable the policy must be deliberate (fail-open: continue unlimited — availability first; fail-closed: reject — protection first). That decision is made at design time, not during the incident. Gateway-level limit modules carry the base; business-aware quotas (per tenant plan) are completed in the application layer.
Load shedding: the last defense inside
Rate limiting is the contract at the door; load shedding is the system watching its own health and dropping work: when queue depth, in-flight requests or latency thresholds are crossed, a share of new requests is rejected fast — because a fast 'no' is worth more than a slow 'yes': a request dragged into timeout burns resources and triggers client retries that multiply the load. Shedding has priorities too: health checks and critical flows like payment/checkout are protected; ornaments like search suggestions, recommendation boxes and analytics are shed first. That priority list is a business decision, not a technical one — and it must be written before campaign day.
Graceful degradation: serving while shrinking
The user-facing face of shedding is graceful degradation: if the recommendation service is down, the page arrives without recommendations but fully purchasable; if the stock service is slow, 'stock info shortly' appears but the cart works; static/cached content backs live content. The precondition is a dependency map: without which components does the page 'still do its job'? Answer that per critical flow and code the fallback — degradation is a pre-written scenario, not incident-day improvisation.
Retry storms: the assault of well-meaning clients
Under overload, the most dangerous traffic is your own clients retrying aggressively on errors — every failure wave amplifies the next. The contract is two-sided: the server returns Retry-After on 429/503; clients apply exponential backoff + jitter and cap retries with a budget. If your mobile apps and integration clients were not written with this discipline, your rate limiter ends up fighting your own ecosystem.
Business impact: controlled shrinkage is cheaper than uncontrolled collapse
Overload is a choice between two scenarios: deliberately reject 20% of traffic and serve 80% fully, or try to serve 100% and give 100% timeouts. The first preserves most of the campaign's revenue; the second risks all of it — and post-crash recovery is its own cost. Rate limiting is also commercial infrastructure: API plan quotas, tenant fairness and abuse protection all feed off the same mechanism.
Frequently asked questions
How are limits determined?
Backwards from load-tested safe capacity: system capacity → endpoint budgets → per-actor shares. Guessed limits either cut too early or protect too late.
Doesn't 429 hurt user experience?
Done right, it protects it: burst tolerance keeps normal users from ever seeing it; at the boundary, a clear message + Retry-After beats a silent timeout.
Do load shedding and autoscaling conflict?
They complement: autoscaling adds capacity on a scale of minutes; shedding bridges on a scale of seconds. In a sudden spike, shedding holds first, then new capacity takes over.
Does rate limiting solve bot traffic?
Not alone: limits cap volume; bot management (behavior analysis, verification) filters the source. Together they preserve stock fairness on campaign day.
Overload design checklist
Limit policy written: who, which endpoint, which algorithm, which response
Distributed counter in place; fail-open/closed decided deliberately
429/503 responses carry Retry-After; clients apply backoff+jitter
Shedding thresholds and priority list business-approved
Fallback behavior of critical flows coded and tested
Retry budgets defined in internal clients and SDKs
Pre-campaign overload drills (chaos/load tests) run
SSH Yazılım builds overload resilience end to end — from limit policy to degradation scenarios. Let us secure your peak day together.