Backend Load Shedding: Protecting Systems Under Overload

An overloaded backend does not merely become slower; it can enter a self-reinforcing failure loop. Queued requests retain connections, memory, and workers for longer. Timeouts trigger retries, retries add more load, health checks fail, and traffic shifts to the remaining instances. When a service attempts to accept everything until the final moment, a short spike can turn into a prolonged incident.

Load shedding deliberately rejects a portion of incoming work as resources approach an unsafe region, preserving controlled latency for the work that remains. It does not replace capacity planning, autoscaling, or code optimization. It is a safety valve for moments when demand exceeds immediate capacity, a dependency slows unexpectedly, or part of the fleet disappears. This guide explains how to select pressure signals, place admission control, define priorities, coordinate retries, and test the system so intentional rejection is preferable to a total collapse.

1. Overload is a dynamic state, not simply 100% CPU

A service can be overloaded while CPU remains available. Its database pool may be exhausted, its event loop may be blocked, its heap may approach a limit, its disk queue may grow, a downstream quota may be depleted, or in-flight work may exceed what can finish before its deadlines. Conversely, a short period of high CPU is not necessarily dangerous if queues remain small and latency stays stable.

The most useful general signal is that the system is no longer completing work as quickly as it accepts work. By Little's Law, when completion throughput is bounded while the amount of work in the system rises, waiting time rises with it. An unbounded queue only delays the error: requests still fail, but fail later after consuming memory, sockets, and user time.

2. Distinguish shedding from rate limits, backpressure, and circuit breakers

These mechanisms are related but protect different boundaries. Rate limiting applies a policy to a client, tenant, or API over time, commonly for fairness and quota enforcement. Backpressure slows a producer according to consumer capacity when the protocol supports feedback. A circuit breaker stops calls to a failing or slow dependency. Load shedding protects the component accepting work by rejecting it when observed local pressure exceeds a safe region.

A production system commonly needs all four. The gateway limits an abusive tenant, a consumer regulates its read rate, a client uses a circuit breaker for a downstream, and each service instance retains local admission control. A gateway alone is insufficient because load distribution is imperfect, request costs vary, and an individual instance can develop a local fault.

Load shedding answers: “Given resource conditions right now, which requests should still enter so the service completes the most useful work?”

3. Put admission control before scarce resources

Rejection should happen before expensive work. If a request has already parsed a large payload, opened a transaction, acquired a database connection, and called two services before checking pressure, most of the cost has already been paid. Admission control should run after the minimum checks required to identify and classify the request, but before scarce resources are allocated.

function handle(request):
  context = authenticateCheaply(request)
  class = classify(context, request.route)

  if deadlineAlreadyExpired(request):
    return error(408, "deadline expired")

  permit = admission.tryAcquire(class, estimatedCost(request))
  if permit is null:
    return overloadResponse(class)

  try:
    return executeBusinessWork(request)
  finally:
    permit.release()

The permit must be released on every path, including exceptions and client disconnects. If several resource permits are required, define a consistent acquisition order to prevent deadlocks. Do not retain a permit while waiting on work that no longer matters: after a deadline expires or a client disconnects, cancel downstream work when the protocol allows it.

4. Concurrency limits often model reality better than QPS

Queries per second are easy to measure but assume roughly equal request costs. A cache read may finish in milliseconds while a report export holds CPU and a database connection for seconds. The same 100 QPS can therefore produce radically different pressure. A concurrency limit more directly constrains the workers, connections, and memory retained by active work.

Separate semaphores can protect interactive reads, writes, batch jobs, and administration. A single small pool lets batch work block user traffic, while completely isolated pools can leave capacity idle. A practical design combines per-class limits with a global ceiling. Heavy operations can consume multiple permit units based on estimated cost rather than treating every request as one unit.

• Derive initial limits from load tests and finite resource counts, not intuition.
• Observe permit hold time, rejection rate, and completed throughput.
• Reserve headroom for health checks, telemetry, and recovery operations.
• Do not increase concurrency automatically merely because a queue grows; growth may mean admission should tighten.

5. Queues must be bounded and deadline-aware

A queue absorbs short bursts when average capacity remains sufficient. A long queue, however, converts load into latency and hides saturation. Every queue needs an item or byte bound, a waiting timeout, and a metric for the oldest item. If a request can no longer finish before its deadline, removing it early is better than letting it occupy capacity.

Suppose a request has 200 ms remaining while estimated queue waiting time is already 300 ms. Appending it creates useless work. The admission controller can combine remaining budget, estimated service time, and queue depth. For asynchronous jobs, the deadline can represent the end of business value, such as a notification that should not be sent after a campaign ends.

remaining = request.deadline - now()
predicted = queue.waitEstimate() + serviceTimeEstimate(request.class)

if remaining <= predicted:
  reject("cannot finish before deadline")
if queue.isFull():
  reject("queue capacity exhausted")

An estimate need not be perfect to help, but it should be conservative and observable. For long-tailed workloads, use an appropriate percentile instead of the mean. Separate priority queues also prevent a flood of background jobs from consuming every waiting slot needed by interactive requests.

6. Shed according to priority and business value

Random rejection is simple, but it may discard a health-critical write while serving a decorative image. Define a small number of explicit classes: recovery and control traffic, core transactions, ordinary interactive requests, optional features, and batch work. Give each class an appropriate threshold, minimum share, or concurrency pool.

A client-provided priority label is not trustworthy. The server should derive priority from the route, authenticated identity, and managed policy. Prevent starvation as well: high-priority work still needs a maximum, while lower classes may need a minimum share to make business progress. Keep the policy simple enough for on-call engineers to reason about during an incident.

7. Graceful degradation preserves value at lower cost

Many requests have more than two possible outcomes. At a soft pressure threshold, a service can omit enrichment, use a slightly stale cache, return fewer results, switch to a cheaper algorithm, or serve precomputed data. This is graceful degradation: intentionally reducing quality while preserving the core function.

Every fallback must truly be cheaper and more independent than the primary path. Calling a second “backup” service that shares the same database does not reduce pressure and may double it. Cached fallback data requires explicit staleness bounds and semantics. A write must never claim success before durable commitment; it can enter a durable queue if the API contract permits that behavior, or return a retryable failure.

• Design degradation modes before incidents and exercise them routinely.
• Measure the proportion of responses served in degraded mode.
• Avoid recursive fallback chains between services.
• Use hysteresis during recovery so modes do not flap around one threshold.

8. Overload responses must guide clients safely

For HTTP, 429 Too Many Requests generally fits a client- or quota-specific limit, whereas 503 Service Unavailable better represents temporary service-wide capacity loss. Retry-After can help when the server has a meaningful delay, but it should not claim false precision. Internal APIs can return a distinct OVERLOADED code and a retryable field.

Clients must not retry immediately without a bound. Retries require exponential backoff, jitter, an overall deadline, and a retry budget. Retry only idempotent operations or requests protected by a correct idempotency key. When the whole cluster is overloaded, retries amplify traffic. When only one task is saturated, one bounded retry on a different task may help. Clients and load balancers should use telemetry to distinguish these cases.

if response.isOverloadError():
  if !request.isSafeToRetry() or budget.exhausted():
    return response
  delay = min(backoff.nextWithJitter(), request.remainingDeadline())
  if delay <= 0:
    return response
  wait(delay)
  return retryOnEligibleTarget()

9. Thresholds need hysteresis and staged actions

A single switch at 90% tends to flap: shedding lowers utilization, the gate opens, traffic floods back, and the gate closes again. Hysteresis uses a higher entry threshold than exit threshold and may require a stable period before recovery. Multiple stages are better than jumping from normal operation to rejecting everything.

1. Normal: accept work under quotas and record a baseline.
2. Moderate pressure: disable optional features and reduce prefetch and batch concurrency.
3. High pressure: shed low classes, shorten queues, and favor work that can meet its deadline.
4. Emergency: retain only control traffic and bounded core operations.

Do not depend on one noisy metric. A policy can combine a hard memory limit with in-flight count and queue age, but precedence must be explicit. A local hard safety condition must not be overridden by healthy fleet averages when one instance is near an out-of-memory failure.

10. Observe rejection and useful work separately

A dashboard that shows only error rate makes load shedding appear harmful even when it prevents a complete outage. Separate intentional overload responses from exceptions and dependency failures, and measure how much work completes within the SLO. The goal is not zero rejection under every offered load; it is maximizing useful completion while keeping failure bounded.

• Accepted, queued, shed, and completed counts by route, tenant class, and priority.
• Queue depth, queue age, in-flight work, permit wait, and permit hold time.
• CPU, memory, pool saturation, thread or event-loop pressure, and downstream latency.
• Retry rate, retry success, request amplification, and work completed after deadlines.
• Degraded-response rate, fallback latency, and fallback data age.

Logging every shed request can itself create I/O overload. Prefer counters, histograms, sampling, and aggregated logs. Traces can record the admission decision and pressure snapshot while controlling cardinality. Alerts should account for both shedding rate and duration; a brief burst differs from a sustained capacity deficit.

11. Test beyond saturation and verify recovery

Load tests often stop when they reach target throughput, but overload protection is proven only beyond that point. Increase offered load in steps through saturation, hold each step long enough for queues and autoscaling to react, and then reduce it to observe recovery. In a sound design, useful throughput reaches a plateau rather than collapsing, accepted-request latency remains bounded, shedding rises intentionally, and the service recovers without a restart.

Include mixed workloads: cheap and expensive requests, one large tenant, a slow dependency, partial instance loss, a reduced database pool, and a retry storm. Confirm that critical traffic retains its reserved headroom. Fault injection needs guardrails and should be exercised in an appropriate environment before a production canary.

assert completed_throughput does not collapse after saturation
assert accepted_request_latency remains within overload objective
assert queue_age is bounded
assert low_priority sheds before critical traffic
assert retry_amplification stays within budget
assert service returns to normal after offered load drops

12. A safe rollout sequence

1. Identify the real bottleneck and build a repeatable load test.
2. Add admission metrics in observation-only mode before rejecting traffic.
3. Set conservative queue and concurrency bounds on one low-risk route.
4. Canary a subset of instances and compare completed throughput, latency, and errors.
5. Add priorities and degradation incrementally instead of shipping several policies together.
6. Write a runbook covering threshold adjustment, policy disablement, false positives, and rollback.
7. Run periodic game days to verify retries, autoscaling, and dashboards still work together.

Load shedding succeeds when users see a bounded portion of requests fail quickly and predictably instead of watching every request stall and fail. It must be designed with deadlines, retries, idempotency, queues, and capacity planning. A safety valve does not make the engine stronger, but it prevents the engine from destroying itself when demand exceeds its limits.

References

• Google SRE Book: Handling Overload
• Google SRE Book: Addressing Cascading Failures
• Envoy Documentation: Overload Manager