An API can remain available while doing almost no useful work. Requests enter a queue, exceed their deadlines, and consume resources after their callers have already given up. Adding retries makes the graph busier while reducing the number of completed user operations.

Google’s SRE chapter on handling overload describes why raw queries per second can be a poor capacity model: requests consume different amounts of resources. It also discusses degraded responses, criticality, and client-side throttling. The design exercise below applies those ideas to a hypothetical report service.

Budget the scarce resource

Imagine two report types. A small report needs 10 milliseconds of CPU; a large report needs 500 milliseconds. A limit of 100 requests per second could represent either one CPU-second or fifty CPU-seconds of demand per second. The request count is identical; the capacity requirement is not.

Start with an admission estimate based on report size, then compare estimates with observed execution cost. Use conservative limits for unknown requests. CPU estimates do not replace separate memory, connection, and downstream concurrency limits: a report that waits on a database can exhaust connections while CPU remains low.

A proposed admission loop. Capacity signals influence which work enters the bounded execution pool.

A proposed admission loop. Capacity signals influence which work enters the bounded execution pool.

Bound waiting as well as execution

A concurrency cap protects workers, but an unbounded queue moves failure into waiting time. Give each request a deadline and reject work that cannot plausibly begin in time. Queue age matters more than queue length when jobs have different durations.

For example, if the oldest waiting request has already consumed most of its deadline, admitting another large job probably worsens the outcome. Returning an explicit overload response early gives the caller a chance to choose a cheaper report or try later.

Control Question it answers
Concurrency ceiling How much work may execute now?
Queue bound How much waiting work can we retain?
Deadline check Can this request still be useful?
Priority policy Which work should receive scarce capacity?

Measure goodput during a load test

For this exercise, define goodput as successfully completed operations within their deadlines. Count rejected work separately from accepted work that times out. Otherwise an apparently improved latency percentile may simply reflect that slower requests disappeared from the measured population.

I would test a changing mix of small and large reports, a slow dependency, and a burst of retries. The useful result is a stable amount of timely work under stress, with an explainable rejection policy. A throughput number measured only below saturation says little about that behavior.

A practical starting point is one bounded queue, one explicit timeout budget, and a dashboard showing arrival rate, goodput, rejection rate, and oldest queue age together.

Related reading: Cloud cost optimization.

Archive note: added November 18, 2026, and filed under 2025-12 as part of the systems reading series.