There is an asterisk in my first report that deserves more than a footnote: the load generator was closed-loop.
Each client sent one request, waited for the response, and only then sent the next request. That sounds harmless. It is also how many useful benchmarks work. But it creates a relationship that production traffic often does not have: when the server slows down, the clients waiting on it also slow down their request generation.
The benchmark withdraws some pressure precisely when the system is struggling.
That does not make the result false. It defines the question the result can answer, and the question it cannot.
Two ways to generate load
Imagine ten clients testing an API.
In a closed-loop test, each client is allowed one request in flight:
send request → wait for response → send next requestConcurrency is controlled. Arrival rate is an outcome. If responses take longer, each client completes the loop less frequently and sends fewer new requests.
In an open-loop test, requests are scheduled independently of previous responses:
scheduled arrival → send request
scheduled arrival → send request
scheduled arrival → send requestArrival rate is controlled. Concurrency and queue depth are outcomes. If the server slows while requests keep arriving, they wait. That waiting becomes part of the latency users experience.
Neither model is universally better. They hold different things constant.
| Load model | What is controlled | What it is good for |
|---|---|---|
| Closed loop | In-flight clients | Saturation studies, concurrency limits and maximum completed work |
| Open loop | Externally offered rate | Arrival-driven queueing, overload behaviour and SLO capacity |
The mistake is not using a closed loop. The mistake is treating closed-loop tail latency as though demand remained constant during slowdowns. It does not: waiting clients stop creating new requests, so part of the queue and the latency it would create is omitted.
Closed loop means the benchmark quietly reduces demand when the server becomes slow.
Suppose production traffic sends 50 requests every second and the server stalls for one second:
- In production/open loop, another 50 requests still arrive during the stall. They queue, and some wait much longer than one second while the server catches up.
- In a closed-loop benchmark, clients with requests in flight wait for responses. They do not generate their next requests during the stall. Much of that potential queue never exists.
The closed-loop p99.9 may capture the stall experienced by requests already in flight, but it omits some waiting that independently arriving requests would experience.
What happens during a stall
Suppose the server pauses for one second.
In a closed-loop test, every client affected by that pause waits. Those clients do not issue their next requests until their current requests complete. The pause appears in the latency of work already in flight, but some of the demand that would have arrived during the pause is never generated.
In an open-loop test at 50 requests per second, roughly 50 new requests are scheduled during that same second. They accumulate behind the stalled work. When the server resumes, it must process both the normal arrival stream and the queue created during the pause.
The server did the same thing in both cases. The users did not experience the same system.
This measurement problem is commonly called coordinated omission: the load generator’s request timing becomes coordinated with the latency of the system being measured, omitting some of the waiting that independent arrivals would have observed.
Illustrative simulation, not benchmark evidence. The instrument below runs the same server capacity and global stall under closed-loop and open-loop arrivals. It explains the measurement mechanism; it does not claim production numbers.
ILLUSTRATIVE SIMULATION / NOT MEASURED
What arrives while the server is stalled?
The server is identical in both lanes. Only the rule that creates the next request changes.
Playback expands the selected stall to 4.5 seconds on screen. Simulation time remains actual.
At 50 requests per second, a 1,000 millisecond stall schedules 50 additional open-loop requests and zero additional closed-loop requests.
Why operators should care
A vendor benchmark can honestly report maximum throughput from a closed-loop test. But many production requests, including users, agents, scheduled jobs, and retries, continue arriving according to their own clocks when the system slows down.
The vendor cannot know that arrival pattern unless you provide it. If infrastructure is purchased using maximum throughput alone, the resulting capacity plan may underestimate queueing, latency, headroom, and replica count.
Treat the vendor number as a starting point. Before making the decision, replay your workload shape and ask: how much traffic can this system serve while meeting our latency objective?
This principle applies to any platform your system depends on: evaluate it using the workload shape and service objectives you will actually impose on it.
Why this matters for AI serving
An LLM server has at least two visibly different waiting costs.
- Time to first token (TTFT): how long a request waits before output begins.
- Time per output token (TPOT): how quickly subsequent tokens arrive.
At light load, both may look healthy. As offered request rate rises, continuous batching and queueing can buy more aggregate token throughput while making individual requests wait longer. The largest token-per-second number and the largest request rate that still meets a TTFT or TPOT objective need not be the same operating point.
That is the AI-serving version of the distinction in my Cassandra experiment: maximum completed throughput is not automatically deployable capacity.
My next campaign will use vLLM’s serving benchmark with controlled request
rates. With burstiness set to 1, the client schedules arrivals using a Poisson
process instead of waiting for each previous response before deciding when the
next request exists.
The command will look approximately like this:
vllm bench serve \
--backend vllm \
--model <pinned-model-and-revision> \
--dataset-name random \
--random-input-len 1024 \
--random-output-len 128 \
--request-rate <requests-per-second> \
--burstiness 1 \
--goodput ttft:1000 tpot:50 \
--save-result \
--save-detailedThis command has not produced a result yet. The model revision, vLLM version, container digest, GPU, driver and request-rate sweep will be pinned in the experiment manifest before the first measured run.
The two provisional service objectives (TTFT at or below one second and TPOT at or below 50 milliseconds) are experimental decision boundaries, not universal standards. Their purpose is to distinguish raw throughput from goodput: requests completed while satisfying the latency requirements.
The manipulation check moves with the method
Calling a generator “open-loop” does not make the delivered workload correct. The client can become CPU-bound, scheduling can drift, requests can fail, and actual prompt or output lengths can differ from their configuration.
Before trusting each run, I will verify:
- The requested and observed arrival rates.
- The actual prompt- and output-token distributions.
- Successful, failed and incomplete requests.
- The delivered server configuration and model revision.
- Whether the client itself fell behind its arrival schedule.
The principle is the same as the write/read experiment: verify that the treatment happened before explaining the outcome.
The question the experiment will answer
The eventual headline will not be “this GPU produces X tokens per second.”
It will be:
At a documented request shape and arrival process, how much traffic can this GPU serve while keeping TTFT within one second and TPOT within 50 milliseconds and how far is that number below maximum throughput?
That gap is not benchmark trivia. It becomes replica count, headroom, cost per request and the probability that a traffic spike turns into a queue the system cannot drain.
The Cassandra result established the question. The vLLM experiment will change the arrival model, pin the AI workload, and measure the answer.
Status: methodology note and experiment pre-registration; no vLLM measurements are reported here. The closed-loop Cassandra setup and its other claim boundaries are documented in Who is your benchmark for?. The standing rules are on the Methods page.