Last week I measured how fast Cassandra runs on a cheap cloud box. I can defend two different answers.
The first answer is 87,311 operations per second. That’s the most the machine would do, flat out, 256 client threads pushing as hard as the closed loop allows.
The second answer is 80,712. That’s what the same machine does if you refuse to let the 99.9th-percentile latency climb above ~15 milliseconds, which btw is the kind of promise you’d actually have to make if real users were on the other end. The last 8% of throughput cost 3.5× the tail latency. Same hardware, same software, same afternoon.
What does “deployable capacity” mean?
I repeated the benchmark with progressively more concurrent clients, each step applying more pressure to the same machine. Throughput initially increased, but the slowest requests eventually became much slower. I call the highest throughput that still kept p99.9 latency within ~15 milliseconds the deployable capacity. In this test, that was 80,712 operations per second.
Neither number is wrong. They answer different questions, asked by different people. And once you start looking at benchmarks this way: whose question does this number answer? a lot of things about the benchmarking world that seem strange start making perfect sense.
Benchmarks are built for one side of the table
Look at how the big public benchmarks are constructed.
MLPerf freezes the model, the dataset, the accuracy target i.e. everything about the workload and lets submitters vary exactly one thing: the hardware and the software stack underneath it. TPC-C has run the same warehouse schema and nearly the same transaction mix for over thirty years. SPEC fixes the source code down to the compiler flags you’re allowed to report. And every chip launch deck you’ve ever seen (“1.4× faster than the previous generation,” the bar chart against the rival’s part) is built on the same discipline: identical workload on both bars, or the comparison means nothing.
That’s not an accident and it’s not a flaw. If you want to compare five platforms across three hardware generations, the workload must be held identical everywhere. The mix is frozen so that the hardware can vary.
Freeze the workload, sweep the hardware. That is the vendor’s question: how does my product compare? and the entire public benchmarking apparatus is a well-engineered machine for answering it rigorously.
The interesting part is what happens when you ask the transpose.
The operator’s question is the transpose
If you run software rather than sell hardware, your situation is inverted. Your hardware is a constant—you bought it, or you committed to the instances. What varies is your workload: the read/write mix shifts through the day, the key skew sharpens when something goes viral, the dataset outgrows RAM a little more every week.
So the operator’s question is: freeze the hardware, sweep the mix. How does my machine behave across the workloads I actually experience?
Here’s a small, concrete example of why the difference isn’t academic. The standard lore about LSM-tree databases like Cassandra says writes are cheap: an append to the commit log and a memtable insert, no read-before-write. It’s in every talk. I believed it.
The stock stress profile I started from runs 80% reads, 20% writes, frozen mix, in the classic style. So I unfroze it: same hardware, same everything, three write/read ratios.
| write/read mix | throughput (op/s) | p99.9 latency |
|---|---|---|
| 20/80 | 80,409 ± 615 (n=3) | 10.8 ms |
| 50/50 | 61,100 | 15.9 ms |
| 80/20 | 51,770 | 18.1 ms |
How is the write cost inferred?
This is a capacity model, not a comparison of individual request latencies. Assume throughput is inversely proportional to the mix-weighted operation cost:
T(x) = K / ((1 − x)r + xw), wherexis the write fraction,ris the cost of one read,wis the cost of one write, andKis total system capacity.The first two measurements imply
w/r ≈ 2.33. That model predicts roughly 49.3K ops/s at 80% writes; the measured result was 51,770, within 5%. This is why the defensible claim is that writes consume roughly twice the system capacity of reads at this operating point—not that every write takes twice as long as every read.
LIVE MODEL / MIX-AXIS EXPLORER
What happens when the workload writes more?
Move the mix. The curve is a two-parameter capacity model; the three points are measured runs.
Each tile represents 5% of the operation mix—not a row, request timeline, or individual latency.
MODEL T(x) = K / ((1 − x)r + xw)
FIT w / r = 2.33
ANCHOR 20% writes = 80,409 ops/s
Write-heavier traffic was slower, a lot slower. At this operating point, a write costs roughly twice as much as a read, the opposite of the lore. The mechanism, in brief: this stress profile’s “one insert” is a ~10-row batch into one of only 100 hot partitions—contended memtable partitions, commit-log volume, memtable flushes landing inside the measurement window. The lore isn’t wrong in general. It’s wrong here, at this dataset size, with this access pattern. Which is the point.
The satisfying part: after measuring the first two rows, I fitted the simplest possible cost model (throughput as capacity over a mix-weighted cost of reads and writes) and sealed a prediction for the third row before running it: 47-52K. Measured: 51,770. Two measurements plus one law predicted the third within 5%.
No published benchmark would ever have told me any of this, and it’s not because published benchmarks are dishonest. It’s because my mix axis is, by construction, the axis their methodology holds frozen.
Why nobody serves the other side
If the operator’s question is so useful, why does essentially every published benchmark answer the vendor’s question instead?
Because the economics only work one way. A frozen-mix result is reusable: one number, presentable to every customer, comparable across platforms and years. That’s what lets a vendor amortize a benchmarking team across every sale they’ll ever make. An operator’s result is bespoke by definition: it’s about your mix on your hardware, so it can’t be amortized, can’t go on a slide for anyone else, and might even land somewhere unflattering. Every incentive points the machine the same direction, and so that’s the direction the whole industry’s benchmarking machinery points.
Which leaves the operator’s side of the table oddly empty. Enterprises make seven-figure hardware and platform decisions using numbers that were generated to answer someone else’s question or worse, using the vendor’s number as if it answered theirs. Sometimes it’s close enough. Sometimes, as the table above shows, the truth at your operating point is the opposite of the lore.
What I’m doing about it
This post is the first of a series. The plan is simple: measure things from the operator’s side of the table, and show all the work: hardware, configs, commands, raw numbers, and every asterisk.
Today’s asterisks, for the record: the box was a 16-vCPU shared cloud instance, the load generator was closed-loop and colocated with the server, and the dataset was small enough to live in memory. Each of those caveats is a future post.
Coming next: how much can you trust a single benchmark number? I ran the exact same benchmark five times back to back, then compared against the previous day’s run of the identical setup, and the day-to-day gap was five times larger than the run-to-run noise. That gap is where most benchmark misreadings live, and it has a number you can measure.
If you make capacity, procurement, or platform decisions and have ever wondered why the benchmark numbers never look like your production numbers, this series is for you. The short answer is: those numbers were never for you. Let’s make some that are.
Method and caveats: Cassandra 4.1 under cassandra-stress on a 16-vCPU shared cloud instance; closed-loop generator colocated with the server; in-memory dataset. The standing measurement rules are documented on the Methods page.