test: measure throughput, not just per-operation cost - #44
Merged
Conversation
…e wrong question Every benchmark here measured one operation at a time. A client that costs less per operation only goes faster if the client was what the operations were waiting for, and against a database on the same machine it is not: the last two rounds of work halved the client's share of a query and moved the clock by about one percent. `minato_saturation` points a growing number of callers at a fixed pool and counts what comes back. Reading it in the order it prints says two things that the per-operation tables cannot. At sixteen connections every pooled client stops at about sixteen thousand queries a second with four percent of the machine in use, and the client that is faster per operation is not faster. The first ceiling anybody meets is the size of their pool. epgsql is the tell: it has no pool so it gets a connection per caller, and past thirty-two callers it walks through the ceiling the others are sitting under. At sixty-four connections the per-operation work finally shows up, and twice: the socket transport does about a fifth more queries a second than the driver while spending a sixth less of the machine. pgo is the argument in one row - it gets slower above thirty-two callers while taking half the machine, because reading a header and then a payload per message is CPU that is not there to spend once connections stop being the constraint. Two things this found about measuring rather than about minato. Scheduler busy has to be a delta across the run; read straight it is cumulative since the node booted and says two percent no matter what is happening. And two callers sharing an epgsql connection interleave their parse and bind on the unnamed statement, which the server answers with 26000, so each caller here takes one and keeps it - and taking it has to be a single atomic operation, because a read after an add is two and the race was the first thing this measured.
🟡 Code Coverage — 88.6%1709 of 1928 lines covered. ✅ ELP LintNo diagnostics. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
bench/minato_saturation.erland the section ofbench/README.mdthatsays what the per-operation numbers are worth.
Why
Every benchmark in here measured one operation at a time. That answers what a
query costs; it does not answer whether the cost mattered. The last two rounds
of work (#42, #43) halved minato's share of a query and moved the wall clock by
about a percent, and I could not say from the existing benchmarks whether that
was a good trade or a wasted afternoon.
What it measures
A fixed pool, a growing number of callers, and a count of what comes back —
plus scheduler busy as a delta, which is the number that says whether the BEAM
was the constraint at all.
Sixteen connections, which is a normal pool:
Everything with a pool stops at the same number with four percent of the machine
in use, and the client that is faster per operation is not faster. The first
ceiling anybody meets is the size of their pool. epgsql is the tell: no pool,
so it gets a connection per caller, and past 32 callers it walks straight
through the ceiling the others are sitting under.
Sixty-four connections, so the pool is not the answer:
Now it shows up, and twice: the socket transport does about a fifth more queries
a second than the driver while spending a sixth less of the machine.
pgoisthe whole argument in one row — it gets slower above 32 callers while taking
half the machine, because a header read and then a payload read per message is
CPU that is not there to spend once connections stop being the constraint.
So: per-operation cost buys throughput, not latency, and only after the pool
is no longer the thing in the way.
Two things it found about measuring
cumulative since the node booted and reports about two percent whatever is
happening — which is what I first reported, and it was meaningless.
the unnamed statement and the server answers
26000. Each caller takes oneand keeps it, and taking it has to be a single atomic operation: a read after
an add is two, and that race was the first thing this benchmark measured.