Skip to content

test: measure throughput, not just per-operation cost - #44

Merged
Taure merged 1 commit into
mainfrom
bench/saturation
Aug 10, 2026
Merged

test: measure throughput, not just per-operation cost#44
Taure merged 1 commit into
mainfrom
bench/saturation

Conversation

@Taure

@Taure Taure commented Aug 10, 2026

Copy link
Copy Markdown
Owner

Adds bench/minato_saturation.erl and the section of bench/README.md that
says what the per-operation numbers are worth.

Why

Every benchmark in here measured one operation at a time. That answers what a
query costs; it does not answer whether the cost mattered. The last two rounds
of work (#42, #43) halved minato's share of a query and moved the wall clock by
about a percent, and I could not say from the existing benchmarks whether that
was a good trade or a wasted afternoon.

What it measures

A fixed pool, a growing number of callers, and a count of what comes back —
plus scheduler busy as a delta, which is the number that says whether the BEAM
was the constraint at all.

Sixteen connections, which is a normal pool:

client 1 8 32 64 busy % at 64
minato (inet) 2476 12372 16591 16362 3.5
minato (socket) 2446 11322 16104 16212 3.4
pgo 1624 11229 15223 15048 4.9
epgsql 1344 6446 12359 23588 7.0

Everything with a pool stops at the same number with four percent of the machine
in use, and the client that is faster per operation is not faster. The first
ceiling anybody meets is the size of their pool.
epgsql is the tell: no pool,
so it gets a connection per caller, and past 32 callers it walks straight
through the ceiling the others are sitting under.

Sixty-four connections, so the pool is not the answer:

client 1 8 32 64 busy % at 64
minato (inet) 2407 12960 23176 32610 7.8
minato (socket) 2619 11977 24517 39263 6.6
pgo 1486 11624 19312 13374 45.1
epgsql 1516 6568 12414 24008 7.0

Now it shows up, and twice: the socket transport does about a fifth more queries
a second than the driver while spending a sixth less of the machine. pgo is
the whole argument in one row — it gets slower above 32 callers while taking
half the machine, because a header read and then a payload read per message is
CPU that is not there to spend once connections stop being the constraint.

So: per-operation cost buys throughput, not latency, and only after the pool
is no longer the thing in the way.

Two things it found about measuring

  • Scheduler busy has to be a delta across the run. Read straight it is
    cumulative since the node booted and reports about two percent whatever is
    happening — which is what I first reported, and it was meaningless.
  • Two callers sharing an epgsql connection interleave their parse and bind on
    the unnamed statement and the server answers 26000. Each caller takes one
    and keeps it, and taking it has to be a single atomic operation: a read after
    an add is two, and that race was the first thing this benchmark measured.

…e wrong question

Every benchmark here measured one operation at a time. A client that costs less
per operation only goes faster if the client was what the operations were
waiting for, and against a database on the same machine it is not: the last two
rounds of work halved the client's share of a query and moved the clock by about
one percent.

`minato_saturation` points a growing number of callers at a fixed pool and
counts what comes back. Reading it in the order it prints says two things that
the per-operation tables cannot.

At sixteen connections every pooled client stops at about sixteen thousand
queries a second with four percent of the machine in use, and the client that is
faster per operation is not faster. The first ceiling anybody meets is the size
of their pool. epgsql is the tell: it has no pool so it gets a connection per
caller, and past thirty-two callers it walks through the ceiling the others are
sitting under.

At sixty-four connections the per-operation work finally shows up, and twice:
the socket transport does about a fifth more queries a second than the driver
while spending a sixth less of the machine. pgo is the argument in one row - it
gets slower above thirty-two callers while taking half the machine, because
reading a header and then a payload per message is CPU that is not there to
spend once connections stop being the constraint.

Two things this found about measuring rather than about minato. Scheduler busy
has to be a delta across the run; read straight it is cumulative since the node
booted and says two percent no matter what is happening. And two callers sharing
an epgsql connection interleave their parse and bind on the unnamed statement,
which the server answers with 26000, so each caller here takes one and keeps it
- and taking it has to be a single atomic operation, because a read after an add
is two and the race was the first thing this measured.
@github-actions

Copy link
Copy Markdown

🟡 Code Coverage — 88.6%

1709 of 1928 lines covered.


✅ ELP Lint

No diagnostics.

@Taure
Taure merged commit ab0c75f into main Aug 10, 2026
19 checks passed
@Taure
Taure deleted the bench/saturation branch August 10, 2026 18:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant