feat: a socket-module transport, opt in - #43
Merged
Conversation
A cached statement is described once and run many times, and the shape of its rows cannot change while it lives: the same fields, in the same order, with the same formats. Every execute still built the decoder for them from scratch, and built it twice - once for the framer and once for the caller's own decode - which is 64 codec resolutions a query for a 64 column row that answered the same way every time. The statement now carries the decoder for the default decoding options, made when it is described, and an execute takes it. Options that differ from the defaults still build one, because they describe different work. Bind was recomputing the parameter formats from their OIDs on every execute too. They belong to the statement for the same reason its result formats do. The four command tags every workload ends on are matched whole instead of being split into a list, lowercased into a copy and looked up in the atom table, and CommandComplete reads its tag without a search for the null terminator it knows the position of. Per operation, against the same PostgreSQL 17 (reductions, 2000 iterations): wide_row 2559 -> 1335, cached 518 -> 434, insert 477 -> 426, simple 471 -> 439. wide_row was behind epgsql and is now ahead of it.
Typed as `binary() | undefined` because the record had no default, which eqwalize reads as a name that may be built from an atom. Every connection is made by `new/3` and every `new/3` sets it.
A client benchmark against a database on the same machine measures a round trip, and the client's own share of it is small enough that a busy machine moves it further than a real change does. Two runs of the same suite an hour apart disagreed by forty percent on rows that had not been touched. `minato_profile:cost/0` runs the same workloads and reports total system reductions and scheduler run time per operation instead. Reductions do not move with the machine, so a five percent difference is a difference. `minato_profile:eprof/1` profiles one workload when the question is which function. The Python clients could not run here at all: Debian and Ubuntu ship a python3 whose venv cannot bootstrap pip without a package that needs root, so the runner silently had no asyncpg or psycopg3 row. `bench/py/install.sh` uses uv when it is there and falls back to venv, and the runner prefers the venv it makes. psycopg spells the database `dbname`, which is why it never connected even when it was installed. The report now has all seven clients, and a second table for what the work costs rather than what the wait costs.
Not the default, and not merged on its own: this is the measurement the
argument needed, in a form that can be run.
The per-operation budget of a cached query is 434 reductions, and 323 of them
are one gen_tcp send and one gen_tcp recv. prim_inet monitors the port,
serialises a reference, hands the bytes over and waits for a reply message. The
same round trip through the socket NIFs is 97.
The pool pays it twice over. A driver port has a connected process and only
that process may use it, so lending a connection out and taking it back is two
controlling_process/2 calls: 389 reductions a query, more than the query. A
socket handle has an owner only for how long it lives - any process holding one
can read and write - so moving it is a NIF call.
Against PostgreSQL 17, reductions per operation:
inet socket
direct 443.7 339.5 -23%
pooled 1023.2 534.8 -48%
rows_1000 20182.8 20110.2 unchanged, it is decode work
`transport => socket` picks it. gen_tcp stays the default and TLS is refused
rather than silently downgraded, because ssl wants an inet socket. `{active,
once}` is refused too, so a listener cannot run on it yet: socket has no active
mode, it has nowait and a select token, which is a different shape and belongs
with the listener rather than in a compatibility shim.
A trap worth writing down: a socket is closed when the process that opened it
terminates. The first cut of controlling_process/2 was a no-op on the argument
that ownership does not gate reads and writes, which is true and beside the
point - the pool's connector opens the connection and then exits, so every
pooled connection died on handover. 57 disconnects for 50 queries, and a pooled
query six times more expensive than through the driver rather than half.
minato_socket_SUITE proves the transport is invisible from above - same
queries, same errors leaving a connection usable, same transactions, same pool
- and proves the two refusals are refusals.
The spike left two things on the driver. Both are done, so `transport =>
socket` is now the whole client rather than the plain-TCP half of it.
TLS turned out to be almost nothing. `ssl:connect/3` takes a socket handle
directly, so the SSLRequest exchange only had to stop naming gen_tcp: the
transport sends the request, reads the one byte answer and hands the socket to
ssl, which gives back an ordinary sslsocket. Everything below TLS is then ssl's
business as before, including the certificate hash SCRAM binds to - a
connection with `channel_binding => require` negotiates SCRAM-SHA-256-PLUS over
TLS 1.3 on a socket handle, which is the case the suite runs.
LISTEN was the real work. socket has no active mode; it has a read that either
hands back what has arrived or registers the caller for one
`{'$socket', Socket, select, Handle}` message. That is the same arrangement
`{active, once}` is - exactly one message, and the owner asks again - so
`activate/1` is now that read, and the select message is answered by
`handle_message/2` in the process that made the call. No reader process between
the socket and the connection, which would have contradicted the one thing
`m:minato_conn` says about itself.
`activate/1` returns the connection now rather than `ok`, because what it is
waiting for is part of the connection: `deactivate/1` has to cancel the wait
before the connection can be read again, and cancelling needs the token. Bytes
are not lost either way - a cancelled wait drops the message, not the socket's
own buffer.
Windows completes reads rather than selecting on them and that shape is not
implemented, so `activate/1` refuses it. Everything else works on either.
The numbers are unchanged by all of this: 433.8 -> 339.8 reductions on a query,
1022.7 -> 532.4 on a pooled one.
Two eqWAlizer escapes, both named where they are: nothing expresses that the
transport field decides what the socket field holds, and ssl:connect/3 is
specified for the sockets that existed before it learned to take a handle.
🟡 Code Coverage — 88.6%1708 of 1928 lines covered. ✅ ELP LintNo diagnostics. |
POSIX says a socket is readable and leaves the bytes in it. Windows completes the read and puts the bytes in the message. `activate/1` refused the second shape, which made `LISTEN` over this transport a POSIX arrangement. Both shapes are handled now. `t:minato_socket:waiting/0` is a select token or a completion token, and which one a machine hands back is that machine's business rather than the caller's: `handle_message/2` re-reads on a select and takes the bytes out of a completion. Cancelling is where the difference bites. Dropping a select loses nothing, because the bytes are still in the socket. Dropping a completion that already fired would lose them, so `cancel/2` answers `pending` when it was too late, and `deactivate/1` then waits for the message it knows is coming and puts the bytes in the buffer. Everything the mailbox holds is absorbed the same way rather than discarded. None of this is testable on a Linux runner, so it runs on Windows: a second workflow starts the PostgreSQL the runner already has and runs eunit and the whole common test suite on it. Two changes so that a green Windows run cannot be an empty one. The socket suite asks whether a server is there over the *driver* transport, so a skip means "no PostgreSQL" and can never mean "the socket transport could not connect". And the notification test asserts which kind of message woke it - `select` on POSIX, `completion` on Windows - so a run that took the other path fails rather than passing without touching this one.
…verywhere Linux reports the socket closed, Windows reports the connection aborted. The test asserted the word rather than the outcome, so it failed on the first Windows run of a suite that had never had one.
Two things the first Windows run found, neither of them the completion path it was added for. Windows will not connect a socket that was never bound, and says `not_bound` rather than binding it: every socket-transport connection failed. Binding to the wildcard first is what the driver does everywhere and costs nothing where it was not needed. The differential harness compared NaN encodings byte for byte. IEEE 754 says nothing about the payload a NaN gets when one is made from the string "NaN", so glibc writing 7fc00000 and Windows writing 7fffffff are both right and both mean NaN. The harness now tells that apart from a real disagreement by decoding both: every NaN decodes to the atom, so bytes that differ and values that do not is the payload and nothing else. What minato itself writes is still asserted exactly - it writes the canonical quiet NaN - and what the server writes is asserted to be a quiet NaN rather than to be a particular one.
…ther The span test waited for the next [minato, query, start] and called it its own. A pool opens connections when it needs them and the first connection of one asks the server how many it will allow, which is a query, and a span. On a slower machine that span got there first. It now waits for the query it ran. The TLS case over the socket transport read the test CA before deciding whether there was a server to use it with, so a machine without the compose service that generates the certificates got an enoent instead of a skip.
The start event was filtered but the stop was still whichever arrived first, so the pool's own first-connection query answered for the test's on a machine slow enough to interleave them. A span carries a context through both halves, which is the thing that says they are the same span.
A server refusing the startup exchange writes why and then the backend exits.
On Windows a socket closed with anything still unread is reset rather than shut
down, and a reset discards what the far end already received, so the reason can
be gone before it is read: the same connect answers {socket, closed} there and
{pgsql_error, 3D000} on Linux.
The test asserted the message. It asserts the refusal now, and accepts a socket
failure only on the machine that does this. The connection is refused either
way and the server's log has the reason, which is now written down in
minato_conn rather than left for somebody to find at three in the morning.
Eleven of the twelve socket transport cases ran on Windows and the twelfth skipped, because the TLS server on Linux comes from the compose file and there is no compose here. The one server it does have now has a certificate and serves both, which is what PostgreSQL does anyway: TLS is the same port.
…ves Linux Turning TLS on for the single server broke the case that asserts a server declining TLS is refused, which needs a server that declines. The runner's own server stays plain and a second cluster serves TLS on 55433, so the suites see the topology they were written against instead of being edited to fit a runner.
The run stopped finishing when the runner gained a TLS server, which is also when two sets of cases stopped skipping. Bounded steps say which.
The folded scalar handed rebar3 a single suite path with spaces in it.
The compose file leaves both the CA and the server certificate in test/certs and the TLS suite reads the second one to check the hash SCRAM binds to. Only the CA was landing there.
The split was for finding out which suite stopped finishing. It was the TLS one, reading a certificate that was being written somewhere else, and the suites after it wore the delay. A bound on the step is what was worth keeping.
Owner
Author
Windows
The completion path is what the job exists for. POSIX says a socket is readable Cancelling is where the difference bites: dropping a select loses nothing, What running it foundEverything below was found by the first Windows runs, and only one of them was
None of these were caused by the socket transport. They were assumptions about
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows #42.
transport => socketruns minato over thesocketNIFs instead ofthe inet driver. gen_tcp stays the default.
Why
The per-operation budget of a
cachedquery is 434 reductions and 323 of themare one send and one recv through the driver: it monitors the port, serialises a
reference, hands the bytes over and waits for a reply message. The same round
trip through
socketis 97.The pool pays it again. A driver port has a connected process and only that
process may use it, so lending a connection out and taking it back is two
controlling_process/2calls: 389 reductions a query, more than the query. Asocket handle has an owner only for how long it lives, so moving it is a NIF
call.
Numbers
Against PostgreSQL 17, reductions per operation:
The pooled row is the one that matters: it is the path kura and shigoto take.
Everything, not half of it
The first cut left TLS and
LISTENon the driver. Both are done.TLS was almost nothing:
ssl:connect/3takes a socket handle directly, sothe SSLRequest exchange only had to stop naming gen_tcp. What comes back is an
ordinary
sslsocket, so the certificate hash SCRAM binds to is the same hash —the suite connects with
channel_binding => requireand negotiatesSCRAM-SHA-256-PLUS over TLS 1.3 on a socket handle.
LISTENwas the real work.sockethas no active mode; it has a read thateither hands back what arrived or registers the caller for one
{'$socket', Socket, select, Handle}message. That is what{active, once}is,so
activate/1is now that read and the select message is answered in theprocess that made the call — no reader process between the socket and the
connection, which would have contradicted the one thing
minato_connsays aboutitself.
activate/1returns the connection rather thanok, because what it is waitingfor is part of the connection and
deactivate/1needs the token to cancel.What it will not do
Windows completes reads rather than selecting on them, and that shape is not
implemented —
activate/1refuses it, soLISTENover this transport is POSIXonly. Everything else works on either.
The trap, for the next person
A socket is closed when the process that opened it terminates. Ownership does
not gate reads and writes — any process holding the handle can use it — so the
first
controlling_process/2was a no-op. The pool's connector opens theconnection and then exits, so every pooled connection died on handover: 57
disconnects for 50 queries, and a pooled query six times more expensive rather
than half. It reads as a flaky pool, not as a broken transport.
Checks
minato_socket_SUITE(12 cases) holds the transport to the same behaviour fromabove — queries, errors leaving a connection usable, transactions, the pool
lending and taking back, a borrower dying — plus TLS and a notification arriving
through a select, end to end through a real listener.
fmt, xref, dialyzer,
elp eqwalize-all,elp lint, ex_doc, eunit (1398), ct(153) all green.
Two eqWAlizer escapes, each named at the line: nothing expresses that the
transport field decides what the socket field holds, and
ssl:connect/3isspecified for the sockets that existed before it learned to take a handle.