feat: metrics for the tasks of background workers - #2637
Open
nicolas-grekas wants to merge 33 commits into
Open
nicolas-grekas wants to merge 33 commits into
nicolas-grekas wants to merge 33 commits into
Conversation
Member
|
Didn't GitHub introduce support for stacked PRs lately? I think that it would greatly help reviewing those |
Contributor
Author
|
They do exist since July, but a stack needs all its branches in the same repository, cross-fork stacks aren't supported, and I can't push branches to php/frankenphp. If a maintainer pushes the four branches here (or gives me push rights), I'll recreate these as a proper stack, one diff per layer. Until then each PR says which commit to review, and the Commits tab shows that one alone. |
nicolas-grekas
force-pushed
the
bgworker-metrics
branch
11 times, most recently
from
September 14, 2026 06:16
de9d9a3 to
2514be8
Compare
nicolas-grekas
force-pushed
the
bgworker-metrics
branch
4 times, most recently
from
September 18, 2026 09:24
3972b24 to
b45bf77
Compare
nicolas-grekas
force-pushed
the
bgworker-metrics
branch
7 times, most recently
from
September 18, 2026 16:58
2a08254 to
dd71a7c
Compare
… validateId() PHP 8.6 deprecates passing a save handler that has neither, see the RFC deprecations_php_8_6. The notices are printed before the handler is installed, which sends the headers, so session_set_save_handler() then refuses to change session.save_handler and the worker session tests fail on nothing but the deprecation.
nicolas-grekas
force-pushed
the
bgworker-metrics
branch
from
September 18, 2026 20:59
dd71a7c to
b1c5bed
Compare
FRANKENPHP_WORKER and FRANKENPHP_WORKER_BACKGROUND say what is running the script, so FrankenPHP owns them. Dropping the background flag from the worker's own env was not enough: $_SERVER is built from the process environment first, then the php_server env, then the worker's, so a value from any layer below survived and an HTTP worker answered the documented isset() check. They are now removed after all the layers are merged, for the kind of thread that must not carry them. The test declares the flag in the worker env, the server env and the process environment.
…olds Two small ones on the restart path. The quadratic backoff multiplied before capping, which overflows a duration of nanoseconds past some 300k consecutive failures and sleeps for a negative one; a background worker crashing past its ready point counts without bound and reaches that in a few days of retries. The cap now comes first, for the same schedule. Closing the Go side of a handle only lands as EOF on the script's end while no other process holds a copy, and a pcntl_fork() child inherits every descriptor of the process, the pairs of the other threads included. Shutting the write direction down first sends the FIN regardless, so a parked script still wakes up instead of waiting out the force-kill. Also drops a platform conditional: php_network.h maps closesocket to close outside Windows.
An explicit max_threads gets the reservation added on top, so the HTTP capacity it describes is preserved. The automatic limit did not: it resolved from a num_threads that already included the background threads, so declaring three of them turned a limit of 4 into 10, and a memory-derived estimate could be swallowed whole, leaving no room for HTTP autoscaling at all. The main thread now knows what part of its count is reserved, resolves the estimate on the rest, and adds the reservation back, like the explicit path.
resetForReboot() was redundant, setupWorkerScript() already resets the request count before every run, so the lifecycle interface is down to the one step that differs. A worker always has a server now, so the extension dispatch has a single path. zend_alter_ini_entry_chars() takes the literal, and php_network.h maps closesocket to close outside Windows. The ready_workers help text and docs said fetching the handle marks a background worker ready; the first wait on its stream does, as the validation test asserts. A parked fixture no longer disables max_execution_time itself, which is how the engine disabling it went untested, and a new test parks past a one-second limit with max_input_time set, the case where php_execute_script() re-arms it.
Two paths the suite took for granted. A worker parked on its handle must survive default_socket_timeout as well as max_execution_time, so the fixture that disables neither now runs with both set to one second. And a run gets one handle: the second fetch is the same stream, a fetch after closing it is a fresh one, and the drain still reaches the script through that one.
The directive, the Go option and the docs section carry the flag, the function did not.
…d workers Readiness rode on intercepting every PHP path that waits on the handle: the read op, the select cast, the transport receive missed at first, and whatever a future PHP adds. The contract was also read three different ways during review. It is now a call the script makes, the background analog of frankenphp_handle_request(): the first frankenphp_worker_tick() of a run marks the worker ready, every call returns false once the worker is drained, and the read, cast and transport hooks are gone. The tick never blocks and never hands out work. It consumes whatever the runtime wrote on the handle to wake the script up, so the script only ever selects on the handle, alone or with its own streams, and the protocol on it stays private.
…ched FRANKENPHP_WORKER stays what it always was, "1" in HTTP workers, and is not set in background workers, where FRANKENPHP_WORKER_BACKGROUND holds the declared name instead. A script serving both roles tests which of the two is set. The removal of inherited values is gone with the change that motivated it: nothing about HTTP workers moves in this PR anymore.
A script that registers its handle with an event loop and runs it only ticks when the handle is readable, so it never became ready before the drain and Init() waited for it. One wake-up written at run setup makes such a loop tick on its own: readiness then means the loop serviced the handle once. The first frankenphp_worker_tick() consumes it, and a script parked in a blocking read without ticking now fails its boot fast instead of hanging the start.
The shared lifecycle keeps its struct, embedded by both worker handlers; the interface that named the one step they supply is gone, that step is a parameter.
The channel was set to nil once Init() had decided, read without synchronization by the handlers, and the send blocked on a buffer sized to the thread count. A background worker can tick, exit and fail its next boot while Init() finishes, which an HTTP worker cannot since it blocks once ready: that exit could race the nil write, or block its thread on a full buffer. The channel now stays, an atomic startup flag gates the sends, and the send never blocks.
Until its first frankenphp_worker_tick(), a run is under the limit like any request: a setup that outlives it ends as a boot failure, with the backoff and the cap. The first tick disarms the timer, and nothing re-arms it past that point, so the loop has no time limit, like the CLI. This replaces the per-request ini override, which exempted the bootstrap too.
Cancelling the handle's watcher only removes that one callback, and run() keeps going while any other referenced watcher exists. A drained worker has to leave its loop, which is the driver's stop().
… fire The limit is PHP's: on Windows CI the busy bootstrap ran its full five seconds without the timer ending it, so the test now runs only with the Zend max execution timers of ZTS builds on Linux, where it passes.
A background worker serves no requests, so it does not scale with the CPUs like an HTTP one and nearly every declaration wrote "num 1". It is now the default, and declaring "background" is enough; a pool still asks for the threads it wants.
The missing stop socket of a background worker is an invariant, not a runtime error: the pair is opened before the script starts and closed at the next run setup, so the two guards are asserts now. A thread reaching the ready callback without a background handler would wait out Init() silently, so it panics instead. The context of a background run is not a dummy request, and the field says so. The scope of a name collision is a local variable rather than a method, and the Caddyfile reference keeps the short version of the "background" line, the long one lives in the worker documentation.
A loop selecting on the handle blocks rather than spinning, because the tick consumed the wake-ups, the one sent at start included. The fixture polls the handle before and after a tick and the docs say so.
Two workers on one script, told apart by a matcher, are the documented way to give slow endpoints their own thread pool. The Caddy module used to make their generated names unique, this PR moved the collision check into the core and dropped that, so the configuration stopped booting. A name generated from the script path is not a declaration: it gets a numeric suffix, as before. A declared name still collides, which is what a background worker needs to keep its identity.
Packing the server into the worker name changed every label value of a php_server worker, which breaks the dashboards and alerts built on them. The two are separate labels now, worker="<name>" and server="<name>", empty for a global worker, so a query on the worker name alone selects that worker in every server and the values are the ones FrankenPHP always reported.
- pace a run that ends right after its ready point, clean or crashed, and cut the wait short on drain - route extension SendRequest() to its worker directly and make SendMessage() fail after Shutdown() - keep the declared path as the default name of a global Caddy worker - guard the background run context with contextMu - one drain owner on phpThread, one background TLS reset in C, the public read-timeout stream option - docs: metric labels, the platform condition of the bootstrap bound, stream_select() and FD_SETSIZE
A worker without a name is reported under the path of its script, which newWorker() resolved through symlinks. Deployments that publish releases behind a symlink then move every worker label at each deploy, since the resolved path names the release directory. The default is now the path as declared, made absolute, for the Caddy module and the Go API alike, so naming them in the module is no longer needed; it would also make two workers sharing a script collide, where an undeclared name gets a numeric suffix instead.
Every exit past the ready point was counted toward the restart backoff unless the run outlived its one second cap, so a worker processing a batch and returning, which is how a script keeps its memory fresh, was throttled to one run per second after four of them. A run now counts only when it ended too fast to have done anything, a tenth of a second, which is what tells a spinning script from a working one. A script returning at once is still paced the same way.
Splitting the identity of a worker into two labels changed the signature of every worker method of the Metrics interface, which an implementation living outside this repository has to follow. Those methods keep the single identifier they always took. One method carries the labels instead: DeclareWorker() names them once, before anything else mentions the worker, and the Prometheus implementation resolves the identifier through them. An outside implementation adds that method, empty when it has no use for the labels, and keeps the rest untouched. The identifier is the qualified name again, so two workers that would report under one are rejected at startup, as before.
… interface DeclareWorker() restored the signatures of the worker methods but still added a method to Metrics, which an implementation outside this repository has to grow before it compiles again. Metrics is now the interface it was, a worker scoped to a server being reported as "<server name>:<name>" as before. An implementation that also satisfies ServerMetrics receives the two names apart, which is what PrometheusMetrics does to label its series; the runtime picks the right shape once, in WithMetrics().
frankenphp_get_worker_handle() and frankenphp_worker_tick() are gone, replaced by FrankenPHP\WorkerHandle: tick() is the ready point and the liveness check, getStream() the stream to wait on, isValid() whether the run still holds its socket. One object instead of two global functions, and a place for the task API to land. Only waiting on the stream is supported, what it carries is not part of the contract and tick() consumes it, so the descriptor stays out of the contract. On PHP 8.6 the class can implement Io\Poll\Handle without moving anything else, which is what the polling API discussion asked for. A run still has one stream whatever the number of handles, so a script may take one wherever it needs it.
The shared-state half of php#2287, on top of the background workers: a worker publishes a snapshot with frankenphp_set_vars(), requests and other workers read it with frankenphp_get_vars(). The persistent-zval toolkit from php#2366 does the cross-thread copies; this adds the two functions and a per-worker slot. set_vars() validates the tree, persists it and swaps it into the slot under a write lock; readers copy it into request memory under the read lock, so the previous table is only freed once no reader is on it. The slot belongs to the worker rather than a thread: it survives script restarts, serving the last snapshot meanwhile, and several threads of one worker simply publish last-writer-wins. The tables are freed in drainPHPThreads() once every PHP thread is gone and before the engine is, since freeing walks string headers. get_vars() resolves the name the way requests do, within the caller's server then among global workers. It blocks until the worker reached its ready point once: activateServers() runs after initWorkers(), so requests never wait, and a blocked caller is another background worker still booting. Those waits form a graph and a cycle is refused with an exception instead of deadlocking Init(); the wait also aborts on shutdown. A ready worker that never published throws. Publishing before the first wait on the handle therefore guarantees the snapshot exists before the server accepts requests. Being the first consumer keeping persistent trees across requests and exposing them repeatedly, this also fixes two fast paths of the toolkit: opcache-immutable arrays were exposed through refcounted zvals, and opcache only keeps their refcount at 2, so the second reader's release destroyed shared memory; and every interned string was shared by pointer, while only permanent ones (opcache, startup) outlive the request that interned them, so trees built from request-interned literals dangled once that request ended (the Windows job runs the embed without opcache). Immutable arrays now go through zvals without type flags, as php-src does for literals, and sharing a string requires IS_STR_PERMANENT. Left out on purpose, see php#2287: the per-request cache with === identity, the unchanged-data skip in set_vars(), ensure_background_worker() and lazy or catch-all workers, CLI hiding of the functions.
The task half of php#2319, on top of the background workers and their shared vars: a request, an HTTP worker or another background worker hands work to a named background worker with frankenphp_send_task(), which returns a stream carrying the updates the worker sends back with frankenphp_update_task(). The worker dequeues tasks with frankenphp_receive_task() once frankenphp_worker_tick() returned: the one handle of php#2617 carries both the drain EOF and the wake-ups, so a script keeps a single stream_select() loop, and the tick consumes what the runtime wrote on it. A wake-up is not a count: a pool wakes one thread per task and the others get null, and in a pool it may belong to a task a sibling took. The tick also parks the thread for the senders, or makes the coming wait return at once when tasks are already queued. send_task() blocks until a thread of the worker picks the task up and throws on timeout, so a busy worker pushes back on its senders instead of queueing without bounds; tasks queued while a thread restarts are signaled again on its next run. The wait also ends when the sender's own thread is drained for a restart or the shutdown, since the target's threads are drained too. Names resolve like frankenphp_get_vars() does. Each task gets a socket pair. The sender's stream is a socket stream over one end, one byte per update and EOF at completion, so stream_select() bounds the wait or multiplexes tasks, and a blocking read parks as well; closing it abandons the task. The receiver's stream is a socket stream over the other end: updates go through update_task(), the stream itself reports the sender's close as EOF to stream_select() and feof(), so a long task learns that nobody waits for its result, and update_task() throws. Closing it completes the task, unless the close is the resource cleanup of request shutdown, which means the script ended with the task open: the sender's next read throws instead of returning null. Sixteen updates are buffered per task, past that update_task() waits for the sender to read. Waking threads is what a task costs, so wake-ups are kept to a minimum. A send wakes one parked thread of the worker, round-robin, with the line on its handle; a thread that reads its handle while tasks are queued gets the line from the read op itself, so no wake-up is lost whichever loop shape the script uses, and after 10ms without pickup every thread is woken as a fallback. The sender waits for the pickup in the kernel, on its end of the task's pair, rather than in a Go select: waking a PHP thread parked inside a cgo callback costs Go a P hand-off, a byte on a socket does not. The thread taking the task writes that byte, a watcher goroutine does when the wait must end without a pickup. The queue mutex is never held across a syscall and taken once per wake-up, as a thread inside a cgo callback that loses it parks the same expensive way. In the Docker builder image this takes a task from 549 to 285us with one thread, a pool of 8 from 1745 to 320us, and 8 senders on 8 threads from 2k to 32k tasks/s. Each side of a task waits on its own descriptor of the task's channel, an eventfd on Linux, one end of a socket pair elsewhere for Windows's php_select(): the streams carry no data, they are what stream_select() waits on and what fclose() ends, the Go side holds the state the functions report. The descriptors belong to the task until both sides closed, then the pair is drained and pooled, so a task costs no socketpair, fcntl or close: about 12 syscalls instead of 18, 13% off the latency and up to a third more throughput under load in the same measurement. Payloads and updates follow the set_vars() whitelist and travel as persistent tables through the Go side, which owns them until they are copied into request memory. The streams reference their task through a cgo handle; the task is freed once both sides closed, or by the sender when no thread picked it up. The stop sockets of a worker's threads are now guarded by its task queue mutex, since senders write to them. Two more savings on the wake-ups. The sender polls a first 10ms slice on its own: only a pickup that outlasts it brings the Go side in, to wake every thread of the worker and to start the goroutine that ends the wait on a drain or the shutdown, so the common case, a pickup within microseconds, spawns no goroutine and wakes no M. And a side's descriptor gets one signal per sleep: an event signals it only while the other side sleeps on it with no signal outstanding, that one is consumed on the next event and the rest is read from the task's state, so a completion behind an update, or a burst of updates, costs no syscall. Together they take a task from 24 to 15 syscalls and its futexes from 4.7 to 0.6: 15% off a plain task, 29% off one carrying 16 updates, and 5 to 15% more throughput under load. Compared to php#2319: no queue ahead of pickup and no cancellation before it, no dedicated signaling stream, no global task table.
The existing worker metrics apply to background workers as they are:
busy_workers counts a thread holding a task, from pickup to the close of
the task's stream, and worker_queue_depth counts the tasks waiting for a
thread, which is the only queue a background worker has. Two new ones
break tasks down: worker_task_count{worker,outcome} with completed,
aborted (the script ended with the task open), abandoned (the sender
closed its stream first) or timeout (no thread picked the task up in
time), settled by whichever side closes first so every task counts once,
and worker_task_time, the seconds spent on tasks from pickup to close.
The threads endpoint follows: a background thread is busy while it holds
a task and waiting otherwise, counted per thread since a script may hold
several.
nicolas-grekas
force-pushed
the
bgworker-metrics
branch
from
September 19, 2026 10:28
b1c5bed to
7ffc2b5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #2617, #2635 and #2636, themselves on #2658: review the last commit (
7ffc2b5).Metrics for the tasks of #2636, reusing what exists where the meaning carries over.
frankenphp_busy_workersnow counts a background worker thread holding a task, from pickup to the close of the task's stream, andfrankenphp_worker_queue_depthcounts the tasks waiting for a thread, the only queue a background worker has; their help texts say so. Two counters break tasks down:frankenphp_worker_task_count{worker,outcome}withcompleted,aborted(the script ended with the task open),abandoned(the sender closed its stream first) andtimeout(no thread picked the task up in time), settled by whichever side closes first so every task counts exactly once, andfrankenphp_worker_task_time{worker}, the seconds spent on tasks. The/frankenphp/threadsendpoint follows: a background thread is busy while it holds a task, waiting otherwise.The
ServerMetricsinterface of #2617 gainsStartWorkerTask,StopWorkerTaskandWorkerTaskOutcome. Tested through a Prometheus registry passed withWithMetrics(): one task of each outcome, then the gauges back to zero.