Relay is an event-driven function runner. It consumes events from a Redis Stream, matches each event against declarative patterns, selects the matching handlers, executes them with managed runtimes in isolated containers, and acknowledges successfully processed messages.
Relay does not know or care where events originate. It only reads from Redis Streams and runs functions:
Event Producer → Redis Streams → Relay → Function Handler
At startup Relay discovers the functions under /functions (one directory
per function), builds one image per function, and creates (if missing) the Redis
consumer group. It then watches /functions for changes and reconciles each
function on the fly. It then blocks
on the stream with a consumer group, decodes each message, and for every event:
- evaluates every function's rules (declarative patterns in
template.yaml), - executes each matching handler sequentially in an isolated container,
- acknowledges the message (XACK) only after every matching invocation is terminal — it either succeeded or was exhausted and routed to the DLQ.
Cron schedules (schedules in template.yaml) ride the same stream: every
worker evaluates the cron locally, but the due occurrence is published to the
stream exactly once cluster-wide (atomic publish-if-new), and the consumer
group delivers that single entry to one worker for execution — see
Schedules below.
Relay is distributed as one binary. relay start runs the long-running
process — it consumes events, loads functions, builds images, and reconciles
/functions live, blocking in the foreground until signalled. The remaining
subcommands are administrative/inspection commands around the same binary;
they never start the runtime. Their writes are the local secrets store
(relay secret set/rm), the DLQ stream (relay dlq rm, and relay dlq replay
on success), and — read-only otherwise — the state database they read from.
relay start # start Relay in the foreground
relay health # check Relay dependencies (Redis, Docker)
relay stats # show current operational statistics
relay stats reset # reset persisted cumulative statistics
relay function ls # list functions
relay function inspect <name>
relay function invoke <name> --event '{...}' # run matching handlers on the live worker
relay dlq ls # list dead-lettered entries
relay dlq inspect <id> # show one entry's metadata and original event
relay dlq replay <id> # re-run one entry's exact handler on the live worker
relay dlq rm <id> # delete one dead-lettered entry
relay secret ls|set|rm # manage local secrets
relay git keygen # generate an SSH deploy key
relay git set <repository> # set the git source to sync from (optionally --webhook-secret)
relay git sync # manually sync the source into /functions
relay git status # show git sync state
relay git remove # forget the source and drop the checkout
# build the single binary
go build -o relay ./cmd
# or, from the module root with defaults
go build ./...
# run Relay in the foreground (reads REDIS_* from the environment)
./relay startRelay runs in the foreground by design: relay start blocks until the
process is interrupted (SIGINT/SIGTERM). It does not background itself, write a
PID file, or fork. Background execution and supervision belong to Docker,
systemd, Kubernetes, etc. — e.g. docker compose up -d, docker run -d ..., or
systemctl start relay — not to Relay itself.
Relay talks to the Docker Engine API directly (via the moby client) to
build images and run containers; it does not require the Docker CLI to be
installed. A running Docker daemon must be reachable on the host. At startup
Relay pings the daemon and fails fast with a clear error if it cannot connect.
Image builds and handler invocations use the local Docker daemon on the
host that runs Relay. The client is configured from the environment
(DOCKER_HOST, DOCKER_TLS_VERIFY, DOCKER_CERT_PATH) and negotiates the API
version automatically.
The daemon must be permitted to execute arbitrary containers, so Relay needs
full host-level Docker permission (e.g. the user running Relay must be a member
of the docker group or otherwise have access to the Docker socket).
When Relay itself runs inside a container, do not mount the host's Docker
socket into Relay. Instead, run Relay against a Docker-socket proxy that
allow-lists the Engine API endpoints Relay actually uses
(DOCKER_HOST=tcp://socket-proxy:2375 over an internal network):
services:
socket-proxy:
image: tecnativa/docker-socket-proxy
environment:
- PING=1
- VERSION=1
- BUILD=1
- CONTAINERS=1
- POST=1
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
relay:
image: relay:latest
depends_on:
- socket-proxy
environment:
- DOCKER_HOST=tcp://socket-proxy:2375
- REDIS_URI=redis:6379Security warning: the Docker socket is privileged. The socket is mounted read-only into a dedicated proxy container, and the proxy forwards only a curated allow-list of Docker Engine API requests (an endpoint allow-list plus read-only access) instead of granting Relay the full socket. This reduces Relay's attack surface against the daemon, but it is not a strong security boundary and it does not make Docker execution unprivileged: Relay still builds images and creates/runs containers, which is effectively root-equivalent on the host. This tradeoff is accepted at this stage of the project so Relay can build and run function images against the host daemon. Separating the executor into a remote/privileged sidecar or a properly isolated build+run service is out of scope for this iteration.
compose.dev.yaml runs the Relay runtime in a container (the image's CMD is
relay start) alongside a Redis service and a Docker-socket proxy, so
you can develop against the same containerized deployment the README above
describes without installing Go or Redis locally:
docker compose -f compose.dev.yaml up --build -d
docker compose -f compose.dev.yaml logs -f relayRelay does not mount the Docker socket. Instead the topology is:
Relay → socket-proxy → Docker daemon
Relay reaches the host daemon through the proxy over the internal compose
network with DOCKER_HOST=tcp://socket-proxy:2375 (Relay's moby client honors
DOCKER_HOST via FromEnv). The socket itself is mounted read-only and only
into the socket-proxy container, which forwards just the Engine API
endpoints Relay needs (ping/version, image build, and container create/attach/
start/wait/kill/remove). Port 2375 is not exposed to the host, so the
proxy is reachable only from the compose network. ./examples/functions is still
mounted read-only into Relay at /functions. The socket mount is privileged (see the security
warning above); this is a dev-only convenience. The Relay container's healthcheck
runs relay health (see Health check), so docker compose ps reports it
healthy only while both Redis and the Docker daemon are reachable. Tear down with
docker compose -f compose.dev.yaml down -v.
| Env var | Required | Description |
|---|---|---|
REDIS_URI |
yes | Redis address or DSN (see below). |
REDIS_STREAM |
yes | Redis stream to consume. |
REDIS_GROUP |
yes | Consumer group name. |
REDIS_STREAM_RETENTION |
no | Stream retention window; unset disables trimming. |
METRICS_ADDR |
no | Metrics HTTP listen address; unset disables Prometheus. |
GIT_WEBHOOK_ADDR |
no | GitHub webhook listen address; unset disables the webhook server (see Git). |
LOG_LEVEL |
no | Log verbosity: DEBUG, INFO, WARN, or ERROR (case-insensitive); default INFO. |
MAX_CONCURRENCY |
no | Max concurrent function invocations per worker; default 8. |
MAX_BUFFERED_EVENTS |
no | Max events read from Redis and held locally before completion; default 16. |
NETWORKS |
no | Comma-separated Docker networks every execution container joins at create time; unset = no extra networks (default bridge). |
WARM_CONTAINER_IDLE_TIMEOUT |
no | How long a healthy idle warm execution container is kept before eviction; Go duration, default 5m. |
TRAEFIK_NETWORK |
no | Docker network Traefik is attached to; required only when a service declares host. |
TRAEFIK_ENTRYPOINTS |
no | One or more comma-separated Traefik entrypoint names (e.g. websecure or web,websecure) for the router entrypoints label; unset = label omitted. |
TRAEFIK_CERTRESOLVER |
no | Traefik router tls/tls.certresolver labels on routed services; unset = omitted. |
TRAEFIK_PRIORITY |
no | Traefik router priority label on routed services; unset = omitted. Positive integer. |
TRAEFIK_HOST_OVERRIDE |
no | Replaces the declared host's domain for local/development routing while preserving its left-most label; e.g. issuer.example.com → issuer.localhost. Unset = declared host unchanged. |
The first three REDIS_* variables are required: Relay fails startup (exits
immediately) if any of them is unset or empty. REDIS_STREAM_RETENTION is
optional and enables internal stream retention (see below). METRICS_ADDR is
optional and opt-in: when set to a non-empty listen address it starts the
Prometheus HTTP endpoint on that address, and when unset or empty no HTTP
server is started. An unbindable address is logged and retried, never fatal.
GIT_WEBHOOK_ADDR is likewise opt-in: when set it starts the GitHub webhook
endpoint on that address (see Git), and when unset or empty the webhook
server is not started. Unlike the metrics server, a webhook bind failure (a
taken port) is fatal at startup.
WARM_CONTAINER_IDLE_TIMEOUT is optional and controls warm-container idle
eviction (see Execution container lifecycle): it takes a Go duration (for
example 90s, 5m, 1h) and defaults to 5m when unset or empty. A
malformed or non-positive duration is a configuration error that fails startup
(unlike REDIS_STREAM_RETENTION, which logs and disables).
NETWORKS is optional and sets the Docker networks every execution
container (one per event/schedule invocation) joins at create time (see Docker
networks). It is a comma-separated list: entries are trimmed of surrounding
whitespace, empty entries are ignored, duplicates are removed, and declaration
order is preserved. Every configured network is verified to exist at startup —
Relay never creates networks, and a missing one fails startup. Unset means no
extra networks (the default bridge behavior). Persistent service containers are
not attached to NETWORKS: service networking is owned by the routing layer
via TRAEFIK_NETWORK (see Routing).
MAX_CONCURRENCY and MAX_BUFFERED_EVENTS bound how much work a single Relay
worker keeps in flight or in its local buffer. Both default to a positive value
(8 and 16 respectively); a value of 0 is invalid (it does not mean
"unbounded") and a non-integer/negative/zero value is a configuration error
that fails startup. These limits are per worker: with N replica workers
in a cluster, the effective totals multiply (N * MAX_CONCURRENCY global
capacity, N * MAX_BUFFERED_EVENTS * ... local buffering), and the per-function
concurrency (see Template format) applies per function per worker.
Redis stream → bounded local buffer → matcher/dispatcher → global worker
concurrency → per-function concurrency → runner/container → completion/ACK
MAX_BUFFERED_EVENTSbounds the number of messages read from Redis and held locally (handed to a handler but not yet ACKed / DLQ'd / left-pending). When the buffer is full, the consumer stops reading until capacity is released, so the backlog stays in Redis (not in local memory) and backpressure flows naturally. The reclaimed-message path is also buffer-aware: a message skipped for capacity is simply retried next reclaim tick.MAX_CONCURRENCYbounds the total number of function invocations executing concurrently in one worker, and the per-functionconcurrencybounds how many concurrent invocations a single function's handlers may run in that worker. The effective limit is the intersection of both: a function's per-function limit ismin(concurrency, MAX_CONCURRENCY), and that same effective value sizes both its runner semaphore and its warm container pool (so the pool's capacity gauge,function inspectsnapshot, and admission all agree). A templateconcurrencyaboveMAX_CONCURRENCY(e.g.15withMAX_CONCURRENCY=8) is therefore capped at8.MAX_CONCURRENCYis startup configuration: a change requires a worker restart, unlike a hot-swapped templateconcurrency, which is re-clipped live. If a slot cannot be acquired within a bounded wait (well below the reclaimMinPendingIdle), the message is left pending and replayed by a later reclaim — so a locally buffered event never sits long enough to defeat the reclaim pacing, and no retry/exhaustion accounting is charged for a merely-blocked invocation.- A hot-swapped template's
concurrencychange is applied live, without a restart: a successful reconcile propagates the resolved bound to the function's runner semaphore (new acquisitions use the resized semaphore; an in-flight invocation releases to the semaphore it started on) and to its warm container pool. An increase admits more concurrent invocations and stays lazy (no container is started eagerly); a decrease immediately retires only excess idle containers and never interrupts a running invocation — containers returned to an over-capacity pool are retired instead of pooled, so the pool converges as the busy leases drain. Changes to a function that has never warmed take effect when its pool is first created.
LOG_LEVEL controls log verbosity. Valid values are DEBUG, INFO, WARN,
and ERROR, parsed case-insensitively with surrounding whitespace trimmed. The
default is INFO: only INFO, WARN and ERROR lines are shown unless DEBUG is
explicitly requested. There is no WARNING alias (use WARN). Only messages at
or above the configured level are emitted; each record is rendered in slog's
native text format, e.g. time=2026-09-13T22:47:41+02:00 level=INFO msg="Loaded 3 function(s)". An invalid value is a configuration error: Relay fails startup
with a message naming the accepted values.
DEBUG— detailed internal execution flow: Redis reads/claims and internal retry decisions, function matching, container/image lifecycle details, periodic metrics snapshots, cleanup and orphan-sweep details, and other high-volume diagnostic lines that would be too noisy for production.INFO— expected high-level lifecycle: Relay starting/stopping, worker and metrics server started/stopped, functions discovered/updated/removed, function executions completed, successful reconciliation, and other major state changes an operator normally wants to see.WARN— unexpected but recoverable situations: retryable failures, transient Redis/Docker/network errors Relay continues through, failed cleanup that may be retried, and degraded-but-operational behavior.ERROR— a failed operation that materially affects the work: function execution ultimately failed, DLQ write failed, handler panics, unrecoverable processing failures, and failures returned upward that abort the workflow.
Fatal startup failures (missing required variables, invalid LOG_LEVEL, an
unbindable metrics address, an unreachable Redis or Docker daemon) always log
and exit regardless of the configured level.
LOG_LEVEL governs only Relay's own operational logs. Function container
stdout/stderr forwarding is NOT routed through the logger, so it is unaffected
by LOG_LEVEL and streams at every level (see Handler contract); the level
filters only Relay's diagnostic and lifecycle lines.
REDIS_STREAM_RETENTION is an optional duration (e.g. 6h) that enables
internal, periodic trimming of the configured REDIS_STREAM. Relay runs the
retention job internally — there is no external cronjob and no per-message
timer. While relay start runs, a single goroutine with a periodic
time.Ticker trims the stream with XTRIM <stream> MINID ~ <cutoff-id>, where
cutoff-id is <unix-milliseconds>-0 for now - retention. One initial trim
runs shortly after startup so an already-large stream does not wait a full
interval.
The tick interval is derived automatically from the retention window
(retention / 24, clamped to [1m, 1h]) — it is not another environment
variable. For 6h that is 15 minutes. Trim failures are logged and retried on
the next tick; they never stop the worker.
The trim is approximate (~): Redis removes whole internal stream nodes
(listpack blocks of up to stream-node-max-entries, default 100), so entries
older than the window that share a node with fresh entries are removed on a
later pass rather than immediately. Repeated ticks make progress toward the
cutoff one node at a time; an entry may linger at most about one tick interval
plus one node past its expiry.
Warning: retention applies to the WHOLE stream, not just Relay's consumer group. Other consumer groups on the same stream may lose unprocessed entries older than the window. Fan-out is preserved for entries inside the window.
Unset or empty REDIS_STREAM_RETENTION disables retention entirely (no
goroutine, no trims). A malformed duration or a zero/negative value fails
startup like any other configuration error.
REDIS_URI accepts either a plain address or a Redis DSN:
host:port(e.g.redis:6379)redis://user:password@host:portrediss://user:password@host:port(TLS)
DOCKER_HOST (and the other Docker client variables DOCKER_TLS_VERIFY,
DOCKER_CERT_PATH) are consumed by Relay through the Docker client at startup
(see Docker requirement); Relay itself does not parse them.
relay health is an operational/container healthcheck command. It checks the
two dependencies the runtime needs at startup — Redis connectivity (a PING to
REDIS_URI) and Docker daemon connectivity (an Engine API Ping) — and exits
0 when both are reachable, 1 otherwise (reporting the first failing check to
stderr). It is not a public API and no HTTP server runs; it only creates
clients and pings, so it never starts consumption, loads functions, builds
images, or touches the state database. compose.dev.yaml uses it as the Relay
container's healthcheck.
Relay's reliability settings — retry/delivery limits and the recovery loop — are fixed internals, not env-configurable. See Reliability defaults below.
Multiple Relay instances may share the same REDIS_GROUP with different
consumer names to scale out consuming; each worker uses its hostname as its
consumer name automatically (the container ID / pod name under
Docker/Kubernetes), so replicas are distinct without any configuration. The
consumer group is created automatically (with MKSTREAM) if the stream or
group does not exist; the group is created at position 0, so only messages
added after startup are consumed.
Each direct subdirectory of /functions is one function. The directory name
is the function name. Function names must be valid: they must match
[a-z0-9][a-z0-9._-]*, be at most 63 characters, and not end in a dot (so
user-events, welcome_email, and jobs.v2 are fine, while User Events,
hello/world, and .hidden are not). Validation happens at load time — a name
is never silently sanitized — so valid names are already safe to use as docker
image tags. Each function directory must contain a template.yaml that declares
which runtime to use and which events it handles.
- A directory without a
template.yamlis ignored. - An invalid name is logged and skipped — it never prevents Relay from starting.
- An invalid
template.yamlis logged and skipped — it never prevents Relay from starting. - A function whose image cannot be built is logged and marked unavailable; the other functions continue to be served.
Function images are versioned by source fingerprint. Each function's
selected source is hashed (SHA-256 over file paths + bytes) and the image is
tagged relay-fn-<name>:<first-16-hex-of-fingerprint>; the full 64-hex
fingerprint stays authoritative in the local state database and on the prepared
function. template.yaml is hashed verbatim: any template or source edit
changes the tag and rebuilds.
Selection is governed by .gitignore rules (the same policy git uses): a source
file matched by an applicable rule is not source, so its bytes never enter the
fingerprint and never enter the image. The applicable .gitignore files
themselves are hashed, so editing a rule changes the fingerprint even when no
included file changed — a rule edit can change the source set, and must therefore
gate a rebuild. The same selection drives materialization, the build context, and
the fingerprint, so all three always agree on which files are source.
- A rebuild produces a new immutable image version; an existing image for the exact fingerprint is reused without rebuilding.
- Relay swaps to the new version only after preparation succeeds — the old version keeps serving until then, and a failed build leaves the old version active.
- Old Relay-owned images are removed only once they are no longer in use (in-flight executions are protected), including on function removal and at startup.
- Relay manages only its own
relay-fn-*images — it never prunes globally or touches other apps' images or layers.
Dependency layers (requirements.txt / pyproject.toml + uv.lock /
package-lock.json / package.json) are installed once into reusable
relay-dep-* images, fingerprinted by:
- runtime + architecture + manifest contents, plus the runtime's pinned external install tool (e.g. uv), whose version shapes the installed payload.
Function images build FROM those layers, so only dependency or source changes
rebuild the top layers; a changed manifest yields a new relay-dep-* tag, an
unchanged one is reused across every version of a function (and across functions
with identical dependency sets). Dependency images are content-addressed and,
being shared bases, are not auto-pruned by the startup sweep — a removed
function image never removes a layer another function may still need. The
fingerprint keys on the base image tag (e.g. python:3.14-slim), not its
digest, so a newer pull of the same tag reuses the cached relay-dep-* image —
operators wanting a refresh must remove those images (a future digest-pinning
feature is the proper fix).
Managed images carry Relay-ownership labels: relay.type=function|dependency
(classifies a function vs a dependency image), relay.function,
relay.runtime, relay.fingerprint, and relay.dependency (a function image's
relay.dependency names the exact relay-dep-* image it was built FROM).
Labels — not repository names — are the source of truth for ownership:
dependency images are garbage-collected lifecycle-driven, once at worker
startup (after the boot sweep removed superseded function images) and once after
each successful function-image removal — never periodically, never forced, and
only when no managed function image references them. An image with no
relay.type label (e.g. a pre-label dependency image) is never touched.
Each function keeps a bounded warm pool of reused execution containers, up
to its effective concurrency — min(template concurrency, MAX_CONCURRENCY) —
per image version. Containers are created lazily: the first invocation starts
one, and additional containers are started only when concurrent demand requires
them, up to that effective limit. A template asking for more than the
worker-global cap (e.g. concurrency: 15 with MAX_CONCURRENCY=8) therefore
warms, reports, and admits only the cap's worth.
Subsequent invocations reuse healthy idle containers instead of paying container startup on every event. Concurrent invocations of the same function lease distinct containers; each individual container still processes one invocation at a time.
Because the interpreter process persists, function code must not assume process-global state is fresh per invocation. Module-level state may survive between invocations, and per-invocation environment values are applied to the long-running process for each request.
A container is returned to the idle pool only while it remains healthy. Timeouts, process exits, protocol errors, image changes, and shutdown invalidate the container and it is discarded instead of reused.
A container's version is its image. On an image change, the old version enters a draining state. Idle old-version containers are discarded immediately, while busy containers are allowed to finish their current invocation and are discarded when released. No new invocation is leased to a draining version, and all new invocations use the current version.
Idle containers are not kept forever. A healthy container is evicted once it
has remained idle longer than WARM_CONTAINER_IDLE_TIMEOUT; eviction never
interrupts a busy invocation, and later demand simply starts a container
lazily. WARM_CONTAINER_IDLE_TIMEOUT is global and defaults to 5m. It
accepts Go duration syntax such as 90s, 10m, or 1h. An unset or empty
value uses the default; malformed or non-positive values are configuration
errors and fail startup.
When a function is removed, no new containers are created for it. Idle containers are discarded immediately, busy containers are allowed to finish their current invocation and are discarded on release, and the function's pool state is removed once fully drained. If the function is later recreated — even with the same content and image — a successful prepare re-activates it, so its new containers warm normally rather than being treated as permanently retired.
Every function execution container is created with Docker AutoRemove, so the daemon removes the container once its process exits. Relay relies on AutoRemove for normal process-exit cleanup and does not explicitly remove containers that exit on their own.
Explicit removal is used only as a backstop for abnormal lifecycle paths where a container may still be running or may never have started correctly, such as start failures, timeouts, cancellation, or wait errors. Backstop removal is idempotent: a container that was already removed, or is already being removed by the daemon, is treated as successfully cleaned up.
Execution containers carry diagnostic-only Docker labels. Because a reused
container outlives individual invocations, per-invocation labels such as
relay.handler, relay.message_id, relay.event_id, and relay.event_name
cannot represent the currently running invocation and remain empty at container
creation. Stable container-level labels such as relay.type,
relay.function, relay.hostname, and relay.image identify ownership and
image generation.
At startup, before functions begin serving invocations, Relay runs a conservative orphan sweep bounded to 30 seconds. It removes only stale Relay containers whose ownership labels match the current worker hostname. Containers belonging to other workers and non-Relay containers are never touched, and Relay performs no global Docker pruning.
Every execution container is hardened: it runs as a non-root user (uid 10001),
is limited to 128 MiB memory / 1 CPU / 128 PIDs, drops all Linux capabilities,
has a read-only root filesystem with a bounded /tmp tmpfs, and keeps outbound
networking enabled.
runtime: python3.14
concurrency: 2
events:
- handler: events.created.handler
pattern:
event_name: [INSERT]
table_name: [users]
- handler: events.updated.handler
pattern:
event_name: [MODIFY]
table_name: [users]
timeout: 20s
- handler: events.deleted.handler
pattern:
event_name: [REMOVE]
table_name: [users]
schedules:
- handler: jobs.cleanup.handler
cron: "0 3 * * *"
- handler: jobs.report.handler
cron: "0 8 * * 1-5"
timezone: "Europe/Rome"
timeout: 20s
services:
- entrypoint: service.js
port: 3000
replicas: 2runtimeselects the execution runtime. Onlypython3.14andnode24are supported; any other value fails validation. It is required whenever Relay must launch work through a runtime: anyevents/schedulesentry, or any service using anentrypointsource. A services-only template whose services all usebuildorimagesources needs noruntime. Onnode24a handler may be JavaScript or TypeScript (see TypeScript handlers); thehandlersyntax is identical for both.eventsis a list of rules. Each rule has a requiredhandler(of the formmodule.function), a requiredpattern, and optionaltimeoutandretries.schedules(optional) is a list of cron-triggered handlers; each entry requireshandler(module.function) andcron.cronis a cron expression in either the standard 5-field formminute hour day-of-month month day-of-weekor the 6-field (seconds) formsecond minute hour day-of-month month day-of-week. The exact expression is shown byrelay function inspect, alongside a human-readable description in 24-hour time.timezone(optional) is an IANA timezone (e.g.Europe/Rome,America/Sao_Paulo) resolved with Go'stime.LoadLocation; omitted means UTC. Scheduling respects DST and offset changes of the configured zone. An invalid timezone or cron expression fails template validation (the function is logged and skipped).timeout(optional, per schedule) follows exactly the same rules as event rules (default 6s, max 5m).- Scheduled handlers receive a deterministic payload
{"source":"relay.schedule","scheduled_at":"<RFC3339 UTC instant>"}on stdin (same handler contract as events: RELAY_HANDLER + JSON on stdin, exit code decides). concurrency(optional, top-level) bounds how many of this function's handler invocations may execute concurrently within a single Relay worker (per function, per worker). It is additionally clipped to the worker-globalMAX_CONCURRENCY, so the effective limit ismin(concurrency, MAX_CONCURRENCY)(e.g.15with the defaultMAX_CONCURRENCY=8is capped at8). It must be a positive integer; a zero, negative, or non-integer value (e.g.0,-1,1.5,true) fails the function's template validation (the function is logged and skipped). Omitted templates use a2default. A hot-swapped change is applied live, without a restart (see Concurrency and backpressure); aMAX_CONCURRENCYchange requires a worker restart.timeout(optional, per rule) is a Go duration string bounding a single invocation of that rule's handler (e.g.20s,1m30s). It must be positive. Zero, negative, unparseable, or values above5m(MaxTimeout) fail the function's template validation (the function is logged and skipped). Omitted rules use a6sdefault.retries(optional, per rule) is the number of additional executions attempted after the initial one (0= only the initial attempt). It must be a non-negative integer; a negative or non-integer value (e.g.-1,abc,1.5) fails template validation. Omitted rules use a4default, so a failing invocation is attempted1 + 4 = 5times in total before it is considered exhausted and the message is routed to the DLQ.handleris split at the last dot:events.created.handler→ moduleevents.created, functionhandler. Handlers may live in nested modules (for example theevents/package), not only in top-level files. On thenode24runtime the module may be a.js/.mjsfile or a.ts/.mtsfile (see TypeScript handlers); the handler string never carries an extension.services(optional) is a list of persistent long-running HTTP services (see Services below). Each entry declares exactly one source —entrypoint(an application entrypoint file, not themodule.functionform),build(a Dockerfile path relative to the function directory), orimage(an external image reference) — plus optionalport(default80,1–65535) andreplicas(default1, positive integer). The configured source is the service identity. The template example above shows a service alongside events and schedules.
Cron schedules reuse the exact event execution path: a scheduled invocation runs through the same runtime image lifecycle, secrets, timeout cap, per-function and global concurrency slots, and handler metrics as any event-driven invocation.
gocron (every worker) → atomic publish-if-new → Redis Stream
→ existing consumer group → one worker → runner → function container
Distributed coordination. Every Relay worker evaluates the function's configured cron schedules locally, but a worker does not execute the handler itself. Instead, when gocron determines an occurrence is due, the worker attempts an atomic publish-if-new into the same Relay event stream — a single Lua script checks a dedup key (with a 7-day TTL) and writes the stream entry in one atomic step, so there is never a window where the dedup key exists without its stream entry. Exactly one worker wins and publishes one stream entry per logical occurrence; every other worker's simultaneous evaluation of the same tick is a clean no-op.
An occurrence's identity is deterministic — schedule:<function>:<handler>:<scheduled_at RFC3339 UTC> —
derived from the absolute instant (normalized to UTC), never from a timezone
representation. The configured timezone affects when the schedule fires,
never the identity, so DST and offset changes cannot split or merge
occurrences. Dedup keys live under relay:schedule:<occurrence_id>, are
history only, and expire solely by TTL — they are never deleted when the
handler completes, so a worker whose cron callback runs slightly later cannot
re-publish an occurrence the fleet already completed.
Once the stream entry exists, it is an ordinary Relay stream message and
follows the full consumer-group model: one worker receives it, the PEL and
XAUTOCLAIM recovery hand it to another worker if that one crashes, and
retries/exhaustion route failures to the DLQ like any other message.
The guarantee is therefore:
One schedule occurrence is published once cluster-wide, while handler execution remains at-least-once — exactly-once handler execution is not claimed (a crash between a handler's side effect and its completion re-runs the handler, so handlers must stay idempotent).
Publication is best-effort across the fleet: every worker evaluates the cron
independently, so a publish failure on one worker only loses that worker's
tick — other workers still publish the same occurrence. Scheduled handlers do
not advance the event-classification counters (events_received_total,
events_matched_total, events_unmatched_total) or
function_events_matched_total: schedule occurrences bypass event matching
entirely, so they have no matched/unmatched class. Schedule coordination has its
own counters (see Observability).
Adding, changing, or removing a function's schedules (or handler/cron/ timezone/timeout) converges live through the reconciler: the worker's cron jobs are replaced in place, so future occurrences use the current definition. Already-published occurrences are not purged from Redis — those entries were valid when published and expire via the dedup-key TTL / stream retention. An occurrence that is still pending when its function or schedule handler is removed is treated as obsolete: it is acknowledged (terminal) rather than retried forever or dead-lettered, because its removal was an intentional configuration change.
A template may define per-function environment variables and secret references:
runtime: python3.14
env:
API_URL: https://api.example.com
secrets:
DATABASE_URL: database-url
events:
- handler: events.created.handler
pattern:
event_name: [INSERT]env(optional) maps an env-var name to a literal string value, injected into every execution container at runtime. Values are literal — never masked, never treated as secret-looking. Empty values are allowed (flag-like variables). Env-var names must match[A-Za-z_][A-Za-z0-9_]*.secrets(optional) maps an env-var name to a secret reference name. The reference is resolved to a value immediately before each execution and injected into the container. Secret references must be valid secret names (lowercase letters, digits,.,_,-; no leading or trailing.; at most 63 chars).- A variable may not be defined in both
envandsecrets. RELAY_HANDLERis reserved by Relay (it carries the rule's handler identity); a template may not set it.- Never put secret VALUES in
template.yaml. The template holds only the reference name; the value lives in the secrets store (seerelay secretbelow). Editingtemplate.yaml(including env values) changes the function's fingerprint and triggers a rebuild by design; rotating a secret value never changes the fingerprint and never requires a rebuild or restart. - Env values and secret references are injected at runtime only — they are never baked into the function image (template.yaml is excluded from the build context), never stored in the state database, and never logged.
A pattern is a tree of field conditions:
- A plain YAML list is implicit equality:
status: [COMPLETED, FAILED]. - A map with only operator keys (
equals,prefix,suffix,exists,gt/gte/lt/lte) holds operators. - A nested map without operator keys holds nested field conditions.
| Operator | Semantics |
|---|---|
equals |
Value equals any of the listed values (type-preserving). |
prefix |
String value starts with any of the listed prefixes. |
suffix |
String value ends with any of the listed suffixes. |
exists |
Key presence check. Takes a boolean, not a list. |
gt |
Value is greater than a numeric threshold or now()-relative cutoff. |
gte |
Value is greater than or equal (numeric or now() cutoff). |
lt |
Value is less than a numeric threshold or now()-relative cutoff. |
lte |
Value is less than or equal (numeric or now() cutoff). |
gt, gte, lt, and lte take either a number (compared numerically) or
a now()-relative expression (compared as instants). Operands may be a
list, in which case the value matches if any operand holds (ordinary OR).
pattern:
new_image:
created_at:
gt: "now()-5m"now()expressions:now(),now()-5m,now()+10m,now()-1h,now()+24h, and any Gotime.ParseDurationsuffix (compound forms likenow()-1h30mare fine). They are evaluated as a UTC instant at match time (never at template load), so the cutoff moves as time passes.now()+0s(zero offset) is valid.- Only the exact
now()/now()±durationsyntax triggers temporal comparison. The event value must be an RFC3339 string; offsets are respected and values are compared as instants, not lexicographically. A numeric operand stays purely numeric. - The template pattern never accepts an arbitrary timestamp string (e.g.
gt: "2026-09-12T10:00:00Z"); onlynow()-syntax is valid for string operands, and anything else fails template validation rather than silently never matching. The old barenow/now-5msyntax is not accepted and is a validation error; rewrite it asnow()/now()-5m. - A missing,
null, non-string, or invalid-RFC3339 event value never matches a temporal comparison (and never panics).
Examples:
pattern:
price:
lte: 100.5 # numeric: value <= 100.5
age:
gt: 18 # numeric: value > 18
created_at:
gt: "now()-5m" # temporal: value after (now() - 5m)
updated_at:
gte: "now()-1h"
lt: "now()" # two operators on one field are OR, not a rangeexists checks key presence only — the value is irrelevant. null still
counts as "exists":
pattern:
new_image:
name:
exists: true # new_image.name must be present (any value, incl. null)pattern:
new_image:
name:
exists: false # new_image.name must be absent from new_imageexists works recursively at any nesting depth: new_image: { exists: true }
checks the top-level new_image key; new_image: { name: { exists: true } }
checks name inside new_image. A nested exists: true fails when a parent is
missing (the nested key cannot be present). A nested exists: false matches
when the nested key is absent — including when the parent map itself is missing
(a missing parent means the nested key is necessarily absent).
The value must be a strict boolean; exists: "true", exists: 1, and
exists: null fail template validation.
- Values within one operator's list are OR.
- Different fields (sibling keys) are AND.
- Multiple operators on the same field are OR (alternatives).
- Each rule's pattern is evaluated independently; multiple rules may match the same event, and no deduplication is performed (two matching rules referencing the same handler are both invoked).
- A missing event field fails that condition.
- Extra event fields are ignored.
prefixandsuffixonly match string values; non-string values never match.existscomposes with other operators on the same field under the normal OR rule:name: { exists: false, prefix: ["12"] }matches whennameis absent or present with a value starting with12.
For example, a rule with status: [COMPLETED, FAILED] matches while
id: { prefix: ["user_"] } matches user_123 but not 123. The implemented
operators are equals, prefix, suffix, exists, and gt/gte/lt/lte.
The NETWORKS environment variable attaches every execution container (one
per event/schedule invocation) this worker creates to operator-provided Docker
networks. It is worker-wide configuration, not template configuration:
NETWORKS=backend,monitoring- Every execution container joins the listed networks at create time, via the
Docker Engine
NetworkingConfig. A worker withNETWORKSunset sends noNetworkingConfig— the default bridge behavior is unchanged. Warm containers are reused as-is; joining the networks happens only at container creation, so changingNETWORKSrequires a worker restart. - The value is a comma-separated list: entries are trimmed of surrounding whitespace, empty entries are ignored, duplicates are removed, and declaration order is preserved.
- The networks are infrastructure owned outside Relay: Relay never creates them. It verifies every configured network exists at startup, before any function is prepared or any container created; a missing network (or a daemon error) is a fatal startup error rather than a per-invocation skip.
- Persistent service containers are deliberately not attached to
NETWORKS: a service is reached through its routing layer, which owns the network it joins (TRAEFIK_NETWORK). An unrouted service joins no network on its own.TRAEFIK_NETWORKis independent ofNETWORKS. - Each service container carries a
relay.networkslabel recording its routing network, so the service reconciler can detect a routing-network change and replace the stale container.
A template's optional services list declares persistent long-running
containers: unlike events and schedules, a service is not an invocation
container that exits after one request. It is a long-lived process — an HTTP
server, for example — that Relay keeps running and reconciling continuously.
Each service declares exactly one source, and the configured source descriptor is the service's identity (it keys containers, routing, and the persisted service rows — there is no synthetic entrypoint):
runtime: node24
services:
- entrypoint: service.js # runtime-managed application entrypoint file
port: 3000
replicas: 2
- build: Dockerfile # a user-supplied Dockerfile (relative to the fn dir)
port: 8080
- image: ghcr.io/acme/api:1.2 # an external image reference
port: 9090
replicas: 3entrypointis the application entrypoint file the runtime starts as the long-lived process (service.json the Node runtime, or a nested file such asapp/main.pyfor Python). It is not themodule.functionevent-handler form, and it is not invoked per-request: HTTP requests are handled directly by the user's application inside the container. It must be a relative path inside the application directory (e.g.service.js,app/main.py) — no whitespace, no absolute paths, and no..path elements. Each runtime decides how the file is executed: Node runs it directly (node <entrypoint>within the application directory), while Python executes it as a module (python -m <module>— e.g.app/main.pyruns aspython -m app.main) so package-relative imports work. Anentrypointservice needs aruntime:(Relay launches it with a runtime-specific command).buildis a Dockerfile path relative to the function directory (e.g.Dockerfileordocker/Dockerfile.prod). Relay builds an image from the function's selected source (the same.gitignore-driven selection that fingerprints a function — ignored files never enter the image) using the Docker Engine API, not the Docker CLI. The image is content-addressed by a fingerprint that folds the Dockerfile identity into the function's source fingerprint, so abuildservice rebuilds when any selected file (the Dockerfile included) changes and reuses the existing local image otherwise. A build-service image lives in the function's ownrelay-fn-<name>repository, so the existing image-retirement machinery covers it. The image's ownENTRYPOINT/CMDare preserved — Relay does not override them. The path must be relative (no absolute paths, no.., no whitespace).imageis an external image reference (e.g.nginx:1.27,ghcr.io/acme/api@sha256:…). Relay inspects the local image and pulls from its registry when the image is missing locally or the hourly freshness window has elapsed (see External image freshness); the image's ownENTRYPOINT/CMDare preserved. Relay never removes external images — cleanup only ever touches Relay's ownrelay-fn-*/relay-dep-*namespaces. Abuildorimageservice does not need aruntime:.- A template whose only services use
build/imagesources needs noruntime:at all. A mixed template (events or schedules, or anyentrypointservice) still requires it, because those run through a runtime. An explicitly configured runtime is always validated, so a typo in an otherwise build/image-only template is still a parse error. port(optional) is the internal TCP port the service application listens on. It defaults to80and must be between1and65535. Relay injects it as thePORTenvironment variable (it cannot be overridden by template env or secrets), and exposes the port as container metadata only — no host port is published. When the service declares ahost, Traefik routes traffic to that port (see Traefik routing below).host(optional) is a hostname (e.g.api.example.com) exposing this service through Traefik. It is validated as a hostname at template parse time (empty or omitted = an internal unrouted service). At reconcile time a routed service requiresTRAEFIK_NETWORK: if the variable is unset Relay reportsservice "app/main.py": TRAEFIK_NETWORK is required when Traefik routing is configured, and if the configured network does not exist on the daemon it reportsservice "app/main.py": Traefik network "proxy" does not exist— Relay never creates the network. Changinghost(or the port orTRAEFIK_NETWORK) reconciles: the running container is replaced.path(optional) is a URL path prefix (e.g./v2) under which this service is exposed on itshost. It requires ahost; apathwithout one is rejected at parse time (service "service.js": path requires host), becausePathPrefixalone is not an externally addressable route. Empty or omitted preserves the exact host-only behavior. A configured path must start with/; whitespace, a query (?), a fragment (#), a backslash, or an empty (//) segment is rejected. It is canonicalized at parse time: trailing slashes are removed from non-root paths (/v2/→/v2), while/stays exactly/. Changingpathreconciles: the running container is replaced with updated routing labels.replicas(optional) is the desired replica count Relay maintains. It defaults to1and must be a positive integer. No autoscaling — the count is always exactly what the template declares.
All three source kinds share one cohesive service reconciler and lifecycle — there are no separate reconcilers, pollers, or caches per source. The only difference is how the desired image is resolved before convergence:
- an
entrypointservice runs the function image prepared exactly as for invocations, with its entrypoint overridden per container to the service entrypoint (e.g.node /app/service.js; Python overrides topython -m app.main); - a
buildservice runs a content-addressed image Relay builds from the user's Dockerfile over the selected source; - an
imageservice runs the external reference (inspected locally and pulled when due).
An entrypoint service keeps every version of a function in the function's own
image repository, so the existing image-retirement machinery covers it; a
build service image lives there too. External images are never Relay-cleaned.
At startup and on every reconcile of the owning function, Relay lists its service containers and converges them to the template:
desired replicas (template) vs actual Relay-owned service containers
The desired image is resolved before any container action: if a source cannot be resolved (a failed build or pull, a missing local image, an unlaunchable entrypoint, an unresolved secret), the pass reports the failure and preserves the service's existing healthy containers rather than tearing them down. A transient registry outage therefore never degrades a working service.
- Containers whose image (or, for an external tag, image content), port,
effective environment (
relay.env_hash), or routing labels no longer match the current version are replaced (stop + remove, then start fresh replicas). Containers whose image, port, and environment are unchanged are preserved — no unnecessary restarts. The environment comparison is what makes a changed templateenvvalue or a rotated secret value replace a service's container: the image reference and source fingerprint do not change for either, but a long-lived container would otherwise keep serving its old environment forever. A container created beforerelay.env_hashexisted carries no label and is replaced once. - Scaling up starts the missing replica slots; scaling down stops and removes exactly the excess containers (the lowest-numbered replicas are kept).
- A replica whose process exits (a crash) is detected by the same reconciliation — Relay recreates the missing slot on the next reconcile of the owning function, which includes the periodic pass (default every 30s) — so a crashed service self-heals without a tight restart loop and without any event-style retry/DLQ semantics.
- A service removed from the template, or its whole function removed, stops
and removes all of its containers. Relay then retires obsolete
relay-fn-<name>images — but only after no active container still references them, and never for images outside Relay's own namespace.
Image retirement is ordered after service convergence: on a rebuild the new
version is swapped in, schedules and persistent services converge to it, and
only then is the superseded image retired. Removal is additionally guarded — an
image is never removed while any Relay-owned (event, schedule, or service)
container references it via its relay.image label. When a rebuild's retire
would remove an image a service container still uses (e.g. a partial reconcile),
removal is skipped at debug level, retried with a short bounded backoff, and
finally deferred to a later natural cleanup pass (the next boot sweep, next
rebuild, or function removal). Image removal is never forced.
Containers are identified by deterministic Relay-owned labels
(relay.type=service, relay.function, relay.identity, plus the image, the
image content id, port, replica slot, and relay.env_hash), never by name alone.
relay.identity is the configured source descriptor (entrypoint file, Dockerfile
path, or image reference) — an honest identity for every source kind, never a
synthetic entrypoint. Service containers carry no
relay.handler label (that key identifies event/schedule handlers) — the source
IS the service. relay.env_hash is a one-way digest of the replica's effective
environment (see below); it carries no value. Generated container names
(relay-svc-<function>-<identity>…) are for human greppability only: like the
routing ids, they end in a 64-bit hash of the full identity so distinct
identities that sanitize or truncate to the same readable prefix still never
share a name. On graceful shutdown, Relay stops and removes the service
containers owned by that worker (scoped by relay.hostname, so other workers'
containers are untouched); stale containers left behind by a crashed Relay
process are swept at the next startup (per-function reconcile plus the startup
orphan sweep).
The environment each replica gets: the runtime's plan environment (e.g.
PYTHONDONTWRITEBYTECODE=1 for Python; empty for build/image sources), then the
template's env values, then resolved secrets values, then PORT. Containers
run under the same hardening as invocation containers: non-root user, dropped
capabilities, memory/CPU/pids limits, read-only rootfs, and a bounded /tmp.
External image freshness. For an image service, Relay checks its registry
at most once per hour per independent service (per function + identity). A
successful remote check is recorded in memory; while the window is open no
further remote check runs, and a running container is preserved. The window is
not persisted (a worker restart checks again), and changing the configured
source (a new identity) is checked immediately. A failed check does not
advance the window, so it is retried at the next reconcile — a transient
registry outage recovers promptly and never advances the clock. This is an
in-memory, per-worker policy: there is no separate poller, cache table, or
persisted timestamp.
relay function inspect shows the effective values (defaults included). A
build or image source is rendered with its kind prefix:
Services:
app/service.js port=3000 replicas=2
build:docker/Dockerfile.prod port=8080 replicas=1
image:ghcr.io/acme/api:1.2 port=9090 replicas=3
Out of scope for this first version: host port publishing, autoscaling, and request-level handler invocation. Routing is Traefik-only (see below).
A service that declares a host is routed through Traefik (operator-provided
infrastructure outside Relay). Relay attaches four labels to the service
container so Traefik picks it up from its Docker provider:
traefik.enable = true
traefik.docker.network = <TRAEFIK_NETWORK>
traefik.http.routers.<id>.rule = Host(`api.example.com`)
traefik.http.services.<id>.loadbalancer.server.port = <port>
Routed services also pick up the optional HTTPS labels, whenever the
corresponding value is set — nothing is defaulted (no implicit websecure,
letsencrypt, or fallback priority; Traefik's own default behavior applies
when a label is omitted):
traefik.http.routers.<id>.entrypoints = <TRAEFIK_ENTRYPOINTS> (only when set)
traefik.http.routers.<id>.tls = true (only when TRAEFIK_CERTRESOLVER is set)
traefik.http.routers.<id>.tls.certresolver = <TRAEFIK_CERTRESOLVER> (only when set)
traefik.http.routers.<id>.priority = <TRAEFIK_PRIORITY> (only when set)
So a routed service with
TRAEFIK_NETWORK=proxy TRAEFIK_ENTRYPOINTS=websecure TRAEFIK_CERTRESOLVER=letsencrypt TRAEFIK_PRIORITY=100
gets all eight labels; with network-only config it gets exactly the four above.
For local or development environments, TRAEFIK_HOST_OVERRIDE changes only the
effective host in the Traefik routing rule. For example, with
TRAEFIK_HOST_OVERRIDE=localhost, issuer.example.com routes as
issuer.localhost while api.example.com routes as
api.localhost. The template host is not modified. If a service declares a
path, the existing PathPrefix and StripPrefix behavior is unchanged.
When the service also declares a path, the router rule is constrained to that
prefix and a StripPrefix middleware is attached to the router, so the upstream
service receives the request as if the prefix were not part of it:
traefik.http.routers.<id>.rule = Host(`api.example.com`) && PathPrefix(`/v2`)
traefik.http.routers.<id>.middlewares = <middleware>
traefik.http.middlewares.<middleware>.stripprefix.prefixes = /v2
<middleware> is the deterministic <id base>-path-<hash> (Traefik-safe, capped
at 100 characters; see <id> below). It is distinct from <id>, so adding a
path never overwrites the router/service slices, and it is per-service (the id
already encodes function + service identity), so two services on the same host
with different paths get distinct router and middleware names. An empty/omitted
path adds nothing — the label set is byte-for-byte the host-only one.
<id>is a single deterministic Traefik-safe router/service id shared by the router and service slices: it is derived from the service identity (the function name + source descriptor — entrypoint file, Dockerfile path, or image reference — never the host or path) asrelay-<function>-<identity>-<hash>, lowercased, with every character outside[a-z0-9-]sanitized to-(identities likeapp/main.pyorghcr.io/acme/api:1.2contain/,., and:), consecutive-collapsed, and the readable part trimmed/capped at 100 characters. Because the id is deterministic, reconciliation produces stable labels.<hash>is a fixed 16-hex-character (64-bit) suffix hashed from the full, untruncated function name and identity, so distinct identities that sanitize to the same readable base (e.g.ghcr.io/acme/a/b:1andghcr.io/acme/a-b:1) or that would truncate to the same prefix still get distinct ids. The readable base is trimmed to make room, so the hash (and therefore collision resistance) is always preserved at the cap.traefik.docker.networktells Traefik which network the container routes on (theTRAEFIK_NETWORKDocker network). The container is created attached to that network. The network itself is owned outside Relay: Relay never creates it, and verifies it exists before starting routed containers — withTRAEFIK_NETWORK=proxyand no such network, the reconcile reportsservice "app/main.py": Traefik network "proxy" does not exist.- A service without
hostgets no Traefik labels at all and joins no extra network: it stays internal.TRAEFIK_ENTRYPOINTS,TRAEFIK_CERTRESOLVER, andTRAEFIK_PRIORITYonly affect routed services and always appear on the same<id>as the router/service slices. - Reconciliation is label-aware: changing
host,path,port, or any routing value (TRAEFIK_NETWORK,TRAEFIK_ENTRYPOINTS,TRAEFIK_CERTRESOLVER,TRAEFIK_PRIORITY) makes the running container stale and it is replaced with one carrying the updated routing labels. Removinghostreplaces the routed container with an internal (unlabeled) one; clearing an optional value (e.g. unsettingTRAEFIK_CERTRESOLVER) converges its labels away the same way.
| Runtime | Base image | Dependency handling |
|---|---|---|
python3.14 |
python:3.14-slim |
uv.lock + pyproject.toml → native uv project; else requirements.txt → uv pip install --system |
node24 |
node:24-alpine |
package-lock.json → npm ci --omit=dev; else package.json → npm install --omit=dev; else none |
Base images are fixed; arbitrary base images are not allowed. Dependencies are installed inside the image at build time, never on the host.
node24 accepts both JavaScript and TypeScript handlers — TypeScript is
transpiled and bundled by Relay at function build time with a pinned esbuild
into generated .mjs JavaScript that the unchanged Node bootstrap executes (see
TypeScript handlers below).
Python dependencies are installed with uv, never pip. The pinned uv binary
(ghcr.io/astral-sh/uv:0.12.17) is copied into every Python runtime image (even
one with no dependencies) by the generic Dockerfile renderer, and the dependency
image inherits it, so installs are fast and reproducible. Python has two
dependency manifests, resolved deterministically:
- Native uv project (
uv.lock+pyproject.toml): when a committeduv.lockis present it is authoritative andrequirements.txtis ignored. The locked set is installed without re-resolving —uv export --locked --no-dev --no-emit-projectthenuv pip install --system— so the image matches the lock exactly and the build fails if the lock is out of date. Always commituv.lock(runuv lock); Relay never generates or updates it. requirements.txt: the classic manifest remains fully supported and is installed withuv pip install --system --no-cache -r requirements.txt. It is still valid alongside an unrelatedpyproject.toml(e.g. tool configuration).- Incomplete native project: a
uv.lockwithoutpyproject.toml, or apyproject.tomlwithoutuv.lock, is an error (the function is skipped and logged) rather than a guess — Relay installs only from a committed lock or arequirements.txt.
Both paths install into the system site-packages (--system), not a
project-local .venv, so the existing python -u /relay/bootstrap.py entrypoint
sees the packages and the dependency image stays a reusable, source-independent
base.
For Node functions, if no package.json exists a minimal {"type":"module"}
package.json is injected so .js files are treated as ESM; if a user
package.json exists, its type field is respected.
Both runtimes support sync and async handlers. The Python bootstrap invokes
the handler and, if the result is awaitable, runs it with asyncio.run; the
Node bootstrap resolves the handler module by filesystem lookup (not by
attempting an import), imports it, and awaits a returned Promise.
Adding a future runtime (e.g. python3.15 or node26) requires only a new
entry in the runtime registry map; the engine is reused.
On node24, a handler module may be a TypeScript file instead of a JavaScript
one; the template syntax is identical and never carries an extension:
runtime: node24
events:
- handler: src.handler.handler # resolves to src/handler.ts
pattern:
type: [order.created]A module is resolved the same way the runtime bootstrap resolves it: the
JavaScript candidates base.mjs, base.js, base/index.mjs, base/index.js
are checked first (in that order). Only when none exists are the TypeScript
candidates base.mts, base.ts, base/index.mts, base/index.ts checked.
A module that resolves to both a JavaScript and a TypeScript source, or to
neither, fails the build with a clear error instead of failing later per
invocation.
- TypeScript is transpiled and bundled by Relay at function build time, never
per invocation. A pinned esbuild (
0.28.2) is installed into an ephemeral layer with the base image's ownnpm, run once for all TypeScript handlers, and removed again in the same image layer — the user'spackage.jsonnever needs esbuild and no build tooling reaches the execution layer. - The output is generated
.mjsJavaScript written beside the source inside the image (e.g.src/handler.ts→src/handler.mjs), which the existing Node bootstrap resolves first and executes unchanged. Your function directory is never modified (the build stages a temporary context). - Local
imports across.tsmodules work: the handler and its local module graph are bundled together. Bare package imports stay external and resolve at runtime from/app/node_modules, so dependencies continue to flow through the normal dependency-image path. - A
tsconfig.jsonpresent in the function directory is honored forcompilerOptions(e.g.strict,experimentalDecorators). Relay does not type-check: esbuild only strips types. Runtsc --noEmityourself in development or CI if you want type checking. - This is a handler-level transpile only for event/schedule handlers. A
service that needs a real frontend/build pipeline declares a
build:source with its own Dockerfile (see Services), which Relay builds from the selected source and runs with its own ENTRYPOINT.
Each execution runs a container with the environment variable
RELAY_HANDLER set to the rule's handler (e.g. events.created.handler) and
the event JSON written to the container's stdin. An embedded bootstrap resolves
the module and function from RELAY_HANDLER, reads the event from stdin, and
invokes the function. The container's exit code decides the result: 0 is
success, non-zero is failure. The container's stdout and stderr are forwarded
verbatim to Relay's process output as a raw transport while the handler runs —
one line per process line, each prefixed with the function/handler (and
message/event id when available), independent of LOG_LEVEL, with no severity
inferred from the stream.
- One image per function, never per handler or event. A function's single image is built at startup and, afterwards, rebuilt only when its directory changes (see Hot reload below); the rebuilt image serves all of its handlers.
- Sequential execution: for each event, functions are iterated in order, then rules in order, and each matching handler runs one at a time (no concurrency).
- Timeout: each invocation is bounded by the matching rule's
timeout(default6s). A timeout kills the invocation and is treated as an execution failure. Multiple matching rules each use their own rule's timeout. - Retries: each rule's
retries(default4) controls how many additional executions are attempted after the initial one. A failing invocation is retried with a per-invocation backoff (1m, 2m, 5m, then 10m capped) until its1 + retriesattempts are exhausted, at which point the message is routed to the DLQ (see Recovery and retries below). - Failure: any non-zero container exit is a failure; errors include the function and handler names. Every matching handler still runs: a failure in one invocation does not stop the remaining rules for that event. The message is not acknowledged and returns to the pending entries list until every matching invocation succeeds or is exhausted to the DLQ. See Acknowledgment semantics below.
For example, examples/functions/user-events-python/ declares three handlers
(events.created.handler, events.updated.handler,
events.deleted.handler), all served by the same function image.
Relay watches /functions (with fsnotify) and reconciles each function on the
fly, without a restart:
- Auto-discovery: a new directory under
/functionsis detected and its image built, then it starts matching events. - Per-function rebuild on change: edits to a function's template, source, or dependency files trigger a rebuild of that function's image only. Events are debounced (750ms) so a burst of editor saves coalesces into one rebuild.
- Fingerprinting: each function's content is hashed (
SHA-256over file paths + bytes); an unchanged function is skipped, so a rebuild happens only when its inputs actually changed. - Failure safety: if a rebuild fails (invalid template or failed image build), the previous, still-working version is retained and keeps serving events. It is retried on the next change or periodic pass.
- Removal: deleting a function's directory removes it from matching. In-flight invocations are never interrupted; they finish against the snapshot they started with.
- Missing template: a directory present but with no
template.yamlyet is treated as "not ready" — Relay waits for more events rather than dropping a previously-active function. - Periodic fallback: a 30s reconciliation pass re-scans
/functionsas a backstop for watch events that were missed. - Nested directories: the watcher covers files in nested subdirectories of a function, so sources split into packages are tracked too.
Functions are read-only to Relay (the directory is mounted read-only in the
container); all rebuilds happen in temporary build contexts, so Relay never
writes into /functions.
Relay keeps a small local SQLite database describing its current view of the
loaded functions — a read-only state view, not the source of truth. The
/functions directory remains authoritative; the local state database is
rebuilt automatically when empty and never drives matching, image building, or
reconciliation. It exists so operators can introspect what Relay has loaded and
how the last reconcile of each function went without touching Redis or Docker.
- Location:
/var/lib/relay/db.sqlite3(a fixed internal path, not env-configurable). The parent directory is created automatically, so the file also works for host-side runs. It is not external infrastructure — it is a local file you can volume-mount to persist across restarts.compose.dev.yamlmounts a named volumerelay-dataat/var/lib/relay. - Schema: a
functionstable (name,data,updated_at) wheredatais a single stored JSON snapshot of the whole function — runtime/status/image/ fingerprint/prepared_at/last-reconcile outcome, env/secret mappings, handlers (name/timeout/retries), schedules (handler/cron/timezone/ timeout/retries), and services (entrypoint/build/image/host/path/port/ replicas) — with only the stable name key and write timestamp kept as columns. There are no per-handler/per-schedule/per-service child tables, so a template change replaces one row atomically. A single-rowstatstable (id,updated_at, and a JSONdatapayload holding the current global operational counters plus backlog gauges), and afunction_statstable (function_name,updated_at, and a JSONdatapayload holding the per-function counters). Thedatapayloads are stored in SQLite's binary JSON (JSONB) format (jsonb(?)on write,json(data)on read). Only stable relational metadata is a column; the evolving payload is JSON so new instrumentation needs no schema change, and absent fields decode to zero. Secret references (never values) live inside the function snapshot. These are current snapshots only — no per-event rows, no metric history (Prometheus is the time-series source). - State model:
statusisready(an active version is built and serving) orpending(loaded but not yet built).last_reconcile_statusissuccess/failed(the last MEANINGFUL reconcile outcome; unchanged periodic checks are not recorded). A failed rebuild never marks a whole function unavailable: the previously active image and fingerprint are retained, so the last good version keeps serving whilelast reconcileshows the failure. All timestamps are RFC3339. - Fault-tolerance: state errors are logged and never fatal — Relay runs without the state database if the DB is missing or broken (Open recreates a missing DB).
- Stats data flow: operational stats accumulate in memory in the
Prometheus registry (the single source of truth);
GET /metricsreflects them immediately. SQLite receives the current absolute snapshot every 5 seconds (fixed, non-configurable) — snapshots, not history.relay statsreads the latest global snapshot (may lag live Prometheus by up to ~5s);relay function inspect <name>reads the latest persisted per-function stats. Graceful shutdown performs a final bounded flush; a hard crash may lose up to ~5s of telemetry. Redis event-processing correctness never depends on SQLite stats.
Relay persists state at startup and on every reconcile. Three read-only CLI
commands expose it (no Redis, Docker, or /functions needed — they read the
state database file only):
relay function lsNAME RUNTIME STATUS UPDATED
user-events-python python3.14 ready 12s ago
welcome-email-node node24 ready 12s ago
The UPDATED column is prepared_at (else updated_at) as a relative age
(12s ago, 3m ago, 2h ago, 5d ago), falling back to an absolute date
beyond ~30 days. Rows are sorted by name; fingerprints, images, errors, and
per-function workload counts are deliberately omitted from ls — detailed
workloads live in relay function inspect.
relay function inspect user-events-pythonName: user-events-python
Runtime: python3.14
Status: ready
Image: relay-fn-user-events-python
Fingerprint: <sha256>
Prepared: 2026-09-08T12:00:00Z (12s ago)
Last reconcile: success (12s ago)
Last error: <error>
Events:
events.created.handler timeout=6s
events.updated.handler timeout=20s
events.deleted.handler timeout=6s
Schedules:
jobs.cleanup.handler cron="0 3 * * *" (At 03:00) timezone=UTC
jobs.report.handler cron="0 8 * * 1-5" (At 08:00, Monday through Friday) timezone=Europe/Rome timeout=20s
The human-readable description in parentheses is display-only: it renders the
same cron in 24-hour time and never affects scheduling. If a description cannot
be generated, inspect falls back to printing just the raw cron="..."
expression (no parentheses) and never fails.
The Image, Fingerprint, and Prepared lines are omitted while a function is
pending (never built); the Last error line is omitted when there is none. A
failed reconcile with an active version keeps Status: ready and shows
Last reconcile: failed (...) — the function is never marked unavailable.
relay function inspect <name> also includes the current per-function
operational stats (zeros until the function has matched events):
Stats:
Events matched: 12493
Handler successes: 12470
Handler failures: 23
Retries: 17
DLQ entries: 2
relay function inspect <name> also shows the function's env and secret
mappings (from its template) when it defines any — literal env values and
secret references, never secret values:
Environment:
API_URL=https://api.example.com
Secrets:
DATABASE_URL=database-url
relay function inspect <name> appends a compact Runtime pool section for
the function. The cumulative acquire/discard counters always come from the
persisted per-function stats row under /var/lib/relay — never from a live
provider or the socket — so the CLI renders them identically whether or not a
worker is running. The authoritative live gauges are worker-local and are
fetched over a Unix socket (/run/relay/relay.sock) that the running worker
serves; when inspect runs in-process with a pool snapshot provider, it uses that
for the live gauges directly instead of a socket round trip. With a worker
reachable, inspect shows the live gauges:
Runtime pool:
Capacity: 4
Containers: 2
Busy: 1
Idle: 1
Starting: 1
Warm acquires: 7
Cold starts: 3
Discarded: 2
Capacity is the function's effective concurrency limit:
min(template concurrency, MAX_CONCURRENCY).
Containers is the number of currently running execution containers owned by
the function pool, while Busy and Idle describe their current lease state.
Starting is shown only while a lazy container start is in progress.
During image transitions, Busy may temporarily include retiring containers
from an older image generation, so the live container count can briefly exceed
the active generation's capacity.
Warm acquires, Cold starts, and Discarded mirror the corresponding
cumulative Prometheus counters, persisted in the per-function stats snapshot and
restored on worker restart so they stay monotonic.
When no worker is reachable (no socket, or the function has no live pool), the live gauges are rendered explicitly unavailable while the cumulative counters still come from the persisted stats snapshot — never a stale number:
Runtime pool:
Capacity: unknown
Containers: unknown
Busy: unknown
Idle: unknown
Warm acquires: 7
Cold starts: 3
Discarded: 2
Known live zero values render as 0; only an unavailable worker renders
unknown. The live gauges are deliberately not persisted to SQLite (a persisted
live gauge would go stale between flushes); only the cumulative counters are.
Starting is transient and is usually absent.
relay function invoke <name> runs a function's matching event handlers
synchronously on the running worker's live runtime pool, without publishing
anything to the event stream:
relay function invoke user-events-python --event '{"event_name":"INSERT"}'
relay function invoke user-events-python --file event.json
echo '{"event_name":"INSERT"}' | relay function invoke user-events-pythonThe event is supplied as inline JSON (--event), a JSON file (--file), or
piped on stdin when neither flag is given (the two flags are mutually
exclusive). It must be a JSON object, because event matching is defined over
an object's fields; arrays, scalars, and null are rejected. A bare invocation
with no payload and an interactive terminal fails fast rather than waiting for a
typed line.
The worker selects every event rule of <name> whose pattern matches the event
(in declaration order) and executes each through the same runtime path as a
stream event — the same concurrency limits, timeout (capped like event rules),
per-invocation env/secret resolution, and handler execution metrics. A failure
in one handler does not prevent the others from running, and the command reports
the first failure.
Unlike stream consumption, manual invocation is not part of the at-least-once
delivery lifecycle: it never touches Redis, never claims event classification
(the events_* counters), never schedules a retry, and never writes to the
DLQ. It is a synchronous operator action against the live runtime; there is no
offline fallback, so a worker must be running.
Output is a single concise line on success:
No matching handlers
Invoked 1 handler
Invoked N handlers
An unknown function, an unavailable function (its image could not be built), a handler failure, or a missing worker are reported as errors.
Relay separates persistent state from ephemeral runtime state:
/var/lib/relay— volume-mounted persistent state containing the SQLite database, secrets store, and git material. It survives container restarts./run/relay— ephemeral process state that must not be persisted:/run/relay/relay.lock— the process-levelflockheld byrelay startfor the worker lifetime. The kernel releases the lock when the process exits, so a leftover file is inert./run/relay/relay.sock— the live worker control socket. It provides runtime-pool state torelay function inspect, handles the semantic stats reset used byrelay stats reset, and serves synchronous manual invocations fromrelay function invoke. It is removed on graceful shutdown and recreated on startup.
relay start creates /run/relay and acquires the process lock before binding
the socket. A second process therefore fails on the lock before it can interfere
with the active worker's socket. If a worker is killed without cleanup, the
leftover socket is stale and can be safely replaced after the next process has
acquired the lock.
Relay stores secrets as files on disk, one per secret, under a fixed directory.
Templates reference secrets by name; the runtime resolves each reference to its
value immediately before an execution and injects it into the container's
environment. Resolved values live only in the container's Config.Env — they
are never baked into images, never stored in the state database, never logged,
and never shown by relay function inspect (which shows only the reference).
- Location:
/var/lib/relay/secrets(a fixed internal path, not env-configurable). The directory is created on first write with mode0700; each secret file is written atomically with mode0600.compose.dev.yamlmounts the named volumerelay-dataat/var/lib/relay, so secrets survive container restarts. Deleting the volume deletes the secrets. - Rotation: for event/schedule invocations, changing a secret's value takes effect on the next invocation — no rebuild, no restart, no fingerprint change (secrets are resolved per execution). For a persistent service container, the rotate takes effect at the next reconcile of the owning function (the periodic pass; default every 30s): the container's environment is part of its configuration, so a value change makes the running container stale and Relay replaces it — still with no rebuild and no fingerprint change.
- Provider: the local filesystem provider is the current (single-host, beta) implementation. The provider interface is deliberately tiny so a future external provider (Vault, a secrets API, ...) can be added without changing templates or the runner.
Manage secrets with the relay CLI:
relay secret lsNAME
database-url
api-key
relay secret set database-urlrelay secret set NAME reads the value from the terminal with echo disabled
(hidden), or — when stdin is not a terminal — from all of stdin (the safe
non-interactive path, e.g. printf 'value' | relay secret set foo). The value
is never echoed and never printed.
relay secret rm database-url- Secret values are never stored in SQLite, never baked into images, never
in labels, logs, metrics, or
relay function inspectoutput. They exist only as files under/var/lib/relay/secrets(mode0600) and, transiently, in the environment of a running execution container (visible viadocker inspectof that running container only). A service container'srelay.env_hashlabel holds a one-way digest of its effective environment (including resolved secret values), never a value; it exists so a rotated secret value replaces the stale container, and a digest cannot be reversed into the value. - Secret references (the names) are configuration metadata: they appear in
template.yaml, in the fingerprint, and inrelay function inspect. - Never put secret VALUES in
template.yaml— the template is copied into the function's build context and its content is fingerprinted.
Relay can source its functions from an SSH git repository. Git synchronization
is manual by default: relay start never polls, never watches a repository,
and never calls a git remote on its own. The runtime only watches /functions
for filesystem changes (see relay function/the reconciler). Without an
explicit trigger the only thing that updates /functions from git is an
explicit relay git sync.
relay git keygen # generate an SSH deploy key (once)
# add the printed public key as a read-only Deploy Key with your provider
relay git set git@github.com:acme/repo.git # remember the SSH source
relay git sync # when you want, materialize into /functions
relay git status # inspect the sync state
relay git remove # forget the source + drop the checkout
relay git set accepts an SSH URL only — either scp-like (git@host:org/repo.git)
or ssh://git@host/org/repo.git — this iteration does not support HTTPS. It
stores the repository, an optional --ref (default main, a branch/tag/commit),
an optional --path monorepo subdirectory, and an optional --webhook-secret
name (see GitHub webhook below). Calling it again overwrites the source.
Relay can additionally expose a GitHub webhook endpoint that triggers the same
sync an operator would run manually. It is opt-in twice over: it only starts
when both GIT_WEBHOOK_ADDR is set to a non-empty listen address and a git
source is configured. When the webhook server starts it logs
Webhook http server listening on <addr>; when disabled it logs the reason
(missing address, no git source, or a configured webhook secret with no secret
resolver) and binds nothing.
- Endpoint:
POST /githubonGIT_WEBHOOK_ADDR. Onlypushevents for the configured repository and ref schedule a sync; anything else (pings, other events, other repos/refs, deleted refs) is acknowledged with 200 and ignored. - No sync in the request path: a valid delivery only schedules the sync
through a coalescing scheduler — at most one sync runs at a time and pushes
arriving mid-sync collapse into a single follow-up run, so a burst of pushes
converges to the latest commit with exactly one extra sync. The HTTP handler
returns
202 Acceptedimmediately. - Secret is optional: without
relay git set --webhook-secret, deliveries are accepted unauthenticated (a GitHub webhook created without a secret sends no signature header) — anyone who can reach the endpoint can trigger a sync. With a webhook secret configured, every delivery must carry a validX-Hub-Signature-256HMAC; bad or missing signatures are rejected with 401. - Secret resolution: the secret is read from Relay's local secret store by
reference name on every delivery (so
relay secret setrotations apply without a restart). The value is never logged, never persisted in the git config, and never included in any error.
relay secret set github-webhook # store the webhook secret value
relay git set --webhook-secret github-webhook git@github.com:acme/repo.git
# then set GIT_WEBHOOK_ADDR (e.g. :8081) and restart `relay start`
Bind failures (a taken webhook port) are fatal at startup, exactly like the
metrics server, so a port conflict surfaces instead of healing invisibly. Like
relay git set itself, a webhook secret configured later requires a worker
restart to take effect.
Git lives under the fixed convention paths (mirroring the secrets/state layout);
the compose volume mount at /var/lib/relay persists them.
/var/lib/relay/ssh/id_ed25519— the private deploy key (0700 dir, 0600 file, written atomically). Only ever generated byrelay git keygen./var/lib/relay/ssh/known_hosts— Relay's own host-key pins (0600), maintained by TOFU; see below./var/lib/relay/git/source.json— the persisted sync config (0600)./var/lib/relay/git/checkout— the managed git clone/worktree. Sync deletes and re-clones it when the configured repository changes.
Sync authenticates over SSH only, using the local ed25519 key. Host-key
verification is never disabled — but Relay does not read the operator's system
known_hosts and never needs ssh-keyscan. Instead it maintains its own
/var/lib/relay/ssh/known_hosts with a Trust On First Use (TOFU) policy,
working with GitHub, GitLab, or any SSH git server:
- The first sync reaches a host, it trusts that key, prints the fingerprint
(
Trusted new host <host> (fingerprint SHA256:...)), and persists it. - Every later sync verifies against that pinned key.
- A changed host key fails the sync with a clear man-in-the-middle warning
and is never auto-replaced — a genuinely rotated key is resolved by the
operator editing
/var/lib/relay/ssh/known_hostsand syncing again.
relay git sync checks out the configured ref (detached HEAD, a hard reset — the
remote is always the source of truth; it never pulls, so no merge state is ever
produced), then locates the functions source at the repo root or the monitored
--path. Each direct subdirectory containing a template.yaml is one function.
Deterministic replace: while a git source is configured and synced, /functions
is owned by git. A sync rewrites /functions to reflect exactly the repository
and path: every function directory is refreshed from the checkout, and any
directory present in /functions but absent from the source is removed. Operator
placed directly in /functions are subject to the same rule once a git source is
configured and synced. Each function is copied into /functions atomically-ish
(temp copy + rename), so the reconciler never sees a half-written function. A
valid checkout containing zero functions is allowed — sync succeeds and leaves
/functions empty.
relay git status prints the configured source, whether the SSH key and checkout
exist, the resolved commit (if a checkout exists), and the last sync time — never
any key material. With nothing configured it prints "No git source configured."
and exits 0.
relay git remove deletes the persisted config and the checkout directory. It
leaves /functions untouched and keeps the SSH key (the operator registered
its public half as a Deploy Key; removing the source is unrelated to the key's
lifecycle, and re-keying is an explicit relay git keygen operation). It is
idempotent: running it again reports nothing configured and fails only if the
config is corrupt.
Relay's observability is logs plus Prometheus metrics plus the local state
snapshot. There is no HTTP health/readiness endpoint — relay health (above)
remains the health check.
- Structured logs: execution, retry, failure, DLQ,
reconciliation, and build lines carry structured
slogattributes —function,handler,message_id,attempt,duration, and containerexit_codewhere available. Handler stdout/stderr is forwarded unconditionally as a raw transport: it is not routed through slog, so it is unaffected byLOG_LEVEL, and only Relay's own operational logs are governed by the level. - Prometheus metrics: when
METRICS_ADDRis set to a non-empty listen address, the Relay runtime exposesGET /metricson that address in Prometheus text format via the official Prometheus client. Counters:events_received_total,events_matched_total,events_unmatched_total,handler_success_total,handler_failure_total,retries_total,dlq_entries_total,handler_invocations_total{outcome,function,handler},build_failures_total{function}, and per-functionfunction_events_matched_total{function}plus the otherfunction_*_total{function}counters. Histograms:handler_duration_seconds{function,handler},function_build_seconds{function}. Gauges:pending_entries,pending_oldest_age_seconds— sampled from the Redis consumer group (XPENDING) every 15s, not per event — plusbuffered_events(the current local buffer occupancy, set on each acquire/release) andin_flight_invocations(the current number of executing invocations in this worker), and theconcurrency_waits_totalcounter (each time an invocation's concurrency-slot acquisition had to block). Schedule coordination counters (schedule_occurrences_published_total,schedule_occurrences_duplicate_total,schedule_publish_failures_total) track the distributed publish-if-new path per worker: published/duplicate counts converge across workers toward one published entry per logical occurrence, while failures flag workers that cannot reach Redis. These are Prometheus-only and not part of the SQLite snapshot. Labels are bounded tofunction/handler/outcomeplus the small closed runtime-pool value sets below; IDs (message, event, container, fingerprint) are never labels. The metrics server is operationally isolated: bind failures are logged and retried, scrape errors never stop event consumption, and shutdown is graceful. Prometheus is the source for time-series metrics. - Event classification counters:
events_received_total,events_matched_total, andevents_unmatched_totalform a closed partition of the logical incoming events the runner handled:received == matched + unmatched. Each logical event is classified exactly once across redeliveries and retries — a message reclaimed by recovery (or redelivered after a failed handler/ACK) is the same logical event, so it does not increment them again. The once-only claim is an atomic RedisHSETNXfield in the message's invocation-state hash, which also makes the claim safe across replicas; if the claim cannot be written (Redis error) nothing is counted, because the counters are an exact partition and a missed count is preferable to a double count. The class is decided from matching alone, before any execution, so a handler failure staysmatched;unmatchedevents are acknowledged and never retried.function_events_matched_totalattributes a function once per matched logical event (deduped across that function's multiple matching rules), so an event matching two functions counts once globally and once per function. Schedule occurrences bypass event matching and are not part of this partition; malformed messages never reach the runner and are likewise not classified. These three counters (plus the per-function matched counter) are persisted in the SQLite snapshot and restored at startup like the other cumulative counters. - Warm-container pool metrics: the per-function warm container pool
(see Execution container lifecycle) publishes its own series, all
function-scoped and low-cardinality:
runtime_pool_capacity{function}(the function's effective concurrency — its resolved templateconcurrencyclipped toMAX_CONCURRENCY— the pool's bound),runtime_containers{function,state}(a gauge of currently pooled containers split bystate=idle|busy|starting),runtime_container_acquires_total{function,outcome=warm|cold}(warm= an existing idle container was leased, including after a capacity wait;cold= a fresh container was started),runtime_container_discards_total{function,reason}wherereasonis one of the finite teardown causes (timeout,process_exit,protocol_error,image_changed,idle_timeout,concurrency_shrink,shutdown; removal-time discards are tombstoned, see below) — the reason label is strictly causal, never a synthetic value,runtime_container_acquire_duration_seconds{function}(a histogram observed for successful acquires only, end-to-end including any wait), andruntime_container_waits_total{function}(acquires that blocked at the pool bound). The gauges are authoritative at the moment they are written: astartingreservation is rolled back on start success, failure, or panic. A function's series are deleted when the function is removed, atomically with the removal transition: the pool becomes a metric tombstone under the same lock that deletes the series, so a late release/discard from a busy removed container can never recreate them. The removal-timefunction_removeddiscard is therefore not counted (it would recreate a deleted series); other functions' series and a genuinely reactivated function's fresh series are unaffected. The CUMULATIVE pool counters (warm acquires, cold starts, discards) are persisted per function so the standalone CLI can render them and so they stay monotonic across restarts; the LIVE pool gauges (capacity and the idle/busy/starting counts) are never persisted to SQLite — a persisted live gauge would go stale between flushes. Restored discards are an a-causal aggregate, so at startup they are held in an internal per-function baseline and folded into the cumulative discard total read by the CLI rather than emitted as a syntheticreasonseries; the per-reason series stay causal. - SQLite operational snapshots: the local state database also keeps the
current operational counters (
stats) and per-function counters (function_stats) — latest totals only, never history or per-event rows. The worker flushes the in-memory registry into SQLite every 5 seconds (fixed, non-configurable), so these rows may lag live Prometheus by up to ~5s; a graceful shutdown performs a final bounded flush, while a hard crash may lose up to ~5s of telemetry. A read-only CLI command renders the global snapshot:
relay statsEvents received: 153000
Events matched: 152934
Events unmatched: 66
Handler successes: 152801
Handler failures: 133
Retries: 82
DLQ entries: 4
Pending entries: 17
Oldest pending age: 2m14s
Updated: 10s ago
relay stats reads the state database file only (no Redis, Docker, or
/functions); it works even when the runtime is down. A fresh database renders
zeroes with Updated: never. Backlog gauges (pending_entries,
oldest_pending_age_seconds) are global — the consumer-group backlog is not
attributed to individual functions.
relay stats reset resets the totals in place: the global cumulative counters
are zeroed and every per-function function_stats row — including the four
per-function Last* execution timestamps and the cumulative warm/cold/discarded
pool counters — is zeroed in its existing row (the rows are not deleted). The
backlog gauges are not reset: they are point-in-time snapshots of the live
Redis backlog, and the next worker flush refreshes them. With a running worker,
the reset goes through the worker's Unix socket so the worker records a
worker-owned reset baseline under the same lock as the periodic flush — a
captured pre-reset snapshot can never be written after the reset, and the
persisted statistics continue from zero. Without a running worker, the state
database is rewritten directly. It does not touch Redis or pending events,
containers/runtime pools, schedules/services, or the worker's Prometheus
counters (those stay monotonic for the process lifetime; the worker subtracts
its Relay-side baseline when snapshotting, never mutating Prometheus). After a
reset, stats accumulate normally again.
A message is acknowledged (XACK) only after all matching invocations reach a terminal state: every handler either succeeds, or is routed to the DLQ because its retries were exhausted (and no other matched invocation is still unresolved):
event → handler A ✓ → handler B ✓ → ... → XACK
event → handler A ✓ → handler B ✗ (retries exhausted) → DLQ write → XACK
If any invocation fails in a way that will be retried, the message is not acknowledged and remains pending for redelivery (at-least-once semantics). A failure in one handler does not stop the other matching handlers from running on the same delivery; each matching invocation gets its own independent attempt and they are aggregated afterwards:
event → handler A ✗ (will retry) → handler B ✓ → no XACK (message stays pending)
An event that matches no rules is a success and is acknowledged. Because delivery is at-least-once, handlers should tolerate duplicate delivery.
Relay consumes with a consumer group, so every delivered message records an entry in the group's Pending Entries List (PEL) until it is acknowledged. A message a consumer reads but never acknowledges — a crash, an outage, or a handler failure — stays in the PEL.
- Recovery loop (
XAUTOCLAIM): a background goroutine runs everyDefaultReclaimInterval(1m) and reclaims messages that have sat pending for longer thanMinPendingIdle(default 1m). This is a message-level recovery-pacing backstop, not the concurrency guard and not the retry timer: reclaiming transfers ownership of the message and replays it through the same processing path as a fresh read, but whether an individual handler actually executes is decided per-invocation from the invocation state (see below). Rule timeouts are capped at 5m (timeoutvalues above5mfail template validation). This makes Relay survive restarts: a message left pending by a dead consumer is picked up and retried by a live one. - Retry counting: the per-message delivery count is read from Redis
(
XPENDINGfull form / retry counter), not kept in process memory, so the count survives restarts. Each reclaim of an idle message increments the count. This delivery count is diagnostic only: it does not drive retry or exhaustion decisions and it is not the handler attempt count. Retry timing and exhaustion are defined per-invocation (see below). A failed attempt records anext_attempt_atdeadline in the invocation state, and a redelivery before that deadline is skipped. Actual retry timing is quantized by the reclaim cadence (~1m granularity), so a 1m backoff effectively fires at the first redelivery after 1m. - Per-invocation retries and backoff: each rule's
retries(default4) bounds the number of additional executions after the initial one. A failing invocation is retried with a fixed backoff schedule — 1m after attempt 1, 2m after attempt 2, 5m after attempt 3, then 10m (capped) — persisted as anext_attempt_atmarker in the invocation state. Once1 + retriesattempts are exhausted, the invocation is markedexhausted(terminal). - Exhaustion → DLQ: when every non-complete matched invocation is exhausted,
the message is dead-lettered and the original is then acknowledged, removing it
from the PEL. A message is written to the dead-letter stream
relay:<stream>:dlq(a Relay-ownedrelay:-prefixed key) as one entry per exhausted invocation: a message matching several functions or handlers that all exhaust produces one precisely-attributed entry each. The XACK is issued only after every required DLQ entry is persisted (and never before it); a failed DLQ write leaves the message pending (see DLQ write ordering). Because each exhausted invocation is terminal, a redelivery of a message whose DLQ write or post-DLQ XACK failed skips re-execution and re-reports exhaustion, so the message is re-routed to the DLQ rather than acknowledged without an entry. - DLQ entry format (flat fields):
original_stream,original_id,group,consumer,event(the original payload string),reason,function,handler,deliveries,handler_attempts,timestamp(RFC 3339).functionandhandlername the exact exhausted invocation the entry attributes;handler_attemptsis the handler execution attempt that exhausted that invocation's per-invocation retry state and drove the DLQ decision — the real execution count.deliveriesis the Redis Stream/PEL delivery count (the retry counter read fromXPENDING, passed through the consumer/reclaim flow): it is diagnostic only, counts every redelivery including redeliveries that skipped a protected invocation, and is therefore>= handler_attempts. They are distinct on purpose: a reclaimed message can be redelivered many times while the handler attempt advances only on real executions. A DLQ path with no handler retry state (a malformed message routed pre-handler) carries the-placeholder forfunction/handlerand an explicithandler_attemptsof0, never a fabricated value derived fromdeliveries. - DLQ write ordering and idempotent retry: the DLQ is written before the
original is acknowledged. If a DLQ write fails, the original is left pending so
the next recovery cycle redelivers it; the exhausted invocation is skipped
without re-running and exhaustion is re-reported, so the message is re-routed
instead of being lost. Once an invocation's entry is successfully written, its
invocation-state marker becomes
exhausted:<attempts>:dlq; a redelivery skips the already-persisted entries (without scanning the DLQ) and writes only the missing ones, so retrying a partially-written multi-entry DLQ (after an XACK failure, a crash, or a one-of-N write failure) neither duplicates nor loses entries. The XACK and the invocation-state clear happen only after every required entry is persisted. - Non-retryable failures: a message whose
eventfield is missing, is not a string, or is not a JSON object can never succeed. It is routed straight to the DLQ on first encounter — without running any handler — and acknowledged. - Malformed input never consumes retry cycles.
These recovery defaults are a fixed part of the stream package and cannot be
overridden by environment variables. A zero-valued ConsumerConfig field
falls back to them in NewConsumer.
The at-least-once contract from the ACK table above is unchanged: XACK happens
only after all matching invocations succeed or the message is successfully
routed to the DLQ. To avoid re-running work that already succeeded, Relay records
per-handler invocation state in Redis: each message has a TTL'd hash keyed by
message (relay:invocation:{stream}:{group}:{msgID}, field
<function>/<handler>; stream/group names are percent-encoded in the key). The
field value describes the invocation's lifecycle for this message:
ok— the invocation completed on a previous delivery; redeliveries skip it.running:<unix-nano deadline>#<attempts>— an attempt is (or was) executing, protected until that absolute deadline;<attempts>is the 1-based attempt number. A deadline marker without#<attempts>does not parse (treated as absent/eligible).next_attempt_at:<unix-nano deadline>#<attempts>— a failed attempt is waiting out its retry backoff, protected until that absolute deadline.exhausted:<attempts>— the invocation's attempts are exhausted; it is terminal and never eligible again, and its DLQ entry has not yet been persisted. A redelivery of an exhausted invocation skips re-execution but still re-reports exhaustion, so a message whose DLQ write or post-DLQ XACK failed is re-routed to the DLQ rather than being acknowledged without an entry.exhausted:<attempts>:dlq— the invocation's attempts are exhausted and its DLQ entry has been persisted. It parses exactly likeexhausted:<attempts>(same terminal state and attempt count); the suffix only lets a redelivery skip the already-written entry without scanning the DLQ stream, so retrying a partially-written multi-entry DLQ is idempotent.- absent — eligible to execute.
Before executing an invocation, the runner claims it via TryStart, which
persists running:<now+timeout>#<attempts> — the same capped timeout the local
context.WithTimeout enforces, so the persisted deadline and the local timer
match by construction. On success MarkComplete overwrites the marker with ok;
on failure RecordFailure persists next_attempt_at:<now+backoff>#<attempts>
with the rule's backoff (1m/2m/5m/10m); once attempts are exhausted
MarkExhausted writes exhausted:<attempts>. Another worker that redelivers
the message while now < running_until or now < next_attempt_at skips that
invocation, because a live attempt (this or another replica) may be executing it
or it is waiting out its backoff. A crashed worker's marker self-expires at its
deadline, so recovery waits it out (bounded by at most one timeout) instead of
racing a live attempt. Bookkeeping failures fail open: a Redis error on the read
or write never blocks delivery, preserving at-least-once. The message is
acknowledged once every matching invocation is complete, or once all are
terminal and at least one exhausted (the message is successfully routed to the
DLQ); the invocation-state key is cleared after a successful XACK on success or
DLQ routing. The keys expire after 7 days as a fallback cleanup for abandoned
messages.
This is still at-least-once, not exactly-once: there is a crash window between a handler's side effect and its state being recorded, so a handler can still run twice. Handlers must therefore remain idempotent. Renaming a function or handler invalidates old invocation state (old entries simply never match), and a rule removed from a template no longer gates the acknowledgment.
The repository's examples/functions/ directory contains development / example
functions to copy and adapt. These are samples for local development and the
README examples — not the production function directory. Relay always reads
its functions from /functions inside the container; for production, mount
your own function directory there (see the deployment section above).
examples/functions/ exists only so the repository ships working, testable
examples; each direct subdirectory is one function, deployed as described
above.
-
examples/functions/user-events-python/(python3.14): three rules on theuserstable, each mapping an event to a module inside theevents/namespace package (no__init__.py):events.created.handleronevent_name: INSERT.events.updated.handleronevent_name: MODIFY(with an explicittimeout: 20sdemonstrating the per-rule timeout).events.deleted.handleronevent_name: REMOVE.
Each module defines a single
handler(event)function.created/updatedreadevent["new_image"];deletedreadsevent["old_image"]. Plain stdlib only, norequirements.txt. -
examples/functions/welcome-email-node/(node24): a single rulehandler.handleronevent_name: INSERT/table_name: users. The handler readsevent.new_imageand logs a welcome email. Nopackage.jsonis provided, so Relay injects the ESMpackage.json. It omitstimeout, so it exercises the6sdefault. -
examples/functions/order-confirmation-typescript/(node24): a TypeScript handler. A single rulesrc.handler.handlerontype: [order.created]resolves tosrc/handler.ts, which imports a local message module and a shared event-type module (both.ts). Relay transpiles the graph tosrc/handler.mjsat build time with the pinned esbuild;tsconfig.jsonand a committedpackage-lock.jsonare present. -
examples/functions/users-api-node/(node24): a persistent service — a small Fastify HTTP server (service.js) exposingGET /healthandGET /users, listening on the template-configured port (3000, injected asPORT). Noevents-style invocation: Relay keeps the container running and reconciled. -
examples/functions/fastapi-service/(python3.14): a persistent service demonstrating thatentrypoint: app/main.pyruns aspython -m app.main, so package-relative imports work:main.pyimportsfrom .deps import get_settingsand servesGET /health(started viauvicorn.runin user code, readingPORT). FastAPI/Uvicorn live in the user's code — Relay only decides how the entrypoint file is executed. -
examples/functions/custom-build-service/(no runtime): a persistent build-source service — a userDockerfile(built by Relay from the function's selected source via the Docker Engine API) with its ownENTRYPOINT, aserver.js, and atemplate.yamlthat declaresbuild: Dockerfileand noruntimeat all. An equivalentimage:service would reference an external image instead.
A single generic, cross-engine event matches both functions:
{
"event_id": "evt_123",
"event_name": "INSERT",
"table_name": "users",
"new_image": {
"id": "user_123",
"name": "John Doe",
"email": "john@example.com"
}
}It matches user-events-python → events.created.handler (Python, prints the
user id) and welcome-email-node → handler.handler (Node, logs the
welcome email).
With Relay consuming stream events in group relay (as configured in
compose.dev.yaml), add the example event with redis-cli:
redis-cli XADD events '*' event '{"event_id":"evt_123","event_name":"INSERT","table_name":"users","new_image":{"id":"user_123","name":"John Doe","email":"john@example.com"}}'
Relay then logs the matched rules and the forwarded handler output:
relay: function "user-events-python" rule "events.created.handler" matched event "<msg-id>"
relay: function "user-events-python" handler "events.created.handler": User created: user_123
relay: function "user-events-python" handler "events.created.handler" executed for event "<msg-id>"
relay: function "welcome-email-node" rule "handler.handler" matched event "<msg-id>"
relay: function "welcome-email-node" handler "handler.handler": Sending welcome email to john@example.com
relay: function "welcome-email-node" handler "handler.handler" executed for event "<msg-id>"
Other runtimes, poetry/pipenv/pnpm/yarn/bun,
build caching, source hashing,
k8s, configurable retry policies per rule (delays/attempt counts
are fixed internals — a rule's retries count is configurable, the backoff
schedule is not), idempotency, exactly-once, per-function
resource limits/networking, external secret-management providers (Vault/AWS/K8s
— the local file provider is the current backend), HTTP API (beyond the
Prometheus /metrics scrape endpoint), UI, full observability platforms
(tracing, log shippers), and additional operators
(anything-but/regex/glob/scripts) are not implemented in this iteration.
Git is supported via the manual relay git sync workflow plus the opt-in
GitHub webhook endpoint (see "Git"); what remains out of scope is polling-based
automatic synchronization — periodic polling, filesystem watchers on the
repository, providers other than GitHub, and webhook providers requiring
payload-level filtering beyond ref/repo matching. Relay never syncs on its own;
only an explicit relay git sync or an accepted webhook delivery updates
/functions.