What happened?
On a stock ghcr.io/toeverything/affine:stable (0.27.4) self-host, every AI BYOK provider fails at "Connection failed" — Anthropic, Gemini, and OpenAI, with keys independently verified working against each vendor's API from outside the container.
While debugging this end to end I found two packaging defects not covered by #15505 / #15506 / #15530 / #15534 / #15552, independently confirmed a third, and hit a fourth meta-problem that made all of it far harder than it needed to be. Sharing everything in case it helps triage the cluster of "Connection failed" reports.
Defect 1 — The published image ships with a completely empty CA certificate store
/etc/ssl/certs in the running container contains zero files:
$ docker exec <affine> sh -c 'ls -la /etc/ssl/certs'
total 8
drwxr-xr-x 2 root root 4096 Jun 6 19:56 .
drwxr-xr-x 4 root root 4096 Aug 18 15:47 ..
$ docker exec <affine> sh -c 'ls /etc/ssl/certs/*.pem 2>/dev/null | wc -l'
0
ca-certificates is not installed. Base image is Debian 12 (bookworm).
This matters because dist/server-native.x64.node is the component that makes provider calls, and it is built on rustls + reqwest:
rustls -> 38 matches
webpki -> 7
reqwest -> 12
hyper -> 29
BEGIN_CERTIFICATE -> 0 (no roots embedded in the binary)
The binary also contains the rustls trust-anchor error strings (UnknownIssuer, CaUsedAsEndEntity, invalid peer certificate), so a missing system trust store is a reachable failure path for it.
Node is unaffected because it bundles its own CA store, which is why naive egress tests pass and mask this:
# Node, bundled CA
api.anthropic.com -> HTTP 401 (reached)
# Node, forced to the system store, BEFORE installing ca-certificates
# -> nothing to trust
Fix: apt-get install -y ca-certificates in the image, or compile the native runtime against rustls-tls-webpki-roots so it carries its own anchors.
Note on scope: I fixed this on the deployment (store now populated, verified with node --use-openssl-ca reaching api.anthropic.com) and it did not by itself resolve the BYOK failure. Reporting it as an independent packaging defect, not as the root cause.
Defect 2 — config.json is absent from every path the native runtime searches
Confirms and extends #15506 on a stock deploy. On a clean install there is no config.json anywhere:
absent /app/config.json <- Rust runtime: <cwd>/config.json
absent /usr/local/bin/config.json <- Rust runtime: <dir of current_exe>/config.json
absent /root/.affine/config/config.json <- Nest reader
The container's config volume contains only private.key. Enabling Copilot through Admin → Settings → AI BYOK writes to the app_configs table (copilot.enabled = true), which the Nest side honours — but the native runtime reads neither the database nor any file that exists, so the two halves disagree by construction on a default install.
Writing the file to all three paths and restarting did not resolve the connection failure either, but the divergence is real and should probably be a startup warning at minimum.
Defect 3 — Anthropic /v1 duplication is still present in :stable
#15505 is closed, but the fix is not in any released stable build. v0.27.4 (2026-08-18) is still the newest non-prerelease tag, and #15505 was closed 2026-08-21. The base URL is still visible in the shipped binary:
https://api.openai.com/v1
https://api.anthropic.com/v1 <- joined with "/v1/messages"
https://generativelanguage.googleapis.com/v1beta
Independently verified from outside:
| URL |
Response |
https://api.anthropic.com/v1/messages |
401 (exists, needs auth) |
https://api.anthropic.com/v1/v1/messages |
404 |
So every Anthropic BYOK attempt on the current stable image is unfixable by the operator. Worth either a backported patch release or a note in the self-host docs.
Defect 4 (meta) — The native provider path emits no diagnostics at all
This is what turned a config problem into a multi-hour blind hunt, and I suspect it is why so many of these reports stall.
A failing "Test connection" produces nothing in docker logs — no error, no warning, no request line. The native binary has no logging hooks:
RUST_LOG -> 0
tracing -> 0
env_logger -> 0
So there is no supported way for an operator to see the upstream status code, URL, or error body. Everything above had to be recovered by running grep against a 64 MB compiled artifact.
Suggested fix: honour RUST_LOG, or have NativeProviderAdapter surface the upstream HTTP status and response body into the Nest logger on probe failure. Even just logging the final request URL would have made #15505 self-diagnosing.
What I ruled out
Environment
- AFFiNE
0.27.4, ghcr.io/toeverything/affine:stable, deployed via a hosting provider's one-click Docker template
- Debian 12 container, Postgres (pgvector) + Redis, Traefik reverse proxy, valid Let's Encrypt cert
- No Manticore/Elasticsearch indexer (the template does not deploy one)
- BYOK storage mode: Server
AFFINE_SERVER_HOST is set by the template to a full URL (https://<host>) rather than a bare FQDN, which contradicts the schema description (Where the server get deployed(FQDN)) — possibly worth validating at startup
Happy to provide further captures from this deployment.
What happened?
On a stock
ghcr.io/toeverything/affine:stable(0.27.4) self-host, every AI BYOK provider fails at "Connection failed" — Anthropic, Gemini, and OpenAI, with keys independently verified working against each vendor's API from outside the container.While debugging this end to end I found two packaging defects not covered by #15505 / #15506 / #15530 / #15534 / #15552, independently confirmed a third, and hit a fourth meta-problem that made all of it far harder than it needed to be. Sharing everything in case it helps triage the cluster of "Connection failed" reports.
Defect 1 — The published image ships with a completely empty CA certificate store
/etc/ssl/certsin the running container contains zero files:ca-certificatesis not installed. Base image is Debian 12 (bookworm).This matters because
dist/server-native.x64.nodeis the component that makes provider calls, and it is built on rustls + reqwest:The binary also contains the rustls trust-anchor error strings (
UnknownIssuer,CaUsedAsEndEntity,invalid peer certificate), so a missing system trust store is a reachable failure path for it.Node is unaffected because it bundles its own CA store, which is why naive egress tests pass and mask this:
Fix:
apt-get install -y ca-certificatesin the image, or compile the native runtime againstrustls-tls-webpki-rootsso it carries its own anchors.Note on scope: I fixed this on the deployment (store now populated, verified with
node --use-openssl-careachingapi.anthropic.com) and it did not by itself resolve the BYOK failure. Reporting it as an independent packaging defect, not as the root cause.Defect 2 —
config.jsonis absent from every path the native runtime searchesConfirms and extends #15506 on a stock deploy. On a clean install there is no
config.jsonanywhere:The container's config volume contains only
private.key. Enabling Copilot through Admin → Settings → AI BYOK writes to theapp_configstable (copilot.enabled = true), which the Nest side honours — but the native runtime reads neither the database nor any file that exists, so the two halves disagree by construction on a default install.Writing the file to all three paths and restarting did not resolve the connection failure either, but the divergence is real and should probably be a startup warning at minimum.
Defect 3 — Anthropic
/v1duplication is still present in:stable#15505 is closed, but the fix is not in any released stable build.
v0.27.4(2026-08-18) is still the newest non-prerelease tag, and #15505 was closed 2026-08-21. The base URL is still visible in the shipped binary:Independently verified from outside:
https://api.anthropic.com/v1/messages401(exists, needs auth)https://api.anthropic.com/v1/v1/messages404So every Anthropic BYOK attempt on the current stable image is unfixable by the operator. Worth either a backported patch release or a note in the self-host docs.
Defect 4 (meta) — The native provider path emits no diagnostics at all
This is what turned a config problem into a multi-hour blind hunt, and I suspect it is why so many of these reports stall.
A failing "Test connection" produces nothing in
docker logs— no error, no warning, no request line. The native binary has no logging hooks:So there is no supported way for an operator to see the upstream status code, URL, or error body. Everything above had to be recovered by running
grepagainst a 64 MB compiled artifact.Suggested fix: honour
RUST_LOG, or haveNativeProviderAdaptersurface the upstream HTTP status and response body into the Nest logger on probe failure. Even just logging the final request URL would have made #15505 self-diagnosing.What I ruled out
server.externalUrlmismatch (Defect 1 of [Bug]: [Self-host BYOK] Three undocumented failure modes: CORS externalUrl mismatch, 16-token structured probe, Peer disconnected on long action turns #15552) — not applicable here.externalUrlmatches the browsing origin and noBlocked CORS requestwarnings appear./v1/modelsand/v1/messageswithclaude-haiku-4-5from a separate host. Account has credit.api.anthropic.comandgenerativelanguage.googleapis.com.claude-haiku-4-5,claude-sonnet-5,claude-opus-5,gpt-4.1,gemini-3.7-flash,gemini-3.6-flash.Environment
0.27.4,ghcr.io/toeverything/affine:stable, deployed via a hosting provider's one-click Docker templateAFFINE_SERVER_HOSTis set by the template to a full URL (https://<host>) rather than a bare FQDN, which contradicts the schema description (Where the server get deployed(FQDN)) — possibly worth validating at startupHappy to provide further captures from this deployment.