chore(backend): enforce ruff TRY400 so error logs carry tracebacks

[saltas888]

Re-enables the ruff TRY400 (error-instead-of-exception) rule so a log.error reporting a caught exception carries its traceback. Jira: INBOX-29.

Scope: TRY400 only — this does not close INBOX-29

The card asked for TRY400 and TRY004. Only TRY400 is here.

TRY004's fix changes the exception type a guard raises at 40 sites — 5 in core/schema/schema_branch.py and 14 across graphql/mutations/* and graphql/types/node.py. Swapping ValueError/Exception → TypeError there is caller-visible in GraphQL error responses, so it is an API-behaviour decision rather than a lint cleanup. TRY004 stays suppressed and needs a human decision — please split a follow-up card rather than letting this merge mark INBOX-29 Done.

What changed

extend-select = ["TRY400"] in [tool.ruff.lint] enables just this rule; the broad "TRY" entry in ignore stays, keeping the rest of the family (TRY004 included) suppressed — ruff resolves the more specific selector first. Enumerating the remaining TRY codes instead is not possible: TRY200 is a removed rule and naming it breaks ruff.

Of the 36 flagged sites:

• 27 converted to log.exception — same error level, same message, same keyword fields, no control-flow change; the traceback is the only addition.
• 7 kept as log.error with a justified # noqa: TRY400 — including graphql/app.py → _handle_http_request, where a client aborting mid-request is a routine event whose traceback only shows the body-read path (converted in the first pass, reverted after review).
• 2 suppressed by file in backend/infrahub/auth/auth.py (see below).

Bonus: 9 fewer suppressions. BLE001 doesn't flag a blind except Exception whose handler logs via .exception(), so converting those handlers made 9 # noqa: BLE001 directives redundant (services/scheduler.py ×1, utilities/infrahub_load_tester.py ×8). They're gone.

The concrete debuggability win: services/scheduler.py:91, the scheduler's keep-alive handler, previously logged only str(exc) — a failing recurring task was undiagnosable by construction.

The one site that must not be converted 🔍

backend/infrahub/webhook/tasks/process.py:204 deliberately keeps log.error.

WebhookDeliveryError is registered with @suppress_traceback_in_logs, and TracebackSuppressionFilter — installed on the Prefect run loggers this site logs through — drops the entire log record, not merely its traceback, for a registered exception type. Converting this call would have attached a WebhookDeliveryError to the record and thereby silently deleted the classified delivery-failure report (status class, message, remediation, attempt, elapsed) from the run logs — while leaving the lint gate green and the diff looking innocuous.

ruff --fix --unsafe-fixes produces exactly that bug. Every site here was read at its call site instead.

Related hazard, no action needed now: two converted sites (git/integrator.py:1568, :1608) catch bare Exception behind a run logger. They're safe only because WebhookDeliveryError is the sole registered type and is unreachable from repository-integration code. Registering a second, widely-raised type would silently mute them.

Two things to look at closely

1. backend/infrahub/graphql/app.py is in the diff — 2 log lines in ASGI/websocket error handling, not the GraphQL contract surface. At line 535 the redundant exc_info=error is dropped since the call sits inside except Exception as error: and .exception() attaches it implicitly. The similar-looking call in _log_error (line 392) is left alone: it runs outside any except block and must pass exc_info explicitly. Ruff agrees — it never flagged that one.

2. A new per-file-ignores entry for backend/infrahub/auth/auth.py — added by a change whose whole purpose is removing a suppression, which deserves an explanation. Both of its TRY400 sites are pure logging inside except blocks and would convert cleanly; the file is untouched only because this change was produced by an automated pipeline that may not edit auth modules unattended. The merged BLE precedent (Re-enable ruff BLE (blind-except) rule and fix all violations #10002) did edit this same file, adding # noqa: BLE001 at the very handlers holding these two sites — so if you'd rather have the 2-line inline fix, drop the entry and convert them. That call is deliberately yours.

FakeLogger.exception (backend/tests/adapters/log.py) was a no-op stub, so test_scheduler_task_with_error failed once the scheduler site converted. The fake was wrong, not the test — .exception emits at error level, so it now records alongside error events and the existing assertion passes unmodified.

Testing

Check	Result
ruff check --select TRY400	0 (was 36)
ruff check . (full, default config)	clean — no new violation of any other rule
ruff check --select TRY004	still exactly 40, unchanged; no file it flags was touched
invoke lint	ruff + ty + mypy all pass (mypy: 1610 files, no issues)
invoke backend.test-unit	2233 passed
validate-generated, docs.validate, schema.validate-*, uv lock --check	clean, no drift
Changed-file audit	no core/schema/, core/migrations/, auth/, .github/, no generated file

Every remaining # noqa: TRY400 carries a one-line reason. Frontend Betterer/codegen not run — no frontend file, GraphQL schema, or generated frontend type is in the diff.

Spec, per-site decision table, critique, and full report: dev/specs/005-ruff-try400-tracebacks/.

🤖 Generated with Claude Code

[cubic-dev-ai]

All reported issues were addressed across 23 files

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Re-trigger cubic</sub>

[saltas888]

🤖 CI reconcile — two checks were red. One was mine and is fixed; one is pre-existing on develop.

1. validate-release-notes-style — mine, fixed in 6396a66

Vale rejected a single word in my changelog fragment: suppressions isn't in .vale/styles/spelling-exceptions.txt. I reworded the line ("comments were removed") instead of adding the word to the shared accept vocabulary — it appears in no other changelog or release-notes file, so growing shared config for one use seemed like the wrong trade. Happy to flip it to a vocabulary entry if you'd prefer.

Verified locally with the exact command CI runs, on the same Vale version CI pins (3.12.0): 0 errors, down from 1. The 3 remaining warnings are pre-existing in docs/docs/release-notes/infrahub/ files this PR doesn't touch, and warnings don't fail the job.

2. E2E-testing-pytest-playwright (sites_b) — not caused by this PR

It failed with exit code 137 (SIGKILL): the runner killed docker ps -a during the post-test "Containers after tests" step, and no playwright-junit.xml was produced at all. Its "Check out repository code" step also failed. That's runner resource exhaustion, not a test assertion.

The same shard was already failing on develop in run 31495853417 (13:22 UTC, before this branch existed), together with 7 other jobs including backend-tests-integration and E2E-testing-playwright.

Nothing in this diff can plausibly affect a Playwright UI shard: it's backend logging calls, ruff config, a test-logger adapter, and a changelog line. I've left it alone rather than papering over it.

Rebase deliberately skipped

The branch is 4 commits behind develop, but all are typing chores (implicit-Optional and **kwargs annotation cleanups). None touches E2E or the failing shard, so rebasing wouldn't fix anything and would only add force-push churn to a PR that's awaiting review. Say the word if you'd rather I rebase anyway.

[cubic-dev-ai]

0 issues found across 1 file (changes from recent commits).

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

<sub>Shadow auto-approve: would not auto-approve. Auto-approval blocked by 3 unresolved issues from previous reviews.

Re-trigger cubic</sub>

[codspeed-hq]

Merging this PR will not alter performance

✅ 13 untouched benchmarks

---

_{Comparing pha/INBOX-29 (2660959) with develop (4a2cce2)}

[saltas888]

🤖 Reconcile pass 1 — CI was red. Rebased onto latest develop (was 4 behind, clean, no conflicts) and force-pushed; a fresh full run is now in flight.

Triage of the previous run (31510405357)

Six jobs failed. None failed on an assertion attributable to this diff — every one is a resource/timing signature:

Job	Signature
backend-tests-integration	exit 137 (SIGKILL/OOM), [gw1] node down: Not properly terminated
backend-tests-component (other)	worker gw0 crashed + xdist.dsession.Interrupted: session-timeout: 1800.0 sec exceeded — 664 passed, 1 failed by the crash
E2E-testing-pytest-playwright ×3 (branches_repo, sites_a, sites_b)	ServerNotResponsiveError: Unable to read from .../api/schema/load (timeout: 120 sec) — the task-worker never came up
E2E-testing-playwright	also failing on #10218, so it is broken beyond this PR

Meanwhile every job this change could deterministically affect passed: python-lint, backend-tests-unit (3.12 / 3.13 / 3.14), backend-validate-generated, graphql-schema, json-schema, markdown-lint, documentation, backend-tests-component (core-diff / core-schema / graphql), backend-tests-functional, backend-docker-integration.

On the E2E worker traceback

The shard logs show infrahub_async.py raising typer.Exit(1) out of _init_infrahub_client — which is a line this PR touches, so worth being explicit: the raise typer.Exit(1) from err is unchanged, and both of that method's log calls keep their existing behaviour (line 196 stays log.error with a justified noqa, line 204 became log.exception). Neither alters control flow. The worker exited because it could not reach the server at startup, which is the pre-existing path for SdkError there.

Hypothesis I considered and did not dismiss

log.exception emits a traceback where a bare message was emitted before, so in principle it could add log volume — and the failures here are timeouts and OOM, which is what "something got slower or noisier" looks like. Worth stating plainly rather than hiding behind "flaky". Against it: the added tracebacks only fire on already-failing paths, the equivalent suites passed on the same commit for the shards that did run (core-diff, core-schema, graphql, functional, docker-integration), and the 2233-test unit suite plus ty and mypy are clean locally. If the rebased run reproduces the same three-way failure, that stops being flake and I will escalate with the two log-volume suspects named (services/scheduler.py:91 in the keep-alive loop, database/__init__.py:446 on ServiceUnavailable).

No code changed in this pass — rebase only. Nothing merged.

[saltas888]

CI triage — failures look environmental, failed jobs re-run

Automated reconcile pass. CI came back red on 7 jobs; I looked at each failure's actual cause before touching anything, and none of them is an assertion failure related to this diff.

Failing job(s)	Root cause in the log
backend-tests-component (core-diff, core-schema, other)	Exit code 137 — SIGKILL/OOM. Logs show [gw0] node down: Not properly terminated and replacing crashed worker before Process completed with exit code 137. The xdist workers were killed, not failed.
backend-tests-integration	neo4j.exceptions.TransientError: {neo4j_code: Neo.TransientError.Transaction.DeadlockDetected} in test_profile_lifecycle.py — a transaction deadlock, explicitly a transient error class.
E2E-testing-pytest-playwright (branches_repo, foundation, sites_a)	infrahub_sdk.exceptions.ServerNotResponsiveError: Unable to read from '.../api/schema/load?branch=main'. (timeout: 120 sec) — the server did not respond within 120s, consistent with the same resource starvation.

Three independent signatures — OOM kill, a Neo4j transient deadlock, and a server read timeout — all consistent with a resource-constrained runner. A log.error → log.exception substitution cannot produce any of them: it changes no control flow, no query, no transaction, and keeps the same log level.

Corroborating: #10218 (an unrelated, human-authored ruff-ignores cleanup) shows the same E2E-testing-pytest-playwright + backend-tests-component failures on the same day.

Branch staleness was checked first and ruled out — pha/INBOX-29 is 0 commits behind develop.

Action taken: re-ran the failed jobs (run 31515711302). No code changed. If the re-run comes back green, this was runner flake and the PR stands as reviewed. If the same jobs fail again with the same signatures, that points at CI capacity rather than this PR, and I'll escalate for a human rather than keep re-running.

For reference, local verification on this branch was and remains green: full ruff check clean, ty + mypy clean (1610 files), and 2233 backend unit tests pass.

[saltas888]

🤖 CI triage — the 4 red checks are infrastructure, not this change. Re-ran the failed jobs; no code or branch change.

E2E-testing-pytest-playwright (foundation, sites_a, branches_repo) — conclusive. Every test in each shard reported ERROR at setup, i.e. none of them ran. The per-shard docker stack never came up:

dependency failed to start: container infrahub-test-861af0a0-message-queue-1 is unhealthy
subprocess.CalledProcessError: Command '['docker','compose','--project-name','infrahub-test-861af0a0','up','--wait']' returned non-zero exit status 1
scraper-1 | error in A lookup for "infrahub-server": ... server misbehaving

An unhealthy RabbitMQ container and DNS resolution failures on the runner. Nothing in a logging change can prevent a container from passing its healthcheck.

backend-tests-component (other) — same conclusion, slightly weaker evidence. The pytest session died with INTERNALERROR> KeyError: <WorkerController gw2> in xdist's loadscope scheduler — that's the crash signature of a worker process dying mid-collection. Because the session aborted, no tracebacks were emitted for the 3 reported failures, so there is nothing in the log tying them to any code:

• core/test_node.py::test_node_create_with_multiple_relationship
• core/test_node_get_list_query.py::test_query_NodeGetListQuery_filter_with_generic_profiles
• core/test_node_get_list_query.py::test_query_NodeGetListQuery_order_with_profiles[main]

All three are node-creation / list-query tests. This PR touches no code they exercise: the only changed file anywhere near that path is backend/infrahub/database/__init__.py, and the change there is confined to the except ServiceUnavailable handler, which re-raises identically and cannot alter a query result. The same runner was demonstrably unhealthy for the E2E jobs in this run.

For reference, develop is currently red too (backend-tests-component (other), two E2E shards, backend-docker-integration, backend-benchmark, E2E-version-upgrade) — though on a different test (trigger/test_zombie_detection.py), so I'm not claiming an identical pre-existing failure, only that the infrastructure is unstable right now.

If the re-run comes back red on the same three tests with actual tracebacks, that changes the picture and I'll dig into the code rather than the runner.

The checks that actually gate this change were green in the same run, and locally: ruff (0 TRY400, no new violations), ty, mypy (1610 files), and 2233 backend unit tests.

[saltas888]

CI status: 43/45 green — the 2 failures are a repo-wide infrastructure problem, not this PR

Flagging this rather than pushing blind fixes at it.

What fails: E2E-testing-pytest-playwright shards foundation and branches_repo. Neither ran a single test — every case errored in the session fixture at tests/e2e/conftest.py:222:

RuntimeError: Failed to start docker compose:
Stdout:

Stderr:

The stack never booted and compose returned empty stdout/stderr. foundation bailed after 5 fixture errors with 207 tests deselected.

Why it isn't this branch:

• develop fails identically. Run 31506529544 hits the same conftest.py:222 empty-log Failed to start docker compose in shards tutorial and sites_b, alongside backend-docker-integration, backend-tests-component, backend-benchmark and E2E-testing-version-upgrade. develop last went green at 09:40 UTC; 13:22 and 15:20 both failed.
• Not stale — 0 commits behind develop, so there's nothing to rebase onto.
• Re-runs exhausted — already on attempt 3, failed all three. Not a transient runner flake.
• This diff cannot affect container startup — no Dockerfile, no compose file, no YAML, no dependency change, no uv.lock edit. Only Python logging calls, ruff config in pyproject.toml, spec docs and a changelog fragment. uv lock --check passes.
• Everything the change actually touches is green — ruff, ty, mypy, and the 2233-test backend unit suite, in CI and locally.

Ask: this needs either a fix to the e2e docker-compose boot on the runners (which is blocking develop too, so it's bigger than this PR), or a call that the failure is unrelated and this merges on the strength of the 43 passing checks. I won't merge and won't keep retrying — INBOX-29 is flagged pha-needs-human.

Reminder: this PR is the TRY400 half only. TRY004 remains suppressed and needs its own card, so merging this shouldn't close INBOX-29.

[saltas888]

CI triage — root cause found: runner Docker address-pool exhaustion. Escalating.

Automated reconcile pass 4. Down to 2 red jobs from 7, both E2E-testing-pytest-playwright (foundation, branches_repo). This pass I got a definitive root cause, so I'm stopping the rerun cycle and handing it to a human.

The failure is docker network create, not a test

Both shards die in the session-scoped compose fixture before a single test runs (10.8s and 20.9s). The --log-failed output shows empty stdout/stderr, but the full job log has the real cause:

Error response from daemon: all predefined address pools have been fully subnetted

The runner's Docker daemon cannot allocate another network. And the reason is in the same log's "Containers after tests" step: the runner is carrying 86 distinct leaked infrahub-test-* stacks — 170 containers still Up 2–4 hours, nearly all (unhealthy):

02ed631b3a45  rabbitmq:4.2.1-management  Up 2 hours (unhealthy)  infrahub-test-dd41feb5-message-queue-1
a3900b440fdd  rabbitmq:4.2.1-management  Up 3 hours (unhealthy)  infrahub-test-ac481617-message-queue-1
4aed856c60a2  rabbitmq:4.2.1-management  Up 4 hours (unhealthy)  infrahub-test-bf3e2b93-message-queue-1
…

Each orphaned stack holds a Docker network. Enough of them and the address space is gone, after which every E2E job scheduled on that runner fails at boot regardless of its diff. Affected runners here: ghrunner_1 and ghrunner_2.

Why this is not this PR

• The failure is at the daemon level, before any container — let alone any Infrahub code — starts.
• The leaked stacks are 2–4 hours old, predating this run.
• The same Failed to start docker compose is currently failing develop (run 31506529544, shards tutorial + sites_b, with dependency failed to start: container …message-queue-1 is unhealthy) and PRs release-1.10.7, po-remove-dead-ruff-ignores, bgi-academy-doc.
• The failing shard set keeps shifting between runs (sites_a, sites_b, branches_repo, foundation, tutorial) and shrinking (7 → 2) with no code change. A deterministic regression does not move around.

This closes out the one credible code-related hypothesis

Reconcile pass 2 flagged, and deliberately did not dismiss, the possibility that log.exception adds log volume and that the timeouts/OOM were "something got slower or noisier" — naming services/scheduler.py:91 and database/__init__.py:446 as suspects, and committing to escalate with them if the failure recurred.

That hypothesis is now dead, not merely unconfirmed: the blocked resource is a Docker network, allocated before any process in any container executes. The containers exhausting the pool are rabbitmq:4.2.1-management instances holding zero Infrahub code. No amount of log volume from this diff can subnet a network. I'm retiring the suspicion explicitly rather than leaving it hanging.

Why I'm not re-running again

Pass 3 already re-ran the failed jobs; this is that rerun's result. A rerun lands on the same self-hosted runners with the same exhausted pool, so further reruns are thrash, not progress. No fix is available inside this diff, and I'm not pushing a speculative one.

What a human needs to do

1. Prune the leaked stacks on the E2E self-hosted runners — docker compose -p <project> down -v --remove-orphans per orphan, or a docker network prune / docker system prune sweep. That should unblock this PR and the others.
2. Fix the teardown leak that produced 86 orphans — jobs cancelled or SIGKILLed mid-run skip fixture teardown and leave the stack (and its network) behind. Worth a separate ticket; this will keep recurring.

State of this PR

Everything this change can deterministically affect is green: python-lint, backend-tests-unit (3.12/3.13/3.14), backend-validate-generated, graphql-schema, json-schema, markdown-lint, documentation, validate-release-notes-style, backend-tests-component, backend-tests-functional, backend-docker-integration — 43 pass, 11 skipped, 2 infra-red. Branch is level with develop (0 behind), working tree clean, nothing force-pushed this pass.

No code changed in this pass. Nothing merged. Escalated on INBOX-29 with pha-needs-human.

[saltas888]

Response to cubic-dev-ai review — 2 valid (1 is a real inconsistency in my own reasoning), 1 rebutted

Triaged all three findings against the code. No changes pushed in this pass: the card carries pha-needs-human and is escalated on the CI infra blocker above, so the content is a human's call now. Each item below is stated precisely enough to apply in seconds.

---

✅ VALID — backend/infrahub/graphql/app.py:195 (ClientDisconnect) — and it contradicts my own stated rule

The bot is right, and this is the finding I most want to own rather than defend.

My own rule in research.md §R4 was: convert unless the traceback would be actively harmful or worthless. A Starlette ClientDisconnect is a routine event — a user navigating away or cancelling a slow query mid-request-read. Its traceback shows the body-read path and is non-actionable. By my own criterion it belonged with the six noqa sites, not with the conversions. My planning notes even called it "a routine, expected condition" and then converted it anyway without a good reason. That's an inconsistency, not a judgement call I'd defend.

Recommended fix (one line + reason, matching the other six):

        except ClientDisconnect as exc:
            # A client aborting mid-request is routine; its traceback is non-actionable noise.
            self.logger.error("Exception ClientDisconnect in _handle_http_request")  # noqa: TRY400
            return JSONResponse({"errors": [str(exc)]}, status_code=400)

Note the pre-existing question the bot gestures at but doesn't ask: this site arguably shouldn't be ERROR at all. Changing the level is out of scope here (this PR preserves levels by design), but it's worth a follow-up.

✅ VALID — dev/specs/.../tasks.md stale line numbers

Confirmed exactly, every claim checks out against the merged tree:

tasks.md says	actually now
process.py:204	:208
infrahub_async.py:194	:196
integrator.py noqa 456 / 638	458 / 644
integrator.py noqa 459 / 641	463 / 649
integrator.py:810	:818 (and no longer a log.error site)

The numbers were measured pre-conversion and the conversions shifted them. As an archived record that's misleading — someone opening integrator.py:810 later finds nothing. The suggestion to use symbol anchors (get_check_definition, get_python_transforms, the except ValidationError handler in _get_jinja2_transforms) instead of absolute lines is the right shape.

Deferred rather than dismissed: it's a docs-only edit to dev/specs/, and pushing now would restart CI onto the same exhausted runners and add churn to an escalated PR. Worth doing in the same pass that applies the app.py fix above.

❌ REBUTTED — "should target stable, not develop"

Two independent reasons this is wrong here:

1. The premise is factually incorrect. The finding characterises this as "primarily repo tooling — lint config plus spec docs — which cannot affect a running product." It isn't. The diff converts 28 runtime logging call sites across 12 backend modules (git/, core/merge/, core/branch/, database/, graphql/, services/, webhook/, workers/). log.error → log.exception changes what production emits — that is exactly why the webhook site's record-dropping hazard mattered enough to document. Lint config is the enabler; the runtime logging change is the substance.

2. The closest precedent went to develop. #10002 — "Re-enable ruff BLE rule and fix all violations", the near-identical predecessor to this change — merged to develop. So did #10147 and the currently-open #10218 ("remove ruff ignore rules that no longer suppress anything"). The two ruff PRs that did target stable (#9199, #8669) were pure config/docs with no source changes — the distinction the guidance is actually drawing, and this change falls on the develop side of it.

Keeping develop. Happy to be overruled by a maintainer who owns the release-vehicle policy.

---

Net: one genuine code inconsistency to fix (app.py:195), one docs accuracy fix, both deferred to the human who unblocks the CI infra. Nothing pushed, nothing merged.

[saltas888]

🤖 Re-ran the two stale E2E shards — the blocker is confirmed clear on today's develop

No code changed, nothing pushed, nothing rebased, nothing merged. This card is escalated (pha-needs-human), so content decisions stay with a human; a CI re-run mutates nothing here, which is why it's the one action taken.

The two red checks are a 3-day-old result, and the shards pass on current develop

The failing run on this PR is 31515711302, attempt 3, created 2026-08-11 17:04 UTC — the run that produced the "Docker address-pool exhaustion" diagnosis. Nothing has run here since.

develop's newest run (31787850882, 2026-08-14 09:23 UTC) is itself a failure, which could look like the outage returning. It isn't — the failures are different jobs:

Job	develop, today 09:23 UTC	this PR
E2E-testing-pytest-playwright (foundation)	✅ success	❌ (stale, 08-11)
E2E-testing-pytest-playwright (branches_repo)	✅ success	❌ (stale, 08-11)
other 3 pytest shards + version-upgrade	✅ success	✅
frontend-tests, backend-docker-integration, E2E-testing-playwright	❌ failure	✅ / not red here

Both shards red here pass on current develop, so the compose fixture boots on the runners again.

Action: gh run rerun 31515711302 --failed → attempt 4 queued (11:03 UTC)

An earlier pass declined to re-run because "a rerun lands on the same runners with the same exhausted pool, so further reruns are thrash." The table above falsifies that premise, so the abstention no longer applies.

What the outcome will mean:

• Green → 45/45 checks pass; ready to merge on review.
• Red, same empty-log Failed to start docker compose / address-pool signature → runner problem recurred; escalation stands.
• Red with real test tracebacks → falsifies the infra explanation and the code needs a look. That falsifier has been on the table since pass 3 and still stands.

Shards take ~2.5h, so the result lands after this pass.

Still with a human

1. Merge decision — this agent never merges.
2. The two conceded cubic-dev-ai findings — graphql/app.py:195 (the ClientDisconnect inconsistency I conceded) and the stale tasks.md line numbers. Both are one-liners, both still unpushed on purpose: the card is escalated, and pushing them restarts a full CI run, so they're best applied together with whatever follows this re-run.
3. TRY004 — 40 sites on core/schema/ and GraphQL mutation error surfaces, untouched and still suppressed. Merging this PR delivers the TRY400 half only; it should not close INBOX-29.
4. The runner teardown leak — 86 orphaned infrahub-test-* stacks were what exhausted the address pool. Deserves its own ticket; it will recur.

Branch is now 5 commits behind develop (all frontend-only: popover/focus-ring/background tokenization, logo, offscreen-diff perf). No rebase — staleness isn't what's failing, and rebasing is gated off by the escalation. Local verification unchanged: 0 TRY400, full ruff clean, ty + mypy clean (1610 files), 2233 backend unit tests pass.

[saltas888]

🤖 Reconcile pass — CI was red, cause was a stale branch. Rebased onto develop; nothing in this PR changed.

The single failure was E2E-testing-pytest-playwright (branches_repo):

FAILED tests/e2e/branches/test_branch_details.py::TestBranchDetailsNonDefaultBranch::
       test_display_branch_name_and_no_default_badge[chromium]
AssertionError: Locator expected not to be visible — waiting for get_by_text("default")

Unrelated to this PR's diff (Python logging calls only, zero frontend files). The actual history:

• Deprecate Branch.has_schema_changes use Branch.schema_differs_from_default_branch #10086 (Aug 10) renamed the branch attribute label Has schema changes → Schema differs from **default** branch. The assertion in that test used a non-exact get_by_text("default"), which then also matched the new label — and the same test requires that label to be visible three lines later. Self-contradictory once it paints, so it passed or failed on render timing.
• perf: skip rendering offscreen diff cards and drop per-icon visibility observers [INFP-677] #10252 (today) fixed it by making the matcher exact=True, matching how the sibling default-branch test already did it.

This branch was 7 commits behind develop and so was still running the pre-fix test. Rebasing picked up the fix.

What I did: rebased onto develop (7 commits, all frontend, no conflicts) and force-pushed with --force-with-lease. No source or test file was edited — the earlier commits replayed unchanged.

Re-verified on the new base:

Check	Result
ruff check . --exclude python_sdk (CI's exact command)	clean
ruff check --select TRY400	0 in repo scope

CI is re-running. I'll reconcile again when it settles.

[saltas888]

CI status — the blocker changed, and it now points at this PR

Nothing pushed in this pass. This PR is escalated (pha-needs-human on INBOX-29) and had activity 33 minutes ago, so the automated pass deliberately did not touch the branch. Posting the diagnosis so it isn't lost.

What's failing now

The previously-escalated failure — E2E-testing-pytest-playwright (branches_repo), attributed to exhausted self-hosted runners — now passes. A different job fails instead:

backend-tests-component (core-schema) — 1178 passed, 8 skipped, 7 errors, every one Failed: Timeout >300.0s, all in backend/tests/component/core/convert_object_type/test_convert_object_type.py. The first is a class-scoped setup timeout. (run)

Why this is harder to write off as infrastructure

• The same fixture takes 33.7s setup on a green develop run 90 minutes earlier — hitting 300s is a ~9× blowup, not a marginal threshold flake.
• That green run's commit is f7b5f884, which is exactly this branch's merge-base. Same develop, same shard, same test: green there, timeout here. The only difference is this PR's 3 commits.
• core-schema passed on the last 5 develop runs (5/5) — the shard isn't generally flaky.
• A concrete path from the hang to changed code: the timing-out fixture runs lifespan(app) → app_initialization → service.scheduler.start_schedule() (server.py:129-130). services/scheduler.py:91 is one of the sites this PR converted to log.exception, and tests/helpers/test_app.py:141 notes the lifespan deliberately emits an ERROR through a local worker — so the hung fixture does exercise changed code on an error path.

The honest counter-argument

Every failure is a timeout, never an assertion; 1178 tests in the same job passed; and I could not identify a mechanism by which log.error → log.exception yields a 9× slowdown — traceback rendering is bounded work. A contended runner can push one heavy fixture past a timeout while lighter tests pass. Causation is not proven either way.

Cheapest decisive next step

Re-run just backend-tests-component (core-schema).

• Green → it was contention; the prior infra escalation stands.
• Reproduces → revert the one-line services/scheduler.py conversion to log.error + a justified # noqa: TRY400 and re-run. That isolates the scheduler hypothesis in one cycle without disturbing the other 27 conversions.

Also still queued from the previous pass and unaffected by this: the app.py:195 ClientDisconnect fix (agreed valid) and the tasks.md stale-line-number anchors.

_{🤖 platform-health-agent, INBOX-29}

[saltas888]

The core-schema timeout was runner contention, not this PR — and the two agreed review findings are now fixed

The CI question is settled

The previous pass laid out a decisive test: re-run only backend-tests-component (core-schema) on an unchanged tree.

• Green → contention.
• Reproduces → revert the services/scheduler.py conversion to isolate the scheduler hypothesis.

I re-ran it on the identical tree (4af080e6, no new commit — deliberately, since this workflow uses concurrency: cancel-in-progress: true, so pushing first would have cancelled the experiment).

Result: green, in 9m27s — against a previous failure of 7 × Timeout >300.0s in the same job. A run that needed ≥35 minutes of timeouts finished in under ten.

That refutes the scheduler hypothesis. My prior comment argued the circumstantial case for this PR being the cause — same merge-base green on develop, 5/5 recent develop runs green, and a real code path from the hung fixture into services/scheduler.py:91. That case was wrong: identical tree, identical shard, opposite outcome, with only the runner differing. The honest counter-argument in that comment — every failure a timeout, never an assertion, and no mechanism by which log.error → log.exception yields a 9× slowdown — was the correct read.

No revert needed. services/scheduler.py keeps its conversion. The whole PR was green at that point.

Two review findings fixed (0688bc1)

1. graphql/app.py — ClientDisconnect reverted to log.error + # noqa: TRY400. Agreed, and this one contradicted the change's own stated criterion rather than being a judgement call worth defending. research.md §R4 says convert unless the traceback would be actively harmful or worthless; a client aborting mid-request is routine and the traceback only shows the body-read path. It belonged with the other suppressed sites from the start. Counts are now 27 conversions / 7 in-line noqa (was 28/6), propagated through research.md §R4, plan.md, and the implementation report.

2. tasks.md — sites re-anchored on enclosing function names. Every claim in that finding checked out; the numbers were measured pre-conversion and the conversions shifted them (process.py:204 → :208, infrahub_async.py:194 → :196, and integrator.py:810 no longer a log site at all). Fixing finding 1 shifted them again, which makes the point better than I could. The task list now names functions (get_check_definition, webhook_send, run_schedule, …) and tells the reader how to regenerate the set: ruff check --select TRY400 . before, grep -n 'noqa: TRY400' after.

The third finding (target stable rather than develop) remains rebutted and resolved — the diff changes 27 runtime logging call sites across 12 backend modules, so it is not repo-tooling-only. Happy to be overruled by whoever owns that policy.

Verification of this push

ruff check . --exclude python_sdk (CI's exact invocation) clean · ty clean · mypy clean, 1610 files · 2233 unit tests pass · TRY400 violations still 0, TRY004 still suppressed at 40.

CI is re-running now on 0688bc1.

_{🤖 platform-health-agent, INBOX-29}

[cubic-dev-ai]

All reported issues were addressed across 5 files (changes from recent commits).

<sub>Shadow auto-approve: would not auto-approve because issues were found.
Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[saltas888]

CI triage: the one red check is infra flake, not this change

E2E-testing-pytest-playwright (foundation) failed on run 31808295556. Everything else on that run is green — 45 pass / 1 fail, including all four other pytest-playwright shards (branches_repo, tutorial, sites_a, sites_b), all three backend-tests-unit matrix legs, backend-tests-integration, backend-tests-functional, every backend-tests-component shard, and python-lint.

Why it is not this change

The stack trace lands in backend/infrahub/workers/infrahub_async.py — a file this PR touches — so that needed ruling out explicitly. The root cause is:

infrahub_sdk.exceptions.ServerNotResponsiveError:
  Unable to read from 'http://infrahub-server-lb:8000/graphql/main'. (timeout: 60 sec)

The task-worker could not reach the server during stack startup, so it exited via the pre-existing raise typer.Exit(1) from err; the five test errors that follow are all ServerNotResponsiveError on /api/schema/load (120 s timeout), then stopping after 5 failures. That is the compose stack not coming up in time — nothing in this diff affects whether the server is reachable, only how the failure is logged.

Two further points:

• The shard failures move around. This PR fails foundation while sites_a passes; develop at head fails sites_a (plus frontend-tests, backend-tests-integration, backend-tests-functional). A given shard failing on one run and passing on the next is the signature of a startup-timing flake, not a regression.
• develop is currently red on its own, and on jobs this PR passes. That is also why I have not rebased: the branch is 5 commits behind, but none of those commits touch e2e, CI, worker startup, or docker/compose, so staleness cannot explain this failure — and rebasing onto a base that is failing backend-tests-integration/functional would import real breakage in order to chase a flake. Happy to rebase once develop is green again.

An accidental demonstration

The ServerNotResponsiveError traceback visible in that CI log is produced by this PR. The converted call is:

except SdkError as err:
    self._logger.exception(f"Error in communication with Infrahub: {err.message}")
    raise typer.Exit(1) from err

Before this change that handler logged the message with no stack, so a worker that failed to reach the server gave you one line and nothing to trace. The failing job is, incidentally, a live example of the debuggability the rule is meant to restore.

Action taken

1. Re-ran the failed shard only (gh run rerun --failed) rather than rebasing or pushing a code change, since the diagnosis points at environment rather than content.
2. Addressed the five open review threads — all valid, all the same stale-count issue: the earlier ClientDisconnect revert moved one site from converted to noqa, and plan.md, research.md, tasks.md, alignment-check.md and the implementation report still said 28/6. Fixed in d085964 and reconciled against the tree, not just internally: 7 noqa: TRY400 comments + 2 deferred via the auth/auth.py per-file-ignore + 27 converted = 36. The PR description above has been corrected too — it still carried the stale 28/6. The dated critique keeps its original figure, marked "then-28", rather than being rewritten after the fact.

Note that pushing (2) cancelled the re-run from (1), so the flake retest is inconclusive; a full fresh run is now in flight on d08596484 and will exercise the shard again. No source or config change was needed for the CI failure itself — the only commit here is documentation.

[cubic-dev-ai]

0 issues found across 6 files (changes from recent commits).

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

<sub>Shadow auto-approve: would auto-approve. Re-enables ruff TRY400, converting log.error→log.exception with documented noqas at ~36 sites, preserving log level, message, and keyword fields. Verified by clean lint and 2233 passing tests; no contract, auth, or operational behavior change requiring human tradeoff.

Re-trigger cubic</sub>

[saltas888]

🤖 platform-health-agent — automated reconcile pass (CI red).

Action: rebased onto latest develop and force-pushed (d0859648 → 2660959d9). The branch was 6 commits behind; CI has re-run from the rebased head.

Why a rebase rather than a code fix

I checked each of the 5 failures against this diff first. None of them implicate it — python-lint, all three backend-tests-unit matrices, backend-tests-integration, backend-tests-functional and backend-docker-integration were all green on the previous run:

Failing job	Actual failure	Related to this PR?
backend-tests-component (core-schema)	exit 137, node down: Not properly terminated, replacing crashed worker gw1 — the xdist worker was SIGKILLed (OOM)	No — runner resource crash, no assertion failed
backend-tests-component (other)	test_git_askpass.py::test_askpass_password — unexpected POST /api/logs/, "Error logging to API"	No — passes locally on this branch, 2/2; test-isolation flake
E2E-testing-pytest-playwright (branches_repo)	test_breadcrumb.py::TestIpamBreadcrumb — get_by_test_id("breadcrumb-ipam") not visible	No — frontend; this diff touches zero frontend files
E2E-testing-pytest-playwright (sites_b)	ServerNotResponsiveError on /graphql/main and /api/schema/load (60s/120s timeouts)	No — the stack never came up
E2E-testing-playwright	branch-selector.spec.ts — toHaveValue, element not found	No — frontend

The two UI failures are the informative ones: three of the six commits this branch was missing are frontend token refactors — "tokenize the sheet frame and surface colors", "tokenize primary and muted text colors", "add Card secondary and panel surface variants" — i.e. exactly the surfaces those locators target. develop has also been red repeatedly today on its own. A stale branch is the boring explanation, so it was the first thing to eliminate.

Rebase outcome

Clean — no conflicts, despite develop concurrently landing #10218 / da5150c0c in the same pyproject.toml ruff-ignore region. Both this PR's extend-select = ["TRY400"] and the auth/auth.py per-file-ignore survived intact and were re-verified.

Re-verified locally on the rebased head:

• ruff check . — clean; 0 TRY400 in the repo
• TRY004 — still exactly 40, unchanged
• invoke lint — ruff + ty + mypy all pass (mypy: 1610 files)
• invoke backend.test-unit — 2233 passed
• Changed-file audit — no core/schema/, core/migrations/, auth/, .github/, no generated file

One clarification for anyone re-running the numbers: ruff check --select TRY400 . reports 12 hits in python_sdk/. Those are in the submodule (its own repo, its own ruff config) and CI lints with --exclude python_sdk; the default project lint is clean. Likewise 2 of the 42 TRY004 hits are SDK-side — 40 in this repo, as before.

Nothing in the working tree changed beyond the replay: the diff is byte-for-byte the same set of files as before the rebase.

If CI comes back red on the same UI specs from the rebased head, that points at develop rather than this branch and I'll escalate to a human rather than keep retrying.

[saltas888]

CI triage: red, but not from this PR

Reconcile pass on the red CI. None of the 6 failures is attributable to this change, and I've re-run the failed jobs. Evidence below.

Branch is not stale

origin/pha/INBOX-29 is 0 commits behind origin/develop — no rebase needed, so staleness isn't the cause.

What each failure actually is

Job	Actual failure	This PR?
backend-tests-component (core-schema)	10 errors, all Failed: Timeout >300.0s in fixture setup (test_relationship_profiles_kind.py, test_relationship_peer_relatives.py); 1175 passed, 0 assertion failures	No — fixture/runner hang
backend-tests-integration	Reached 98% with every test PASSED, then killed at 30m18s ("Terminate orphan process")	No — job timeout
E2E-pytest-playwright (branches_repo)	test_create_a_branch_with_a_name_that_does_not_exist — AssertionError: Locator expected to have Value 'quick-branch-form'	No — a frontend UI assertion; this PR contains zero frontend files
E2E-playwright, (sites_a), (sites_b)	Same E2E suite	No

This diff is Python logging calls (log.error → log.exception), one ruff config block, a changelog fragment, and spec docs. It cannot produce a frontend locator assertion, and a level-preserving logging substitution does not hang a database fixture for 300s.

develop is red right now, independently

Latest develop CI run (31813036391) fails on:

• E2E-testing-pytest-playwright (branches_repo) — RuntimeError: Failed to start docker compose ← same job that fails here
• E2E-testing-pytest-playwright (tutorial)
• backend-tests-component (other) — a sibling shard of the one failing here
• backend-docker-integration

The three preceding develop runs are also failure/cancelled.

The timeline is the clincher

This branch was fully green at 12:04 UTC today — run 31798777919, SHA 4af080e66. It went red only after being rebased onto the currently-broken develop. The rebase pulled in ~120 changed files of develop commits (frontend token refactors, ruff-ignore cleanups); my own commits since that green run were the ClientDisconnect noqa revert plus docs.

Action taken

Re-ran the 6 failed jobs on run 31817672397 — currently in progress. The timeout-class failures (component, integration) have a good chance of clearing on a healthier runner. branches_repo may well stay red, since it is broken on develop too — that one needs a fix on develop, not here.

No code changes made: there is nothing in this diff to fix.

[saltas888]

CI triage — both failures are external to this PR

Automated reconcile pass. Two jobs were red on run 31817672397; I've re-run just those two. Neither is caused by this change, and I've made no code changes in this pass.

E2E-testing-playwright — broken on develop

Mass 180s timeouts across unrelated frontend UI specs (object-hierarchy-tree-list, file-upload, object-metadata, object-header-sort, object-sort, object-relationships, object-update), all locator.click / toBeVisible timeouts.

The same job fails on develop (run 31823106064) with the identical signature — Test timeout of 180000ms exceeded, toBeVisible failures on a different set of UI specs (global-activities, object-groups). develop's last three CI runs are all red, that one failing 9 jobs.

This PR changes no frontend file, no GraphQL schema, and no generated frontend type. frontend-lint passes and frontend-tests / frontend-validate-error-catalogue are skipped precisely because no frontend path is touched. This is a pre-existing develop breakage that arrived at this branch through its base.

backend-tests-integration — flaky

2 failed / 451 passed, alongside a pytest-xdist scheduler crash:

INTERNALERROR> KeyError: <WorkerController gw4>

i.e. a test worker died and took the run's reporting with it. The two tests were test_step02_remove_on_default_then_rebase and test_cross_branch_cardinality_one_never_exceeds_one.

Those two live in schema-rebase and merge-cardinality paths, and this PR does touch two files in the merge/rebase code (core/merge/orchestrator.py:152, core/branch/tasks.py:119), so I checked that link rather than assuming. It's ruled out:

• Commit 4af080e66 on this branch had a fully green CI run (2026-08-14 12:04Z) and already contained both of those log.exception conversions. Identical code, green.
• The only code change since then (d9893d27) reverts graphql/app.py's ClientDisconnect handler back to log.error — an HTTP-disconnect path that cannot reach schema-rebase or merge-cardinality logic.
• The previous red run on this branch (d0859648) failed a disjoint set of jobs — component (other), component (core-schema), and three E2E jobs — with backend-tests-integration passing. Two consecutive runs of near-identical code failing different job sets is flakiness, not a regression.
• develop's own backend-tests-integration also fails, on a different test (test_run_generators_validate_requested_jobs — ResourceNotFoundError: Diff summary for pipeline … not found in the cache).

Both log lines only execute on failure/rollback paths, and both are level-preserving (.exception emits at error, adding only the traceback).

What passes

python-lint, all three backend-tests-unit matrix jobs (3.12/3.13/3.14), all four backend-tests-component shards, backend-tests-functional, backend-docker-integration, backend-validate-generated, validate-generated-documentation, graphql-schema, json-schema, markdown-lint, documentation, all five E2E-testing-pytest-playwright shards, and E2E-testing-version-upgrade.

Not rebased, deliberately

This branch is only 1 commit behind develop — not meaningfully stale — and develop is currently red on 9 jobs. Rebasing onto a broken base would import more failures, not fewer. Once develop is green again, a rebase here is the right move and should clear E2E-testing-playwright.

What would falsify the flakiness call: if the re-run fails again on those same two integration tests, treat it as a real regression and ping me — I'd then bisect the two merge-path log conversions directly.

[saltas888]

🤖 CI note — the red check is a pre-existing flake, not a regression from this PR. Re-ran the failed job; no code changed.

Exactly one job failed: E2E-testing-playwright (the legacy TS suite) — 241 passed, 1 failed. Everything else is green, including all backend unit / component / functional / docker-integration jobs and all five E2E-testing-pytest-playwright shards.

The failing test was branches/merged-branch-permissions.spec.ts, and it failed in beforeAll with:

apiRequestContext.post: socket hang up
  → POST http://localhost:33879/graphql
  at executeGraphQLMutation (tests/e2e/utils/graphql.ts:27)
  at mergeBranchAPI (tests/e2e/utils/graphql.ts:88)

I did not wave this off as flake on sight, because it fails on a branch-merge call and this PR does touch core/merge/orchestrator.py. Why it's unrelated:

1. The line changed in that file sits inside except BaseException — the merge failure path. On a successful merge it never executes, and log.exception cannot drop a TCP connection.
2. Every job that exercises merges far more directly passed: backend-tests-component (core-diff) and (graphql), backend-tests-functional, backend-docker-integration, and the branches_repo e2e shard.
3. The same job fails on develop without this change (run 31787850882) — there on a different single test (object-relationships.spec.ts, 238 passed). develop is broadly red at the moment with a rotating cast of failing jobs, which is the signature of flaky infrastructure rather than one regression.
4. executeGraphQLMutation retries only GraphQL-level MERGE_IN_PROGRESS errors — a transport-level socket hang up throws straight through the retry loop, so a busy server during a heavy merge fails the test outright.

No rebase: this branch is 2 commits behind develop and both are frontend-only CSS changes, which can't affect a transport error on a GraphQL call.

Two things a reviewer may want to pick up separately from this PR:

• the legacy E2E-testing-playwright suite dropping one arbitrary test per run is its own debt item;
• making executeGraphQLMutation retry transport errors (not just MERGE_IN_PROGRESS) is a small, concrete fix that would remove this failure mode.

[saltas888]

🤖 Reconcile pass — the re-run passed. This PR is fully green and unblocked.

Closing the loop on my previous comment: the E2E-testing-playwright flake (merged-branch-permissions.spec.ts, socket hang up in beforeAll) cleared on re-run.

45 success · 10 skipped · 1 neutral · 0 failed · 0 pending. First fully-green run on this branch.

Nothing pushed this pass — no code change, no rebase, no force-push.

State

• CI: green across the board, including all backend-tests-* and all five E2E-testing-pytest-playwright shards.
• Review threads: all 8 resolved. The two agreed cubic-dev-ai findings were fixed in d9893d27 / 2660959d (the ClientDisconnect site reverted to log.error + noqa, and the spec's per-site anchors switched from line numbers to enclosing symbols). Latest bot review: 0 issues, would auto-approve.
• Mergeable: yes.

On the 3 commits this branch is behind

Deliberately not rebased. All three are frontend-only (#10273, #10274, and the semantic-color refactor) — 146 files, zero Python, no pyproject.toml. They cannot introduce a TRY400 violation, so the green result is not stale in any way that matters, and a force-push would re-roll a 45-job CI cycle on a PR that has only just gone green after three days of runner-infrastructure flakes.

Ready for a human review decision. As always: I don't merge.