fix: remove generated template and profile schemas with their node (closes #10049)

[lancamat1]

Why

Loading a schema change onto a branch that removes a node with generate_template: true (via state: absent) leaves the branch permanently unable to converge: workers reloading the branch schema keep the orphaned Template<Kind> entry, crash with SchemaNotFoundError during process(), and never recover — the UI shows "Loading schemas…" forever, GraphQL on non-initiating workers misses fields from that and later loads, and every subsequent load on the branch re-triggers the failure. Reproduced on 1.10.1, 1.10.5, and current stable.

Goal: removing a node with generated kinds converges on every worker, and the branch stays usable.

Non-goals: the wider no-snapshot/no-rollback design of SchemaManager.load_schema (a mid-process() failure still leaves the in-memory registry half-updated) is a separate robustness issue and is not addressed here.

Closes #10049

What changed

Behavioral changes:

• Removing a node or generic from a branch schema now removes the Profile<Kind> and Template<Kind> generated from it at the same time, so a worker refreshing the branch rebuilds a consistent schema instead of failing on every refresh.
• A schema that somehow already holds a generated template with no backing node reprocesses cleanly and drops the orphan, rather than raising.
• An object template generated for a generic is now removed with that generic. It never was before, so a deleted or renamed generic left its Template<Kind> in the generics map permanently — and in the branch hash.

Implementation notes:

The production change is 20 lines in backend/infrahub/core/schema/schema_branch.py, in two layers:

1. SchemaBranch.delete() cascades to the generated kinds (_delete_generated_kinds). This enforces the invariant at the source: a kind leaving the schema map takes its generated kinds with it, so the orphaned state never exists. The template lookup checks self.templates and falls back to self.generics, because generate_object_template_from_node returns a GenericSchema for a generic and set() routes it into the generics map.
2. _generate_weight_templates skips a template whose backing node is missing. generate_weight() runs before manage_object_template_schemas() in process_pre_validation, so this keeps process() total for a registry that already holds an orphan. It mirrors the guard already present in _generate_generics_templates_weight.

An earlier revision of this branch shipped layer 2 alone, after layer 1 was reverted for breaking test_exhaustive_handles_same_uuid_vertices_after_kind_migration in CI. That failure was a test-harness artifact, not a production signal, so layer 1 is back.

Keeping both layers is deliberate. Layer 1 prevents the orphan; layer 2 means a reader that meets one anyway does not crash. Making each reader tolerant instead does not scale — _generate_generics_templates_weight already needed the same guard.

What stayed the same: no schema-format or API changes, process_pre_validation() ordering untouched, profiles and templates remain excluded from the branch hash.

Suggested review order

1. backend/infrahub/core/schema/schema_branch.py — the whole production change.
2. backend/tests/unit/core/schema/test_schema_branch.py — the cheapest expression of both invariants.
3. backend/tests/component/core/schema/schema_branch/test_process_idempotency.py — the save/reload behaviour.
4. backend/tests/component/graph_traversal/test_path_traversal_query.py — two lines, per above.

How to review

The tests are the part worth scrutinising. Each layer of the fix has exactly one dedicated guard, verified by removing each layer and re-running:

Code state	What fails
both layers present	everything passes
cascade removed	the two TestDeleteRemovesGeneratedKinds cascade tests (2 failed, 9 passed); the round-trip tests still pass, since process() prunes the orphan
skip removed	TestProcessWithOrphanedTemplate (1 failed, 10 passed)
both removed	TestSchemaBranchDbRoundtrip::test_removing_node_drops_its_generated_kinds — SchemaNotFoundError: Unable to find the schema 'TestGadget' at manager.py:1123

TestSchemaBranchDbRoundtrip saves the base schema to the database once for the class; each test then applies only its own change, on its own branch, writing a single kind. Two disjoint standalone nodes (TestGadget removed, TestDoodad flipped) keep the tests independent of execution order.

A functional test added earlier on this branch (backend/tests/functional/schemas/test_load_remove_template_node.py) was removed. Functional tests use a single shared registry (and SchemaManager), so this would not be capable for reproducing the bug. Several component tests that compare SchemaBranchs before and after updates have been added instead.

Unable to reproduce the generate_template flip causing a schema update crash

The issue states that flipping generate_template: true → false without deleting the node triggers the same failure, and concludes the trigger is "removal of the generated template, not node deletion per se". I was unable to reproduce this either in tests or against a local Infrahub app. Three independent checks:

1. _generate_weight_templates resolves template.name, which is the node kind. A flag flip leaves the node in place, so the lookup succeeds and manage_object_template_schemas drops the template normally.
2. On a live stack the flip converged cleanly on a clean instance while state: absent failed on that same instance. An earlier flip run on a contaminated instance did time out, but zero error lines named the flip branch and all 24 named the earlier state: absent branch — the sync check is global, so one wedged branch keeps it red. The issue's own steps run the removal first, which is the likely origin of the observation.
3. At infrahub-v1.10.1, _generate_weight_templates, delete(), and the removal handling in manager.py are materially identical to stable, so the flip could not have failed that way there either.

test_disabling_generate_template_drops_template covers the flip anyway, since the issue lists it under Expected Behavior — but it passes with both layers removed, so it is a convergence guard, not a regression test for this bug.

Testing

Verified end to end on a local deployment with 1 API worker and 2 task workers, same schemas and steps against both versions:

Scenario	stable @ c918af76e	this branch
Remove a generate_template node (state: absent) on a branch	"Schema is still not in sync"; both workers looping SchemaNotFoundError: Unable to find the schema 'DemoDoohickey'	Schema updated on all workers
Subsequent additive load on that same branch	n/a — branch already wedged	Schema updated on all workers
Flip generate_template: true → false on a branch	converges (never the bug)	Schema updated on all workers
SchemaNotFoundError in task-worker logs	18 branch-tagged failures	0

Impact & rollout

• Backward compatibility: none broken. No API or schema-format changes. A worker whose registry is already poisoned self-heals on its next schema refresh.
• Performance: negligible — two dict lookups per delete, and a try/except per template during weight generation. The component test consolidation cut that file's database work from three full save + reset + core-model-registration cycles to one (70.8s for the file).
• Config/env changes: none.
• Deployment notes: safe to deploy. Clean backport candidate for the 1.10.x line — the touched code is unchanged since 1.10.1.

Checklist

• Tests added/updated
• Changelog entry added (changelog/10049.fixed.md)
• External docs updated (if user-facing or ops-facing change) — N/A
• Internal .md docs updated (internal knowledge and AI code tools knowledge) — N/A
• I have reviewed AI generated content — pending human review

🤖 Generated with Claude Code

[opsmill-bug-pipeline]

AGENT_REVIEW_VERDICT: FIX_APPROVED

Overall verdict: APPROVED WITH SUGGESTIONS

The fix correctly addresses the root cause from #10049. A branch that removes a generate_template: true node no longer leaves an orphaned Template/Profile entry that crashes _generate_weight_templates() with SchemaNotFoundError during process(). Both changes are sound and complementary. One test-coverage gap is noted as a non-blocking suggestion.

A. Correctness — PASS

• Cascade in delete() (schema_branch.py:480-503) is the primary fix. Only the node/generic branches cascade to _delete_generated_kinds; the profiles/templates branches do not, so there is no recursion and no double-delete.
• Safety of the cascade holds under review. SchemaBranch.diff() (schema_branch.py:245-248) builds its kind map with nodes_and_generics_only=True, so diff.removed in update_schema_to_db (manager.py:314-317) and removed_nodes/removed_generics in the worker-refresh path (manager.py:1089-1094) only ever contain nodes/generics. A derived Template/Profile can therefore never appear as an independent removal that would collide with the cascade — the PR's claim checks out.
• The other delete() call sites are safe with the new cascade: process_nodes_state (schema_branch.py:2101-2105, the state: absent path) and the rename path (schema_branch.py:691-696) both benefit from dropping the derived kinds early, and the management functions regenerate profiles/templates as needed afterward.
• _generate_weight_templates skip (schema_branch.py:2159-2165) is a genuinely distinct self-heal path, not redundant with the cascade: a registry poisoned by the old code holds TemplateX with no X node. On refresh, X is already absent from the in-memory schema, so it never enters removed_nodes and the cascade never fires — only the skip lets process() reach manage_object_template_schemas (schema_branch.py:3036-3039), which then removes the orphan. Correct.
• _get_profile_kind/_get_object_template_kind are pure formatters, and _delete_generated_kinds guards with membership checks, so it is idempotent and cannot raise.

B. Code quality — PASS

• Minimal, targeted change (~16 lines); no unnecessary refactoring.
• The skip guard deliberately mirrors the pre-existing _generate_generics_templates_weight() pattern (schema_branch.py:2187-2190), keeping the two weight-generation methods consistent.
• Negligible performance cost (two dict lookups per node/generic deletion). No security concerns.

C. Documentation alignment — PASS

• No public API, schema-format, or GraphQL changes → no external docs needed. Changelog fragment changelog/10049.fixed.md is present and describes user-visible impact.
• Both new docstrings/comments follow .claude/rules/code-doc-style.md: they explain the why (the invariant "must not outlive the node they derive from"; "removed when the template schemas are managed") without naming specific collaborators or embedding issue IDs.

D. Test quality — APPROVED, one gap noted

• test_load_remove_template_node.py exercises the real worker-refresh path (registry.schema.load_schema) against a registry holding the pre-removal schema, asserts the correct end-state (node + Template + Profile gone, sibling Widget retained), and would fail on the buggy code with SchemaNotFoundError. It asserts expected, not buggy, behavior and is deterministic. Placement and TestInfrahubApp usage match project conventions.
• Gap (non-blocking): the test only covers the cascade path (change #1). Because the cascade removes the orphan before process() runs, the _generate_weight_templates skip (change #2) is never executed by this test — the self-heal path the PR body specifically calls out ("already-poisoned worker registries self-heal ... via the _generate_weight_templates skip") is untested. A regression that reverted only the try/except would still pass. Consider adding a case that seeds a SchemaBranch with a template whose backing node is already absent (no delete() call), then runs process() / generate_weight() and asserts it does not raise and the orphan is cleaned up. This is a suggestion, not a merge blocker: the fix is correct and the primary regression is well covered.

Recommended next steps

1. Merge is not blocked. Optionally add the self-heal test case described in D to lock in change #2 against future regressions.
2. No production-code changes required.

AGENT_REVIEW_ITERATION: fix-1

Generated by Bug reviewer agent for #10051 · 176.6 AIC · ⌖ 28.7 AIC · ⊞ 12.3K · ◷

[codspeed-hq]

Merging this PR will not alter performance

✅ 12 untouched benchmarks

---

_{Comparing ai-bug-pipeline-10049-template-node-removal (ef5ae26) with stable (0ad9131)}

[lancamat1]

Addressed the test-coverage gap from the review (section D): added TestProcessWithOrphanedTemplate in backend/tests/unit/core/schema/test_schema_branch.py — it seeds a processed SchemaBranch, drops the backing node from the map without the delete() cascade (mirroring a registry poisoned before this fix), reprocesses, and asserts the orphaned template and profile are removed. Verified it fails with SchemaNotFoundError when only the _generate_weight_templates try/except is reverted, so change #2 is now locked in independently of the cascade.

[cubic-dev-ai]

0 issues found across 1 file (changes from recent commits).

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

<sub>Shadow auto-approve: would auto-approve. Focused bug fix for orphaned template/profile schemas when a node is removed. The change adds a cascade in delete and a safety skip in template generation; tests cover the fix. Bounded and clearly beneficial.

Re-trigger cubic</sub>

[lancamat1]

CI failure analysis: backend-tests-component (other) failed on test_exhaustive_handles_same_uuid_vertices_after_kind_migration — the delete() cascade broke the delete-then-re-set idiom kind migrations use (delete(name=X) → modify → set(name=X)), dropping ProfileTestHub with nothing regenerating it since that flow doesn't reprocess. The cascade was also redundant: the _generate_weight_templates skip lets process() reach the template/profile management steps, which remove orphans from the maps on every reload path. Reverted the cascade; the fix is now the skip alone (~5 lines). Verified locally: the previously failing component test, the functional repro, and all schema unit tests pass. (The agent job failure was infra noise in the bot workflow — permission denied on its cache dir.)

[cubic-dev-ai]

0 issues found across 1 file (changes from recent commits).

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

<sub>Shadow auto-approve: would auto-approve. This is a targeted bug fix (5 lines) that prevents a schema reload crash by skipping orphaned templates, letting cleanup proceed. Bounded and clearly beneficial, with focused tests.

Re-trigger cubic</sub>

[cubic-dev-ai]

0 issues found across 3 files (changes from recent commits).

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

<sub>Shadow auto-approve: would auto-approve. Fixes a bug where removing a node with generate_template:true left orphaned template/profile schemas, causing worker crashes. The 5-line skip in _generate_weight_templates and the cascade deletion in delete() clean up generated kinds, with tests covering both paths.

Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 2 files (changes from recent commits).

Confidence score: 4/5

• In backend/tests/component/core/schema/schema_branch/test_process_idempotency.py, there’s a meaningful uncertainty around SchemaBranch.get_hash() potentially incorporating DB-assigned in-memory IDs after load_schema_to_db, which could break idempotency checks across save/load cycles and cause false change detection—add or tighten a round-trip assertion that hash input excludes transient database identifiers.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="backend/tests/component/core/schema/schema_branch/test_process_idempotency.py">

<violation number="1" location="backend/tests/component/core/schema/schema_branch/test_process_idempotency.py:279">
P2: Confirm that SchemaBranch.get_hash() is idempotent after a DB save/load cycle. If load_schema_to_db sets in-memory DB id attributes (e.g., node/relationship database identifiers) that feed into the hash computation, then a fresh SchemaBranch(cache={}, ...) loaded via load_schema_from_db will get different internal ids and produce a different hash than the 'saved_schema' that was mutated in-place by load_schema_to_db. If the hash is pure schema-model content (name, namespace, attributes, etc.), this is fine; if not, test_reload_matches_saved_schema may be flaky or perma-fail.</violation>
</file>

<sub>Shadow auto-approve: would not auto-approve because issues were found.
Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: Confirm that SchemaBranch.get_hash() is idempotent after a DB save/load cycle. If load_schema_to_db sets in-memory DB id attributes (e.g., node/relationship database identifiers) that feed into the hash computation, then a fresh SchemaBranch(cache={}, ...) loaded via load_schema_from_db will get different internal ids and produce a different hash than the 'saved_schema' that was mutated in-place by load_schema_to_db. If the hash is pure schema-model content (name, namespace, attributes, etc.), this is fine; if not, test_reload_matches_saved_schema may be flaky or perma-fail.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At backend/tests/component/core/schema/schema_branch/test_process_idempotency.py, line 279:

<comment>Confirm that SchemaBranch.get_hash() is idempotent after a DB save/load cycle. If load_schema_to_db sets in-memory DB id attributes (e.g., node/relationship database identifiers) that feed into the hash computation, then a fresh SchemaBranch(cache={}, ...) loaded via load_schema_from_db will get different internal ids and produce a different hash than the 'saved_schema' that was mutated in-place by load_schema_to_db. If the hash is pure schema-model content (name, namespace, attributes, etc.), this is fine; if not, test_reload_matches_saved_schema may be flaky or perma-fail.</comment>

<file context>
@@ -222,34 +253,128 @@ def test_process_idempotency(register_core_models_schema: SchemaBranch, idempote
+        schema = registry.schema.register_schema(schema=schema_root, branch=default_branch_scope_class.name)
+        # load_schema_to_db mutates the schema in place to assign DB ids, so the saved state is
+        # only final once it returns
+        await registry.schema.load_schema_to_db(schema=schema, db=db, branch=default_branch_scope_class)
+        return schema
+
</file context>

[ajtmccarty]

the IDs are persisted to the database and loaded from the database. so this is not an issue

[ajtmccarty]

pushed some changes and updated the description

1. re-added the Profile and Template schema deletes when the generating schema is deleted
2. replaced the functional test with a few component tests