bug: removing a node with generate_template: true on a branch permanently breaks schema convergence across workers

[lancamat1]

Component

API Server / GraphQL

Infrahub version

1.10.1 (reproduced identically on 1.10.5; defect present in current stable)

Current Behavior

Loading a schema change onto a branch that removes a node with generate_template: true (via state: absent) leaves the branch permanently unable to converge across workers:

• Every worker other than the one that processed the load fails to refresh the branch schema in a loop:

  SchemaNotFoundError: Unable to find the schema 'DemoDoohickey' in the registry
  Schema had not converged after 6.00 seconds, refreshing schema on local worker manually
  Finished in state Failed(... SchemaNotFoundError ...)
  

  raised from SchemaBranch._generate_weight_templates (schema_branch.py:2150 → SchemaBranch.get at :405), reached via refresh_branches from the schema-updated, hfid-setup, display-labels-setup-jinja2 and computed-attribute flows.

• infrahubctl schema load --wait times out with "Schema is still not in sync after the specified waiting time".

• Every subsequent schema load on that branch — even a purely additive one — re-triggers the same failure on whichever worker holds a stale copy, so the branch never becomes usable: the UI shows "Loading schemas…" indefinitely and GraphQL on non-initiating workers returns Cannot query field '<new_field>' on type '<Kind>' for fields added in the same or later loads.

• On 1.10.1 the orphaned Template<Kind>/Profile<Kind> are also served in the branch's /api/schema after the node deletion, and the breakage survives pod restarts and branch delete + recreate. On a multi-pod deployment (3× api-server, 3× task-worker) this manifests as InfrahubStatus.summary.schema_hash_synced: false permanently.

• Flipping generate_template: true → false without deleting the node triggers the same failure (verified on 1.10.1), so the trigger is removal of the generated template, not node deletion per se.

Removing a plain node (no generate_template) or removing an attribute via state: absent converges fine — this is specific to generated object templates.

Expected Behavior

A schema load that removes a generate_template: true node (or disables the flag) on a branch should converge on all workers, removing the node together with its generated Template<Kind>/Profile<Kind>, and the branch should remain fully usable.

Steps to Reproduce

On a clean instance from the official docker-compose (VERSION=1.10.1 or 1.10.5, ≥2 task-workers):

1. Load a base schema on main containing a node with generate_template: true:

   # base.yml
   version: "1.0"
   nodes:
     - name: Doohickey
       namespace: Demo
       generate_template: true
       attributes:
         - name: name
           kind: Text
           unique: true

   infrahubctl schema load --wait 60 base.yml → syncs fine.

2. infrahubctl branch create demo-absent-test

3. Load the removal onto the branch:

   # change.yml
   version: "1.0"
   nodes:
     - name: Doohickey
       namespace: Demo
       state: absent

   infrahubctl schema load --branch demo-absent-test --wait 90 change.yml

4. The wait times out; task-worker logs show the SchemaNotFoundError loop above. Any further schema load on the branch (even additive) fails to converge again.

Additional Information

Observed mechanism (verified on stable @ d8ba8dc15): the initiating worker processes a full candidate schema and persists a clean result, but other workers refresh via SchemaManager.load_schema → load_schema_from_db, which deletes only the removed node kind itself from the maps (manager.py:1085-1090; same in update_schema_to_db, manager.py:314-317). The derived Template<Kind> entry survives, and the subsequent schema.process() hits _generate_weight_templates — which unconditionally resolves each template's backing node — before manage_object_template_schemas (the step that would remove the orphan) runs. Because the reload mutates the live registry object in place, the failed worker is left half-updated with a matching top-level hash, so it never self-repairs.

Note: InfrahubStatus.summary.schema_hash_synced can report true while workers hold stale schemas in this state (the branch hash covers only nodes and generics), so it is not a reliable regression signal.

Related: #9899 (node removal → SchemaNotFoundError, fixed in v1.10.5 via #9905) covered the merge path; this branch-load path is unaffected by that fix. backend/tests/integration/schema_lifecycle/ has no test loading a generate_template node removal onto a branch via /schema/load.

Traceback (task-worker, 1.10.5)

File "/source/backend/infrahub/schema/tasks.py", line 25, in schema_updated
  await wait_for_schema_to_converge(branch_name=branch_name, component=service.component, db=db, log=log)
File "/source/backend/infrahub/workflows/utils.py", line 99, in wait_for_schema_to_converge
  await refresh_branches(db=db)
File "/source/backend/infrahub/tasks/registry.py", line 99, in refresh_branches
  await update_branch_registry(db=db, branch=active_branch)
File "/source/backend/infrahub/tasks/registry.py", line 77, in update_branch_registry
  await registry.schema.load_schema(db=db, branch=branch)
File "/source/backend/infrahub/core/schema/manager.py", line 1036, in load_schema
  branch_schema = await self.load_schema_from_db(...)
File "/source/backend/infrahub/core/schema/manager.py", line 1123, in load_schema_from_db
  schema.process(validate_schema=validate_schema)
File "/source/backend/infrahub/core/schema/schema_branch.py", line 707, in process
  self.process_pre_validation()
File "/source/backend/infrahub/core/schema/schema_branch.py", line 739, in process_pre_validation
  self.generate_weight()
File "/source/backend/infrahub/core/schema/schema_branch.py", line 2193, in generate_weight
  self._generate_weight_templates()
File "/source/backend/infrahub/core/schema/schema_branch.py", line 2150, in _generate_weight_templates
  node = self.get(name=template.name, duplicate=False)
File "/source/backend/infrahub/core/schema/schema_branch.py", line 405, in get
  raise SchemaNotFoundError(
infrahub.exceptions.SchemaNotFoundError:
      Unable to find the schema 'DemoDoohickey' in the registry
      demo-absent-template-node | DemoDoohickey

[lancamat1]

Fix with a failing-first reproduction test is up in draft PR #10051. Confirmed root cause: deleting a node with generate_template: true left its generated Template<Kind>/Profile<Kind> entries behind on the worker schema-refresh path, and process() crashed on the orphan before the template-management cleanup step could run. The fix cascades the deletion in SchemaBranch.delete() and lets the weight pass skip orphaned templates so already-poisoned registries self-heal.