No envtest coverage for the JobReconciler and SkyhookReconciler handoff

[ayuskauskas]

Note

The Jobs migration (#223) merged to main in #459 on 2026-08-15. The
feature/package-as-jobs branch has been deleted — base this work on main,
which now executes packages as Jobs.

Part of #223. Refs #411 ("Not covered by the review").

Gap

NewJobReconciler and NewPodReconciler are registered only in cmd/manager/main.go (:215,219), not in internal/controller/suite_test.go, so no envtest spec exercises the JobReconciler → SkyhookReconciler handoff end to end. The new coverage is fake-client unit tests plus chainsaw, which means the concurrent-writer behavior between the two controllers is only observable in e2e.

Scope

• Register the Job and Pod reconcilers in the envtest suite.
• Add specs for the handoff: terminal Job → state-recorded → node-state update → the heavy pass observing complete.
• Cover the concurrent-writer path on nodewright.nvidia.com/nodeState_<name> that fix(operator): merge node-state deltas instead of restamping a stale snapshot #413 addresses, so the guarantee is asserted rather than argued.

[ayuskauskas]

Raised to priority/p1 (2026-08-11).

Not test debt in the ordinary sense: the Job and Pod reconcilers are registered only in cmd/manager/main.go, never in internal/controller/suite_test.go, so nothing in the unit suite ever runs the two controllers against each other. Every guard on the handoff — the state-recorded marker, the optimistic-locked node-state patches, the create-nothing rule — is currently proven only by fake-client specs that drive one reconciler at a time, or by chainsaw, which cannot stage the races at all.

Two of this epic's real defects came out of exactly that seam and were caught by reading rather than by tests: #413 (two controllers writing the same node-state key concurrently) and #426 (a completion recorded against an entry another writer had removed). Both would have been mechanical to catch with an envtest that runs both reconcilers.

Treating this as feature completeness rather than coverage-for-its-own-sake: the Jobs execution model is the handoff between these two controllers, and it ships with no test that exercises it.