Successful uninstall Jobs are deleted immediately, losing their logs

[ayuskauskas]

Note

The Jobs migration (#223) merged to main in #459 on 2026-08-15. The
feature/package-as-jobs branch has been deleted — base this work on main,
which now executes packages as Jobs.

Part of #223. Found by manual validation on 2026-08-12 (93caf2ac, kind v1.36.1, operator run locally), Case 7a of the Jobs validation plan.

What happens

A successful uninstall stage's Job and its pod are deleted the moment the uninstall completes, so the uninstall script's output is unrecoverable. Every other stage's Job and pod are retained until ttlSecondsAfterFinished.

Observed on a CR with two packages — cleanly (uninstall: {enabled: true, apply: true}) and stays (no uninstall block):

# after the uninstall completed:
job/case7-uninstall-stays-1-2-5-apply-…    Complete 1/1   102s   ← retained
job/case7-uninstall-stays-1-2-5-config-…   Complete 1/1    84s   ← retained
pod/case7-uninstall-stays-…-apply          Completed       102s   ← retained
# the cleanly uninstall Job and pod: gone, ~5s after they succeeded

Node state was correct throughout: cleanly|1.2.3 removed, stays|1.2.5 preserved, CR back to complete. The uninstall worked — only its evidence is gone.

Why

shouldDeleteFinishedJob (skyhook_controller.go) ends with:

recordedDone := found && (status.Stage != pkg.Stage || status.State == v1alpha1.StateComplete)
return !recordedDone

An absent entry means "the stage should re-run", which is right for every stage except this one: a successful uninstall's success condition is that the entry is gone. So the sweep reads the completed uninstall as work to redo, and foreground-deletes the Job. Nothing re-runs (uninstall work is driven from node-state entries, and there is no longer one), so the only visible effect is the lost logs.

Scope

• Only successful uninstalls. A failed one leaves the entry at erroring, so its Job is retained normally and the logs survive.
• No functional impact on the lifecycle: no churn, no stuck stage, node state correct.
• It does contradict the migration's stated contract — "finished work is retained instead of deleted on completion", the headline reason for moving to Jobs — for exactly one stage.

Options

1. Exempt StageUninstall from the absent-entry arm of the rerun predicate, so a processed uninstall Job lives out its success TTL like any other stage. Smallest change; the Job is inert either way.
2. Have the uninstall completion path leave a tombstone the predicate can read, rather than special-casing the stage.
3. Accept it and document that uninstall logs are not retained — defensible, since there is no state left to explain, but it should then be said out loud in docs/uninstall.md rather than discovered.

Option 1 unless someone wants the tombstone for other reasons.

Not covered

Whether the same applies to uninstall-interrupt. Untested — it would follow the same predicate, so probably yes.