Operator gives no log line or event when it deletes a package Job

[ayuskauskas]

Important

Base this work on feature/package-as-jobs, not main.

The Jobs migration (#223) is integrating on that branch. main does not yet execute
packages as Jobs, so a PR opened against main will be missing the code this depends
on. feature/package-as-jobs merges to main once #305 lands.

That branch squash-merges, so if your branch is stacked on another PR in this epic,
replay only your own commits when restacking:
git rebase --onto origin/feature/package-as-jobs <last-inherited-commit>

Part of #223. Found by manual validation on 2026-08-13 (945b0061), cases 3, 11 and 13 of the Jobs
validation plan.

What happens

The operator deletes package-stage Jobs on several paths and explains none of them. deleteJobForeground
(operator/internal/controller/job_controller.go:752) logs nothing at any level and records no event:

func deleteJobForeground(ctx context.Context, c client.Client, job *batchv1.Job) error {
	if err := c.Delete(ctx, job, client.PropagationPolicy(metav1.DeletePropagationForeground)); client.IgnoreNotFound(err) != nil {
		return fmt.Errorf("deleting job %s: %w", job.Name, err)
	}
	return nil
}

Only the error path is observable. Every successful deletion is silent, including:

• the orphaned-node sweep — case 13: the Job for a deleted node was simply gone 5s later;
• the stale-spec / spec-edit clear of a terminal Job — case 3 (editing a failing package to unpark
  it) and case 4 (raising stageTimeout);
• config invalidation of an in-flight Job — case 11.

Why it matters

The asymmetry is the problem, not the silence in isolation. Job creation and stage transitions are
well covered — roughly 296 events over a 70-minute run (NodeWright/State, NodeWright/Apply,
Node/State, Node/Apply), plus Job/SuccessfulCreate from the Job controller. So a reader trusts
the event stream, and then a Job disappears from it with no explanation.

At 3am a vanished Job is indistinguishable from:

• a TTL collection (ttlSecondsAfterFinished firing),
• the orphan sweep,
• a stale-spec clear,
• someone running kubectl delete job.

Those have very different meanings and the same observable signature. The Jobs validation plan's own
step for case 13 — "check the operator log for the sweep line rather than inferring from absence" —
cannot be satisfied today; the case can only be confirmed by watching the object vanish.

For scale: across the whole run the operator log contained 10 distinct messages, all
controller-runtime startup noise plus one reconcile error.

Suggested fix

One logger.Info in deleteJobForeground with the job name, node, and a reason supplied by the
caller — e.g. orphaned-node, stale-spec, config-update, timed-out-cleared. A caller-supplied
reason is the important half: the function itself cannot distinguish the cases, and the reason is
exactly what the reader is missing.

Diagnosability only — no lifecycle impact, nothing wedges, node state is correct on every path. Same
family as #443 and #449: these cost the ability to explain what happened, not correctness.

[ayuskauskas]

Fixed in #459 (merged as 3c13b4a). deleteJobForeground now takes a reason and logs it at info level with the Job name, and each of the seven call sites passes a distinct one: package marked invalid, unfinished job no longer matches the package spec, finished job superseded by a recreate attempt, clearing an unfinished or terminally failed stage, removing all jobs on the node, node no longer exists, and finished job no longer recorded as done in node state.