LocalProcessBackend silently ignores initializer, breaking the train() API contract

[kingabdulsalamsas111-lgtm]

What happened?

While testing the different Trainer backends, I noticed that LocalProcessBackend.train() exposes an initializer parameter in its public API, but the parameter is never used anywhere in the implementation.

The method accepts:

def train(
self,
runtime: str | types.Runtime | None = None,
initializer: types.Initializer | None = None,
trainer: ... = None,
options: list | None = None,
) -> str:

However, after tracing the entire execution flow, there is no code path that references initializer.

The implementation proceeds directly from runtime validation to training job creation.

LocalProcessBackend.train()

→ validate runtime
→ process options
→ create virtual environment
→ get_local_train_job_script(...)
→ LocalJob(...)
→ start()

The generated training script is created with:

training_command = local_utils.get_local_train_job_script(
trainer=trainer,
runtime=runtime,
train_job_name=trainjob_name,
venv_dir=venv_dir,
cleanup_venv=self.cfg.cleanup_venv,
)

Only the following values are forwarded:

• `trainer
• runtime
• train_job_name
• venv_dir
• cleanup_venv

initializer is never passed into the script generation, never validated, and never consumed before the training process starts.

A repository-wide search inside the LocalProcess backend also confirms this:

LocalProcess backend:

initializer → only appears in the function signature

No dataset initialization, model initialization, download, preparation, or execution logic exists anywhere in the LocalProcess backend.

Backend inconsistency

This behavior is inconsistent with the other backends.

Container backend

if initializer:
self._run_initializers(...)

which executes dataset/model initializers before training.

Kubernetes backend

trainjob_spec.initializer = ...

which embeds the initializer into the generated TrainJob specification.

Only the LocalProcess backend silently ignores the same API parameter.

Why this is a problem

From the public API, users reasonably expect that supplying an Initializer will prepare the required dataset/model before training starts.

Instead:

• the argument is accepted,
• no warning or error is produced,
• no initializer logic is executed,
• training starts immediately.

This creates a silent behavioral mismatch where users believe initialization has occurred even though it has been completely skipped.

If the training code depends on artifacts that should have been prepared by the initializer, the job may fail later or execute with an unexpected environment, while the original cause is hidden because the API accepted the initializer without ever using it.

Expected behavior

The LocalProcess backend should either:

1. implement initializer handling consistent with the Container and Kubernetes backends, or
2. explicitly reject the initializer argument (for example by raising NotImplementedError or ValueError) instead of silently accepting and ignoring it.

What did you expect to happen?

I expected LocalProcessBackend to handle the initializer consistently with the other backends.

Specifically, when an Initializer is supplied to train(), the backend should either:

• execute the configured dataset/model initialization before starting the training job, as implemented by the ContainerBackend and KubernetesBackend, or
• explicitly reject the parameter with a clear error if initializers are not supported by the LocalProcessBackend.

The backend should not silently accept the initializer argument while completely ignoring it, as this creates inconsistent behavior across implementations of the same public API.

Environment

Kubernetes version:

$ kubectl version

Kubeflow Trainer version:

$ kubectl get pods -n kubeflow -l app.kubernetes.io/name=trainer -o jsonpath="{.items[*].spec.containers[*].image}"

Kubeflow Python SDK version:

$ pip show kubeflow

Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

[kingabdulsalamsas111-lgtm]

I am currently working on a fix and will submit a pull request shortly.

[kingabdulsalamsas111-lgtm]

/assign