[bug] Spark KubernetesBackend ignores client_configuration, context filtering when config_file is set, and default target namespace resolution in KubernetesBackendConfig

[vishalmore90]

What happened?

In kubeflow.spark, initialization of KubernetesBackend in kubeflow/spark/backends/kubernetes/backend.py handles KubernetesBackendConfig inconsistently compared to KubernetesBackend in kubeflow.trainer and kubeflow.optimizer.

Specifically, in kubeflow/spark/backends/kubernetes/backend.py:

class KubernetesBackend(RuntimeBackend):
    def __init__(self, backend_config: KubernetesBackendConfig):
        self.namespace = backend_config.namespace or "default"

        if backend_config.config_file:
            config.load_kube_config(config_file=backend_config.config_file)
        elif backend_config.context:
            config.load_kube_config(context=backend_config.context)
        else:
            try:
                config.load_incluster_config()
            except config.ConfigException:
                config.load_kube_config()

        self.custom_api = client.CustomObjectsApi()
        self.core_api = client.CoreV1Api()

1. client_configuration ignored completely: When custom client settings are provided in KubernetesBackendConfig(client_configuration=...) (e.g. for proxy/SSL setup or mock clients), Spark's KubernetesBackend ignores backend_config.client_configuration and creates default client.CustomObjectsApi() and client.CoreV1Api() clients without passing an ApiClient.
2. context ignored when config_file is specified: If both config_file and context are passed in KubernetesBackendConfig(config_file="...", context="..."), the if backend_config.config_file branch triggers config.load_kube_config(config_file=backend_config.config_file) without passing context=backend_config.context.
3. Namespace resolution defaults to "default" statically: If namespace is None in KubernetesBackendConfig, Spark's KubernetesBackend defaults to "default" instead of dynamically resolving the active namespace via common_utils.get_default_target_namespace(cfg.context) (which inspects the active kubeconfig context or the in-cluster service account namespace file /var/run/secrets/kubernetes.io/serviceaccount/namespace).

What did you expect to happen?

SparkClient / Spark's KubernetesBackend should process KubernetesBackendConfig in alignment with TrainerBackend and OptimizerBackend:

class KubernetesBackend(RuntimeBackend):
    def __init__(self, backend_config: KubernetesBackendConfig):
        if backend_config.namespace is None:
            backend_config.namespace = common_utils.get_default_target_namespace(backend_config.context)

        if backend_config.client_configuration is None:
            if backend_config.config_file or not common_utils.is_running_in_k8s():
                config.load_kube_config(config_file=backend_config.config_file, context=backend_config.context)
            else:
                config.load_incluster_config()

        k8s_client = client.ApiClient(backend_config.client_configuration)
        self.custom_api = client.CustomObjectsApi(k8s_client)
        self.core_api = client.CoreV1Api(k8s_client)
        self.namespace = backend_config.namespace

Environment

Environment

Kubernetes version:

$ kubectl version
Client Version: v1.31.0

Kubeflow Trainer version:

$ kubectl get pods -n kubeflow -l app.kubernetes.io/name=trainer -o jsonpath="{.items[*].spec.containers[*].image}"

Kubeflow Python SDK version:

$ pip show kubeflow

Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

[github-actions]

🎉 Welcome to the Kubeflow SDK! 🎉

Thanks for opening your first issue! We're happy to have you as part of our community 🚀

Here's what happens next:

• Our team will review your issue soon! cc @kubeflow/kubeflow-sdk-team
• If you'd like to contribute to this issue, check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards

Join the community:

• Slack: Join our #kubeflow-ml-experience and #kubeflow-trainer Slack channels
• Meetings: Attend the Kubeflow SDK and ML Experience bi-weekly meetings

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

[vishalmore90]

/assign