Add ability to configure custom init containers

[akurinnoy]

What does this PR do?

This PR adds the ability for cluster administrators to configure custom init containers that run in all workspace pods via the DWOC.
So, now administrators can:

• Inject arbitrary init containers into all workspaces via config.workspace.initContainers.
• Override the built-in init-persistent-home logic by providing a custom container with the same name.

What issues does this PR fix or reference?

https://issues.redhat.com/browse/CRW-9373
https://issues.redhat.com/browse/CRW-9367

Is it tested? How?

Deploy the DevWorkspace Operator with these changes

Test 1: Custom init-persistent-home script execution

1. Apply a DWOC with a custom init container:

   kubectl apply -f - <<EOF
   apiVersion: controller.devfile.io/v1alpha1
   kind: DevWorkspaceOperatorConfig
   metadata:
     name: devworkspace-operator-config
     namespace: $OPERATOR_NAMESPACE
   config:
     workspace:
       persistUserHome:
         enabled: true
       initContainers:
         - name: init-persistent-home
           args:
             - |
               echo "Custom home init executed" > /home/user/.custom_init_marker
               mkdir -p /home/user/custom-config
               echo "enterprise-config" > /home/user/custom-config/settings.txt
   EOF

2. Create a test workspace and wait for it to start

   kubectl apply -f - <<EOF
   apiVersion: workspace.devfile.io/v1alpha2
   kind: DevWorkspace
   metadata:
     name: test1-custom-home
     namespace: $WORKSPACE_NAMESPACE
   spec:
     started: true
     template:
       attributes:
         controller.devfile.io/storage-type: per-user
       components:
         - name: tooling
           container:
             image: quay.io/devfile/universal-developer-image:latest
             memoryLimit: 2Gi
             memoryRequest: 256Mi
   EOF

   Wait for workspace to run:

   kubectl wait --for=condition=Ready devworkspace/test1-custom-home -n $WORKSPACE_NAMESPACE --timeout=5m

3. Verify the init container was injected:

   POD_NAME=$(kubectl get pods -n $WORKSPACE_NAMESPACE -l controller.devfile.io/devworkspace_name=test1-custom-home -o jsonpath='{.items[0].metadata.name}')

   • Verify marker file

     kubectl exec -n $WORKSPACE_NAMESPACE $POD_NAME -c tooling -- cat /home/user/.custom_init_marker

     Expected output

     Custom home init executed
     

   • Verify config file

     kubectl exec -n $WORKSPACE_NAMESPACE $POD_NAME -c tooling -- cat /home/user/custom-config/settings.txt

     Expected output

     enterprise-config

4. Cleanup

   kubectl delete devworkspace test1-custom-home -n $WORKSPACE_NAMESPACE

Test 2: Default init-persistent-home behavior

1. Reset DWOC to default (no custom init)

   kubectl apply -f - <<EOF
   apiVersion: controller.devfile.io/v1alpha1
   kind: DevWorkspaceOperatorConfig
   metadata:
     name: devworkspace-operator-config
     namespace: $OPERATOR_NAMESPACE
   config:
     workspace:
       persistUserHome:
         enabled: true
   EOF

2. Create workspace with persistent home enabled

   kubectl apply -f - <<EOF
   apiVersion: workspace.devfile.io/v1alpha2
   kind: DevWorkspace
   metadata:
     name: test2-backward-compat
     namespace: $WORKSPACE_NAMESPACE
   spec:
     started: true
     template:
       attributes:
         controller.devfile.io/storage-type: per-user
       components:
         - name: tooling
           container:
             image: quay.io/devfile/universal-developer-image:latest
             memoryLimit: 2Gi
             memoryRequest: 256Mi
   EOF

   Wait for workspace to run:

   kubectl wait --for=condition=Ready devworkspace/test2-backward-compat -n $WORKSPACE_NAMESPACE --timeout=5m

3. Verify default stow logic still works

   kubectl logs -n $WORKSPACE_NAMESPACE -l controller.devfile.io/devworkspace_name=test2-backward-compat -c init-persistent-home

   Expected output

   Checking for stow command
   Running stow command
   

4. Cleanup

   kubectl delete devworkspace test2-backward-compat -n $WORKSPACE_NAMESPACE

Test 3: Validation of init-persistent-home configuration

1. Apply DWOC with invalid command

   kubectl apply -f - <<EOF
   apiVersion: controller.devfile.io/v1alpha1
   kind: DevWorkspaceOperatorConfig
   metadata:
     name: devworkspace-operator-config
     namespace: $OPERATOR_NAMESPACE
   config:
     workspace:
       persistUserHome:
         enabled: true
       initContainers:
         - name: init-persistent-home
           image: busybox:latest
           command: ["/bin/bash"]  # Invalid - must be ["/bin/sh", "-c"]
           args:
             - echo "test"
   EOF

2. Create workspace

   kubectl apply -f - <<EOF
   apiVersion: workspace.devfile.io/v1alpha2
   kind: DevWorkspace
   metadata:
     name: test3-invalid-command
     namespace: $WORKSPACE_NAMESPACE
   spec:
     started: true
     template:
       attributes:
         controller.devfile.io/storage-type: per-user
       components:
         - name: tooling
           container:
             image: quay.io/devfile/universal-developer-image:latest
             memoryLimit: 2Gi
             memoryRequest: 256Mi
   EOF

   Wait a few seconds for reconciliation

3. Verify workspace fails with validation error

   kubectl get devworkspace test3-invalid-command -n $WORKSPACE_NAMESPACE -o jsonpath='{.status.phase}'

   Expected output

   Failed

   kubectl get devworkspace test3-invalid-command -n $WORKSPACE_NAMESPACE -o jsonpath='{.status.message}'

   Expected output

   Invalid init-persistent-home container: command must be exactly [/bin/sh, -c]

4. Cleanup

   kubectl delete devworkspace test3-invalid-command -n $WORKSPACE_NAMESPACE

Test 4: Multiple Custom Init Containers

1. Apply DWOC with multiple init containers

   kubectl apply -f - <<EOF
   apiVersion: controller.devfile.io/v1alpha1
   kind: DevWorkspaceOperatorConfig
   metadata:
     name: devworkspace-operator-config
     namespace: $OPERATOR_NAMESPACE
   config:
     workspace:
       persistUserHome:
         enabled: true
       initContainers:
         - name: init-persistent-home
           args:
             - |
               echo "Step 1: Home init" > /home/user/init-sequence.log
               date >> /home/user/init-sequence.log
               echo "Home directory initialized" >> /home/user/init-sequence.log
         - name: install-tools
           image: quay.io/devfile/universal-developer-image:latest
           command: ["/bin/sh", "-c"]
           args:
             - |
               echo "Step 2: Installing tools" >> /home/user/init-sequence.log
               date >> /home/user/init-sequence.log
               echo "Tools setup completed" > /home/user/tools-installed.txt
               echo "wget simulation completed" >> /home/user/init-sequence.log
           volumeMounts:
             - name: persistent-home
               mountPath: /home/user/
         - name: config-setup
           image: quay.io/devfile/universal-developer-image:latest
           command: ["/bin/sh", "-c"]
           args:
             - |
               echo "Step 3: Final config" >> /home/user/init-sequence.log
               date >> /home/user/init-sequence.log
               echo "Configuration setup completed" > /home/user/config-complete.txt
           volumeMounts:
             - name: persistent-home
               mountPath: /home/user/
   EOF

2. Create workspace

   kubectl apply -f - <<EOF
   apiVersion: workspace.devfile.io/v1alpha2
   kind: DevWorkspace
   metadata:
     name: test4-multiple-init
     namespace: $WORKSPACE_NAMESPACE
   spec:
     started: true
     template:
       attributes:
         controller.devfile.io/storage-type: per-user
       components:
         - name: tooling
           container:
             image: quay.io/devfile/universal-developer-image:latest
             memoryLimit: 2Gi
             memoryRequest: 256Mi
   EOF

   Wait for workspace to run:

   kubectl wait --for=condition=Ready devworkspace/test4-multiple-init -n $WORKSPACE_NAMESPACE --timeout=5m

3. Verify all init containers were injected and ran in order

   POD_NAME=$(kubectl get pods -n $WORKSPACE_NAMESPACE -l controller.devfile.io/devworkspace_name=test4-multiple-init -o jsonpath='{.items[0].metadata.name}')

   • Verify all 3 init containers are present

     kubectl get pod -n $WORKSPACE_NAMESPACE $POD_NAME -o jsonpath='{.spec.initContainers[*].name}'

     Expected output

     init-persistent-home install-tools config-setup

   • Verify the sequence log shows all 3 steps

     kubectl exec -n $WORKSPACE_NAMESPACE $POD_NAME -c tooling -- cat /home/user/init-sequence.log

     Expected output

     Step 1: Home init
     [timestamp]
     Home directory initialized
     Step 2: Installing tools
     [timestamp]
     wget simulation completed
     Step 3: Final config
     [timestamp]
     

   • Verify marker files from each init container

     kubectl exec -n $WORKSPACE_NAMESPACE $POD_NAME -c tooling -- cat /home/user/tools-installed.txt

     Expected output

     Tools setup completed

     kubectl exec -n $WORKSPACE_NAMESPACE $POD_NAME -c tooling -- cat /home/user/config-complete.txt

     Expected output

     Configuration setup completed

4. Cleanup

   kubectl delete devworkspace test4-multiple-init -n $WORKSPACE_NAMESPACE

PR Checklist

• E2E tests pass (when PR is ready, comment /test v8-devworkspace-operator-e2e, v8-che-happy-path to trigger)
  • v8-devworkspace-operator-e2e: DevWorkspace e2e test
  • v8-che-happy-path: Happy path for verification integration with Che

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[rohanKanojia]

I tested with the steps provided in the PR description and can confirm it works as expected ✔️

[akurinnoy]

/retest

[tolusha]

I think we shouldn't take this clause into account
cc @dkwon17

[akurinnoy]

To me, it appears to align with the documentation:

Note: If persistUserHome.enabled is false, any init-persistent-home container is ignored.

[dkwon17]

To keep it simple, I agree with @tolusha ,

Note: If persistUserHome.enabled is false, any init-persistent-home container is ignored.

Unless I'm mistaken, the above documentation is introduced in this PR, so removing the clause is okay on my end

[akurinnoy]

@dkwon17 you're right about the documentation. It comes from this:

devworkspace-operator/apis/controller/v1alpha1/devworkspaceoperatorconfig_types.go

Lines 230 to 243 in 1110865

	type PersistentHomeConfig struct {
	// Determines whether the `/home/user/` directory in workspaces should persist between
	// workspace shutdown and startup.
	// Must be used with the 'per-user'/'common' or 'per-workspace' storage class in order to take effect.
	// Disabled by default.
	Enabled *bool `json:"enabled,omitempty"`
	// Determines whether the init container that initializes the persistent home directory should be disabled.
	// When the `/home/user` directory is persisted, the init container is used to initialize the directory before
	// the workspace starts. If set to true, the init container will not be created.
	// This field is not used if the `workspace.persistUserHome.enabled` field is set to false.
	// Enabled by default.
	DisableInitContainer *bool `json:"disableInitContainer,omitempty"`
	}

I understand that PersistentHomeConfig.Enabled and PersistentHomeConfig.DisableInitContainer serve different purposes and work together.

Without this check, custom init-persistent-home container will be processed even when Enabled: false, which is inconsistent.

[dkwon17]

@akurinnoy thank you for the PR,

I want to suggest a different approach for this feature:

After all of the initconatiners have been accumulated:

• initcontainers create from devfile
• container for persistent home (if enabled)
• container for project-clone

We can do a strategic merge around here for the init containers, where the patch is the content of config.workspace.initContainers.

We already do something similar here with container contributions, the code uses strategic merge from the devfile api to merge containers

WDYT cc @rohanKanojia @tolusha ?

[akurinnoy]

/retest

[tolusha]

Good job!

[dkwon17]

Is there a reason why we limit command and args for init persistent home container?

[akurinnoy]

I did it this way because I wanted to have a simple implementation without needing complex validation logic for various command and argument combinations. By restricting the command and args, both the default init container and custom ones are created with the same command format. This makes the behavior more predictable.

[akurinnoy]

Do you think we need to relax these restrictions?

[dkwon17]

IMHO for this feature I would prefer if there were no specific restrictions just for the persistent home init container to allow more freedom for configuration. If something is misconfigured the user would likely know immediately (maybe by an error when starting the workspace pod) when starting a workspace which I think is fine in this case

cc @tolusha @rohanKanojia

[tolusha]

I don't mind

[akurinnoy]

@dkwon17 I removed the restrictions, please take a look.

[dkwon17]

Thank you for the detailed steps to test the PR! They worked as expected. I have a couple of comments ^

[dkwon17]

Hi @tolusha , I'm wondering, what are your thoughts about putting a new field in the CheCluster to edit the init containers for all Eclipse Che workspaces?

For example:

devEnvironments:
  initContainers: ...

which would update the Che-owned DWOC?

A problem with using a global DWOC for persistent init home, is that, DevWorkspaces that are not Che workspaces will fail, for example with web terminals I am getting this error because of course persistent home volume does not exist for web terminals:

[tolusha]

@dkwon17

I'm wondering, what are your thoughts about putting a new field in the CheCluster

I don’t mind.
I might be mistaken, but the global configuration is eventually merged with the Che-owned DWOC.
And it will cause errors like above anyway, right?

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: akurinnoy, rohanKanojia, tolusha
Once this PR has been reviewed and has the lgtm label, please assign dkwon17 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rohanKanojia]

@akurinnoy : Thank you for your patience with the review comments and for addressing everything so thoroughly! Much appreciated.

[openshift-ci]

New changes are detected. LGTM label has been removed.