feat(23570): Add controller for workspace backup #1530

Allda · 2025-10-17T08:53:31Z

A new backup controller orchestrates a backup process for workspace PVC. A new configuration option is added to DevWorkspaceOperatorConfig that enables running regular cronjob that is responsible for backup mechanism. The job executes following steps:

Find a workspaces
Finds out that workspace has been recently stopped
Detect a workspace PVC
Execute a job in the same namespace that does the backup

The last step is currently not fully implemented as it requires running a buildah inside the container and it will be delivered as a separate feature.

Issue: eclipse-che/che#23570

What does this PR do?

What issues does this PR fix or reference?

Is it tested? How?

The feature has been tested locally and using integration tests. Following configuration should be added to the config to enable this feature:

config:                                                                         
  workspace:                                                                    
    backupCronJob:                                                              
      enable: true                                                              
      registry: kind-registry:5000/backup                                       
      schedule: '* * * * *'

After a config is added, stop any workspace and wait till a backup job is created.

$ kubectl get jobs
devworkspace-backup-2l679   Running    0/1           138m       138m
devworkspace-backup-2xvgl   Running    0/1           139m       139m
devworkspace-backup-45vxb   Running    0/1           145m       145m

The job creates a backup and push image to registry

+ set -e
+ exec /workspace-recovery.sh --backup
+ set -e
+ for i in "$@"
+ case $i in
+ backup
+ BACKUP_IMAGE=kind-registry:5000/backup/backup-default-common-pvc-test:latest
++ buildah from scratch
+ NEW_IMAGE=working-container
+ buildah copy working-container /workspace/workspacedfd9f53065ea452c//projects /
f099c09f924cf051a01d78cd34ca87a4c161d7c217df5ac627e90e66926fbe9f
+ buildah config --label DEVWORKSPACE=common-pvc-test working-container
+ buildah config --label NAMESPACE=default working-container
+ buildah commit working-container kind-registry:5000/backup/backup-default-common-pvc-test:latest
Getting image source signatures
Copying blob sha256:137b2a0909654325b7eff0a9dfe623e5abdc685c4d6ad8e4c8d163e0984cf805
Copying config sha256:86693ca728855121a4dce059d91c6c9a196b4611fea4cb17d7b38015310cf193
Writing manifest to image destination
86693ca728855121a4dce059d91c6c9a196b4611fea4cb17d7b38015310cf193
+ buildah umount working-container
+ buildah push --tls-verify=false kind-registry:5000/backup/backup-default-common-pvc-test:latest
Getting image source signatures
Copying blob sha256:137b2a0909654325b7eff0a9dfe623e5abdc685c4d6ad8e4c8d163e0984cf805
Copying config sha256:86693ca728855121a4dce059d91c6c9a196b4611fea4cb17d7b38015310cf193
Writing manifest to image destination
stream closed: EOF for default/devworkspace-backup-zjzk5-82psq (backup-workspace)

PR Checklist

E2E tests pass (when PR is ready, comment /test v8-devworkspace-operator-e2e, v8-che-happy-path to trigger)
- v8-devworkspace-operator-e2e: DevWorkspace e2e test
- v8-che-happy-path: Happy path for verification integration with Che

openshift-ci · 2025-10-17T08:53:38Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Allda
Once this PR has been reviewed and has the lgtm label, please assign dkwon17 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

rohanKanojia · 2025-10-25T09:04:19Z

@Allda : Really appreciate you taking the time to contribute this in such a short time. 🎉

Could you please also fill out the “Is it tested? How?” section in the PR template? It’ll help reviewers and future contributors verify the change more easily.

Thanks again for your effort! 🙌

controllers/backupcronjob/backupcronjob_controller.go

apis/controller/v1alpha1/devworkspaceoperatorconfig_types.go

controllers/backupcronjob/backupcronjob_controller.go

rohanKanojia · 2025-10-27T14:46:43Z

I tested this PR and it seems to work.

Created DevWorkspaceOperatorConfig with this BackupCronJobConfig (backup every 3 minutes)

config:
  workspace:
    backupCronJob:
      enable: true
      schedule: "*/3 * * * *"

Created a DevWorkspace and wait for it to get running
Stopped workspace
Controller detected stopped workspace and started creating jobs for backups:

NAME               STATUS    COMPLETIONS   DURATION   AGE
backup-job-8tnsp   Running   0/1                      0s
backup-job-8tnsp   Running   0/1           0s         0s
backup-job-8tnsp   Running   0/1           16s        16s
backup-job-8tnsp   Running   0/1           17s        17s
backup-job-8tnsp   Running   0/1           18s        18s
backup-job-8tnsp   Complete   1/1           18s        18s
backup-job-kc8rm   Running    0/1                      0s
backup-job-kc8rm   Running    0/1           0s         0s
backup-job-kc8rm   Running    0/1           6s         6s
backup-job-kc8rm   Running    0/1           7s         7s
backup-job-kc8rm   Running    0/1           8s         8s
backup-job-kc8rm   Complete   1/1           8s         8s

Allda · 2025-10-29T11:26:56Z

/retest

Makefile

.github/workflows/next-build.yml

tolusha · 2025-10-29T14:51:16Z

apis/controller/v1alpha1/devworkspaceoperatorconfig_types.go

+	// A registry where backup images are stored. Images are stored
+	// in {registry}/backup-${DEVWORKSPACE_NAMESPACE}-${DEVWORKSPACE_NAME}
+	// +kubebuilder:validation:Optional
+	Registry string `json:"registry,omitempty"`


What if registry is not public and requires authentication and/or certificate to access.

I am currently working on the second phase of this feature, where I cover all the use cases, including authentication. I wanted to submit a PR as early as possible to get initial feedback. The auth part should be ready soon.

The registry authorization was added to the controller. You can check the latest commits.

Thank you, I think it make sense to move Registry and RegistryAuthSecret to a dedicated structure

tolusha · 2025-10-29T15:15:24Z

controllers/backupcronjob/backupcronjob_controller.go

+		return err
+	}
+
+	job := &batchv1.Job{


I do believe we need a dedicated SA for this job and delegate only required permissions.
@dkwon17 ?

Yes, that makes sense to me, could you please take a look @Allda ?

@dkwon17 Do you want to create a brand new SA for each namespace or for each job? Or is there any existing SA that I should use here? Also, what permissions should I delegate to it?

Also, what permissions should I delegate to it?

Maybe we don't need any.

In the latest commit, I added a separate SA for the workspace namespace and use it for the Job definition.

controllers/backupcronjob/backupcronjob_controller.go

dkwon17 · 2025-10-29T22:14:07Z

controllers/backupcronjob/backupcronjob_controller.go

+	backUpConfig := dwOperatorConfig.Config.Workspace.BackupCronJob
+
+	// Find a PVC with the name "claim-devworkspace" or based on the name from the operator config
+	pvcName := "claim-devworkspace"


The PVC name will not always be claim-devworkspace,

There are two main types of storage strategies for DevWorkspaces, common (or, per-user), and per-workspace

Here are some more details about the storage strategies: https://eclipse.dev/che/docs/stable/administration-guide/configuring-the-storage-strategy/

For common, the default PVC name is claim-devworkspace, and for per-workspace, the PVC name is storage-<devworkspaceid>

I suggest using for example GetProvisioner to help determine the storage policy,

and to determine the PVC name, the code is currently determining that like so:

devworkspace-operator/pkg/provision/storage/commonStorage.go

Lines 58 to 65 in b61eaed

usingAlternatePVC, pvcName, err := checkForAlternatePVC(workspace.Namespace, clusterAPI)

if err != nil {

return err

}

if pvcName == "" {

pvcName = workspace.Config.Workspace.PVCName

}

devworkspace-operator/pkg/provision/storage/perWorkspaceStorage.go

Lines 61 to 65 in b61eaed

perWorkspacePVC, err := syncPerWorkspacePVC(workspace, clusterAPI)

if err != nil {

return err

}

pvcName := perWorkspacePVC.Name

I added a dynamic PVC logic that selects the right PVC name based on the used type. Please check again.

codecov · 2025-11-03T11:46:08Z

Codecov Report

❌ Patch coverage is 64.13043% with 165 lines in your changes missing coverage. Please review.
✅ Project coverage is 35.30%. Comparing base (d92e750) to head (2679783).
⚠️ Report is 16 commits behind head on main.

Files with missing lines	Patch %	Lines
...trollers/backupcronjob/backupcronjob_controller.go	71.95%	87 Missing and 19 partials ⚠️
apis/controller/v1alpha1/zz_generated.deepcopy.go	0.00%	43 Missing ⚠️
main.go	0.00%	9 Missing ⚠️
internal/images/image.go	0.00%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1530      +/-   ##
==========================================
+ Coverage   34.09%   35.30%   +1.21%     
==========================================
  Files         160      161       +1     
  Lines       13348    13802     +454     
==========================================
+ Hits         4551     4873     +322     
- Misses       8487     8599     +112     
- Partials      310      330      +20

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ibuziuk

@Allda great job!
discussed the overall PR with @dkwon17 and I believe we should target it to be merged in the DWO 0.39.0 version

project-backup/Containerfile

tolusha · 2025-11-05T09:06:04Z

apis/controller/v1alpha1/devworkspaceoperatorconfig_types.go

 	Schedule string `json:"schedule,omitempty"`
 }

+type BackupCronJobConfig struct {


@dkwon17
Does it make sense to create a completely new API for backup and don't use devworkspaceoperatorconfig?

tolusha · 2025-11-05T09:13:32Z

controllers/backupcronjob/backupcronjob_controller.go

+}
+
+func (r *BackupCronJobReconciler) copySecret(workspace *dw.DevWorkspace, ctx context.Context, sourceSecret *corev1.Secret, logger logr.Logger) (namespaceSecret *corev1.Secret, err error) {
+	log := logger.WithName("copySecret")


I think it makes sense to use a common logger name for a backup component, so it can be easily identified in the DevWorkspaceController logs.

controllers/backupcronjob/backupcronjob_controller.go

Signed-off-by: Anatolii Bazko <[email protected]>

Signed-off-by: Rohan Kumar <[email protected]>

…de to v0.22.1 Signed-off-by: Rohan Kumar <[email protected]>

A new backup controller orchestrates a backup process for workspace PVC. A new configuration option is added to DevWorkspaceOperatorConfig that enables running regular cronjob that is responsible for backup mechanism. The job executes following steps: - Find a workspaces - Finds out that workspace has been recently stopped - Detect a workspace PVC - Execute a job in the same namespace that does the backup The last step is currently not fully implemented as it requires running a buildah inside the container and it will be delivered as a separate feature. Issue: eclipse-che/che#23570 Signed-off-by: Ales Raszka <[email protected]>

A backup of workspace is done using Buildah and storing a content of the workspace PVC into a container image. The image is later stored in a registry and can be used to recover data. A prototype script was updated and stored under project-backup directory and is build alongside the controller. The backup job calls the script and execute following steps: - mount a volume with workspace data - build container image using buildah - push image to registry configured by the operator admin Signed-off-by: Ales Raszka <[email protected]>

A new sub-object was added to the operator config that reflect a current status of the backup controller and stores a last time the backup was executed. This value is used to determine whether a backup of the workspace is needed or if it already has been executed. Signed-off-by: Ales Raszka <[email protected]>

A backup job use a PVC name from a default value or from the config if user configured custom name. Signed-off-by: Ales Raszka <[email protected]>

The backup job can now push to registries which requires auth token. The token is provided as a secret in operator namespace and added to the operator config. Signed-off-by: Ales Raszka <[email protected]>

Signed-off-by: Ales Raszka <[email protected]>

A backup job now determines the name of pvc based on used storage type. It distinguish between different storage types (common and per-workspace) and mount the volume dynamically. Signed-off-by: Ales Raszka <[email protected]>

It turns out the capabilities from the prototype are not needed. Signed-off-by: Ales Raszka <[email protected]>

A new SA is created for the backup jobs to limit the permission to just what is necessary. Signed-off-by: Ales Raszka <[email protected]>

Signed-off-by: Ales Raszka <[email protected]>

Allda · 2025-11-05T13:05:56Z

/retest

Allda requested review from akurinnoy, dkwon17, ibuziuk and rohanKanojia as code owners October 17, 2025 08:53

Allda force-pushed the 23570 branch 2 times, most recently from 42dd45c to dffd7e6 Compare October 17, 2025 11:06

rohanKanojia reviewed Oct 27, 2025

View reviewed changes

controllers/backupcronjob/backupcronjob_controller.go Outdated Show resolved Hide resolved

rohanKanojia reviewed Oct 27, 2025

View reviewed changes

controllers/backupcronjob/backupcronjob_controller.go Show resolved Hide resolved

rohanKanojia reviewed Oct 27, 2025

View reviewed changes

controllers/backupcronjob/backupcronjob_controller.go Outdated Show resolved Hide resolved

rohanKanojia reviewed Oct 27, 2025

View reviewed changes

controllers/backupcronjob/backupcronjob_controller.go Outdated Show resolved Hide resolved

rohanKanojia reviewed Oct 27, 2025

View reviewed changes

apis/controller/v1alpha1/devworkspaceoperatorconfig_types.go Show resolved Hide resolved

rohanKanojia reviewed Oct 27, 2025

View reviewed changes

controllers/backupcronjob/backupcronjob_controller.go Outdated Show resolved Hide resolved

rohanKanojia reviewed Oct 27, 2025

View reviewed changes

controllers/backupcronjob/backupcronjob_controller.go Outdated Show resolved Hide resolved

rohanKanojia reviewed Oct 27, 2025

View reviewed changes

controllers/backupcronjob/backupcronjob_controller.go Outdated Show resolved Hide resolved

Allda force-pushed the 23570 branch 3 times, most recently from 0bc74b1 to 8427ba5 Compare October 29, 2025 10:24

tolusha reviewed Oct 29, 2025

View reviewed changes

Makefile Outdated Show resolved Hide resolved

tolusha reviewed Oct 29, 2025

View reviewed changes

controllers/backupcronjob/backupcronjob_controller.go Outdated Show resolved Hide resolved

dkwon17 reviewed Oct 29, 2025

View reviewed changes

ibuziuk reviewed Nov 4, 2025

View reviewed changes

tolusha reviewed Nov 5, 2025

View reviewed changes

project-backup/Containerfile Outdated Show resolved Hide resolved

tolusha reviewed Nov 5, 2025

View reviewed changes

project-backup/Containerfile Outdated Show resolved Hide resolved

tolusha reviewed Nov 5, 2025

View reviewed changes

controllers/backupcronjob/backupcronjob_controller.go Show resolved Hide resolved

tolusha reviewed Nov 5, 2025

View reviewed changes

controllers/backupcronjob/backupcronjob_controller.go Outdated Show resolved Hide resolved

tolusha and others added 14 commits November 5, 2025 12:51

chore: build multi-arch index image

2080169

Signed-off-by: Anatolii Bazko <[email protected]>

chore: build multi-arch index image

5428fb5

Signed-off-by: Anatolii Bazko <[email protected]>

chore (deps) : update controller-runtime to v0.22.1

87d624d

Signed-off-by: Rohan Kumar <[email protected]>

chore (deploy) : Update manifests to reflect controller-runtime upgra…

24ce2c0

…de to v0.22.1 Signed-off-by: Rohan Kumar <[email protected]>

feat: Use PVC name from config if provided

1765b4c

A backup job use a PVC name from a default value or from the config if user configured custom name. Signed-off-by: Ales Raszka <[email protected]>

feat: Allow using registry auth for private registies

d9533a7

The backup job can now push to registries which requires auth token. The token is provided as a secret in operator namespace and added to the operator config. Signed-off-by: Ales Raszka <[email protected]>

fix: Use correct repository for backup job

0f957f2

Signed-off-by: Ales Raszka <[email protected]>

Dynamic PVC selection base on workspace

e5fc07f

A backup job now determines the name of pvc based on used storage type. It distinguish between different storage types (common and per-workspace) and mount the volume dynamically. Signed-off-by: Ales Raszka <[email protected]>

Remove unnecessary capabilities from backup Job

24f510e

It turns out the capabilities from the prototype are not needed. Signed-off-by: Ales Raszka <[email protected]>

Create SA for backup jobs

b84b069

A new SA is created for the backup jobs to limit the permission to just what is necessary. Signed-off-by: Ales Raszka <[email protected]>

Replace custom pointer fn with k8s ptr package

f9297b6

Signed-off-by: Ales Raszka <[email protected]>

Allda force-pushed the 23570 branch from ade80a0 to f9297b6 Compare November 5, 2025 11:53

	usingAlternatePVC, pvcName, err := checkForAlternatePVC(workspace.Namespace, clusterAPI)
	if err != nil {
	return err
	}

	if pvcName == "" {
	pvcName = workspace.Config.Workspace.PVCName
	}

	perWorkspacePVC, err := syncPerWorkspacePVC(workspace, clusterAPI)
	if err != nil {
	return err
	}
	pvcName := perWorkspacePVC.Name

feat(23570): Add controller for workspace backup #1530

Are you sure you want to change the base?

feat(23570): Add controller for workspace backup #1530

Uh oh!

Conversation

Allda commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

What issues does this PR fix or reference?

Is it tested? How?

PR Checklist

Uh oh!

openshift-ci bot commented Oct 17, 2025

Uh oh!

rohanKanojia commented Oct 25, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rohanKanojia commented Oct 27, 2025

Uh oh!

Allda commented Oct 29, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Nov 3, 2025

Codecov Report

Uh oh!

ibuziuk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Allda commented Nov 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Allda commented Oct 17, 2025 •

edited

Loading