feat(23570): Add controller for workspace backup #1530
+2,985
−22
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Allda
requested review from
akurinnoy,
dkwon17,
ibuziuk and
rohanKanojia
as code owners
Allda
force-pushed
the
23570
branch
2 times, most recently
from
42dd45c to
dffd7e6
Compare
Member
|
@Allda : Really appreciate you taking the time to contribute this in such a short time. 🎉 Could you please also fill out the “Is it tested? How?” section in the PR template? It’ll help reviewers and future contributors verify the change more easily. Thanks again for your effort! 🙌 |
Member
|
I tested this PR and it seems to work.
config:
workspace:
backupCronJob:
enable: true
schedule: "*/3 * * * *"
|
Allda
force-pushed
the
23570
branch
3 times, most recently
from
0bc74b1 to
8427ba5
Compare
Collaborator
Author
|
/retest |
tolusha
reviewed
Oct 29, 2025
tolusha
reviewed
Oct 29, 2025
tolusha
reviewed
Oct 29, 2025
tolusha
reviewed
Oct 29, 2025
dkwon17
reviewed
Oct 29, 2025
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #1530 +/- ##
==========================================
+ Coverage 34.09% 35.30% +1.21%
==========================================
Files 160 161 +1
Lines 13348 13802 +454
==========================================
+ Hits 4551 4873 +322
- Misses 8487 8599 +112
- Partials 310 330 +20 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
tolusha
reviewed
Nov 5, 2025
tolusha
reviewed
Nov 5, 2025
tolusha
reviewed
Nov 5, 2025
Collaborator
|
@tolusha for some reason GitHub is not letting me comment on your comment: #1530 (comment) IMHO I would prefer if we didn't add an annotation to the DevWorkspaces to avoid potentially sending a high number of requests to the apiserver |
- Moving extraArgs to Oras config section - Unify default values - Change UBI base image - Use constant for the PVC name Signed-off-by: Ales Raszka <[email protected]>
tolusha
reviewed
Dec 2, 2025
Instead of using global secret for a whole cluster the controller search for namespace specific secret and use it if available. If not found it fallback to the global secret. Signed-off-by: Ales Raszka <[email protected]>
Signed-off-by: Ales Raszka <[email protected]>
Signed-off-by: Ales Raszka <[email protected]>
In case user uses internal registry in the OCP the image repository path aligns with OCP namespace and creates a image stream given by the workspace name. Signed-off-by: Ales Raszka <[email protected]>
The internal OCP registry is supported by default without a need of providing any registry auth secret. The backup image is pushed to the same namespace where the workspace is running. The token is auto-generated and mounted from the SA definition. Signed-off-by: Ales Raszka <[email protected]>
Collaborator
Author
|
/retest |
Collaborator
Author
|
With this comment I would like to give you an overview of the feature, how to set it up and show you results. The current implementation was focused on priorities to support Build in OCP registry or any other OCI compliant registry. First let's start with the default OCP registry. For this variant user don't need to provide any auth secrets. In next cronjob iteration the workspace will be backed up with the Job. Logs from the Job: The image is then available in the internal registry The second option is to use any generic OCI compliant registry with custom authentication. Here is the example of the config with custom quay.io registry. User needs to provide access token to either a worskspace namespace or to the operator namespace. Here is the log of the backup job: |
Contributor
|
@Allda could you please clarify if the backup is expected to work with both per-user & per-workspace PVC strategies? Also, please consider contributing documentation (could be a separate PR) - https://github.com/devfile/devworkspace-operator/blob/main/docs/dwo-configuration.md |
Collaborator
Author
Yes, the controller supports both types of volumes and is based on the volume provisioner logic that was already available in the operator. I'll create a separate PR with documentation. |
Member
|
@Allda : Thanks for providing the steps. I followed these steps on CRC ( External Registry Backup Scenario
DWOC Configuration: config:
workspace:
backupCronJob:
enable: true
registry:
authSecret: dockerhub-push-secret
path: docker.io/rohankanojia
schedule: '* * * * *'When I stop DevWorkspace, I see job pod logs in When inspecting individual pod logs , it seems it's trying to push to wrong URL (maybe DockerHub doesn't support it): OpenShift Internal Registry Backup Scenario❌ I wasn't able to get it working on CRC When I stop DevWorkspace, I see job pod logs in I see error below in job pod logs, it seems I can see ImageStream gets created but tag is empty: |
tolusha
reviewed
Dec 5, 2025
| log.Info("Backup Job created for DevWorkspace", "id", dwID) | ||
|
|
||
| } | ||
| origConfig := client.MergeFrom(dwOperatorConfig.DeepCopy()) |
Contributor
There was a problem hiding this comment.
Could you explain why do we need to make a deep copy?
Collaborator
Author
There was a problem hiding this comment.
Based on what I read the MergeFrom requires a deepCopy in order to create a diff for Patch(). If you know a better way please let me know.
Collaborator
|
@rohanKanojia for the error you're facing, I was able to work around it by having this in my config: |
dkwon17
reviewed
Dec 6, 2025
- use caching client for secrets - use TTLSecondsAfterFinished to remove old Jobs - update logger Signed-off-by: Ales Raszka <[email protected]>
Collaborator
Author
|
Signed-off-by: Ales Raszka <[email protected]>
rohanKanojia
approved these changes
Dec 9, 2025
ibuziuk
approved these changes
Dec 9, 2025
dkwon17
approved these changes
Dec 9, 2025
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: Allda, dkwon17, ibuziuk, rohanKanojia The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
tolusha
reviewed
Dec 10, 2025
|
|
||
| _, err := sync.SyncObjectWithCluster(sa, clusterAPI) | ||
| if err != nil { | ||
| if _, ok := err.(*sync.NotInSyncError); !ok { |
Contributor
There was a problem hiding this comment.
@dkwon17
Do we really need to ignore NotInSyncError?
Should we try on the next reconcile loop?
Collaborator
There was a problem hiding this comment.
For me, it doesn't seem like a simple change, since the cron job function and the reconciliation function can act separately?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A new backup controller orchestrates a backup process for workspace PVC. A new configuration option is added to DevWorkspaceOperatorConfig that enables running regular cronjob that is responsible for backup mechanism. The job executes following steps:
The last step is currently not fully implemented as it requires running a buildah inside the container and it will be delivered as a separate feature.
Issue: eclipse-che/che#23570
What does this PR do?
What issues does this PR fix or reference?
Is it tested? How?
The feature has been tested locally and using integration tests. Following configuration should be added to the config to enable this feature:
After a config is added, stop any workspace and wait till a backup job is created.
The job creates a backup and push image to registry
PR Checklist
/test v8-devworkspace-operator-e2e, v8-che-happy-pathto trigger)v8-devworkspace-operator-e2e: DevWorkspace e2e testv8-che-happy-path: Happy path for verification integration with Che