Skip to content

testing.md

Latest commit

 

History

History
396 lines (314 loc) · 13.6 KB

testing.md

File metadata and controls

396 lines (314 loc) · 13.6 KB
Raw

Testing guide

This guide covers two things: running this repo's own automated test suites (unit tests and the kubebuilder-scaffolded e2e suite), and setting up a full end-to-end environment to manually exercise a real VM backup and restore through CBT. The manual flow is the same one the automated OADP e2e suite runs, just done by hand so you can watch each step and inspect intermediate state.

Automated tests in this repo

Unit tests

make test

This runs go test against every package returned by go list ./... except test/e2e (Makefile:74), which currently means internal/controller/... and pkg/..., but also picks up any new package added elsewhere in the module. It uses envtest, which spins up a real kube-apiserver and etcd (no full cluster, no kubelet) so reconcilers can be tested against actual API server behavior instead of a fake client. make test also regenerates manifests and DeepCopy methods first (manifests generate fmt vet setup-envtest), so it's safe to run right after editing *_types.go or +kubebuilder markers.

Individual packages can be tested directly, for example:

go test ./internal/controller/... -run TestKubeVirtDataUpload -v
go test ./pkg/uploader/... -v
go test ./pkg/downloader/... -v

Lint

make lint       # report only
make lint-fix   # auto-fix what it can

Kubebuilder-scaffolded e2e suite

test/e2e/e2e_test.go is Kubebuilder's default scaffolded suite (Ginkgo/Gomega). test/e2e/e2e_suite_test.go's BeforeSuite builds and loads its own manager image (example.com/kubevirt-datamover-controller:v0.0.1, hardcoded) and installs cert-manager if it isn't already present. e2e_test.go's own BeforeAll then installs the CRDs and deploys the controller with make deploy, and its test cases check that the controller-manager pod comes up and serves metrics. Run it against a disposable Kind cluster, never a real dev or prod cluster:

kind create cluster --name kdm-e2e
export KIND_CLUSTER=kdm-e2e
go test ./test/e2e/... -tags e2e -v

Setting KIND_CLUSTER matters: the suite's own image-loading step (utils.LoadImageToKindClusterWithName) loads into whatever cluster that variable names, and falls back to a cluster literally named kind if it isn't set. You do not need to docker-build/kind load anything yourself first, the suite does that for you with its own hardcoded image name.

If cert-manager is already installed on the cluster, or you'd rather skip the automatic install/teardown, set CERT_MANAGER_INSTALL_SKIP=true before running the suite.

This suite validates the manager deployment and metrics endpoint. It does not exercise the CBT backup/restore flow, since that needs a real KubeVirt installation with a running VM, which a plain Kind cluster doesn't provide. For that, use the manual flow below.

End-to-end manual test: backup and restore a VM with CBT

This walks through the same flow the OADP e2e suite automates, using the sample manifests kept in openshift/oadp-operator/tests/e2e/sample-applications/virtual-machines/kubevirt-dm. Because this needs OpenShift Virtualization, KubeVirt CBT, and a real object store, run it against a dedicated OpenShift cluster, not a local Kind cluster.

Prerequisites

  • An OpenShift cluster with OpenShift Virtualization installed (HCO >= 1.18, KubeVirt >= v1.8.2). HCO 1.18+ and the backup.kubevirt.io CRDs are required for CBT support, and KubeVirt >= v1.8.2 includes a QEMU backup-abort fix this controller depends on.
  • The OADP operator installed.
  • A working BackupStorageLocation (an S3 bucket, or an S3-compatible/Azure/GCP equivalent, with credentials already set up).
  • oc configured against the target cluster.

1. Enable CBT on the HyperConverged Operator (HCO)

Two separate HCO configurations are needed.

Enable the incrementalBackup feature gate. spec.featureGates on the HCO CR is a list of gate objects (not a map keyed by name), so it must be patched as an array entry:

oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv --type merge -p \
  '{"spec":{"featureGates":[{"name":"incrementalBackup"}]}}'

This also turns on the IncrementalBackup and UtilityVolumes feature gates on the underlying KubeVirt CR.

Enable the CBT label selector. This field lives on the KubeVirt CR, which HCO manages, so it has to go through a jsonpatch annotation on the HCO CR instead of a direct field:

oc annotate hyperconverged kubevirt-hyperconverged -n openshift-cnv --overwrite \
  kubevirt.kubevirt.io/jsonpatch='[{"op":"add","path":"/spec/configuration/changedBlockTrackingLabelSelectors","value":{"virtualMachineLabelSelector":{"matchLabels":{"changedBlockTracking":"true"}}}}]'

Verify it took effect:

oc get kubevirt kubevirt-hyperconverged -n openshift-cnv \
  -o jsonpath='{.spec.configuration.changedBlockTrackingLabelSelectors}'

Expected output:

{"virtualMachineLabelSelector":{"matchLabels":{"changedBlockTracking":"true"}}}

2. Configure the DataProtectionApplication (DPA)

The DPA needs both the kubevirt and kubevirt-datamover default plugins. Adding kubevirt-datamover is what causes the OADP operator to deploy the kubevirt-datamover-plugin (as a Velero init container) and the kubevirt-datamover-controller Deployment from this repo.

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: velero-test
  namespace: openshift-adp
spec:
  configuration:
    velero:
      defaultPlugins:
      - openshift
      - csi
      - aws
      - kubevirt
      - kubevirt-datamover
    nodeAgent:
      enable: true
      uploaderType: kopia
  backupLocations:
  - velero:
      provider: aws
      default: true
      objectStorage:
        bucket: <YOUR_BUCKET>
        prefix: velero
      config:
        region: <YOUR_REGION>
      credential:
        name: cloud-credentials
        key: cloud

Confirm the controller deployed:

oc get deployment -n openshift-adp | grep datamover
oc get pods -n openshift-adp | grep datamover

3. Deploy a test VM with the CBT label

Using the CirrOS sample VM (small, boots fast) as an example:

oc apply -f cirros-vm-cbt.yaml
oc get vm -n cirros-test cirros-test -w

Wait for status.printableStatus to reach Running.

4. Verify CBT is enabled on the VM

With KubeVirt >= v1.8.2 and the feature gate and label selector from step 1 in place, CBT activates the first time the VM boots, no manual restart needed.

oc get vm cirros-test -n cirros-test -o jsonpath='{.status.changedBlockTracking.state}'

Expected output: Enabled. If it isn't (older KubeVirt, or CBT didn't activate on boot), trigger it with a stop/start cycle:

virtctl stop cirros-test -n cirros-test
oc wait vm cirros-test -n cirros-test --for=jsonpath='{.status.printableStatus}'=Stopped --timeout=5m

virtctl start cirros-test -n cirros-test
oc wait vm cirros-test -n cirros-test --for=jsonpath='{.status.printableStatus}'=Running --timeout=5m

5. Create the volume policy

This tells Velero to route the VM's PVCs to the KubeVirt datamover plugin's custom action instead of a CSI snapshot:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kubevirt-volume-policy
  namespace: openshift-adp
data:
  policy.yaml: |
    version: v1
    volumePolicies:
      - conditions: {}
        action:
          type: custom
          parameters:
            datamover: kubevirt
oc apply -f volume-policy.yaml

6. Run the backup

apiVersion: velero.io/v1
kind: Backup
metadata:
  name: kubevirt-dm-backup-1
  namespace: openshift-adp
spec:
  includedNamespaces:
  - cirros-test
  defaultVolumesToFsBackup: false
  snapshotMoveData: true
  resourcePolicy:
    kind: ConfigMap
    name: kubevirt-volume-policy
oc apply -f backup-cirros.yaml
oc get backup kubevirt-dm-backup-1 -n openshift-adp -w

While it's running, you can confirm the datamover path is active by watching for the CRs this controller and its companion plugin create — the controller creates and reconciles the VMBT and VMB, while the DataUpload itself is created by the kubevirt-datamover-plugin (Velero's BackupItemAction) for this controller to act on:

oc get virtualmachinebackuptrackers -A
oc get virtualmachinebackups -A
oc get datauploads -n openshift-adp

Confirm it finished:

oc get backup kubevirt-dm-backup-1 -n openshift-adp -o jsonpath='{.status.phase}'

Expected output: Completed.

7. Run a second backup to confirm incremental behavior

Run another backup against the same VM (a new Backup CR, kubevirt-dm-backup-2, pointing at the same namespace):

apiVersion: velero.io/v1
kind: Backup
metadata:
  name: kubevirt-dm-backup-2
  namespace: openshift-adp
spec:
  includedNamespaces:
  - cirros-test
  defaultVolumesToFsBackup: false
  snapshotMoveData: true
  resourcePolicy:
    kind: ConfigMap
    name: kubevirt-volume-policy
oc apply -f backup-cirros-2.yaml
oc get backup kubevirt-dm-backup-2 -n openshift-adp -w

Check the checkpoint index in your bucket at <bsl-prefix>-kubevirt-datamover/checkpoints/cirros-test/cirros-test/index.json to confirm the second backup was recorded as "type": "incremental" with a parent pointing at the first checkpoint. That confirms the checkpoint chain logic described in docs/architecture.md is working end to end.

8. Restore the VM

Restoring from kubevirt-dm-backup-2 exercises downloading, rebasing, and flattening the full incremental chain (the full checkpoint plus the incremental on top of it), rather than just the single full checkpoint from the first backup.

Before restoring, either delete the original VM's namespace for an in-place restore, or configure a namespace mapping (spec.namespaceMapping on the Restore) to restore into a different namespace; a namespace cannot simply be "scaled down". This example deletes the original namespace:

oc delete namespace cirros-test

Then create a Restore pointing at the second backup, saving it as restore-cirros.yaml:

apiVersion: velero.io/v1
kind: Restore
metadata:
  name: kubevirt-dm-restore-1
  namespace: openshift-adp
spec:
  backupName: kubevirt-dm-backup-2
oc apply -f restore-cirros.yaml
oc get restore kubevirt-dm-restore-1 -n openshift-adp -w

Watch the download side the same way as the upload side:

oc get datadownloads -n openshift-adp

Once the restore completes, confirm the VM comes back up:

oc get vm cirros-test -n cirros-test -o jsonpath='{.status.printableStatus}'

Checking that the VM is Running only confirms the disk was reattached, not that its data survived the restore intact. To actually verify data integrity, write a marker onto the disk before the first backup (for example, virtctl console cirros-test and echo pre-backup > /tmp/marker.txt from inside the guest, or a distinguishing change to the workload's data for an application VM), then after the restore completes, connect to the guest again and confirm the marker/data is still there.

Cleanup

oc delete restore kubevirt-dm-restore-1 -n openshift-adp
oc delete backup kubevirt-dm-backup-2 -n openshift-adp
oc delete backup kubevirt-dm-backup-1 -n openshift-adp
oc delete configmap kubevirt-volume-policy -n openshift-adp
oc delete namespace cirros-test

Other sample VMs

The same oadp-operator sample directory also has manifests for a Fedora and a CentOS Stream 10 VM, both running a todolist/MariaDB workload, useful for testing backup/restore of a VM with an actual application and database rather than an empty CirrOS image. They follow the same steps as above, just against a different namespace and VM name; see the README.md in that directory for the exact file names and target namespaces.

Debugging tips

  • The datamover and downloader pods are short-lived and get cleaned up automatically after a successful backup or restore, but you don't need to catch them mid-flight to see their output. The controller streams each pod's logs into its own log output (as "Datamover pod log" entries with the source pod name) right before it removes the pod. This forwarding is best-effort, not guaranteed: it only requests the pod's last 100 log lines, and a failure to collect them is itself only logged rather than blocking pod cleanup. If you need complete diagnostics, or the forwarded output looks incomplete, catch the pod live (oc get pods -n openshift-adp -w) and read its logs directly before it's removed.

  • A stuck DataUpload/DataDownload will eventually fail once spec.operationTimeout elapses. If you want a phase to sit and let you inspect it (VMB status, PVC state, and so on), watch the phase transitions with oc get dataupload <name> -n openshift-adp -w -o jsonpath='{.status.phase}{"\n"}' rather than deleting resources mid-flight.

  • If a backup or restore fails, check oc get events -n openshift-adp --sort-by=.lastTimestamp and the controller manager's own logs:

    oc logs -n openshift-adp deployment/kubevirt-datamover-controller-manager

    This single log stream has both the reconciler's own phase-transition and failure-reason messages and the datamover/downloader pod's forwarded output, since the controller emits the pod's log lines into its own logger before deleting the pod. Cross-reference the reconciliation phases described in docs/architecture.md to figure out which step failed.