Skip to content

Commit b34f093

Browse files
weizhouapacheweizhouapache
authored
veeam: fix some issues with restoring volume from backup and attaching it to VM (#8570)
* veeam: detach only the restored volume during backup restore Steps to reproduce the issue 1. create a VM (A) with ROOT and DATA disk 2. assign to a backup offering 3. create backup 4. create another VM (B) 5. restore the DATA disk of VM A, and attach to VM B 6. When operation is done, check the datastore Without this change, the ROOT image is not removed and left over on the datastore. ``` [root@ref-trl-5933-v-Mr8-wei-zhou-esxi2:/vmfs/volumes/5f60667d-18d828eb] ls -l /vmfs/volumes/5f60667d-18d828eb/CS-RSTR-dfb6f21c-a941-49db-9963-4f0286a17dac total 1784840 -rw------- 1 root root 5242880000 Jan 24 09:23 ROOT-722_2-flat.vmdk -rw------- 1 root root 499 Jan 24 09:23 ROOT-722_2.vmdk ``` With this change, the whole temporary vm has been destroyed. ``` [root@ref-trl-5933-v-Mr8-wei-zhou-esxi2:/vmfs/volumes/5f60667d-18d828eb] ls -l /vmfs/volumes/5f60667d-18d828eb/CS-RSTR-734bee3b-640c-4ff0-a34b-bc45358565b2 ls: /vmfs/volumes/5f60667d-18d828eb/CS-RSTR-734bee3b-640c-4ff0-a34b-bc45358565b2: No such file or directory ``` * veeam: fix wrong disk size in debug message * veeam: sync backup repository after operations are done got exception of some operations which succeeds due to the following error ``` 2024-01-19 10:59:52,846 DEBUG [o.a.c.b.v.VeeamClient] (API-Job-Executor-42:ctx-716501bb job-4373 ctx-2359b76d) (logid:b5e19a17) Veeam response for PowerShell commands [PowerShell Import-Module Veeam.Backup.PowerShell -WarningAction SilentlyContinue;$restorePoint = Get-VBRRestorePoint ^| Where-Object { $_.Id -eq '1d99106a-b5c8-4a1e-958d-066a987caa5f' };if ($restorePoint) { Remove-VBRRestorePoint -Oib $restorePoint -Confirm:$false;$repo = Get-VBRBackupRepository;Sync-VBRBackupRepository -Repository $repo;} else { ; Write-Output 'Failed to delete'; Exit 1;}] is: [^M Restore Type Job Name State Start Time End Time Description ^M ------------ -------- ----- ---------- -------- ----------- ^M ConfResynchronize Configuration Dat... Starting 19/01/2024 10:59:52 01/01/1900 00:00:00 ^M ^M ^M Remove-VBRRestorePoint : Win32 internal error "Access is denied" 0x5 occurred while reading the console output buffer. ^M Contact Microsoft Customer Support Services.^M At line:1 char:196^M + ... orePoint) { Remove-VBRRestorePoint -Oib $restorePoint -Confirm:$false ...^M + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^M + CategoryInfo : ReadError: (:) [Remove-VBRRestorePoint], HostException^M + FullyQualifiedErrorId : ReadConsoleOutput,Veeam.Backup.PowerShell.Cmdlets.RemoveVBRRestorePoint^M ^M ]. ``` * veeam: fix unable to detach volume when restore backup and attach to vm then detach the volume It also happened when destroy the original or backup VM ``` 2024-01-24 10:10:03,401 ERROR [c.c.s.r.VmwareStorageProcessor] (DirectAgent-74:ctx-95b24ac7 10.0.35.53, job-25995/job-25996, cmd: DettachCommand) (logid:7260ffb8) Failed to detach volume! java.lang.RuntimeException: Unable to access file [de52fdd3386b3d67b27b3960ecdb08f4] i-2-723-VM/7c2197c129464035bab062edec536a09-flat.vmdk at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:426) at com.cloud.hypervisor.vmware.mo.DatastoreMO.moveDatastoreFile(DatastoreMO.java:290) at com.cloud.storage.resource.VmwareStorageLayoutHelper.syncVolumeToRootFolder(VmwareStorageLayoutHelper.java:241) at com.cloud.storage.resource.VmwareStorageProcessor.attachVolume(VmwareStorageProcessor.java:2150) at com.cloud.storage.resource.VmwareStorageProcessor.dettachVolume(VmwareStorageProcessor.java:2408) at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:174) at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:71) at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:589) at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-24 10:10:03,402 INFO [c.c.h.v.u.VmwareHelper] (DirectAgent-74:ctx-95b24ac7 10.0.35.53, job-25995/job-25996, cmd: DettachCommand) (logid:7260ffb8) [ignored]failed to get message for exception: Unable to access file [de52fdd3386b3d67b27b3960ecdb08f4] i-2-723-VM/7c2197c129464035bab062edec536a09-flat.vmdk ``` * vmware: create restored volume with new UUID and attach to VM
1 parent 33bb92a commit b34f093

File tree

6 files changed

+61
-41
lines changed

6 files changed

+61
-41
lines changed

‎plugins/backup/veeam/src/main/java/org/apache/cloudstack/backup/VeeamBackupProvider.java‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -212,6 +212,7 @@ public boolean removeVMFromBackupOffering(final VirtualMachine vm) {
212212
LOG.warn("Failed to remove Veeam job and backup for job: " + clonedJobName);
213213
throw new CloudRuntimeException("Failed to delete Veeam B&R job and backup, an operation may be in progress. Please try again after some time.");
214214
}
215+
client.syncBackupRepository();
215216
return true;
216217
}
217218

@@ -245,6 +246,8 @@ public boolean deleteBackup(Backup backup, boolean forced) {
245246
return false;
246247
}
247248

249+
client.syncBackupRepository();
250+
248251
List<Backup> allBackups = backupDao.listByVmId(backup.getZoneId(), backup.getVmId());
249252
for (Backup b : allBackups) {
250253
if (b.getId() != backup.getId()) {

‎plugins/backup/veeam/src/main/java/org/apache/cloudstack/backup/veeam/VeeamClient.java‎

Lines changed: 13 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -604,6 +604,7 @@ protected String transformPowerShellCommandList(List<String> cmds) {
604604
joiner.add("PowerShell Add-PSSnapin VeeamPSSnapin");
605605
} else {
606606
joiner.add("PowerShell Import-Module Veeam.Backup.PowerShell -WarningAction SilentlyContinue");
607+
joiner.add("$ProgressPreference='SilentlyContinue'");
607608
}
608609
for (String cmd : cmds) {
609610
joiner.add(cmd);
@@ -646,9 +647,7 @@ public boolean deleteJobAndBackup(final String jobName) {
646647
String.format("$job = Get-VBRJob -Name '%s'", jobName),
647648
"if ($job) { Remove-VBRJob -Job $job -Confirm:$false }",
648649
String.format("$backup = Get-VBRBackup -Name '%s'", jobName),
649-
"if ($backup) { Remove-VBRBackup -Backup $backup -FromDisk -Confirm:$false }",
650-
"$repo = Get-VBRBackupRepository",
651-
"Sync-VBRBackupRepository -Repository $repo"
650+
"if ($backup) { Remove-VBRBackup -Backup $backup -FromDisk -Confirm:$false }"
652651
));
653652
return result != null && result.first() && !result.second().contains(FAILED_TO_DELETE);
654653
}
@@ -658,8 +657,6 @@ public boolean deleteBackup(final String restorePointId) {
658657
Pair<Boolean, String> result = executePowerShellCommands(Arrays.asList(
659658
String.format("$restorePoint = Get-VBRRestorePoint ^| Where-Object { $_.Id -eq '%s' }", restorePointId),
660659
"if ($restorePoint) { Remove-VBRRestorePoint -Oib $restorePoint -Confirm:$false",
661-
"$repo = Get-VBRBackupRepository",
662-
"Sync-VBRBackupRepository -Repository $repo",
663660
"} else { ",
664661
" Write-Output 'Failed to delete'",
665662
" Exit 1",
@@ -668,6 +665,17 @@ public boolean deleteBackup(final String restorePointId) {
668665
return result != null && result.first() && !result.second().contains(FAILED_TO_DELETE);
669666
}
670667

668+
public boolean syncBackupRepository() {
669+
LOG.debug("Trying to sync backup repository.");
670+
Pair<Boolean, String> result = executePowerShellCommands(Arrays.asList(
671+
"$repo = Get-VBRBackupRepository",
672+
"$Syncs = Sync-VBRBackupRepository -Repository $repo",
673+
"while ((Get-VBRSession -ID $Syncs.ID).Result -ne 'Success') { Start-Sleep -Seconds 10 }"
674+
));
675+
LOG.debug("Done syncing backup repository.");
676+
return result != null && result.first();
677+
}
678+
671679
public Map<String, Backup.Metric> getBackupMetrics() {
672680
if (isLegacyServer()) {
673681
return getBackupMetricsLegacy();

‎plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/guru/VMwareGuru.java‎

Lines changed: 13 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1022,12 +1022,12 @@ private String getVolumeFullPath(VirtualDisk disk) {
10221022
/**
10231023
* Get dest volume full path
10241024
*/
1025-
private String getDestVolumeFullPath(VirtualDisk restoredDisk, VirtualMachineMO restoredVm, VirtualMachineMO vmMo) throws Exception {
1025+
private String getDestVolumeFullPath(VirtualMachineMO vmMo) throws Exception {
10261026
VirtualDisk vmDisk = vmMo.getVirtualDisks().get(0);
10271027
String vmDiskPath = vmMo.getVmdkFileBaseName(vmDisk);
10281028
String vmDiskFullPath = getVolumeFullPath(vmMo.getVirtualDisks().get(0));
1029-
String restoredVolumePath = restoredVm.getVmdkFileBaseName(restoredDisk);
1030-
return vmDiskFullPath.replace(vmDiskPath, restoredVolumePath);
1029+
String uuid = UUID.randomUUID().toString().replace("-", "");
1030+
return vmDiskFullPath.replace(vmDiskPath, uuid);
10311031
}
10321032

10331033
/**
@@ -1079,17 +1079,18 @@ public boolean attachRestoredVolumeToVirtualMachine(long zoneId, String location
10791079
VirtualDisk restoredDisk = findRestoredVolume(volumeInfo, vmRestored);
10801080
String diskPath = vmRestored.getVmdkFileBaseName(restoredDisk);
10811081

1082-
s_logger.debug("Restored disk size=" + toHumanReadableSize(restoredDisk.getCapacityInKB()) + " path=" + diskPath);
1082+
s_logger.debug("Restored disk size=" + toHumanReadableSize(restoredDisk.getCapacityInKB() * Resource.ResourceType.bytesToKiB) + " path=" + diskPath);
10831083

10841084
// Detach restored VM disks
1085-
vmRestored.detachAllDisks();
1085+
vmRestored.detachDisk(String.format("%s/%s.vmdk", location, diskPath), false);
10861086

10871087
String srcPath = getVolumeFullPath(restoredDisk);
1088-
String destPath = getDestVolumeFullPath(restoredDisk, vmRestored, vmMo);
1088+
String destPath = getDestVolumeFullPath(vmMo);
10891089

10901090
VirtualDiskManagerMO virtualDiskManagerMO = new VirtualDiskManagerMO(dcMo.getContext());
10911091

10921092
// Copy volume to the VM folder
1093+
s_logger.debug(String.format("Moving volume from %s to %s", srcPath, destPath));
10931094
virtualDiskManagerMO.moveVirtualDisk(srcPath, dcMo.getMor(), destPath, dcMo.getMor(), true);
10941095

10951096
try {
@@ -1103,11 +1104,13 @@ public boolean attachRestoredVolumeToVirtualMachine(long zoneId, String location
11031104
vmRestored.destroy();
11041105
}
11051106

1106-
VirtualDisk attachedDisk = getAttachedDisk(vmMo, diskPath);
1107+
s_logger.debug(String.format("Attaching disk %s to vm %s", destPath, vm.getId()));
1108+
VirtualDisk attachedDisk = getAttachedDisk(vmMo, destPath);
11071109
if (attachedDisk == null) {
1108-
s_logger.error("Failed to get the attached the (restored) volume " + diskPath);
1110+
s_logger.error("Failed to get the attached the (restored) volume " + destPath);
11091111
return false;
11101112
}
1113+
s_logger.debug(String.format("Creating volume entry for disk %s attached to vm %s", destPath, vm.getId()));
11111114
createVolume(attachedDisk, vmMo, vm.getDomainId(), vm.getDataCenterId(), vm.getAccountId(), vm.getId(), poolId, vm.getTemplateId(), backup, false);
11121115

11131116
if (vm.getBackupOfferingId() == null) {
@@ -1119,9 +1122,9 @@ public boolean attachRestoredVolumeToVirtualMachine(long zoneId, String location
11191122
return true;
11201123
}
11211124

1122-
private VirtualDisk getAttachedDisk(VirtualMachineMO vmMo, String diskPath) throws Exception {
1125+
private VirtualDisk getAttachedDisk(VirtualMachineMO vmMo, String diskFullPath) throws Exception {
11231126
for (VirtualDisk disk : vmMo.getVirtualDisks()) {
1124-
if (vmMo.getVmdkFileBaseName(disk).equals(diskPath)) {
1127+
if (getVolumeFullPath(disk).equals(diskFullPath)) {
11251128
return disk;
11261129
}
11271130
}

‎plugins/hypervisors/vmware/src/main/java/com/cloud/storage/resource/VmwareStorageLayoutHelper.java‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -218,7 +218,10 @@ public static void syncVolumeToRootFolder(DatacenterMO dcMo, DatastoreMO ds, Str
218218
}
219219

220220
public static void syncVolumeToRootFolder(DatacenterMO dcMo, DatastoreMO ds, String vmdkName, String vmName, String excludeFolders) throws Exception {
221-
String fileDsFullPath = ds.searchFileInSubFolders(vmdkName + ".vmdk", false, excludeFolders);
221+
String fileDsFullPath = ds.searchFileInSubFolders(String.format("%s/%s.vmdk", vmName, vmdkName), false, excludeFolders);
222+
if (fileDsFullPath == null) {
223+
fileDsFullPath = ds.searchFileInSubFolders(vmdkName + ".vmdk", false, excludeFolders);
224+
}
222225
if (fileDsFullPath == null)
223226
return;
224227

@@ -409,6 +412,22 @@ public static String getVmwareDatastorePathFromVmdkFileName(DatastoreMO dsMo, St
409412
return String.format("[%s] %s/%s", dsMo.getName(), vmName, vmdkFileName);
410413
}
411414

415+
public static String getDatastoreVolumePath(DatastoreMO dsMo, String vmName, String volumePath) throws Exception {
416+
String datastoreVolumePath = VmwareStorageLayoutHelper.getVmwareDatastorePathFromVmdkFileName(dsMo, vmName, volumePath + ".vmdk");
417+
if (dsMo.folderExists(String.format("[%s]", dsMo.getName()), vmName) && dsMo.fileExists(datastoreVolumePath)) {
418+
return datastoreVolumePath;
419+
}
420+
datastoreVolumePath = VmwareStorageLayoutHelper.getVmwareDatastorePathFromVmdkFileName(dsMo, volumePath, volumePath + ".vmdk");
421+
if (dsMo.folderExists(String.format("[%s]", dsMo.getName()), volumePath) && dsMo.fileExists(datastoreVolumePath)) {
422+
return datastoreVolumePath;
423+
}
424+
datastoreVolumePath = VmwareStorageLayoutHelper.getLegacyDatastorePathFromVmdkFileName(dsMo, volumePath + ".vmdk");
425+
if (dsMo.fileExists(datastoreVolumePath)) {
426+
return datastoreVolumePath;
427+
}
428+
return dsMo.searchFileInSubFolders(volumePath + ".vmdk", false, null);
429+
}
430+
412431
@Override
413432
public String getConfigComponentName() {
414433
return VmwareStorageLayoutHelper.class.getSimpleName();

‎plugins/hypervisors/vmware/src/main/java/com/cloud/storage/resource/VmwareStorageProcessor.java‎

Lines changed: 2 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -2062,17 +2062,7 @@ private Answer attachVolume(Command cmd, DiskTO disk, boolean isAttach, boolean
20622062
datastoreVolumePath = dsMo.getDatastorePath((vmdkPath != null ? vmdkPath : dsMo.getName()) + ".vmdk");
20632063
} else {
20642064
if (dsMo.getDatastoreType().equalsIgnoreCase("VVOL")) {
2065-
datastoreVolumePath = VmwareStorageLayoutHelper.getLegacyDatastorePathFromVmdkFileName(dsMo, volumePath + ".vmdk");
2066-
if (!dsMo.fileExists(datastoreVolumePath)) {
2067-
datastoreVolumePath = VmwareStorageLayoutHelper.getVmwareDatastorePathFromVmdkFileName(dsMo, vmName, volumePath + ".vmdk");
2068-
}
2069-
if (!dsMo.folderExists(String.format("[%s]", dsMo.getName()), vmName) || !dsMo.fileExists(datastoreVolumePath)) {
2070-
datastoreVolumePath = VmwareStorageLayoutHelper.getVmwareDatastorePathFromVmdkFileName(dsMo, volumePath, volumePath + ".vmdk");
2071-
}
2072-
if (!dsMo.folderExists(String.format("[%s]", dsMo.getName()), volumePath) || !dsMo.fileExists(datastoreVolumePath)) {
2073-
datastoreVolumePath = dsMo.searchFileInSubFolders(volumePath + ".vmdk", true, null);
2074-
}
2075-
2065+
datastoreVolumePath = VmwareStorageLayoutHelper.getDatastoreVolumePath(dsMo, vmName, volumePath);
20762066
} else {
20772067
datastoreVolumePath = VmwareStorageLayoutHelper.syncVolumeToVmDefaultFolder(dsMo.getOwnerDatacenter().first(), vmName, dsMo, volumePath, VmwareManager.s_vmwareSearchExcludeFolder.value());
20782068
}
@@ -2101,16 +2091,7 @@ private Answer attachVolume(Command cmd, DiskTO disk, boolean isAttach, boolean
21012091
}
21022092
dsMo = new DatastoreMO(context, morDs);
21032093

2104-
datastoreVolumePath = VmwareStorageLayoutHelper.getLegacyDatastorePathFromVmdkFileName(dsMo, volumePath + ".vmdk");
2105-
if (!dsMo.fileExists(datastoreVolumePath)) {
2106-
datastoreVolumePath = VmwareStorageLayoutHelper.getVmwareDatastorePathFromVmdkFileName(dsMo, vmName, volumePath + ".vmdk");
2107-
}
2108-
if (!dsMo.folderExists(String.format("[%s]", dsMo.getName()), vmName) || !dsMo.fileExists(datastoreVolumePath)) {
2109-
datastoreVolumePath = VmwareStorageLayoutHelper.getVmwareDatastorePathFromVmdkFileName(dsMo, volumePath, volumePath + ".vmdk");
2110-
}
2111-
if (!dsMo.folderExists(String.format("[%s]", dsMo.getName()), volumePath) || !dsMo.fileExists(datastoreVolumePath)) {
2112-
datastoreVolumePath = dsMo.searchFileInSubFolders(volumePath + ".vmdk", true, null);
2113-
}
2094+
datastoreVolumePath = VmwareStorageLayoutHelper.getDatastoreVolumePath(dsMo, vmName, volumePath);
21142095
}
21152096
}
21162097

‎test/integration/smoke/test_backup_recovery_veeam.py‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -235,13 +235,13 @@ def test_03_restore_volume_attach_vm(self):
235235
if self.offering:
236236
self.cleanup.insert(0, self.offering)
237237

238-
self.vm = VirtualMachine.create(self.user_apiclient, self.services["small"], accountid=self.account.name,
239-
domainid=self.account.domainid, serviceofferingid=self.service_offering.id)
240-
241238
self.vm_with_datadisk = VirtualMachine.create(self.user_apiclient, self.services["small"], accountid=self.account.name,
242239
domainid=self.account.domainid, serviceofferingid=self.service_offering.id,
243240
diskofferingid=self.disk_offering.id)
244241

242+
self.vm = VirtualMachine.create(self.user_apiclient, self.services["small"], accountid=self.account.name,
243+
domainid=self.account.domainid, serviceofferingid=self.service_offering.id)
244+
245245
# Assign VM to offering and create ad-hoc backup
246246
self.offering.assignOffering(self.user_apiclient, self.vm_with_datadisk.id)
247247

@@ -296,7 +296,13 @@ def test_03_restore_volume_attach_vm(self):
296296
listall=True
297297
)
298298
self.assertTrue(isinstance(vm_volumes, list), "List volumes should return a valid list")
299-
self.assertEqual(3, len(vm_volumes), "The number of volumes should be 2")
299+
self.assertEqual(3, len(vm_volumes), "The number of volumes should be 3")
300300
finally:
301301
# Delete backup
302302
Backup.delete(self.user_apiclient, backup.id, forced=True)
303+
# Remove VM from offering
304+
self.offering.removeOffering(self.user_apiclient, self.vm_with_datadisk.id)
305+
# Delete vm
306+
self.vm.delete(self.apiclient)
307+
# Delete vm with datadisk
308+
self.vm_with_datadisk.delete(self.apiclient)

0 commit comments

Comments
 (0)