VCF installer deployment failed.

[marcbunyan]

Describe the bug

My deploy is failing at the SDDC stage, is the error below because the holorouter VM requires direct network access to the parent ESXi host?

14-08-2025 10:29:59 VCFInstaller[61306]: [ERROR] VCF Installer deployment failed
14-08-2025 10:29:59 VCFInstaller[61306]: [INFO] Failed to deploy OVF template. Retry #5
Opening OVA source: /holodeck-runtime/bin/9.0/VCF-SDDC-Manager-Appliance-9.0.0.0.24703748.ova
The manifest validates
Opening VI target: vi://user%40domain.local@parent-vcenter:443/LAB-DC/host/LAB_CLUSTER01
Deploying to VI: vi://user%40domain.local@parent-vcenter:443/LAB-DC/host/LAB_CLUSTER01
Transfer Failed
Transfer Failed
Error: Failed to send file [disk-0.vmdk] to the host [host-0001.domain.local], please check the network connection
Completed with errors
14-08-2025 10:30:12 VCFInstaller[61306]: [ERROR] VCF Installer deployment failed
14-08-2025 10:30:12 VCFInstaller[61306]: [ERROR] Failed to deploy OVF template after 6 retries. Terminating

Reproduction steps

1. Connect to vCenter - OK
2. Select Datastore - OK
3. Select Portgroup - OK
   ...nested hosts deploy etc...
   SDDC .ova retry quits after 6 attempts.

Expected behavior

SDDC disks are uploaded and the SDDC/VCF installer appliance boots up?

Additional context

No response

[dhruv-tyagi-broadcom]

Yes, Holorouter VM needs to have access to the target ESX host to transfer the OVA file as mentioned in the docs: https://vmware.github.io/Holodeck/#configuration-requirements

[marcbunyan]

I'm afraid that the instructions below are a little too vague - I see no confirmation of the source:destination:ports required for the holorouter to function... (is it just parent ESXi/vCenter API calls etc?)

"If a vCenter is used as the target for deploying nested VCF lab, then VLANs 0, 10 through 25 and 40 through 58 need to be allowed on the physical switches to allow inter-host communication within the vSphere cluster where the nested VCF deployment will occur."

If these instructions can't be expanded with more details, can you confirm if there is a manual method to get the SDDC appliance up and running so the script can continue?

Thanks,
Marc

[dhruv-tyagi-broadcom]

Can you ping the target ESX hosts from Holorouter and see if they are reachable?

[marcbunyan]

ICMP unfortunately is blocked in this environment...

Is there another method or manual process that could be used to 'get' the SDDC appliance up and running if we cannot API call the hosts... the networking port requirements are missing from the documentation it would appear.

[dhruv-tyagi-broadcom]

We make use of ovftool to deploy the OVA file. Even though we target the vCenter if that is selected, the actual OVA deployment happens on the ESX host within the cluster instead of directly on the vCenter because the datastore is on the ESX hosts and not the vCenter. This is not specific to Holodeck and is just how ovftool works in general even outside Holodeck.

That said, here is the step to do this manually, however still tries the same thing: #28 (comment)

Most likely you'll see the same error. In that case, you can manually deploy the OVA file on the vCenter as you would normally deploy an OVA by right clicking on the vCenter cluster > Deploy new OVF Template, select the OVA and fill out the details manually

[marcbunyan]

If we deploy the SDDC/VCF installer manually, what are the steps to continue the holodeck deploy using the manual deploy process?

[dhruv-tyagi-broadcom]

#28 (comment)

[marcbunyan]

After setting up the sddc manually (and we can open the VCF UI via the htop console OK) and running the code in the #28 comment we now get this error..

15-08-2025 03:39:45 SetHoloDeckConfig[61306]: [INFO] Updating the Global Config, ConfigKey: vSANMode, ConfigValue: ESA
15-08-2025 03:39:45 HoloDeck[61306]: [INFO] Creating Single Site HoloDeck Instance with VCF Management & Workload Domain Only with default configurations!
New-Item: The file '/holodeck-runtime/output/network_manager_config_site_a.yaml' already exists.
15-08-2025 03:39:46 Network-Manager-Init[61306]: [SUCCESS] Network Configuration Generated Successfully
0
15-08-2025 03:39:46 HoloSite[61306]: [INFO] Deploying Full Stack SDDC
15-08-2025 03:39:46 HoloSite[61306]: [INFO] Deploying Management Domain
15-08-2025 03:39:46 SddcMgmtDomain[61306]: [INFO] Beginning SDDC Management Domain Deployment
15-08-2025 03:39:46 SddcMgmtDomain[61306]: [INFO] Verifying VCF Installer availability
15-08-2025 03:39:46 SddcMgmtDomain[61306]: [INFO] VCF Installer is now ready
15-08-2025 03:39:46 SddcMgmtDomain[61306]: [INFO] Setting up depot for VCF Installer
15-08-2025 03:39:46 SddcMgmtDomain[61306]: [INFO] Depot Type selected: online
ConvertFrom-Json: Cannot bind argument to parameter 'InputObject' because it is null.
15-08-2025 03:39:46 SddcMgmtDomain[61306]: [ERROR] Depot setup failed. Error encountered:
Exception: Exiting

It looks like we hit another hard-stop with access to the parent hypervisor with the #28 code block sub-tasks.

15-08-2025 04:22:18 HoloSite[61306]: [INFO] Updating VCF Installer Settings
Invoke-VMScript: 8/15/2025 4:22:35 AM Invoke-VMScript Name or service not known (host-0001.domain.local:443)
15-08-2025 04:22:35 HoloSite[61306]: [INFO] VCF Installer settings updated successfully

[dhruv-tyagi-broadcom]

#28 is using some variables that are not directly defined in that comment but are defined in the comment above it. Here's what you can run first:

$site = "a"
$siteCode = "Site-$site"
$target_host = $config.Target
$vcf_installer_config = $config.'holodeck-sddc'.$siteCode.'vcf-installer'
$instanceID = $config.InstanceID 
$ova_path = $vcf_installer_config.ovaPath
$username = $target_host.username
Add-Type -AssemblyName System.Web
$username_encoded = [System.Web.HttpUtility]::UrlEncode($username)
$password = $target_host.password
$password_encoded = [System.Web.HttpUtility]::UrlEncode($password)
$hostname = $target_host.hostname
$datastore = $target_host.datastore


$vcf_installer_hostname = $vcf_installer_config.hostname
if( $config.InstanceID -ne "Default")
{
    $vcf_installer_vmname = $instanceId +"-"+$vcf_installer_config.hostname
}else{
    $vcf_installer_vmname = $vcf_installer_config.hostname
}

$vcf_installer_network = $vcf_installer_config.network1
$vcf_installer_root_password = $vcf_installer_config.rootPassword
$vcf_installer_local_password = $vcf_installer_config.localUserPassword
$vcf_installer_ntp = $vcf_installer_config.ntp
$vcf_installer_ip0 = $vcf_installer_config.ip0
$vcf_installer_netmask0 = $vcf_installer_config.netmask0
$vcf_installer_gateway = $vcf_installer_config.gateway
$vcf_installer_domain = $vcf_installer_config.domain
$vcf_installer_searchpath = $vcf_installer_config.searchpath
$vcf_installer_DNS = $vcf_installer_config.dns

Then run #28 and it should work

15-08-2025 03:39:46 SddcMgmtDomain[61306]: [INFO] Depot Type selected: online
ConvertFrom-Json: Cannot bind argument to parameter 'InputObject' because it is null.
15-08-2025 03:39:46 SddcMgmtDomain[61306]: [ERROR] Depot setup failed. Error encountered:
Exception: Exiting

This is a known issue that we will fix in the next patch release.

To skip over it, you can run (assuming your config is loaded, you can check by running $config in pwsh to see if you get an output)

vi $config.state

Search for the section:

{
  "VCF-Installer-Depot-Setup": {
    "Status": "InProgress"
  }
}

Change "InProgress" to "Success" and save the file. Then run your New-HoloDeckInstance command again

[marcbunyan]

perfect! ..

Looks like I need to fix the depot file structure now :)

15-08-2025 07:17:18 SddcMgmtDomain[61306]: [INFO] Received Bundles. Checking if all VCF 9 bundles are available
15-08-2025 07:17:18 SddcMgmtDomain[61306]: [INFO] Didn't receive all bundles. Received 0 bundle details. Trying again after 10 seconds
15-08-2025 07:17:28 SddcMgmtDomain[61306]: [INFO] Received Bundles. Checking if all VCF 9 bundles are available
15-08-2025 07:17:28 SddcMgmtDomain[61306]: [INFO] Didn't receive all bundles. Received 0 bundle details. Trying again after 10 seconds
15-08-2025 07:17:38 SddcMgmtDomain[61306]: [INFO] Received Bundles. Checking if all VCF 9 bundles are available
15-08-2025 07:17:38 SddcMgmtDomain[61306]: [INFO] Didn't receive all bundles. Received 0 bundle details. Trying again after 10 seconds
15-08-2025 07:17:48 SddcMgmtDomain[61306]: [INFO] Received Bundles. Checking if all VCF 9 bundles are available
15-08-2025 07:17:48 SddcMgmtDomain[61306]: [INFO] Didn't receive all bundles. Received 0 bundle details. Trying again after 10 seconds

it must be unhappy with the following :

[4.0K] /var/www/
└── [4.0K] PROD/
├── [4.0K] COMP/
│   ├── [4.0K] ESX_HOST/
│   ├── [4.0K] NSX_T_MANAGER/
│   │   ├── [10.0G] nsx-unified-appliance-9.0.0.0.24733065.ova
│   │   └── [292K] VMware-NSX-T-9.0.0.0.24733065.vlcp
│   ├── [4.0K] SDDC_MANAGER_VCF/
│   │   ├── [4.0K] Compatibility/
│   │   │   └── [9.9M] VmwareCompatibilityData.json
│   │   └── [2.0G] VCF-SDDC-Manager-Appliance-9.0.0.0.24703748.ova
│   ├── [4.0K] VCENTER/
│   │   └── [ 12G] VMware-VCSA-all-9.0.0.0.24755230.iso
│   ├── [4.0K] VCF_OPS_CLOUD_PROXY/
│   │   └── [2.7G] Operations-Cloud-Proxy-9.0.0.0.24695833.ova
│   ├── [4.0K] VRA/
│   │   └── [ 21G] vmsp-vcfa-combined-9.0.0.0.24701403.tar
│   ├── [4.0K] VROPS/
│   │   └── [2.6G] Operations-Appliance-9.0.0.0.24695812.ova
│   └── [4.0K] VRSLCM/
│   └── [1.5G] VCF-OPS-Lifecycle-Manager-Appliance-9.0.0.0.24695816.ova
├── [4.0K] metadata/
│   ├── [4.0K] manifest/
│   │   └── [4.0K] v1/
│   │   └──- [109K] vcfManifest.json
│   └── [4.0K] productVersionCatalog/
│   └── [4.0K] v1/
│   ├── [110K] productVersionCatalog.json
│   └── [1.9K] productVersionCatalog.sig
└── [4.0K] vsan/
└── [4.0K] hcl/
├── [ 14M] all.json
└── [ 71] lastupdatedtime.json

[dhruv-tyagi-broadcom]

I'm not able to tell clearly, is your COMP folder within the PROD folder? If not, it should be. You also need to ensure the correct permissions are assigned to the folders by running these commands:

chown -R nginx /var/www
chgrp -R nginx /var/www
chmod -R 750 /var/www
chmod g+s /var/www

[marcbunyan]

apologies, github destroys any thread edit if someone replies mid edit :)

I tried the move into a 'build' folder and that instantly says 401 in the 'Depot' page in the UI...below is my folder structure.

I am attempting to build this offline depot by manually downloading all the files required and they /should/ all be there... I just can't find any official documentation on how to do this 'manually' (so that the lab is a 90day demo using official binaries)

[dhruv-tyagi-broadcom]

Documentation for setting up the offline depot: https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0/deployment/deploying-a-new-vmware-cloud-foundation-or-vmware-vsphere-foundation-private-cloud-/preparing-your-environment/downloading-binaries-to-the-vcf-installer-appliance/connect-to-an-offline-depot-to-download-binaries.html