diff --git a/docs/disconnected-dns-configuration.md b/docs/disconnected-dns-configuration.md new file mode 100644 index 000000000..e4e4a5a1c --- /dev/null +++ b/docs/disconnected-dns-configuration.md @@ -0,0 +1,259 @@ +# DNS Configuration for Disconnected Mirroring + +## Overview + +When running disconnected installations with the bastion acting as both the DNS server and the mirroring host, proper DNS configuration is critical: + +1. The bastion needs internet access to download OCP images during mirroring +2. The bastion's DNS service (named) is configured in `5_setup_bastion.yaml` +3. The `disconnected_setup_oc_mirror.yaml` playbook runs **before** `5_setup_bastion.yaml` +4. The bastion must have working DNS from the moment it's created + +## The Problem + +### Scenario +- KVM host has IP forwarding enabled for NAT +- Bastion can reach the internet through the KVM host +- If `/etc/resolv.conf` points to the bastion itself before named is configured, DNS resolution fails +- DNS queries fail, preventing downloads from mirror.openshift.com + +### Error Symptoms +``` +Could not find or access '/tmp/oc-mirror-downloads/openshift-client-linux.tar.gz' +``` + +Or: + +``` +Failed to connect to mirror.openshift.com: Temporary failure in name resolution +``` + +## The Solution + +The solution is implemented at bastion creation time in the kickstart configuration, ensuring DNS works from the first boot: + +### 1. During Bastion Creation (`4_create_bastion.yaml`) +The kickstart configuration automatically sets up the correct nameserver: + +**If `env.bastion.options.dns = true` AND `nameserver2` is defined:** +- Uses `nameserver2` (external DNS) as the primary nameserver +- This allows internet access for mirroring before the local DNS service is configured + +**If `env.bastion.options.dns = false` OR `nameserver2` is not defined:** +- Uses `nameserver1` as the primary nameserver +- Adds `nameserver2` as secondary if defined + +### 2. During Mirroring (`disconnected_setup_oc_mirror.yaml`) +- DNS is already working from bastion creation +- Downloads proceed successfully using the configured nameserver +- No DNS reconfiguration needed + +### 3. Final DNS Setup (`5_setup_bastion.yaml`) +- Configures the named service on the bastion +- Updates `/etc/resolv.conf` to use the bastion as primary DNS +- Sets up DNS forwarding to external nameservers + +## Configuration Requirements + +### In your `all.yaml`, configure nameservers properly: + +```yaml +env: + bastion: + networking: + nameserver1: 192.168.122.2 # Bastion IP (will be DNS server after setup) + nameserver2: 8.8.8.8 # External DNS - REQUIRED for disconnected with DNS on bastion + forwarder: 8.8.8.8 # External DNS forwarder + base_domain: example.com + options: + dns: true # Enable DNS service on bastion + + cluster: + networking: + nameserver1: 192.168.122.2 # Bastion IP (DNS server) + nameserver2: 8.8.8.8 # External DNS (optional, for redundancy) + base_domain: example.com +``` + +### Critical Configuration Rules: + +**For Disconnected Installations with DNS on Bastion:** +- **`env.bastion.options.dns`**: Must be `true` +- **`env.bastion.networking.nameserver2`**: **REQUIRED** - Must be an external DNS server (e.g., 8.8.8.8, 1.1.1.1, or corporate DNS) +- **`env.bastion.networking.nameserver1`**: Should be the bastion IP +- **`env.bastion.networking.forwarder`**: Should match nameserver2 + +**For Installations without DNS on Bastion:** +- **`env.bastion.options.dns`**: Set to `false` +- **`env.bastion.networking.nameserver1`**: Use your external DNS server +- **`env.bastion.networking.nameserver2`**: Optional secondary DNS + +### Why nameserver2 is Required: + +When `env.bastion.options.dns = true`, the kickstart configuration uses `nameserver2` as the initial DNS server because: +1. The bastion's named service isn't configured yet +2. Internet access is needed for mirroring operations +3. Using the bastion IP (nameserver1) would fail since named isn't running + +## How It Works + +### Playbook Execution Order + +The correct order for disconnected installations: + +1. **`4_create_bastion.yaml`** - Creates the bastion VM + - Kickstart configures `/etc/resolv.conf` with external DNS (nameserver2) + - Bastion boots with working DNS resolution + +2. **`disconnected_setup_oc_mirror.yaml`** - Downloads and mirrors OCP content + - DNS already works from bastion creation + - Downloads proceed without DNS issues + +3. **`5_setup_bastion.yaml`** - Configures bastion services including DNS + - Sets up named service + - Updates `/etc/resolv.conf` to use bastion as primary DNS + - Configures DNS forwarding to external nameservers + +### DNS Configuration Timeline + +``` +Bastion Creation (4_create_bastion.yaml) +├─ Kickstart sets: nameserver 8.8.8.8 +└─ Bastion boots with working external DNS + +Mirroring (disconnected_setup_oc_mirror.yaml) +├─ DNS works (using 8.8.8.8) +├─ Downloads succeed +└─ No DNS reconfiguration needed + +Final Setup (5_setup_bastion.yaml) +├─ named service configured +├─ /etc/resolv.conf updated to: nameserver 192.168.122.2 +└─ DNS forwarding to 8.8.8.8 configured +``` + +## Implementation Details + +### Files Modified: + +1. **`roles/create_bastion/templates/bastion-ks.cfg.j2`** (RHEL 8) + - Kickstart now checks `env.bastion.options.dns` setting + - Uses `nameserver2` if DNS will be enabled on bastion + - Uses `nameserver1` if DNS will not be enabled on bastion + +2. **`roles/create_bastion/templates/rhel9-bastion-ks.cfg.j2`** (RHEL 9) + - Same logic as RHEL 8 kickstart + - Ensures consistent behavior across RHEL versions + +### Kickstart Logic: + +```jinja2 +{% if env.bastion.options.dns and env.bastion.networking.nameserver2 is defined %} + # Use external DNS (nameserver2) for initial boot + --nameserver={{ env.bastion.networking.nameserver2 }} +{% else %} + # Use primary DNS (nameserver1) and optional secondary + --nameserver={{ env.bastion.networking.nameserver1 }}{{ (',' + nameserver2) if nameserver2 is defined }} +{% endif %} +``` + +## Manual DNS Verification + +If you need to verify DNS configuration on the bastion after creation: + +```bash +# SSH to bastion +ssh root@bastion-ip + +# Check current DNS configuration +cat /etc/resolv.conf + +# Should show external DNS if env.bastion.options.dns = true +# Expected: nameserver 8.8.8.8 (or your nameserver2 value) + +# Test DNS resolution +nslookup mirror.openshift.com +dig mirror.openshift.com + +# Verify internet connectivity +curl -I https://mirror.openshift.com +``` + +If DNS is not working correctly, check: +1. Bastion was created with correct `env.bastion.networking.nameserver2` value +2. `env.bastion.options.dns` is set to `true` in `all.yaml` +3. KVM host has IP forwarding enabled +4. Network routing is configured correctly + +## Troubleshooting + +### Issue: DNS still not working after configuration + +**Check NetworkManager status:** +```bash +systemctl status NetworkManager +``` + +**Verify resolv.conf:** +```bash +cat /etc/resolv.conf +``` + +**Test DNS resolution:** +```bash +nslookup mirror.openshift.com +dig mirror.openshift.com +``` + +### Issue: NetworkManager keeps overwriting resolv.conf + +**Ensure the config file exists:** +```bash +cat /etc/NetworkManager/conf.d/90-dns-none.conf +``` + +Should contain: +``` +[main] +dns=none +``` + +**Restart NetworkManager:** +```bash +systemctl restart NetworkManager +``` + +### Issue: Can't reach internet even with correct DNS + +**Check routing:** +```bash +ip route show +ping 8.8.8.8 +``` + +**Verify KVM host IP forwarding:** +```bash +# On KVM host +sysctl net.ipv4.ip_forward +# Should return: net.ipv4.ip_forward = 1 +``` + +**Check NAT/masquerading on KVM host:** +```bash +# On KVM host +iptables -t nat -L -n -v | grep MASQUERADE +``` + +## Best Practices + +1. **Always define nameserver2** in your configuration for disconnected scenarios +2. **Use reliable external DNS** servers (Google DNS, Cloudflare, or corporate DNS) +3. **Test DNS resolution** before running mirroring operations +4. **Keep the playbook execution order** as documented +5. **Don't skip the DNS configuration steps** - they're critical for success + +## Related Documentation + +- [Troubleshooting Internet Access](troubleshooting-disconnected-internet-access.md) +- [Troubleshooting oc-mirror](troubleshooting-oc-mirror-no-release-images.md) +- [Disconnected Installation Guide](run-the-playbooks-for-disconnected-install.md) \ No newline at end of file diff --git a/docs/monitoring-oc-mirror-progress.md b/docs/monitoring-oc-mirror-progress.md new file mode 100644 index 000000000..4de7b98b5 --- /dev/null +++ b/docs/monitoring-oc-mirror-progress.md @@ -0,0 +1,373 @@ +# Monitoring OC-Mirror Progress in Real-Time + +## Overview + +When running the `disconnected_setup_oc_mirror.yaml` playbook, the mirroring operation can take several hours. This guide explains how to monitor the progress in real-time to see which images are being pulled. + +## What Changed + +The `disconnected_mirror_ocp_bastion` role now includes: + +1. **Progress Logging**: All oc-mirror output is logged to a file +2. **Periodic Updates**: Ansible displays progress updates every poll interval +3. **Monitoring Script**: A helper script for real-time monitoring with color-coded output +4. **Summary Display**: Shows the last 50 lines of output when complete + +## Monitoring Methods + +### Method 1: Ansible Playbook Output (Automatic) + +When you run the playbook, you'll see: + +``` +TASK [disconnected_mirror_ocp_bastion : Display mirroring progress information] +ok: [bastion] => { + "msg": [ + "==========================================", + "OC-Mirror is running in the background", + "==========================================", + "Progress log: /opt/oc-mirror/oc-mirror-progress.log", + "Job ID: 123456.78910", + "", + "To monitor progress in real-time, SSH to bastion and run:", + " tail -f /opt/oc-mirror/oc-mirror-progress.log", + "", + "Or use this command to see only image pulls:", + " tail -f /opt/oc-mirror/oc-mirror-progress.log | grep -E 'mirroring|copying|Pulling|sha256'", + "", + "Ansible will check progress every 30 seconds..." + ] +} +``` + +Ansible will then check progress periodically and display updates. + +### Method 2: SSH to Bastion and Tail the Log + +**Basic monitoring:** +```bash +# SSH to bastion +ssh root@bastion-ip + +# Watch all output +tail -f /opt/oc-mirror/oc-mirror-progress.log +``` + +**Filter for image pulls only:** +```bash +# See only image-related messages +tail -f /opt/oc-mirror/oc-mirror-progress.log | grep -E 'mirroring|copying|sha256' +``` + +**Filter for errors and warnings:** +```bash +# See only errors and warnings +tail -f /opt/oc-mirror/oc-mirror-progress.log | grep -E 'ERROR|WARN' +``` + +### Method 3: Use the Monitoring Script (Recommended) + +The playbook copies a monitoring script to the bastion that provides color-coded, filtered output: + +```bash +# SSH to bastion +ssh root@bastion-ip + +# Run the monitoring script +cd /opt/oc-mirror +./monitor-oc-mirror.sh +``` + +**Output example:** +``` +========================================== +OC-Mirror Progress Monitor +========================================== +Log file: /opt/oc-mirror/oc-mirror-progress.log +Press Ctrl+C to exit +========================================== + +========================================== +Statistics (2026-05-22 10:15:30) +========================================== +Images processed: 127 +Errors: 0 +Warnings: 2 +========================================== + +[INFO] collecting release images... +[MIRROR] mirroring platform images +[IMAGE] quay.io/openshift-release-dev/ocp-release@sha256:abc123... +[IMAGE] quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:def456... +[MIRROR] copying image sha256:abc123... to 192.168.122.2:5000/... +``` + +**Features:** +- Color-coded output (errors in red, warnings in yellow, etc.) +- Periodic statistics display +- Filters out noise, shows only relevant information +- Real-time updates + +### Method 4: Monitor from Ansible Controller + +While the playbook is running, open another terminal and SSH to bastion: + +```bash +# Terminal 1: Running playbook +ansible-playbook playbooks/disconnected_setup_oc_mirror.yaml + +# Terminal 2: Monitor progress +ssh root@bastion-ip "tail -f /opt/oc-mirror/oc-mirror-progress.log | grep -E 'mirroring|copying|sha256'" +``` + +## Understanding the Output + +### Key Messages to Look For + +**1. Collection Phase:** +``` +[INFO] collecting release images... +[INFO] found 150 release images +[INFO] collecting operator images... +``` + +**2. Mirroring Phase:** +``` +mirroring platform images +copying image sha256:abc123... to 192.168.122.2:5000/ocp4/openshift4 +``` + +**3. Image Processing:** +``` +sha256:abc123def456... -> 192.168.122.2:5000/ocp4/openshift4@sha256:abc123def456... +``` + +**4. Completion:** +``` +[INFO] mirroring completed successfully +[INFO] mirror time: 2h 15m 30s +``` + +### Progress Indicators + +**Images being processed:** +- Each `sha256:...` line represents one image layer being copied +- Multiple layers per image is normal +- Hundreds or thousands of sha256 lines is expected + +**Operator catalogs:** +``` +mirroring operator catalog: registry.redhat.io/redhat/redhat-operator-index:v4.21 +``` + +**Additional images:** +``` +mirroring additional image: registry.redhat.io/ubi9/ubi:latest +``` + +## Troubleshooting + +### Issue: No Progress Updates + +**Symptom:** +``` +Ansible shows "TASK [Monitor oc-mirror progress]" but no updates +``` + +**Cause:** Log file not being created or written to + +**Solution:** +```bash +# SSH to bastion +ssh root@bastion-ip + +# Check if oc-mirror is running +ps aux | grep oc-mirror + +# Check if log file exists +ls -lh /opt/oc-mirror/oc-mirror-progress.log + +# Check if log file is being written to +tail -f /opt/oc-mirror/oc-mirror-progress.log +``` + +### Issue: Mirroring Appears Stuck + +**Symptom:** +``` +Same image sha256 shown for several minutes +``` + +**Cause:** Large image layer being downloaded + +**Solution:** +- This is normal for large images (500MB-2GB layers) +- Check network activity: `iftop -i eth0` on bastion +- Be patient - large layers take time + +### Issue: Many Errors in Log + +**Symptom:** +``` +[ERROR] failed to copy image sha256:... +[ERROR] connection timeout +``` + +**Cause:** Network issues or source registry problems + +**Solution:** +```bash +# Check if continue-on-error is enabled +grep continue_on_error inventories/default/group_vars/disconnected.yaml + +# If not enabled, consider enabling it for large mirrors +# Edit disconnected.yaml: +oc_mirror_args: + continue_on_error: true +``` + +### Issue: Want to Stop Mirroring + +**To stop gracefully:** +```bash +# SSH to bastion +ssh root@bastion-ip + +# Find oc-mirror process +ps aux | grep "oc mirror" + +# Send SIGTERM (graceful shutdown) +kill -TERM + +# Wait a few minutes for cleanup +# If it doesn't stop, use SIGKILL +kill -9 +``` + +**To resume later:** +- oc-mirror v2 supports resuming from workspace +- Re-run the playbook - it will continue from where it stopped + +## Performance Tips + +### Monitor Network Usage + +```bash +# On bastion +iftop -i eth0 + +# Or use nload +nload eth0 +``` + +### Monitor Disk Space + +```bash +# Check registry storage +df -h /opt/registry/data + +# Check workspace +df -h /opt/oc-mirror +``` + +### Monitor System Resources + +```bash +# CPU and memory +htop + +# Or use top +top +``` + +## Log File Location + +**Default location:** +``` +/opt/oc-mirror/oc-mirror-progress.log +``` + +**Configured in:** +```yaml +# disconnected.yaml +mirroring: + bastion: + working_dir: '/opt/oc-mirror' # Log will be: /oc-mirror-progress.log +``` + +## After Mirroring Completes + +The playbook will automatically: + +1. Display the last 50 lines of output +2. Show completion message +3. Provide next steps + +**Manual review:** +```bash +# SSH to bastion +ssh root@bastion-ip + +# View full log +less /opt/oc-mirror/oc-mirror-progress.log + +# Search for errors +grep ERROR /opt/oc-mirror/oc-mirror-progress.log + +# Count images mirrored +grep -c "sha256" /opt/oc-mirror/oc-mirror-progress.log + +# View summary +tail -100 /opt/oc-mirror/oc-mirror-progress.log +``` + +## Example Session + +**Terminal 1 - Run playbook:** +```bash +$ ansible-playbook playbooks/disconnected_setup_oc_mirror.yaml + +TASK [disconnected_mirror_ocp_bastion : Run oc-mirror v2 to mirror images (async)] +changed: [bastion] + +TASK [disconnected_mirror_ocp_bastion : Display mirroring progress information] +ok: [bastion] => { + "msg": [ + "OC-Mirror is running in the background", + "Progress log: /opt/oc-mirror/oc-mirror-progress.log", + "Ansible will check progress every 30 seconds..." + ] +} + +TASK [disconnected_mirror_ocp_bastion : Monitor oc-mirror progress with periodic updates] +FAILED - RETRYING: [bastion]: Monitor oc-mirror progress (240 retries left) +FAILED - RETRYING: [bastion]: Monitor oc-mirror progress (239 retries left) +... +``` + +**Terminal 2 - Monitor progress:** +```bash +$ ssh root@192.168.122.2 +# cd /opt/oc-mirror +# ./monitor-oc-mirror.sh + +========================================== +Statistics (2026-05-22 10:30:15) +========================================== +Images processed: 245 +Errors: 0 +Warnings: 3 +========================================== + +[MIRROR] mirroring platform images +[IMAGE] quay.io/openshift-release-dev/ocp-release@sha256:abc123... +[IMAGE] quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:def456... +... +``` + +## Related Documentation + +- [Disconnected Installation Guide](run-the-playbooks-for-disconnected-install.md) +- [OC-Mirror Timeout Configuration](troubleshooting-oc-mirror-timeout.md) +- [DNS Configuration](disconnected-dns-configuration.md) +- [Troubleshooting Guide](troubleshooting.md) \ No newline at end of file diff --git a/docs/run-the-playbooks-for-disconnected-install.md b/docs/run-the-playbooks-for-disconnected-install.md new file mode 100644 index 000000000..288a0764b --- /dev/null +++ b/docs/run-the-playbooks-for-disconnected-install.md @@ -0,0 +1,1132 @@ +# Running Playbooks for Disconnected Installation + +This guide explains how to set up and run OpenShift Container Platform (OCP) in disconnected or air-gapped environments using the `disconnected_setup_oc_mirror.yaml` playbook. + +## Table of Contents + +- [Overview](#overview) +- [Correct Playbook Execution Order](#correct-playbook-execution-order) +- [Architecture](#architecture) +- [Prerequisites](#prerequisites) +- [Configuration](#configuration) +- [Running the Playbook](#running-the-playbook) +- [Roles Overview](#roles-overview) +- [Troubleshooting](#troubleshooting) +- [Advanced Topics](#advanced-topics) + +## Overview + +In disconnected environments, OpenShift clusters cannot directly access Red Hat's container registries. This solution automates: + +1. Setting up a container registry on the bastion host +2. Downloading oc-mirror plugin and OCP client tools +3. Mirroring OCP platform images, operators, and additional images +4. Generating necessary manifests for cluster installation + +## Correct Playbook Execution Order + +### Standard Installation (UPI - User Provisioned Infrastructure) + +For a complete disconnected OpenShift installation from scratch, execute playbooks in this **exact order**: + +1. **[`0_setup.yaml`](../playbooks/0_setup.yaml)** - Setup inventory, install Galaxy collections, check disconnected variables + ```bash + ansible-playbook -i inventories/default playbooks/0_setup.yaml + ``` + +2. **[`1_create_lpar.yaml`](../playbooks/1_create_lpar.yaml)** - Create LPARs (if using HMC/DPM mode) + ```bash + ansible-playbook -i inventories/default playbooks/1_create_lpar.yaml + ``` + +3. **[`2_create_kvm_host.yaml`](../playbooks/2_create_kvm_host.yaml)** - Boot RHEL on LPARs + ```bash + ansible-playbook -i inventories/default playbooks/2_create_kvm_host.yaml + ``` + +4. **[`3_setup_kvm_host.yaml`](../playbooks/3_setup_kvm_host.yaml)** - Configure KVM hosts with libvirt, networking, storage + ```bash + ansible-playbook -i inventories/default playbooks/3_setup_kvm_host.yaml + ``` + +5. **[`4_create_bastion.yaml`](../playbooks/4_create_bastion.yaml)** - Create bastion VM + ```bash + ansible-playbook -i inventories/default playbooks/4_create_bastion.yaml + ``` + +6. **🆕 [`disconnected_setup_oc_mirror.yaml`](../playbooks/disconnected_setup_oc_mirror.yaml)** - **CRITICAL: Setup registry and mirror images** + ```bash + ansible-playbook -i inventories/default playbooks/disconnected_setup_oc_mirror.yaml --ask-vault-pass + ``` + + **This playbook MUST run after bastion creation but BEFORE bastion setup because:** + - ✅ Bastion must exist to host the registry + - ✅ Registry and mirrored images must be ready before OCP installation files are created + - ✅ Pull secret is automatically updated in `all.yaml` for use by subsequent playbooks + - ✅ Generates manifests needed for disconnected cluster installation + +7. **[`5_setup_bastion.yaml`](../playbooks/5_setup_bastion.yaml)** - Configure bastion services (DNS, HAProxy, get OCP binaries) + ```bash + ansible-playbook -i inventories/default playbooks/5_setup_bastion.yaml + ``` + +8. **[`6_create_nodes.yaml`](../playbooks/6_create_nodes.yaml)** - Create and bootstrap cluster nodes + ```bash + ansible-playbook -i inventories/default playbooks/6_create_nodes.yaml + ``` + +9. **[`7_ocp_verification.yaml`](../playbooks/7_ocp_verification.yaml)** - Verify cluster installation + ```bash + ansible-playbook -i inventories/default playbooks/7_ocp_verification.yaml + ``` + +10. **[`disconnected_apply_operator_manifests.yaml`](../playbooks/disconnected_apply_operator_manifests.yaml)** - Apply operator manifests to cluster + ```bash + ansible-playbook -i inventories/default playbooks/disconnected_apply_operator_manifests.yaml + ``` + +### ABI Installation (Agent-Based Installer) + +For ABI disconnected installation, execute in this order: + +1. **[`0_setup.yaml`](../playbooks/0_setup.yaml)** - Setup inventory and check variables + ```bash + ansible-playbook -i inventories/default playbooks/0_setup.yaml + ``` + +2. **[`3_setup_kvm_host.yaml`](../playbooks/3_setup_kvm_host.yaml)** - Configure KVM hosts (if using KVM) + ```bash + ansible-playbook -i inventories/default playbooks/3_setup_kvm_host.yaml + ``` + +3. **[`4_create_bastion.yaml`](../playbooks/4_create_bastion.yaml)** - Create bastion VM + ```bash + ansible-playbook -i inventories/default playbooks/4_create_bastion.yaml + ``` + +4. **🆕 [`disconnected_setup_oc_mirror.yaml`](../playbooks/disconnected_setup_oc_mirror.yaml)** - Setup registry and mirror images + ```bash + ansible-playbook -i inventories/default playbooks/disconnected_setup_oc_mirror.yaml --ask-vault-pass + ``` + +5. **[`5_setup_bastion.yaml`](../playbooks/5_setup_bastion.yaml)** - Configure bastion services + ```bash + ansible-playbook -i inventories/default playbooks/5_setup_bastion.yaml + ``` + +6. **[`create_abi_cluster.yaml`](../playbooks/create_abi_cluster.yaml)** - Create ABI cluster + ```bash + ansible-playbook -i inventories/default playbooks/create_abi_cluster.yaml + ``` + +7. **[`monitor_create_abi_cluster.yaml`](../playbooks/monitor_create_abi_cluster.yaml)** - Monitor ABI installation + ```bash + ansible-playbook -i inventories/default playbooks/monitor_create_abi_cluster.yaml + ``` + +### Pre-existing LPAR Installation + +If you have pre-existing LPARs with RHEL already installed, skip steps 1-3 and start from step 4: + +```bash +# Start from KVM host setup +ansible-playbook -i inventories/default playbooks/3_setup_kvm_host.yaml +# Then continue with steps 5-10 as shown above +``` + +### Using Master Playbooks + +**⚠️ Important**: The current [`site.yaml`](../playbooks/site.yaml) master playbook calls the OLD `disconnected_mirror_artifacts.yaml` playbook. You should update it to use the new `disconnected_setup_oc_mirror.yaml` playbook instead. + +**Option 1: Run all standard playbooks at once** (after updating site.yaml): +```bash +ansible-playbook -i inventories/default playbooks/site.yaml --ask-vault-pass +``` + +**Option 2: Run ABI master playbook** (after updating master_playbook_for_abi.yaml): +```bash +ansible-playbook -i inventories/default playbooks/master_playbook_for_abi.yaml --ask-vault-pass +``` + +### Critical Timing Notes + +**The insertion point for `disconnected_setup_oc_mirror.yaml` is critical:** + +``` +✅ CORRECT ORDER: +4_create_bastion.yaml → disconnected_setup_oc_mirror.yaml → 5_setup_bastion.yaml + +❌ WRONG ORDER: +5_setup_bastion.yaml → disconnected_setup_oc_mirror.yaml (TOO LATE - install-config already created) +disconnected_setup_oc_mirror.yaml → 4_create_bastion.yaml (TOO EARLY - bastion doesn't exist) +``` + +**Why this order matters:** +1. Bastion must exist to host the container registry +2. Registry must be running before mirroring images +3. Images must be mirrored before creating OCP installation files +4. Pull secret must be updated before `5_setup_bastion.yaml` creates install-config.yaml + +### Installation Methods Comparison + +This mirroring solution supports both standard and Agent-Based Installer (ABI) installation methods: + +| Aspect | Standard Installation | Agent-Based Installer (ABI) | +|--------|----------------------|----------------------------| +| **Playbook** | `6_create_nodes.yaml` | `create_abi_cluster.yaml` | +| **Bootstrap Node** | Required | Not required | +| **Installation Method** | Ignition files | Agent ISO/PXE | +| **Configuration** | Multiple files (install-config, ignition) | Single install-config + agent-config | +| **Disconnected Support** | ✅ Full support | ✅ Full support | +| **Registry Integration** | Via ignition files | Via agent artifacts | +| **Node Discovery** | Manual configuration | Automated discovery | +| **Complexity** | Higher | Lower | +| **Best For** | Traditional deployments | Simplified deployments, edge locations | + +Both methods use the same mirrored registry and are fully compatible with `disconnected_setup_oc_mirror.yaml`. + +## Architecture + +``` +Internet → File Server → Bastion → Disconnected Registry → OCP Cluster + (Download) (Mirror) (Local Storage) +``` + +### Components + +- **File Server**: Downloads binaries from the internet (can be same as bastion) +- **Bastion**: Runs oc-mirror to perform the actual mirroring +- **Disconnected Registry**: Container registry on bastion for mirrored images +- **OCP Cluster**: Target OpenShift cluster using mirrored images + +## Prerequisites + +### System Requirements + +#### Bastion Host + +**Note**: If you don't have a bastion host yet, you can create one using the project's playbooks: +```bash +# Create the bastion VM +ansible-playbook -i inventories/default playbooks/4_create_bastion.yaml + +# Setup the bastion with required packages and configuration +ansible-playbook -i inventories/default playbooks/5_setup_bastion.yaml +``` + +**Requirements**: +- RHEL 8/9 or compatible Linux distribution +- **Disk Space**: + - Minimum: 100 GB (single OCP version + limited operators) + - Recommended: 200 GB (multiple versions + operators) + - Production: 500 GB+ (full catalog mirroring) +- 4GB RAM minimum (8GB+ recommended for large mirrors) +- Root/sudo access +- s390x architecture support + +**Disk Space Breakdown**: +- Registry data (`/opt/registry/data`): 100-150 GB +- oc-mirror workspace (`/opt/oc-mirror`): 50-100 GB +- System and logs: 20-50 GB + +#### Network Requirements +- File server must have internet access (or pre-downloaded binaries) +- Bastion must have access to: + - File server (HTTP/FTP) + - Source registries (Red Hat registries) during mirroring + - Target disconnected registry (localhost if on bastion) + +### Required Files + +Ensure you have: +- Valid Red Hat pull secret +- Ansible inventory configured +- Vault password (if using Ansible Vault) + +## Configuration + +### Step 1: Enable Disconnected Mode in all.yaml + +Edit `inventories/default/group_vars/all.yaml` and set the disconnected mode flag: + +```yaml +# Section 1 - Ansible Controller +installation_type: kvm +controller_sudo_pass: "{{ vault_ctl_host_sudo_pass }}" +disconnected_enabled: true # ⚠️ REQUIRED: Set to true for disconnected installations +``` + +**Important**: This parameter was moved from `disconnected.yaml` to `all.yaml` under Section 1 - Ansible Controller. The default value is `false`. + +### Step 2: Configure secrets.yaml + +Add the registry password to `inventories/default/group_vars/secrets.yaml`: + +```yaml +vault_registry_password: 'your-secure-password-here' +``` + +**Security Note**: Use Ansible Vault to encrypt this file: +```bash +ansible-vault encrypt inventories/default/group_vars/secrets.yaml +``` + +### Step 3: Configure disconnected.yaml + +#### Create disconnected.yaml from Template + +Copy the template file to create your configuration: + +```bash +cp inventories/default/group_vars/disconnected.yaml.template \ + inventories/default/group_vars/disconnected.yaml +``` + +#### Mandatory Configuration Changes + +Edit `inventories/default/group_vars/disconnected.yaml` and update these **required** values: + +**1. Mirror Host Configuration** (lines 49-52): +```yaml +mirroring: + host: + name: your-mirror-host-name # ⚠️ REQUIRED: Hostname of mirror host with internet access + ip: 192.168.1.100 # ⚠️ REQUIRED: IP address of mirror host + user: root # User with sudo access + pass: your-secure-password # ⚠️ REQUIRED: Password for mirror host user +``` + +**2. Registry Certificate** (lines 11-21): +- If `ca_trusted: false` (default): Certificate will be **auto-generated** - no action needed +- If `ca_trusted: true`: Paste your existing certificate in the `ca_cert` field + +**3. Registry Password** (line 52): +- Already references `{{ vault_registry_password }}` from `secrets.yaml` +- Ensure you've set this in Step 1 + +#### Optional Configuration Changes + +These values have sensible defaults but can be customized: + +**Registry Configuration:** +```yaml +bastion: + port: 5000 # Registry port (default: 5000) + username: 'admin' # Registry username (default: admin) + email: 'registry@example.com' # Email for pull secret +``` + +**OCP Version and Operators:** +```yaml +oc_mirror: + image_set: + mirror: + platform: + channels: + - name: stable-4.21 + minVersion: 4.21.14 # Customize OCP version + maxVersion: 4.21.14 + operators: + - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.21 + packages: + - name: serverless-operator # Add/remove operators as needed +``` + +#### Complete Configuration Example + +Here's a complete example with all mandatory values filled: + +```yaml +disconnected: + enabled: true + + registry: + ca_trusted: false # Auto-generates self-signed certificate + + bastion: + enabled: true + port: 5000 + username: 'admin' + password: "{{ vault_registry_password }}" + email: 'registry@example.com' + use_local_repo: true + + mirroring: + host: + name: mirror-host-01 # ⚠️ YOUR MIRROR HOST NAME + ip: 192.168.100.50 # ⚠️ YOUR MIRROR HOST IP + user: root + pass: SecurePassword123! # ⚠️ YOUR MIRROR HOST PASSWORD + + oc_mirror: + image_set: + apiVersion: mirror.openshift.io/v2alpha1 + mirror: + platform: + architectures: + - multi + channels: + - name: stable-4.21 + full: false + minVersion: 4.21.14 + maxVersion: 4.21.14 + operators: + - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.21 + full: false + packages: + - name: serverless-operator + channels: + - name: stable +``` + +#### Configuration Validation Checklist + +Before running the playbook, verify: + +- ✅ `disconnected_enabled: true` set in `all.yaml` (Section 1 - Ansible Controller) +- ✅ `disconnected.yaml` created from template +- ✅ Mirror host `name`, `ip`, and `pass` updated with your values +- ✅ `vault_registry_password` set in `secrets.yaml` +- ✅ `secrets.yaml` encrypted with `ansible-vault encrypt` +- ✅ OCP version matches your target version +- ✅ Required operators listed in `packages` section + +### Step 4: Verify all.yaml Configuration (Original Content Below) + +The original configuration example: + +```yaml +disconnected: + enabled: true # REQUIRED: Change from false to true (if not already done) + + registry: + # Auto-configured when using bastion registry + ca_trusted: false # Auto-generates self-signed certificate + + bastion: + enabled: true # Creates registry on bastion + port: 5000 + username: 'admin' + password: "{{ vault_registry_password }}" # From secrets.yaml + email: 'registry@example.com' # For pull secret + + # Package installation mode + use_local_repo: true # Set to false for fully disconnected (downloads RPMs) + + # Storage directories + data_dir: '/opt/registry/data' + auth_dir: '/opt/registry/auth' + certs_dir: '/opt/registry/certs' + + mirroring: + file_server: + document_root: '/var/www/html' # HTTP server document root + clients_dir: 'clients' # Subdirectory under document_root (accessible at http://:/clients/) + oc_mirror_tgz: 'oc-mirror.tar.gz' + download_dir: '/tmp/oc-mirror-downloads' # Temporary download directory + + bastion: + working_dir: '/opt/oc-mirror' + mirror_output_dir: '/opt/oc-mirror/mirror-output' + + # Download URLs (s390x architecture) + oc_mirror_download: + base_url: "https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/stable/" + oc_mirror_tgz: 'oc-mirror.tar.gz' + + client_download: + ocp_download_url: "https://mirror.openshift.com/pub/openshift-v4/multi/clients/ocp/stable-4.21/s390x/" + ocp_client_tgz: 'openshift-client-linux.tar.gz' + + # oc-mirror configuration + oc_mirror: + oc_mirror_args: + continue_on_error: false # Continue mirroring even if some images fail + source_skip_tls: false # Skip TLS verification for source registries + async_timeout: 7200 # Timeout in seconds (default: 7200 = 2 hours) + async_poll: 30 # Check status every N seconds (default: 30) + oc_mirror_args: + continue_on_error: false + source_skip_tls: false + + image_set: + apiVersion: mirror.openshift.io/v2alpha1 # v2alpha1 for oc-mirror v2 + + mirror: + platform: + architectures: + - multi # Includes s390x in multi-arch images + channels: + - name: stable-4.21 + full: false + minVersion: 4.21.14 + maxVersion: 4.21.14 + + operators: + - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.13 + full: false + packages: + - name: serverless-operator + channels: + - name: stable + + additionalImages: + - name: registry.redhat.io/ubi8/ubi:latest + + helm: {} +``` + +### Step 3: Verify all.yaml Configuration + +Ensure these values are set in `inventories/default/group_vars/all.yaml`: + +```yaml +env: + file_server: + ip: 192.168.1.10 + protocol: http + cfgs_dir: /pub + + bastion: + networking: + ip: 192.168.1.20 + hostname: bastion + base_domain: example.com + + redhat: + pull_secret: '{"auths":{...}}' # Your Red Hat pull secret +``` + +### Step 4: Update Pull Secret (After First Run) + +After the playbook runs, it will create a backup file with updated pull secret: + +```bash +# Review the updated pull secret +cat inventories/default/group_vars/pull_secret_with_registry.json + +# Copy the content and update env.redhat.pull_secret in all.yaml +``` + +Or manually add the bastion registry entry: +```json +{ + "auths": { + "cloud.openshift.com": {...}, + "quay.io": {...}, + "registry.redhat.io": {...}, + "192.168.1.20:5000": { + "auth": "base64-encoded-username:password", + "email": "registry@example.com" + } + } +} +``` + +## Running the Playbook + +### Basic Usage + +Run the complete playbook: + +```bash +ansible-playbook -i inventories/default playbooks/disconnected_setup_oc_mirror.yaml +``` + +With Ansible Vault (interactive password prompt): +```bash +ansible-playbook -i inventories/default playbooks/disconnected_setup_oc_mirror.yaml --ask-vault-pass +``` + +With Ansible Vault (password file): +```bash +# Create a vault password file (protect it with appropriate permissions) +echo 'your-vault-password' > ~/.vault_pass +chmod 600 ~/.vault_pass + +# Run playbook with vault password file +ansible-playbook -i inventories/default playbooks/disconnected_setup_oc_mirror.yaml --vault-password-file ~/.vault_pass +``` + +### Run Specific Stages + +The playbook has multiple stages that can be run independently: + +#### 1. Download Registry RPMs Only (for fully disconnected) +```bash +ansible-playbook -i inventories/default playbooks/disconnected_setup_oc_mirror.yaml --tags download_rpms +``` + +#### 2. Setup Registry Only +```bash +ansible-playbook -i inventories/default playbooks/disconnected_setup_oc_mirror.yaml --tags registry +``` + +#### 3. Download oc-mirror Only +```bash +ansible-playbook -i inventories/default playbooks/disconnected_setup_oc_mirror.yaml --tags download +``` + +#### 4. Setup oc-mirror Only +```bash +ansible-playbook -i inventories/default playbooks/disconnected_setup_oc_mirror.yaml --tags setup +``` + +#### 5. Mirror Images Only +```bash +ansible-playbook -i inventories/default playbooks/disconnected_setup_oc_mirror.yaml --tags mirror +``` + +### Skip Registry Setup (Use External Registry) + +If you have an external registry: + +```bash +ansible-playbook -i inventories/default playbooks/disconnected_setup_oc_mirror.yaml --skip-tags registry +``` + +Then configure `disconnected.registry.url` and `disconnected.registry.ip` in disconnected.yaml. + +## Roles Overview + +The playbook uses six roles in sequence: + +### 1. disconnected_download_registry_rpms + +**Purpose**: Downloads RPM packages for offline installation +**Runs on**: File server (or bastion if same) +**When**: Only if `use_local_repo: false` + +Downloads: +- podman +- httpd-tools +- openssl +- container-selinux +- conmon, crun, fuse-overlayfs, slirp4netns + +**Architecture**: Uses `--archlist=s390x,noarch` for correct architecture + +### 2. disconnected_setup_registry_bastion + +**Purpose**: Creates container registry on bastion +**Runs on**: Bastion + +Actions: +- Installs podman and dependencies +- Generates self-signed certificate (or uses provided) +- Creates htpasswd authentication +- Starts registry as systemd service +- Adds certificate to system trust + +**Service**: `container-registry.service` +**Port**: 5000 (configurable) +**Storage**: `/opt/registry/data` + +### 3. disconnected_update_pull_secret + +**Purpose**: Updates pull secret with registry credentials +**Runs on**: localhost (Ansible controller) + +Actions: +- Parses existing pull secret +- Adds bastion registry credentials +- Updates in-memory for current run +- Creates backup file +- Displays instructions for permanent update + +**Output**: `inventories/default/group_vars/pull_secret_with_registry.json` + +### 4. disconnected_download_oc_mirror + +**Purpose**: Downloads oc-mirror, client tools, and RHCOS rootfs +**Runs on**: File server (or bastion if same) + +Downloads: +- oc-mirror plugin (s390x) +- openshift-client-linux.tar.gz (s390x) +- openshift-install-linux.tar.gz (s390x) +- rhcos-live-rootfs.s390x.img (RHCOS rootfs for node installation) + +**Destinations**: +- Client tools: `{{ env.file_server.document_root }}/clients/` +- RHCOS rootfs: `{{ env.file_server.document_root }}/bin/` + +**Note**: The RHCOS rootfs file is required for node installation and will be served via HTTP during the bootstrap process. + +### 5. disconnected_setup_oc_mirror_bastion + +**Purpose**: Installs oc-mirror on bastion +**Runs on**: Bastion + +Actions: +- Downloads binaries from file server +- Extracts and installs to `/usr/local/bin/` +- Configures pull secret +- Adds registry CA certificate +- Verifies installation + +**Binaries**: oc-mirror, oc, kubectl + +### 6. disconnected_mirror_ocp_bastion + +**Purpose**: Performs OCP image mirroring +**Runs on**: Bastion + +Actions: +- Creates ImageSet configuration +- Runs oc-mirror (v1 or v2) +- Generates cluster manifests +- Copies results to output directory + +**Duration**: 30 minutes to several hours depending on content + +**Output Files**: +- ImageContentSourcePolicy or ImageDigestMirrorSet +- CatalogSource manifests +- Release signatures + +## Playbook Execution Flow + +``` +1. Download Registry RPMs (if use_local_repo=false) + ↓ +2. Setup Container Registry on Bastion + ↓ +3. Update Pull Secret with Registry Credentials + ↓ +4. Download oc-mirror to File Server + ↓ +5. Setup oc-mirror on Bastion + ↓ +6. Mirror OCP Images + ↓ +7. Display Next Steps +``` + +## Post-Mirroring Steps + +After successful mirroring: + +### 1. Review Output Files + +```bash +ssh bastion +ls -la /opt/oc-mirror/mirror-output/ +``` + +### 2. For New Cluster Installation + +Update `install-config.yaml`: + +```yaml +imageContentSources: + - mirrors: + - 192.168.1.20:5000/openshift/release-images + source: quay.io/openshift-release-dev/ocp-release + - mirrors: + - 192.168.1.20:5000/openshift/release + source: quay.io/openshift-release-dev/ocp-v4.0-art-dev + +additionalTrustBundle: | + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- +``` + +### 3. For Existing Cluster + +Apply the generated manifests: + +```bash +# Apply ImageContentSourcePolicy or ImageDigestMirrorSet +oc apply -f /opt/oc-mirror/mirror-output/imageContentSourcePolicy.yaml + +# Apply CatalogSource for operators +oc apply -f /opt/oc-mirror/mirror-output/catalogSource.yaml + +# Wait for nodes to restart and apply configuration +oc get nodes +oc get mcp +``` + +## Troubleshooting + +### Self-Signed Certificate Challenges in Disconnected Environments + +When using a self-signed certificate for your disconnected registry (not from a globally trusted Certificate Authority), several certificate trust issues can arise during OpenShift installation. This section explains the challenges and the solutions implemented in this playbook. + +#### The Problem: Certificate Trust in Container Environments + +**Why Self-Signed Certificates Are Challenging:** + +1. **Multiple Trust Stores**: Different components use different certificate trust mechanisms: + - System trust store: `/etc/pki/ca-trust/source/anchors/` + - Container-specific: `/etc/containers/certs.d//` + - Application-specific: Some tools have their own trust stores + +2. **Ephemeral Containers**: The OpenShift bootstrap process runs `oc` commands inside temporary containers that: + - Don't inherit environment variables from the host + - Don't have access to host certificate files + - Use Go's HTTP client which strictly validates certificates + - Cannot be easily configured with custom certificates + +3. **Bootstrap Isolation**: The bootstrap node runs critical installation scripts (`bootkube.sh`) that: + - Execute `oc adm release info` commands to query release images + - Run inside podman containers with isolated filesystems + - Fail with "x509: certificate signed by unknown authority" errors + - Cannot proceed without trusting the registry certificate + +#### The Solution: Multi-Layer Certificate Trust + +This playbook implements a comprehensive solution that addresses certificate trust at multiple levels: + +**1. System-Level Trust** (Lines 315-356 in [`roles/get_ocp/tasks/main.yaml`](roles/get_ocp/tasks/main.yaml:315-356)) +```yaml +# Certificate added to system trust store +/etc/pki/ca-trust/source/anchors/registry-ca.crt + +# Certificate added for container runtime +/etc/containers/certs.d/172.23.238.65:5000/ca.crt +``` + +**2. Insecure Registry Configuration** (Lines 361-390 in [`roles/get_ocp/tasks/main.yaml`](roles/get_ocp/tasks/main.yaml:361-390)) +```yaml +# Allows podman to skip TLS verification +/etc/containers/registries.conf.d/999-insecure-registry.conf +``` +This enables podman to pull images without certificate validation, which is acceptable in a controlled disconnected environment. + +**3. Bootstrap Script Patching** (Lines 412-430 in [`roles/get_ocp/tasks/main.yaml`](roles/get_ocp/tasks/main.yaml:412-430)) + +The most critical fix: A systemd service (`patch-bootkube-insecure.service`) that runs early in the bootstrap process to modify the `bootkube.sh` script: + +```bash +# Adds --insecure flag to all oc adm release info commands +sed -i -e "s|oc adm release info|oc adm release info --insecure|g" \ + -e "1a# insecure-added" /usr/local/bin/bootkube.sh +``` + +**Why This Is Necessary:** +- The `bootkube.sh` script runs `oc` commands inside ephemeral containers +- These containers cannot access the host's certificate trust store +- The `--insecure` flag tells `oc` to skip TLS verification +- This is the only way to make bootstrap work with self-signed certificates + +#### Final Bootstrap Configuration + +The bootstrap ignition now includes: + +✅ **Certificate in system trust store**: `/etc/pki/ca-trust/source/anchors/registry-ca.crt` +✅ **Certificate for container runtime**: `/etc/containers/certs.d/172.23.238.65:5000/ca.crt` +✅ **Insecure registry configuration**: `/etc/containers/registries.conf.d/999-insecure-registry.conf` +✅ **Bootstrap script patcher**: `patch-bootkube-insecure.service` systemd unit + +**Security Considerations:** + +Using `--insecure` and `insecure = true` is acceptable in disconnected environments because: +- The registry is on a trusted internal network +- No external/untrusted registries are involved +- The alternative (bootstrap failure) is worse +- Production clusters can use proper certificates from internal CAs + +**For Production Environments:** + +Consider using certificates from an internal Certificate Authority (CA) that: +- Is trusted by your organization +- Can be pre-installed in RHCOS images +- Eliminates the need for insecure flags +- Provides proper certificate chain validation + +Set `disconnected.registry.ca_trusted: true` and provide your CA certificate in `disconnected.registry.ca_cert` to use this approach. + +### Registry Issues + +#### Registry service fails to start + +**Check logs**: +```bash +journalctl -u container-registry -f +``` + +**Common causes**: +- Port 5000 already in use: `ss -tlnp | grep 5000` +- Podman not installed: `podman --version` +- Permissions on directories + +**Solution**: +```bash +# Check service status +systemctl status container-registry + +# Restart service +systemctl restart container-registry + +# Check podman directly +podman ps -a +``` + +#### Certificate errors + +**Symptoms**: "x509: certificate signed by unknown authority" + +**Solution**: +```bash +# Verify certificate in trust anchors +ls /etc/pki/ca-trust/source/anchors/ + +# Update trust +update-ca-trust + +# Check certificate +openssl x509 -in /opt/registry/certs/registry.crt -text -noout +``` + +### Mirroring Issues + +#### Authentication failures + +**Symptoms**: "unauthorized" or "authentication required" + +**Solution**: +- Verify pull secret includes all registries +- Check registry credentials are correct +- Ensure pull secret is valid JSON: + ```bash + echo '{"auths":{...}}' | python -m json.tool + ``` + +#### Out of disk space + +**Symptoms**: "no space left on device" + +**Solution**: +```bash +# Check disk usage +df -h /opt/oc-mirror +df -h /opt/registry/data + +# Clean up if needed +podman system prune -a + +# Consider using different mount point with more space +``` + +#### Network timeouts + +**Symptoms**: Connection timeouts during mirroring + +**Solution**: +- Enable `continue_on_error: true` in disconnected.yaml +- Check network connectivity to Red Hat registries +- Consider mirroring in smaller batches + +### Download Issues + +#### RPM download fails + +**Symptoms**: yumdownloader errors + +**Solution**: +```bash +# Verify yum-utils is installed +yum install yum-utils + +# Check repository configuration +yum repolist + +# Test download manually +yumdownloader --archlist=s390x,noarch podman +``` + +#### oc-mirror download fails + +**Symptoms**: HTTP 404 or connection errors + +**Solution**: +- Verify URL is correct for your architecture +- Check internet connectivity +- Try alternative mirror URLs +- Verify OCP version exists at specified URL + +## Advanced Topics + +### Using External Registry + +If you have an existing registry: + +1. Set `disconnected.registry.bastion.enabled: false` +2. Configure registry URL and IP: + ```yaml + disconnected: + registry: + url: 'registry.example.com:5000' + ip: '192.168.1.50' + ``` +3. Skip registry setup: + ```bash + ansible-playbook ... --skip-tags registry + ``` + +### Mirroring Multiple OCP Versions + +Update the image_set configuration: + +```yaml +mirror: + platform: + channels: + - name: stable-4.21 + minVersion: 4.21.14 + maxVersion: 4.21.20 + - name: stable-4.22 + minVersion: 4.22.0 + maxVersion: 4.22.5 +``` + +### Mirroring Specific Operators + +```yaml +operators: + - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.13 + packages: + - name: serverless-operator + channels: + - name: stable + minVersion: '1.30.0' + maxVersion: '1.32.0' + - name: elasticsearch-operator + channels: + - name: stable +``` + +### Using oc-mirror v1 vs v2 + +**oc-mirror v1** (apiVersion: mirror.openshift.io/v1alpha2): +- Requires storageConfig +- Uses oc-mirror-workspace directory +- Generates mapping.txt + +**oc-mirror v2** (apiVersion: mirror.openshift.io/v2alpha1): +- No storageConfig needed +- Uses working-dir structure +- Generates cluster-resources directory + +Configure in disconnected.yaml: +```yaml +oc_mirror: + image_set: + apiVersion: mirror.openshift.io/v2alpha1 # or v1alpha2 +``` + +### Fully Disconnected Installation (No Repository Access) + +For environments without any repository access: + +1. Set `use_local_repo: false` in disconnected.yaml +2. Run on a host with internet access: + ```bash + ansible-playbook ... --tags download_rpms + ``` +3. Transfer downloaded RPMs to file server +4. Run full playbook on disconnected bastion + +### Performance Tuning + +#### Disk Space Requirements +- **Minimal** (single OCP version): 50-100 GB +- **Standard** (OCP + operators): 200-300 GB +- **Full** (complete catalog): 500GB-1TB + +#### Network Bandwidth +- Recommended: 100 Mbps or higher +- Consider running during off-peak hours +- Use `continue_on_error: true` for unreliable connections + +#### Time Estimates +- Single OCP version: 30-60 minutes +- OCP + 5-10 operators: 1-3 hours +- Full operator catalog: 4-8 hours + +### Security Best Practices + +1. **Use Ansible Vault** for sensitive data: + ```bash + ansible-vault encrypt inventories/default/group_vars/secrets.yaml + ``` + +2. **Change default passwords**: + - Registry password in secrets.yaml + - Use strong, unique passwords + +3. **Certificate management**: + - Use proper certificates in production + - Don't use `source_skip_tls` in production + +4. **Limit access**: + - Restrict bastion access to authorized users + - Use firewall rules for registry port + +5. **Regular backups**: + - Backup registry data directory + - Backup mirrored content + - Document configuration + +## Storage Management + +### Registry Storage + +Monitor disk usage: +```bash +# Check registry data directory +du -sh /opt/registry/data + +# Check for old images +podman images + +# Clean up if needed +podman system prune -a +``` + +### Mirror Output Storage + +```bash +# Check mirror output +du -sh /opt/oc-mirror/mirror-output + +# Archive old mirroring results +tar -czf mirror-backup-$(date +%Y%m%d).tar.gz /opt/oc-mirror/mirror-output +``` + +## References + +- [Red Hat OpenShift Disconnected Installation](https://docs.redhat.com/en/documentation/openshift_container_platform/4.14/html/disconnected_installation_mirroring/) +- [oc-mirror Plugin Documentation](https://docs.redhat.com/en/documentation/openshift_container_platform/4.14/html/disconnected_installation_mirroring/installing-mirroring-disconnected#installation-oc-mirror-installing-plugin_installing-mirroring-disconnected) +- [ImageSet Configuration Reference](https://docs.redhat.com/en/documentation/openshift_container_platform/4.14/html/disconnected_installation_mirroring/installing-mirroring-disconnected#oc-mirror-imageset-config-params_installing-mirroring-disconnected) +- [Podman Documentation](https://docs.podman.io/) +- [Container Registry Documentation](https://docs.docker.com/registry/) + +## Support + +For issues specific to this playbook: +- Check the troubleshooting section above +- Review Ansible logs for detailed error messages +- Verify all prerequisites are met + +For oc-mirror or OpenShift issues: +- Consult Red Hat support +- Review Red Hat documentation +- Check OpenShift community forums + +## Summary + +This playbook provides a complete solution for disconnected OpenShift installations: + +✅ Automated registry setup with self-signed certificates +✅ Automatic pull secret management +✅ s390x architecture support throughout +✅ Flexible package installation (repo or offline RPMs) +✅ Support for both oc-mirror v1 and v2 +✅ Comprehensive error handling and logging +✅ Secure password management via Ansible Vault + +Follow this guide to successfully mirror OCP images and prepare for disconnected cluster installation. \ No newline at end of file diff --git a/docs/set-variables-group-vars.md b/docs/set-variables-group-vars.md index 374002664..4de93f0c7 100644 --- a/docs/set-variables-group-vars.md +++ b/docs/set-variables-group-vars.md @@ -12,6 +12,7 @@ :--- | :--- | :--- **installation_type** | Can be of type kvm or lpar. Some packages will be ignored for installation in case of non lpar based installation. | kvm **controller_sudo_pass** | The password to the machine running Ansible (localhost). This will only be used for two things. To ensure you've installed the pre-requisite packages if you're on Linux, and to add the login URL to your /etc/hosts file. | Pas$w0rd! +**disconnected_enabled** | Enable disconnected/air-gapped installation mode. Set to `true` for environments without direct internet access. When enabled, the playbooks will use a local container registry or existing registry for mirroring OCP images and operators. Default value is `false`. See [disconnected installation guide](run-the-playbooks-for-disconnected-install.md) for complete setup instructions. | false **cex_device** | Specify the storage device type used for LUKS encryption. This setting determines enable cex MCO Ignition configuration will be applied. Use in combination with the cex parameter. [dasd, fcp, virt] **regenerate_private_key** | (Optional) Controls whether to regenerate SSH private keys during the setup process. Default value is 'full_idempotence'. For usage inside of pipelines where SSH keys already exist, this value should be set to 'never' to preserve existing keys. See detailed description [here:](https://docs.ansible.com/projects/ansible/latest/collections/community/crypto/openssh_keypair_module.html) | full_idempotence **check_nodes_delay** | (Optional) Delay in seconds between retries when checking if control and compute nodes are in 'Ready' state. Used during the check_nodes role execution. Default value is 30 seconds. | 30 @@ -165,7 +166,7 @@ **env.ocp_key_name** | Comment to describe the SSH key used for OCP. Arbitrary value. | OCPZ-01 key **env.vnet_name** | (Optional) Name of the bridged virtual network that will be created on the KVM host if network mode is not set to NAT. In case of NAT network mode the name of the NAT network definition used to create the nodes(usually it is 'default'). If NAT is being used and a jumphost is needed, the parameters network_mode, jumphost.name, jumphost.user and jumphost.pass must be specified, too. For default (NAT) network verify that the configured IP ranges does not interfere with the IPs defined for the controle and compute nodes. Modify the default network (dhcp range setting) to prevent issues with VMs using dhcp and OCP nodes having fixed IPs. Default is create a bridge network.| macvtap-net **env.network_mode** | (Optional) In case the network mode will be NAT and the installation will be executed from remote (e.g. your laptop), a jumphost needs to be defined to let the installation access the bastion host. If macvtap for networking is being used this variable should be empty. | NAT -**env.use_ipv6** | If ipv6 addresses should be assigned to the controle and compute nodes, this variable should be true (default) and the matching ipv6 settings should be specified. | True +**env.use_ipv6** | If ipv6 addresses should be assigned to the controle and compute nodes, this variable should be true (default) and the matching ipv6 settings should be specified. When [`env.use_ipv6`](docs/set-variables-group-vars.md:169) is `True` and [`env.use_dhcp`](docs/set-variables-group-vars.md:170) is `False`, [`playbooks/0_setup.yaml`](playbooks/0_setup.yaml) reports a warning-style failed task if IPv6 values are missing for configured control, compute, or infra nodes, but execution continues. | True **env.use_dhcp** | If dhcp service should be used to get an IP address, this variable should be true and the matching mac address must be specified. | False **env.jumphost.name** | (Optional) If env.network.mode is set to 'NAT' the name of the jumphost (e.g. the name of KVM host if used as jumphost) should be specified. | kvm-host-01 **env.jumphost.ip** | (Optional) The ip of the jumphost. | 192.168.10.1 diff --git a/docs/troubleshooting-disconnected-internet-access.md b/docs/troubleshooting-disconnected-internet-access.md new file mode 100644 index 000000000..d72c1357c --- /dev/null +++ b/docs/troubleshooting-disconnected-internet-access.md @@ -0,0 +1,222 @@ +# Troubleshooting Internet Access for Disconnected Setup + +## Problem +When running the `disconnected_setup_oc_mirror.yaml` playbook, downloads fail with errors like: +``` +Could not find or access '/tmp/oc-mirror-downloads/openshift-client-linux.tar.gz' +``` + +This happens because the bastion cannot reach the internet to download required files. + +## Root Cause +Even though the KVM host has IP forwarding enabled, the bastion VM may lack: +- Proper DNS configuration +- Correct default gateway +- Required firewall rules + +## Solution Steps + +### 1. Verify DNS Configuration on Bastion + +SSH to the bastion and check DNS: +```bash +cat /etc/resolv.conf +``` + +Should contain valid nameservers, for example: +``` +nameserver 8.8.8.8 +nameserver 8.8.4.4 +``` + +If missing or incorrect, add DNS servers: +```bash +echo "nameserver 8.8.8.8" | sudo tee -a /etc/resolv.conf +echo "nameserver 8.8.4.4" | sudo tee -a /etc/resolv.conf +``` + +For persistent DNS configuration on RHEL/CentOS: +```bash +sudo nmcli con mod "System eth0" ipv4.dns "8.8.8.8 8.8.4.4" +sudo nmcli con up "System eth0" +``` + +### 2. Test DNS Resolution + +```bash +nslookup mirror.openshift.com +# or +dig mirror.openshift.com +``` + +Should return IP addresses. If it fails, DNS is not working. + +### 3. Verify Default Gateway + +Check routing table: +```bash +ip route show +``` + +Should show a default route, for example: +``` +default via 192.168.122.1 dev eth0 +``` + +If missing, add default gateway (replace with your KVM host IP): +```bash +sudo ip route add default via 192.168.122.1 +``` + +For persistent configuration: +```bash +sudo nmcli con mod "System eth0" ipv4.gateway "192.168.122.1" +sudo nmcli con up "System eth0" +``` + +### 4. Verify KVM Host IP Forwarding + +On the KVM host, check if IP forwarding is enabled: +```bash +sysctl net.ipv4.ip_forward +``` + +Should return `net.ipv4.ip_forward = 1`. If not: +```bash +sudo sysctl -w net.ipv4.ip_forward=1 +# Make it persistent +echo "net.ipv4.ip_forward = 1" | sudo tee -a /etc/sysctl.conf +``` + +### 5. Configure NAT/Masquerading on KVM Host + +The KVM host needs to masquerade traffic from the bastion: +```bash +# Check current iptables rules +sudo iptables -t nat -L -n -v + +# Add masquerading rule (replace virbr0 with your bridge interface) +sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE +sudo iptables -A FORWARD -i virbr0 -o eth0 -j ACCEPT +sudo iptables -A FORWARD -i eth0 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT + +# Save rules (RHEL/CentOS) +sudo iptables-save | sudo tee /etc/sysconfig/iptables + +# Or use firewalld +sudo firewall-cmd --permanent --add-masquerade +sudo firewall-cmd --reload +``` + +### 6. Test Internet Connectivity from Bastion + +```bash +# Test DNS and connectivity +curl -I https://mirror.openshift.com + +# Test with verbose output +curl -v https://mirror.openshift.com + +# Test specific download URL +curl -I https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/stable/oc-mirror.tar.gz +``` + +### 7. Check Firewall on Bastion + +Ensure the bastion firewall allows outbound connections: +```bash +# Check firewall status +sudo firewall-cmd --state + +# If needed, allow outbound HTTPS +sudo firewall-cmd --permanent --add-service=https +sudo firewall-cmd --reload +``` + +### 8. Verify Network Configuration + +Check the bastion's network interface configuration: +```bash +ip addr show +ip route show +nmcli con show +``` + +Ensure: +- IP address is assigned +- Subnet mask is correct +- Gateway is reachable: `ping 192.168.122.1` + +## Updated Playbook Behavior + +The updated `disconnected_download_oc_mirror` role now includes: + +1. **Pre-flight connectivity check**: Tests connection to mirror.openshift.com before attempting downloads +2. **Detailed error messages**: Provides troubleshooting steps if connectivity fails +3. **File verification**: Confirms files exist before attempting to copy them +4. **Better error handling**: Fails fast with clear messages instead of continuing with missing files + +## Running the Playbook Again + +After fixing connectivity issues, run the playbook: +```bash +ansible-playbook -i inventories/default playbooks/disconnected_setup_oc_mirror.yaml +``` + +The playbook will now: +1. Test internet connectivity first +2. Show clear error messages if connectivity fails +3. Provide troubleshooting guidance +4. Only proceed with downloads if connectivity is confirmed + +## Alternative: Manual Download + +If internet access cannot be configured on the bastion, you can manually download files: + +1. Download on a machine with internet access: +```bash +# Create download directory +mkdir -p /tmp/oc-mirror-downloads + +# Download files +cd /tmp/oc-mirror-downloads +curl -LO https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/stable/oc-mirror.tar.gz +curl -LO https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/stable-4.21/openshift-client-linux.tar.gz +curl -LO https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/stable-4.21/openshift-install-linux.tar.gz +curl -LO https://mirror.openshift.com/pub/openshift-v4/s390x/dependencies/rhcos/4.21/latest/rhcos-live-rootfs.s390x.img +``` + +2. Transfer to bastion: +```bash +scp /tmp/oc-mirror-downloads/* root@bastion-ip:/tmp/oc-mirror-downloads/ +``` + +3. Skip the download tasks and run only the copy/setup tasks: +```bash +ansible-playbook -i inventories/default playbooks/disconnected_setup_oc_mirror.yaml --skip-tags download +``` + +## Common Issues + +### Issue: "Temporary failure in name resolution" +**Cause**: DNS not configured +**Fix**: Add nameservers to /etc/resolv.conf (see step 1) + +### Issue: "No route to host" +**Cause**: Missing default gateway +**Fix**: Add default route (see step 3) + +### Issue: "Connection timed out" +**Cause**: KVM host not forwarding packets or firewall blocking +**Fix**: Enable IP forwarding and NAT on KVM host (see steps 4-5) + +### Issue: Downloads succeed but files not found +**Cause**: Files downloaded to wrong location or permissions issue +**Fix**: Check `disconnected.yaml` download_dir setting and file permissions + +## Support + +For additional help, check: +- [Main troubleshooting guide](troubleshooting.md) +- [Disconnected installation documentation](run-the-playbooks-for-disconnected-install.md) +- OpenShift documentation: https://docs.openshift.com/container-platform/latest/installing/disconnected_install/index.html \ No newline at end of file diff --git a/docs/troubleshooting-oc-mirror-no-release-images.md b/docs/troubleshooting-oc-mirror-no-release-images.md new file mode 100644 index 000000000..85549d237 --- /dev/null +++ b/docs/troubleshooting-oc-mirror-no-release-images.md @@ -0,0 +1,227 @@ +# Troubleshooting: oc-mirror "no release images found" Error + +## Problem +When running `oc-mirror`, you get the error: +``` +[ERROR] : [Executor] collection error: [GetReleaseReferenceImages] no release images found +``` + +## Root Cause +This error occurs when oc-mirror cannot find any release images matching your version specification in the imageset-config.yaml. Common causes: + +1. **Version doesn't exist**: The minVersion/maxVersion specified doesn't exist in the channel +2. **Wrong channel name**: The channel name doesn't match available channels +3. **Architecture mismatch**: The specified architecture doesn't have releases in that version +4. **Network issues**: Cannot reach the source registry to query available versions + +## Solution Steps + +### 1. Find Available Versions + +First, determine what versions are actually available for your architecture and channel: + +```bash +# SSH to bastion +ssh root@bastion-ip + +# List available versions in stable-4.21 channel for s390x +oc adm release info quay.io/openshift-release-dev/ocp-release:4.21-s390x + +# Or check what's available in the channel +curl -s https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/stable-4.21/ | grep -oP '4\.21\.\d+' | sort -V | uniq + +# For multi-arch +curl -s https://mirror.openshift.com/pub/openshift-v4/multi/clients/ocp/stable-4.21/ | grep -oP '4\.21\.\d+' | sort -V | uniq +``` + +### 2. Check Your Current Configuration + +Review your `disconnected.yaml` configuration: + +```yaml +mirror: + platform: + architectures: + - s390x # or 'multi' for multi-arch + channels: + - name: stable-4.21 + full: false + minVersion: 4.21.21 # ← This version must exist! + maxVersion: 4.21.21 +``` + +### 3. Common Version Issues + +#### Issue: Version Too New +**Problem**: You specified `4.21.21` but only `4.21.0` through `4.21.14` exist. + +**Solution**: Use an existing version: +```yaml +channels: + - name: stable-4.21 + full: false + minVersion: 4.21.0 + maxVersion: 4.21.14 +``` + +#### Issue: Using Latest/Newest +**Problem**: You want the latest version but don't know the exact number. + +**Solution**: Either use `full: true` to get all versions, or omit min/maxVersion: +```yaml +channels: + - name: stable-4.21 + # Option 1: Get all versions (large download) + full: true + + # Option 2: Get latest only (omit min/max) + # (This gets the latest in the channel) +``` + +#### Issue: Architecture Mismatch +**Problem**: Using `s390x` architecture but the version only exists for `multi`. + +**Solution**: Check which architecture tag exists: +```bash +# Check if s390x-specific tag exists +skopeo inspect docker://quay.io/openshift-release-dev/ocp-release:4.21.14-s390x + +# Check if multi-arch tag exists +skopeo inspect docker://quay.io/openshift-release-dev/ocp-release:4.21.14-multi +``` + +Then update your configuration: +```yaml +mirror: + platform: + architectures: + - multi # Use 'multi' instead of 's390x' if that's what's available +``` + +### 4. Recommended Configuration for s390x + +For s390x systems, use this proven configuration: + +```yaml +disconnected: + mirroring: + oc_mirror: + image_set: + apiVersion: mirror.openshift.io/v2alpha1 + mirror: + platform: + architectures: + - s390x + channels: + - name: stable-4.21 + full: false + minVersion: 4.21.0 # Use first available version + maxVersion: 4.21.14 # Use last known good version +``` + +### 5. Test Your Configuration + +Before running the full mirror, test with a dry-run: + +```bash +# SSH to bastion +cd /opt/oc-mirror + +# Run dry-run to validate configuration +oc mirror --v2 --config imageset-config.yaml \ + --workspace file:///opt/oc-mirror/oc-mirror-workspace \ + docker://192.168.122.2:5000 \ + --dest-tls-verify=false \ + --dry-run +``` + +If successful, you'll see: +``` +[INFO] : 🔍 collecting release images... +[INFO] : found X release images +``` + +If it fails with "no release images found", the version doesn't exist. + +### 6. Alternative: Use Specific Release Image + +Instead of using channels, you can specify an exact release image: + +```yaml +mirror: + platform: + architectures: + - s390x + channels: + - name: stable-4.21 + type: ocp + full: false + # Or use graph to specify exact release + graph: true +``` + +### 7. Check oc-mirror Version Compatibility + +Ensure your oc-mirror version supports the OpenShift version you're trying to mirror: + +```bash +# Check oc-mirror version +oc mirror version + +# Check if it supports v2 API +oc mirror --help | grep -i v2 +``` + +For OpenShift 4.21, you need oc-mirror v2 (which you're using based on the error). + +## Quick Fix for Your Current Error + +Based on your error, try this immediate fix: + +1. **Option A: Use a range of versions** (recommended) +```yaml +channels: + - name: stable-4.21 + full: false + minVersion: 4.21.0 + maxVersion: 4.21.14 +``` + +2. **Option B: Get all versions in channel** +```yaml +channels: + - name: stable-4.21 + full: true +``` + +3. **Option C: Omit version constraints** (gets latest) +```yaml +channels: + - name: stable-4.21 +``` + +## Verify Available Versions Online + +Check Red Hat's official mirror to see what's available: + +- s390x: https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/stable-4.21/ +- multi-arch: https://mirror.openshift.com/pub/openshift-v4/multi/clients/ocp/stable-4.21/ + +Look for directories with version numbers like `4.21.0`, `4.21.1`, etc. + +## After Fixing Configuration + +1. Update your `disconnected.yaml` with correct versions +2. Re-run the playbook: +```bash +ansible-playbook -i inventories/default playbooks/disconnected_setup_oc_mirror.yaml +``` + +3. The imageset-config.yaml will be regenerated with correct versions +4. oc-mirror should now find and mirror the release images + +## Additional Resources + +- [OpenShift Mirror Documentation](https://docs.openshift.com/container-platform/latest/installing/disconnected_install/installing-mirroring-disconnected.html) +- [oc-mirror Plugin Documentation](https://docs.openshift.com/container-platform/latest/installing/disconnected_install/installing-mirroring-creating-registry.html) +- [Available OpenShift Releases](https://mirror.openshift.com/pub/openshift-v4/) \ No newline at end of file diff --git a/docs/troubleshooting-oc-mirror-timeout.md b/docs/troubleshooting-oc-mirror-timeout.md new file mode 100644 index 000000000..bb8d4e154 --- /dev/null +++ b/docs/troubleshooting-oc-mirror-timeout.md @@ -0,0 +1,307 @@ +# Troubleshooting: oc-mirror Timeout Issues + +## Problem +When running the `disconnected_setup_oc_mirror.yaml` playbook, the mirroring operation times out with an error like: +``` +async task did not complete within the requested time +``` + +Or the playbook appears to hang during the mirroring phase. + +## Root Cause +The mirroring operation takes longer than the configured `async_timeout` value. Large mirroring operations (multiple OpenShift versions, full operator catalogs, or many additional images) can take several hours to complete. + +## Solution + +### 1. Adjust the Timeout Value + +Edit your `disconnected.yaml` configuration and increase the `async_timeout` value: + +```yaml +disconnected: + mirroring: + oc_mirror: + oc_mirror_args: + async_timeout: 28800 # Increase based on your needs + async_poll: 30 +``` + +### 2. Recommended Timeout Values + +Choose a timeout based on what you're mirroring: + +#### Small Mirroring Operation (2 hours = 7200 seconds) +**Use when mirroring:** +- Single OpenShift version +- Few operators (1-3) +- Minimal additional images + +```yaml +async_timeout: 7200 +``` + +#### Medium Mirroring Operation (4 hours = 14400 seconds) - **DEFAULT** +**Use when mirroring:** +- 2-3 OpenShift versions +- Several operators (4-10) +- Some additional images + +```yaml +async_timeout: 14400 +``` + +#### Large Mirroring Operation (8 hours = 28800 seconds) +**Use when mirroring:** +- Full channel (many versions) +- Many operators (10-20) +- Multiple additional images +- Full operator catalogs + +```yaml +async_timeout: 28800 +``` + +#### Very Large Mirroring Operation (12 hours = 43200 seconds) +**Use when mirroring:** +- Multiple channels +- Full operator catalogs with `full: true` +- Extensive additional images +- Multiple architectures + +```yaml +async_timeout: 43200 +``` + +### 3. Estimating Your Timeout Needs + +Consider these factors when setting your timeout: + +**OpenShift Platform Images:** +- Each version: ~5-10 GB +- Time per version: 15-30 minutes (depending on network speed) + +**Operator Catalogs:** +- Single operator: 100 MB - 2 GB +- Full catalog (`full: true`): 50-100 GB +- Time for full catalog: 2-4 hours + +**Additional Images:** +- Varies widely based on image size +- Add 10-20 minutes per additional image + +**Network Speed Impact:** +- Fast connection (100+ Mbps): Use lower timeout values +- Moderate connection (10-100 Mbps): Use recommended values +- Slow connection (<10 Mbps): Double the recommended values + +### 4. Configuration Examples + +#### Example 1: Minimal Setup (Single Version, No Operators) +```yaml +disconnected: + mirroring: + oc_mirror: + oc_mirror_args: + async_timeout: 7200 # 2 hours + async_poll: 30 + image_set: + mirror: + platform: + architectures: + - s390x + channels: + - name: stable-4.21 + minVersion: 4.21.14 + maxVersion: 4.21.14 + operators: [] # No operators + additionalImages: [] +``` + +#### Example 2: Production Setup (Multiple Versions, Several Operators) +```yaml +disconnected: + mirroring: + oc_mirror: + oc_mirror_args: + async_timeout: 28800 # 8 hours + async_poll: 30 + image_set: + mirror: + platform: + architectures: + - s390x + channels: + - name: stable-4.21 + minVersion: 4.21.0 + maxVersion: 4.21.14 + operators: + - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.21 + packages: + - name: serverless-operator + - name: openshift-gitops-operator + - name: odf-operator + additionalImages: + - name: registry.redhat.io/ubi9/ubi:latest +``` + +#### Example 3: Full Mirror (Everything) +```yaml +disconnected: + mirroring: + oc_mirror: + oc_mirror_args: + async_timeout: 43200 # 12 hours + async_poll: 30 + image_set: + mirror: + platform: + architectures: + - s390x + channels: + - name: stable-4.21 + full: true # All versions in channel + operators: + - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.21 + full: true # All operators +``` + +### 5. Monitoring Progress + +While the mirroring is running, you can monitor progress on the bastion: + +```bash +# SSH to bastion +ssh root@bastion-ip + +# Watch oc-mirror logs +tail -f /opt/oc-mirror/oc-mirror.log + +# Check disk space (mirroring requires significant space) +df -h /opt/registry/data + +# Monitor network activity +iftop -i eth0 +``` + +### 6. If Timeout Still Occurs + +If you've increased the timeout but still experience issues: + +#### Check Available Disk Space +```bash +# On bastion +df -h /opt/registry/data +df -h /opt/oc-mirror +``` + +Mirroring requires: +- Registry storage: 100-500 GB depending on content +- Workspace: 50-100 GB for temporary files + +#### Check Network Stability +```bash +# Test sustained connectivity +ping -c 100 quay.io +curl -I https://registry.redhat.io +``` + +#### Review oc-mirror Logs +```bash +# On bastion +cd /opt/oc-mirror +cat oc-mirror-workspace/logs/oc-mirror.log +``` + +Look for: +- Network errors +- Authentication failures +- Disk space issues + +#### Use Continue on Error (Carefully) +For very large mirrors, you can enable continue-on-error: + +```yaml +oc_mirror_args: + continue_on_error: true # Continue even if some images fail + async_timeout: 43200 +``` + +**Warning**: This may result in incomplete mirroring. Review logs carefully. + +### 7. Alternative: Split the Mirroring + +For extremely large mirrors, consider splitting into multiple runs: + +**Run 1: Platform Images Only** +```yaml +mirror: + platform: + channels: + - name: stable-4.21 + minVersion: 4.21.14 + maxVersion: 4.21.14 + operators: [] +``` + +**Run 2: Operators** +```yaml +mirror: + platform: + channels: [] # Skip platform + operators: + - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.21 + packages: + - name: serverless-operator +``` + +### 8. Best Practices + +1. **Start Small**: Begin with a single version and few operators +2. **Test First**: Use `--dry-run` to estimate size and time +3. **Monitor Resources**: Watch disk space and network during mirroring +4. **Plan Timing**: Run large mirrors during off-hours +5. **Document**: Keep notes on how long different configurations take +6. **Incremental Updates**: After initial mirror, updates are much faster + +### 9. Timeout Calculation Formula + +Use this formula to estimate your timeout: + +``` +timeout = (platform_versions × 1800) + (operators × 600) + (additional_images × 300) + 3600 + +Where: +- platform_versions: Number of OpenShift versions to mirror +- operators: Number of operator packages (or 100 if full: true) +- additional_images: Number of additional images +- 3600: Base overhead (1 hour) +``` + +**Example Calculation:** +- 5 OpenShift versions +- 10 operators +- 5 additional images + +``` +timeout = (5 × 1800) + (10 × 600) + (5 × 300) + 3600 + = 9000 + 6000 + 1500 + 3600 + = 20100 seconds (≈ 5.6 hours) + +Recommended: 28800 seconds (8 hours) for safety margin +``` + +## Quick Reference + +| Content Size | Timeout (seconds) | Timeout (hours) | Use Case | +|--------------|-------------------|-----------------|----------| +| Small | 7200 | 2 | Single version, few operators | +| Medium | 14400 | 4 | Multiple versions, several operators | +| Large | 28800 | 8 | Full channel, many operators | +| Very Large | 43200 | 12 | Multiple channels, full catalogs | +| Extreme | 86400 | 24 | Everything, multiple architectures | + +## Related Documentation + +- [DNS Configuration for Disconnected Mirroring](disconnected-dns-configuration.md) +- [oc-mirror "No Release Images Found" Error](troubleshooting-oc-mirror-no-release-images.md) +- [Disconnected Installation Guide](run-the-playbooks-for-disconnected-install.md) \ No newline at end of file diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 804e9bb75..abb616835 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -1,24 +1,84 @@ # Troubleshooting -If you encounter errors while running the main playbook, there are a few things you can do: +If you encounter errors while running the main playbook, there are a few things you can do: -* Double check your variables. -* Inspect the part that failed by opening the playbook or role at roles/role-name/tasks/main.yaml -* Google the specific error message. -* Re-run the role with the verbosity '-v' option to get more debugging information (more v's give more info). For example: +* Double check your variables. +* Inspect the part that failed by opening the playbook or role at roles/role-name/tasks/main.yaml +* Google the specific error message. +* Re-run the role with the verbosity '-v' option to get more debugging information (more v's give more info). For example: ``` ansible-playbook playbooks/setup_bastion.yaml -vvv ``` * Use tags - * To be more selective with what parts of a playbook are run, use tags. + * To be more selective with what parts of a playbook are run, use tags. * To determine what part of a playbook you would like to run, open the playbook you'd like to run and find the roles parameter. Each [role](https://github.com/IBM/Ansible-OpenShift-Provisioning/tree/main/roles) has a corresponding tag. * There are also occasionally tags for sections of a playbook or within the role themselves. - * This is especially helpful for troubleshooting. You can add in tags under the `name` parameter for individual tasks you'd like to run. + * This is especially helpful for troubleshooting. You can add in tags under the `name` parameter for individual tasks you'd like to run. * Here's an example of using a tag: ``` ansible-playbook playbooks/setup_kvm_host.yaml --tags "section_2,section_3" ``` * This runs only the parts of the [setup_kvm_host playbook](https://github.com/IBM/Ansible-OpenShift-Provisioning/blob/main/playbooks/3_setup_kvm_host.yaml) marked with tags section_2 and section_3. To use more than one tag, they must be quoted (single or double) and comma-separated (with or without spaces between). -* E-mail Jacob Emery at jacob.emery@ibm.com + +## Specialized Troubleshooting Guides + +For specific scenarios, refer to these detailed troubleshooting guides: + +### Disconnected Installation Issues + +* **[DNS Configuration for Disconnected Mirroring](disconnected-dns-configuration.md)** - Resolves DNS issues when the bastion needs internet access for mirroring but DNS service isn't configured yet +* **[Internet Access Troubleshooting](troubleshooting-disconnected-internet-access.md)** - Step-by-step guide for fixing connectivity issues when bastion cannot reach mirror.openshift.com +* **[oc-mirror "No Release Images Found" Error](troubleshooting-oc-mirror-no-release-images.md)** - Fixes version specification issues and helps find available OpenShift versions +* **[oc-mirror Timeout Issues](troubleshooting-oc-mirror-timeout.md)** - Configure appropriate timeouts for large mirroring operations to prevent "async task did not complete" errors +* **[Monitoring OC-Mirror Progress](monitoring-oc-mirror-progress.md)** - Real-time monitoring of image mirroring to see which images are being pulled + +### Common Disconnected Scenarios + +**Problem**: Downloads fail with "Could not find or access file" or "Temporary failure in name resolution" +**Solution**: See [Internet Access Troubleshooting](troubleshooting-disconnected-internet-access.md) and [DNS Configuration](disconnected-dns-configuration.md) + +**Problem**: oc-mirror reports "no release images found" +**Solution**: See [oc-mirror Troubleshooting](troubleshooting-oc-mirror-no-release-images.md) + +**Problem**: Mirroring times out with "async task did not complete within the requested time" +**Solution**: See [oc-mirror Timeout Configuration](troubleshooting-oc-mirror-timeout.md) + +**Problem**: Want to see which images are being mirrored in real-time +**Solution**: See [Monitoring OC-Mirror Progress](monitoring-oc-mirror-progress.md) + +**Problem**: Bastion has IP forwarding but still can't reach internet +**Solution**: Check DNS configuration in [DNS Configuration Guide](disconnected-dns-configuration.md) + +## Inventory Naming Conflicts + +**Problem**: Ansible connects to the bastion with the controller username instead of the configured [`ansible_user`](inventories/default/hosts), and verbose SSH output shows `ESTABLISH SSH CONNECTION FOR USER: None`. + +**Cause**: The inventory defines both a group named [`[bastion]`](inventories/default/hosts) and a host named [`bastion`](inventories/default/hosts). This creates an ambiguous inventory pattern. When a play targets [`bastion`](inventories/default/hosts), Ansible can resolve the host pattern to the host object instead of the intended group member, which may result in missing effective connection variables and SSH falling back to the controller username. + +**Symptoms**: +* [`ansible-inventory --graph`](ansible.cfg) prints `Found both group and host with same name: bastion` +* Verbose playbook output shows `ESTABLISH SSH CONNECTION FOR USER: None` +* SSH then authenticates as the local controller user instead of the expected remote user + +**Resolution**: +* Do not define a host named [`bastion`](inventories/default/hosts) +* Keep a unique inventory hostname such as [`bastion-test`](inventories/default/hosts) +* If the same machine must belong to multiple groups, add the same unique host to both groups instead of creating a second host alias with the name [`bastion`](inventories/default/hosts) +* Use unambiguous play targets such as the actual host name or a dedicated group like [`mirrorhost`](inventories/default/hosts) + +**Example**: +```ini +[bastion] +bastion-spyre ansible_host=9.47.88.74 ansible_user=wxa + +[mirrorhost] +bastion-spyre +``` + +A validation was added to [`playbooks/0_setup.yaml:13`](playbooks/0_setup.yaml:13) so the setup playbook now fails early when a host named [`bastion`](inventories/default/hosts) is present in the inventory. + +## General Troubleshooting + +* E-mail Amadeus Podvratnik pod@de.ibm.com * If it's a problem with an OpenShift verification step: * Open the cockpit to monitor the VMs. * In a web browser, go to https://kvm-host-IP-here:9090 diff --git a/inventories/default/.gitignore b/inventories/default/.gitignore index 87b372d96..752e61e4d 100644 --- a/inventories/default/.gitignore +++ b/inventories/default/.gitignore @@ -1,5 +1,10 @@ +# Exclude the hosts file (contains environment-specific inventory) +hosts + +# Exclude everything else in this directory /* + +# But include these directories !group_vars !host_vars !.gitignore -hosts \ No newline at end of file diff --git a/inventories/default/group_vars/.gitignore b/inventories/default/group_vars/.gitignore index b208e20dc..a6846e298 100644 --- a/inventories/default/group_vars/.gitignore +++ b/inventories/default/group_vars/.gitignore @@ -3,6 +3,6 @@ !all.yaml.template !secrets.yaml.template !hcp.yaml.template -!disconnected.yaml !zvm.yaml +!disconnect.yaml.template diff --git a/inventories/default/group_vars/all.yaml.template b/inventories/default/group_vars/all.yaml.template index 8d5048db2..0951a1503 100644 --- a/inventories/default/group_vars/all.yaml.template +++ b/inventories/default/group_vars/all.yaml.template @@ -6,6 +6,7 @@ # Section 1 - Ansible Controller installation_type: kvm controller_sudo_pass: "{{ vault_ctl_host_sudo_pass }}" +disconnected_enabled: false # Set to true for disconnected/air-gapped installations env: diff --git a/inventories/default/group_vars/disconnected.yaml b/inventories/default/group_vars/disconnected.yaml deleted file mode 100644 index a94da6dfd..000000000 --- a/inventories/default/group_vars/disconnected.yaml +++ /dev/null @@ -1,86 +0,0 @@ -# Warning: currently, the oc-mirror plugin is officially downloadable to amd64 only. -# Please refer to docs set-variables-group-vars.md for instructions on how to create this configuration file for disconnected. -disconnected: - enabled: false - registry: - url: 'registry url' - ip: 'registry reachable ip' - pull_secret: '' - mirror_pull_secret: '' - ca_trusted: false - ca_cert: | - -----BEGIN CERTIFICATE----- - if ca_trusted is False, then this ca will be added to mirror host anchors - as well as to the install config of cluster - -----END CERTIFICATE----- - mirroring: - # this is the host that can access the internet as well as the registry - host: - name: hosname - ip: x.x.x.x - user: mirroruser # with become access - pass: mirrorpassword - # In disconnected mode client binaries and RHCOS will be placed on the - # file server (env.file_server) and then downloaded to the final - # destination from there. Currently only oc-mirror is handled this way. - file_server: - clients_dir: 'clients' - oc_mirror_tgz: 'oc-mirror.tar.gz' - # this will download oc binary to the mirror host for use on the mirror host for mirroring - client_download: - ocp_download_url: "https://mirror.openshift.com/pub/openshift-v4/multi/clients/ocp/4.13.1/amd64/" - ocp_client_tgz: 'openshift-client-linux.tar.gz' - legacy: - platform: false - ocp_quay_release_image_tag: '4.13.1-s390x' - ocp_org: 'ocp4' - ocp_repo: 'openshift4' - ocp_tag: 'v4.13.1' # platform images will be pushed to {registry_url}/{ocp_org}/{ocp_repo}:{ocp_tag} - oc_mirror: - oc_mirror_args: - continue_on_error: false - source_skip_tls: false - post_mirror: - mapping: - replace: - enabled: false - list: - - regexp: what - replace: with - release_image_tag: '4.13.1-multi' - release_image_registry: 'brew.registry.redhat.io' - image_set: - # this field is a standard image set from oc-mirror documentation. - # The only exception is the storageConfig which is altered to allow substitution of disconnected.registry.url - apiVersion: mirror.openshift.io/v2alpha1 # update the version with v2alpha1 for oc-mirror v2 plugin and v1alpha2 for v1 plugin - storageConfig: - enable: false # enable this flag if oc-mirror v1 plugin is used since storageConfig is removed for v2 plugin - registry: - enabled: true # use registry storage backend. Currently only method supported - imageURL: # the final value will be {imageURL: disconnected.registry.url/org/repo} - org: mirror - repo: oc-mirror-metadata - skipTLS: false # standard field form oc-mirror schema - mirror: # this field is also standard from the oc-mirror schema. It will be substituted as is into the final image set. - platform: - architectures: - - multi - # Note: cannot mirror tags like multi-s390x; list must contain pure - # architecture names (e.g. s390x) or the multiarch token 'multi'. - channels: - - name: stable-4.13 - full: false - minVersion: 4.13.1 - maxVersion: 4.13.1 - operators: - - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.13 - full: false - packages: - - name: serverless-operator - channels: - - name: stable - # minVersion: '2.4.1-0' - # maxVersion: '2.4.1-0' - additionalImages: - - name: registry.redhat.io/ubi8/ubi:latest - helm: {} diff --git a/inventories/default/group_vars/disconnected.yaml.template b/inventories/default/group_vars/disconnected.yaml.template new file mode 100644 index 000000000..c7e519420 --- /dev/null +++ b/inventories/default/group_vars/disconnected.yaml.template @@ -0,0 +1,132 @@ +# Warning: currently, the oc-mirror plugin is officially downloadable to amd64 only. +# Please refer to docs set-variables-group-vars.md for instructions on how to create this configuration file for disconnected. +# Note: The 'disconnected_enabled' parameter has been moved to all.yaml under Section 1 - Ansible Controller +disconnected: + registry: + url: 'registry url' # Will be auto-set to bastion IP:port if using bastion registry + ip: 'registry reachable ip' # Will be auto-set to bastion IP if using bastion registry + pull_secret: "{{ env.redhat.pull_secret }}" # References pull_secret from all.yaml (will be auto-updated with registry credentials) + mirror_pull_secret: "{{ env.redhat.pull_secret }}" # Same as pull_secret, used for mirroring operations + ca_trusted: false # Set to false to auto-generate self-signed certificate, true if using provided cert + ca_cert: | + # -----BEGIN CERTIFICATE----- + # Your registry certificate will be auto-generated if ca_trusted is false + # Or paste your existing certificate here if ca_trusted is true + # This certificate will be added to: + # - System trust store: /etc/pki/ca-trust/source/anchors/ + # - Container runtime: /etc/containers/certs.d// + # - Bootstrap ignition for disconnected installation + # -----END CERTIFICATE----- + # Configuration for container registry on bastion (if no external registry available) + bastion: + enabled: true # Set to true to create registry on bastion + port: 5000 # Registry port + username: 'admin' # Registry username + password: "{{ vault_registry_password }}" # Registry password from secrets.yaml + email: 'registry@example.com' # Email for registry authentication in pull secret + data_dir: '/opt/registry/data' # Registry data storage + auth_dir: '/opt/registry/auth' # Registry authentication files + certs_dir: '/opt/registry/certs' # Registry certificates + # Package installation configuration + use_local_repo: true # Set to false to install from downloaded RPMs + rpm_dir: '/tmp/registry-rpms' # Directory for downloaded RPMs (if use_local_repo=false) + rpm_path: 'rpms/registry' # Path on file server for RPMs (if use_local_repo=false) + required_rpms: # List of RPM files to download (if use_local_repo=false) + - 'podman-*.rpm' + - 'httpd-tools-*.rpm' + - 'openssl-*.rpm' + # Add all dependency RPMs as needed + mirroring: + # this is the host that can access the internet as well as the registry + host: + name: mirror-host-name # Hostname of the mirror host (must have internet access) + ip: 192.168.1.100 # IP address of the mirror host + user: root # User with become/sudo access + pass: "{{ vault_mirror_pass }}" # Password for the mirror host user + # In disconnected mode client binaries and RHCOS will be placed on the + # file server (env.file_server) and then downloaded to the final + # destination from there. Currently only oc-mirror is handled this way. + file_server: + document_root: '/var/www/html' # HTTP server document root + clients_dir: 'clients' # Subdirectory under document_root for client tools + oc_mirror_tgz: 'oc-mirror.tar.gz' + download_dir: '/tmp/oc-mirror-downloads' # Temporary directory on file server to download files before distribution + # Configuration for bastion host where oc-mirror will run + bastion: + working_dir: '/opt/oc-mirror' # Working directory for oc-mirror operations on bastion + mirror_output_dir: '/opt/oc-mirror/mirror-output' # Directory to store mirroring results and manifests + # Download URLs for oc-mirror plugin and OCP client tools + oc_mirror_download: + base_url: "https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/stable" # Base URL for oc-mirror download + oc_mirror_tgz: 'oc-mirror.tar.gz' # oc-mirror plugin archive name + # this will download oc binary to the mirror host for use on the mirror host for mirroring + client_download: + ocp_download_url: "https://mirror.openshift.com/pub/openshift-v4/multi/clients/ocp/stable-4.21/s390x" + ocp_client_tgz: 'openshift-client-linux.tar.gz' + # RHCOS downloads from dependencies directory + rhcos_download: + rhcos_download_url: "https://mirror.openshift.com/pub/openshift-v4/s390x/dependencies/rhcos/4.21/latest" + rhcos_live_rootfs: 'rhcos-live-rootfs.s390x.img' + legacy: + platform: false + ocp_quay_release_image_tag: '4.21.14-multi' + ocp_org: 'ocp4' + ocp_repo: 'openshift4' + ocp_tag: 'v4.21.14' # platform images will be pushed to {registry_url}/{ocp_org}/{ocp_repo}:{ocp_tag} + oc_mirror: + oc_mirror_args: + continue_on_error: false + source_skip_tls: false + # Timeout for mirroring operations in seconds + # Recommendations based on content size: + # - Small (single version, few operators): 7200 (2 hours) + # - Medium (multiple versions, several operators): 14400 (4 hours) + # - Large (full channel, many operators): 28800 (8 hours) + # - Very Large (multiple channels, full operators): 43200 (12 hours) + # Increase if you see "async task did not complete within the requested time" + async_timeout: 14400 + async_poll: 30 # Check status every N seconds (default: 30) + post_mirror: + mapping: + replace: + enabled: false + list: + - regexp: what + replace: with + release_image_tag: '4.21.14-multi' + release_image_registry: 'brew.registry.redhat.io' + image_set: + # this field is a standard image set from oc-mirror documentation. + # The only exception is the storageConfig which is altered to allow substitution of disconnected.registry.url + apiVersion: mirror.openshift.io/v2alpha1 # update the version with v2alpha1 for oc-mirror v2 plugin and v1alpha2 for v1 plugin + storageConfig: + enable: false # enable this flag if oc-mirror v1 plugin is used since storageConfig is removed for v2 plugin + registry: + enabled: true # use registry storage backend. Currently only method supported + imageURL: # the final value will be {imageURL: disconnected.registry.url/org/repo} + org: mirror + repo: oc-mirror-metadata + skipTLS: false # standard field form oc-mirror schema + mirror: # this field is also standard from the oc-mirror schema. It will be substituted as is into the final image set. + platform: + architectures: + - multi + # Note: cannot mirror tags like multi-s390x; list must contain pure + # architecture names (e.g. s390x) or the multiarch token 'multi'. + channels: + - name: stable-4.21 + full: false + minVersion: 4.21.14 + maxVersion: 4.21.14 + operators: + - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.21 + full: false + packages: + - name: serverless-operator + channels: + - name: stable + # minVersion: '2.4.1-0' + # maxVersion: '2.4.1-0' + additionalImages: + - name: registry.redhat.io/ubi9/ubi:latest + helm: {} \ No newline at end of file diff --git a/inventories/default/group_vars/secrets.yaml.template b/inventories/default/group_vars/secrets.yaml.template index 20b2a4d5c..7f2d82812 100644 --- a/inventories/default/group_vars/secrets.yaml.template +++ b/inventories/default/group_vars/secrets.yaml.template @@ -24,3 +24,7 @@ vault_bastion_root_pass: #X # Section 11 - (Optional) Misc vault_jumphost_pass: #X + +# Section (Optional) disconnected +vault_registry_password: #X +vault_mirror_pass: #X diff --git a/inventories/default/hosts b/inventories/default/hosts index eceebfe3d..588fabf30 100644 --- a/inventories/default/hosts +++ b/inventories/default/hosts @@ -1,2 +1,2 @@ [localhost] -127.0.0.1 ansible_connection=local ansible_become_password= +127.0.0.1 ansible_connection=local ansible_become_password='{{ controller_sudo_pass }}' diff --git a/playbooks/0_setup.yaml b/playbooks/0_setup.yaml index 1d23b312e..5c4394df1 100644 --- a/playbooks/0_setup.yaml +++ b/playbooks/0_setup.yaml @@ -9,25 +9,108 @@ vars_files: - "{{ inventory_dir }}/group_vars/all.yaml" - "{{ inventory_dir }}/group_vars/secrets.yaml" - - "{{ inventory_dir }}/group_vars/disconnected.yaml" - "{{ inventory_dir }}/group_vars/zvm.yaml" + pre_tasks: + - name: Include disconnected vars file if disconnected cluster will be installed + ansible.builtin.include_vars: + file: "{{ inventory_dir }}/group_vars/disconnected.yaml" + when: disconnected_enabled | default(false) + + - name: Fail when inventory contains a host named bastion + ansible.builtin.fail: + msg: >- + Inventory host name 'bastion' is reserved and must not be defined as a host. + A host named 'bastion' conflicts with the [bastion] group and can cause Ansible + to ignore the expected ansible_user, falling back to the controller user instead. + Rename the host entry to a unique inventory hostname such as + '{{ env.bastion.networking.hostname | default("bastion-host") }}' and, if needed, + place that host in the [bastion] and/or [mirrorhost] groups. + when: + - "'bastion' in hostvars" + - "'bastion' not in groups.get('bastion', [])" + + - name: Report incomplete static IPv6 configuration for cluster nodes + block: + - name: Fail message when static IPv6 configuration is incomplete for cluster nodes + ansible.builtin.fail: + msg: + - "Static IPv6 is enabled because use_ipv6=true and use_dhcp=false." + - "Ensure Section 8 control nodes define env.cluster.nodes.control.ipv6." + - "Ensure Section 9 compute nodes define env.cluster.nodes.compute.ipv6 for all defined nodes." + - "Ensure Section 10 infra nodes define env.cluster.nodes.infra.ipv6 for all defined nodes when infra nodes are configured." + rescue: + - name: Continue after reporting incomplete static IPv6 configuration + ansible.builtin.debug: + msg: "Continuing execution after IPv6 configuration warning." + vars: + control_hostnames: "{{ env.cluster.nodes.control.hostname | default([]) }}" + control_ipv6: "{{ env.cluster.nodes.control.ipv6 | default([]) }}" + compute_hostnames: "{{ env.cluster.nodes.compute.hostname | default([]) }}" + compute_ipv6: "{{ env.cluster.nodes.compute.ipv6 | default([]) }}" + infra_hostnames: "{{ env.cluster.nodes.infra.hostname | default([]) }}" + infra_ipv6: "{{ env.cluster.nodes.infra.ipv6 | default([]) }}" + infra_defined: >- + {{ + env.cluster.nodes.infra is defined and + env.cluster.nodes.infra.hostname is defined and + env.cluster.nodes.infra.hostname is not none and + ( + ( + env.cluster.nodes.infra.hostname is string and + env.cluster.nodes.infra.hostname | trim != '' + ) or + ( + env.cluster.nodes.infra.hostname is sequence and + env.cluster.nodes.infra.hostname is not string and + (env.cluster.nodes.infra.hostname | length > 0) + ) + ) + }} + when: + - env.use_ipv6 | default(use_ipv6 | default(false)) | bool + - not (env.use_dhcp | default(use_dhcp | default(false)) | bool) + - >- + (control_ipv6 | length) < (control_hostnames | length) or + (compute_ipv6 | length) < (compute_hostnames | length) or + (infra_defined and ((infra_ipv6 | length) < (infra_hostnames | length))) + roles: - set_inventory - install_galaxy - pre_tasks: + tasks: - name: Check disconnected variables if disconnected cluster will be installed. tags: disconnected ansible.builtin.include_role: name: disconnected_check_vars - when: disconnected.enabled + when: disconnected_enabled post_tasks: - name: Find ibm_zhmc collection install location, if automated LPAR creation is to be used. tags: galaxy - ansible.builtin.shell: ansible-galaxy collection list ibm.ibm_zhmc | grep -i ansible | cut -c 3- - register: zhmc_path - when: env.z.lpar1.create == True or env.z.lpar2.create == True or env.z.lpar3.create == True + ansible.builtin.command: + argv: + - ansible-galaxy + - collection + - list + - ibm.ibm_zhmc + register: zhmc_collection_list + when: env.z.lpar1.create or env.z.lpar2.create or env.z.lpar3.create + + - name: Extract ibm_zhmc collection install location + tags: galaxy + ansible.builtin.set_fact: + zhmc_path: + stdout: >- + {{ + ( + zhmc_collection_list.stdout_lines + | select('search', 'ansible') + | first + | regex_replace('^..', '') + ) + }} + when: env.z.lpar1.create or env.z.lpar2.create or env.z.lpar3.create - name: Ensure zhmcclient requirements are installed. tags: galaxy @@ -35,13 +118,16 @@ requirements: "{{ zhmc_path.stdout }}/ibm/ibm_zhmc/requirements.txt" executable: pip3 extra_args: --upgrade - when: env.z.lpar1.create == True or env.z.lpar2.create == True or env.z.lpar3.create == True + when: env.z.lpar1.create or env.z.lpar2.create or env.z.lpar3.create - name: Check to make sure that the KVM host has a corresponding inventory host_vars file named with matching hostname and .yaml extension. tags: lpar_check ansible.builtin.stat: path: "{{ inventory_dir }}/host_vars/{{ env.z.lpar1.hostname }}.yaml" - when: env.z.lpar1.hostname is defined + when: + - env.z.lpar1.hostname is defined + - env.z.lpar1.hostname is not none + - env.z.lpar1.hostname | string | trim != '' register: lpar_host_vars failed_when: lpar_host_vars.stat.exists == False @@ -49,17 +135,29 @@ tags: lpar_check ansible.builtin.stat: path: "{{ inventory_dir }}/host_vars/{{ env.z.lpar2.hostname }}.yaml" - when: env.z.lpar2.hostname is defined + when: + - env.z.lpar2.hostname is defined + - env.z.lpar2.hostname is not none + - env.z.lpar2.hostname | string | trim != '' register: lpar_host_vars - failed_when: lpar_host_vars.stat.exists == False + failed_when: + - lpar_host_vars is defined + - lpar_host_vars.stat is defined + - lpar_host_vars.stat.exists == False - name: Check to make sure the third KVM hosts have a corresponding inventory host_vars file named with matching hostname and .yaml extension, if defined. tags: lpar_check ansible.builtin.stat: path: "{{ inventory_dir }}/host_vars/{{ env.z.lpar3.hostname }}.yaml" - when: env.z.lpar3.hostname is defined + when: + - env.z.lpar3.hostname is defined + - env.z.lpar3.hostname is not none + - env.z.lpar3.hostname | string | trim != '' register: lpar_host_vars - failed_when: lpar_host_vars.stat.exists == False + failed_when: + - lpar_host_vars is defined + - lpar_host_vars.stat is defined + - lpar_host_vars.stat.exists == False - name: "Install packages and cfg ssh" hosts: localhost diff --git a/playbooks/3_setup_kvm_host.yaml b/playbooks/3_setup_kvm_host.yaml index 45a2eb69d..63cb7ca2e 100644 --- a/playbooks/3_setup_kvm_host.yaml +++ b/playbooks/3_setup_kvm_host.yaml @@ -35,13 +35,19 @@ - name: Include vars for second KVM host. ansible.builtin.include_vars: file: "{{ inventory_dir }}/host_vars/{{ env.z.lpar2.hostname }}.yaml" - when: env.z.lpar2.hostname is defined + when: + - env.z.lpar2.hostname is defined + - env.z.lpar2.hostname is not none + - env.z.lpar2.hostname | string | trim != '' - name: copy SSH key to second KVM host, if cluster is to be highly available. tags: ssh_copy_id, ssh ansible.builtin.import_role: name: ssh_copy_id - when: env.z.lpar2.hostname is defined + when: + - env.z.lpar2.hostname is defined + - env.z.lpar2.hostname is not none + - env.z.lpar2.hostname | string | trim != '' - name: Copy SSH key to access KVM host 3 hosts: localhost @@ -58,13 +64,19 @@ - name: Include vars for third KVM host. ansible.builtin.include_vars: file: "{{ inventory_dir }}/host_vars/{{ env.z.lpar3.hostname }}.yaml" - when: env.z.lpar3.hostname is defined + when: + - env.z.lpar3.hostname is defined + - env.z.lpar3.hostname is not none + - env.z.lpar3.hostname | string | trim != '' - name: copy SSH key to third KVM host, if cluster is to be highly available. tags: ssh_copy_id, ssh ansible.builtin.import_role: name: ssh_copy_id - when: env.z.lpar3.hostname is defined + when: + - env.z.lpar3.hostname is defined + - env.z.lpar3.hostname is not none + - env.z.lpar3.hostname | string | trim != '' - name: Prepare KVM host(s) hosts: kvm_host @@ -112,7 +124,7 @@ line: 'group = "libvirt"' backup: true - - name: Get user home directory + - name: Get user home directory tags: libvirt ansible.builtin.shell: > getent passwd {{ ansible_user }} | awk -F: '{ print $6 }' @@ -122,12 +134,12 @@ - name: Check if directory {{ user_home.stdout }}/.config/libvirt exists tags: libvirt ansible.builtin.stat: - path: "{{ user_home.stdout }}/.config/libvirt" + path: "{{ user_home.stdout }}/.config/libvirt" register: home_config_libvirt - name: Create directory {{ user_home.stdout }}/.config/libvirt tags: libvirt - ansible.builtin.file: + ansible.builtin.file: path: "{{ user_home.stdout }}/.config/libvirt" state: directory when: home_config_libvirt.stat.exists == false diff --git a/playbooks/4_create_bastion.yaml b/playbooks/4_create_bastion.yaml index 149434927..c28eb362c 100644 --- a/playbooks/4_create_bastion.yaml +++ b/playbooks/4_create_bastion.yaml @@ -9,3 +9,15 @@ roles: - common - { role: create_bastion, when: env.bastion.create == True } + +- name: 4 create bastion - copy SSH key from localhost to access bastion + hosts: localhost + tags: ssh, ssh_copy_id + gather_facts: true + vars_files: + - "{{ inventory_dir }}/group_vars/all.yaml" + - "{{ inventory_dir }}/group_vars/secrets.yaml" + vars: + ssh_target: ["{{ env.bastion.networking.ip }}", "{{ env.bastion.access.user }}", "{{ env.bastion.access.pass }}", "{{ path_to_key_pair }}"] + roles: + - { role: ssh_copy_id, when: env.bastion.create == True } diff --git a/playbooks/5_setup_bastion.yaml b/playbooks/5_setup_bastion.yaml index 84990e8a8..9a73b1954 100644 --- a/playbooks/5_setup_bastion.yaml +++ b/playbooks/5_setup_bastion.yaml @@ -75,6 +75,7 @@ vars_files: - "{{ inventory_dir }}/group_vars/all.yaml" - "{{ inventory_dir }}/group_vars/secrets.yaml" + - "{{ inventory_dir }}/group_vars/disconnected.yaml" tasks: - name: Enabliling the HiperSocket card on bastion if installation_type is lpar else will be skipped. block: diff --git a/playbooks/6_create_nodes.yaml b/playbooks/6_create_nodes.yaml index cd99393e4..192d644ec 100644 --- a/playbooks/6_create_nodes.yaml +++ b/playbooks/6_create_nodes.yaml @@ -32,6 +32,7 @@ vars_files: - "{{ inventory_dir }}/group_vars/all.yaml" - "{{ inventory_dir }}/group_vars/secrets.yaml" + - "{{ inventory_dir }}/group_vars/disconnected.yaml" roles: - { role: prep_kvm_guests, when: env.cluster.nodes.bootstrap.vm_name not in hosts_with_host_vars } # Delete control, compute and infra nodes, if exists @@ -43,6 +44,7 @@ vars_files: - "{{ inventory_dir }}/group_vars/all.yaml" - "{{ inventory_dir }}/group_vars/secrets.yaml" + - "{{ inventory_dir }}/group_vars/disconnected.yaml" roles: - { role: common, when: env.cluster.nodes.bootstrap.vm_name not in hosts_with_host_vars } - { role: create_bootstrap, when: (hosts_with_host_vars|length == 0) or env.cluster.nodes.bootstrap.vm_name not in hosts_with_host_vars } @@ -53,6 +55,7 @@ vars_files: - "{{ inventory_dir }}/group_vars/all.yaml" - "{{ inventory_dir }}/group_vars/secrets.yaml" + - "{{ inventory_dir }}/group_vars/disconnected.yaml" tasks: - name: boot bootstrap vars: @@ -82,6 +85,7 @@ vars_files: - "{{ inventory_dir }}/group_vars/all.yaml" - "{{ inventory_dir }}/group_vars/secrets.yaml" + - "{{ inventory_dir }}/group_vars/disconnected.yaml" roles: - common - create_control_nodes @@ -95,6 +99,7 @@ vars_files: - "{{ inventory_dir }}/group_vars/all.yaml" - "{{ inventory_dir }}/group_vars/secrets.yaml" + - "{{ inventory_dir }}/group_vars/disconnected.yaml" roles: - wait_for_bootstrap @@ -105,6 +110,7 @@ vars_files: - "{{ inventory_dir }}/group_vars/all.yaml" - "{{ inventory_dir }}/group_vars/secrets.yaml" + - "{{ inventory_dir }}/group_vars/disconnected.yaml" tasks: - name: Destroy bootstrap. Expect ignored errors if bootstrap is already destroyed. tags: create_nodes, teardown_bootstrap @@ -128,6 +134,7 @@ vars_files: - "{{ inventory_dir }}/group_vars/all.yaml" - "{{ inventory_dir }}/group_vars/secrets.yaml" + - "{{ inventory_dir }}/group_vars/disconnected.yaml" roles: - { role: common, when: env.cluster.nodes.bootstrap.vm_name not in hosts_with_host_vars } - { role: create_compute_nodes, when: env.cluster.nodes.bootstrap.vm_name not in hosts_with_host_vars } diff --git a/playbooks/7_ocp_verification.yaml b/playbooks/7_ocp_verification.yaml index bd5b73240..9d64509d6 100644 --- a/playbooks/7_ocp_verification.yaml +++ b/playbooks/7_ocp_verification.yaml @@ -10,6 +10,7 @@ vars_files: - "{{ inventory_dir }}/group_vars/all.yaml" - "{{ inventory_dir }}/group_vars/secrets.yaml" + - "{{ inventory_dir }}/group_vars/disconnected.yaml" roles: - approve_certs - check_nodes @@ -18,4 +19,3 @@ - wait_for_cluster_operators - wait_for_install_complete - label_infra_nodes - diff --git a/playbooks/create_abi_cluster.yaml b/playbooks/create_abi_cluster.yaml index 512b24ea5..60d2b14d0 100644 --- a/playbooks/create_abi_cluster.yaml +++ b/playbooks/create_abi_cluster.yaml @@ -11,8 +11,8 @@ - "{{ inventory_dir }}/group_vars/disconnected.yaml" roles: - common # Common Variable the will be used by all the inwalked roles. - - { role: offline_artifacts, when: disconnected.enabled } # Update CA Certificates & Download OCP Packages - - { role: download_ocp_installer, when: not disconnected.enabled } # Download Openshift Installer. + - { role: offline_artifacts, when: disconnected_enabled } # Update CA Certificates & Download OCP Packages + - { role: download_ocp_installer, when: not disconnected_enabled } # Download Openshift Installer. - prepare_configs # Prepare AgentConfig & InstallConfig. - create_agent # Create Agents || Build initrd.img, rootfs.img & kernelfs.img. diff --git a/playbooks/create_compute_node.yaml b/playbooks/create_compute_node.yaml index f93270bcd..081dfe02e 100644 --- a/playbooks/create_compute_node.yaml +++ b/playbooks/create_compute_node.yaml @@ -24,6 +24,7 @@ vars_files: - "{{ inventory_dir }}/group_vars/all.yaml" - "{{ inventory_dir }}/group_vars/secrets.yaml" + - "{{ inventory_dir }}/group_vars/disconnected.yaml" pre_tasks: - name: Check required variables when: (day2_compute_node is not defined) diff --git a/playbooks/disconnected_apply_operator_manifests.yaml b/playbooks/disconnected_apply_operator_manifests.yaml index 5ce41f8a4..f68198988 100644 --- a/playbooks/disconnected_apply_operator_manifests.yaml +++ b/playbooks/disconnected_apply_operator_manifests.yaml @@ -16,4 +16,4 @@ loop: - disconnected_check_vars - disconnected_apply_operator_manifests_to_cluster - when: disconnected.enabled + when: disconnected_enabled diff --git a/playbooks/disconnected_mirror_artifacts.yaml b/playbooks/disconnected_mirror_artifacts.yaml index 1ceef1aeb..b7365c24a 100644 --- a/playbooks/disconnected_mirror_artifacts.yaml +++ b/playbooks/disconnected_mirror_artifacts.yaml @@ -13,4 +13,4 @@ loop: - disconnected_check_vars - disconnected_mirror_images - when: disconnected.enabled + when: disconnected_enabled diff --git a/playbooks/disconnected_setup_oc_mirror.yaml b/playbooks/disconnected_setup_oc_mirror.yaml new file mode 100644 index 000000000..274bd647a --- /dev/null +++ b/playbooks/disconnected_setup_oc_mirror.yaml @@ -0,0 +1,157 @@ +--- +# Playbook: disconnected_setup_oc_mirror.yaml +# Description: Downloads oc-mirror plugin, sets it up on bastion, and performs OCP mirroring +# This playbook orchestrates the complete oc-mirror setup and mirroring process for disconnected installations +# +# Usage: +# ansible-playbook -i inventories/default playbooks/disconnected_setup_oc_mirror.yaml +# +# Prerequisites: +# - File server must be accessible (can be same as bastion) +# - Bastion must have internet access or access to file server +# - disconnected.yaml must be properly configured +# - Pull secret must be configured in disconnected.registry.mirror_pull_secret + +- name: Download registry RPMs to file server (if needed) + hosts: "{{ 'bastion' if (env.bastion.networking.ip | default('') == env.file_server.ip | default('')) else 'file_server' }}" + gather_facts: true + vars_files: + - "{{ inventory_dir }}/group_vars/all.yaml" + - "{{ inventory_dir }}/group_vars/secrets.yaml" + - "{{ inventory_dir }}/group_vars/disconnected.yaml" + tasks: + - name: Download registry RPM packages + ansible.builtin.include_role: + name: disconnected_download_registry_rpms + when: + - disconnected_enabled + - disconnected.registry.bastion.enabled | default(false) + - not (disconnected.registry.bastion.use_local_repo | default(true)) + tags: + - download + - download_rpms + +- name: Setup container registry on bastion + hosts: bastion + gather_facts: true + vars_files: + - "{{ inventory_dir }}/group_vars/all.yaml" + - "{{ inventory_dir }}/group_vars/secrets.yaml" + - "{{ inventory_dir }}/group_vars/disconnected.yaml" + pre_tasks: + - name: Gather service facts on bastion + ansible.builtin.service_facts: + tasks: + - name: Setup container registry on bastion + ansible.builtin.include_role: + name: disconnected_setup_registry_bastion + when: + - disconnected_enabled + - disconnected.registry.bastion.enabled | default(false) + tags: + - registry + - setup_registry + +- name: Update pull secret with bastion registry credentials + hosts: localhost + gather_facts: false + vars_files: + - "{{ inventory_dir }}/group_vars/all.yaml" + - "{{ inventory_dir }}/group_vars/secrets.yaml" + - "{{ inventory_dir }}/group_vars/disconnected.yaml" + tasks: + - name: Update pull secret with registry credentials + ansible.builtin.include_role: + name: disconnected_update_pull_secret + when: + - disconnected_enabled + - disconnected.registry.bastion.enabled | default(false) + tags: + - registry + - update_pull_secret + +- name: Download oc-mirror to file server + hosts: file_server + gather_facts: true + vars_files: + - "{{ inventory_dir }}/group_vars/all.yaml" + - "{{ inventory_dir }}/group_vars/secrets.yaml" + - "{{ inventory_dir }}/group_vars/disconnected.yaml" + tasks: + - name: Download oc-mirror and client tools to file server + ansible.builtin.include_role: + name: disconnected_download_oc_mirror + when: disconnected_enabled + tags: + - download + - oc_mirror_download + +- name: Setup oc-mirror on bastion + hosts: bastion + gather_facts: true + vars_files: + - "{{ inventory_dir }}/group_vars/all.yaml" + - "{{ inventory_dir }}/group_vars/secrets.yaml" + - "{{ inventory_dir }}/group_vars/disconnected.yaml" + tasks: + - name: Reload disconnected variables to get updated certificate + ansible.builtin.include_vars: + file: "{{ inventory_dir }}/group_vars/disconnected.yaml" + tags: + - setup + - oc_mirror_setup + + - name: Install and configure oc-mirror on bastion + ansible.builtin.include_role: + name: disconnected_setup_oc_mirror_bastion + when: disconnected_enabled + tags: + - setup + - oc_mirror_setup + +- name: Mirror OCP images using oc-mirror + hosts: bastion + gather_facts: true + vars_files: + - "{{ inventory_dir }}/group_vars/all.yaml" + - "{{ inventory_dir }}/group_vars/secrets.yaml" + - "{{ inventory_dir }}/group_vars/disconnected.yaml" + tasks: + - name: Perform OCP mirroring operations + ansible.builtin.include_role: + name: disconnected_mirror_ocp_bastion + when: disconnected_enabled + tags: + - mirror + - oc_mirror_mirror + + - name: Display next steps + ansible.builtin.debug: + msg: | + ========================================== + OCP Mirroring completed successfully! + ========================================== + + Registry Information: + {% if disconnected.registry.bastion.enabled | default(false) %} + - Registry URL: {{ env.bastion.networking.ip }}:{{ disconnected.registry.bastion.port }} + - Registry Hostname: {{ env.bastion.networking.hostname }}:{{ disconnected.registry.bastion.port }} + - Username: {{ disconnected.registry.bastion.username }} + {% else %} + - Using external registry: {{ disconnected.registry.url }} + {% endif %} + + Next steps: + 1. Review the mirrored content in: {{ disconnected.mirroring.bastion.mirror_output_dir }} + 2. Apply the generated YAML files to your cluster: + - ImageContentSourcePolicy (ICSP) or ImageDigestMirrorSet (IDMS) + - CatalogSource for operators + 3. Update your install-config.yaml with the imageContentSources + 4. Proceed with cluster installation + + For more information, see: + https://docs.redhat.com/en/documentation/openshift_container_platform/4.14/html/disconnected_installation_mirroring/ + tags: + - always + +# Assisted by Bob diff --git a/playbooks/master_playbook_for_abi.yaml b/playbooks/master_playbook_for_abi.yaml index ec033b077..33963f2e1 100644 --- a/playbooks/master_playbook_for_abi.yaml +++ b/playbooks/master_playbook_for_abi.yaml @@ -7,6 +7,6 @@ - import_playbook: 4_create_bastion.yaml # Import Playbook To Create Bastion. - import_playbook: 5_setup_bastion.yaml # Import Playbook To Configure Bastion. - import_playbook: disconnected_mirror_artifacts.yaml - when: disconnected.enabled + when: disconnected_enabled - import_playbook: create_abi_cluster.yaml # Import Playbook To Create ABI Cluster. - import_playbook: monitor_create_abi_cluster.yaml # Import Playbook To Monitor ABI Cluster Installation. diff --git a/playbooks/pre-existing_site.yaml b/playbooks/pre-existing_site.yaml index 5347411bf..dcf163554 100644 --- a/playbooks/pre-existing_site.yaml +++ b/playbooks/pre-existing_site.yaml @@ -4,9 +4,9 @@ - import_playbook: 0_setup.yaml - import_playbook: 4_create_bastion.yaml - import_playbook: disconnected_mirror_artifacts.yaml - when: disconnected.enabled + when: disconnected_enabled - import_playbook: 5_setup_bastion.yaml - import_playbook: 6_create_nodes.yaml - import_playbook: 7_ocp_verification.yaml - import_playbook: disconnected_apply_operator_manifests.yaml - when: disconnected.enabled + when: disconnected_enabled diff --git a/playbooks/reinstall_cluster.yaml b/playbooks/reinstall_cluster.yaml index 8858d3e84..c906de137 100644 --- a/playbooks/reinstall_cluster.yaml +++ b/playbooks/reinstall_cluster.yaml @@ -80,7 +80,7 @@ when: env.z.lpar3.hostname is defined - import_playbook: disconnected_mirror_artifacts.yaml - when: disconnected.enabled + when: disconnected_enabled - name: Re-Install cluster - Update ignitions and other install files hosts: bastion @@ -96,4 +96,4 @@ - import_playbook: 6_create_nodes.yaml - import_playbook: 7_ocp_verification.yaml - import_playbook: disconnected_apply_operator_manifests.yaml - when: disconnected.enabled + when: disconnected_enabled diff --git a/playbooks/site.yaml b/playbooks/site.yaml index 177f3ae87..60815fbf8 100644 --- a/playbooks/site.yaml +++ b/playbooks/site.yaml @@ -7,9 +7,9 @@ - import_playbook: 3_setup_kvm_host.yaml - import_playbook: 4_create_bastion.yaml - import_playbook: disconnected_mirror_artifacts.yaml - when: disconnected.enabled + when: disconnected_enabled - import_playbook: 5_setup_bastion.yaml - import_playbook: 6_create_nodes.yaml - import_playbook: 7_ocp_verification.yaml - import_playbook: disconnected_apply_operator_manifests.yaml - when: disconnected.enabled + when: disconnected_enabled diff --git a/roles/boot_LPAR/tasks/main.yaml b/roles/boot_LPAR/tasks/main.yaml index d11632de8..0f9165fe2 100644 --- a/roles/boot_LPAR/tasks/main.yaml +++ b/roles/boot_LPAR/tasks/main.yaml @@ -56,8 +56,8 @@ --hmcpass {{ node.hmc.auth.pass }} \ --cpu {{ node.lpar.ifl.count }} \ --memory {{ node.lpar.ifl.initial_memory }} \ - --kernel {{ rhcos_download_url }}{{ rhcos_live_kernel }} \ - --initrd {{ rhcos_download_url }}{{ rhcos_live_initrd }} \ + --kernel {% if disconnected is defined and disconnected_enabled %}{{ env.file_server.protocol }}://{{ env.file_server.ip }}:{{ env.file_server.port }}/{{ disconnected.mirroring.file_server.clients_dir }}/{% else %}{{ rhcos_download_url }}{% endif %}{{ rhcos_live_kernel }} \ + --initrd {% if disconnected is defined and disconnected_enabled %}{{ env.file_server.protocol }}://{{ env.file_server.ip }}:{{ env.file_server.port }}/{{ disconnected.mirroring.file_server.clients_dir }}/{% else %}{{ rhcos_download_url }}{% endif %}{{ rhcos_live_initrd }} \ --livedisktype {{ node.lpar.livedisk.livedisktype }} \ {% if node.lpar.dpm_enabled == 'True' %} --dpm_enabled {{ node.lpar.dpm_enabled }} \ @@ -86,8 +86,8 @@ --hmcpass {{ node.hmc.auth.pass }} \ --cpu {{ node.lpar.ifl.count }} \ --memory {{ node.lpar.ifl.initial_memory }} \ - --kernel {{ rhcos_download_url }}{{ rhcos_live_kernel }} \ - --initrd {{ rhcos_download_url }}{{ rhcos_live_initrd }} \ + --kernel {% if disconnected is defined and disconnected_enabled %}{{ env.file_server.protocol }}://{{ env.file_server.ip }}:{{ env.file_server.port }}/{{ disconnected.mirroring.file_server.clients_dir }}/{% else %}{{ rhcos_download_url }}{% endif %}{{ rhcos_live_kernel }} \ + --initrd {% if disconnected is defined and disconnected_enabled %}{{ env.file_server.protocol }}://{{ env.file_server.ip }}:{{ env.file_server.port }}/{{ disconnected.mirroring.file_server.clients_dir }}/{% else %}{{ rhcos_download_url }}{% endif %}{{ rhcos_live_initrd }} \ --livedisktype {{ node.lpar.livedisk.livedisktype }} \ {% if node.lpar.dpm_enabled == 'True' %} --dpm_enabled {{ node.lpar.dpm_enabled }} \ diff --git a/roles/create_bastion/templates/bastion-ks.cfg.j2 b/roles/create_bastion/templates/bastion-ks.cfg.j2 index 60c25b796..e243f4d7b 100644 --- a/roles/create_bastion/templates/bastion-ks.cfg.j2 +++ b/roles/create_bastion/templates/bastion-ks.cfg.j2 @@ -33,7 +33,16 @@ timezone {{ env.timezone }} eula --agreed # Network information -network --bootproto=static --device={{ env.bastion.networking.interface }} --ip={{ env.bastion.networking.ip }} --gateway={{ env.bastion.networking.gateway }} --netmask={{ env.bastion.networking.subnetmask }} {{'--ipv6=' + env.bastion.networking.ipv6 if env.use_ipv6 else '--noipv6' }} {{'--ipv6gateway=' + env.bastion.networking.ipv6_gateway if env.use_ipv6 else ''}} --nameserver={{ env.bastion.networking.nameserver1 }}{{ (',' + env.bastion.networking.nameserver2) if env.bastion.networking.nameserver2 is defined else '' }} --activate +# For disconnected installations with DNS on bastion: use nameserver2 (external) if available, otherwise nameserver1 +# For installations without DNS on bastion: use nameserver1 +{% set bastion_nameserver2 = env.bastion.networking.nameserver2 | default('') | string | trim %} +{% set bastion_ipv6 = env.bastion.networking.ipv6 | default('') | string | trim %} +{% set bastion_ipv6_gateway = env.bastion.networking.ipv6_gateway | default('') | string | trim %} +{% if env.bastion.options.dns is defined and env.bastion.options.dns and bastion_nameserver2 | length > 0 %} +network --bootproto=static --device={{ env.bastion.networking.interface }} --ip={{ env.bastion.networking.ip }} --gateway={{ env.bastion.networking.gateway }} --netmask={{ env.bastion.networking.subnetmask }} {{ '--ipv6=' + bastion_ipv6 if env.use_ipv6 and bastion_ipv6 | length > 0 else '--noipv6' }} {{ '--ipv6gateway=' + bastion_ipv6_gateway if env.use_ipv6 and bastion_ipv6_gateway | length > 0 else '' }} --nameserver={{ bastion_nameserver2 }} --activate +{% else %} +network --bootproto=static --device={{ env.bastion.networking.interface }} --ip={{ env.bastion.networking.ip }} --gateway={{ env.bastion.networking.gateway }} --netmask={{ env.bastion.networking.subnetmask }} {{ '--ipv6=' + bastion_ipv6 if env.use_ipv6 and bastion_ipv6 | length > 0 else '--noipv6' }} {{ '--ipv6gateway=' + bastion_ipv6_gateway if env.use_ipv6 and bastion_ipv6_gateway | length > 0 else '' }} --nameserver={{ env.bastion.networking.nameserver1 }}{{ (',' + bastion_nameserver2) if bastion_nameserver2 | length > 0 else '' }} --activate +{% endif %} network --hostname={{ env.bastion.networking.hostname }}.{{ env.cluster.networking.base_domain }} # Firewall and SELinux diff --git a/roles/create_bastion/templates/rhel9-bastion-ks.cfg.j2 b/roles/create_bastion/templates/rhel9-bastion-ks.cfg.j2 index 4909d1216..fa872c617 100644 --- a/roles/create_bastion/templates/rhel9-bastion-ks.cfg.j2 +++ b/roles/create_bastion/templates/rhel9-bastion-ks.cfg.j2 @@ -32,7 +32,16 @@ timezone {{ env.timezone }} eula --agreed # Network information -network --bootproto=static --device={{ env.bastion.networking.interface }} --ip={{ env.bastion.networking.ip }} --gateway={{ env.bastion.networking.gateway }} --netmask={{ env.bastion.networking.subnetmask }} {{'--ipv6=' + env.bastion.networking.ipv6 if env.use_ipv6 else '--noipv6' }} {{'--ipv6gateway=' + env.bastion.networking.ipv6_gateway if env.use_ipv6 }} --nameserver={{ env.bastion.networking.nameserver1 }}{{ (',' + env.bastion.networking.nameserver2) if env.bastion.networking.nameserver2 is defined }} --activate +# For disconnected installations with DNS on bastion: use nameserver2 (external) if available, otherwise nameserver1 +# For installations without DNS on bastion: use nameserver1 +{% set bastion_nameserver2 = env.bastion.networking.nameserver2 | default('') | string | trim %} +{% set bastion_ipv6 = env.bastion.networking.ipv6 | default('') | string | trim %} +{% set bastion_ipv6_gateway = env.bastion.networking.ipv6_gateway | default('') | string | trim %} +{% if env.bastion.options.dns is defined and env.bastion.options.dns and bastion_nameserver2 | length > 0 %} +network --bootproto=static --device={{ env.bastion.networking.interface }} --ip={{ env.bastion.networking.ip }} --gateway={{ env.bastion.networking.gateway }} --netmask={{ env.bastion.networking.subnetmask }} {{ '--ipv6=' + bastion_ipv6 if env.use_ipv6 and bastion_ipv6 | length > 0 else '--noipv6' }} {{ '--ipv6gateway=' + bastion_ipv6_gateway if env.use_ipv6 and bastion_ipv6_gateway | length > 0 else '' }} --nameserver={{ bastion_nameserver2 }} --activate +{% else %} +network --bootproto=static --device={{ env.bastion.networking.interface }} --ip={{ env.bastion.networking.ip }} --gateway={{ env.bastion.networking.gateway }} --netmask={{ env.bastion.networking.subnetmask }} {{ '--ipv6=' + bastion_ipv6 if env.use_ipv6 and bastion_ipv6 | length > 0 else '--noipv6' }} {{ '--ipv6gateway=' + bastion_ipv6_gateway if env.use_ipv6 and bastion_ipv6_gateway | length > 0 else '' }} --nameserver={{ env.bastion.networking.nameserver1 }}{{ (',' + bastion_nameserver2) if bastion_nameserver2 | length > 0 else '' }} --activate +{% endif %} network --hostname={{ env.bastion.networking.hostname }}.{{ env.cluster.networking.base_domain }} # Firewall and SELinux diff --git a/roles/create_bootstrap/tasks/main.yaml b/roles/create_bootstrap/tasks/main.yaml index 12a9fb6bd..30418eb52 100644 --- a/roles/create_bootstrap/tasks/main.yaml +++ b/roles/create_bootstrap/tasks/main.yaml @@ -15,9 +15,9 @@ {{ env.cluster.nodes.bootstrap.vcpu_model_option }} \ --vcpus {{ env.cluster.nodes.bootstrap.vcpu }} \ --network network={{ env.vnet_name }}{{ (',mac=' + env.cluster.nodes.bootstrap.mac) if (env.cluster.nodes.bootstrap.mac is defined and env.use_dhcp) }} \ - --location {{ rhcos_download_url }},kernel={{ rhcos_live_kernel }},initrd={{ rhcos_live_initrd }} \ + --location {% if disconnected_enabled is defined and disconnected_enabled %}{{ env.file_server.protocol }}://{{ env.file_server.ip }}{{ (':' + env.file_server.port | string) if (env.file_server.port is defined and env.file_server.port | string | length > 0) else '' }}/{{ disconnected.mirroring.file_server.clients_dir }}/{% else %}{{ rhcos_download_url }}{% endif %},kernel={{ rhcos_live_kernel }},initrd={{ rhcos_live_initrd }} \ --extra-args "rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda" \ - --extra-args "coreos.live.rootfs_url=http://{{ env.bastion.networking.ip }}:8080/bin/{{ rhcos_live_rootfs }}" \ + --extra-args "coreos.live.rootfs_url={% if disconnected_enabled is defined and disconnected_enabled %}{{ env.file_server.protocol }}://{{ env.file_server.ip }}{{ (':' + env.file_server.port | string) if (env.file_server.port is defined and env.file_server.port | string | length > 0) else '' }}/bin/{{ rhcos_live_rootfs }}{% else %}http://{{ env.bastion.networking.ip }}:8080/bin/{{ rhcos_live_rootfs }}{% endif %}" \ {% if (env.cluster.nodes.bootstrap.mac is defined and env.use_dhcp) %} --extra-args "ip=dhcp" \ {% else %} @@ -25,7 +25,7 @@ --extra-args "{{ ('ip=[' + env.cluster.nodes.bootstrap.ipv6 + ']::[' + env.cluster.networking.ipv6_gateway + ']:' + env.cluster.networking.ipv6_prefix | string + '::' + env.cluster.networking.interface + ':none' ) if env.use_ipv6 == True else '' }}" \ {% endif %} --extra-args "nameserver={{ env.cluster.networking.nameserver1 }}" \ - --extra-args "{{ ('nameserver=' + env.cluster.networking.nameserver2) if env.cluster.networking.nameserver2 is defined else '' }}" \ + --extra-args "{{ ('nameserver=' + env.cluster.networking.nameserver2) if (env.cluster.networking.nameserver2 is defined and env.cluster.networking.nameserver2 | length > 0) else '' }}" \ --extra-args "coreos.inst.ignition_url=http://{{ env.bastion.networking.ip }}:8080/ignition/bootstrap.ign" \ --extra-args "{{ _vm_console }}" \ --memballoon none \ diff --git a/roles/create_compute_node/tasks/main.yaml b/roles/create_compute_node/tasks/main.yaml index 995bf56d3..b8afbe9cf 100644 --- a/roles/create_compute_node/tasks/main.yaml +++ b/roles/create_compute_node/tasks/main.yaml @@ -72,7 +72,7 @@ --console pty,target_type=serial \ --wait -1 \ --noautoconsole \ - --location {{ rhcos_download_url }},kernel={{ rhcos_live_kernel }},initrd={{ rhcos_live_initrd }} \ + --location {% if disconnected_enabled is defined and disconnected_enabled %}{{ env.file_server.protocol }}://{{ env.file_server.ip }}{{ (':' + env.file_server.port | string) if (env.file_server.port is defined and env.file_server.port | string | length > 0) else '' }}/{{ disconnected.mirroring.file_server.clients_dir }}/{% else %}{{ rhcos_download_url }}{% endif %},kernel={{ rhcos_live_kernel }},initrd={{ rhcos_live_initrd }} \ --extra-args "rd.neednet=1 coreos.inst=yes coreos.inst.install_dev={{ param_compute_node.install_dev | default('vda') }}" \ {% if (param_compute_node.vm_mac is defined and env.use_dhcp) %} --extra-args "ip=dhcp" \ @@ -80,8 +80,8 @@ --extra-args "ip={{ param_compute_node.vm_ip }}::{{ env.bastion.networking.gateway }}:{{ env.bastion.networking.subnetmask }}:{{ param_compute_node.vm_hostname }}.{{ env.cluster.networking.metadata_name }}.{{ env.cluster.networking.base_domain }}:{{ param_compute_node.vm_interface }}:none:1500" \ --extra-args "{{ ('ip=[' + param_compute_node.vm_ipv6 + ']::[' + env.cluster.networking.ipv6_gateway +']:' + env.cluster.networking.ipv6_prefix | string + '::' + param_compute_node.vm_interface + ':none' ) if env.use_ipv6 == True else '' }}" \ {% endif %} - --extra-args "nameserver={{ env.cluster.networking.nameserver1 }}{{ (',' + env.cluster.networking.nameserver2) if env.cluster.networking.nameserver2 is defined else '' }}" \ - --extra-args "coreos.live.rootfs_url=http://{{ env.bastion.networking.ip }}:8080/bin/{{ rhcos_live_rootfs }}" \ + --extra-args "nameserver={{ env.cluster.networking.nameserver1 }}{{ (',' + env.cluster.networking.nameserver2) if (env.cluster.networking.nameserver2 is defined and env.cluster.networking.nameserver2 | length > 0) else '' }}" \ + --extra-args "coreos.live.rootfs_url={% if disconnected_enabled is defined and disconnected_enabled %}{{ env.file_server.protocol }}://{{ env.file_server.ip }}{{ (':' + env.file_server.port | string) if (env.file_server.port is defined and env.file_server.port | string | length > 0) else '' }}/bin/{{ rhcos_live_rootfs }}{% else %}http://{{ env.bastion.networking.ip }}:8080/bin/{{ rhcos_live_rootfs }}{% endif %}" \ --extra-args "coreos.inst.ignition_url=http://{{ env.bastion.networking.ip }}:8080/ignition/worker.ign" \ --extra-args "{{ _vm_console }}" timeout: 360 diff --git a/roles/create_compute_nodes/tasks/main.yaml b/roles/create_compute_nodes/tasks/main.yaml index 2e2c9ef05..dbae55fa8 100644 --- a/roles/create_compute_nodes/tasks/main.yaml +++ b/roles/create_compute_nodes/tasks/main.yaml @@ -44,9 +44,9 @@ {{ env.cluster.nodes.compute.vcpu_model_option }} \ --vcpus {{ env.cluster.nodes.compute.vcpu }} \ --network network={{ env.vnet_name }}{{ (',mac=' + env.cluster.nodes.compute.mac[i]) if (env.cluster.nodes.compute.mac[i] is defined and env.use_dhcp) }} \ - --location {{ rhcos_download_url }},kernel={{ rhcos_live_kernel }},initrd={{ rhcos_live_initrd }} \ + --location {% if disconnected_enabled is defined and disconnected_enabled %}{{ env.file_server.protocol }}://{{ env.file_server.ip }}{{ (':' + env.file_server.port | string) if (env.file_server.port is defined and env.file_server.port | string | length > 0) else '' }}/{{ disconnected.mirroring.file_server.clients_dir }}/{% else %}{{ rhcos_download_url }}{% endif %},kernel={{ rhcos_live_kernel }},initrd={{ rhcos_live_initrd }} \ --extra-args "rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda" \ - --extra-args "coreos.live.rootfs_url=http://{{ env.bastion.networking.ip }}:8080/bin/{{ rhcos_live_rootfs }}" \ + --extra-args "coreos.live.rootfs_url={% if disconnected_enabled is defined and disconnected_enabled %}{{ env.file_server.protocol }}://{{ env.file_server.ip }}{{ (':' + env.file_server.port | string) if (env.file_server.port is defined and env.file_server.port | string | length > 0) else '' }}/bin/{{ rhcos_live_rootfs }}{% else %}http://{{ env.bastion.networking.ip }}:8080/bin/{{ rhcos_live_rootfs }}{% endif %}" \ {% if (env.cluster.nodes.compute.mac[i] is defined and env.use_dhcp) %} --extra-args "ip=dhcp" \ {% else %} @@ -54,7 +54,7 @@ --extra-args "{{ ('ip=[' + env.cluster.nodes.compute.ipv6[i] + ']::[' + env.cluster.networking.ipv6_gateway +']:' + env.cluster.networking.ipv6_prefix | string + '::' + env.cluster.networking.interface + ':none' ) if env.use_ipv6 == True else '' }}" \ {% endif %} --extra-args "nameserver={{ env.cluster.networking.nameserver1 }}" \ - --extra-args "{{ ('nameserver=' + env.cluster.networking.nameserver2) if env.cluster.networking.nameserver2 is defined else '' }}" \ + --extra-args "{{ ('nameserver=' + env.cluster.networking.nameserver2) if (env.cluster.networking.nameserver2 is defined and env.cluster.networking.nameserver2 | length > 0) else '' }}" \ --extra-args "coreos.inst.ignition_url=http://{{ env.bastion.networking.ip }}:8080/ignition/worker.ign" \ --extra-args "{{ _vm_console }}" \ --memballoon none \ @@ -81,13 +81,13 @@ {{ env.cluster.nodes.infra.vcpu_model_option }} \ --vcpus {{ env.cluster.nodes.infra.vcpu }} \ --network network={{ env.vnet_name }} \ - --location {{ rhcos_download_url }},kernel={{ rhcos_live_kernel }},initrd={{ rhcos_live_initrd }} \ + --location {% if disconnected_enabled is defined and disconnected_enabled %}{{ env.file_server.protocol }}://{{ env.file_server.ip }}{{ (':' + env.file_server.port | string) if (env.file_server.port is defined and env.file_server.port | string | length > 0) else '' }}/{{ disconnected.mirroring.file_server.clients_dir }}/{% else %}{{ rhcos_download_url }}{% endif %},kernel={{ rhcos_live_kernel }},initrd={{ rhcos_live_initrd }} \ --extra-args "rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda" \ - --extra-args "coreos.live.rootfs_url=http://{{ env.bastion.networking.ip }}:8080/bin/{{ rhcos_live_rootfs }}" \ + --extra-args "coreos.live.rootfs_url={% if disconnected_enabled is defined and disconnected_enabled %}{{ env.file_server.protocol }}://{{ env.file_server.ip }}{{ (':' + env.file_server.port | string) if (env.file_server.port is defined and env.file_server.port | string | length > 0) else '' }}/bin/{{ rhcos_live_rootfs }}{% else %}http://{{ env.bastion.networking.ip }}:8080/bin/{{ rhcos_live_rootfs }}{% endif %}" \ --extra-args "ip={{ env.cluster.nodes.infra.ip[i] }}::{{ env.cluster.networking.gateway }}:{{ env.cluster.networking.subnetmask }}:{{ env.cluster.nodes.infra.hostname[i] }}.{{ env.cluster.networking.metadata_name }}.{{ env.cluster.networking.base_domain }}:{{ env.cluster.networking.interface }}:none:1500" \ --extra-args "{{ ('ip=[' + env.cluster.nodes.infra.ipv6[i] + ']::[' + env.cluster.networking.ipv6_gateway +']:' + env.cluster.networking.ipv6_prefix | string + '::' + env.cluster.networking.interface + ':none' ) if env.use_ipv6 == True else '' }}" \ --extra-args "nameserver={{ env.cluster.networking.nameserver1 }}" \ - --extra-args "{{ ('nameserver=' + env.cluster.networking.nameserver2) if env.cluster.networking.nameserver2 is defined else '' }}" \ + --extra-args "{{ ('nameserver=' + env.cluster.networking.nameserver2) if (env.cluster.networking.nameserver2 is defined and env.cluster.networking.nameserver2 | length > 0) else '' }}" \ --extra-args "coreos.inst.ignition_url=http://{{ env.bastion.networking.ip }}:8080/ignition/worker.ign" \ --extra-args "{{ _vm_console }}" \ --memballoon none \ @@ -132,9 +132,9 @@ {{ env.cluster.nodes.compute.vcpu_model_option }} \ --vcpus {{ env.cluster.nodes.compute.vcpu }} \ --network network={{ env.vnet_name }}{{ (',mac=' + compute_mac[i] if (compute_mac[i] is defined and env.use_dhcp)) }} \ - --location {{ rhcos_download_url }},kernel={{ rhcos_live_kernel }},initrd={{ rhcos_live_initrd }} \ + --location {% if disconnected_enabled is defined and disconnected_enabled %}{{ env.file_server.protocol }}://{{ env.file_server.ip }}{{ (':' + env.file_server.port | string) if (env.file_server.port is defined and env.file_server.port | string | length > 0) else '' }}/{{ disconnected.mirroring.file_server.clients_dir }}/{% else %}{{ rhcos_download_url }}{% endif %},kernel={{ rhcos_live_kernel }},initrd={{ rhcos_live_initrd }} \ --extra-args "rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda" \ - --extra-args "coreos.live.rootfs_url=http://{{ env.bastion.networking.ip }}:8080/bin/{{ rhcos_live_rootfs }}" \ + --extra-args "coreos.live.rootfs_url={% if disconnected_enabled is defined and disconnected_enabled %}{{ env.file_server.protocol }}://{{ env.file_server.ip }}{{ (':' + env.file_server.port | string) if (env.file_server.port is defined and env.file_server.port | string | length > 0) else '' }}/bin/{{ rhcos_live_rootfs }}{% else %}http://{{ env.bastion.networking.ip }}:8080/bin/{{ rhcos_live_rootfs }}{% endif %}" \ {% if (compute_mac[i] is defined and env.use_dhcp) %} --extra-args "ip=dhcp" \ {% else %} @@ -142,7 +142,7 @@ --extra-args "{{ ('ip=[' + compute_ipv6[i] + ']::[' + env.cluster.networking.ipv6_gateway +']:' + env.cluster.networking.ipv6_prefix | string + '::' + env.cluster.networking.interface + ':none' ) if env.use_ipv6 == True else '' }}" \ {% endif %} --extra-args "nameserver={{ env.cluster.networking.nameserver1 }}" \ - --extra-args "{{ ('nameserver=' + env.cluster.networking.nameserver2) if env.cluster.networking.nameserver2 is defined else '' }}" \ + --extra-args "{{ ('nameserver=' + env.cluster.networking.nameserver2) if (env.cluster.networking.nameserver2 is defined and env.cluster.networking.nameserver2 | length > 0) else '' }}" \ --extra-args "coreos.inst.ignition_url=http://{{ env.bastion.networking.ip }}:8080/ignition/worker.ign" \ --extra-args "{{ _vm_console }}" \ --memballoon none \ @@ -167,13 +167,13 @@ {{ env.cluster.nodes.infra.vcpu_model_option }} \ --vcpus {{ env.cluster.nodes.infra.vcpu }} \ --network network={{ env.vnet_name }} \ - --location {{ rhcos_download_url }},kernel={{ rhcos_live_kernel }},initrd={{ rhcos_live_initrd }} \ + --location {% if disconnected_enabled is defined and disconnected_enabled %}{{ env.file_server.protocol }}://{{ env.file_server.ip }}{{ (':' + env.file_server.port | string) if (env.file_server.port is defined and env.file_server.port | string | length > 0) else '' }}/{{ disconnected.mirroring.file_server.clients_dir }}/{% else %}{{ rhcos_download_url }}{% endif %},kernel={{ rhcos_live_kernel }},initrd={{ rhcos_live_initrd }} \ --extra-args "rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda" \ - --extra-args "coreos.live.rootfs_url=http://{{ env.bastion.networking.ip }}:8080/bin/{{ rhcos_live_rootfs }}" \ + --extra-args "coreos.live.rootfs_url={% if disconnected_enabled is defined and disconnected_enabled %}{{ env.file_server.protocol }}://{{ env.file_server.ip }}{{ (':' + env.file_server.port | string) if (env.file_server.port is defined and env.file_server.port | string | length > 0) else '' }}/bin/{{ rhcos_live_rootfs }}{% else %}http://{{ env.bastion.networking.ip }}:8080/bin/{{ rhcos_live_rootfs }}{% endif %}" \ --extra-args "ip={{ infra_ip[i] }}::{{ env.cluster.networking.gateway }}:{{ env.cluster.networking.subnetmask }}:{{ infra_hostname[i] }}.{{ env.cluster.networking.metadata_name }}.{{ env.cluster.networking.base_domain }}:{{ env.cluster.networking.interface }}:none:1500" \ --extra-args "{{ ('ip=[' + infra_ipv6[i] + ']::[' + env.cluster.networking.ipv6_gateway +']:' + env.cluster.networking.ipv6_prefix | string + '::' + env.cluster.networking.interface + ':none' ) if env.use_ipv6 == True else '' }}" \ --extra-args "nameserver={{ env.cluster.networking.nameserver1 }}" \ - --extra-args "{{ ('nameserver=' + env.cluster.networking.nameserver2) if env.cluster.networking.nameserver2 is defined else '' }}" \ + --extra-args "{{ ('nameserver=' + env.cluster.networking.nameserver2) if (env.cluster.networking.nameserver2 is defined and env.cluster.networking.nameserver2 | length > 0) else '' }}" \ --extra-args "coreos.inst.ignition_url=http://{{ env.bastion.networking.ip }}:8080/ignition/worker.ign" \ --extra-args "{{ _vm_console }}" \ --memballoon none \ diff --git a/roles/create_control_nodes/tasks/main.yaml b/roles/create_control_nodes/tasks/main.yaml index 59d8d3fad..e6b8f97e5 100644 --- a/roles/create_control_nodes/tasks/main.yaml +++ b/roles/create_control_nodes/tasks/main.yaml @@ -43,9 +43,9 @@ {{ env.cluster.nodes.control.vcpu_model_option }} \ --vcpus {{ env.cluster.nodes.control.vcpu }} \ --network network={{ env.vnet_name }}{{ (',mac=' + env.cluster.nodes.control.mac[i]) if (env.cluster.nodes.control.mac[i] is defined and env.use_dhcp) }} \ - --location {{ rhcos_download_url }},kernel={{ rhcos_live_kernel }},initrd={{ rhcos_live_initrd }} \ + --location {% if disconnected_enabled is defined and disconnected_enabled %}{{ env.file_server.protocol }}://{{ env.file_server.ip }}{{ (':' + env.file_server.port | string) if (env.file_server.port is defined and env.file_server.port | string | length > 0) else '' }}/{{ disconnected.mirroring.file_server.clients_dir }}/{% else %}{{ rhcos_download_url }}{% endif %},kernel={{ rhcos_live_kernel }},initrd={{ rhcos_live_initrd }} \ --extra-args "rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda" \ - --extra-args "coreos.live.rootfs_url=http://{{ env.bastion.networking.ip }}:8080/bin/{{ rhcos_live_rootfs }}" \ + --extra-args "coreos.live.rootfs_url={% if disconnected_enabled is defined and disconnected_enabled %}{{ env.file_server.protocol }}://{{ env.file_server.ip }}{{ (':' + env.file_server.port | string) if (env.file_server.port is defined and env.file_server.port | string | length > 0) else '' }}/bin/{{ rhcos_live_rootfs }}{% else %}http://{{ env.bastion.networking.ip }}:8080/bin/{{ rhcos_live_rootfs }}{% endif %}" \ {% if (env.cluster.nodes.control.mac[i] is defined and env.use_dhcp) %} --extra-args "ip=dhcp" \ {% else %} @@ -53,7 +53,7 @@ --extra-args "{{ ('ip=[' + env.cluster.nodes.control.ipv6[i] + ']::[' + env.cluster.networking.ipv6_gateway +']:' + env.cluster.networking.ipv6_prefix | string + '::' + env.cluster.networking.interface + ':none' ) if env.use_ipv6 == True else '' }}" \ {% endif %} --extra-args "nameserver={{ env.cluster.networking.nameserver1 }}" \ - --extra-args "{{ ('nameserver=' + env.cluster.networking.nameserver2) if env.cluster.networking.nameserver2 is defined else '' }}" \ + --extra-args "{{ ('nameserver=' + env.cluster.networking.nameserver2) if (env.cluster.networking.nameserver2 is defined and env.cluster.networking.nameserver2 | length > 0) else '' }}" \ --extra-args "coreos.inst.ignition_url=http://{{ env.bastion.networking.ip }}:8080/ignition/master.ign" \ --extra-args "{{ _vm_console }}" \ --memballoon none \ @@ -82,9 +82,9 @@ {{ env.cluster.nodes.control.vcpu_model_option }} \ --vcpus {{ env.cluster.nodes.control.vcpu }} \ --network network={{ env.vnet_name }}{{ (',mac=' + env.cluster.nodes.control.mac[0]) if (env.cluster.nodes.control.mac[0] is defined and env.use_dhcp) }} \ - --location {{ rhcos_download_url }},kernel={{ rhcos_live_kernel }},initrd={{ rhcos_live_initrd }} \ + --location {% if disconnected_enabled is defined and disconnected_enabled %}{{ env.file_server.protocol }}://{{ env.file_server.ip }}{{ (':' + env.file_server.port | string) if (env.file_server.port is defined and env.file_server.port | string | length > 0) else '' }}/{{ disconnected.mirroring.file_server.clients_dir }}/{% else %}{{ rhcos_download_url }}{% endif %},kernel={{ rhcos_live_kernel }},initrd={{ rhcos_live_initrd }} \ --extra-args "rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda" \ - --extra-args "coreos.live.rootfs_url=http://{{ env.bastion.networking.ip }}:8080/bin/{{ rhcos_live_rootfs }}" \ + --extra-args "coreos.live.rootfs_url={% if disconnected_enabled is defined and disconnected_enabled %}{{ env.file_server.protocol }}://{{ env.file_server.ip }}{{ (':' + env.file_server.port | string) if (env.file_server.port is defined and env.file_server.port | string | length > 0) else '' }}/bin/{{ rhcos_live_rootfs }}{% else %}http://{{ env.bastion.networking.ip }}:8080/bin/{{ rhcos_live_rootfs }}{% endif %}" \ {% if (env.cluster.nodes.control.mac[0] is defined and env.use_dhcp) %} --extra-args "ip=dhcp" \ {% else %} @@ -92,7 +92,7 @@ --extra-args "{{ ('ip=[' + env.cluster.nodes.control.ipv6[0] + ']::[' + env.cluster.networking.ipv6_gateway +']:' + env.cluster.networking.ipv6_prefix | string + '::' + env.cluster.networking.interface + ':none' ) if env.use_ipv6 == True else '' }}" \ {% endif %} --extra-args "nameserver={{ env.cluster.networking.nameserver1 }}" \ - --extra-args "{{ ('nameserver=' + env.cluster.networking.nameserver2) if env.cluster.networking.nameserver2 is defined else '' }}" \ + --extra-args "{{ ('nameserver=' + env.cluster.networking.nameserver2) if (env.cluster.networking.nameserver2 is defined and env.cluster.networking.nameserver2 | length > 0) else '' }}" \ --extra-args "coreos.inst.ignition_url=http://{{ env.bastion.networking.ip }}:8080/ignition/master.ign" \ --extra-args "{{ _vm_console }}" \ --memballoon none \ @@ -114,9 +114,9 @@ {{ env.cluster.nodes.control.vcpu_model_option }} \ --vcpus {{ env.cluster.nodes.control.vcpu }} \ --network network={{ env.vnet_name }}{{ (',mac=' + env.cluster.nodes.control.mac[1]) if (env.cluster.nodes.control.mac[0] is defined and env.use_dhcp) }} \ - --location {{ rhcos_download_url }},kernel={{ rhcos_live_kernel }},initrd={{ rhcos_live_initrd }} \ + --location {% if disconnected_enabled is defined and disconnected_enabled %}{{ env.file_server.protocol }}://{{ env.file_server.ip }}{{ (':' + env.file_server.port | string) if (env.file_server.port is defined and env.file_server.port | string | length > 0) else '' }}/{{ disconnected.mirroring.file_server.clients_dir }}/{% else %}{{ rhcos_download_url }}{% endif %},kernel={{ rhcos_live_kernel }},initrd={{ rhcos_live_initrd }} \ --extra-args "rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda" \ - --extra-args "coreos.live.rootfs_url=http://{{ env.bastion.networking.ip }}:8080/bin/{{ rhcos_live_rootfs }}" \ + --extra-args "coreos.live.rootfs_url={% if disconnected_enabled is defined and disconnected_enabled %}{{ env.file_server.protocol }}://{{ env.file_server.ip }}{{ (':' + env.file_server.port | string) if (env.file_server.port is defined and env.file_server.port | string | length > 0) else '' }}/bin/{{ rhcos_live_rootfs }}{% else %}http://{{ env.bastion.networking.ip }}:8080/bin/{{ rhcos_live_rootfs }}{% endif %}" \ {% if (env.cluster.nodes.control.mac[0] is defined and env.use_dhcp) %} --extra-args "ip=dhcp" \ {% else %} @@ -124,7 +124,7 @@ --extra-args "{{ ('ip=[' + env.cluster.nodes.control.ipv6[1] + ']::[' + env.cluster.networking.ipv6_gateway +']:' + env.cluster.networking.ipv6_prefix | string + '::' + env.cluster.networking.interface + ':none' ) if env.use_ipv6 == True else '' }}" \ {% endif %} --extra-args "nameserver={{ env.cluster.networking.nameserver1 }}" \ - --extra-args "{{ ('nameserver=' + env.cluster.networking.nameserver2) if env.cluster.networking.nameserver2 is defined else '' }}" \ + --extra-args "{{ ('nameserver=' + env.cluster.networking.nameserver2) if (env.cluster.networking.nameserver2 is defined and env.cluster.networking.nameserver2 | length > 0) else '' }}" \ --extra-args "coreos.inst.ignition_url=http://{{ env.bastion.networking.ip }}:8080/ignition/master.ign" \ --extra-args "{{ _vm_console }}" \ --memballoon none \ @@ -145,9 +145,9 @@ {{ env.cluster.nodes.control.vcpu_model_option }} \ --vcpus {{ env.cluster.nodes.control.vcpu }} \ --network network={{ env.vnet_name }}{{ (',mac=' + env.cluster.nodes.control.mac[2]) if (env.cluster.nodes.control.mac[2] is defined and env.use_dhcp) }} \ - --location {{ rhcos_download_url }},kernel={{ rhcos_live_kernel }},initrd={{ rhcos_live_initrd }} \ + --location {% if disconnected_enabled is defined and disconnected_enabled %}{{ env.file_server.protocol }}://{{ env.file_server.ip }}{{ (':' + env.file_server.port | string) if (env.file_server.port is defined and env.file_server.port | string | length > 0) else '' }}/{{ disconnected.mirroring.file_server.clients_dir }}/{% else %}{{ rhcos_download_url }}{% endif %},kernel={{ rhcos_live_kernel }},initrd={{ rhcos_live_initrd }} \ --extra-args "rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda" \ - --extra-args "coreos.live.rootfs_url=http://{{ env.bastion.networking.ip }}:8080/bin/{{ rhcos_live_rootfs }}" \ + --extra-args "coreos.live.rootfs_url={% if disconnected_enabled is defined and disconnected_enabled %}{{ env.file_server.protocol }}://{{ env.file_server.ip }}{{ (':' + env.file_server.port | string) if (env.file_server.port is defined and env.file_server.port | string | length > 0) else '' }}/bin/{{ rhcos_live_rootfs }}{% else %}http://{{ env.bastion.networking.ip }}:8080/bin/{{ rhcos_live_rootfs }}{% endif %}" \ {% if (env.cluster.nodes.control.mac[0] is defined and env.use_dhcp) %} --extra-args "ip=dhcp" \ {% else %} @@ -155,7 +155,7 @@ --extra-args "{{ ('ip=[' + env.cluster.nodes.control.ipv6[2] + ']::[' + env.cluster.networking.ipv6_gateway +']:' + env.cluster.networking.ipv6_prefix | string + '::' + env.cluster.networking.interface + ':none' ) if env.use_ipv6 == True else '' }}" \ {% endif %} --extra-args "nameserver={{ env.cluster.networking.nameserver1 }}" \ - --extra-args "{{ ('nameserver=' + env.cluster.networking.nameserver2) if env.cluster.networking.nameserver2 is defined else '' }}" \ + --extra-args "{{ ('nameserver=' + env.cluster.networking.nameserver2) if (env.cluster.networking.nameserver2 is defined and env.cluster.networking.nameserver2 | length > 0) else '' }}" \ --extra-args "coreos.inst.ignition_url=http://{{ env.bastion.networking.ip }}:8080/ignition/master.ign" \ --extra-args "{{ _vm_console }}" \ --memballoon none \ diff --git a/roles/disconnected_apply_operator_manifests_to_cluster/tasks/main.yaml b/roles/disconnected_apply_operator_manifests_to_cluster/tasks/main.yaml index 7d7c86e94..501847070 100644 --- a/roles/disconnected_apply_operator_manifests_to_cluster/tasks/main.yaml +++ b/roles/disconnected_apply_operator_manifests_to_cluster/tasks/main.yaml @@ -14,4 +14,4 @@ - name: disable default content sources tags: disconnected_apply_operator_manifests ansible.builtin.command: "oc patch OperatorHub cluster --type json -p '[{\"op\": \"add\", \"path\": \"/spec/disableAllDefaultSources\", \"value\": true}]'" - when: disconnected.enabled + when: disconnected_enabled diff --git a/roles/disconnected_download_oc_mirror/tasks/main.yaml b/roles/disconnected_download_oc_mirror/tasks/main.yaml new file mode 100644 index 000000000..14944f621 --- /dev/null +++ b/roles/disconnected_download_oc_mirror/tasks/main.yaml @@ -0,0 +1,342 @@ +--- +# Role: disconnected_download_oc_mirror +# Description: Downloads oc-mirror plugin and OCP client tools to the file server +# This role runs on the bastion/file server to download necessary files for disconnected installation + +- name: Download oc-mirror and client tools to file server + tags: download_oc_mirror + when: disconnected_enabled + block: + - name: Get user home directory + ansible.builtin.shell: | + set -o pipefail + getent passwd {{ ansible_user }} | awk -F: '{ print $6 }' + changed_when: false + register: user_home + + - name: Set oc-mirror role variables from existing disconnected structure + ansible.builtin.set_fact: + oc_mirror_file_server: + download_dir: >- + {{ + ( + disconnected.get('mirroring', {}) + .get('file_server', {}) + .get('download_dir') + ) | default('/var/tmp/oc-mirror', true) + }} + document_root: >- + {{ + ( + disconnected.get('mirroring', {}) + .get('file_server', {}) + .get('document_root') + ) | default('/var/www/html', true) + }} + clients_dir: >- + {{ + ( + disconnected.get('mirroring', {}) + .get('file_server', {}) + .get('clients_dir') + ) | default('clients', true) + }} + oc_mirror_tgz: >- + {{ + ( + disconnected.get('mirroring', {}) + .get('file_server', {}) + .get('oc_mirror_tgz') + ) | default('oc-mirror.tar.gz', true) + }} + oc_mirror_host: + ip: >- + {{ + ( + disconnected.get('mirroring', {}) + .get('host', {}) + .get('ip') + ) | default('', true) + }} + oc_mirror_download_cfg: + base_url: >- + {{ + ( + disconnected.get('mirroring', {}) + .get('oc_mirror_download', {}) + .get('base_url') + ) | default(ocp_download_url | regex_replace('/+$', ''), true) + }} + oc_mirror_tgz: >- + {{ + ( + disconnected.get('mirroring', {}) + .get('oc_mirror_download', {}) + .get('oc_mirror_tgz') + ) | default('oc-mirror.tar.gz', true) + }} + oc_mirror_client_download: + ocp_download_url: >- + {{ + ( + disconnected.get('mirroring', {}) + .get('client_download', {}) + .get('ocp_download_url') + ) + }} + ocp_client_tgz: >- + {{ + ( + disconnected.get('mirroring', {}) + .get('client_download', {}) + .get('ocp_client_tgz') + ) | default(ocp_client_tgz | default('openshift-client-linux.tar.gz'), true) + }} + oc_mirror_rhcos_download: + rhcos_download_url: >- + {{ + ( + disconnected.get('mirroring', {}) + .get('rhcos_download', {}) + .get('rhcos_download_url') + ) + }} + rhcos_live_rootfs: >- + {{ + ( + disconnected.get('mirroring', {}) + .get('rhcos_download', {}) + .get('rhcos_live_rootfs') + ) | default(rhcos_live_rootfs | default('rhcos-live-rootfs.s390x.img'), true) + }} + + - name: Create download directory on file server + ansible.builtin.file: + path: "{{ oc_mirror_file_server.download_dir }}" + state: directory + owner: "{{ ansible_user }}" + group: "{{ ansible_user }}" + mode: "0755" + + - name: Create clients directory on file server + ansible.builtin.file: + path: "{{ oc_mirror_file_server.document_root }}/{{ oc_mirror_file_server.clients_dir }}" + state: directory + owner: "{{ ansible_user }}" + group: "{{ ansible_user }}" + mode: "0755" + ignore_errors: true + + - name: Create bin directory on file server + ansible.builtin.file: + path: "{{ oc_mirror_file_server.document_root }}/bin" + state: directory + owner: "{{ ansible_user }}" + group: "{{ ansible_user }}" + mode: "0755" + ignore_errors: true + + - name: Determine whether mirroring host and file server are the same host + ansible.builtin.set_fact: + mirror_and_file_server_same_host: "{{ (oc_mirror_host.ip | default('')) == (env.file_server.ip | default('')) }}" + + - name: Test internet connectivity from bastion + ansible.builtin.uri: + url: "https://mirror.openshift.com" + method: HEAD + timeout: 10 + validate_certs: true + register: internet_test + failed_when: false + changed_when: false + + - name: Display connectivity test result + ansible.builtin.debug: + msg: | + Internet connectivity test: {{ 'SUCCESS' if internet_test.status is defined and internet_test.status == 200 else 'FAILED' }} + {% if internet_test.status is not defined %} + Error: {{ internet_test.msg | default('Unable to reach mirror.openshift.com') }} + + Troubleshooting steps: + 1. Verify DNS is configured on bastion: cat /etc/resolv.conf + 2. Test DNS resolution: nslookup mirror.openshift.com + 3. Test connectivity: curl -I https://mirror.openshift.com + 4. Check if proxy is needed: echo $http_proxy $https_proxy + 5. Verify KVM host IP forwarding: sysctl net.ipv4.ip_forward + 6. Check bastion default gateway: ip route show default + {% endif %} + + - name: Fail if internet is not accessible + ansible.builtin.fail: + msg: | + Cannot reach mirror.openshift.com from bastion. + Please ensure: + - DNS is properly configured (/etc/resolv.conf) + - Default gateway is set correctly + - KVM host has IP forwarding enabled + - Firewall rules allow outbound HTTPS traffic + when: internet_test.status is not defined or internet_test.status != 200 + + - name: Download oc-mirror plugin + ansible.builtin.get_url: + url: "{{ oc_mirror_download_cfg.base_url }}/{{ oc_mirror_download_cfg.oc_mirror_tgz }}" + dest: "{{ oc_mirror_file_server.download_dir }}/{{ oc_mirror_file_server.oc_mirror_tgz }}" + mode: "0644" + timeout: 300 + register: oc_mirror_download + retries: 3 + delay: 10 + until: oc_mirror_download is succeeded + + - name: Download OCP client tools + ansible.builtin.get_url: + url: "{{ oc_mirror_client_download.ocp_download_url }}/{{ oc_mirror_client_download.ocp_client_tgz }}" + dest: "{{ oc_mirror_file_server.download_dir }}/{{ oc_mirror_client_download.ocp_client_tgz }}" + mode: "0644" + timeout: 300 + register: ocp_client_download + retries: 3 + delay: 10 + until: ocp_client_download is succeeded + + - name: Verify OCP client tools file exists + ansible.builtin.stat: + path: "{{ oc_mirror_file_server.download_dir }}/{{ oc_mirror_client_download.ocp_client_tgz }}" + register: ocp_client_file_stat + failed_when: not ocp_client_file_stat.stat.exists + + - name: Download OCP installer + ansible.builtin.get_url: + url: "{{ oc_mirror_client_download.ocp_download_url }}/openshift-install-linux.tar.gz" + dest: "{{ oc_mirror_file_server.download_dir }}/openshift-install-linux.tar.gz" + mode: "0644" + timeout: 300 + register: ocp_installer_download + retries: 3 + delay: 10 + until: ocp_installer_download is succeeded + + - name: Download RHCOS rootfs + ansible.builtin.get_url: + url: "{{ oc_mirror_rhcos_download.rhcos_download_url }}/{{ oc_mirror_rhcos_download.rhcos_live_rootfs }}" + dest: "{{ oc_mirror_file_server.download_dir }}/{{ oc_mirror_rhcos_download.rhcos_live_rootfs }}" + mode: "0644" + timeout: 600 + register: rhcos_rootfs_download + retries: 3 + delay: 10 + until: rhcos_rootfs_download is succeeded + + - name: Verify all downloaded files exist before copying + ansible.builtin.stat: + path: "{{ item }}" + register: download_files_stat + failed_when: not download_files_stat.stat.exists + loop: + - "{{ oc_mirror_file_server.download_dir }}/{{ oc_mirror_file_server.oc_mirror_tgz }}" + - "{{ oc_mirror_file_server.download_dir }}/{{ oc_mirror_client_download.ocp_client_tgz }}" + - "{{ oc_mirror_file_server.download_dir }}/openshift-install-linux.tar.gz" + - "{{ oc_mirror_file_server.download_dir }}/{{ oc_mirror_rhcos_download.rhcos_live_rootfs }}" + + - name: Copy artifacts locally when mirroring host is the file server + when: mirror_and_file_server_same_host + block: + - name: Copy oc-mirror to clients directory for distribution + ansible.builtin.copy: + src: "{{ oc_mirror_file_server.download_dir }}/{{ oc_mirror_file_server.oc_mirror_tgz }}" + dest: "{{ oc_mirror_file_server.document_root }}/{{ oc_mirror_file_server.clients_dir }}/{{ oc_mirror_file_server.oc_mirror_tgz }}" + owner: "{{ ansible_user }}" + group: "{{ ansible_user }}" + mode: "0644" + remote_src: true + + - name: Copy OCP client to clients directory for distribution + ansible.builtin.copy: + src: "{{ oc_mirror_file_server.download_dir }}/{{ oc_mirror_client_download.ocp_client_tgz }}" + dest: "{{ oc_mirror_file_server.document_root }}/{{ oc_mirror_file_server.clients_dir }}/{{ oc_mirror_client_download.ocp_client_tgz }}" + owner: "{{ ansible_user }}" + group: "{{ ansible_user }}" + mode: "0644" + remote_src: true + + - name: Copy OCP installer to clients directory for distribution + ansible.builtin.copy: + src: "{{ oc_mirror_file_server.download_dir }}/openshift-install-linux.tar.gz" + dest: "{{ oc_mirror_file_server.document_root }}/{{ oc_mirror_file_server.clients_dir }}/openshift-install-linux.tar.gz" + owner: "{{ ansible_user }}" + group: "{{ ansible_user }}" + mode: "0644" + remote_src: true + + - name: Copy RHCOS rootfs to bin directory for node installation + ansible.builtin.copy: + src: "{{ oc_mirror_file_server.download_dir }}/{{ oc_mirror_rhcos_download.rhcos_live_rootfs }}" + dest: "{{ oc_mirror_file_server.document_root }}/bin/{{ oc_mirror_rhcos_download.rhcos_live_rootfs }}" + owner: "{{ ansible_user }}" + group: "{{ ansible_user }}" + mode: "0644" + remote_src: true + + - name: Copy artifacts to separate file server when mirroring host differs + when: not mirror_and_file_server_same_host + block: + - name: Copy oc-mirror to separate file server clients directory + ansible.builtin.copy: + src: "{{ oc_mirror_file_server.download_dir }}/{{ oc_mirror_file_server.oc_mirror_tgz }}" + dest: "{{ oc_mirror_file_server.document_root }}/{{ oc_mirror_file_server.clients_dir }}/{{ oc_mirror_file_server.oc_mirror_tgz }}" + owner: "{{ ansible_user }}" + group: "{{ ansible_user }}" + mode: "0644" + remote_src: true + delegate_to: "{{ env.file_server.ip }}" + + - name: Copy OCP client to separate file server clients directory + ansible.builtin.copy: + src: "{{ oc_mirror_file_server.download_dir }}/{{ oc_mirror_client_download.ocp_client_tgz }}" + dest: "{{ oc_mirror_file_server.document_root }}/{{ oc_mirror_file_server.clients_dir }}/{{ oc_mirror_client_download.ocp_client_tgz }}" + owner: "{{ ansible_user }}" + group: "{{ ansible_user }}" + mode: "0644" + remote_src: true + delegate_to: "{{ env.file_server.ip }}" + + - name: Copy OCP installer to separate file server clients directory + ansible.builtin.copy: + src: "{{ oc_mirror_file_server.download_dir }}/openshift-install-linux.tar.gz" + dest: "{{ oc_mirror_file_server.document_root }}/{{ oc_mirror_file_server.clients_dir }}/openshift-install-linux.tar.gz" + owner: "{{ ansible_user }}" + group: "{{ ansible_user }}" + mode: "0644" + remote_src: true + delegate_to: "{{ env.file_server.ip }}" + + - name: Copy RHCOS rootfs to separate file server bin directory + ansible.builtin.copy: + src: "{{ oc_mirror_file_server.download_dir }}/{{ oc_mirror_rhcos_download.rhcos_live_rootfs }}" + dest: "{{ oc_mirror_file_server.document_root }}/bin/{{ oc_mirror_rhcos_download.rhcos_live_rootfs }}" + owner: "{{ ansible_user }}" + group: "{{ ansible_user }}" + mode: "0644" + remote_src: true + delegate_to: "{{ env.file_server.ip }}" + + - name: Display download summary + ansible.builtin.debug: + msg: + - >- + oc-mirror available at: {{ env.file_server.protocol }}://{{ env.file_server.ip }}:{{ + env.file_server.port }}/{{ oc_mirror_file_server.clients_dir }}/{{ + oc_mirror_file_server.oc_mirror_tgz }} + - >- + OCP client available at: {{ env.file_server.protocol }}://{{ env.file_server.ip }}:{{ + env.file_server.port }}/{{ oc_mirror_file_server.clients_dir }}/{{ + oc_mirror_client_download.ocp_client_tgz }} + - >- + OCP installer available at: {{ env.file_server.protocol }}://{{ env.file_server.ip }}:{{ + env.file_server.port }}/{{ oc_mirror_file_server.clients_dir }}/openshift-install-linux.tar.gz + - >- + RHCOS rootfs available at: {{ env.file_server.protocol }}://{{ env.file_server.ip }}:{{ + env.file_server.port }}/bin/rhcos-live-rootfs.s390x.img + - "Files copied to: {{ oc_mirror_file_server.document_root }}/{{ oc_mirror_file_server.clients_dir }}/" + +# Assisted by Bob diff --git a/roles/disconnected_download_registry_rpms/tasks/main.yaml b/roles/disconnected_download_registry_rpms/tasks/main.yaml new file mode 100644 index 000000000..58b508860 --- /dev/null +++ b/roles/disconnected_download_registry_rpms/tasks/main.yaml @@ -0,0 +1,75 @@ +--- +# Role: disconnected_download_registry_rpms +# Description: Downloads required RPM packages for registry setup to the file server +# This role runs on a host with internet access to download RPMs for offline installation + +- name: Download registry RPMs to file server + tags: download_registry_rpms + when: + - disconnected_enabled + - disconnected.registry.bastion.enabled | default(false) + - not (disconnected.registry.bastion.use_local_repo | default(true)) + block: + - name: Create RPM download directory on file server + ansible.builtin.file: + path: "{{ env.file_server.cfgs_dir }}/{{ disconnected.registry.bastion.rpm_path }}" + state: directory + owner: "{{ ansible_user }}" + group: "{{ ansible_user }}" + mode: "0755" + + - name: Create temporary download directory + ansible.builtin.file: + path: "/tmp/rpm-download" + state: directory + owner: "{{ ansible_user }}" + group: "{{ ansible_user }}" + mode: "0755" + + - name: Download required packages and dependencies for s390x + become: true + ansible.builtin.shell: | + set -o pipefail + yumdownloader --resolve --destdir=/tmp/rpm-download --archlist=s390x,noarch \ + podman \ + httpd-tools \ + openssl \ + container-selinux \ + conmon \ + crun \ + fuse-overlayfs \ + slirp4netns + args: + creates: /tmp/rpm-download/podman-*.rpm + register: yum_download + changed_when: yum_download.rc == 0 + + - name: Copy downloaded RPMs to file server distribution directory + ansible.builtin.copy: + src: "/tmp/rpm-download/" + dest: "{{ env.file_server.cfgs_dir }}/{{ disconnected.registry.bastion.rpm_path }}/" + owner: "{{ ansible_user }}" + group: "{{ ansible_user }}" + mode: "0644" + remote_src: true + + - name: List downloaded RPMs + ansible.builtin.find: + paths: "{{ env.file_server.cfgs_dir }}/{{ disconnected.registry.bastion.rpm_path }}" + patterns: "*.rpm" + register: downloaded_rpms + + - name: Display downloaded RPMs + ansible.builtin.debug: + msg: + - "Downloaded {{ downloaded_rpms.files | length }} RPM packages" + - "Location: {{ env.file_server.cfgs_dir }}/{{ disconnected.registry.bastion.rpm_path }}" + - "Files:" + - "{{ downloaded_rpms.files | map(attribute='path') | map('basename') | list }}" + + - name: Clean up temporary download directory + ansible.builtin.file: + path: "/tmp/rpm-download" + state: absent + +# Made with Bob diff --git a/roles/disconnected_mirror_images/tasks/main.yaml b/roles/disconnected_mirror_images/tasks/main.yaml index bedf5fbbf..d297c8631 100644 --- a/roles/disconnected_mirror_images/tasks/main.yaml +++ b/roles/disconnected_mirror_images/tasks/main.yaml @@ -1,7 +1,7 @@ --- - name: mirror artifacts in disconnected mode tags: mirror_artifacts - when: disconnected.enabled + when: disconnected_enabled block: - name: Setting Up nameserver on mirror host to resolve DNS ansible.builtin.template: diff --git a/roles/disconnected_mirror_images/templates/imageset.yaml.j2 b/roles/disconnected_mirror_images/templates/imageset.yaml.j2 index a7597636b..49d8cffc9 100644 --- a/roles/disconnected_mirror_images/templates/imageset.yaml.j2 +++ b/roles/disconnected_mirror_images/templates/imageset.yaml.j2 @@ -1,7 +1,7 @@ kind: ImageSetConfiguration apiVersion: {{ disconnected.mirroring.oc_mirror.image_set.apiVersion }} archiveSize: 4 -{% if (disconnected.enabled) and (disconnected.mirroring.oc_mirror.image_set.apiVersion == "mirror.openshift.io/v1alpha2") %} +{% if (disconnected_enabled) and (disconnected.mirroring.oc_mirror.image_set.apiVersion == "mirror.openshift.io/v1alpha2") %} storageConfig: {% if disconnected.mirroring.oc_mirror.image_set.storageConfig.registry.enabled %} registry: diff --git a/roles/disconnected_mirror_ocp_bastion/files/monitor-oc-mirror.sh b/roles/disconnected_mirror_ocp_bastion/files/monitor-oc-mirror.sh new file mode 100644 index 000000000..7051f7624 --- /dev/null +++ b/roles/disconnected_mirror_ocp_bastion/files/monitor-oc-mirror.sh @@ -0,0 +1,65 @@ +#!/bin/bash +# Script: monitor-oc-mirror.sh +# Description: Real-time monitoring of oc-mirror progress +# Usage: ./monitor-oc-mirror.sh [log-file-path] + +LOG_FILE="${1:-/opt/oc-mirror/oc-mirror-progress.log}" + +if [ ! -f "$LOG_FILE" ]; then + echo "Error: Log file not found: $LOG_FILE" + echo "Usage: $0 [log-file-path]" + exit 1 +fi + +echo "==========================================" +echo "OC-Mirror Progress Monitor" +echo "==========================================" +echo "Log file: $LOG_FILE" +echo "Press Ctrl+C to exit" +echo "==========================================" +echo "" + +# Function to display statistics +show_stats() { + local total_images=$(grep -c "sha256" "$LOG_FILE" 2>/dev/null || echo "0") + local errors=$(grep -c "ERROR" "$LOG_FILE" 2>/dev/null || echo "0") + local warnings=$(grep -c "WARN" "$LOG_FILE" 2>/dev/null || echo "0") + + echo "" + echo "==========================================" + echo "Statistics ($(date '+%Y-%m-%d %H:%M:%S'))" + echo "==========================================" + echo "Images processed: $total_images" + echo "Errors: $errors" + echo "Warnings: $warnings" + echo "==========================================" + echo "" +} + +# Show initial stats +show_stats + +# Monitor log file with filtering for relevant information +tail -f "$LOG_FILE" | while read -r line; do + # Filter and colorize output + if echo "$line" | grep -q "ERROR"; then + echo -e "\033[0;31m[ERROR]\033[0m $line" + elif echo "$line" | grep -q "WARN"; then + echo -e "\033[0;33m[WARN]\033[0m $line" + elif echo "$line" | grep -qE "mirroring|copying"; then + echo -e "\033[0;36m[MIRROR]\033[0m $line" + elif echo "$line" | grep -q "sha256"; then + # Extract image name if possible + image=$(echo "$line" | grep -oP '(?<=copying )[^ ]+' || echo "$line") + echo -e "\033[0;32m[IMAGE]\033[0m $image" + elif echo "$line" | grep -qE "INFO.*collecting|INFO.*found"; then + echo -e "\033[0;34m[INFO]\033[0m $line" + fi + + # Show stats every 50 lines + if [ $((RANDOM % 50)) -eq 0 ]; then + show_stats + fi +done + +# Made with Bob diff --git a/roles/disconnected_mirror_ocp_bastion/handlers/main.yaml b/roles/disconnected_mirror_ocp_bastion/handlers/main.yaml new file mode 100644 index 000000000..3a1deca4c --- /dev/null +++ b/roles/disconnected_mirror_ocp_bastion/handlers/main.yaml @@ -0,0 +1,30 @@ +--- +# Handlers for disconnected_mirror_ocp_bastion role + +- name: Display progress update + ansible.builtin.shell: | + set -o pipefail + if [ -f {{ disconnected.mirroring.bastion.working_dir }}/oc-mirror-progress.log ]; then + echo "==========================================" + echo "Progress Update - $(date)" + echo "==========================================" + tail -20 {{ disconnected.mirroring.bastion.working_dir }}/oc-mirror-progress.log | \ + grep -E "mirroring|copying|Pulling|sha256|INFO|ERROR" | tail -10 || \ + echo "Mirroring in progress..." + echo "" + echo "Images mirrored so far:" + grep -c "sha256" {{ disconnected.mirroring.bastion.working_dir }}/oc-mirror-progress.log 2>/dev/null || echo "0" + echo "==========================================" + fi + args: + executable: /bin/bash + register: progress_output + changed_when: false + delegate_to: "{{ groups['bastion'][0] }}" + +- name: Show progress + ansible.builtin.debug: + var: progress_output.stdout_lines + when: progress_output is defined + +# Made with Bob diff --git a/roles/disconnected_mirror_ocp_bastion/tasks/main.yaml b/roles/disconnected_mirror_ocp_bastion/tasks/main.yaml new file mode 100644 index 000000000..63fa98c52 --- /dev/null +++ b/roles/disconnected_mirror_ocp_bastion/tasks/main.yaml @@ -0,0 +1,254 @@ +--- +# Role: disconnected_mirror_ocp_bastion +# Description: Performs OCP mirroring operations using oc-mirror on the bastion +# This role runs on the bastion to mirror OCP images to the disconnected registry + +- name: Mirror OCP images using oc-mirror on bastion + tags: mirror_ocp_bastion + when: disconnected_enabled + block: + - name: Get user home directory + ansible.builtin.shell: | + set -o pipefail + getent passwd {{ ansible_user }} | awk -F: '{ print $6 }' + changed_when: false + register: user_home + + - name: Set registry URL based on configuration + ansible.builtin.set_fact: + registry_url: >- + {{ + env.bastion.networking.ip ~ ':' ~ disconnected.registry.bastion.port + if disconnected.registry.bastion.enabled | default(false) + else disconnected.registry.url + }} + + - name: Create imageset configuration file + ansible.builtin.template: + src: imageset-config.yaml.j2 + dest: "{{ disconnected.mirroring.bastion.working_dir }}/imageset-config.yaml" + owner: "{{ ansible_user }}" + group: "{{ ansible_user }}" + mode: "0644" + + - name: Copy monitoring script to bastion + ansible.builtin.copy: + src: monitor-oc-mirror.sh + dest: "{{ disconnected.mirroring.bastion.working_dir }}/monitor-oc-mirror.sh" + owner: "{{ ansible_user }}" + group: "{{ ansible_user }}" + mode: "0755" + + - name: Create registries.conf for oc-mirror v2 if needed + when: + - disconnected.mirroring.oc_mirror.image_set.apiVersion == "mirror.openshift.io/v2alpha1" + - disconnected.mirroring.oc_mirror.post_mirror.mapping.replace.enabled + block: + - name: Create .config/containers directory + ansible.builtin.file: + path: "{{ user_home.stdout }}/.config/containers" + state: directory + owner: "{{ ansible_user }}" + group: "{{ ansible_user }}" + mode: "0700" + + - name: Create registries.conf + ansible.builtin.template: + src: registries.conf.j2 + dest: "{{ user_home.stdout }}/.config/containers/registries.conf" + owner: "{{ ansible_user }}" + group: "{{ ansible_user }}" + mode: "0644" + + - name: Mirror images using oc-mirror v1 + when: disconnected.mirroring.oc_mirror.image_set.apiVersion == "mirror.openshift.io/v1alpha2" + block: + - name: Run oc-mirror v1 to mirror images + ansible.builtin.shell: | + set -o pipefail + oc mirror --config {{ disconnected.mirroring.bastion.working_dir }}/imageset-config.yaml \ + docker://{{ registry_url }} \ + --ignore-history{{ ' --continue-on-error' if disconnected.mirroring.oc_mirror.oc_mirror_args.continue_on_error }} \ + {{ ' --source-skip-tls' if disconnected.mirroring.oc_mirror.oc_mirror_args.source_skip_tls }} + args: + chdir: "{{ disconnected.mirroring.bastion.working_dir }}" + register: oc_mirror_v1_output + async: "{{ disconnected.mirroring.oc_mirror.oc_mirror_args.async_timeout | default(7200) }}" + poll: "{{ disconnected.mirroring.oc_mirror.oc_mirror_args.async_poll | default(30) }}" + + - name: Display oc-mirror v1 output + ansible.builtin.debug: + var: oc_mirror_v1_output.stdout_lines + + - name: Get results directory name + ansible.builtin.shell: | + set -o pipefail + ls {{ disconnected.mirroring.bastion.working_dir }}/oc-mirror-workspace/ | grep results + register: oc_mirror_results_dir + changed_when: false + + - name: Copy results to mirror output directory + ansible.builtin.copy: + src: "{{ disconnected.mirroring.bastion.working_dir }}/oc-mirror-workspace/{{ oc_mirror_results_dir.stdout }}/" + dest: "{{ disconnected.mirroring.bastion.mirror_output_dir }}/" + owner: "{{ ansible_user }}" + group: "{{ ansible_user }}" + mode: preserve + remote_src: true + + - name: Mirror images using oc-mirror v2 + when: disconnected.mirroring.oc_mirror.image_set.apiVersion == "mirror.openshift.io/v2alpha1" + block: + - name: Run oc-mirror v2 dry-run to generate mapping + ansible.builtin.shell: | + set -o pipefail + export SSL_CERT_FILE=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem + oc mirror --v2 --config {{ disconnected.mirroring.bastion.working_dir }}/imageset-config.yaml \ + --workspace file://{{ disconnected.mirroring.bastion.working_dir }}/oc-mirror-workspace \ + docker://{{ registry_url }} \ + --dest-tls-verify=false \ + --dry-run + args: + chdir: "{{ disconnected.mirroring.bastion.working_dir }}" + register: oc_mirror_v2_dryrun + changed_when: false + + - name: Display oc-mirror v2 dry-run output + ansible.builtin.debug: + var: oc_mirror_v2_dryrun.stdout_lines + + - name: Run oc-mirror v2 to mirror images (async) + ansible.builtin.shell: | + set -o pipefail + export SSL_CERT_FILE=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem + oc mirror --v2 \ + --config {{ disconnected.mirroring.bastion.working_dir }}/imageset-config.yaml \ + --workspace file://{{ disconnected.mirroring.bastion.working_dir }}/oc-mirror-workspace \ + docker://{{ registry_url }} \ + --dest-tls-verify=false \ + {{ '--continue-on-error' if disconnected.mirroring.oc_mirror.oc_mirror_args.continue_on_error else '' }} \ + {{ '--src-tls-verify=false' if disconnected.mirroring.oc_mirror.oc_mirror_args.source_skip_tls else '' }} \ + 2>&1 | tee {{ disconnected.mirroring.bastion.working_dir }}/oc-mirror-progress.log + args: + chdir: "{{ disconnected.mirroring.bastion.working_dir }}" + register: oc_mirror_v2_job + async: "{{ disconnected.mirroring.oc_mirror.oc_mirror_args.async_timeout | default(7200) }}" + poll: 0 + + - name: Display mirroring progress information + ansible.builtin.debug: + msg: + - "==========================================" + - "OC-Mirror is running in the background" + - "==========================================" + - "Progress log: {{ disconnected.mirroring.bastion.working_dir }}/oc-mirror-progress.log" + - "Job ID: {{ oc_mirror_v2_job.ansible_job_id }}" + - "" + - "To monitor progress in real-time, SSH to bastion and run:" + - " tail -f {{ disconnected.mirroring.bastion.working_dir }}/oc-mirror-progress.log" + - "" + - "Or use this command to see only image pulls:" + - >- + tail -f {{ disconnected.mirroring.bastion.working_dir }}/oc-mirror-progress.log | + grep -E 'mirroring|copying|Pulling|sha256' + - "" + - >- + Ansible will check progress every + {{ disconnected.mirroring.oc_mirror.oc_mirror_args.async_poll | default(30) }} seconds... + + - name: Monitor oc-mirror progress with periodic updates + ansible.builtin.async_status: + jid: "{{ oc_mirror_v2_job.ansible_job_id }}" + register: oc_mirror_v2_result + until: oc_mirror_v2_result.finished + retries: >- + {{ + ( + disconnected.mirroring.oc_mirror.oc_mirror_args.async_timeout | default(7200) / + disconnected.mirroring.oc_mirror.oc_mirror_args.async_poll | default(30) + ) | int + }} + delay: "{{ disconnected.mirroring.oc_mirror.oc_mirror_args.async_poll | default(30) }}" + notify: Display progress update + + - name: Set final output variable + ansible.builtin.set_fact: + oc_mirror_v2_output: "{{ oc_mirror_v2_result }}" + + - name: Display oc-mirror v2 final output + ansible.builtin.debug: + msg: + - "==========================================" + - "OC-Mirror completed successfully!" + - "==========================================" + - >- + Full log available at: + {{ disconnected.mirroring.bastion.working_dir }}/oc-mirror-progress.log + - "" + - "Summary of last 50 lines:" + + - name: Display last 50 lines of mirroring output + ansible.builtin.shell: | + tail -50 {{ disconnected.mirroring.bastion.working_dir }}/oc-mirror-progress.log + register: mirror_summary + changed_when: false + + - name: Show mirroring summary + ansible.builtin.debug: + var: mirror_summary.stdout_lines + + - name: Copy cluster resources to mirror output directory + ansible.builtin.copy: + src: "{{ disconnected.mirroring.bastion.working_dir }}/oc-mirror-workspace/working-dir/cluster-resources/" + dest: "{{ disconnected.mirroring.bastion.mirror_output_dir }}/" + owner: "{{ ansible_user }}" + group: "{{ ansible_user }}" + mode: preserve + remote_src: true + + - name: Perform post-mirror mapping replacements for v1 + when: + - disconnected.mirroring.oc_mirror.image_set.apiVersion == "mirror.openshift.io/v1alpha2" + - disconnected.mirroring.oc_mirror.post_mirror.mapping.replace.enabled + block: + - name: Create copy of mapping file + ansible.builtin.copy: + src: "{{ disconnected.mirroring.bastion.mirror_output_dir }}/mapping.txt" + dest: "{{ disconnected.mirroring.bastion.mirror_output_dir }}/post_mapping.txt" + owner: "{{ ansible_user }}" + group: "{{ ansible_user }}" + mode: preserve + remote_src: true + + - name: Apply replacements to mapping file + ansible.builtin.replace: + path: "{{ disconnected.mirroring.bastion.mirror_output_dir }}/post_mapping.txt" + regexp: "{{ item.regexp }}" + replace: "{{ item.replace }}" + loop: "{{ disconnected.mirroring.oc_mirror.post_mirror.mapping.replace.list }}" + + - name: Mirror patched mappings + ansible.builtin.shell: | + set -o pipefail + oc image mirror -a {{ user_home.stdout }}/.docker/config.json \ + -f {{ disconnected.mirroring.bastion.mirror_output_dir }}/post_mapping.txt \ + --insecure \ + --filter-by-os='.*' \ + --continue-on-error || true + args: + chdir: "{{ disconnected.mirroring.bastion.working_dir }}" + register: oc_image_mirror_output + + - name: Display oc image mirror output + ansible.builtin.debug: + var: oc_image_mirror_output.stdout_lines + + - name: Display mirroring summary + ansible.builtin.debug: + msg: + - "Mirroring completed successfully" + - "Mirror output directory: {{ disconnected.mirroring.bastion.mirror_output_dir }}" + - "ImageSet configuration: {{ disconnected.mirroring.bastion.working_dir }}/imageset-config.yaml" + - "Apply the YAML files in {{ disconnected.mirroring.bastion.mirror_output_dir }} to your cluster" + +# Made with Bob diff --git a/roles/disconnected_mirror_ocp_bastion/templates/imageset-config.yaml.j2 b/roles/disconnected_mirror_ocp_bastion/templates/imageset-config.yaml.j2 new file mode 100644 index 000000000..c957a1a9a --- /dev/null +++ b/roles/disconnected_mirror_ocp_bastion/templates/imageset-config.yaml.j2 @@ -0,0 +1,107 @@ +--- +# ImageSet configuration for oc-mirror +# Generated by Ansible for disconnected OCP mirroring +apiVersion: {{ disconnected.mirroring.oc_mirror.image_set.apiVersion }} +kind: ImageSetConfiguration +{% if disconnected.mirroring.oc_mirror.image_set.storageConfig.enable %} +storageConfig: + registry: + imageURL: {{ registry_url }}/{{ disconnected.mirroring.oc_mirror.image_set.storageConfig.registry.imageURL.org }}/{{ disconnected.mirroring.oc_mirror.image_set.storageConfig.registry.imageURL.repo }} + skipTLS: {{ disconnected.mirroring.oc_mirror.image_set.storageConfig.registry.skipTLS | default(false) | lower }} +{% endif %} +mirror: +{% if disconnected.mirroring.oc_mirror.image_set.mirror.platform is defined %} + platform: + architectures: +{% for arch in disconnected.mirroring.oc_mirror.image_set.mirror.platform.architectures %} + - {{ arch }} +{% endfor %} + channels: +{% for channel in disconnected.mirroring.oc_mirror.image_set.mirror.platform.channels %} + - name: {{ channel.name }} +{% if channel.type is defined %} + type: {{ channel.type }} +{% endif %} +{% if channel.full is defined %} + full: {{ channel.full | lower }} +{% endif %} +{% if channel.minVersion is defined %} + minVersion: '{{ channel.minVersion }}' +{% endif %} +{% if channel.maxVersion is defined %} + maxVersion: '{{ channel.maxVersion }}' +{% endif %} +{% if channel.shortestPath is defined %} + shortestPath: {{ channel.shortestPath | lower }} +{% endif %} +{% endfor %} +{% if disconnected.mirroring.oc_mirror.image_set.mirror.platform.graph is defined %} + graph: {{ disconnected.mirroring.oc_mirror.image_set.mirror.platform.graph | lower }} +{% endif %} +{% endif %} +{% if disconnected.mirroring.oc_mirror.image_set.mirror.operators is defined and disconnected.mirroring.oc_mirror.image_set.mirror.operators | length > 0 %} + operators: +{% for operator in disconnected.mirroring.oc_mirror.image_set.mirror.operators %} + - catalog: {{ operator.catalog }} +{% if operator.full is defined %} + full: {{ operator.full | lower }} +{% endif %} +{% if operator.packages is defined and operator.packages | length > 0 %} + packages: +{% for package in operator.packages %} + - name: {{ package.name }} +{% if package.channels is defined and package.channels | length > 0 %} + channels: +{% for channel in package.channels %} + - name: {{ channel.name }} +{% if channel.minVersion is defined %} + minVersion: '{{ channel.minVersion }}' +{% endif %} +{% if channel.maxVersion is defined %} + maxVersion: '{{ channel.maxVersion }}' +{% endif %} +{% if channel.minBundle is defined %} + minBundle: '{{ channel.minBundle }}' +{% endif %} +{% if channel.maxBundle is defined %} + maxBundle: '{{ channel.maxBundle }}' +{% endif %} +{% endfor %} +{% endif %} +{% if package.minVersion is defined %} + minVersion: '{{ package.minVersion }}' +{% endif %} +{% if package.maxVersion is defined %} + maxVersion: '{{ package.maxVersion }}' +{% endif %} +{% endfor %} +{% endif %} +{% if operator.targetName is defined %} + targetName: {{ operator.targetName }} +{% endif %} +{% if operator.targetTag is defined %} + targetTag: {{ operator.targetTag }} +{% endif %} +{% endfor %} +{% endif %} +{% if disconnected.mirroring.oc_mirror.image_set.mirror.additionalImages is defined and disconnected.mirroring.oc_mirror.image_set.mirror.additionalImages | length > 0 %} + additionalImages: +{% for image in disconnected.mirroring.oc_mirror.image_set.mirror.additionalImages %} + - name: {{ image.name }} +{% endfor %} +{% endif %} +{% if disconnected.mirroring.oc_mirror.image_set.mirror.helm is defined %} + helm: {{ disconnected.mirroring.oc_mirror.image_set.mirror.helm | to_nice_yaml(indent=2) | indent(4) }} +{% endif %} +{% if disconnected.mirroring.oc_mirror.image_set.mirror.blockedImages is defined and disconnected.mirroring.oc_mirror.image_set.mirror.blockedImages | length > 0 %} + blockedImages: +{% for image in disconnected.mirroring.oc_mirror.image_set.mirror.blockedImages %} + - name: {{ image.name }} +{% endfor %} +{% endif %} +{% if disconnected.mirroring.oc_mirror.image_set.mirror.samples is defined %} + samples: +{% for sample in disconnected.mirroring.oc_mirror.image_set.mirror.samples %} + - image: {{ sample.image }} +{% endfor %} +{% endif %} \ No newline at end of file diff --git a/roles/disconnected_mirror_ocp_bastion/templates/registries.conf.j2 b/roles/disconnected_mirror_ocp_bastion/templates/registries.conf.j2 new file mode 100644 index 000000000..69e82c9d9 --- /dev/null +++ b/roles/disconnected_mirror_ocp_bastion/templates/registries.conf.j2 @@ -0,0 +1,22 @@ +[[registry]] + prefix = "" + location = "{{ registry_url }}" + insecure = false + blocked = false + mirror-by-digest-only = true + +{% if disconnected.mirroring.oc_mirror.post_mirror.mapping.replace.enabled %} +{% for item in disconnected.mirroring.oc_mirror.post_mirror.mapping.replace.list %} +[[registry]] + prefix = "" + location = "{{ item.regexp }}" + insecure = false + blocked = false + mirror-by-digest-only = true + + [[registry.mirror]] + location = "{{ item.replace }}" + insecure = false + +{% endfor %} +{% endif %} \ No newline at end of file diff --git a/roles/disconnected_setup_oc_mirror_bastion/tasks/main.yaml b/roles/disconnected_setup_oc_mirror_bastion/tasks/main.yaml new file mode 100644 index 000000000..688da8ba5 --- /dev/null +++ b/roles/disconnected_setup_oc_mirror_bastion/tasks/main.yaml @@ -0,0 +1,236 @@ +--- +# Role: disconnected_setup_oc_mirror_bastion +# Description: Installs and configures oc-mirror on the bastion for mirroring operations +# This role runs on the bastion to set up the mirroring environment + +- name: Setup oc-mirror on bastion for mirroring + tags: setup_oc_mirror_bastion + when: disconnected_enabled + block: + - name: Get user home directory + ansible.builtin.shell: | + set -o pipefail + getent passwd {{ ansible_user }} | awk -F: '{ print $6 }' + changed_when: false + register: user_home + + - name: Create oc-mirror working directory on bastion + ansible.builtin.file: + path: "{{ disconnected.mirroring.bastion.working_dir }}" + state: directory + owner: "{{ ansible_user }}" + group: "{{ ansible_user }}" + mode: "0755" + + - name: Create mirror output directory on bastion + ansible.builtin.file: + path: "{{ disconnected.mirroring.bastion.mirror_output_dir }}" + state: directory + owner: "{{ ansible_user }}" + group: "{{ ansible_user }}" + mode: "0755" + + - name: Download oc-mirror from file server + ansible.builtin.get_url: + url: >- + {{ env.file_server.protocol }}://{{ + (env.file_server.user ~ ':' ~ env.file_server.pass ~ '@' if env.file_server.protocol == 'ftp' else '') ~ + env.file_server.ip ~ (':' ~ env.file_server.port if env.file_server.port | default('') | length > 0 else '') ~ + '/' ~ disconnected.mirroring.file_server.clients_dir ~ '/' ~ disconnected.mirroring.file_server.oc_mirror_tgz }} + dest: "{{ disconnected.mirroring.bastion.working_dir }}/{{ disconnected.mirroring.file_server.oc_mirror_tgz }}" + mode: "0644" + timeout: 300 + register: oc_mirror_download + retries: 3 + delay: 10 + until: oc_mirror_download is succeeded + + - name: Download OCP client from file server + ansible.builtin.get_url: + url: >- + {{ env.file_server.protocol }}://{{ + (env.file_server.user ~ ':' ~ env.file_server.pass ~ '@' if env.file_server.protocol == 'ftp' else '') ~ + env.file_server.ip ~ (':' ~ env.file_server.port if env.file_server.port | default('') | length > 0 else '') ~ + '/' ~ disconnected.mirroring.file_server.clients_dir ~ '/' ~ disconnected.mirroring.client_download.ocp_client_tgz }} + dest: "{{ disconnected.mirroring.bastion.working_dir }}/{{ disconnected.mirroring.client_download.ocp_client_tgz }}" + mode: "0644" + timeout: 300 + register: ocp_client_download + retries: 3 + delay: 10 + until: ocp_client_download is succeeded + + - name: Extract oc-mirror binary + ansible.builtin.unarchive: + src: "{{ disconnected.mirroring.bastion.working_dir }}/{{ disconnected.mirroring.file_server.oc_mirror_tgz }}" + dest: "{{ disconnected.mirroring.bastion.working_dir }}" + remote_src: true + mode: "0755" + + - name: Extract OCP client binaries + ansible.builtin.unarchive: + src: "{{ disconnected.mirroring.bastion.working_dir }}/{{ disconnected.mirroring.client_download.ocp_client_tgz }}" + dest: "{{ disconnected.mirroring.bastion.working_dir }}" + remote_src: true + mode: "0755" + + - name: Install oc-mirror to system path + become: true + ansible.builtin.copy: + src: "{{ disconnected.mirroring.bastion.working_dir }}/oc-mirror" + dest: /usr/local/bin/oc-mirror + owner: root + group: root + mode: "0755" + remote_src: true + + - name: Install oc to system path + become: true + ansible.builtin.copy: + src: "{{ disconnected.mirroring.bastion.working_dir }}/oc" + dest: /usr/local/bin/oc + owner: root + group: root + mode: "0755" + remote_src: true + + - name: Install kubectl to system path + become: true + ansible.builtin.copy: + src: "{{ disconnected.mirroring.bastion.working_dir }}/kubectl" + dest: /usr/local/bin/kubectl + owner: root + group: root + mode: "0755" + remote_src: true + + - name: Create .docker directory for pull secrets + ansible.builtin.file: + path: "{{ user_home.stdout }}/.docker" + state: directory + owner: "{{ ansible_user }}" + group: "{{ ansible_user }}" + mode: "0700" + + - name: Create pull secret file for mirroring + ansible.builtin.copy: + content: "{{ env.redhat.pull_secret }}" + dest: "{{ user_home.stdout }}/.docker/config.json" + owner: "{{ ansible_user }}" + group: "{{ ansible_user }}" + mode: "0600" + + - name: Setup CA certificate for registry access + become: true + block: + - name: Check if certificate exists in disconnected config + ansible.builtin.set_fact: + has_ca_cert: "{{ (disconnected.registry.ca_cert | default('') | trim | length > 0) }}" + + - name: Debug certificate availability + ansible.builtin.debug: + msg: + - "Certificate in config: {{ has_ca_cert }}" + - "Certificate length: {{ disconnected.registry.ca_cert | default('') | length }}" + - "Bastion registry enabled: {{ disconnected.registry.bastion.enabled | default(false) }}" + + - name: Read certificate from bastion registry if not in config + ansible.builtin.slurp: + src: "{{ disconnected.registry.bastion.certs_dir }}/registry.crt" + register: registry_cert_from_file + when: + - not has_ca_cert + - disconnected.registry.bastion.enabled | default(false) + + - name: Set certificate content from config or file + ansible.builtin.set_fact: + registry_cert_content: "{{ disconnected.registry.ca_cert if has_ca_cert else (registry_cert_from_file.content | b64decode) }}" + when: has_ca_cert or (registry_cert_from_file is defined and registry_cert_from_file.content is defined) + + - name: Fail if no certificate available + ansible.builtin.fail: + msg: "No certificate available - neither in config nor in registry certs directory" + when: registry_cert_content is not defined + + - name: Copy CA certificate to trust anchors + ansible.builtin.copy: + content: "{{ registry_cert_content }}" + dest: /etc/pki/ca-trust/source/anchors/registry.crt + owner: root + group: root + mode: "0644" + + - name: Update CA trust + ansible.builtin.command: update-ca-trust + changed_when: true + + - name: Get registry URL for certificate directory + ansible.builtin.set_fact: + registry_host: "{{ env.bastion.networking.ip if disconnected.registry.bastion.enabled | default(false) else disconnected.registry.ip }}" + registry_port: "{{ disconnected.registry.bastion.port if disconnected.registry.bastion.enabled | default(false) else disconnected.registry.url.split(':')[-1] }}" + + - name: Create containers certs.d directory for registry + ansible.builtin.file: + path: "/etc/containers/certs.d/{{ registry_host }}:{{ registry_port }}" + state: directory + owner: root + group: root + mode: "0755" + + - name: Copy certificate to containers certs.d + ansible.builtin.copy: + content: "{{ registry_cert_content }}" + dest: "/etc/containers/certs.d/{{ registry_host }}:{{ registry_port }}/ca.crt" + owner: root + group: root + mode: "0644" + + - name: Verify certificate installation + ansible.builtin.debug: + msg: + - "Certificate installed to: /etc/pki/ca-trust/source/anchors/registry.crt" + - "Certificate installed to: /etc/containers/certs.d/{{ registry_host }}:{{ registry_port }}/ca.crt" + - "CA trust updated" + + - name: Test certificate with curl + ansible.builtin.command: > + curl -v https://{{ registry_host }}:{{ registry_port }}/v2/ + register: curl_test + failed_when: false + changed_when: false + + - name: Display curl test result + ansible.builtin.debug: + msg: + - "Curl test result: {{ curl_test.rc }}" + - "Curl stderr: {{ curl_test.stderr }}" + + - name: Test registry with podman + ansible.builtin.command: > + podman login --get-login {{ registry_host }}:{{ registry_port }} + register: podman_test + failed_when: false + changed_when: false + + - name: Display podman test result + ansible.builtin.debug: + var: podman_test + + - name: Verify oc-mirror installation + ansible.builtin.command: oc-mirror version + register: oc_mirror_version + changed_when: false + + - name: Display oc-mirror version + ansible.builtin.debug: + msg: "oc-mirror installed successfully: {{ oc_mirror_version.stdout }}" + + - name: Display setup summary + ansible.builtin.debug: + msg: + - "oc-mirror working directory: {{ disconnected.mirroring.bastion.working_dir }}" + - "Mirror output directory: {{ disconnected.mirroring.bastion.mirror_output_dir }}" + - "oc-mirror binary: /usr/local/bin/oc-mirror" + - "Pull secret configured: {{ user_home.stdout }}/.docker/config.json" + +# Assisted by Bob diff --git a/roles/disconnected_setup_registry_bastion/README.md b/roles/disconnected_setup_registry_bastion/README.md new file mode 100644 index 000000000..f359180e4 --- /dev/null +++ b/roles/disconnected_setup_registry_bastion/README.md @@ -0,0 +1,251 @@ +# Role: disconnected_setup_registry_bastion + +## Description + +This role sets up a container registry on the bastion host for disconnected OpenShift installations. It installs podman, creates a self-signed certificate (or uses a provided one), configures authentication, and runs a container registry as a systemd service. + +## Requirements + +- RHEL 8/9 or compatible Linux distribution +- Root/sudo access +- Sufficient disk space for container images (100GB+ recommended) +- Podman, httpd-tools, and openssl packages (either from repo or pre-downloaded RPMs) + +## Role Variables + +All variables are defined in `inventories/default/group_vars/disconnected.yaml`: + +### Required Variables + +```yaml +disconnected: + enabled: true + registry: + ca_trusted: false # Set to true if using provided certificate for the bastion host + ca_cert: | + -----BEGIN CERTIFICATE----- + ... + -----END CERTIFICATE----- + bastion: + enabled: true # Enable registry on bastion + port: 5000 # Registry port + username: 'admin' # Registry username + password: 'redhat123' # Registry password (CHANGE THIS!) + data_dir: '/opt/registry/data' # Registry data storage + auth_dir: '/opt/registry/auth' # Registry authentication files + certs_dir: '/opt/registry/certs' # Registry certificates + # Package installation options + use_local_repo: true # Set to false for fully disconnected (use downloaded RPMs) + rpm_dir: '/tmp/registry-rpms' # Directory for RPMs (if use_local_repo=false) + rpm_path: 'rpms/registry' # Path on file server (if use_local_repo=false) + required_rpms: # RPM files on file server (if use_local_repo=false) + - 'podman-*.rpm' + - 'httpd-tools-*.rpm' + - 'openssl-*.rpm' + +env: + bastion: + networking: + hostname: 'bastion' + base_domain: 'example.com' + ip: '192.168.1.100' +``` + +## Dependencies + +- `community.crypto` collection (for certificate generation) +- `community.general` collection (for htpasswd module) + +Install with: +```bash +ansible-galaxy collection install community.crypto community.general +``` + +## Example Playbook + +```yaml +- name: Setup container registry on bastion + hosts: bastion + gather_facts: true + vars_files: + - "{{ inventory_dir }}/group_vars/all.yaml" + - "{{ inventory_dir }}/group_vars/disconnected.yaml" + tasks: + - name: Setup container registry + ansible.builtin.include_role: + name: disconnected_setup_registry_bastion + when: + - disconnected.enabled + - disconnected.registry.bastion.enabled +``` + +## Package Installation Modes + +### Mode 1: Using Local Repository (use_local_repo: true) +- Installs packages from configured yum/dnf repositories +- Requires bastion to have access to RHEL repositories or local mirror +- Simplest method if repositories are available + +### Mode 2: Using Downloaded RPMs (use_local_repo: false) +- Downloads RPMs from file server +- For fully disconnected environments +- Requires RPMs to be pre-downloaded to file server using `disconnected_download_registry_rpms` role + +## Tasks Overview + +1. Checks package installation mode +2. Installs required packages (podman, httpd-tools, openssl) from repo OR downloaded RPMs +2. Creates registry directories for data, auth, and certificates +3. Generates self-signed certificate (if ca_trusted=false) or uses provided certificate +4. Adds certificate to system trust anchors +5. Creates htpasswd file for registry authentication +6. Creates systemd service for container registry +7. Starts and enables the registry service +8. Verifies registry is accessible + +## Files Created + +### Directories +- `{{ disconnected.registry.bastion.data_dir }}` - Registry image storage +- `{{ disconnected.registry.bastion.auth_dir }}` - Authentication files +- `{{ disconnected.registry.bastion.certs_dir }}` - TLS certificates + +### Files +- `{{ disconnected.registry.bastion.certs_dir }}/registry.crt` - Registry certificate +- `{{ disconnected.registry.bastion.certs_dir }}/registry.key` - Registry private key +- `{{ disconnected.registry.bastion.auth_dir }}/htpasswd` - Authentication credentials +- `/etc/systemd/system/container-registry.service` - Systemd service file +- `/etc/pki/ca-trust/source/anchors/registry.crt` - System-trusted certificate + +## Registry Service + +The registry runs as a systemd service named `container-registry`: + +```bash +# Check status +systemctl status container-registry + +# View logs +journalctl -u container-registry -f + +# Restart registry +systemctl restart container-registry + +# Stop registry +systemctl stop container-registry +``` + +## Registry Configuration + +The registry is configured with: +- **TLS**: Enabled with self-signed or provided certificate +- **Authentication**: Basic auth using htpasswd +- **Storage**: Local filesystem storage +- **Delete**: Image deletion enabled +- **Port**: Configurable (default: 5000) + +## Testing the Registry + +After installation, test the registry: + +```bash +# Login to registry +podman login {{ env.bastion.networking.ip }}:{{ disconnected.registry.bastion.port }} \ + -u {{ disconnected.registry.bastion.username }} \ + -p {{ disconnected.registry.bastion.password }} + +# Pull a test image +podman pull docker.io/library/hello-world:latest + +# Tag for local registry +podman tag docker.io/library/hello-world:latest \ + {{ env.bastion.networking.ip }}:{{ disconnected.registry.bastion.port }}/hello-world:latest + +# Push to local registry +podman push {{ env.bastion.networking.ip }}:{{ disconnected.registry.bastion.port }}/hello-world:latest + +# Verify +curl -u {{ disconnected.registry.bastion.username }}:{{ disconnected.registry.bastion.password }} \ + https://{{ env.bastion.networking.ip }}:{{ disconnected.registry.bastion.port }}/v2/_catalog +``` + +## Pull Secret Configuration + +After the registry is created, you need to add its credentials to your pull secret. The role displays the required authentication string in base64 format. + +Add to `env.redhat.pull_secret` in all.yaml: + +```json +{ + "auths": { + "cloud.openshift.com": {...}, + "quay.io": {...}, + "registry.redhat.io": {...}, + "192.168.1.100:5000": { + "auth": "base64-encoded-username:password", + "email": "registry@example.com" + } + } +} +``` + +To generate the auth string: +```bash +echo -n "admin:redhat123" | base64 +``` + +## Security Considerations + +1. **Change Default Password**: Always change the default registry password +2. **Certificate Management**: Use proper certificates in production +3. **Firewall Rules**: Ensure port 5000 (or configured port) is accessible +4. **Disk Space**: Monitor disk usage in data_dir +5. **Backup**: Regularly backup registry data directory + +## Storage Requirements + +Plan for adequate storage based on your needs: +- **Minimal** (single OCP version): 50-100 GB +- **Standard** (OCP + operators): 200-300 GB +- **Full** (complete catalog): 500GB-1TB + +## Troubleshooting + +### Issue: Registry service fails to start +**Solution**: Check logs with `journalctl -u container-registry -f` +- Verify port is not in use: `ss -tlnp | grep 5000` +- Check podman is installed: `podman --version` +- Verify directories exist and have correct permissions + +### Issue: Certificate errors when accessing registry +**Solution**: +- Verify certificate is in trust anchors: `ls /etc/pki/ca-trust/source/anchors/` +- Update trust: `update-ca-trust` +- Check certificate validity: `openssl x509 -in /opt/registry/certs/registry.crt -text -noout` + +### Issue: Authentication failures +**Solution**: +- Verify htpasswd file exists: `cat /opt/registry/auth/htpasswd` +- Test credentials: `htpasswd -v /opt/registry/auth/htpasswd admin` +- Check service environment variables in systemd file + +### Issue: Out of disk space +**Solution**: +- Check disk usage: `df -h /opt/registry/data` +- Clean up old images if needed +- Consider using different mount point with more space + +## Tags + +- `setup_registry_bastion` +- `registry` + +## References + +- [Podman Documentation](https://docs.podman.io/) +- [Docker Registry Documentation](https://docs.docker.com/registry/) +- [Red Hat Container Registry](https://access.redhat.com/documentation/en-us/red_hat_quay/) + +## Author + +Generated for Ansible-OpenShift-Provisioning project \ No newline at end of file diff --git a/roles/disconnected_setup_registry_bastion/handlers/main.yaml b/roles/disconnected_setup_registry_bastion/handlers/main.yaml new file mode 100644 index 000000000..c4fc2d395 --- /dev/null +++ b/roles/disconnected_setup_registry_bastion/handlers/main.yaml @@ -0,0 +1,11 @@ +--- +# Handlers for disconnected_setup_registry_bastion role + +- name: Restart container registry + become: true + ansible.builtin.systemd: + name: container-registry + state: restarted + daemon_reload: true + +# Made with Bob diff --git a/roles/disconnected_setup_registry_bastion/tasks/main.yaml b/roles/disconnected_setup_registry_bastion/tasks/main.yaml new file mode 100644 index 000000000..2672f1a89 --- /dev/null +++ b/roles/disconnected_setup_registry_bastion/tasks/main.yaml @@ -0,0 +1,323 @@ +--- +# Role: disconnected_setup_registry_bastion +# Description: Sets up a container registry on the bastion host for disconnected OCP installations +# This role installs and configures podman and creates a local container registry + +- name: Setup container registry on bastion + tags: setup_registry_bastion + when: disconnected_enabled + block: + - name: Check if packages need to be downloaded + ansible.builtin.set_fact: + use_local_repo: "{{ disconnected.registry.bastion.use_local_repo | default(true) }}" + + - name: Install required packages from configured repositories + become: true + when: use_local_repo + ansible.builtin.package: + name: + - podman + - httpd-tools + - openssl + - python3-cryptography # Required for community.crypto Ansible modules + - python3-pip # Required to install passlib + state: present + + - name: Install passlib Python library via pip + become: true + when: use_local_repo + ansible.builtin.pip: + name: passlib + state: present + executable: pip3 + + - name: Install packages from downloaded RPMs (if not using repo) + become: true + when: not use_local_repo + block: + - name: Create RPM download directory + ansible.builtin.file: + path: "{{ disconnected.registry.bastion.rpm_dir }}" + state: directory + owner: root + group: root + mode: "0755" + + - name: Download RPMs from file server + ansible.builtin.get_url: + url: >- + {{ env.file_server.protocol }}://{{ + (env.file_server.user ~ ':' ~ env.file_server.pass ~ '@' if env.file_server.protocol == 'ftp' else '') ~ + env.file_server.ip ~ (':' ~ env.file_server.port if env.file_server.port | default('') | length > 0 else '') ~ + '/' ~ disconnected.registry.bastion.rpm_path ~ '/' ~ item }} + dest: "{{ disconnected.registry.bastion.rpm_dir }}/{{ item }}" + mode: "0644" + loop: "{{ disconnected.registry.bastion.required_rpms }}" + register: rpm_downloads + retries: 3 + delay: 5 + + - name: Install RPMs from local directory + ansible.builtin.yum: + name: "{{ disconnected.registry.bastion.rpm_dir }}/*.rpm" + state: present + disable_gpg_check: true + + - name: Create registry directories + become: true + ansible.builtin.file: + path: "{{ item }}" + state: directory + owner: root + group: root + mode: "0755" + loop: + - "{{ disconnected.registry.bastion.data_dir }}" + - "{{ disconnected.registry.bastion.auth_dir }}" + - "{{ disconnected.registry.bastion.certs_dir }}" + + - name: Open firewall port for registry + become: true + ansible.posix.firewalld: + port: "{{ disconnected.registry.bastion.port }}/tcp" + permanent: true + state: enabled + immediate: true + when: ansible_facts.services['firewalld.service'] is defined and ansible_facts.services['firewalld.service'].state == 'running' + + - name: Generate self-signed certificate for registry + become: true + when: not disconnected.registry.ca_trusted + block: + - name: Generate private key + community.crypto.openssl_privatekey: + path: "{{ disconnected.registry.bastion.certs_dir }}/registry.key" + size: 4096 + mode: "0600" + + - name: Generate certificate signing request + community.crypto.openssl_csr: + path: "{{ disconnected.registry.bastion.certs_dir }}/registry.csr" + privatekey_path: "{{ disconnected.registry.bastion.certs_dir }}/registry.key" + common_name: "{{ env.bastion.networking.hostname }}" + basic_constraints: + - "CA:TRUE" + basic_constraints_critical: true + key_usage: + - digitalSignature + - keyEncipherment + - keyCertSign + - cRLSign + key_usage_critical: true + subject_alt_name: + - "DNS:{{ env.bastion.networking.hostname }}" + - "DNS:{{ env.bastion.networking.hostname }}.{{ env.bastion.networking.base_domain }}" + - "IP:{{ env.bastion.networking.ip }}" + + - name: Generate self-signed certificate + community.crypto.x509_certificate: + path: "{{ disconnected.registry.bastion.certs_dir }}/registry.crt" + privatekey_path: "{{ disconnected.registry.bastion.certs_dir }}/registry.key" + csr_path: "{{ disconnected.registry.bastion.certs_dir }}/registry.csr" + provider: selfsigned + selfsigned_not_after: "+3650d" + selfsigned_create_subject_key_identifier: always_create + mode: "0644" + + - name: Copy certificate to system trust anchors + ansible.builtin.copy: + src: "{{ disconnected.registry.bastion.certs_dir }}/registry.crt" + dest: /etc/pki/ca-trust/source/anchors/registry.crt + owner: root + group: root + mode: "0644" + remote_src: true + + - name: Read generated certificate + ansible.builtin.slurp: + src: "{{ disconnected.registry.bastion.certs_dir }}/registry.crt" + register: registry_cert_content + + - name: Update CA trust + ansible.builtin.command: update-ca-trust + changed_when: true + + - name: Create container registry certificate directory + become: true + ansible.builtin.file: + path: "/etc/containers/certs.d/{{ env.bastion.networking.ip }}:{{ disconnected.registry.bastion.port }}" + state: directory + owner: root + group: root + mode: "0755" + + - name: Trust generated registry certificate for containers + become: true + ansible.builtin.copy: + src: "{{ disconnected.registry.bastion.certs_dir }}/registry.crt" + dest: "/etc/containers/certs.d/{{ env.bastion.networking.ip }}:{{ disconnected.registry.bastion.port }}/ca.crt" + owner: root + group: root + mode: "0644" + remote_src: true + + - name: Update disconnected.registry.ca_cert with generated certificate + ansible.builtin.set_fact: + disconnected: + "{{ disconnected | combine({ + 'registry': disconnected.registry | combine({ + 'ca_cert': registry_cert_content.content | b64decode + }, recursive=True) + }, recursive=True) }}" + + - name: Replace ca_cert block in disconnected.yaml with generated certificate + ansible.builtin.lineinfile: + path: "{{ inventory_dir }}/group_vars/disconnected.yaml" + insertafter: " ca_cert" + backup: true + line: "{{ (registry_cert_content.content | b64decode).split('\n') | reject('equalto', '') | map('regex_replace', '^(.*)$', ' \\1') | join('\n') ~ '\n' }}" + delegate_to: localhost + run_once: true + + - name: Use provided certificate if CA is trusted + become: true + when: disconnected.registry.ca_trusted + block: + - name: Create certificate from provided CA cert + ansible.builtin.copy: + content: "{{ disconnected.registry.ca_cert }}" + dest: "{{ disconnected.registry.bastion.certs_dir }}/registry.crt" + owner: root + group: root + mode: "0644" + + - name: Create private key placeholder (using provided cert) + ansible.builtin.copy: + content: "# Using provided certificate" + dest: "{{ disconnected.registry.bastion.certs_dir }}/registry.key" + owner: root + group: root + mode: "0600" + + - name: Create htpasswd file for registry authentication + become: true + ansible.builtin.shell: | + htpasswd -nbB {{ disconnected.registry.bastion.username }} {{ disconnected.registry.bastion.password }} > {{ disconnected.registry.bastion.auth_dir }}/htpasswd + chown root:root {{ disconnected.registry.bastion.auth_dir }}/htpasswd + chmod 0640 {{ disconnected.registry.bastion.auth_dir }}/htpasswd + args: + creates: "{{ disconnected.registry.bastion.auth_dir }}/htpasswd" + no_log: true # Don't log password + + - name: Create registry systemd service file + become: true + ansible.builtin.template: + src: registry.service.j2 + dest: /etc/systemd/system/container-registry.service + owner: root + group: root + mode: "0644" + notify: Restart container registry + + - name: Enable and start container registry service + become: true + ansible.builtin.systemd: + name: container-registry + enabled: true + state: started + daemon_reload: true + + - name: Wait for registry to be ready + ansible.builtin.wait_for: + host: "{{ env.bastion.networking.ip }}" + port: "{{ disconnected.registry.bastion.port }}" + delay: 5 + timeout: 60 + + - name: Verify htpasswd file was created + ansible.builtin.stat: + path: "{{ disconnected.registry.bastion.auth_dir }}/htpasswd" + register: htpasswd_file + + - name: Display htpasswd file status + ansible.builtin.debug: + msg: "htpasswd file exists: {{ htpasswd_file.stat.exists }}, size: {{ htpasswd_file.stat.size | default('N/A') }}" + + - name: Test registry connectivity (allow 401 as it means auth is working) + ansible.builtin.uri: + url: "https://{{ env.bastion.networking.ip }}:{{ disconnected.registry.bastion.port }}/v2/" + method: GET + user: "{{ disconnected.registry.bastion.username }}" + password: "{{ disconnected.registry.bastion.password }}" + force_basic_auth: true + validate_certs: false + status_code: [200, 401] # 401 means registry is up and requiring auth + register: registry_test + retries: 5 + delay: 10 + until: registry_test.status in [200, 401] + + - name: Restart registry service to ensure htpasswd is loaded + become: true + ansible.builtin.systemd: + name: container-registry + state: restarted + daemon_reload: true + when: registry_test.status == 401 + + - name: Wait after restart + ansible.builtin.pause: + seconds: 15 + when: registry_test.status == 401 + + - name: Test registry connectivity again after restart + ansible.builtin.uri: + url: "https://{{ env.bastion.networking.ip }}:{{ disconnected.registry.bastion.port }}/v2/" + method: GET + user: "{{ disconnected.registry.bastion.username }}" + password: "{{ disconnected.registry.bastion.password }}" + force_basic_auth: true + validate_certs: false + status_code: 200 + register: registry_test_final + retries: 3 + delay: 5 + until: registry_test_final.status == 200 + when: registry_test.status == 401 + failed_when: false # Don't fail the playbook if auth still doesn't work + + - name: Display authentication test result + ansible.builtin.debug: + msg: + - "Registry authentication test: {{ 'PASSED' if (registry_test_final.status | default(registry_test.status)) == 200 else 'FAILED (continuing anyway)' }}" + - "Status code: {{ registry_test_final.status | default(registry_test.status) }}" + - "Registry is running and accessible at: https://{{ env.bastion.networking.ip }}:{{ disconnected.registry.bastion.port }}" + - "" + - "If authentication failed, you can manually test with:" + - " podman login --username {{ disconnected.registry.bastion.username }} --password '{{ disconnected.registry.bastion.password }}' {{ env.bastion.networking.ip }}:{{ disconnected.registry.bastion.port }}" + - "" + - "Or check the htpasswd file:" + - " cat {{ disconnected.registry.bastion.auth_dir }}/htpasswd" + - " podman logs registry" + + - name: Create registry pull secret + ansible.builtin.set_fact: + registry_auth: "{{ (disconnected.registry.bastion.username + ':' + disconnected.registry.bastion.password) | b64encode }}" + + - name: Display registry information + ansible.builtin.debug: + msg: + - "Container registry successfully configured!" + - "Registry URL: {{ env.bastion.networking.ip }}:{{ disconnected.registry.bastion.port }}" + - "Registry hostname: {{ env.bastion.networking.hostname }}:{{ disconnected.registry.bastion.port }}" + - "Username: {{ disconnected.registry.bastion.username }}" + - "Data directory: {{ disconnected.registry.bastion.data_dir }}" + - "Certificate: {{ disconnected.registry.bastion.certs_dir }}/registry.crt" + - "" + - "Add this to your pull secret:" + - " '{{ env.bastion.networking.ip }}:{{ disconnected.registry.bastion.port }}': {" + - " 'auth': '{{ registry_auth }}'," + - " 'email': 'registry@example.com'" + - " }" + +# Assisted with Bob diff --git a/roles/disconnected_setup_registry_bastion/templates/registry.service.j2 b/roles/disconnected_setup_registry_bastion/templates/registry.service.j2 new file mode 100644 index 000000000..99f3624ba --- /dev/null +++ b/roles/disconnected_setup_registry_bastion/templates/registry.service.j2 @@ -0,0 +1,33 @@ +[Unit] +Description=Container Registry for Disconnected OpenShift +After=network-online.target +Wants=network-online.target + +[Service] +Type=simple +TimeoutStartSec=5m +ExecStartPre=-/usr/bin/podman rm -f registry +ExecStart=/usr/bin/podman run \ + --name registry \ + --net host \ + --privileged \ + -v {{ disconnected.registry.bastion.data_dir }}:/var/lib/registry:z \ + -v {{ disconnected.registry.bastion.auth_dir }}:/auth:z \ + -v {{ disconnected.registry.bastion.certs_dir }}:/certs:z \ + -e REGISTRY_AUTH=htpasswd \ + -e REGISTRY_AUTH_HTPASSWD_REALM="Registry Realm" \ + -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \ + -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/registry.crt \ + -e REGISTRY_HTTP_TLS_KEY=/certs/registry.key \ + -e REGISTRY_HTTP_ADDR=0.0.0.0:{{ disconnected.registry.bastion.port }} \ + -e REGISTRY_STORAGE_DELETE_ENABLED=true \ + docker.io/library/registry:2 + +ExecStop=/usr/bin/podman stop registry +ExecStopPost=/usr/bin/podman rm -f registry + +Restart=always +RestartSec=10 + +[Install] +WantedBy=multi-user.target \ No newline at end of file diff --git a/roles/disconnected_update_pull_secret/tasks/main.yaml b/roles/disconnected_update_pull_secret/tasks/main.yaml new file mode 100644 index 000000000..2b18106ac --- /dev/null +++ b/roles/disconnected_update_pull_secret/tasks/main.yaml @@ -0,0 +1,133 @@ +--- +# Role: disconnected_update_pull_secret +# Description: Automatically updates the pull secret in all.yaml with bastion registry credentials +# This role modifies the pull secret to include authentication for the bastion registry + +- name: Update pull secret with bastion registry credentials + tags: update_pull_secret + when: + - disconnected_enabled + - disconnected.registry.bastion.enabled | default(false) + delegate_to: localhost + run_once: true + block: + - name: Parse existing pull secret + ansible.builtin.set_fact: + pull_secret_json: "{{ env.redhat.pull_secret | from_json }}" + + - name: Generate registry authentication string + ansible.builtin.set_fact: + registry_auth: "{{ (disconnected.registry.bastion.username + ':' + disconnected.registry.bastion.password) | b64encode }}" + + - name: Set registry URL + ansible.builtin.set_fact: + registry_url: "{{ env.bastion.networking.ip }}:{{ disconnected.registry.bastion.port }}" + + - name: Check if bastion registry already exists in pull secret + ansible.builtin.set_fact: + registry_exists: "{{ registry_url in pull_secret_json.auths.keys() }}" + + - name: Add bastion registry to pull secret + when: not registry_exists + block: + - name: Create updated pull secret with bastion registry + ansible.builtin.set_fact: + updated_pull_secret: >- + {{ + pull_secret_json | combine({ + 'auths': pull_secret_json.auths | combine({ + registry_url: { + 'auth': registry_auth, + 'email': disconnected.registry.bastion.email + } + }) + }) + }} + + - name: Display registry credentials being added + ansible.builtin.debug: + msg: + - "Adding bastion registry to pull secret:" + - " Registry: {{ registry_url }}" + - " Username: {{ disconnected.registry.bastion.username }}" + - " Email: {{ disconnected.registry.bastion.email }}" + - "" + - "The pull secret will be updated in memory for this playbook run." + - "To persist this change, add the following to env.redhat.pull_secret in all.yaml:" + - "" + - " '{{ registry_url }}': {" + - " 'auth': '{{ registry_auth }}'," + - " 'email': '{{ disconnected.registry.bastion.email }}'" + - " }" + + - name: Update pull secret in memory for current playbook run + ansible.builtin.set_fact: + env: "{{ env | combine({'redhat': env.redhat | combine({'pull_secret': updated_pull_secret | to_json})}, recursive=True) }}" + + - name: Create pull secret backup file + ansible.builtin.copy: + content: "{{ updated_pull_secret | to_nice_json }}" + dest: "{{ inventory_dir }}/group_vars/pull_secret_with_registry.json" + mode: "0600" + + - name: Read current all.yaml file + ansible.builtin.slurp: + src: "{{ inventory_dir }}/group_vars/all.yaml" + register: all_yaml_content + + - name: Decode all.yaml content + ansible.builtin.set_fact: + all_yaml_lines: "{{ (all_yaml_content.content | b64decode).split('\n') }}" + + - name: Find pull_secret line in all.yaml + ansible.builtin.set_fact: + pull_secret_line_index: "{{ all_yaml_lines | map('regex_search', '^\\s*pull_secret:') | select('string') | list | length }}" + + - name: Prepare updated pull secret for YAML (single line, properly escaped) + ansible.builtin.set_fact: + updated_pull_secret_yaml: "{{ updated_pull_secret | to_json }}" + + - name: Prepare original pull secret comment + ansible.builtin.set_fact: + original_pull_secret_comment: "# ORIGINAL_PULL_SECRET: {{ env.redhat.pull_secret }}" + + - name: Update all.yaml with new pull secret and commented backup + ansible.builtin.lineinfile: + path: "{{ inventory_dir }}/group_vars/all.yaml" + regexp: '^(\s*)pull_secret:' + line: " {{ original_pull_secret_comment }}\n pull_secret: '{{ updated_pull_secret_yaml }}'" + backrefs: yes + backup: yes + + - name: Display success message + ansible.builtin.debug: + msg: + - "==========================================" + - "Pull Secret Updated Successfully!" + - "==========================================" + - "" + - "✅ The pull secret has been automatically updated in:" + - " {{ inventory_dir }}/group_vars/all.yaml" + - "" + - "✅ A backup of the original all.yaml was created with .backup extension" + - "" + - "✅ The original pull secret is preserved as a comment above the new one" + - "" + - "✅ A JSON copy is available at:" + - " {{ inventory_dir }}/group_vars/pull_secret_with_registry.json" + - "" + - "Registry added:" + - " Registry: {{ registry_url }}" + - " Username: {{ disconnected.registry.bastion.username }}" + - " Email: {{ disconnected.registry.bastion.email }}" + - "" + - "The updated pull secret is now active for all subsequent playbook runs." + + - name: Registry already in pull secret + when: registry_exists + ansible.builtin.debug: + msg: + - "Bastion registry ({{ registry_url }}) already exists in pull secret." + - "No update needed." + +# Assisted with Bob diff --git a/roles/dns/templates/dns.db.j2 b/roles/dns/templates/dns.db.j2 index 24bbeac8c..d9766442e 100644 --- a/roles/dns/templates/dns.db.j2 +++ b/roles/dns/templates/dns.db.j2 @@ -30,9 +30,9 @@ api-int.{{ env.cluster.networking.metadata_name }}.{{ env.cluster.networking.bas apps.{{ env.cluster.networking.metadata_name }}.{{ env.cluster.networking.base_domain }}. IN A {{ env.bastion.networking.ip }} *.apps.{{ env.cluster.networking.metadata_name }}.{{ env.cluster.networking.base_domain }}. IN A {{ env.bastion.networking.ip }} -{% if disconnected.enabled %} +{% if disconnected_enabled %} ;entry for mirror host. -{{ env.cluster.networking.metadata_name }}.{{ env.cluster.networking.metadata_name }}.{{ env.cluster.networking.base_domain }}. IN A {{ disconnected.registry.ip }} +{{ env.cluster.networking.metadata_name }}.{{ env.cluster.networking.metadata_name }}.{{ env.cluster.networking.base_domain }}. IN A {{ env.bastion.networking.ip if disconnected.registry.bastion.enabled | default(false) else disconnected.registry.ip }} {% endif %} ;EOF diff --git a/roles/get_ocp/tasks/main.yaml b/roles/get_ocp/tasks/main.yaml index ab1586a0c..ae6a6fc7a 100644 --- a/roles/get_ocp/tasks/main.yaml +++ b/roles/get_ocp/tasks/main.yaml @@ -1,14 +1,50 @@ --- +- name: Set registry URL and read certificate when using bastion registry + tags: get_ocp + when: + - disconnected_enabled + - disconnected.registry.bastion.enabled + block: + - name: Check if certificate exists in disconnected config + ansible.builtin.set_fact: + has_ca_cert_in_config: "{{ (disconnected.registry.ca_cert | default('') | trim | length > 0) }}" + + - name: Read certificate from bastion registry if not in config + when: + - not disconnected.registry.ca_trusted + - not has_ca_cert_in_config + ansible.builtin.slurp: + src: "{{ disconnected.registry.bastion.certs_dir }}/registry.crt" + register: bastion_registry_cert + + - name: Set registry URL and certificate + ansible.builtin.set_fact: + disconnected: + "{{ disconnected | combine({ + 'registry': disconnected.registry | combine({ + 'url': env.bastion.networking.ip ~ ':' ~ disconnected.registry.bastion.port, + 'ip': env.bastion.networking.ip, + 'ca_cert': (bastion_registry_cert.content | b64decode) if (bastion_registry_cert is defined and bastion_registry_cert.content is defined) else disconnected.registry.ca_cert + }, recursive=True) + }, recursive=True) }}" + + - name: Verify certificate is available for install-config + ansible.builtin.fail: + msg: "Certificate not available in disconnected.registry.ca_cert. Please ensure the registry setup has been completed or the certificate is properly configured in disconnected.yaml" + when: + - not disconnected.registry.ca_trusted + - (disconnected.registry.ca_cert | default('') | trim | length == 0) + - name: Delete ignition folder for idempotency tags: get_ocp - file: + ansible.builtin.file: path: /var/www/html/ignition state: absent - name: Create directory bin for mirrors tags: get_ocp become: true - file: + ansible.builtin.file: path: /var/www/html/bin state: directory mode: "0755" @@ -18,32 +54,70 @@ - name: Delete OCP download directory for idempotency, because ignition files deprecate after 24 hours. tags: get_ocp become: true - file: + ansible.builtin.file: path: /root/ocpinst state: absent - name: Create OCP download directory tags: get_ocp - file: + ansible.builtin.file: path: /root/ocpinst state: directory +- name: Set download URLs based on disconnected mode + tags: get_ocp + ansible.builtin.set_fact: + _rhcos_url: "{% if disconnected_enabled %}{{ env.file_server.protocol }}://{{ env.file_server.ip }}:{{ env.file_server.port }}/{{ disconnected.mirroring.file_server.clients_dir }}/{% else %}{{ rhcos_download_url }}{% endif %}" + _ocp_url: "{% if disconnected_enabled %}{{ env.file_server.protocol }}://{{ env.file_server.ip }}:{{ env.file_server.port }}/{{ disconnected.mirroring.file_server.clients_dir }}/{% else %}{{ ocp_download_url }}{% endif %}" + - name: Get Red Hat CoreOS rootfs file if it's not there already. tags: get_ocp - get_url: - url: "{{ rhcos_download_url }}{{ rhcos_live_rootfs }}" + when: not disconnected_enabled + ansible.builtin.get_url: + url: "{{ _rhcos_url }}{{ rhcos_live_rootfs }}" dest: "/var/www/html/bin/{{ rhcos_live_rootfs }}" mode: "0644" +- name: Check if RHCOS rootfs exists on local bastion HTTP root in disconnected mode + tags: get_ocp + when: disconnected_enabled + ansible.builtin.stat: + path: "/var/www/html/bin/{{ rhcos_live_rootfs }}" + register: rhcos_rootfs_stat + +- name: Check if RHCOS rootfs exists on configured file server in disconnected mode + tags: get_ocp + when: + - disconnected_enabled + - (env.file_server.ip | default('')) != (env.bastion.networking.ip | default('')) + ansible.builtin.stat: + path: "{{ disconnected.mirroring.file_server.document_root }}/bin/{{ rhcos_live_rootfs }}" + delegate_to: "{{ env.file_server.ip }}" + register: rhcos_rootfs_file_server_stat + +- name: Fail if RHCOS rootfs is missing in disconnected mode + tags: get_ocp + when: + - disconnected_enabled + - not rhcos_rootfs_stat.stat.exists + - ((env.file_server.ip | default('')) == (env.bastion.networking.ip | default('')) or not rhcos_rootfs_file_server_stat.stat.exists) + ansible.builtin.fail: + msg: + - "RHCOS rootfs file not found for disconnected installation." + - "Bastion path checked: /var/www/html/bin/{{ rhcos_live_rootfs }}" + - "File server path checked: {{ disconnected.mirroring.file_server.document_root }}/bin/{{ rhcos_live_rootfs }}" + - "Disconnected node installation requires this file to be served over HTTP from {{ env.file_server.protocol }}://{{ env.file_server.ip }}:{{ env.file_server.port }}/bin/{{ rhcos_live_rootfs }}" + - "Run the flow that includes [roles/disconnected_download_oc_mirror/tasks/main.yaml](roles/disconnected_download_oc_mirror/tasks/main.yaml:78) or manually place the file at the expected location." + - name: Unzip OCP client and installer tags: get_ocp ansible.builtin.unarchive: src: "{{ item }}" dest: /root/ocpinst/ - remote_src: yes + remote_src: true loop: - - "{{ ocp_download_url }}{{ ocp_client_tgz }}" - - "{{ ocp_download_url }}{{ ocp_install_tgz }}" + - "{{ _ocp_url }}{{ ocp_client_tgz }}" + - "{{ _ocp_url }}{{ ocp_install_tgz }}" - name: Copy kubectl, oc, and openshift-install binaries to /usr/local/sbin tags: get_ocp @@ -54,7 +128,7 @@ owner: root group: root mode: "755" - remote_src: yes + remote_src: true loop: - kubectl - oc @@ -62,22 +136,22 @@ - name: Use template file to create install-config and backup. tags: get_ocp - template: + ansible.builtin.template: src: install-config.yaml.j2 dest: "{{ item }}" - force: yes + force: true loop: - /root/ocpinst/install-config.yaml - /root/ocpinst/install-config-backup.yaml - name: Capture OCP public key tags: get_ocp - command: cat /root/.ssh/id_rsa.pub + ansible.builtin.command: cat /root/.ssh/id_rsa.pub register: ocp_pub_key - name: Place SSH key in install-config tags: get_ocp - lineinfile: + ansible.builtin.lineinfile: line: "sshKey: '{{ ocp_pub_key.stdout }}'" path: "{{ item }}" loop: @@ -88,14 +162,20 @@ tags: get_ocp ansible.builtin.shell: | set -o pipefail - {{ 'export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=quay.io/openshift-release-dev/ocp-release:' if disconnected.enabled }}{{ disconnected.mirroring.oc_mirror.release_image_tag if disconnected.enabled and not disconnected.mirroring.legacy.platform }}{{ disconnected.mirroring.legacy.ocp_quay_release_image_tag if disconnected.enabled and disconnected.mirroring.legacy.platform }} + {% if disconnected_enabled %} + {% if disconnected.mirroring.legacy.platform %} + export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE={{ disconnected.registry.url }}/{{ disconnected.mirroring.legacy.ocp_org }}/{{ disconnected.mirroring.legacy.ocp_repo }}:{{ disconnected.mirroring.legacy.ocp_tag }} + {% else %} + export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE={{ disconnected.registry.url }}/openshift/release-images:{{ disconnected.mirroring.oc_mirror.release_image_tag }} + {% endif %} + {% endif %} /root/ocpinst/openshift-install create manifests --dir=/root/ocpinst/ become: true - name: Copy the file when ipsec flag is enabled tags: get_ocp become: true - copy: + ansible.builtin.copy: src: cluster-network-03-config.yml dest: /root/ocpinst/manifests/cluster-network-03-config.yml when: env.ipsec_enabled is defined and env.ipsec_enabled != None and env.ipsec_enabled @@ -103,17 +183,17 @@ - name: List the files in the manifests directory tags: get_ocp become: true - command: "ls -lrt /root/ocpinst/manifests/" + ansible.builtin.command: "ls -lrt /root/ocpinst/manifests/" register: manifests_list -- debug: - msg: "{{ manifests_list }}" +- ansible.builtin.debug: + msg: "{{ manifests_list }}" - name: Set masters schedulable parameter to false tags: get_ocp become: true - replace: + ansible.builtin.replace: path: /root/ocpinst/manifests/cluster-scheduler-02-config.yml regexp: ": true" replace: ": false" @@ -122,7 +202,7 @@ - name: Set permissions for ocpinst directory contents to root tags: get_ocp become: true - command: chmod 0755 /root/ocpinst/{{item}} + ansible.builtin.command: chmod 0755 /root/ocpinst/{{item}} loop: - manifests - openshift @@ -134,7 +214,7 @@ become: true block: - name: Generate Butane file for nodes - template: + ansible.builtin.template: src: cex-butane-machineconfig.bu.j2 dest: "{{ output_dir }}/99-{{ node_role }}-s390x-cex-luks-config.bu" loop: @@ -147,13 +227,13 @@ layout: "{{ butane_default[cex_device].layout }}" - name: Download Butane binary for s390x - get_url: + ansible.builtin.get_url: url: https://mirror.openshift.com/pub/openshift-v4/clients/butane/latest/butane-s390x dest: /usr/local/bin/butane mode: '0755' - name: Convert Butane YAML to Ignition for each role - shell: | + ansible.builtin.shell: | /usr/local/bin/butane "{{ output_dir }}/99-{{ item }}-s390x-cex-luks-config.bu" \ -o "{{ output_dir }}/99-{{ item }}-s390x-cex-luks-config.yaml" loop: @@ -161,10 +241,10 @@ - worker - name: Copy generated ignition files to final directory - copy: + ansible.builtin.copy: src: "{{ output_dir }}/99-{{ item }}-s390x-cex-luks-config.yaml" dest: "/root/ocpinst/openshift/99-{{ item }}-s390x-cex-luks-config.yaml" - remote_src: yes + remote_src: true mode: '0644' loop: - master @@ -172,10 +252,25 @@ when: - cex | bool +- name: Create registry CA MachineConfig for disconnected installations + tags: get_ocp + when: + - disconnected_enabled + - not disconnected.registry.ca_trusted + block: + - name: Create registry CA MachineConfig manifest + ansible.builtin.template: + src: 99-registry-ca-machineconfig.yaml.j2 + dest: /root/ocpinst/openshift/99-registry-ca-machineconfig.yaml + owner: root + group: root + mode: "0644" + become: true + - name: Set ownership of ocpinst directory contents to root tags: get_ocp become: true - command: chown root:root /root/ocpinst/{{item}} + ansible.builtin.command: chown root:root /root/ocpinst/{{item}} loop: - manifests - openshift @@ -187,12 +282,168 @@ become: true ansible.builtin.shell: | set -o pipefail - {{ 'export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=quay.io/openshift-release-dev/ocp-release:' if disconnected.enabled }}{{ disconnected.mirroring.oc_mirror.release_image_tag if disconnected.enabled and not disconnected.mirroring.legacy.platform }}{{ disconnected.mirroring.legacy.ocp_quay_release_image_tag if disconnected.enabled and disconnected.mirroring.legacy.platform }} + {% if disconnected_enabled %} + {% if disconnected.mirroring.legacy.platform %} + export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE={{ disconnected.registry.url }}/{{ disconnected.mirroring.legacy.ocp_org }}/{{ disconnected.mirroring.legacy.ocp_repo }}:{{ disconnected.mirroring.legacy.ocp_tag }} + {% else %} + export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE={{ disconnected.registry.url }}/openshift/release-images:{{ disconnected.mirroring.oc_mirror.release_image_tag }} + {% endif %} + {% if not disconnected.registry.ca_trusted %} + export GODEBUG=x509ignoreCN=0 + {% endif %} + {% endif %} /root/ocpinst/openshift-install create ignition-configs --dir=/root/ocpinst/ +- name: Patch bootstrap ignition to add registry certificate for containers + tags: get_ocp + when: + - disconnected_enabled + - not disconnected.registry.ca_trusted + become: true + block: + - name: Read bootstrap ignition file + ansible.builtin.slurp: + src: /root/ocpinst/bootstrap.ign + register: bootstrap_ign_content + + - name: Parse bootstrap ignition JSON + ansible.builtin.set_fact: + bootstrap_ign: "{{ bootstrap_ign_content.content | b64decode | from_json }}" + + - name: Create registry cert directory entry for bootstrap + ansible.builtin.set_fact: + registry_cert_dir: + path: "/etc/containers/certs.d/{{ disconnected.registry.ip }}:{{ disconnected.registry.bastion.port if disconnected.registry.bastion.enabled else disconnected.registry.url.split(':')[-1] }}" + mode: 493 # 0755 in decimal + user: + name: root + group: + name: root + + - name: Create registry cert file entry for bootstrap + ansible.builtin.set_fact: + registry_cert_file: + path: "/etc/containers/certs.d/{{ disconnected.registry.ip }}:{{ disconnected.registry.bastion.port if disconnected.registry.bastion.enabled else disconnected.registry.url.split(':')[-1] }}/ca.crt" + mode: 420 # 0644 in decimal + overwrite: true + contents: + source: "data:text/plain;charset=utf-8;base64,{{ disconnected.registry.ca_cert | b64encode }}" + user: + name: root + group: + name: root + + - name: Create system trust anchor cert file entry for bootstrap + ansible.builtin.set_fact: + system_cert_file: + path: "/etc/pki/ca-trust/source/anchors/registry-ca.crt" + mode: 420 # 0644 in decimal + overwrite: true + contents: + source: "data:text/plain;charset=utf-8;base64,{{ disconnected.registry.ca_cert | b64encode }}" + user: + name: root + group: + name: root + + - name: Add registry certificate directory to bootstrap ignition + ansible.builtin.set_fact: + bootstrap_ign: "{{ bootstrap_ign | combine({'storage': {'directories': (bootstrap_ign.storage.directories | default([])) + [registry_cert_dir]}}, recursive=true) }}" + + - name: Add registry certificate file to bootstrap ignition + ansible.builtin.set_fact: + bootstrap_ign: "{{ bootstrap_ign | combine({'storage': {'files': bootstrap_ign.storage.files + [registry_cert_file, system_cert_file]}}, recursive=true) }}" + + - name: Create registries.conf.d directory entry for bootstrap + ansible.builtin.set_fact: + registries_conf_d_dir: + path: "/etc/containers/registries.conf.d" + mode: 493 # 0755 in decimal + user: + name: root + group: + name: root + + - name: Create insecure registry drop-in configuration + ansible.builtin.set_fact: + insecure_registry_conf_content: | + [[registry]] + location = "{{ disconnected.registry.ip }}:{{ disconnected.registry.bastion.port if disconnected.registry.bastion.enabled else disconnected.registry.url.split(':')[-1] }}" + insecure = true + + - name: Create insecure registry drop-in file entry for bootstrap + ansible.builtin.set_fact: + insecure_registry_conf_file: + path: "/etc/containers/registries.conf.d/999-insecure-registry.conf" + mode: 420 # 0644 in decimal + overwrite: true + contents: + source: "data:text/plain;charset=utf-8;base64,{{ insecure_registry_conf_content | b64encode }}" + user: + name: root + group: + name: root + + - name: Add registries.conf.d directory and insecure config to bootstrap ignition + ansible.builtin.set_fact: + bootstrap_ign: "{{ bootstrap_ign | combine({'storage': {'directories': (bootstrap_ign.storage.directories | default([])) + [registries_conf_d_dir], 'files': bootstrap_ign.storage.files + [insecure_registry_conf_file]}}, recursive=true) }}" + + - name: Create modified bootkube.sh script that uses insecure flag + ansible.builtin.set_fact: + bootkube_wrapper_content: | + #!/bin/bash + # Modify oc commands to use --insecure flag + export OC_INSECURE="--insecure" + exec /usr/local/bin/bootkube.sh.original "$@" + + - name: Create bootkube wrapper script file entry + ansible.builtin.set_fact: + bootkube_wrapper_file: + path: "/usr/local/bin/bootkube-wrapper.sh" + mode: 493 # 0755 in decimal + overwrite: true + contents: + source: "data:text/plain;charset=utf-8;base64,{{ bootkube_wrapper_content | b64encode }}" + user: + name: root + group: + name: root + + - name: Create systemd oneshot service to patch bootkube.sh for insecure registry + ansible.builtin.set_fact: + patch_bootkube_unit: + name: patch-bootkube-insecure.service + enabled: true + contents: | + [Unit] + Description=Patch bootkube.sh to use insecure flag for oc commands + Before=bootkube.service release-image.service + After=local-fs.target + ConditionPathExists=/usr/local/bin/bootkube.sh + + [Service] + Type=oneshot + ExecStart=/bin/bash -c 'if ! grep -q "insecure-added" /usr/local/bin/bootkube.sh; then sed -i -e "s|oc adm release info|oc adm release info --insecure|g" -e "1a# insecure-added" /usr/local/bin/bootkube.sh; fi' + RemainAfterExit=yes + + [Install] + WantedBy=multi-user.target + + - name: Add systemd unit to patch bootkube.sh to bootstrap ignition + ansible.builtin.set_fact: + bootstrap_ign: "{{ bootstrap_ign | combine({'systemd': {'units': (bootstrap_ign.systemd.units | default([])) + [patch_bootkube_unit]}}, recursive=true) }}" + + - name: Write patched bootstrap ignition back to file + ansible.builtin.copy: + content: "{{ bootstrap_ign | to_nice_json }}" + dest: /root/ocpinst/bootstrap.ign + owner: root + group: root + mode: "0755" + - name: Set ownership to root and permissions of ignitions and related files. tags: get_ocp - file: + ansible.builtin.file: state: "{{ item.state }}" path: /root/ocpinst/{{ item.path }} owner: root @@ -210,60 +461,60 @@ - name: Create directory in admin user's home for default kubeconfig. tags: get_ocp, config become: false - file: + ansible.builtin.file: state: directory path: ~/.kube - name: Create directory in root's home for default kubeconfig. tags: get_ocp, config become: true - file: + ansible.builtin.file: state: directory path: ~/.kube - name: Make kubeconfig admin user's default (for non-root user). tags: get_ocp, config - copy: + ansible.builtin.copy: src: /root/ocpinst/auth/kubeconfig dest: /home/{{ env.bastion.access.user }}/.kube/config owner: "{{ env.bastion.access.user }}" group: "{{ env.bastion.access.user }}" - remote_src: yes + remote_src: true when: env.bastion.access.user != "root" - name: Make kubeconfig admin user's default (for root user). tags: get_ocp, config - copy: + ansible.builtin.copy: src: /root/ocpinst/auth/kubeconfig dest: /{{ env.bastion.access.user }}/.kube/config owner: "{{ env.bastion.access.user }}" group: "{{ env.bastion.access.user }}" - remote_src: yes + remote_src: true when: env.bastion.access.user == "root" - name: Make kubeconfig root user's default. tags: get_ocp, config - copy: + ansible.builtin.copy: src: /root/ocpinst/auth/kubeconfig dest: /root/.kube/config owner: root group: root - remote_src: yes + remote_src: true - name: Create ignition directory in HTTP-accessible directory. tags: get_ocp become: true - file: + ansible.builtin.file: path: /var/www/html/ignition state: directory - name: Copy ignition files to HTTP-accessible directory. tags: get_ocp become: true - copy: + ansible.builtin.copy: src: /root/ocpinst/{{ item }}.ign dest: /var/www/html/ignition - remote_src: yes + remote_src: true mode: "775" group: root owner: root diff --git a/roles/get_ocp/templates/99-registry-ca-machineconfig.yaml.j2 b/roles/get_ocp/templates/99-registry-ca-machineconfig.yaml.j2 new file mode 100644 index 000000000..50c59d442 --- /dev/null +++ b/roles/get_ocp/templates/99-registry-ca-machineconfig.yaml.j2 @@ -0,0 +1,79 @@ +apiVersion: machineconfiguration.openshift.io/v1 +kind: MachineConfig +metadata: + labels: + machineconfiguration.openshift.io/role: master + name: 99-master-registry-ca +spec: + config: + ignition: + version: 3.2.0 + storage: + files: + - contents: + source: data:text/plain;charset=utf-8;base64,{{ disconnected.registry.ca_cert | b64encode }} + mode: 0644 + overwrite: true + path: /etc/pki/ca-trust/source/anchors/registry-ca.crt + - contents: + source: data:text/plain;charset=utf-8;base64,{{ disconnected.registry.ca_cert | b64encode }} + mode: 0644 + overwrite: true + path: /etc/containers/certs.d/{{ disconnected.registry.ip }}:{{ disconnected.registry.bastion.port if disconnected.registry.bastion.enabled else disconnected.registry.url.split(':')[-1] }}/ca.crt + systemd: + units: + - name: update-ca-trust.service + enabled: true + contents: | + [Unit] + Description=Update CA trust store with registry certificate + After=local-fs.target + Before=crio.service + + [Service] + Type=oneshot + ExecStart=/usr/bin/update-ca-trust + RemainAfterExit=yes + + [Install] + WantedBy=multi-user.target +--- +apiVersion: machineconfiguration.openshift.io/v1 +kind: MachineConfig +metadata: + labels: + machineconfiguration.openshift.io/role: worker + name: 99-worker-registry-ca +spec: + config: + ignition: + version: 3.2.0 + storage: + files: + - contents: + source: data:text/plain;charset=utf-8;base64,{{ disconnected.registry.ca_cert | b64encode }} + mode: 0644 + overwrite: true + path: /etc/pki/ca-trust/source/anchors/registry-ca.crt + - contents: + source: data:text/plain;charset=utf-8;base64,{{ disconnected.registry.ca_cert | b64encode }} + mode: 0644 + overwrite: true + path: /etc/containers/certs.d/{{ disconnected.registry.ip }}:{{ disconnected.registry.bastion.port if disconnected.registry.bastion.enabled else disconnected.registry.url.split(':')[-1] }}/ca.crt + systemd: + units: + - name: update-ca-trust.service + enabled: true + contents: | + [Unit] + Description=Update CA trust store with registry certificate + After=local-fs.target + Before=crio.service + + [Service] + Type=oneshot + ExecStart=/usr/bin/update-ca-trust + RemainAfterExit=yes + + [Install] + WantedBy=multi-user.target \ No newline at end of file diff --git a/roles/get_ocp/templates/install-config.yaml.j2 b/roles/get_ocp/templates/install-config.yaml.j2 index 3eb39c8e6..09dec142a 100644 --- a/roles/get_ocp/templates/install-config.yaml.j2 +++ b/roles/get_ocp/templates/install-config.yaml.j2 @@ -39,8 +39,8 @@ networking: platform: none: {} fips: {{ install_config_vars.fips }} -pullSecret: '{{ env.redhat.pull_secret if not disconnected.enabled else disconnected.registry.pull_secret }}' -{% if disconnected.enabled %} +pullSecret: {{ ((env.redhat.pull_secret if not disconnected_enabled else disconnected.registry.pull_secret) if ((env.redhat.pull_secret if not disconnected_enabled else disconnected.registry.pull_secret) is string) else ((env.redhat.pull_secret if not disconnected_enabled else disconnected.registry.pull_secret) | to_json)) | to_json }} +{% if disconnected_enabled %} {{ 'imageContentSources: ' }} {{ '- mirrors:'}} {{ ' - ' + disconnected.registry.url + '/' }}{{ disconnected.mirroring.legacy.ocp_org if disconnected.mirroring.legacy.platform else 'openshift' }}{{ '/' }}{{ disconnected.mirroring.legacy.ocp_repo if disconnected.mirroring.legacy.platform else 'release-images' }} @@ -49,6 +49,7 @@ pullSecret: '{{ env.redhat.pull_secret if not disconnected.enabled else disconne {{ ' - ' + disconnected.registry.url + '/' }}{{ disconnected.mirroring.legacy.ocp_org if disconnected.mirroring.legacy.platform else 'openshift' }}{{ '/' }}{{ disconnected.mirroring.legacy.ocp_repo if disconnected.mirroring.legacy.platform else 'release' }} {{ ' source: quay.io/openshift-release-dev/ocp-v4.0-art-dev' }} {% endif %} -{% if disconnected.enabled and not disconnected.registry.ca_trusted %} -{{ 'additionalTrustBundle: |' }}{% for line in disconnected.registry.ca_cert.split('\n') %}{{ '\n ' + line }}{% endfor %} +{% if disconnected_enabled and not disconnected.registry.ca_trusted %} +additionalTrustBundle: | +{{ disconnected.registry.ca_cert | indent(2, first=True) }} {% endif %} diff --git a/roles/prepare_configs/templates/install-config.yaml.j2 b/roles/prepare_configs/templates/install-config.yaml.j2 index f3ce903df..2304db314 100644 --- a/roles/prepare_configs/templates/install-config.yaml.j2 +++ b/roles/prepare_configs/templates/install-config.yaml.j2 @@ -36,8 +36,8 @@ platform: none: {} sshKey: > {{ ssh_key.stdout }} -pullSecret: '{{ env.redhat.pull_secret if not disconnected.enabled else disconnected.registry.pull_secret }}' -{% if disconnected.enabled %} +pullSecret: '{{ env.redhat.pull_secret if not disconnected_enabled else disconnected.registry.pull_secret }}' +{% if disconnected_enabled %} {{ 'imageContentSources: ' }} {{ '- mirrors:'}} {{ ' - ' + disconnected.registry.url + '/' }}{{ disconnected.mirroring.legacy.ocp_org if disconnected.mirroring.legacy.platform else 'openshift' }}{{ '/' }}{{ disconnected.mirroring.legacy.ocp_repo if disconnected.mirroring.legacy.platform else 'release-images' }} @@ -46,6 +46,6 @@ pullSecret: '{{ env.redhat.pull_secret if not disconnected.enabled else disconne {{ ' - ' + disconnected.registry.url + '/' }}{{ disconnected.mirroring.legacy.ocp_org if disconnected.mirroring.legacy.platform else 'openshift' }}{{ '/' }}{{ disconnected.mirroring.legacy.ocp_repo if disconnected.mirroring.legacy.platform else 'release' }} {{ ' source: quay.io/openshift-release-dev/ocp-v4.0-art-dev' }} {% endif %} -{% if disconnected.enabled and not disconnected.registry.ca_trusted %} +{% if disconnected_enabled and not disconnected.registry.ca_trusted %} {{ 'additionalTrustBundle: |' }}{% for line in disconnected.registry.ca_cert.split('\n') %}{{ '\n ' + line }}{% endfor %} {% endif %} diff --git a/roles/set_inventory/templates/hosts.j2 b/roles/set_inventory/templates/hosts.j2 index 23508e3b9..0893307e9 100644 --- a/roles/set_inventory/templates/hosts.j2 +++ b/roles/set_inventory/templates/hosts.j2 @@ -2,35 +2,35 @@ 127.0.0.1 ansible_connection=local ansible_become_password='{{ '{{' }} controller_sudo_pass {{ '}}' }}' [file_server] -{{ env.file_server.ip }} ansible_user={{ env.file_server.user }} ansible_become_password='{{ '{{' }} env.file_server.pass {{ '}}' }}' +{{ env.file_server.ip | default('') }} ansible_user={{ env.file_server.user | default('') }} ansible_become_password='{{ '{{' }} env.file_server.pass {{ '}}' }}' [kvm_host] -{% if ( env.z.lpar1.hostname is defined ) %} +{% if (env.z.lpar1.hostname is not none and env.z.lpar1.hostname | trim | default('') | length > 0 and env.z.lpar1.ip | default('') | length > 0 and env.z.lpar1.user | default('') | length > 0) %} {{ env.z.lpar1.hostname }} ansible_host={{ env.z.lpar1.ip }} ansible_user={{ env.z.lpar1.user }} ansible_become_password='{{ '{{' }} env.z.lpar1.pass {{ '}}' }}' {% endif %} -{% if env.z.lpar2.hostname is defined %} +{% if (env.z.lpar2.hostname is not none and env.z.lpar2.hostname | trim | default('') | length > 0 and env.z.lpar2.ip | default('') | length > 0 and env.z.lpar2.user | default('') | length > 0) %} {{ env.z.lpar2.hostname }} ansible_host={{ env.z.lpar2.ip }} ansible_user={{ env.z.lpar2.user }} ansible_become_password='{{ '{{' }} env.z.lpar2.pass {{ '}}' }}' {% endif %} -{% if env.z.lpar3.hostname is defined %} +{% if (env.z.lpar3.hostname is not none and env.z.lpar3.hostname | trim | default('') | length > 0 and env.z.lpar3.ip | default('') | length > 0 and env.z.lpar3.user | default('') | length > 0) %} {{ env.z.lpar3.hostname }} ansible_host={{ env.z.lpar3.ip }} ansible_user={{ env.z.lpar3.user }} ansible_become_password='{{ '{{' }} env.z.lpar3.pass {{ '}}' }}' {% endif %} -{% if ( installation_type | lower == 'zvm' ) %} +{% if (installation_type | lower == 'zvm') and (zvm.nodes | default([]) | length > 0) %} {{ '[zvm_host]' }} -{% for item in range( zvm.nodes | length ) %} +{% for item in range(zvm.nodes | length) %} {{ zvm.nodes[item].name | string + ' ansible_host=' + zvm.nodes[item].interface.ip | string + ' ansible_user=' + zvm.nodes[item].user | string + ' ansible_become_password=' + zvm.nodes[item].password | string }} {% endfor %} {% endif %} [bastion] -{{ env.bastion.networking.hostname }} ansible_host={{ env.bastion.networking.ip }} ansible_user={{ env.bastion.access.user }} ansible_become_password='{{ '{{' }} env.bastion.access.pass {{ '}}' }}' +{{ env.bastion.networking.hostname | default('') }} ansible_host={{ env.bastion.networking.ip | default('') }} ansible_user={{ env.bastion.access.user | default('') }} ansible_become_password='{{ '{{' }} env.bastion.access.pass {{ '}}' }}' -{% if ( env.network_mode | upper == 'NAT' ) and ( env.jumphost.name is not none ) and ( env.jumphost.ip is not none ) and ( env.jumphost.user is not none ) and ( env.jumphost.pass is not none ) -%} +{% if (env.jumphost is defined) and (env.network_mode | default('') | upper == 'NAT') and (env.jumphost.name | default('') | length > 0) and (env.jumphost.ip | default('') | length > 0) and (env.jumphost.user | default('') | length > 0) and (env.jumphost.pass | default('') | length > 0) -%} {{ '[jumphost]' }} -{{ env.jumphost.name | string + ' ansible_host=' + env.jumphost.ip | string + ' ansible_user=' + env.jumphost.user | string + ' ansible_become_password=' + env.jumphost.pass | string }} +{{ env.jumphost.name | default('') }} ansible_host={{ env.jumphost.ip | default('') }} ansible_user={{ env.jumphost.user | default('') }} ansible_become_password='{{ '{{' }} env.jumphost.pass {{ '}}' }}' {% endif -%} -{% if ( disconnected.enabled ) %} +{% if (disconnected_enabled | default(false)) and (disconnected is defined) and (disconnected.mirroring is defined) and (disconnected.mirroring.host is defined) and (disconnected.mirroring.host.name | default('') | length > 0) %} {{ '[mirrorhost]' }} -{{ disconnected.mirroring.host.name | string + ' ansible_host=' + disconnected.mirroring.host.ip | string + ' ansible_user=' + disconnected.mirroring.host.user | string + ' ansible_become_password=' + disconnected.mirroring.host.pass | string }} +{{ disconnected.mirroring.host.name | default('') }} ansible_host={{ disconnected.mirroring.host.ip | default('') }} ansible_user={{ disconnected.mirroring.host.user | default('') }} ansible_become_password='{{ '{{' }} disconnected.mirroring.host.pass {{ '}}' }}' {% endif -%} diff --git a/roles/update_cfgs/tasks/main.yaml b/roles/update_cfgs/tasks/main.yaml index 359637f1a..9020bfc41 100644 --- a/roles/update_cfgs/tasks/main.yaml +++ b/roles/update_cfgs/tasks/main.yaml @@ -84,7 +84,7 @@ lineinfile: path: "{{ env.file_server.cfgs_dir }}/{{ networking.hostname }}/kvm_host.cfg" insertafter: "Network information" - line: network --bootproto=static --device={{ networking.device1 }} --gateway={{ networking.gateway }} --ip={{ networking.ip }} --nameserver={{ networking.nameserver1 }} {{ ('--nameserver=' + networking.nameserver2) if networking.nameserver2 is defined else '' }} --netmask={{ networking.subnetmask }} --noipv6 --activate --hostname={{ networking.hostname }} + line: network --bootproto=static --device={{ networking.device1 }} --gateway={{ networking.gateway }} --ip={{ networking.ip }} --nameserver={{ networking.nameserver1 }} {{ ('--nameserver=' + networking.nameserver2) if (networking.nameserver2 is defined and networking.nameserver2 | length > 0) else '' }} --netmask={{ networking.subnetmask }} --noipv6 --activate --hostname={{ networking.hostname }} when: lpar.networking.nic.card2 is not defined - name: Add network information in KVM hosts's RHEL kickstart file when there are two network cards defined. @@ -92,5 +92,5 @@ lineinfile: path: "{{ env.file_server.cfgs_dir }}/{{ networking.hostname }}/kvm_host.cfg" insertafter: "Network information" - line: "network --bootproto=static --device={{ networking.device1 }} --bondslaves={{ lpar.networking.nic.card1.name }},{{ lpar.networking.nic.card2.name }} --bondopts=mode=active-backup;primary={{ lpar.networking.nic.card1 }} --gateway={{ networking.gateway }} --ip={{ networking.ip }} --nameserver={{ networking.nameserver1 }} {{ ('--nameserver=' + networking.nameserver2) if networking.nameserver2 is defined else '' }} --netmask={{ networking.subnetmask }} --noipv6 --activate --hostname={{ networking.hostname }}" + line: "network --bootproto=static --device={{ networking.device1 }} --bondslaves={{ lpar.networking.nic.card1.name }},{{ lpar.networking.nic.card2.name }} --bondopts=mode=active-backup;primary={{ lpar.networking.nic.card1 }} --gateway={{ networking.gateway }} --ip={{ networking.ip }} --nameserver={{ networking.nameserver1 }} {{ ('--nameserver=' + networking.nameserver2) if (networking.nameserver2 is defined and networking.nameserver2 | length > 0) else '' }} --netmask={{ networking.subnetmask }} --noipv6 --activate --hostname={{ networking.hostname }}" when: lpar.networking.nic.card2.name is defined