upgrade fails during puppet_runonce

[logicminds]

Describe the Bug

During a upgrade there is a bash script that fails and causes the plan to fail with an unfinished upgrade.

# Plan Step: upgrade-primary-compilers
Starting: task peadm::puppet_infra_upgrade on pe-std.puppet.vm
Finished: task peadm::puppet_infra_upgrade with 0 failures in 1.22 sec
# Plan Step: upgrade-replica
Starting: task peadm::puppet_runonce on pe-std.puppet.vm
Finished: task peadm::puppet_runonce with 0 failures in 29.89 sec
Starting: command 'if [ -e /opt/puppetlabs/server/apps/puppetdb/cli/apps/delete-reports -a ! -h /opt/puppetlabs/server/apps/puppetdb/cli/apps/delete-reports ]
then
  mv /opt/puppetlabs/server/apps/puppetdb/cli/apps/delete-reports /opt/puppetlabs/server/apps/puppetdb/cli/apps/delete-reports.original
  ln -s $(which true) /opt/puppetlabs/server/apps/puppetdb/cli/apps/delete-reports
fi
' on pe-std-replica.puppet.vm
Finished: command 'if [ -e /opt/puppetlabs/server/apps/puppetdb/cli/apps/delete-reports -a ! -h /opt/puppetlabs/server/apps/puppetdb/cli/apps/delete-reports ]
then
  mv /opt/puppetlabs/server/apps/puppetdb/cli/apps/delete-reports /opt/puppetlabs/server/apps/puppetdb/cli/apps/delete-reports.original
  ln -s $(which true) /opt/puppetlabs/server/apps/puppetdb/cli/apps/delete-reports
fi
' with 1 failure in 0.91 sec
Finished: plan peadm::upgrade in 2 min, 21 sec
Failed on pe-std-replica.puppet.vm:
  The command failed with exit code 126
  OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "if": executable file not found in $PATH: unknown


  This might be caused by the default tmpdir being mounted using 'noexec'. See http://pup.pt/task-failure for details and workarounds.
Failed on 1 target: pe-std-replica.puppet.vm

Expected Behavior

The upgrade plan with finish without errors. I don't really know what other steps are required to complete the replica upgrade.

Steps to Reproduce

Steps to reproduce the behavior:

1. From peadm codebase, cd spec/docker
2. bash install.sh (should install 2019.8.5)
3. Choose standard-ha
4. Once complete, will need to upgrade next
5. change the version in standard-ha/upgrade_params.json to 2019.8.8
6. From spec/docker, bash upgrade.sh, select standard-ha
7. Wait for upgrade and watch for error to occur towards end of upgrade.

{
  "primary_host": "pe-std.puppet.vm",
  "replica_host": "pe-std-replica.puppet.vm",
  "version": "2019.8.8"
}

Environment

CentOS:8
PE 2019.8.5, 2019.8.8

Additional Context

The current dockerfile doesn't have centos:8 since I have not created a PR for it yet. So this might just be a centos:8 problem as I haven't tested with centos:7. The steps above will use centos:7 which might have different results.

[mcka1n]

Internal ticket: SOLARCH-871