Need help running fio benchmark using static VM configuration

[rtadakam]

I am running a fio test against a pair of static VMs in an OnPrem environment. Oddly, PKB assumes the VM to be from GCP environment. I have tried specifying os_type as "rhel" in the static VM configuration, but that does not help. The same static VM definition works well with iperf tests.

Could someone please review my configuration file and let me know the error?

Config file

# cat static_configs/fio_config.yaml
static_vms:
 - &vm1
   ip_address: x.x.x.x
   internal_ip: 192.168.150.201
   user_name: root
   ssh_private_key: /root/.ssh/id_ecdsa
   zone: local
   os_type: rhel8
   disk_specs:
    - mount_point: /test_data
 - &vm2
   ip_address: y.y.y.y
   internal_ip: 192.168.150.203
   user_name: root
   ssh_private_key: /root/.ssh/id_ecdsa
   zone: local
   os_type: rhel8
   disk_specs:
    - mount_point: /test_data

benchmarks:
  - fio: {vm_groups: {default: {static_vms: [*vm1]}},
          flags: {against_device: True}}
  - fio: {vm_groups: {default: {static_vms: [*vm2]}},
          flags: {against_device: True}}

PKB Run command
# ./pkb.py --benchmarks=fio --benchmark_config_file=static_configs/fio_config.yaml --accept_licenses --log_level=debug
PKB Log File

# cat /tmp/perfkitbenchmarker/runs/6f6585c1/pkb.log
2024-05-06 03:03:56,841 6f6585c1 MainThread pkb.py:1304 INFO     PerfKitBenchmarker version: v1.12.0-4489-gd50322ee
2024-05-06 03:03:56,842 6f6585c1 MainThread pkb.py:1277 INFO     Flag values:
--benchmark_config_file=static_configs/fio_config.yaml
--accept_licenses
--benchmarks=fio
--log_level=debug
2024-05-06 03:03:57,201 6f6585c1 MainThread vm_util.py:244 INFO     Running: ssh-keygen -t rsa -N  -m PEM -q -f /tmp/perfkitbenchmarker/runs/6f6585c1/perfkitbenchmarker_keyfile
2024-05-06 03:03:57,300 6f6585c1 MainThread vm_util.py:244 INFO     Ran: {ssh-keygen -t rsa -N  -m PEM -q -f /tmp/perfkitbenchmarker/runs/6f6585c1/perfkitbenchmarker_keyfile}
ReturnCode:0
STDOUT:
STDERR:
2024-05-06 03:03:57,301 6f6585c1 MainThread pkb.py:1348 INFO     Setting --max_concurrent_threads=200.
2024-05-06 03:03:57,366 6f6585c1 MainThread __init__.py:278 WARNING  The key "flags" was not in the default config, but was in user overrides. This may indicate a typo.
2024-05-06 03:03:57,378 6f6585c1 MainThread publisher.py:1073 DEBUG    Using publishers: [<LogPublisher logger=<RootLogger root (DEBUG)> level=20>, <PrettyPrintStreamPublisher stream=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>>, <NewlineDelimitedJSONPublisher file_path="/tmp/perfkitbenchmarker/runs/6f6585c1/perfkitbenchmarker_results.json" mode="w">]
2024-05-06 03:03:57,378 6f6585c1 MainThread pkb.py:1098 INFO
-------------------------------------------------------------------------------------
Starting benchmark 1/1 fio (UID: fio0) attempt 1 of 1
-------------------------------------------------------------------------------------
2024-05-06 03:03:57,378 6f6585c1 MainThread publisher.py:1073 DEBUG    Using publishers: [<LogPublisher logger=<RootLogger root (DEBUG)> level=20>, <PrettyPrintStreamPublisher stream=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>>, <NewlineDelimitedJSONPublisher file_path="/tmp/perfkitbenchmarker/runs/6f6585c1/perfkitbenchmarker_results.json" mode="w">]
2024-05-06 03:03:57,378 6f6585c1 MainThread fio(1/1) pkb.py:451 INFO     Provisioning resources for benchmark fio
2024-05-06 03:03:57,379 6f6585c1 MainThread fio(1/1) util.py:71 INFO     Running: gcloud config list --format=json
2024-05-06 03:03:57,385 6f6585c1 MainThread fio(1/1) pkb.py:933 ERROR    Error during benchmark fio
Traceback (most recent call last):
  File "/pkb/PerfKitBenchmarker/perfkitbenchmarker/pkb.py", line 853, in RunBenchmark
    DoProvisionPhase(spec, detailed_timer)
  File "/pkb/PerfKitBenchmarker/perfkitbenchmarker/pkb.py", line 457, in DoProvisionPhase
    spec.ConstructVirtualMachines()
  File "/pkb/PerfKitBenchmarker/perfkitbenchmarker/benchmark_spec.py", line 620, in ConstructVirtualMachines
    vms = self.ConstructVirtualMachineGroup(group_name, group_spec)
  File "/pkb/PerfKitBenchmarker/perfkitbenchmarker/benchmark_spec.py", line 559, in ConstructVirtualMachineGroup
    vm = self._CreateVirtualMachine(group_spec.vm_spec, os_type, cloud)
  File "/pkb/PerfKitBenchmarker/perfkitbenchmarker/benchmark_spec.py", line 1012, in _CreateVirtualMachine
    return vm_class(vm_spec)
  File "/pkb/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gce_virtual_machine.py", line 1499, in __init__
    super(BaseLinuxGceVirtualMachine, self).__init__(vm_spec)
  File "/pkb/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gce_virtual_machine.py", line 536, in __init__
    self.project = vm_spec.project or util.GetDefaultProject()
  File "/pkb/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/util.py", line 71, in GetDefaultProject
    stdout, _, _ = vm_util.IssueCommand(cmd)
  File "/pkb/PerfKitBenchmarker/perfkitbenchmarker/vm_util.py", line 486, in IssueCommand
    process = subprocess.Popen(
  File "/usr/lib64/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib64/python3.9/subprocess.py", line 1821, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'gcloud'
2024-05-06 03:03:57,387 6f6585c1 MainThread fio(1/1) pkb.py:1111 ERROR    Exception running benchmark
Traceback (most recent call last):
  File "/pkb/PerfKitBenchmarker/perfkitbenchmarker/pkb.py", line 1109, in RunBenchmarkTask
    RunBenchmark(spec_for_run, collector)
  File "/pkb/PerfKitBenchmarker/perfkitbenchmarker/pkb.py", line 853, in RunBenchmark
    DoProvisionPhase(spec, detailed_timer)
  File "/pkb/PerfKitBenchmarker/perfkitbenchmarker/pkb.py", line 457, in DoProvisionPhase
    spec.ConstructVirtualMachines()
  File "/pkb/PerfKitBenchmarker/perfkitbenchmarker/benchmark_spec.py", line 620, in ConstructVirtualMachines
    vms = self.ConstructVirtualMachineGroup(group_name, group_spec)
  File "/pkb/PerfKitBenchmarker/perfkitbenchmarker/benchmark_spec.py", line 559, in ConstructVirtualMachineGroup
    vm = self._CreateVirtualMachine(group_spec.vm_spec, os_type, cloud)
  File "/pkb/PerfKitBenchmarker/perfkitbenchmarker/benchmark_spec.py", line 1012, in _CreateVirtualMachine
    return vm_class(vm_spec)
  File "/pkb/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gce_virtual_machine.py", line 1499, in __init__
    super(BaseLinuxGceVirtualMachine, self).__init__(vm_spec)
  File "/pkb/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gce_virtual_machine.py", line 536, in __init__
    self.project = vm_spec.project or util.GetDefaultProject()
  File "/pkb/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/util.py", line 71, in GetDefaultProject
    stdout, _, _ = vm_util.IssueCommand(cmd)
  File "/pkb/PerfKitBenchmarker/perfkitbenchmarker/vm_util.py", line 486, in IssueCommand
    process = subprocess.Popen(
  File "/usr/lib64/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib64/python3.9/subprocess.py", line 1821, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'gcloud'
2024-05-06 03:03:57,387 6f6585c1 MainThread fio(1/1) pkb.py:1117 ERROR    Benchmark 1/1 fio (UID: fio0) failed. Execution will continue.
2024-05-06 03:03:57,387 6f6585c1 MainThread fio(1/1) pkb.py:1131 INFO     Benchmark should not be retried. Finished 1 runs of 1
2024-05-06 03:03:57,388 6f6585c1 MainThread fio(1/1) pkb.py:1402 INFO     Benchmark run statuses:
------------------------------------
Name  UID   Status  Failed Substatus
------------------------------------
fio   fio0  FAILED  UNCATEGORIZED
------------------------------------
Success rate: 0.00% (0/1)
2024-05-06 03:03:57,388 6f6585c1 MainThread fio(1/1) pkb.py:1404 INFO     Complete logs can be found at: /tmp/perfkitbenchmarker/runs/6f6585c1/pkb.log
2024-05-06 03:03:57,388 6f6585c1 MainThread fio(1/1) pkb.py:1405 INFO     Completion statuses can be found at: /tmp/perfkitbenchmarker/runs/6f6585c1/completion_statuses.json
2024-05-06 03:03:57,388 6f6585c1 MainThread fio(1/1) pkb.py:1439 INFO     PKB exiting with return_code 1

[s-deitz]

It looks like PKB is trying to provision VMs and the default cloud flag is GCP.

You may need to use the --static_vm_file, but I'm surprised this would work for iperf then. Are you still stuck or did you make progress?