Introduction

This script dispatches fence requests from the scoutfs quorum nodes.

All scoutfs installations require fencing to be setup. Including single node systems.

It supports fencing nodes using ipmitool or powerman, depending on the configuration used. SSH access is also required for all of these to properly function.

Documentation for the script, configuration, and operations is available at ScoutFS Fencing.

This script must be installed on ALL active quorum nodes, so make sure to repeat the full installation instructions on every node.

Installation

Steps for installing the script:

1. Install the scoutfs-fence-agent RPM on all quorum nodes.
2. Copy both example configuration files from /usr/share/doc/scoutfs-fence-agent/ to /etc/scoutfs/, dropping the -example suffix, and edit them as needed:
   • scoutfs-ipmi.conf-example -> /etc/scoutfs/scoutfs-ipmi.conf (credentials and binary paths for ssh / ipmitool / powerman)
   • scoutfs-ipmi-hosts.conf-example -> /etc/scoutfs/scoutfs-ipmi-hosts.conf (per-host backend selection and target addresses)
3. Enable and start the ScoutFS fencing service: systemctl enable --now scoutfs-fenced

Repeat on every node.

Testing your Configuration

The fencing configuration can be tested by executing the script directly:

/usr/libexec/scoutfs-fenced/run/scoutfs-fence-agent

Example output:

Example 1: Properly configured (one ipmitool, one powerman, and one plain ssh entry). The agent emits a (ssh) line per node that uses ipmitool or powerman (to confirm the rid probe path) followed by a line for the backend itself:

  Test mode: verifying connection to manage all known nodes.
   ✓ PASS		192.168.124.12 (ssh)
   ✓ PASS		192.168.124.12 (ipmitool)
   ✓ PASS		192.168.124.126 (ssh)
   ✓ PASS		192.168.124.126 (powerman)
   ✓ PASS		192.168.124.80 (ssh)

Example 2: Misconfigured SSH:

  Test mode: verifying connection to manage all known nodes.
  Unable to ssh to host "192.168.124.12"
   ✗ FAIL		192.168.124.12 (ssh)
   ✓ PASS		192.168.124.12 (ipmitool)
  Unable to ssh to host "192.168.124.126"
   ✗ FAIL		192.168.124.126 (ssh)
   ✓ PASS		192.168.124.126 (ipmitool)
  Unable to ssh to host "192.168.124.80"
   ✗ FAIL		192.168.124.80 (ssh)
   ✓ PASS		192.168.124.80 (ipmitool)
  Test mode failed.

The script will test connectivity to the configured control or management plane and succeed if that works. Any errors encountered during the testing will be shown to help diagnose configuration errors.

Passwordless SSH Considerations

The script can utilize passwordless SSH between all nodes that mount the ScoutFS filesystem but adds the ability to use non-root identities. If using a non-root user, the following parameters can be added to the /etc/scoutfs/scoutfs-ipmi.conf configuration file:

SSH_USER=user
SSH_IDENTS="-i /etc/secret/scoutfs_1_rsa -i /etc/secret/scoutfs_2_rsa"

These parameters will modify the SSH command used in the script to log in as user and use the private keys defined in the SSH_IDENTS parameter.

The corresponding public key must be added to the remote hosts authorized_keys file. An easy way to do this is to use ssh-copy-id -i /etc/secret/scoutfs_1_rsa (with or without .pub will work).

Powerman

Powerman can be used in environments that do not allow node-to-node BMC connections. The powerman daemon runs on a designated host with BMC reachability, and the quorum nodes invoke pm to ask it to query or power off a target node by name.

Important

Versity does not directly support Powerman. While we can provide guidance and example configurations, we do not debug or provide fixes for Powerman.

Installation

The powerman package is part of the EPEL repository. Refer to EPEL to install the EPEL repository on your system. Usually, one can do dnf install epel-release.

The powerman package can be installed after the EPEL repository is enabled:

dnf install -y powerman

Important

By default the powerman command installed by powerman package is world executable. We recommend securing access to the powerman command by adjusting the permissions to 700 with chmod 700 /bin/powerman.

Where to Install Powerman

The powerman package will need to be installed on the designated node(s) that have BMC access to the ScoutFS nodes as well as all ScoutFS quorum nodes. The ScoutFS quorum nodes will use the pm command to issue remote power status and off commands to the Powerman servers.

Multiple Powerman servers are supported to enable redundancy if one Powerman server is unavailable.

Configuration

The configuration file for Powerman is /etc/powerman/powerman.conf. Here is an example configuration for Dell systems with iDRAC that support the ipmipower command:

listen "0.0.0.0:10101"
include "/etc/powerman/ipmipower.dev"
device "ipmi_radia" "ipmipower" "/usr/sbin/ipmipower -D LAN_2_0 --wait-until-on --wait-until-off -u admin -p password -h radia[1-5]-ipmi |&"
node "radia[1-5]"    "ipmi_radia" "radia[1-5]-ipmi"

Note

Any change to the /etc/powerman/powerman.conf file requires the powerman service to be restarted.

Starting/Stopping Powerman

The powerman service can be controlled with the systemctl command.

Starting the powerman service:

systemctl start powerman

Stopping the powerman service:

systemctl stop powerman

Verifying Powerman

Once powerman is configured and running on one or more designated Powerman servers, the configuration can be validated by checking the power status of all nodes from a ScoutFS quorum member.

Example querying the power state of all configured nodes:

powerman -h admin -q

Example output:

on:      radia[1-5]
off:
unknown:

ScoutFS Fencing Configuration

After validating the powerman configuration, the host definitions in /etc/scoutfs/scoutfs-ipmi-hosts.conf file are as follows:

10.0.0.100  powerman    v1,172.21.1.51:radia1
10.0.0.101  powerman    v1,172.21.1.51:radia2
10.0.0.102  powerman    v1,172.21.1.51:radia3
10.0.0.103  powerman    v1,172.21.1.51:radia4
10.0.0.104  powerman    v1,172.21.1.51:radia5

The fields for the Powerman configuration are as follows:

SCOUTFS_QUORUM_IP    powerman    PM_SRV1[,PM_SRV2]:PM_NODE

The first parameter SCOUTFS_QUORUM_IP is the IP address that ScoutFS uses for quorum communications. The second parameter is the mode powerman. The third parameter includes a comma separated list of Powerman servers followed by a colon (:) and the name of the node configured in the Powerman configuration.

The PM_SRV1[,PM_SRV2] parameters can be host names or IP addresses. The PM_NODE parameter must match a node parameter in /etc/powerman/powerman.conf on the Powerman servers.