feat: add --subnets flag to deploy multiple nodes per client (#136)

* feat: add isAggregator flag to validator configuration

Add support for configuring nodes as aggregators through validator-config.yaml.
This allows selective designation of nodes to perform aggregation duties by
setting isAggregator: true in the validator configuration.

Changes:
- Add isAggregator field (default: false) to all validators in both local and ansible configs
- Update parse-vc.sh to extract and export isAggregator flag
- Modify all client command scripts to pass --is-aggregator flag when enabled
- Add isAggregator status to node information output

* spin-node: add --subnets flag to deploy multiple nodes per client

Adds --subnets N (1–5) to deploy N nodes of each client on their
associated servers, each on a distinct attestation subnet.

New files:
  - generate-subnet-config.py: expands validator-config.yaml into
    validator-config-subnets-N.yaml with unique node names, incremented
    ports (quic/metrics/api), fresh P2P private keys, and explicit subnet
    membership per entry. Also sets config.attestation_committee_count = N
    so each client correctly partitions validators across N committees.

Changes:
  - parse-env.sh: add --subnets N and --dry-run flags
  - spin-node.sh:
    - expand validator-config before genesis setup when --subnets N given
    - select one aggregator per subnet randomly; print prominent summary
    - --dry-run: simulate full deployment without applying any changes
      (Ansible runs with --check --diff, local execs are echoed only)
  - run-ansible.sh: pass validator_config_basename extra var so playbooks
    use the active (possibly expanded) config; add --check --diff in dry-run
  - ansible/playbooks/deploy-nodes.yml: use validator_config_basename to
    sync the correct config file to remote hosts
  - ansible/playbooks/prepare.yml: open port ranges for all subnet nodes
    on a host by matching entries via IP, not just hostname
  - convert-validator-config.py: fall back to httpPort for Lantern nodes
    when generating Leanpoint upstreams
  - README.md: document --subnets and --dry-run; update --prepare firewall
    table to reflect port ranges when --subnets N is active

Rules enforced by generate-subnet-config.py:
  - No two nodes on the same server may share a subnet (template validated)
  - Each subnet has exactly one node per client
  - N=1 is a no-op expansion (single-subnet baseline)
  - N capped at 5

* ansible: copy only the node's own hash-sig keys to each server

Previously both deploy-nodes.yml and copy-genesis.yml synced the entire
hash-sig-keys/ directory to every remote host, meaning every server
received every validator's sk/pk pair.

Now each playbook:
  1. Reads annotated_validators.yaml on the controller to look up the
     privkey_file entries for the node being deployed (inventory_hostname).
  2. Derives the pk filename by replacing _sk.ssz → _pk.ssz.
  3. Copies only those specific files to the target host.

A server running zeam_0 (validator_0_sk.ssz / validator_0_pk.ssz) no
longer receives validator_1_sk.ssz, validator_2_sk.ssz, etc.

* spin-node: assert exactly 1 aggregator per subnet after selection

* validator-config: add privkey for commented-out gean_0, lean_node_0, peam_0

* spin-node: derive subnet from config 'subnet' field, not node name suffix

The old suffix-based detection (ethlambda_1 → subnet 1) broke when a
config contained multiple nodes for the same client without --subnets
(e.g. ethlambda_0..4 for redundancy), incorrectly creating 5 subnets
and forcing ethlambda nodes as the sole aggregator on subnets 1-4.

Subnet membership is now read from the explicit 'subnet:' field that
generate-subnet-config.py writes for each entry. Nodes without this
field (all standard configs) default to subnet 0, so a single-subnet
deployment always selects exactly one aggregator from all active nodes
regardless of numeric suffixes in their names.

* docs: add client integration guide with link from README

* spin-node: honour pre-existing isAggregator: true when no --aggregator flag is passed

Previously the script always reset all flags and randomly re-selected an
aggregator, ignoring any manual isAggregator: true already set in the
YAML. This caused ethlambda_0 (user's choice) to be silently replaced by
ethlambda_1 (random pick).

Aggregator selection now follows a three-level priority:
  1. --aggregator <node> CLI flag
  2. Pre-existing isAggregator: true in the config (manual YAML edit)
  3. Random selection (fallback when neither is set)

The preset node is validated against the active node list. If it no
longer exists a warning is printed and random selection takes over.

* docs: clarify touch point 1 — both configs required, separate local/ansible examples

* docs: add note to contact zeam team for server IP assignment

* spin-node: fix associative array for bash 3.2 compatibility

* validator-config: use apiPort for lantern instead of httpPort

* fix: cadvisor deploy

* prepare: install jq alongside yq and docker

* fix: grandine address flag

* fix: grandine address flag ansible

* spin-node: skip aggregator selection when using --restart-client

* validator-config: enable gean_0 node

* run-ansible: derive inventory groups dynamically instead of hardcoding

The hardcoded group list (zeam_nodes, ream_nodes, ...) caused newly added
client types (e.g. gean_nodes) to never have their ansible_user updated.
This meant --useRoot was silently ignored for those nodes, causing Ansible
to SSH as the current local user (partha) instead of root, and fail.

* validator-config: add nlean_0 node

* ansible: add gean and nlean roles and wire into deploy

* docs: update adding-a-new-client guide with gean and nlean

* nlean: remove --pull=always for locally-built image

* nlean: use ghcr.io/nleaneth/nlean:latest as docker image

* fix: enable metrics flag for nlean

---------

Co-authored-by: Katya Ryazantseva <sibkatya@gmail.com>

Author: ch4r10t33r

README.md

@@ -11,7 +11,7 @@ A single command line quickstart to spin up lean node(s)
     - Uses PK's `eth-beacon-genesis` docker tool (not custom tooling)
     - Generates PQ keys based on specified configuration in `validator-config.yaml`
         - Force regen with flag `--forceKeyGen` when supplied with `generateGenesis`
-- ✅ Integrates zeam, ream, qlean, lantern, lighthouse, grandine, ethlambda
+- ✅ Integrates zeam, ream, qlean, lantern, lighthouse, grandine, ethlambda, gean, nlean, peam
 - ✅ Configure to run clients in docker or binary mode for easy development
 - ✅ Linux & Mac compatible & tested
 - ✅ Option to operate on single or multiple nodes or `all`
@@ -212,10 +212,17 @@ Every Ansible deployment automatically deploys an observability stack alongside
 15. `--prepare` verify and install the software required to run lean nodes on every remote server, and open + persist the necessary firewall ports.
    - **Ansible mode only** — fails with an error if `deployment_mode` is not `ansible`
    - Installs: `python3` (Ansible requirement), Docker CE + Compose plugin (all clients run as containers), `yq` (required by the `common` role at every deploy)
-   - Opens per-node ports (`quicPort`/UDP, `metricsPort`/TCP, `apiPort`/TCP) read from `validator-config.yaml`, plus fixed observability ports (9090, 9080, 9098, 9100). Enables `ufw` with default deny incoming (persisted across reboots).
+   - Opens per-node ports (`quicPort`/UDP, `metricsPort`/TCP, `apiPort`/TCP) read from the active validator config, plus fixed observability ports (9090, 9080, 9098, 9100). With `--subnets N`, all N nodes' port ranges are opened per host. Enables `ufw` with default deny incoming (persisted across reboots).
    - Prints a per-tool, per-host status summary (`✅ ok` / `❌ missing`) and `ufw status verbose`
-   - `--node` is not required and is ignored; all other flags are also ignored except `--sshKey` and `--useRoot`
+   - `--node` is not required; passing unsupported flags alongside `--prepare` produces a prominent error — only `--sshKey` and `--useRoot` are accepted
    - Example: `NETWORK_DIR=ansible-devnet ./spin-node.sh --prepare --sshKey ~/.ssh/id_ed25519 --useRoot`
+16. `--subnets N` expand the validator config to deploy N nodes of each client on the same server, where N is 1–5.
+   - Generates `validator-config-subnets-N.yaml` from the template (without modifying the original)
+   - Each subnet node gets a unique name (`{client}_0`, `{client}_1`, …), ports incremented by the subnet index, and a fresh P2P identity key for subnets > 0
+   - Subnet assignment rule: each server contributes **exactly one node per subnet** — nodes on the same server are never in the same subnet
+   - Every subnet contains the same set of client types
+   - `N=1` renames nodes to `{client}_0` with no port changes (useful for canonical naming)
+   - Example: `NETWORK_DIR=ansible-devnet ./spin-node.sh --node all --subnets 3 --sshKey ~/.ssh/id_ed25519 --useRoot`
 
 ### Preparing remote servers
 
@@ -237,7 +244,7 @@ NETWORK_DIR=ansible-devnet ./spin-node.sh --prepare --sshKey ~/.ssh/id_ed25519 -
 
 **Constraints:**
 - Only works in ansible mode (`deployment_mode: ansible` in your config, or `--deploymentMode ansible`)
-- Any other flags (e.g., `--node`, `--generateGenesis`) are silently ignored — only `--sshKey` and `--useRoot` are used
+- Passing unsupported flags (e.g. `--node`, `--generateGenesis`) alongside `--prepare` produces a prominent error — only `--sshKey` and `--useRoot` are accepted
 - `--node` is not required; the playbook runs on all remote hosts in the inventory
 
 Once preparation succeeds, proceed with the normal deploy command:
@@ -246,6 +253,43 @@ Once preparation succeeds, proceed with the normal deploy command:
 NETWORK_DIR=ansible-devnet ./spin-node.sh --node all --generateGenesis --sshKey ~/.ssh/id_ed25519 --useRoot
 ```
 
+### Deploying multiple subnets
+
+Use `--subnets N` to run N independent copies of each client on the same server. This is useful for testing multi-subnet P2P scenarios without provisioning additional machines.
+
+```sh
+# Deploy 3 subnets of every client (ansible)
+NETWORK_DIR=ansible-devnet ./spin-node.sh --node all --subnets 3 \
+  --generateGenesis --sshKey ~/.ssh/id_ed25519 --useRoot
+```
+
+**How it works:**
+
+`--subnets N` generates `validator-config-subnets-N.yaml` from the template (the original file is never modified). For each client in the template it creates N entries:
+
+| Subnet index | Name | quicPort | metricsPort | apiPort |
+|---|---|---|---|---|
+| 0 | `zeam_0` | base | base | base |
+| 1 | `zeam_1` | base+1 | base+1 | base+1 |
+| … | … | … | … | … |
+| N-1 | `zeam_N-1` | base+N-1 | base+N-1 | base+N-1 |
+
+**Rules enforced:**
+- `N` must be between 1 and 5
+- Each server contributes exactly one node per subnet (nodes on the same server are never in the same subnet)
+- Every subnet contains the same set of client types
+- Each node beyond subnet 0 gets a fresh P2P identity key
+
+**Running `--prepare` with subnets:**
+
+Always run `--prepare` with the same `--subnets N` value before deploying, so the firewall opens all N port ranges per host:
+
+```sh
+# Prepare firewall for 3 subnets
+NETWORK_DIR=ansible-devnet ./spin-node.sh --prepare --subnets 3 \
+  --sshKey ~/.ssh/id_ed25519 --useRoot
+```
+
 ### Checkpoint sync
 
 Checkpoint sync lets you restart clients by syncing from a remote checkpoint instead of from genesis. This is useful for joining an existing network (e.g., leanpoint mainnet) without replaying the full chain.
@@ -278,7 +322,7 @@ NETWORK_DIR=local-devnet ./spin-node.sh --restart-client zeam_0 \
 - **Local** (`NETWORK_DIR=local-devnet`): Uses Docker directly
 - **Ansible** (`NETWORK_DIR=ansible-devnet`): Uses Ansible to deploy to remote hosts
 
-**Supported clients:** zeam, ream, qlean, lantern, lighthouse, grandine, ethlambda, peam
+**Supported clients:** zeam, ream, qlean, lantern, lighthouse, grandine, ethlambda, gean, nlean, peam
 
 > **Note:** All clients accept `--checkpoint-sync-url`. Client implementations may use different parameter names internally; update client-cmd scripts if parameters change.
 
@@ -293,9 +337,13 @@ Current following clients are supported:
 5. Lighthouse
 6. Grandine
 7. Ethlambda
-8. Peam
+8. Gean
+9. Nlean
+10. Peam
+
+Adding a new client requires 6 small, well-defined steps. See the full integration guide:
 
-However adding a lean client to this setup is very easy. Feel free to do the PR or reach out to the maintainers.
+📖 **[Adding a New Client](docs/adding-a-new-client.md)**
 
 ## How It Works
 
@@ -806,7 +854,7 @@ ansible/
 │       └── all.yml           # Global variables
 ├── playbooks/
 │   ├── site.yml             # Main playbook (clean + copy genesis + deploy)
-│   ├── prepare.yml          # Bootstrap: install Docker, build-essential, yq, etc.
+│   ├── prepare.yml          # Bootstrap: install Docker CE, yq; open firewall ports
 │   ├── clean-node-data.yml  # Clean node data directories
 │   ├── generate-genesis.yml # Generate genesis files
 │   ├── copy-genesis.yml     # Copy genesis files to remote hosts
@@ -846,13 +894,13 @@ The command runs `ansible/playbooks/prepare.yml` against all remote hosts in the
 
 **Firewall rules opened (via `ufw`):**
 
-Each host's ports are read directly from `validator-config.yaml`, so only the ports actually configured for that node are opened:
+Ports are read from the active validator config (the `--subnets`-expanded file when `--subnets N` is used, or `validator-config.yaml` otherwise). Entries are matched by IP address, so all N subnet nodes on a server are found and all their ports are opened:
 
 | Port | Protocol | Source |
 |---|---|---|
-| `quicPort` | UDP | Per-node — QUIC/P2P transport (e.g. 9001) |
-| `metricsPort` | TCP | Per-node — Prometheus scrape endpoint (e.g. 9095) |
-| `apiPort` / `httpPort` | TCP | Per-node — REST API (e.g. 5055) |
+| `quicPort` … `quicPort+N-1` | UDP | Per-node — QUIC/P2P transport (e.g. 9001–9003 for N=3) |
+| `metricsPort` … `metricsPort+N-1` | TCP | Per-node — Prometheus scrape endpoint |
+| `apiPort`/`httpPort` … `+N-1` | TCP | Per-node — REST API |
 | 9090 | TCP | Observability — Prometheus |
 | 9080 | TCP | Observability — Promtail |
 | 9098 | TCP | Observability — cAdvisor |

ansible-devnet/genesis/validator-config.yaml

@@ -54,7 +54,7 @@ validators:
       ip: "65.109.131.177"
       quic: 9001
     metricsPort: 9095
-    httpPort: 5055
+    apiPort: 5055
     isAggregator: false
     count: 1
 
@@ -97,16 +97,28 @@ validators:
     isAggregator: false
     count: 1
 
-  # - name: "gean_0"
-  #   enrFields:
-  #     ip: "204.168.134.201"
-  #     quic: 9001
-  #   metricsPort: 9095
-  #   apiPort: 5055
-  #   isAggregator: false
-  #   count: 1
+  - name: "gean_0"
+    privkey: "df008e968231c25c3938d80fee9bcc93b4b9711312cf471c1b6f77e67ad68d08"
+    enrFields:
+      ip: "204.168.134.201"
+      quic: 9001
+    metricsPort: 9095
+    apiPort: 5055
+    isAggregator: false
+    count: 1
+
+  - name: "nlean_0"
+    privkey: "d94e3dc35e320440c891b66bd82d1aaf2079364162815b32c2633ecae009c84c"
+    enrFields:
+      ip: "95.216.164.165"
+      quic: 9001
+    metricsPort: 9095
+    apiPort: 5055
+    isAggregator: false
+    count: 1
 
   # - name: "lean_node_0"
+  #   privkey: "d94e3dc35e320440c891b66bd82d1aaf2079364162815b32c2633ecae009c84c"
   #   enrFields:
   #     ip: "95.217.19.42"
   #     quic: 9001

ansible/playbooks/copy-genesis.yml

@@ -153,20 +153,31 @@
       loop_control:
         label: "{{ item }}.key"
 
+    - name: Resolve hash-sig key files for this node
+      vars:
+        _av: "{{ lookup('file', genesis_dir + '/annotated_validators.yaml') | from_yaml }}"
+        _assignments: "{{ _av[inventory_hostname] | default([]) }}"
+        _sk_files: "{{ _assignments | map(attribute='privkey_file') | list }}"
+        _pk_files: "{{ _sk_files | map('regex_replace', '_sk\\.ssz$', '_pk.ssz') | list }}"
+      set_fact:
+        node_hash_sig_files: "{{ _sk_files + _pk_files }}"
+      when: hash_sig_keys_stat.stat.exists
+
     - name: Create hash-sig-keys directory on remote
       file:
         path: "{{ actual_remote_genesis_dir }}/hash-sig-keys"
         state: directory
         mode: '0755'
-      when: hash_sig_keys_stat.stat.exists
+      when: hash_sig_keys_stat.stat.exists and (node_hash_sig_files | default([]) | length > 0)
 
-    - name: Copy hash-sig-keys directory to remote host
+    - name: Copy hash-sig key files for this node only
       copy:
-        src: "{{ genesis_dir }}/hash-sig-keys/"
-        dest: "{{ actual_remote_genesis_dir }}/hash-sig-keys/"
-        mode: '0644'
+        src: "{{ genesis_dir }}/hash-sig-keys/{{ item }}"
+        dest: "{{ actual_remote_genesis_dir }}/hash-sig-keys/{{ item }}"
+        mode: '0600'
         force: yes
-      when: hash_sig_keys_stat.stat.exists
+      loop: "{{ node_hash_sig_files | default([]) }}"
+      when: hash_sig_keys_stat.stat.exists and (node_hash_sig_files | default([]) | length > 0)
 
     - name: List files on remote genesis directory
       find:

ansible/playbooks/deploy-nodes.yml

@@ -8,7 +8,7 @@
 #   - node key files (*.key)
 #   - config.yaml, validators.yaml, nodes.yaml
 #   - genesis.ssz, genesis.json
-#   - hash-sig-keys/ directory (if exists, for qlean nodes)
+#   - hash-sig-keys/ directory (if exists): only the sk/pk files for this node's validators
 
 - name: Parse and validate node names
   hosts: localhost
@@ -122,7 +122,10 @@
 
     - name: Sync validator-config.yaml to remote host
       copy:
-        src: "{{ local_genesis_dir }}/validator-config.yaml"
+        # Use the expanded subnet config when --subnets was specified; fall back
+        # to the standard validator-config.yaml otherwise. The destination is
+        # always validator-config.yaml so client roles don't need to change.
+        src: "{{ local_genesis_dir }}/{{ validator_config_basename | default('validator-config.yaml') }}"
         dest: "{{ genesis_dir }}/validator-config.yaml"
         mode: '0644'
         force: yes
@@ -165,23 +168,37 @@
         - deploy
         - sync
 
+    - name: Resolve hash-sig key files for this node
+      vars:
+        _av: "{{ lookup('file', local_genesis_dir + '/annotated_validators.yaml') | from_yaml }}"
+        _assignments: "{{ _av[node_name] | default([]) }}"
+        _sk_files: "{{ _assignments | map(attribute='privkey_file') | list }}"
+        _pk_files: "{{ _sk_files | map('regex_replace', '_sk\\.ssz$', '_pk.ssz') | list }}"
+      set_fact:
+        node_hash_sig_files: "{{ _sk_files + _pk_files }}"
+      when: hash_sig_keys_local.stat.exists
+      tags:
+        - deploy
+        - sync
+
     - name: Create hash-sig-keys directory on remote
       file:
         path: "{{ genesis_dir }}/hash-sig-keys"
         state: directory
         mode: '0755'
-      when: hash_sig_keys_local.stat.exists
+      when: hash_sig_keys_local.stat.exists and (node_hash_sig_files | default([]) | length > 0)
       tags:
         - deploy
         - sync
 
-    - name: Sync hash-sig-keys directory (for qlean nodes)
+    - name: Copy hash-sig key files for this node only
       copy:
-        src: "{{ local_genesis_dir }}/hash-sig-keys/"
-        dest: "{{ genesis_dir }}/hash-sig-keys/"
-        mode: '0644'
+        src: "{{ local_genesis_dir }}/hash-sig-keys/{{ item }}"
+        dest: "{{ genesis_dir }}/hash-sig-keys/{{ item }}"
+        mode: '0600'
         force: yes
-      when: hash_sig_keys_local.stat.exists
+      loop: "{{ node_hash_sig_files | default([]) }}"
+      when: hash_sig_keys_local.stat.exists and (node_hash_sig_files | default([]) | length > 0)
       tags:
         - deploy
         - sync
@@ -199,6 +216,8 @@
         - lighthouse
         - grandine
         - ethlambda
+        - gean
+        - nlean
         - peam
         - deploy
 

ansible/playbooks/helpers/deploy-single-node.yml

@@ -86,6 +86,22 @@
     - ethlambda
     - deploy
 
+- name: Deploy Gean node
+  include_role:
+    name: gean
+  when: client_type == "gean"
+  tags:
+    - gean
+    - deploy
+
+- name: Deploy Nlean node
+  include_role:
+    name: nlean
+  when: client_type == "nlean"
+  tags:
+    - nlean
+    - deploy
+
 - name: Deploy Peam node
   include_role:
     name: peam
@@ -96,5 +112,5 @@
 
 - name: Fail if unknown client type
   fail:
-    msg: "Unknown client type '{{ client_type }}' for node '{{ node_name }}'. Expected: zeam, ream, qlean, lantern, lighthouse, grandine, ethlambda or peam"
-  when: client_type not in ["zeam", "ream", "qlean", "lantern", "lighthouse", "grandine", "ethlambda", "peam"]
+    msg: "Unknown client type '{{ client_type }}' for node '{{ node_name }}'. Expected: zeam, ream, qlean, lantern, lighthouse, grandine, ethlambda, gean, nlean or peam"
+  when: client_type not in ["zeam", "ream", "qlean", "lantern", "lighthouse", "grandine", "ethlambda", "gean", "nlean", "peam"]