Bootnodes discovery in cloud

[tsamsiyu]

Description

I’m trying to connect multiple Besu nodes of 2 organisations into a private network. Each organisation runs nodes in their cloud environment in Kubernetes. However, it seems impossible to configure bootnode discovery between them.

After some debugging I can see that during the bonding process, the node receiving the request attempts to respond to a port extracted from a UDP datagram (while it correctly retrieves the TCP port from the request data)

The root of the problem here is that in cloud environment it's a common practice to have different egress/ingress networks

Reference: https://github.com/hyperledger/besu/blob/main/ethereum/p2p/src/main/java/org/hyperledger/besu/ethereum/p2p/discovery/PeerDiscoveryAgent.java#L288

Acceptance Criteria

1. when a Node A connects to a Node B it shares its P2P IP, P2P TCP port, Discovery UDP port (already works this way)
2. Node B accepts bonding request by responding to a UDP discovery port mentioned in the data package

[jflo]

Have you tried using the internal (private) Kubernetes IP and port in your bootnode list in the genesis file?
Have you tried specifying an --p2p-host, and if so, why is that insufficient for your network topology?
Can you post the configuration being used for both nodes, including the genesis?

[tsamsiyu]

When I connect nodes internally within a cluster, bootnode discovery works well since there is no cloud egress gateway in a such scenario. However, in my case, I need to connect nodes running in different clusters. To achieve this, I expose their P2P and discovery ports via a load balancer on each side.

here's the example of how I expose ports:

// cluster A
LB TCP 30001 -> node 1 p2p
LB UDP 30002 -> node 1 discovery
LB TCP 30003 -> node 2 p2p
LB UDP 30004 -> node 2 discovery

• yes, I use p2p-host, it's equal to the static IP assigned to a load balancer
• If I connect nodes in different cluster without discovery but just with the static-nodes list, they sync successfully

the problem is that when node (1) of cluster A connects via UDP to node (2) of cluster B, the node (2) can't reach the sender of "PING" as it's sending "PONG" packet to the wrong address (cloud egress address)

[tsamsiyu]

    {
      "nonce": "0x0",
      "timestamp": "0x58ee40ba",
      "extraData": "0xf87aa00000000000000000000000000000000000000000000000000000000000000000f85494c20680c38137a1af424850169158fd230a8ef7d2947e13e5d752d9d4ce4b43549b370486955d679ff19457d9a3330593d0cbce09e27eaee7843b0f0af01a94d96786bf2bef3a726ca3be3932fce85c1a23ad65c080c0",
      "gasLimit": "0xFFFFFF",
      "gasUsed": "0x0",
      "number": "0x0",
      "difficulty": "0x1",
      "coinbase": "0x0000000000000000000000000000000000000000",
      "mixHash": "0x63746963616c2062797a616e74696e65206661756c7420746f6c6572616e6365",
      "parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
      "config": {
        "chainId": 1337,
        "homesteadBlock": 0,
        "eip150Block": 0,
        "eip150Hash": "0x0000000000000000000000000000000000000000000000000000000000000000",
        "eip155Block": 0,
        "eip158Block": 0,
        "byzantiumBlock": 0,
        "constantinopleBlock": 0,
        "petersburgBlock": 0,
        "istanbulBlock": 0,
        "muirglacierblock": 0,
        "berlinBlock": 0,
        "londonBlock": 0,
        "arrowGlacierBlock": 0,
        "grayGlacierBlock": 0,
        "zeroBaseFee": true,
        "qbft": {
          "blockperiodseconds": 15,
          "epochlength": 30000,
          "requesttimeoutseconds": 10
        }
      },
      "alloc": {
        "0x4b14b0061942d030b6027e82b948b2630f91e26b": {
          "balance": "1000000000000000000000000000"
        },
        "0x138f1f5c05e04a4d7af601d37ecaf58a972c2f78": {
          "balance": "1000000000000000000000000000"
        },
        "0xf29c13b63f58fa74a1116d6c0fcd68d9e112234a": {
          "balance": "1000000000000000000000000000"
        },
        "0xe28a8b67a7a40f9c78fbffa7a3ed3941b6d32b37": {
          "balance": "1000000000000000000000000000"
        },
        "0x79405f3a3bc26961d31b0f5ba7137e8efc90760d": {
          "balance": "1000000000000000000000000000"
        },
        "0xbb027cf2106b3a77767326756b5f1208c2f90994": {
          "balance": "1000000000000000000000000000"
        },
        "0x95e7cc48c2932d184883d949aecbb76a5d656e68": {
          "balance": "1000000000000000000000000000"
        }
      }
    }

Organisation A Node 1 config

    bootnodes=["enode://519a84c1720f58753b2614f3df5468d5c6b4b0cfa943bcf336d7613fde3f3360d22d47d838b15a7a68843fff0973ee3165254e6bd3d3c73e8f6e880de1268933@{{LOAD_BALANCER_ORG_B_IP}}:30001"]
    #static-nodes-file="/etc/besu/staticnodes/static-nodes.json"
    data-path="/etc/besu/data/storage"
    genesis-file="/etc/besu/genesis/genesis.json"
    node-private-key-file="/etc/besu/identity/key"
    min-gas-price=0
    nat-method="NONE"
    p2p-enabled=true
    p2p-host="{{LOAD_BALANCER_ORG_A_IP}}"
    p2p-port=30303
    host-allowlist=[ "*" ]
    logging="TRACE"
    max-peers=20
    rpc-http-enabled=true
    rpc-http-api=[ "DEBUG", "ETH", "QBFT", "NET", "ADMIN" ]
    rpc-http-authentication-enabled=false
    sync-mode="FULL"
    data-storage-format="FOREST"
    permissions-accounts-config-file-enabled=false
    permissions-accounts-config-file="/etc/besu/permissions/accounts.toml"
    permissions-nodes-config-file-enabled=false
    permissions-nodes-config-file="/etc/besu/permissions/nodes.toml"

[tsamsiyu]

I just realised there’s one more issue related to this.
Even if a node that receives a PING packet sends a PONG response using the correct port, it won’t be able to reach the sender.
This is because the sender's load balancer’s UDP port is different from the TCP but the receiver would still send PONG packet back to the TCP port

Visual representation of the problem (let's assume we have node 1 running in cluster A and node 2 running in cluster B):

node 1 config:

p2p-host={{CLUSTER_A_LB_IP}}
p2p-port=30001

let's say we have the following cluster routing:

Load balancer TCP 30001 -> node 1 30001 TCP (p2p-port)
Load balancer UDP 30002 -> node 1 30001 UDP (p2p-port)

In k8s we can't have single load balancer port for both TCP and UDP protocols

assuming 'PING-recipient' node extracts 'PING-sender' node's port correctly from the packet data, still for the PONG request to reach the destination, load balancer's port MUST match node's p2p-port, i.e. we should have routing Load balancer UDP 30002 -> node 1 30002 UDP, which is currently impossible to do

Discovery flow assuming node 2 sends PONG request to the port from a PING packet data:

1) Node 1   --- PING --->  Node 2
         url={{CLUSTER_B_LB_IP}}:30001
         tcpPort=30001
         udpPort=30001
         host={{CLUSTER_A_LB_IP}}
2) Node 2   ---  PONG ---> Node 1
         url={{pingPacket.host}}:{{pingPacket.udpPort}}
         tcpPort=30001
         udpPort=30001
         host={{CLUSTER_B_LB_IP}}
     >>> Error! Node 1 listens for UDP connections on a different port (30002)

The solution would be to allow setting discovery-port config option