HashMap.Set is ineffective on BPF_MAP_TYPE_HASH declared in XDP program

[yoursunny]

I'm trying to set an entry in a hash map that is declared in an XDP program loaded by a C program.
However, the entry set is ineffective: the entry is not visible to the XDP program or bpftool; gobpfld cannot see the entry if the map is closed then re-opened.

Environment

• Ubuntu 18.04
• Linux kernel 5.4.0-100-generic, installed via linux-generic-hwe-18.04 package
• libxdp commit 971bb03 (2022-02-28)
• clang-11
• Mellanox ConnectX-5 Ethernet adapter

Source Code

Each file must be placed in a separate directory.

xdp.c

#include <stdint.h>

#include <linux/bpf.h>

#include <bpf/bpf_endian.h>
#include <bpf/bpf_helpers.h>

struct
{
  __uint(type, BPF_MAP_TYPE_XSKMAP);
  __uint(max_entries, 64);
  __type(key, int32_t);
  __type(value, int32_t);
} xsks_map SEC(".maps");

struct
{
  __uint(type, BPF_MAP_TYPE_HASH);
  __uint(max_entries, 64);
  __type(key, int32_t);
  __type(value, int32_t);
} face_map SEC(".maps");

SEC("xdp_sock") int xdp_sock_prog(struct xdp_md* ctx)
{
  int32_t key = 0x33333333;
  int32_t* queue = bpf_map_lookup_elem(&face_map, &key);
  if (queue == NULL) {
    return XDP_DROP;
  }
  return bpf_redirect_map(&xsks_map, *queue, XDP_PASS);
}

loader.c

#include <stdbool.h>
#include <stdlib.h>
#include <stdint.h>
#include <stdio.h>

#include <xdp/libxdp.h>

int main(int argc, char** argv)
{
  if (argc != 2) {
    printf("USAGE: %s IFINDEX\n", argv[0]);
    return 1;
  }
  int ifindex = atoi(argv[1]);

  struct xdp_program* prog = xdp_program__open_file("../bpf/xdp.o", "xdp_sock", NULL);
  int err = xdp_program__attach(prog, ifindex, XDP_MODE_NATIVE, 0);
  if (err) {
    printf("xdp_program__attach %d\n", err);
    return 1;
  }
  return 0;
}

main.go

package main

import (
	"flag"
	"fmt"

	"github.com/dylandreimerink/gobpfld"
	"github.com/dylandreimerink/gobpfld/bpfsys"
)

var mid = flag.Int("map", 0, "BPF map id")

func main() {
	flag.Parse()

	m, e := gobpfld.MapFromID(uint32(*mid))
	if e != nil {
		panic(e)
	}
	h := m.(*gobpfld.HashMap)
	if e := h.Load(); e != nil {
		panic(e)
	}
	setKey, setValue := int32(0x33333333), int32(0)
	if e := h.Set(&setKey, &setValue, bpfsys.BPFMapElemAny); e != nil {
		panic(e)
	}
	gobpfld.MapIterForEach(h.Iterator(), int32(0), int32(0), func(key, value interface{}) error {
		fmt.Println("IterA", *key.(*int32), *value.(*int32))
		return nil
	})
	fmt.Println("IterA end")
	h.Close()

	m, e = gobpfld.MapFromID(uint32(*mid))
	if e != nil {
		panic(e)
	}
	h = m.(*gobpfld.HashMap)
	if e := h.Load(); e != nil {
		panic(e)
	}
	gobpfld.MapIterForEach(h.Iterator(), int32(0), int32(0), func(key, value interface{}) error {
		fmt.Println("IterB", *key.(*int32), *value.(*int32))
		return nil
	})
	fmt.Println("IterB end")
	h.Close()
}

Steps to Reproduce

1. Compile the programs:

   clang-11 -g -O2 -target bpf -c xdp.c -o xdp.o
   clang-11 -g -O2 loader.c -o loader $(pkg-config --cflags --libs libxdp)
   go build main.go

2. Load the XDP program to a network interface:

   IFNAME=enp94s0
   IFINDEX=3
   sudo ip link set $IFNAME xdp off
   sudo ./loader $IFINDEX ../bpf/xdp.o

3. Find the id of the hash map:

   $ sudo bpftool prog | tail
   19: cgroup_skb  tag 2a142ef67aaad174  gpl
           loaded_at 2022-03-02T18:25:24+0000  uid 0
           xlated 296B  jited 200B  memlock 4096B  map_ids 18,19
   22: xdp  name xsk_bypass  tag 03b13f331978c78c  gpl
           loaded_at 2022-03-08T14:06:23+0000  uid 0
           xlated 56B  jited 57B  memlock 4096B  map_ids 22
   80: xdp  name xdp_sock_prog  tag b17bd206a4a38408
           loaded_at 2022-03-08T14:51:50+0000  uid 0
           xlated 144B  jited 107B  memlock 4096B  map_ids 62,63
           btf_id 30
   
   $ sudo bpftool map | tail
           key 8B  value 8B  max_entries 1  memlock 4096B
   19: lpm_trie  flags 0x1
           key 20B  value 8B  max_entries 1  memlock 4096B
   22: xskmap  name xskmap  flags 0x0
           key 4B  value 4B  max_entries 5  memlock 4096B
   62: hash  name face_map  flags 0x0
           key 4B  value 4B  max_entries 64  memlock 8192B
           btf_id 30
   63: xskmap  name xsks_map  flags 0x0
           key 4B  value 4B  max_entries 64  memlock 4096B

4. Run the Go program to set an entry in the hash map:

   $ sudo ./main -map 62
   IterA 858993459 0
   IterA end
   IterB end

5. View the hash map via bpftool:

   $ sudo bpftool map dump id 62
   []

Expected Behavior

• The Go program should print IterB 858993459 0, which means the entry is presented after the map is closed and reopened.
• bpftool should print the entry.
• If the network interface receives traffic, it should be visible to tcpdump on the network interface, because there's no XSK socket attached so that the XDP program would return XDP_PASS after finding an entry in the map.

Actual Behavior

• The Go program output suggests that the entry is absent after the map is closed and reopened. Note that the XDP program has a reference to the map, so that closing the map in the Go program should not cause the kernel to free the map.
• bpftool shows the hash map is empty.
• If the network interface receives traffic, it is dropped, which means the XDP program cannot find an entry in the map so that it returns XDP_DROP.
• Also, bpftool map update command could successfully create an entry in this map.