How to work with BPF_MAP_TYPE_ARRAY_OF_MAPS?

[thediveo]

Say, we have...

struct FOO
{
    __u64 foo;
    __u64 bar;
};

struct {
    __uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
    __uint(max_entries, 8192);
    __type(key, __u32);
    __array(values, struct FOO);
} baz SEC(".maps");

...how do I correctly load and set up things on the Go side? I've tried to find examples, but had hits only on testdata and unfortunately can't see how to do it.

Are the inner maps automatically created for all entries of the outer map or do I need to allocate them myself?

Is there a way to resize the outer map at creation time?

Any help greatly appreciated!

[ti-mo]

@thediveo Could you elaborate on what you want to accomplish on the Go side? I assume you want to create new inner maps and store them in the outer map? Either way, the example we have in testdata is quite representative, but this is an alternative version that doesn't explicitly declare the inner map as a separate map in the ELF:

struct {
	__uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
	__type(key, uint32_t);
	__type(value, uint32_t);
	__uint(max_entries, 16);
	__array(values, struct {
		__uint(type, BPF_MAP_TYPE_HASH);
		__type(key, uint64_t);
		__type(value, uint64_t);
		__uint(max_entries, 1024);
	});
} outer_map __section(".maps");

Are the inner maps automatically created for all entries of the outer map or do I need to allocate them myself?

Reading through https://docs.kernel.org/bpf/map_of_maps.html may be helpful, it explains the difference between HASH_OF_MAPS and ARRAY_OF_MAPS. tl;dr: the inner maps themselves (not the array/hash data structures powering the outer map!) are never preallocated. All map creations are performed by user space. A shorthand for this is using .values like in btf_map_init.c to automatically insert pointers to other maps defined in the ELF into the outer map.

Is there a way to resize the outer map at creation time?

The outer map definition can be found in CollectionSpec.Maps before loading and can be modified like any other map.

To build on my previous point: here's how to create more instances of the inner map and insert them into the outer map:

spec, _ := LoadCollectionSpec(file)
inner := spec.Maps["outer_map"].InnerMap

coll, _ := NewCollection(spec)

for i := range 10 {
  m, _ := NewMap(inner)
  _ = coll.Maps["outer_map"].Put(uint32(i), m)
}

This will instantiate 10 inner maps and insert them into outer_map at indices 0 through 9.

Hope this helped!

[thediveo]

The error comes from

ebpf/btf/types.go

Line 643 in 9124d6f

func Sizeof(typ Type) (int, error) {

when it hits the default switch case, when it doesn't know what to do with the *btf.Fwd.

[ti-mo]

Huh, interesting. Looks like the value type declaration of the inner map can't resolve struct FOO. That's not an issue with the lib, rather a limitation of the macros.

In that case, define your inner map declaration as a type only (struct inner { .. }) without declaring it as a variable (struct inner { ... } inner_map SEC(".maps"), like so:

struct foo {
	uint64_t foo;
};

struct inner {
	__uint(type, BPF_MAP_TYPE_HASH);
	__type(key, uint64_t);
	__type(value, struct foo);
	__uint(max_entries, 1024);
};

struct {
	__uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
	__type(key, uint32_t);
	__type(value, struct foo);
	__uint(max_entries, 16);
	__array(values, struct inner);
} outer_map __section(".maps");

This works for me, at least it gets rid of the FWD in the BTF by resolving the definition of struct foo.

[thediveo]

I'm not telling horror stories, I am your horror story (unfortunately).

I've rebuild my devcontainer to be sure; Go 1.24.1, go install github.com/cilium/ebpf/cmd/bpf2go@latest, go.mod has github.com/cilium/ebpf v0.17.3. Kernel is 6.11.0-17-generic, devcontainer base image is Ubuntu 24.04.1LTS.

Looking at your example and mine I still cannot see a difference in the source, __u64 shouldn't matter I assume.

struct foo {
    __u64 foo;
};

struct inner {
    __uint(type, BPF_MAP_TYPE_HASH);
    __type(key, __u64);
    __type(value, struct foo);
    __uint(max_entries, 1024);
};


struct {
    __uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
    __type(key, __u32);
    __type(value, struct foo);
    __uint(max_entries, 16);
    __array(values, struct inner);
} outer_map SEC(".maps");

gives this error:

load BTF maps: map outer_map: value_size must be 0 or 4

[thediveo]

considering that __array(values, struct inner); does a typeof(struct inner) *values[] I wonder how on a 64bit platform this can ever be 4 byte pointers...?!

Also, the values are considered to be file descriptors and thus the restriction to exactly 4 bytes per value in case of the outer map.

Using __type(value, __u32); on the outer map brings me back to square one 🎉:

load BTF maps: map outer_map: can't parse BTF map definition of inner map: can't get size of BTF value: type *btf.Fwd: type is unsized

[ti-mo]

Sorry, as you've noticed, that __type(value, struct foo); on the outer map should've been __type(value, __u32); (or omitted altogether).

Interesting, looks like we've hit a compiler bug. For some reason, struct foo doesn't get included in btf when it's only referred to by an inner map definition:

struct foo {
	uint64_t foo;
};

// Either of the following will work:
// Dummy variable of struct foo.
__hidden struct foo bar;

// Dummy function taking struct foo.
int dummy(struct foo bar) {
	return 0;
}

struct {
	__uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
	__type(key, uint32_t);
	__type(value, uint32_t);
	__uint(max_entries, 16);
	__array(
		values, struct {
			__uint(type, BPF_MAP_TYPE_HASH);
			__type(key, uint64_t);
			__type(value, struct foo);
			__uint(max_entries, 1024);
		});
} outer_map __section(".maps");

Typically, an ELF would contain at least one function that takes a struct foo, which takes care of creating a reference to the type, causing it to be included in BTF and avoiding the FWD. Without that, looks like you need a dummy var or func.

Might be worth reporting to the LLVM project (https://github.com/llvm/llvm-project/issues) and/or the netdev mailing list: https://docs.kernel.org/bpf/bpf_devel_QA.html#q-reporting-llvm-bpf-issues.