Skip to content

Task local data #9315

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 4 commits into
base: bpf-next_base
Choose a base branch
from
Open

Task local data #9315

wants to merge 4 commits into from
+986 −0

Conversation

kernel-patches-daemon-bpf[bot]
Copy link

Pull request for series with
subject: Task local data
version: 6
url: https://patchwork.kernel.org/project/netdevbpf/list/?series=983442

@kernel-patches-daemon-bpf
Copy link
Author

Upstream branch: 0768e98
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=983442
version: 6

@kernel-patches-daemon-bpf
Copy link
Author

Upstream branch: 0768e98
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=983442
version: 6

@kernel-patches-daemon-bpf
Copy link
Author

Upstream branch: 0ee30d9
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=983442
version: 6

@kernel-patches-daemon-bpf
Copy link
Author

Upstream branch: 0ee30d9
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=983442
version: 6

@kernel-patches-daemon-bpf
Copy link
Author

Upstream branch: beb1097
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=983442
version: 6

@kernel-patches-daemon-bpf
Copy link
Author

Upstream branch: 42be23e
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=983442
version: 6

@kernel-patches-daemon-bpf
Copy link
Author

Upstream branch: 42be23e
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=983442
version: 6

@kernel-patches-daemon-bpf
Copy link
Author

Upstream branch: 95993dc
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=983442
version: 6

@kernel-patches-daemon-bpf
Copy link
Author

Upstream branch: 9ea0691
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=983442
version: 6

@kernel-patches-daemon-bpf
Copy link
Author

Upstream branch: 5345e64
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=983442
version: 6

@kernel-patches-daemon-bpf
Copy link
Author

Upstream branch: 5b4c54a
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=983442
version: 6

@kernel-patches-daemon-bpf
Copy link
Author

Upstream branch: 5b4c54a
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=983442
version: 6

@kernel-patches-daemon-bpf
Copy link
Author

Upstream branch: 5b4c54a
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=983442
version: 6

@kernel-patches-daemon-bpf
Copy link
Author

Upstream branch: cd7c97f
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=983442
version: 6

@kernel-patches-daemon-bpf
Copy link
Author

Upstream branch: cd7c97f
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=987135
version: 7

@kernel-patches-daemon-bpf
Copy link
Author

Upstream branch: e8d780d
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=987135
version: 7

ameryhung added 4 commits July 31, 2025 11:39
@ameryhung
bpf: Allow syscall bpf programs to call non-recur helpers
fa1467b 
Allow syscall programs to call non-recur helpers too since syscall bpf
programs runs in process context through bpf syscall, BPF_PROG_TEST_RUN,
and cannot run recursively.

bpf_task_storage_{get,set} have "_recur" versions that call trylock
instead of taking the lock directly to avoid deadlock when called by
bpf programs that run recursively. Currently, only bpf_lsm, bpf_iter,
struct_ops without private stack are allow to call the non-recur helpers
since they cannot be recursively called in another bpf program.

Signed-off-by: Amery Hung <[email protected]>
Reviewed-by: Emil Tsalapatis <[email protected]>
@ameryhung
selftests/bpf: Introduce task local data
62b5975 
Task local data defines an abstract storage type for storing task-
specific data (TLD). This patch provides user space and bpf
implementation as header-only libraries for accessing task local data.

Task local data is a bpf task local storage map with two UPTRs:

- tld_meta_u, shared by all tasks of a process, consists of the total
  count and size of TLDs and an array of metadata of TLDs. A TLD
  metadata contains the size and name. The name is used to identify a
  specific TLD in bpf programs.

- u_tld_data points to a task-specific memory. It stores TLD data and
  the starting offset of data in a page.

  Task local design decouple user space and bpf programs. Since bpf
  program does not know the size of TLDs in compile time, u_tld_data
  is declared as a page to accommodate TLDs up to a page. As a result,
  while user space will likely allocate memory smaller than a page for
  actual TLDs, it needs to pin a page to kernel. It will pin the page
  that contains enough memory if the allocated memory spans across the
  page boundary.

The library also creates another task local storage map, tld_key_map,
to cache keys for bpf programs to speed up the access.

Below are the core task local data API:

                   User space                          BPF
  Define TLD       TLD_DEFINE_KEY(), tld_create_key()  -
  Init TLD object  -                                   tld_object_init()
  Get TLD data     tld_get_data()                      tld_get_data()

- TLD_DEFINE_KEY(), tld_create_key()

  A TLD is first defined by the user space with TLD_DEFINE_KEY() or
  tld_create_key(). TLD_DEFINE_KEY() defines a TLD statically and
  allocates just enough memory during initialization. tld_create_key()
  allows creating TLDs on the fly, but has a fix memory budget,
  TLD_DYN_DATA_SIZE.

  Internally, they all call __tld_create_key(), which iterates
  tld_meta_u->metadata to check if a TLD can be added. The total TLD
  size needs to fit into a page (limit of UPTR), and no two TLDs can
  have the same name. If a TLD can be added, u_tld_meta->cnt is
  increased using cmpxchg as there may be other concurrent
  __tld_create_key(). After a successful cmpxchg, the last available
  tld_meta_u->metadata now belongs to the calling thread. To prevent
  other threads from reading incomplete metadata while it is being
  updated, tld_meta_u->metadata->size is used to signal the completion.

  Finally, the offset, derived from adding up prior TLD sizes is then
  encapsulated as an opaque object key to prevent user misuse. The
  offset is guaranteed to be 8-byte aligned to prevent load/store
  tearing and allow atomic operations on it.

- tld_get_data()

  User space programs can pass the key to tld_get_data() to get a
  pointer to the associated TLD. The pointer will remain valid for the
  lifetime of the thread.

  tld_data_u is lazily allocated on the first call to tld_get_data().
  Trying to read task local data from bpf will result in -ENODATA
  during tld_object_init(). The task-specific memory need to be freed
  manually by calling tld_free() on thread exit to prevent memory leak
  or use TLD_FREE_DATA_ON_THREAD_EXIT.

- tld_object_init() (BPF)

  BPF programs need to call tld_object_init() before calling
  tld_get_data(). This is to avoid redundant map lookup in
  tld_get_data() by storing pointers to the map values on stack.
  The pointers are encapsulated as tld_object.

  tld_key_map is also created on the first time tld_object_init()
  is called to cache TLD keys successfully fetched by tld_get_data().

  bpf_task_storage_get(.., F_CREATE) needs to be retried since it may
  fail when another thread has already taken the percpu counter lock
  for the task local storage.

- tld_get_data() (BPF)

  BPF programs can also get a pointer to a TLD with tld_get_data().
  It uses the cached key in tld_key_map to locate the data in
  tld_data_u->data. If the cached key is not set yet (<= 0),
  __tld_fetch_key() will be called to iterate tld_meta_u->metadata
  and find the TLD by name. To prevent redundant string comparison
  in the future when the search fail, the tld_meta_u->cnt is stored
  in the non-positive range of the key. Next time, __tld_fetch_key()
  will be called only if there are new TLDs and the search will start
  from the newly added tld_meta_u->metadata using the old
  tld_meta_u-cnt.

Signed-off-by: Amery Hung <[email protected]>
Reviewed-by: Emil Tsalapatis <[email protected]>
@ameryhung
selftests/bpf: Test basic task local data operations
a31f432 
Test basic operations of task local data with valid and invalid
tld_create_key().

For invalid calls, make sure they return the right error code and check
that the TLDs are not inserted by running tld_get_data("
value_not_exists") on the bpf side. The call should a null pointer.

For valid calls, first make sure the TLDs are created by calling
tld_get_data() on the bpf side. The call should return a valid pointer.

Finally, verify that the TLDs are indeed task-specific (i.e., their
addresses do not overlap) with multiple user threads. This done by
writing values unique to each thread, reading them from both user space
and bpf, and checking if the value read back matches the value written.

Signed-off-by: Amery Hung <[email protected]>
Reviewed-by: Emil Tsalapatis <[email protected]>
@ameryhung
selftests/bpf: Test concurrent task local data key creation
5c1000f 
Test thread-safety of tld_create_key(). Since tld_create_key() does
not rely on locks but memory barriers and atomic operations to protect
the shared metadata, the thread-safety of the function is non-trivial.
Make sure concurrent tld_key_create(), both valid and invalid, can not
race and corrupt metatada, which may leads to TLDs not being thread-
specific or duplicate TLDs with the same name.

Signed-off-by: Amery Hung <[email protected]>
Reviewed-by: Emil Tsalapatis <[email protected]>
@kernel-patches-daemon-bpf
Copy link
Author

Upstream branch: e8d780d
series: https://patchwork.kernel.org/project/netdevbpf/list/?series=987135
version: 7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
bpf-next new V7-ci-pass V7
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@ameryhung