Fix compatibility with Linux kernel 6.15–6.16 ABI/API changes by sheviks · Pull Request #302 · Xilinx-CNS/onload

sheviks · 2025-08-10T04:41:34Z

This patch is an extension of issue #293, addressing additional ABI/API changes required to successfully build against kernel 6.15+.

In addition to the kbuild ccflags-y issue reported in #293, this update resolves several compatibility problems introduced in recent upstream kernel changes:

Renamed page->index to page->__folio_index
Upstream commit: acc53a0
UNIXCB() usage has been restricted to internal use
Upstream commit: 84960bf
Replaced d_hash_and_lookup() with try_lookup_noperm()
Upstream commit: 06c5674
Corrected EFRM_HAVE_OLD_FAULT usage: vm_operations_struct::fault() has consistently been a single-parameter function; EFRM_HAVE_NEW_FAULT remains valid
Upstream commit: 1c8f422
Resolved Clang build error
When building via env LLVM=1 scripts/onload_build --kernelver <clang-compiled kernel>, compilation would fail due to unsupported Clang extension:
error: fields must have a constant size: variable length array in structure
The affected #if conditions have now been updated to use valid macros.

This patch introduces some new kernel_compact tests and can be merged independently. However, to fully build on Linux 6.15+, the kbuild adjustments from issue #293 are still required.

… structure Building with 'scripts/onload_build --kernelver <clang-compiled kernel>' would fail due to unsupported Clang extension: 'error: fields must have a constant size: variable length array in structure'. This commit updates the affected #if conditions to use valid macros.

- Rename `page->index` to `page->__folio_index` [acc53a0](torvalds/linux@acc53a0) - `UNIXCB()` usage is now restricted to internal only as per upstream change [84960bf](torvalds/linux@84960bf) - Replace `d_hash_and_lookup()` with `try_lookup_noperm()` [06c5674](torvalds/linux@06c5674) - Fix `EFRM_HAVE_OLD_FAULT`; `vm_operations_struct::fault()` has always been single-parameter, and `EFRM_HAVE_NEW_FAULT` remains valid [1c8f422](torvalds/linux@1c8f422)

sheviks · 2025-08-10T15:55:23Z

I've applied @lukester1975's kbuild ccflags-y patch locally (with a small tweak for Clang compatibility).
I tested the build successfully on four kernel versions:

6.16.0 compiled with GCC
6.16.0 compiled with Clang
6.12.40 compiled with GCC
6.12.36 compiled with Clang

If needed, I can commit and push the changes so they can be included and await merge.

Thanks!

sheviks · 2025-08-14T09:11:09Z

I just realized that when I initially addressed the UNIXCB() issue, I missed an important detail.

When copying struct unix_skb_parms, I didn’t notice that its definition ends with __randomize_layout.
This means the structure layout may vary across different kernels (although most distributions disable this feature for compatibility).

The function using UNIXCB() also contains this comment:
Sadly kernel_recvmsg doesn't offer any way to extract files from a unix socket, so we'll need to dig into the socket buffer to find them.

This indicates that using UNIXCB() is essentially a forced workaround.
I searched through the latest kernel source and couldn’t find any alternative API equivalent to retrieving UNIXCB(skb).fp.

If __randomize_layout is enabled, I can think of two methods to accurately get the layout of struct unix_skb_parms:

Using bpftool
bpftool btf dump file /lib/modules/<kernelver>/build/vmlinux format c > vmlinux.h
Then extract struct unix_skb_parms from vmlinux.h
Using pahole
pahole -C unix_skb_parms /lib/modules/<kernelver>/build/vmlinux

Both methods require the kernel to be built with the appropriate debug configuration options enabled (e.g., CONFIG_DEBUG_INFO).
In either case, the extracted struct unix_skb_parms should be placed into a new header file.

If we completely avoid using UNIXCB(), one possible approach is similar to
src/lib/ciul/shrub_socket.c’s ef_shrub_socket_recv_metrics() —
modify it to use kernel_recvmsg() and then parse cmsghdr to retrieve the data.
However, I’m not sure whether the cmsghdr would contain this information in an Onload kernel bypass environment.

Does anyone have other ideas or suggestions?

See Xilinx-CNS#302 (comment)

boyracer2014 · 2025-08-19T08:29:52Z

clone the repo. swtiched and tried the branch but on a rocky linux 6.15 i am getting:

  AR      src/lib/cplane/lib.a
  CC [M]  src/driver/linux_char/vi_resource.o
In file included from src/driver/linux_net/drivers/net/ethernet/sfc/net_driver.h:77,
                 from src/driver/linux_net/drivers/net/ethernet/sfc/efx.c:33:
src/driver/linux_net/drivers/net/ethernet/sfc/efx_client.h:21:10: fatal error: linux/sfc/efx_auxbus.h: No such file or directory
   21 | #include <linux/sfc/efx_auxbus.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[6]: *** [/usr/src/kernels/6.15.7-1.el9.elrepo.x86_64/scripts/Makefile.build:203: src/driver/linux_net/drivers/net/ethernet/sfc/efx.o] Error 1
make[6]: *** Waiting for unfinished jobs....

what am i missing ?

sheviks · 2025-08-19T10:56:31Z

This issue is due to the absence of @lukester1975's kbuild ccflags-y patch mentioned above.
Since I’m not the author of that patch, I wasn’t sure if it was appropriate to include it in this PR.

If you'd like to test the compilation, you can try:
git clone -b test-6.17 https://github.com/sheviks/onload.git

This branch additionally includes:

The kbuild ccflags-y patch
(sheviks@fdcf56e)
A minor tweak to improve compatibility with Clang
(sheviks@9525ca6)
A temporary workaround for __randomize_layout mentioned above
(sheviks@5c53f7b)
Adjustments for kernel 6.17-rc
(sheviks@b29408a)
(sheviks@a3171bf)

Thanks!

clone the repo. swtiched and tried the branch but on a rocky linux 6.15 i am getting:

  AR      src/lib/cplane/lib.a
  CC [M]  src/driver/linux_char/vi_resource.o
In file included from src/driver/linux_net/drivers/net/ethernet/sfc/net_driver.h:77,
                 from src/driver/linux_net/drivers/net/ethernet/sfc/efx.c:33:
src/driver/linux_net/drivers/net/ethernet/sfc/efx_client.h:21:10: fatal error: linux/sfc/efx_auxbus.h: No such file or directory
   21 | #include <linux/sfc/efx_auxbus.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[6]: *** [/usr/src/kernels/6.15.7-1.el9.elrepo.x86_64/scripts/Makefile.build:203: src/driver/linux_net/drivers/net/ethernet/sfc/efx.o] Error 1
make[6]: *** Waiting for unfinished jobs....

what am i missing ?

boyracer2014 · 2025-08-19T11:28:57Z

nice. compiled and tested. works well !

abower-amd · 2025-09-19T08:26:02Z

src/driver/linux_net/drivers/net/ethernet/sfc/debugfs.c

This file is part of the sfc Linux net driver which is managed in a separate project so we would make changes there. Linux 6.15 build compatibility changes have already been made in the net driver so this particular change is no longer needed since 2b28cf9, thanks!

abower-amd · 2025-09-19T08:27:19Z

src/driver/linux_net/drivers/net/ethernet/sfc/kernel_compat.sh

This file is part of the sfc Linux net driver which is managed in a separate project so we would make changes there. Linux 6.15 build compatibility changes have already been made in the net driver so this particular change is no longer needed since 2b28cf9, thanks!

kieranm-xilinx

Thanks for the pull request. This is helpful in identifying the areas of change needed. I've left comments for the individual parts, as I think the Onload team needs to make some changes before merging. Likely we'll split this into small changes so we can immediately merge some parts.

kieranm-xilinx · 2025-09-19T08:24:05Z

src/include/ci/tools/debug.h

 /* verifies compile time expression is 0 or positive -
 * no-op for runtime expressions */
-#if __GNUC__ * 100 + __GNUC_MINOR__ >= 409
+#if defined(__clang__) || __GNUC__ * 100 + __GNUC_MINOR__ >= 409


Although we don't officially support clang I'm inclined to recommend taking this fix as it will make life easier for any clang users and seems harmless for everyone else.

If the Clang-related changes above are accepted, I’d really appreciate it if you could also include this commit: sheviks@9525ca6.

Clang emits more errors than GCC; adding that commit ensures Clang generates the same autocompat.h as GCC.
This is also harmless for users not using Clang.

Thanks!

Hi @sheviks, I don't think that commit 9525ca6 features in a pull request. Would you mind putting it in one if you'd like our team to review it, please? Thanks!

I’ve committed the change. Please see this commit for details: sheviks@dc0b741
Thank you.

After I committed the patch, GitHub reported conflicts. When I resolved them, GitHub automatically added a merge commit.
Because of that, I found that the commit mentioned in issue #293 (2b28cf9) does not include all ccflags-y changes.
For example:

src/driver/linux_net/scripts/kernel_compat_funcs.sh has been updated to use ccflags-y,

but src/driver/linux_resource/kernel_compat_funcs.sh still uses EXTRA_CFLAGS.

A grep for EXTRA_CFLAGS on current master still shows quite a few remaining occurrences.
Could you please take another look and make sure the ccflags-y changes are applied consistently?

Alternatively, if you'd prefer, I can review them and submit the fixes here.

Thank you.

I reconsidered the approach. Since this PR will be split into smaller merges, I went ahead and committed the ccflags-y changes so they’re ready for review.
Please review and decide on the final merge strategy.

Thanks!

kieranm-xilinx · 2025-09-19T08:41:53Z

src/driver/linux_resource/kernel_compat.sh

 EFRM_NET_HAS_USER_NS			member	struct_net user_ns	include/net/net_namespace.h

-EFRM_HAVE_OLD_FAULT			memtype struct_vm_operations_struct	fault	include/linux/mm.h	int (*)(struct vm_area_struct *vma, struct vm_fault *vmf)
+EFRM_HAVE_OLD_FAULT			memtype struct_vm_operations_struct	fault	include/linux/mm.h	int (*)(struct vm_fault *vmf)


I'd like us to understand why this was wrong before changing it. It has been this way for a long time. I appreciate that as this repository doesn't have the history pre-2020, it makes it difficult for external contributors to do that kind of archaeology, so I think this is a task for the AMD team. I'm wondering if when this was written, what was "new" then has become "old" now. I.e. there are potentially three signatures we need to interoperate with, although I'm hoping that the original "old" is now so old that we don't need it.

In summary, I think this change is probably correct, but I want to check history to be sure.

kieranm-xilinx · 2025-09-19T08:44:08Z

src/driver/linux_net/drivers/net/ethernet/sfc/debugfs.c

 #include "ef100_vdpa.h"
 #include "mcdi_filters.h"
 #endif
+#ifdef EFX_HAVE_TRY_LOOKUP_NOPERM


The changes to src/driver/linux_net/* need to go via the AMD net driver team, as we consume the driver unmodified from them. We'll work internally to get this kernel supported with them, and then import a new version of the net driver to the Onload repository.

kieranm-xilinx · 2025-09-19T08:44:43Z

src/driver/linux_resource/kernel_compat.sh

 EFRM_HAVE_FOLLOW_PTE symbol follow_pte include/linux/mm.h
 EFRM_HAVE_FOLLOW_PTE_VMA symtype follow_pte include/linux/mm.h int(struct vm_area_struct*, unsigned long, pte_t**, spinlock_t**)

+EFRM_PAGE_HAS_FOLIO_INDEX		member	struct_page	__folio_index	include/linux/mm_types.h


This looks good to me.

kieranm-xilinx · 2025-09-19T08:47:04Z

src/driver/linux_resource/kernel_compat.sh

 EFRM_HAVE_FOLLOW_PTE_VMA symtype follow_pte include/linux/mm.h int(struct vm_area_struct*, unsigned long, pte_t**, spinlock_t**)

+EFRM_PAGE_HAS_FOLIO_INDEX		member	struct_page	__folio_index	include/linux/mm_types.h
+EFRM_NEED_UNIXCB			nsymbol	UNIXCB	include/net/af_unix.h


This, and the code that uses it in shrub_socket_kernel.c needs closer inspection. While it might succeed in getting it to compile on recent kernels, we need to be careful about introducing new kernel definitions as they can be fragile to maintain. If there's a way we can avoid that, we'd have a strong preference for that. AMD Onload team to have a look at this.

kieranm-xilinx · 2025-09-19T08:47:30Z

src/lib/kernel_utils/hugetlb.c

+#ifdef EFRM_PAGE_HAS_FOLIO_INDEX
+	pgoff_t index = page->__folio_index;
+#else
+	pgoff_t index = page->index;
+#endif
 #if defined(EFRM_HAS_FILEMAP_LOCK_HUGETLB_FOLIO) && ! defined(EFRM_HAS_HUGETLB_BASEPAGE_INDEX)
-	return page->index * PAGE_SIZE;
+	return index * PAGE_SIZE;
 #else
-	return page->index * OO_HUGEPAGE_SIZE;
+	return index * OO_HUGEPAGE_SIZE;


Looks good to me.

kieranm-xilinx · 2025-09-19T08:47:59Z

src/lib/ciul/shrub_socket_kernel.c

+#ifdef EFRM_NEED_UNIXCB
+struct unix_skb_parms {
+	struct pid		*pid;		/* skb credentials	*/
+	kuid_t			uid;
+	kgid_t			gid;
+	struct scm_fp_list	*fp;		/* Passed files		*/
+#ifdef CONFIG_SECURITY_NETWORK
+	u32			secid;		/* Security ID		*/
+#endif
+	u32			consumed;
+} __randomize_layout;
+
+#define UNIXCB(skb)	(*(struct unix_skb_parms *)&((skb)->cb))
+#endif
+


See above comment for UNIXCB - Onload team to have a look

sheviks · 2025-09-19T12:02:09Z

I have replied above regarding the Clang-related part.
Aside from that, everything looks fine to me — please proceed according to your usual process.

Thank you!

…h clang

sheviks added 2 commits August 10, 2025 08:59

sheviks requested a review from a team as a code owner August 10, 2025 04:41

sheviks added a commit to sheviks/onload that referenced this pull request Aug 14, 2025

Temporary workaround for __randomize_layout struct unix_skb_parms

5c53f7b

See Xilinx-CNS#302 (comment)

abower-amd requested changes Sep 19, 2025

View reviewed changes

kieranm-xilinx reviewed Sep 19, 2025

View reviewed changes

sheviks requested a review from abower-amd September 19, 2025 12:00

sheviks added 3 commits September 19, 2025 20:16

Add a check when doing kernel_compact tests with kernels compiled wit…

dc0b741

…h clang

Merge branch 'master' into fix-for-kernel-6.15-and-6.16

7a0eaa8

Replace EXTRA_CFLAGS with ccflags-y

1fb1712

Conversation

sheviks commented Aug 10, 2025

Uh oh!

sheviks commented Aug 10, 2025

Uh oh!

sheviks commented Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boyracer2014 commented Aug 19, 2025

Uh oh!

sheviks commented Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boyracer2014 commented Aug 19, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kieranm-xilinx left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sheviks Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sheviks commented Sep 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sheviks commented Aug 14, 2025 •

edited

Loading

sheviks commented Aug 19, 2025 •

edited

Loading

sheviks Sep 19, 2025 •

edited

Loading