mm/hugetlb: fix copy_hugetlb_page_range() to use ->pt_share_count

jira VULN-46929
cve-bf CVE-2024-57883
commit-author Jane Chu <[email protected]>
commit 14967a9
upstream-diff |
  include/linux/mm_types.h
        Removed the definition of `ptdesc_pmd_is_shared()' function in
        alignment with stable-5.15 backport
        8410996eb6fea116fe1483ed977aacf580eee7b4 (it omits the definition
        of `ptdesc_pmd_pts_*()' functions family, to which
        `ptdesc_pmd_is_shared()' belongs).
  mm/hugetlb.c
        copy_hugetlb_page_range()
              1. Used CONFIG_ARCH_WANT_HUGE_PMD_SHARE instead of
                 CONFIG_HUGETLB_PMD_PAGE_TABLE_SHARING, because the latter
                 was introduced only in the non-backported commit
                 188cac5.
              2. Since `ptdesc_pmd_is_shared()' was not defined, read the
                 `pt_share_count' field directly, as is don in the
                 stable-5.15 backport
                 8410996eb6fea116fe1483ed977aacf580eee7b4. (Compare
                 changes to `huge_pmd_unshare()' in `mm/hugetlb.c' between
                 upstream 59d9094 and
                 stable-5.15 8410996eb6fea116fe1483ed977aacf580eee7b4.)
        huge_pmd_unshare()
              No change to the conditional. It was arguably not needed in
              the upstream as well, probably introduced only for the sake
              of clarity in the presence of `ptdesc_pmd_is_shared()'
              function, which is missing here.

commit 59d9094 ("mm: hugetlb: independent PMD page table shared
count") introduced ->pt_share_count dedicated to hugetlb PMD share count
tracking, but omitted fixing copy_hugetlb_page_range(), leaving the
function relying on page_count() for tracking that no longer works.

When lazy page table copy for hugetlb is disabled, that is, revert commit
bcd51a3 ("hugetlb: lazy page table copies in fork()") fork()'ing with
hugetlb PMD sharing quickly lockup -

[  239.446559] watchdog: BUG: soft lockup - CPU#75 stuck for 27s!
[  239.446611] RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x2e0
[  239.446631] Call Trace:
[  239.446633]  <TASK>
[  239.446636]  _raw_spin_lock+0x3f/0x60
[  239.446639]  copy_hugetlb_page_range+0x258/0xb50
[  239.446645]  copy_page_range+0x22b/0x2c0
[  239.446651]  dup_mmap+0x3e2/0x770
[  239.446654]  dup_mm.constprop.0+0x5e/0x230
[  239.446657]  copy_process+0xd17/0x1760
[  239.446660]  kernel_clone+0xc0/0x3e0
[  239.446661]  __do_sys_clone+0x65/0xa0
[  239.446664]  do_syscall_64+0x82/0x930
[  239.446668]  ? count_memcg_events+0xd2/0x190
[  239.446671]  ? syscall_trace_enter+0x14e/0x1f0
[  239.446676]  ? syscall_exit_work+0x118/0x150
[  239.446677]  ? arch_exit_to_user_mode_prepare.constprop.0+0x9/0xb0
[  239.446681]  ? clear_bhb_loop+0x30/0x80
[  239.446684]  ? clear_bhb_loop+0x30/0x80
[  239.446686]  entry_SYSCALL_64_after_hwframe+0x76/0x7e

There are two options to resolve the potential latent issue:
  1. warn against PMD sharing in copy_hugetlb_page_range(),
  2. fix it.
This patch opts for the second option.
While at it, simplify the comment, the details are not actually relevant
anymore.

Link: https://lkml.kernel.org/r/[email protected]
Fixes: 59d9094 ("mm: hugetlb: independent PMD page table shared count")
	Signed-off-by: Jane Chu <[email protected]>
	Reviewed-by: Harry Yoo <[email protected]>
	Acked-by: Oscar Salvador <[email protected]>
	Acked-by: David Hildenbrand <[email protected]>
	Cc: Jann Horn <[email protected]>
	Cc: Liu Shixin <[email protected]>
	Cc: Muchun Song <[email protected]>
	Signed-off-by: Andrew Morton <[email protected]>
(cherry picked from commit 14967a9)
	Signed-off-by: Marcin Wcisło <[email protected]>

Author: pvts-mat

mm/hugetlb.c

@@ -4808,16 +4808,11 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
 			break;
 		}
 
-		/*
-		 * If the pagetables are shared don't copy or take references.
-		 *
-		 * dst_pte == src_pte is the common case of src/dest sharing.
-		 * However, src could have 'unshared' and dst shares with
-		 * another vma. So page_count of ptep page is checked instead
-		 * to reliably determine whether pte is shared.
-		 */
-		if (page_count(virt_to_page(dst_pte)) > 1)
+#ifdef CONFIG_ARCH_WANT_HUGE_PMD_SHARE
+		/* If the pagetables are shared, there is nothing to do */
+		if (!!atomic_read(&virt_to_page(dst_pte)->pt_share_count))
 			continue;
+#endif
 
 		dst_ptl = huge_pte_lock(h, dst, dst_pte);
 		src_ptl = huge_pte_lockptr(h, src, src_pte);