Skip to content

[hwasan] Add hwasan-all-globals option #149621

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

shuffle2
Copy link

@shuffle2 shuffle2 commented Jul 18, 2025

hwasan-globals does not instrument globals with custom sections, because existing code may use __start_/__stop_ symbols to iterate over globals in such a way which will cause hwasan assertions.

Introduce new hwasan-all-globals option, which instruments all user-defined globals (but not those globals which are generated by the hwasan instrumentation itself), including those with custom sections.

fixes #142442

Copy link

Thank you for submitting a Pull Request (PR) to the LLVM Project!

This PR will be automatically labeled and the relevant teams will be notified.

If you wish to, you can add reviewers by using the "Reviewers" section on this page.

If this is not working for you, it is probably because you do not have write permissions for the repository. In which case you can instead tag reviewers by name in a comment by using @ followed by their GitHub username.

If you have received no comments on your PR for a week, you can request a review by "ping"ing the PR by adding a comment “Ping”. The common courtesy "ping" rate is once a week. Please remember that you are asking for valuable time from other developers.

If you have further questions, they may be answered by the LLVM GitHub User Guide.

You can also ask questions in a comment on this PR, on the LLVM Discord or on the forums.

@llvmbot
Copy link
Member

llvmbot commented Jul 19, 2025

@llvm/pr-subscribers-llvm-transforms

@llvm/pr-subscribers-compiler-rt-sanitizer

Author: None (shuffle2)

Changes

hwasan-globals does not instrument globals with custom sections, because existing code may use __start_/__stop_ symbols to iterate over globals in such a way which will cause hwasan assertions.

Introduce new hwasan-all-globals option, which instruments all user-defined globals (but not those globals which are generated by the hwasan instrumentation itself), including those with custom sections.

fixes #142442
request review from: @pcc


Full diff: https://github.com/llvm/llvm-project/pull/149621.diff

1 Files Affected:

  • (modified) llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp (+22-5)
diff --git a/llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp b/llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp
index 2c34bf2157cdd..33c2860152214 100644
--- a/llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp
+++ b/llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp
@@ -160,6 +160,11 @@ static cl::opt<bool> ClGenerateTagsWithCalls(
 static cl::opt<bool> ClGlobals("hwasan-globals", cl::desc("Instrument globals"),
                                cl::Hidden, cl::init(false));
 
+static cl::opt<bool> ClAllGlobals(
+    "hwasan-all-globals",
+    cl::desc("Instrument globals, even those within user-defined sections"),
+    cl::Hidden, cl::init(false));
+
 static cl::opt<int> ClMatchAllTag(
     "hwasan-match-all-tag",
     cl::desc("don't report bad accesses via pointers with this tag"),
@@ -452,6 +457,7 @@ class HWAddressSanitizer {
   bool InstrumentWithCalls;
   bool InstrumentStack;
   bool InstrumentGlobals;
+  bool InstrumentAllGlobals;
   bool DetectUseAfterScope;
   bool UsePageAliases;
   bool UseMatchAllCallback;
@@ -679,6 +685,7 @@ void HWAddressSanitizer::initializeModule() {
 
   InstrumentGlobals =
       !CompileKernel && !UsePageAliases && optOr(ClGlobals, NewRuntime);
+  InstrumentAllGlobals = InstrumentGlobals && ClAllGlobals;
 
   if (!CompileKernel) {
     createHwasanCtorComdat();
@@ -1780,11 +1787,21 @@ void HWAddressSanitizer::instrumentGlobals() {
     if (GV.hasCommonLinkage())
       continue;
 
-    // Globals with custom sections may be used in __start_/__stop_ enumeration,
-    // which would be broken both by adding tags and potentially by the extra
-    // padding/alignment that we insert.
-    if (GV.hasSection())
-      continue;
+    if (InstrumentAllGlobals) {
+      // Avoid adding metadata emitted for the hwasan instrumentation itself.
+      // Code which makes assumptions about memory layout of globals between
+      // __start_<section>/__end_<section> linker-generated symbols may need
+      // manual adaptation.
+      auto section = GV.getSection();
+      if (section == "hwasan_globals" || section == ".note.hwasan.globals")
+        continue;
+    } else {
+      // Globals with custom sections may be used in __start_/__stop_
+      // enumeration, which would be broken both by adding tags and potentially
+      // by the extra padding/alignment that we insert.
+      if (GV.hasSection())
+        continue;
+    }
 
     Globals.push_back(&GV);
   }

@fmayer
Copy link
Contributor

fmayer commented Jul 19, 2025

What do you need this for? This seems like a very easy to misuse flag.

From AsmPrinter.cpp:

  // If the alignment is specified, we *must* obey it.  Overaligning a global
  // with a specified alignment is a prompt way to break globals emitted to
  // sections and expected to be contiguous (e.g. ObjC metadata).

@fmayer
Copy link
Contributor

fmayer commented Jul 19, 2025

Please remove @pcc from the commit message. Otherwise, in case this gets merged, he will be notified whenever this gets merged into some fork, see also: https://discourse.llvm.org/t/forbidding-username-in-commits/86997

@fmayer
Copy link
Contributor

fmayer commented Jul 19, 2025

If you only need this for specific sections, maybe we should add a flag to enumerate the specific allowed sections?

@fmayer fmayer requested a review from pcc July 19, 2025 00:19
@shuffle2
Copy link
Author

The intent of this change is to allow hwasan globals instrumentation to be effective on embedded targets which tend to extensively use custom sections to have fine grained control of data allocation, and mainly/exclusively use global storage instead of dynamically allocated memory.

I don't think listing all sections you want to instrument would be a good solution. A normal user would then need to duplicate all expected section names in some build scripts. Also, some large codebases which this is intended for auto-create section names in ways which would be annoying to manually upkeep in related build scripts (e.g. generated based on __FILE__ and/or __LINE__).

Copy link
Contributor

@fmayer fmayer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs a test. See llvm/test/Instrumentation/HWAddressSanitizer/globals.ll

I am adding a case for a section global #149625

@shuffle2
Copy link
Author

This needs a test. See llvm/test/Instrumentation/HWAddressSanitizer/globals.ll

I am adding a case for a section global #149625

sure, do you intend me to pick those changes into this PR?

@fmayer
Copy link
Contributor

fmayer commented Jul 19, 2025

This needs a test. See llvm/test/Instrumentation/HWAddressSanitizer/globals.ll
I am adding a case for a section global #149625

sure, do you intend me to pick those changes into this PR?

I'll submit it as soon as CI passes. You can base your test changes (an extra RUN line with your flag) on that.

@shuffle2
Copy link
Author

I'll submit it as soon as CI passes. You can base your test changes (an extra RUN line with your flag) on that.

Here's what I came up with based on those changes, does it look OK?

--- a/llvm/test/Instrumentation/HWAddressSanitizer/globals.ll
+++ b/llvm/test/Instrumentation/HWAddressSanitizer/globals.ll
@@ -1,10 +1,12 @@
 ; RUN: opt < %s -S -passes=hwasan -mtriple=aarch64--linux-android29 | FileCheck --check-prefixes=CHECK,CHECK29 %s
 ; RUN: opt < %s -S -passes=hwasan -mtriple=aarch64--linux-android30 | FileCheck --check-prefixes=CHECK,CHECK30 %s
+; RUN: opt < %s -S -passes=hwasan -mtriple=riscv64-unknown-elf -hwasan-globals=1 -hwasan-all-globals=1 | FileCheck --check-prefixes=CHECKALLGLOBALS %s

 ; CHECK29: @four = global

 ; CHECK: @specialcaselisted = global i16 2, no_sanitize_hwaddress
-
+; CHECK: @insection = global i16 2, section "custom"
+; CHECKALLGLOBALS: @insection.hwasan = private global { i16, [14 x i8] } { i16 2, [14 x i8] c"\00\00\00\00\00\00\00\00\00\00\00\00\00\AF" }, section "custom", align 16
 ; CHECK: @__start_hwasan_globals = external hidden constant [0 x i8]
 ; CHECK: @__stop_hwasan_globals = external hidden constant [0 x i8]

@@ -37,3 +39,4 @@ source_filename = "foo"
 @sixteen = global [16 x i8] zeroinitializer
 @huge = global [16777232 x i8] zeroinitializer
 @specialcaselisted = global i16 2, no_sanitize_hwaddress
+@insection = global i16 2, section "custom"

@fmayer
Copy link
Contributor

fmayer commented Jul 19, 2025

You also want to check @specialcaselisted for your flag

@fmayer
Copy link
Contributor

fmayer commented Jul 21, 2025

Thanks for your patch! I will wait to see if @pcc has any thoughts, and if not, go ahead and merge this later this week.

Copy link
Contributor

@fmayer fmayer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LG, but maybe add a test for llvm.metadata as well? Also fine in a follow up.

@fmayer fmayer requested a review from pcc July 31, 2025 00:26
@fmayer
Copy link
Contributor

fmayer commented Jul 31, 2025

CI failed:

2025-07-31T00:26:18.6558024Z FAIL: LLVM :: Instrumentation/HWAddressSanitizer/X86/globals.ll (32313 of 60638)
2025-07-31T00:26:18.6558790Z ******************** TEST 'LLVM :: Instrumentation/HWAddressSanitizer/X86/globals.ll' FAILED ********************
2025-07-31T00:26:18.6559425Z Exit Code: 1
2025-07-31T00:26:18.6559592Z 
2025-07-31T00:26:18.6559724Z Command Output (stderr):
2025-07-31T00:26:18.6560050Z --
2025-07-31T00:26:18.6561866Z /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/opt < /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/Instrumentation/HWAddressSanitizer/X86/globals.ll -S -passes=hwasan -mtriple=x86_64-unknown-linux-gnu | /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/Instrumentation/HWAddressSanitizer/X86/globals.ll # RUN: at line 1
2025-07-31T00:26:18.6564922Z + /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/opt -S -passes=hwasan -mtriple=x86_64-unknown-linux-gnu
2025-07-31T00:26:18.6566715Z + /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/Instrumentation/HWAddressSanitizer/X86/globals.ll
2025-07-31T00:26:18.6569273Z /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/Instrumentation/HWAddressSanitizer/X86/globals.ll:10:10: error: CHECK: expected string not found in input
2025-07-31T00:26:18.6571050Z ; CHECK: @four.hwasan = private global { i32, [12 x i8] } { i32 1, [12 x i8] c"\00\00\00\00\00\00\00\00\00\00\00\10" }, align 16
2025-07-31T00:26:18.6571821Z          ^
2025-07-31T00:26:18.6572146Z <stdin>:20:136: note: scanning from here
2025-07-31T00:26:18.6573440Z @hwasan.dummy.global = private constant [0 x i8] zeroinitializer, section "hwasan_globals", comdat($hwasan.module_ctor), !associated !3
2025-07-31T00:26:18.6574532Z                                                                                                                                        ^
2025-07-31T00:26:18.6574920Z 
2025-07-31T00:26:18.6575053Z Input file: <stdin>
2025-07-31T00:26:18.6576025Z Check file: /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/Instrumentation/HWAddressSanitizer/X86/globals.ll
2025-07-31T00:26:18.6576952Z 
2025-07-31T00:26:18.6577172Z -dump-input=help explains the following input dump.
2025-07-31T00:26:18.6577563Z 
2025-07-31T00:26:18.6577659Z Input was:
2025-07-31T00:26:18.6578086Z <<<<<<
2025-07-31T00:26:18.6578315Z           .
2025-07-31T00:26:18.6578546Z           .
2025-07-31T00:26:18.6578741Z           .
2025-07-31T00:26:18.6579206Z          15: @llvm.used = appending global [1 x ptr] [ptr @hwasan.module_ctor], section "llvm.metadata" 
2025-07-31T00:26:18.6580167Z          16: @llvm.global_ctors = appending global [1 x { i32, ptr, ptr }] [{ i32, ptr, ptr } { i32 0, ptr @hwasan.module_ctor, ptr @hwasan.module_ctor }] 
2025-07-31T00:26:18.6580976Z          17: @__start_hwasan_globals = external hidden constant [0 x i8] 
2025-07-31T00:26:18.6581523Z          18: @__stop_hwasan_globals = external hidden constant [0 x i8] 
2025-07-31T00:26:18.6583459Z          19: @hwasan.note = private constant { i32, i32, i32, [8 x i8], i32, i32 } { i32 8, i32 8, i32 3, [8 x i8] c"LLVM\00\00\00\00", i32 trunc (i64 sub (i64 ptrtoint (ptr @__start_hwasan_globals to i64), i64 ptrtoint (ptr @hwasan.note to i64)) to i32), i32 trunc (i64 sub (i64 ptrtoint (ptr @__stop_hwasan_globals to i64), i64 ptrtoint (ptr @hwasan.note to i64)) to i32) }, section ".note.hwasan.globals", comdat($hwasan.module_ctor), align 4 
2025-07-31T00:26:18.6585569Z          20: @hwasan.dummy.global = private constant [0 x i8] zeroinitializer, section "hwasan_globals", comdat($hwasan.module_ctor), !associated !3 
2025-07-31T00:26:18.6586475Z check:10                                                                                                                                            X error: no match found
2025-07-31T00:26:18.6587272Z          21: @__hwasan_tls = external thread_local(initialexec) global i64 
2025-07-31T00:26:18.6587905Z check:10     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2025-07-31T00:26:18.6589340Z          22: @llvm.compiler.used = appending global [7 x ptr] [ptr @four.hwasan.descriptor, ptr @sixteen.hwasan.descriptor, ptr @huge.hwasan.descriptor, ptr @huge.hwasan.descriptor.1, ptr @hwasan.note, ptr @hwasan.dummy.global, ptr @__hwasan_tls], section "llvm.metadata" 
2025-07-31T00:26:18.6591099Z check:10     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2025-07-31T00:26:18.6591953Z          23:  
2025-07-31T00:26:18.6592226Z check:10     ~
2025-07-31T00:26:18.6593010Z          24: @four = alias i32, inttoptr (i64 add (i64 ptrtoint (ptr @four.hwasan to i64), i64 2305843009213693952) to ptr) 
2025-07-31T00:26:18.6593863Z check:10     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2025-07-31T00:26:18.6594717Z          25: @sixteen = alias [16 x i8], inttoptr (i64 add (i64 ptrtoint (ptr @sixteen.hwasan to i64), i64 2449958197289549824) to ptr) 
2025-07-31T00:26:18.6595632Z check:10     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2025-07-31T00:26:18.6596235Z           .
2025-07-31T00:26:18.6596517Z           .
2025-07-31T00:26:18.6596796Z           .
2025-07-31T00:26:18.6597078Z >>>>>>
2025-07-31T00:26:18.6597238Z 
2025-07-31T00:26:18.6597352Z --
2025-07-31T00:26:18.6597530Z 
2025-07-31T00:26:18.6597648Z ********************

hwasan-globals does not instrument globals with custom sections,
because existing code may use __start_/__stop_ symbols to iterate
over globals in such a way which will cause hwasan assertions.

Introduce new hwasan-all-globals option, which instruments all
user-defined globals (but not those globals which are generated by
the hwasan instrumentation itself), including those with custom sections.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

hwasan: globals which have been assigned a section cannot be instrumented
4 participants