Adding asan sanitizer support for hip #795

kab163 · 2022-11-23T23:19:39Z

Cmake command on rzvernal:

cmake -DROCM_ROOT_DIR=/opt/rocm-5.3.0 -DHIP_ROOT_DIR=/opt/rocm-5.3.0/hip -DHIP_PATH=/opt/rocm-5.3.0/llvm/bin -DENABLE_HIP=On -DENABLE_OPENMP=Off -DENABLE_CUDA=Off -DUMPIRE_ENABLE_ASAN=On -DCMAKE_CXX_COMPILER=/opt/rocm-5.3.0/llvm/bin/amdclang++ -DCMAKE_C_COMPILER=/opt/rocm-5.3.0/llvm/bin/amdclang -DCMAKE_CXX_FLAGS="-fsanitize=address" -DUMPIRE_ENABLE_SANITIZER_TESTS=ON -DCMAKE_HIP_ARCHITECTURES=gfx90a:xnack+ ../

Warning while compiling: clang-15: warning: ignoring '-fsanitize=address' option for offload arch 'gfx90a' as it is not currently supported there. Use it with an offload arch containing 'xnack+' instead [-Woption-ignored]

Output of sanitizer_tests.cpp

[belcher6@rzvernal11:build]$ ./bin/sanitizer_tests QuickPool write
data[INDEX] = 100
=================================================================
==79729==ERROR: AddressSanitizer: use-after-poison on address 0x154d16201530 at pc 0x0000004f5eb4 bp 0x7fffffff8b10 sp 0x7fffffff8b08
WRITE of size 8 at 0x154d16201530 thread T0
    #0 0x4f5eb3 in test_write_after_free() (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x4f5eb3)
    #1 0x4f67cb in main (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x4f67cb)
    #2 0x15555267acf2 in __libc_start_main (/lib64/libc.so.6+0x3acf2) (BuildId: 64aa558dcdda2d8b0d7b04cef33ddbb2d9d8b8b4)
    #3 0x404e4d in _start (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x404e4d)

Address 0x154d16201530 is a wild pointer inside of access range of size 0x000000000008.
SUMMARY: AddressSanitizer: use-after-poison (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x4f5eb3) in test_write_after_free()
Shadow bytes around the buggy address:
  0x02aa22c38250: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c38260: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c38270: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c38280: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c38290: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
=>0x02aa22c382a0: f7 f7 f7 f7 f7 f7[f7]f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c382b0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c382c0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c382d0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c382e0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c382f0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==79729==ABORTING
[belcher6@rzvernal11:build]$ ./bin/sanitizer_tests QuickPool read
data[INDEX] = 100
=================================================================
==79736==ERROR: AddressSanitizer: use-after-poison on address 0x154d16200800 at pc 0x0000004f5c22 bp 0x7fffffff8b30 sp 0x7fffffff8b28
READ of size 8 at 0x154d16200800 thread T0
    #0 0x4f5c21 in test_read_after_free() (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x4f5c21)
    #1 0x4f679e in main (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x4f679e)
    #2 0x15555267acf2 in __libc_start_main (/lib64/libc.so.6+0x3acf2) (BuildId: 64aa558dcdda2d8b0d7b04cef33ddbb2d9d8b8b4)
    #3 0x404e4d in _start (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x404e4d)

Address 0x154d16200800 is a wild pointer inside of access range of size 0x000000000008.
SUMMARY: AddressSanitizer: use-after-poison (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x4f5c21) in test_read_after_free()
Shadow bytes around the buggy address:
  0x02aa22c380b0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c380c0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c380d0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c380e0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c380f0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
=>0x02aa22c38100:[f7]f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c38110: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c38120: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c38130: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c38140: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c38150: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==79736==ABORTING

kab163 · 2022-12-06T18:03:55Z

Updated cmake command:

cmake -DROCM_ROOT_DIR=/opt/rocm-5.4.0 -DHIP_ROOT_DIR=/opt/rocm-5.4.0/hip -DHIP_PATH=/opt/rocm-5.4.0/llvm/bin -DENABLE_HIP=On -DENABLE_OPENMP=Off -DENABLE_CUDA=Off -DUMPIRE_ENABLE_ASAN=On -DCMAKE_CXX_COMPILER=/opt/rocm-5.4.0/llvm/bin/amdclang++ -DCMAKE_C_COMPILER=/opt/rocm-5.4.0/llvm/bin/amdclang -DCMAKE_CXX_FLAGS="-fsanitize=address" -DUMPIRE_ENABLE_SANITIZER_TESTS=ON -DCMAKE_HIP_ARCHITECTURES="gfx90a:xnack+" -DGPU_TARGETS="gfx90a:xnack+" -DAMDGPU_TARGETS="gfx90a:xnack+" ../

Updated Output (for hip kernel sanitizer test):

bash-4.4$ ./bin/sanitizer_tests QuickPool read
AddressSanitizer:DEADLYSIGNAL
=================================================================
==333094==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000020 (pc 0x155553a4b9cd bp 0x7fffffff7ab0 sp 0x7fffffff7a50 T0)
==333094==The signal is caused by a READ memory access.
==333094==Hint: address points to the zero page.
    #0 0x155553a4b9cd  (/opt/rocm-5.4.0/lib/libamdhip64.so.5+0xd59cd) (BuildId: a4e47799322466785f920489031c315137965c08)
    #1 0x155553a4bb26  (/opt/rocm-5.4.0/lib/libamdhip64.so.5+0xd5b26) (BuildId: a4e47799322466785f920489031c315137965c08)
    #2 0x155553a4c9fe  (/opt/rocm-5.4.0/lib/libamdhip64.so.5+0xd69fe) (BuildId: a4e47799322466785f920489031c315137965c08)
    #3 0x155553a0a5d6  (/opt/rocm-5.4.0/lib/libamdhip64.so.5+0x945d6) (BuildId: a4e47799322466785f920489031c315137965c08)
    #4 0x155553b5de15  (/opt/rocm-5.4.0/lib/libamdhip64.so.5+0x1e7e15) (BuildId: a4e47799322466785f920489031c315137965c08)
    #5 0x155553b39881  (/opt/rocm-5.4.0/lib/libamdhip64.so.5+0x1c3881) (BuildId: a4e47799322466785f920489031c315137965c08)
    #6 0x155553b3af93 in hipLaunchKernel (/opt/rocm-5.4.0/lib/libamdhip64.so.5+0x1c4f93) (BuildId: a4e47799322466785f920489031c315137965c08)
    #7 0x543a21 in __device_stub__test_for_hip(double**, unsigned long) (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x543a21)
    #8 0x544cdf in main (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x544cdf)
    #9 0x15555267dcf2 in __libc_start_main (/lib64/libc.so.6+0x3acf2) (BuildId: 64aa558dcdda2d8b0d7b04cef33ddbb2d9d8b8b4)
    #10 0x45285d in _start (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x45285d)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/opt/rocm-5.4.0/lib/libamdhip64.so.5+0xd59cd) (BuildId: a4e47799322466785f920489031c315137965c08)
==333094==ABORTING

Note: Will not work if HSA_XNACK is set to 1, do not export... explanation TBD

kab163 · 2022-12-07T23:40:59Z

Issue: it seems like running the test with ./bin/sanitizer_tests QuickPool read HOST and ./bin/sanitizer_tests QuickPool write HOST both give an error that detects a write access....
(Whether it is build with hip support or without it)

kab163 · 2022-12-08T00:28:32Z

The way this test is ran in make test might need to be changed...

kab163 · 2022-12-21T17:22:57Z

See AMD bug report (277) - made reproducer for AMD to look in to. Seems like this does not yet work as expected in rocm 5.4

kab163 · 2023-03-02T22:49:36Z

Update: A fix is supposed to be ready in rocm 6.0

…an-poisoning-for-hip

kab163 · 2023-04-03T16:07:23Z

Edited out...

…an-poisoning-for-hip

…-1020-asan-poisoning-for-hip

kab163 · 2024-01-22T18:23:23Z

The fix is in rocm/6.0.0, but we have to add a "-Wl,-rpath=/opt/rocm-6.0.0/llvm/lib/clang/17.0.0/lib/linux/" flag. Once we update the corona/tioga jobs to rocm-6.0.0 and get that in, it should be good to go.

kab163 · 2024-01-24T16:12:28Z

This has been a long time coming! @adrienbernede - once the blt/camp releases come out and we can update umpire to rocm/6.0.0 we should add this test! (No updates yet for the reason behind why the device allocator test is failing with 6.0.0 - ticket has been "submitted to vendor", so it is being looked into at least...)

adrienbernede · 2024-06-10T14:10:14Z

@kab163 @davidbeckingsale now that the allocation is not limiting anymore I’ll let you deal with the rest ;).

…an-poisoning-for-hip

kab163 · 2024-12-17T15:26:17Z

Updated cmake command:

cmake -DROCM_ROOT_DIR=/opt/rocm-6.0.0 -DHIP_ROOT_DIR=/opt/rocm-6.0.0/hip -DHIP_PATH=/opt/rocm-6.0.0/llvm/bin -DENABLE_HIP=On -DENABLE_OPENMP=Off -DENABLE_CUDA=Off -DCMAKE_CXX_COMPILER=/opt/rocm-6.0.0/llvm/bin/amdclang++ -DCMAKE_C_COMPILER=/opt/rocm-6.0.0/llvm/bin/amdclang -DUMPIRE_ENABLE_ASAN=On -DCMAKE_CXX_FLAGS="-fsanitize=address" -DUMPIRE_ENABLE_SANITIZER_TESTS=ON -DCMAKE_HIP_ARCHITECTURES="gfx90a:xnack+" -DGPU_TARGETS="gfx90a:xnack+" -DAMDGPU_TARGETS="gfx90a:xnack+" ../

or

cmake -DROCM_ROOT_DIR=/opt/rocm-6.3.0 -DHIP_ROOT_DIR=/opt/rocm-6.3.0/hip -DHIP_PATH=/opt/rocm-6.3.0/llvm/bin -DENABLE_HIP=On -DENABLE_OPENMP=Off -DENABLE_CUDA=Off -DCMAKE_CXX_COMPILER=/opt/rocm-6.3.0/llvm/bin/amdclang++ -DCMAKE_C_COMPILER=/opt/rocm-6.3.0/llvm/bin/amdclang -DUMPIRE_ENABLE_ASAN=On -DCMAKE_CXX_FLAGS="-fsanitize=address" -DCMAKE_EXE_LINKER_FLAGS="-Wl,-rpath=/opt/rocm-6.3.0/llvm/lib/clang/18/lib/linux/" -DUMPIRE_ENABLE_SANITIZER_TESTS=ON -DCMAKE_HIP_ARCHITECTURES="gfx90a:xnack+" -DGPU_TARGETS="gfx90a:xnack+" -DAMDGPU_TARGETS="gfx90a:xnack+" -DCAMP_ENABLE_TESTS=Off ../

kab163 · 2024-12-23T21:54:24Z

Built with

cmake -DROCM_ROOT_DIR=/opt/rocm-6.3.0 -DHIP_ROOT_DIR=/opt/rocm-6.3.0/hip -DHIP_PATH=/opt/rocm-6.3.0/llvm/bin -DENABLE_HIP=On -DENABLE_OPENMP=Off -DENABLE_CUDA=Off -DCMAKE_CXX_COMPILER=/opt/rocm-6.3.0/llvm/bin/amdclang++ -DCMAKE_C_COMPILER=/opt/rocm-6.3.0/llvm/bin/amdclang -DUMPIRE_ENABLE_ASAN=On -DCMAKE_CXX_FLAGS="-fsanitize=address" -DCMAKE_EXE_LINKER_FLAGS="-Wl,-rpath=/opt/rocm-6.3.0/llvm/lib/clang/18/lib/linux/" -DUMPIRE_ENABLE_SANITIZER_TESTS=ON -DCMAKE_HIP_ARCHITECTURES="gfx90a:xnack+" -DGPU_TARGETS="gfx90a:xnack+" -DAMDGPU_TARGETS="gfx90a:xnack+" -DCAMP_ENABLE_TESTS=Off ../

then run make -j on rzvernal. Upon running make test all hip tests fail.
For example:

[belcher6@rzvernal10:build]$ HSA_XNACK=1 ./tests/blt_hip_smoke 
H
E
L
L
O
 
W
O
R
L
D
!

=================================================================
==888244==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 5400 byte(s) in 75 object(s) allocated from:
    #0 0x34555d in operator new(unsigned long) (/g/g0/belcher6/Umpire/build/tests/blt_hip_smoke+0x34555d)
    #1 0x1555495c67e3  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x617e3) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)

Direct leak of 144 byte(s) in 2 object(s) allocated from:
    #0 0x34555d in operator new(unsigned long) (/g/g0/belcher6/Umpire/build/tests/blt_hip_smoke+0x34555d)
    #1 0x1555495b2c14  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x4dc14) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)
    #2 0x1555495a6160  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x41160) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)
    #3 0x1555495bb7fd  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x567fd) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)
    #4 0x155553ec1262  (/opt/rocm-6.3.0/lib/libamdhip64.so.6+0x2db262) (BuildId: 30c0c486666b96452fa202536d3f49306cd3d993)

Direct leak of 72 byte(s) in 1 object(s) allocated from:
    #0 0x34555d in operator new(unsigned long) (/g/g0/belcher6/Umpire/build/tests/blt_hip_smoke+0x34555d)
    #1 0x1555495b2c14  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x4dc14) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)
    #2 0x1555495a6160  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x41160) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)
    #3 0x1555495a2d00  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x3dd00) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)

Direct leak of 56 byte(s) in 1 object(s) allocated from:
    #0 0x34555d in operator new(unsigned long) (/g/g0/belcher6/Umpire/build/tests/blt_hip_smoke+0x34555d)
    #1 0x1555495c65d6  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x615d6) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)
    #2 0x5040003ddd8f  (<unknown module>)

Direct leak of 56 byte(s) in 1 object(s) allocated from:
    #0 0x34555d in operator new(unsigned long) (/g/g0/belcher6/Umpire/build/tests/blt_hip_smoke+0x34555d)
    #1 0x1555495c65d6  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x615d6) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)
    #2 0x5040003dde0f  (<unknown module>)

Direct leak of 56 byte(s) in 1 object(s) allocated from:
    #0 0x34555d in operator new(unsigned long) (/g/g0/belcher6/Umpire/build/tests/blt_hip_smoke+0x34555d)
    #1 0x1555495c65d6  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x615d6) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)
    #2 0x5040003dddcf  (<unknown module>)

Direct leak of 56 byte(s) in 1 object(s) allocated from:
    #0 0x34555d in operator new(unsigned long) (/g/g0/belcher6/Umpire/build/tests/blt_hip_smoke+0x34555d)
    #1 0x1555495c65d6  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x615d6) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)
    #2 0x5040003ddd4f  (<unknown module>)

Indirect leak of 3504 byte(s) in 73 object(s) allocated from:
    #0 0x3071a9 in calloc (/g/g0/belcher6/Umpire/build/tests/blt_hip_smoke+0x3071a9)
    #1 0x15554966fc66  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x10ac66) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)

SUMMARY: AddressSanitizer: 9344 byte(s) leaked in 155 allocation(s).

or

[belcher6@rzvernal10:build]$ HSA_XNACK=1 ./bin/resource_manager_tests 
Running main() from /g/g0/belcher6/Umpire/blt/thirdparty_builtin/googletest/googletest/src/gtest_main.cc
[==========] Running 6 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 6 tests from ResourceManager
[ RUN      ] ResourceManager.Constructor
[       OK ] ResourceManager.Constructor (181 ms)
[ RUN      ] ResourceManager.findAllocationRecord
🔍 Printing allocation map contents...
0x15553f0fe800 {
  size: 1048576, range: 0x15553f0fe800 -- 0x15553f1fe800, name: , 
}
done.
🔍 Printing allocation map contents...
0x15553f0fe800 {
  size: 1048576, range: 0x15553f0fe800 -- 0x15553f1fe800, name: , 
}
done.
🔍 Printing allocation map contents...
done.
[       OK ] ResourceManager.findAllocationRecord (9 ms)
[ RUN      ] ResourceManager.getAllocatorByName
[       OK ] ResourceManager.getAllocatorByName (1 ms)
[ RUN      ] ResourceManager.getAllocatorById
[       OK ] ResourceManager.getAllocatorById (1 ms)
[ RUN      ] ResourceManager.getAllocatorInvalidId
[       OK ] ResourceManager.getAllocatorInvalidId (1 ms)
[ RUN      ] ResourceManager.aliases
[       OK ] ResourceManager.aliases (6 ms)
[----------] 6 tests from ResourceManager (202 ms total)

[----------] Global test environment tear-down
[==========] 6 tests from 1 test suite ran. (202 ms total)
[  PASSED  ] 6 tests.

=================================================================
==889427==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 648 byte(s) in 9 object(s) allocated from:
    #0 0x5de66d in operator new(unsigned long) (/g/g0/belcher6/Umpire/build/bin/resource_manager_tests+0x5de66d)
    #1 0x1555495c67e3  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x617e3) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)

Indirect leak of 384 byte(s) in 8 object(s) allocated from:
    #0 0x5a02b9 in calloc (/g/g0/belcher6/Umpire/build/bin/resource_manager_tests+0x5a02b9)
    #1 0x15554966fc66  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x10ac66) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)

SUMMARY: AddressSanitizer: 1032 byte(s) leaked in 17 allocation(s).

adayton1 · 2025-01-17T23:51:07Z

What's the current status of this merge request? I'm testing out asan and finding it works for simple tests. I'm working on enabling it in my application, but until this goes in, I'll have to disable memory pools.

…an-poisoning-for-hip

adayton1 · 2025-03-12T16:09:52Z

Is this ready for review?

kab163 · 2025-03-12T16:10:16Z

What's the current status of this merge request? I'm testing out asan and finding it works for simple tests. I'm working on enabling it in my application, but until this goes in, I'll have to disable memory pools.

Whether I'm running on rzvernal or rzadams, I see tests fail with memory leaks detected. So for example, on rzvernal:

[belcher6@rzvernal11:build]$ ctest -T test -R memory_resource_traits_tests --output-on-failure
   Site: rzvernal11
   Build name: Linux-amdclang++
Test project /g/g0/belcher6/Umpire/build
    Start 28: memory_resource_traits_tests
1/1 Test #28: memory_resource_traits_tests .....***Failed    0.51 sec
Running main() from /g/g0/belcher6/Umpire/blt/thirdparty_builtin/googletest/googletest/src/gtest_main.cc
[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from MemoryResourceTraits/MemoryResourceTraitsTest
[ RUN      ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/0
[       OK ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/0 (0 ms)
[ RUN      ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/1
[       OK ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/1 (1 ms)
[ RUN      ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/2
[       OK ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/2 (0 ms)
[ RUN      ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/3
[       OK ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/3 (0 ms)
[----------] 4 tests from MemoryResourceTraits/MemoryResourceTraitsTest (1 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 1 test suite ran. (1 ms total)
[  PASSED  ] 4 tests.

=================================================================
==1444299==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 648 byte(s) in 9 object(s) allocated from:
    #0 0x5ea53d in operator new(unsigned long) (/g/g0/belcher6/Umpire/build/bin/memory_resource_traits_tests+0x5ea53d)
    #1 0x1555495c67e3  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x617e3) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)

Indirect leak of 384 byte(s) in 8 object(s) allocated from:
    #0 0x5ac189 in calloc (/g/g0/belcher6/Umpire/build/bin/memory_resource_traits_tests+0x5ac189)
    #1 0x15554966fc66  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x10ac66) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)

SUMMARY: AddressSanitizer: 1032 byte(s) leaked in 17 allocation(s).

or on rzadams:

[belcher6@rzadams1060:build]$ ctest -T test -R blt_hip_smoke --output-on-failure
   Site: rzadams1001
   Build name: Linux-amdclang++
Test project /g/g0/belcher6/Umpire/build
    Start 3: blt_hip_smoke
1/1 Test #3: blt_hip_smoke ....................***Failed    0.50 sec

=================================================================
==1973496==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 360 byte(s) in 5 object(s) allocated from:
    #0 0x34555d in operator new(unsigned long) (/g/g0/belcher6/Umpire/build/tests/blt_hip_smoke+0x34555d)
    #1 0x1555495c67e3  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x617e3) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)

Indirect leak of 192 byte(s) in 4 object(s) allocated from:
    #0 0x3071a9 in calloc (/g/g0/belcher6/Umpire/build/tests/blt_hip_smoke+0x3071a9)
    #1 0x15554966fc66  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x10ac66) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)

SUMMARY: AddressSanitizer: 552 byte(s) leaked in 9 allocation(s).

kab163 · 2025-03-12T16:11:36Z

I'm using rocm/6.3.0 and the build recipe above from late December last year

kab163 · 2025-03-12T16:15:40Z

See https://rzlc.llnl.gov/jira/browse/ELCAP-622

kab163 · 2025-03-12T23:33:28Z

Update: I updated the cmake to:

cmake -DROCM_ROOT_DIR=/opt/rocm-6.3.0 -DHIP_ROOT_DIR=/opt/rocm-6.3.0/hip -DHIP_PATH=/opt/rocm-6.3.0/llvm/bin -DENABLE_HIP=On -DENABLE_OPENMP=Off -DENABLE_CUDA=Off -DCMAKE_CXX_COMPILER=/opt/rocm-6.3.0/llvm/bin/amdclang++ -DCMAKE_C_COMPILER=/opt/rocm-6.3.0/llvm/bin/amdclang -DUMPIRE_ENABLE_ASAN=On -DCMAKE_CXX_FLAGS="-fsanitize=address" -DCMAKE_EXE_LINKER_FLAGS="-Wl,-rpath=/opt/rocm-6.3.0/llvm/lib/clang/18/lib/linux/" -DUMPIRE_ENABLE_SANITIZER_TESTS=ON -DCMAKE_HIP_ARCHITECTURES="gfx942:xnack+" -DGPU_TARGETS="gfx942:xnack+" -DAMDGPU_TARGETS="gfx942:xnack+" -DCAMP_ENABLE_TESTS=Off ../

so that it reflects the correct hardware for rzadams (gfx942) and I load the rocm/6.3.0 module, but I still end up getting similar failures:

[belcher6@rzadams1098:build]$ ctest -T test -R memory_allocator_tests --output-on-failure
   Site: rzadams1098
   Build name: Linux-amdclang++
Test project /g/g0/belcher6/Umpire/build
    Start 13: memory_allocator_tests
1/1 Test #13: memory_allocator_tests ...........***Failed    0.52 sec
Running main() from /g/g0/belcher6/Umpire/blt/thirdparty_builtin/googletest/googletest/src/gtest_main.cc
[==========] Running 4 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 1 test from Default/MemoryAllocatorTest/0, where TypeParam = umpire::alloc::MallocAllocator
[ RUN      ] Default/MemoryAllocatorTest/0.Allocate
[       OK ] Default/MemoryAllocatorTest/0.Allocate (0 ms)
[----------] 1 test from Default/MemoryAllocatorTest/0 (0 ms total)

[----------] 1 test from Default/MemoryAllocatorTest/1, where TypeParam = umpire::alloc::HipMallocAllocator
[ RUN      ] Default/MemoryAllocatorTest/1.Allocate
[       OK ] Default/MemoryAllocatorTest/1.Allocate (230 ms)
[----------] 1 test from Default/MemoryAllocatorTest/1 (230 ms total)

[----------] 1 test from Default/MemoryAllocatorTest/2, where TypeParam = umpire::alloc::HipMallocManagedAllocator
[ RUN      ] Default/MemoryAllocatorTest/2.Allocate
[       OK ] Default/MemoryAllocatorTest/2.Allocate (0 ms)
[----------] 1 test from Default/MemoryAllocatorTest/2 (0 ms total)

[----------] 1 test from Default/MemoryAllocatorTest/3, where TypeParam = umpire::alloc::HipPinnedAllocator
[ RUN      ] Default/MemoryAllocatorTest/3.Allocate
[       OK ] Default/MemoryAllocatorTest/3.Allocate (0 ms)
[----------] 1 test from Default/MemoryAllocatorTest/3 (0 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 4 test suites ran. (231 ms total)
[  PASSED  ] 4 tests.

=================================================================
==1478672==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 360 byte(s) in 5 object(s) allocated from:
    #0 0x42273d in operator new(unsigned long) (/g/g0/belcher6/Umpire/build/bin/memory_allocator_tests+0x42273d)
    #1 0x1555495c67e3  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x617e3) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)

Indirect leak of 192 byte(s) in 4 object(s) allocated from:
    #0 0x3e4389 in calloc (/g/g0/belcher6/Umpire/build/bin/memory_allocator_tests+0x3e4389)
    #1 0x15554966fc66  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x10ac66) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)

SUMMARY: AddressSanitizer: 552 byte(s) leaked in 9 allocation(s).

adayton1 · 2025-03-12T23:36:18Z

Have you tried rocm 6.3.1? I know at least one internal rocm memory issue was fixed between rocm 6.3.0 and rocm 6.3.1.

kab163 · 2025-03-13T14:52:01Z

Have you tried rocm 6.3.1? I know at least one internal rocm memory issue was fixed between rocm 6.3.0 and rocm 6.3.1.

I have not tried that, but I can try that now!

…an-poisoning-for-hip

kab163 · 2025-03-24T20:35:45Z

Had some troubles compiling all of a sudden, but was able to compile again after adding a new flag. Updated cmake:

cmake -DROCM_ROOT_DIR=/opt/rocm-6.3.1 -DHIP_ROOT_DIR=/opt/rocm-6.3.1/hip -DHIP_PATH=/opt/rocm-6.3.1/llvm/bin -DENABLE_HIP=On -DENABLE_OPENMP=Off -DENABLE_CUDA=Off -DCMAKE_CXX_COMPILER=/opt/rocm-6.3.1/llvm/bin/amdclang++ -DCMAKE_C_COMPILER=/opt/rocm-6.3.1/llvm/bin/amdclang -DUMPIRE_ENABLE_ASAN=On -DCMAKE_CXX_FLAGS="-fsanitize-recover -fsanitize=address -shared-libsan" -DCMAKE_EXE_LINKER_FLAGS="-L/opt/rocm-6.3.1/lib/asan/ -L/opt/rocm-6.3.1/llvm/lib/asan -Wl,-rpath,/opt/rocm-6.3.1/lib/asan/:/opt/rocm-6.3.1/llvm/lib/asan:/opt/rocm-6.3.1/lib/llvm/lib/clang/18/lib/linux -lasan" -DUMPIRE_ENABLE_SANITIZER_TESTS=ON -DCMAKE_HIP_ARCHITECTURES="gfx942:xnack+" -DGPU_TARGETS="gfx942:xnack+" -DAMDGPU_TARGETS="gfx942:xnack+" -DCAMP_ENABLE_TESTS=Off ../

adayton1 · 2025-06-23T17:58:11Z

tests/tools/sanitizers/sanitizer_tests.cpp

+__global__ void test_write_for_hip(double* data_ptr, std::size_t INDEX)
+{
+  if (threadIdx.x == 0) {
+    data_ptr[INDEX] = 256;


Are two writes needed?

adayton1 · 2025-06-23T18:01:40Z

tests/tools/sanitizers/sanitizer_tests.cpp


-  if (strategy.find("QuickPool") != std::string::npos) {
-    auto pool = rm.makeAllocator<umpire::strategy::QuickPool>("test_allocator", rm.getAllocator("HOST"));
+  if ((resource_type.find("DEVICE") != std::string::npos) || (resource_type.find("UM") != std::string::npos)) {


I think we might actually want == instead of find.

adayton1 · 2025-06-23T18:06:06Z

tests/tools/sanitizers/sanitizer_test_runner.py

+    run_sanitizer_test('DynamicPoolList', 'read', 'HOST')
+    run_sanitizer_test('DynamicPoolList', 'write', 'HOST')
+    run_sanitizer_test('QuickPool', 'read', 'HOST')
+    run_sanitizer_test('QuickPool', 'write', 'HOST')


Any device tests?

adayton1 · 2025-06-24T22:56:09Z

What's the current status of this merge request? I'm testing out asan and finding it works for simple tests. I'm working on enabling it in my application, but until this goes in, I'll have to disable memory pools.

Whether I'm running on rzvernal or rzadams, I see tests fail with memory leaks detected. So for example, on rzvernal:

[belcher6@rzvernal11:build]$ ctest -T test -R memory_resource_traits_tests --output-on-failure
   Site: rzvernal11
   Build name: Linux-amdclang++
Test project /g/g0/belcher6/Umpire/build
    Start 28: memory_resource_traits_tests
1/1 Test #28: memory_resource_traits_tests .....***Failed    0.51 sec
Running main() from /g/g0/belcher6/Umpire/blt/thirdparty_builtin/googletest/googletest/src/gtest_main.cc
[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from MemoryResourceTraits/MemoryResourceTraitsTest
[ RUN      ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/0
[       OK ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/0 (0 ms)
[ RUN      ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/1
[       OK ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/1 (1 ms)
[ RUN      ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/2
[       OK ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/2 (0 ms)
[ RUN      ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/3
[       OK ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/3 (0 ms)
[----------] 4 tests from MemoryResourceTraits/MemoryResourceTraitsTest (1 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 1 test suite ran. (1 ms total)
[  PASSED  ] 4 tests.

=================================================================
==1444299==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 648 byte(s) in 9 object(s) allocated from:
    #0 0x5ea53d in operator new(unsigned long) (/g/g0/belcher6/Umpire/build/bin/memory_resource_traits_tests+0x5ea53d)
    #1 0x1555495c67e3  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x617e3) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)

Indirect leak of 384 byte(s) in 8 object(s) allocated from:
    #0 0x5ac189 in calloc (/g/g0/belcher6/Umpire/build/bin/memory_resource_traits_tests+0x5ac189)
    #1 0x15554966fc66  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x10ac66) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)

SUMMARY: AddressSanitizer: 1032 byte(s) leaked in 17 allocation(s).

or on rzadams:

[belcher6@rzadams1060:build]$ ctest -T test -R blt_hip_smoke --output-on-failure
   Site: rzadams1001
   Build name: Linux-amdclang++
Test project /g/g0/belcher6/Umpire/build
    Start 3: blt_hip_smoke
1/1 Test #3: blt_hip_smoke ....................***Failed    0.50 sec

=================================================================
==1973496==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 360 byte(s) in 5 object(s) allocated from:
    #0 0x34555d in operator new(unsigned long) (/g/g0/belcher6/Umpire/build/tests/blt_hip_smoke+0x34555d)
    #1 0x1555495c67e3  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x617e3) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)

Indirect leak of 192 byte(s) in 4 object(s) allocated from:
    #0 0x3071a9 in calloc (/g/g0/belcher6/Umpire/build/tests/blt_hip_smoke+0x3071a9)
    #1 0x15554966fc66  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x10ac66) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)

SUMMARY: AddressSanitizer: 552 byte(s) leaked in 9 allocation(s).

I did not see any leaks with rocm 6.4.1 on rzadams with blt_hip_smoke or resource_manager_tests!

kab163 · 2025-07-03T19:17:22Z

What's the current status of this merge request? I'm testing out asan and finding it works for simple tests. I'm working on enabling it in my application, but until this goes in, I'll have to disable memory pools.

Whether I'm running on rzvernal or rzadams, I see tests fail with memory leaks detected. So for example, on rzvernal:

[belcher6@rzvernal11:build]$ ctest -T test -R memory_resource_traits_tests --output-on-failure
   Site: rzvernal11
   Build name: Linux-amdclang++
Test project /g/g0/belcher6/Umpire/build
    Start 28: memory_resource_traits_tests
1/1 Test #28: memory_resource_traits_tests .....***Failed    0.51 sec
Running main() from /g/g0/belcher6/Umpire/blt/thirdparty_builtin/googletest/googletest/src/gtest_main.cc
[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from MemoryResourceTraits/MemoryResourceTraitsTest
[ RUN      ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/0
[       OK ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/0 (0 ms)
[ RUN      ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/1
[       OK ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/1 (1 ms)
[ RUN      ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/2
[       OK ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/2 (0 ms)
[ RUN      ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/3
[       OK ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/3 (0 ms)
[----------] 4 tests from MemoryResourceTraits/MemoryResourceTraitsTest (1 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 1 test suite ran. (1 ms total)
[  PASSED  ] 4 tests.

=================================================================
==1444299==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 648 byte(s) in 9 object(s) allocated from:
    #0 0x5ea53d in operator new(unsigned long) (/g/g0/belcher6/Umpire/build/bin/memory_resource_traits_tests+0x5ea53d)
    #1 0x1555495c67e3  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x617e3) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)

Indirect leak of 384 byte(s) in 8 object(s) allocated from:
    #0 0x5ac189 in calloc (/g/g0/belcher6/Umpire/build/bin/memory_resource_traits_tests+0x5ac189)
    #1 0x15554966fc66  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x10ac66) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)

SUMMARY: AddressSanitizer: 1032 byte(s) leaked in 17 allocation(s).

or on rzadams:

[belcher6@rzadams1060:build]$ ctest -T test -R blt_hip_smoke --output-on-failure
   Site: rzadams1001
   Build name: Linux-amdclang++
Test project /g/g0/belcher6/Umpire/build
    Start 3: blt_hip_smoke
1/1 Test #3: blt_hip_smoke ....................***Failed    0.50 sec

=================================================================
==1973496==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 360 byte(s) in 5 object(s) allocated from:
    #0 0x34555d in operator new(unsigned long) (/g/g0/belcher6/Umpire/build/tests/blt_hip_smoke+0x34555d)
    #1 0x1555495c67e3  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x617e3) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)

Indirect leak of 192 byte(s) in 4 object(s) allocated from:
    #0 0x3071a9 in calloc (/g/g0/belcher6/Umpire/build/tests/blt_hip_smoke+0x3071a9)
    #1 0x15554966fc66  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x10ac66) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)

SUMMARY: AddressSanitizer: 552 byte(s) leaked in 9 allocation(s).

I did not see any leaks with rocm 6.4.1 on rzadams with blt_hip_smoke or resource_manager_tests!

How did you build it?

adding asan sanitizer support for hip

a7e72b3

kab163 requested a review from davidbeckingsale November 23, 2022 23:20

github-actions and others added 3 commits November 23, 2022 23:20

Apply style updates

f90a7af

WIP adding a hip kernel sanitizer test

00e3f43

Apply style updates

ac5cd62

kab163 and others added 7 commits December 7, 2022 11:39

adding usage checkers to sanitizer test

522a4a9

Apply style updates

3ac6a7c

more updates to sanitizer test for error checking

a4f007a

merging...

a1c0180

Apply style updates

8b9e194

adding hip define macros

6617aba

Apply style updates

4c17280

getting rid of double pointer

5379807

kab163 and others added 2 commits December 21, 2022 09:31

updating sanitizer test

4e4804f

Apply style updates

4b92502

Merge branch 'develop' of github.com:LLNL/Umpire into task/UM-1020-as…

276f15b

…an-poisoning-for-hip

kab163 added 2 commits September 18, 2023 08:06

Merge branch 'develop' of github.com:LLNL/Umpire into task/UM-1020-as…

986c5a1

…an-poisoning-for-hip

Merge branch 'develop' of https://github.com/LLNL/Umpire into task/UM…

3858c0f

…-1020-asan-poisoning-for-hip

kab163 added 3 commits January 22, 2024 10:40

adding flag for hip asan

e9bf9f6

updates for rocm test

0232f2c

had to update the test runner to handle new paramter

f154c08

kab163 marked this pull request as ready for review January 24, 2024 16:10

davidbeckingsale and others added 3 commits June 7, 2024 11:36

remove extra linker flags

f2eb907

Add xnack+ to target for memleak job

1332301

Increase allocation on tioga due to memleak job

9022631

Merge branch 'develop' into task/UM-1020-asan-poisoning-for-hip

c458f02

kab163 dismissed rhornung67’s stale review via c458f02 July 16, 2024 17:58

Merge branch 'develop' of github.com:LLNL/Umpire into task/UM-1020-as…

8a5057e

…an-poisoning-for-hip

Merge branch 'develop' of github.com:LLNL/Umpire into task/UM-1020-as…

9898cb5

…an-poisoning-for-hip

kab163 added 2 commits March 13, 2025 07:52

Merge branch 'develop' of github.com:LLNL/Umpire into task/UM-1020-as…

749cd46

…an-poisoning-for-hip

Merge branch 'develop' of github.com:LLNL/Umpire into task/UM-1020-as…

9b95a36

…an-poisoning-for-hip

adayton1 added 3 commits June 20, 2025 16:05

Resolve merge conflicts

3aeb7f2

Update to 6.4.1 for hip asan job

c0cc484

Move back to rocm 6.3.1

1e42362

adayton1 reviewed Jun 23, 2025

View reviewed changes

merge with develop

3073c9a

Adding asan sanitizer support for hip #795

Are you sure you want to change the base?

Adding asan sanitizer support for hip #795

Uh oh!

Conversation

kab163 commented Nov 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kab163 commented Dec 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kab163 commented Dec 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kab163 commented Dec 8, 2022

Uh oh!

kab163 commented Dec 21, 2022

Uh oh!

kab163 commented Mar 2, 2023

Uh oh!

kab163 commented Apr 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kab163 commented Jan 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kab163 commented Jan 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adrienbernede commented Jun 10, 2024

Uh oh!

kab163 commented Dec 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kab163 commented Dec 23, 2024

Uh oh!

adayton1 commented Jan 17, 2025

Uh oh!

adayton1 commented Mar 12, 2025

Uh oh!

kab163 commented Mar 12, 2025

Uh oh!

kab163 commented Mar 12, 2025

Uh oh!

kab163 commented Mar 12, 2025

Uh oh!

kab163 commented Mar 12, 2025

Uh oh!

adayton1 commented Mar 12, 2025

Uh oh!

kab163 commented Mar 13, 2025

Uh oh!

kab163 commented Mar 24, 2025

Uh oh!

adayton1 Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

adayton1 Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

adayton1 Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

adayton1 commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kab163 commented Jul 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

kab163 commented Nov 23, 2022 •

edited

Loading

kab163 commented Dec 6, 2022 •

edited

Loading

kab163 commented Dec 7, 2022 •

edited

Loading

kab163 commented Apr 3, 2023 •

edited

Loading

kab163 commented Jan 22, 2024 •

edited

Loading

kab163 commented Jan 24, 2024 •

edited

Loading

kab163 commented Dec 17, 2024 •

edited

Loading

adayton1 commented Jun 24, 2025 •

edited

Loading