[Issue]: [Bug] GetSegmentId fails for type 1 (MEMORY) on discrete RX 9060 XT (gfx1200) in WSL2 — VramAvail returns HSA_STATUS_ERROR, PyTorch sees 0 bytes free

[Hyroshima]

Problem Description

Environment

• GPU: AMD Radeon RX 9060 XT (gfx1200 / RDNA 4 / Navi 44)
• VRAM: 16 GB GDDR6
• OS (Windows): Windows 10 22H2 (Build 19045)
• AMD Driver: Adrenalin Edition 26.2.2
• WSL2: Ubuntu 24.04 LTS
• ROCm: 7.2.1
• librocdxg: built from main branch (v1.1.1)
• PyTorch: 2.9.1+rocm7.2.1

Problem Description

On a discrete RX 9060 XT GPU, GetSegmentId consistently fails for segment type 1
(D3DKMT_QUERYSTATISTICS_SEGMENT_TYPE_MEMORY), which is the local/dedicated VRAM segment.
This causes VramAvail() to return HSA_STATUS_ERROR, which PyTorch interprets as
0 bytes free, making all model loading fail with loaded partially; 0.00 MB usable.

The GPU is correctly detected by rocminfo, HSA_ENABLE_DXG_DETECTION=1 is set,
/dev/dxg is present, and torch.cuda.is_available() returns True.
However, torch.cuda.mem_get_info() returns (4096, 16987488256) — only 4 KB free
despite 16 GB total VRAM, making it impossible to load any model.

Operating System

WSL2 Ubuntu 24.04.4

CPU

Ryzen 9 3900

GPU

RX 9060 XT

ROCm Version

7.2.1.70201-81~24.04

ROCm Component

No response

Steps to Reproduce

Reproduction Steps

1. Install ROCm 7.2.1 on WSL2 Ubuntu 24.04 with amdgpu-install --usecase=rocm --no-dkms
2. Build and install librocdxg from source with Windows SDK path
3. Set HSA_ENABLE_DXG_DETECTION=1
4. Run rocminfo — GPU is detected correctly (Agent 2, gfx1200)
5. Run python -c "import torch; print(torch.cuda.mem_get_info())" → (4096, 16987488256)
6. Try loading any model in PyTorch or ComfyUI → loaded partially; 0.00 MB usable, 0.00 MB loaded

Debug Output

Running with HSAKMT_DEBUG_LEVEL=7:

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

WSL environment detected.

HSA System Attributes

Runtime Version: 1.18
Runtime Ext Version: 1.15
System Timestamp Freq.: 1000.000000MHz
Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model: LARGE
System Endianness: LITTLE
Mwaitx: DISABLED
XNACK enabled: NO
DMAbuf Support: YES
VMM Support: YES

==========
HSA Agents

---

Agent 1

---

Name: AMD Ryzen 9 3900 12-Core Processor
Uuid: CPU-XX
Marketing Name: AMD Ryzen 9 3900 12-Core Processor
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 0
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
Cacheline Size: 64(0x40)
BDFID: 0
Internal Node ID: 0
Compute Unit: 8
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
Memory Properties:
Features: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 28739804(0x1b688dc) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED
Size: 28739804(0x1b688dc) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 3
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED
Size: 28739804(0x1b688dc) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 4
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 28739804(0x1b688dc) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
ISA Info:

---

Agent 2

---

Name: gfx1200
Uuid: GPU-aa3d0bc8364af119
Marketing Name: AMD Radeon RX 9060 XT
Vendor Name: AMD
Feature: KERNEL_DISPATCH
Profile: BASE_PROFILE
Float Round Mode: NEAR
Max Queue Number: 128(0x80)
Queue Min Size: 64(0x40)
Queue Max Size: 131072(0x20000)
Queue Type: MULTI
Node: 1
Device Type: GPU
Cache Info:
L1: 32(0x20) KB
L3: 32768(0x8000) KB
Chip ID: 30096(0x7590)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 2700
BDFID: 3072
Internal Node ID: 1
Compute Unit: 32
SIMDs per CU: 2
Shader Engines: 2
Shader Arrs. per Eng.: 2
Coherent Host Access: FALSE
Memory Properties:
Features: KERNEL_DISPATCH
Fast F16 Operation: TRUE
Wavefront Size: 32(0x20)
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Max Waves Per CU: 32(0x20)
Max Work-item Per CU: 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 2147483647(0x7fffffff)
y 65535(0xffff)
z 65535(0xffff)
Max fbarriers/Workgrp: 32
Packet Processor uCode:: 108
SDMA engine uCode:: 0
IOMMU Support:: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 16589344(0xfd2220) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 2
Segment: GROUP
Size: 64(0x40) KB
Allocatable: FALSE
Alloc Granule: 0KB
Alloc Recommended Granule:0KB
Alloc Alignment: 0KB
Accessible by all: FALSE
ISA Info:
ISA 1
Name: amdgcn-am

Additional Information

[QuerySegmentInfo] Total Segments: 4
[GetSegmentId] Failed to get segment id for type 1

The GPU has 4 segments, but none of them matches
D3DKMT_QUERYSTATISTICS_SEGMENT_TYPE_MEMORY (type 1).
The GetSegmentId call in VramAvail() therefore always fails and returns
HSA_STATUS_ERROR.

rocminfo correctly reports 16 GB VRAM in Pool Info:

Pool 1
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 16589344(0xfd2220) KB
Allocatable: TRUE

---

Root Cause Analysis

The issue is in WDDMDevice::VramAvail() in src/wddm/device.cpp:

if (!GetSegmentId(D3DKMT_QUERYSTATISTICS_SEGMENT_TYPE_MEMORY, segmentId))
    return HSA_STATUS_ERROR;

GetSegmentId iterates over segment_infos_ (populated by QuerySegmentInfo)
and looks for a segment whose segment_type matches
D3DKMT_QUERYSTATISTICS_SEGMENT_TYPE_MEMORY. However, on the RX 9060 XT (gfx1200),
none of the 4 reported segments has this type — suggesting that ParseAdapterInfo
(in the closed-source libthunk_proxy.a) is not correctly classifying the dedicated
VRAM segment as type MEMORY for this discrete RDNA 4 GPU in WSL2.

As a result:

• VramAvail() always returns HSA_STATUS_ERROR
• PyTorch's mem_get_info() sees only 4 KB free (near-zero sentinel value)
• All model inference fails — GPU compute works fine but memory allocation does not

---

Additional Notes

• Small allocations work: torch.zeros(100).cuda() succeeds
• Large allocations fail silently (process hangs with no output)
• IsDgpu() returns true correctly (the GPU is recognized as discrete)
• LocalHeapSize() returns the correct 16 GB value (used by rocminfo)
• The problem is isolated to the segment type classification in ParseAdapterInfo
  inside libthunk_proxy.a, which cannot be patched without access to its source

This issue appears specific to the discrete RX 9060 XT (gfx1200) in WSL2.
APU-based setups (Strix Halo, etc.) seem to have a different but related memory
mapping issue (#6022 on ROCm/ROCm).

---

Expected Behavior

torch.cuda.mem_get_info() should return approximately (16_000_000_000, 16_987_488_256),
and model loading should use GPU VRAM as on native Linux or Windows.

[longlyao-amd]

Hi @Hyroshima, Could you please update your Windows OS to Windows 11 (22H2) and try again? Please also refer to the requirements in the README: https://github.com/ROCm/librocdxg#wsl-compatiblity-matrix

[K-H-Tamim]

I ran into a similar problem few weeks back where program with large memory allocation would fail but small memory allocation worked fine on rocm7.2 in WSL(win11 25H2) via librocdxg. Values reported by rocminfo maybe correct but can be misleading..at least in my case it was.

@Hyroshima, You should check 2 things.

1. check whether ReBAR is enabled for you motherboard or not. Use GPU-Z or Adrenalin's Performance section for ReBAR/Smart Access Memory related info. Or find ReBAR/Above-4G Decoding related configs in BIOS.
2. check local_visible_heap_size and local_invisible_heap_size after ParseAdapterInfo() call in device.cpp:565. Add pr_info/pr_err for these, then build and install. Now run rocminfo. That should print the values.

The correct values for your gpu would depend on status of ReBAR-

1. If enabled, you should see 16GB for local_visible_heap_size and 0 for local_invisible_heap_size. or 8GB for each in some cases cause some BIOS can cap the ReBAR at 8GB.
2. If disabled, you should see 256MB for local_visible_heap_size and 16128MB for local_invisible_heap_size,

If ReBAR IS disabled and you see 0 for local_invisible_heap_size, that would mean ParseAdapterInfo() not setting that property correctly for some reason.

[Hyroshima]

@longlyao-amd, I'm going to install Windows 11 on another SSD and start from scratch again (since I can't upgrade my Windows 10 right now).

@K-H-Tamim, The Resizable BAR is enabled in Windows. I'll try to verify if this information is reflected in WSL2, then I can test it. I'll also test it on a new Windows 11 installation to see if what happened to me in Windows 10 also happens in Windows 11.

[Hyroshima]

@K-H-Tamim
Thank you for the suggestion! I tested both scenarios as you recommended.

Results of local_visible_heap_size and local_invisible_heap_size check:

After adding the pr_err logs right after ParseAdapterInfo() in device.cpp:

[ParseDeviceInfo] local_visible_heap_size: 16200 MB
[ParseDeviceInfo] local_invisible_heap_size: 0 MB
[ParseDeviceInfo] is_dgpu: 1

ReBAR is enabled on my system (confirmed via GPU-Z: BAR0 = 16384 MB, Resizable BAR = Enabled), so the values match your expected scenario 1 — local_visible_heap_size = 16GB and local_invisible_heap_size = 0.

ParseAdapterInfo() is correctly populating these fields.

However, VramAvail() still fails because GetSegmentId(D3DKMT_QUERYSTATISTICS_SEGMENT_TYPE_MEMORY, segmentId) returns false for all 4 reported segments — none of them has type MEMORY. The fix that worked for me was:

if (!GetSegmentId(D3DKMT_QUERYSTATISTICS_SEGMENT_TYPE_MEMORY, segmentId)) {
    if (IsDgpu())
        return LocalHeapSize();
    return HSA_STATUS_ERROR;
}

After this fix, torch.cuda.mem_get_info() correctly returns (16987488256, 16987488256) — 16GB free.

@longlyao-amd

Update: Windows 10 investigation + GPU compute hang

Regarding the Windows 11 recommendation from @longlyao-amd:

I am currently on Windows 10 22H2 (Build 19045) and cannot upgrade to Windows 11 at this time. I understand this may be outside the officially supported configuration per the README compatibility matrix. However, I was able to make significant progress and document the behavior in detail, which may help identify the root cause regardless of OS version.

---

Summary of findings

After applying the VramAvail() fix (returning LocalHeapSize() when GetSegmentId fails on a dGPU with ReBAR), memory allocation works correctly:

• torch.cuda.mem_get_info() → (16987488256, 16987488256) ✅
• torch.cuda.is_available() → True ✅
• Small GPU operations (≤ 512×512 matrix multiply) → work correctly ✅
• Large GPU operations (> 512×512) → hang indefinitely ❌

---

Root cause investigation

I added extensive logging to trace the hang. The full execution chain for a 513×513 matrix multiply:

1. Process() called — starts correctly
2. SwitchAql2PM4() — returns HSA_STATUS_SUCCESS
3. UpdateScratch() — private_segment_size=0, no realloc needed
4. SwsSubmit() — D3DKMTSubmitCommand succeeds
5. GpuSignal() — D3DKMTSignalSynchronizationObjectFromGpu returns STATUS_SUCCESS
6. PyTorch hangs waiting for GPU result — cat /proc/<pid>/wchan shows vmbus_teardown_gpadl

The librocdxg side completes successfully. The hang is on the PyTorch/ROCm runtime side waiting for a GPU completion signal that never arrives.

Key data point: The exact boundary is 512×512 elements. Operations of exactly 512×512 complete successfully; 513×513 hangs. This suggests the limit is related to command buffer size or the number of wavefronts dispatched, not memory capacity.

Additional context:

• platform_atomic_support: 1 (confirmed via logging)
• local_visible_heap_size: 16200 MB / local_invisible_heap_size: 0 MB / is_dgpu: 1
• ReBAR fully enabled (BAR0 = 16384 MB per GPU-Z)
• Tested with PyTorch 2.11.0+rocm7.2, 2.9.1+rocm7.2.1, and TheRock nightly (rocm 7.13.0a20260425) — same result in all versions
• Tested with all branches in this repo (develop, longlyao/fix-gpu-memory-usage, longlyao/revert-scratch-imp) — same result

Environment:

• GPU: AMD Radeon RX 9060 XT (gfx1200 / RDNA 4)
• Windows: 10 22H2 (Build 19045)
• Driver: Adrenalin 26.2.2
• WSL2: Ubuntu 24.04
• ROCm: 7.2.1 + librocdxg v1.1.1 (develop branch)

Further investigation: GPU compute hang + full test results

Following up on the VramAvail() fix from my previous comment — while memory allocation now works correctly, I continued investigating why GPU compute still fails for larger operations.

---

ROCm/HIP compute hang — root cause analysis

Exact boundary: 512×512 matrix multiply works, 513×513 hangs forever.

Full execution trace via logging added to librocdxg (513×513 case):

1. Process() → starts correctly ✅
2. SwitchAql2PM4() → HSA_STATUS_SUCCESS ✅
3. UpdateScratch() → private_segment_size=0, no realloc needed ✅
4. SwsSubmit() → D3DKMTSubmitCommand succeeds ✅
5. GpuSignal() → D3DKMTSignalSynchronizationObjectFromGpu returns STATUS_SUCCESS ✅
6. PyTorch hangs — cat /proc/<pid>/wchan shows vmbus_teardown_gpadl ❌

The librocdxg side completes successfully. The hang is in the PyTorch/ROCm runtime waiting for a GPU completion signal that never arrives from the Windows driver side.

Tested with all branches in this repo (develop, longlyao/fix-gpu-memory-usage, longlyao/revert-scratch-imp) and multiple PyTorch versions (2.9.1+rocm7.2.1, 2.11.0+rocm7.2, TheRock nightly rocm7.13.0a20260425) — same result in all.

---

Full test results

❌ Failed (ROCm/HIP backend)

Tool	Result	Notes
PyTorch matrix ops > 512×512	❌ Hangs	vmbus_teardown_gpadl in wchan
ComfyUI (--disable-pinned-memory)	❌ Hangs at 0%	Model loads into VRAM (~9GB), compute never completes
ComfyUI (--directml)	❌ Partial	Only 1024MB VRAM visible, falls back to CPU
Docker + ROCm container	❌ Hangs	Same bug, environment-independent
Ollama (ROCm)	❌ CPU only	Bundled ROCm ignores system librocdxg
PyTorch matrix ops ≤ 512×512	✅ Works	Small operations complete correctly

✅ Working (Vulkan via Microsoft Direct3D12)

After installing Mesa with RADV (ppa:kisak/kisak-mesa), the GPU appears as Microsoft Direct3D12 (AMD Radeon RX 9060 XT) via Vulkan, enabling full GPU acceleration completely bypassing ROCm/HIP:

Tool	Result	Performance
llama.cpp (compiled with -DGGML_VULKAN=ON)	✅ GPU	~34 tokens/s on Qwen2.5 7B
Ollama (with OLLAMA_VULKAN=1)	✅ 100% GPU	~42 tokens/s on Qwen2.5 7B
stable-diffusion.cpp (compiled with -DSD_VULKAN=ON)	✅ GPU	~3.8s/image (SDXL Turbo, 512×512, 4 steps)
ComfyUI	❌ No Vulkan	PyTorch has no Vulkan backend

Mesa installation:

sudo add-apt-repository ppa:kisak/kisak-mesa -y
sudo apt update && sudo apt upgrade -y
sudo apt install -y mesa-vulkan-drivers

---

I will still install another SSD in the computer so I can run tests with Windows 11.
I will post the results soon.

[K-H-Tamim]

@Hyroshima, Interesting findings! Specially how ROCm hanging on compute > 512x512. You should do a sanity check for all the device_info_ props, to see what values are passed down from Windows 10 kernel to WSL ROCm via librocdxg. Print all the props of device_info_ after ParseAdapterInfo() call,

Then Give Claude's sonnet the thunk_proxy.h, ask it to populate the DeviceInfo struct for you GPU(mention gfx with name) by going through open source Linux diver files for AMD GPUs.

Cross check between what was printed and what values Claude populated DeviceInfo with for your gpu. If you see any difference, try overwriting device_info_ props with the values from Claude's answer. Claude could be wrong, so check the reference Claude includes in the answer.

Although, I think it's better to try your tests with Windows 11 if you can manage that more easily than printing, cross checking and overwriting.

[fcui-amd]

The segment type is introduced in WDDM 3.1, which is tied to Windows 11(22H2) (https://learn.microsoft.com/en-us/windows-hardware/drivers/display/windows-vista-display-driver-model-design-guide).
Windows 10 is not supported.
If you still want to experiment on Windows 10 with a dGPU, you can try hard‑coding the segment IDs to 0 and 1.
In addition, GPU hardware scheduling (HWS) is not enabled on your system, whereas on Windows it is normally enabled by default.

[Hyroshima]

Update: Windows 11 Results — ROCm + WSL2 Fully Working

After formatting and installing Windows 11 (Build 28020), I was able to reproduce and resolve all issues reported previously. Here is a complete summary of findings.

---

Environment

• GPU: AMD Radeon RX 9060 XT (gfx1200 / RDNA 4 / Navi 44)
• VRAM: 16 GB GDDR6
• OS: Windows 11 (Build 28020)
• AMD Driver: Adrenalin Edition 26.5.1
• WSL2: Ubuntu 24.04.4 LTS
• Kernel: 6.6.114.1-microsoft-standard-WSL2
• ROCm: 7.2.3
• librocdxg: compiled from source (develop branch)
• PyTorch: 2.11.0+rocm7.2

---

Key Finding: Windows 11 Resolves All Core Issues

As confirmed by @fcui-amd, the segment type classification used by VramAvail() requires WDDM 3.1, which is only available on Windows 11 (22H2+). All memory-related failures on Windows 10 were caused by this OS-level limitation, not by the GPU or ROCm stack.

---

PyTorch — GPU Memory

torch.cuda.is_available() → True
torch.cuda.get_device_name(0) → AMD Radeon RX 9060 XT
torch.cuda.mem_get_info() → (14.15 GB free, 15.81 GB total)

Previously on Windows 10: (4096, 16987488256) — only 4 KB free.

---

PyTorch — Compute Hang Fixed

The 512×512 boundary hang that occurred on Windows 10 is completely resolved:

512x512  matrix multiply → OK ✅
513x513  matrix multiply → OK ✅
2048x2048 matrix multiply → OK ✅

---

ComfyUI — ROCm Native

ComfyUI running with ROCm on WSL2 is faster than Windows native with DirectML:

Environment | Time (ace_step 1.5 turbo) -- | -- Windows (DirectML) | 102 seconds WSL2 ROCm | 72 seconds

3.7x faster with Vulkan.

---

Critical Note for Tool Compatibility

Any tool that uses pre-compiled ROCm binaries (pip packages, official installers, Ollama) will not detect the GPU via librocdxg on WSL2. Only tools compiled manually against the system ROCm installation work correctly:

• ✅ llama.cpp compiled with -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1200
• ✅ PyTorch installed via pip install torch --index-url https://download.pytorch.org/whl/rocm7.2
• ✅ ComfyUI using the PyTorch ROCm installation above
• ❌ Ollama — uses bundled ROCm, ignores system librocdxg (Vulkan workaround available via OLLAMA_VULKAN=1)

---

Installation Steps That Worked

bash

# 1. Install ROCm 7.2.3
wget https://repo.radeon.com/amdgpu-install/7.2.3/ubuntu/noble/amdgpu-install_7.2.3.70203-1_all.deb
sudo apt install -y ./amdgpu-install_7.2.3.70203-1_all.deb
sudo amdgpu-install --usecase=rocm --no-dkms -y

# 2. Build and install librocdxg
git clone https://github.com/ROCm/librocdxg.git
cd librocdxg
export win_sdk='/mnt/c/Program Files (x86)/Windows Kits/10/Include/10.0.28000.0/'
mkdir -p build && cd build
cmake .. -DWIN_SDK="${win_sdk}/shared"
make && sudo make install

# 3. Set environment variable
export HSA_ENABLE_DXG_DETECTION=1

# 4. Verify
rocminfo  # Should show Agent 2: AMD Radeon RX 9060 XT (gfx1200)

# 5. Install PyTorch with ROCm
pip3 install torch torchvision torchaudio \
    --index-url https://download.pytorch.org/whl/rocm7.2

# 6. Build llama.cpp with ROCm
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
HSA_ENABLE_DXG_DETECTION=1 cmake -B build \
    -DGGML_HIP=ON \
    -DAMDGPU_TARGETS=gfx1200 \
    -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc)

---

Conclusion

The RX 9060 XT (gfx1200) works fully with ROCm on WSL2 under Windows 11. The issues reported in the original bug were specific to Windows 10's lack of WDDM 3.1 support. With Windows 11, GPU memory is correctly reported, compute works at all sizes, and performance exceeds Windows native DirectML in real workloads.