Triton update introduced "unknown extension 'SPV_KHR_bfloat16'"

[Stonepia]

Describe the bug

When testing on helion's test, we got an error message:

$cd helion
$python -m pytest -sv test/test_dot.py  -k test_addmm_multiple_small_dims

InvalidModule: Invalid SPIR-V module: input SPIR-V module uses unknown extension 'SPV_KHR_bfloat16'

Reproducer

No Need to install helion. Just run the reproducer is enough.

# Run triton reproducer
cd SPV_KHR_bfloat16_repro
python SPV_KHR_bfloat16_repro.py

Attached files

SPV_KHR_bfloat16_repro.zip

Also attach the log of:

MLIR_ENABLE_DUMP=1 python repro.py

success_triton_3.5.log

failure_triton_3.6.log

Environment details

Workable Triton: pytorch-triton-xpu-3.5.0+git1b0418a9
Failure Triton: 3.6.0+gitd3a28a60 (nightly whl for wheels-pytorch-py3.12-20251126)

Torch: 2.10.0.dev20251123+xpu

Testing machine: PVC
uname -a : Linux 984fee015d7d.jf.intel.com 6.8.0-60-generic #63-Ubuntu SMP PREEMPT_DYNAMIC Tue Apr 15 19:04:15 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

[anmyachev]

The use of some features is determined at build time, so if you build in one environment and run in another, it won't work:

intel-xpu-backend-for-triton/setup.py

Line 499 in ba1d008

cmake_args.append(f"-DAGAMA_VERSION={agama_version}")

@whitneywhtsang it looks like this solution is worse than we thought.

[whitneywhtsang]

I can reproduce the good case with the latest Triton.
@Stonepia What driver is it using? pure LTS2, or new IGC?

[chuanqi129]

The use of some features is determined at build time, so if you build in one environment and run in another, it won't work

@anmyachev @whitneywhtsang Is it true? We also meet similar issue when we do the new round acceptance test by using the triton wheel in https://github.com/pytorch/pytorch/actions/runs/19624769131?pr=168950. If so, I think it's can't be upgraded into pytorch
cc: @etaf @EikanWang @riverliuintel

[Stonepia]

I can reproduce the good case with the latest Triton. @Stonepia What driver is it using? pure LTS2, or new IGC?

Seems that my dev machine is not the LTS2 and on the old driver, but @chuanqi129 's acceptance test should be on the LTS2.

This is the igc on my dev machine:

libigc-dev                           2.10.11-1133~24.04    

This is the igc on LTS2 according to https://dgpu-docs.intel.com/releases/packages.html?release=LTS+2523.31&os=Ubuntu+24.04

libigc-dev_2.11.27-1146~24.04_amd64.deb

[whitneywhtsang]

Seems that my dev machine is not the LTS2 and on the old driver

It is expected that old driver doesn't work. We tried to workaround for LTS2.

The use of some features is determined at build time, so if you build in one environment and run in another, it won't work

Is it true?

@chuanqi129 It is unclear at this moment what is the situation that doesn't work. We added a patch to SPIRV translator to workaround problems when SPV_KHR_bfloat16 is not available on LTS2. Do you build on LTS2 and run on Rolling, or the other way around?

[chuanqi129]

Hi @whitneywhtsang , the triton build in pytorch CD based on the cpu-only docker image, which is same with cuda and rocm, there is no driver installed, refer https://github.com/pytorch/pytorch/blob/main/.github/workflows/build-triton-wheel.yml#L55. Why we need to depends on driver now?

[riverliuintel]

PyTorch 2.10 upgrade block issue, please cherry pick this regression issue fix in release branch.