Run FastFlowLM on the HRX amdxdna runtime - #602
Merged
Conversation
Abhishek-Varma
force-pushed
the
hrx_native_pr
branch
4 times, most recently
from
July 17, 2026 06:29
8cbfc51 to
6695b50
Compare
Abhishek-Varma
force-pushed
the
hrx_native_pr
branch
9 times, most recently
from
July 28, 2026 09:37
99a3d53 to
f2c2dbe
Compare
Abhishek-Varma
added a commit
to Abhishek-Varma/FastFlowLM_JT
that referenced
this pull request
Aug 5, 2026
Rebased onto latest main: reintroduces XRT as the default, selectable NPU dispatch backend alongside HRX behind a CMake flag (FLM_USE_HRX: 0=XRT default, 1=HRX), so PR ROCm#602 stays mergeable and easy to sync with main while HRX remains opt-in. Squashes the prior HRX PR commits + the flag work into a single change on top of origin/main. - CMake: FLM_USE_HRX option + compile definition; runtime discovery/link/ bundle branch between XRT (pkg-config/manual/fetch, xrt_coreutil + aiebu) and HRX (find_package(hrx), hrx::hrx); flag-selected engine lib dir. - device_runtime.hpp: neutral flm_rt alias -> xrt or hrx; app-facing hrx:: replaced with flm_rt:: across models/runner/server/tests. - npu_utils.hpp split into _xrt (from main) / _hrx (PR) + dispatcher; buffer.hpp bo sync gated (xrt sync() vs hrx flush/invalidate). - main.cpp: XRT libxrt preload restored under !FLM_USE_HRX. - Engine libs reorganized into src/lib/xrt (refreshed from latest main) and src/lib/hrx (aiebu XRT-only); wrapper + src/test/common.mk backend-aware. - Folds in main updates: modelscope ctor arg (rest_handler), model_info.json install, refreshed gpt_oss/qwen3_6_moe XRT engine libs and xclbins. Verified on accel0 against latest main: default XRT build links libxrt_coreutil + lib/xrt and HRX build (-DFLM_USE_HRX=ON) links libhrx + lib/hrx; both answer "2 + 2 = 4". Co-authored-by: Cursor <cursoragent@cursor.com>
Abhishek-Varma
added a commit
to Abhishek-Varma/FastFlowLM_JT
that referenced
this pull request
Aug 5, 2026
Rebased onto latest main: reintroduces XRT as the default, selectable NPU dispatch backend alongside HRX behind a CMake flag (FLM_USE_HRX: 0=XRT default, 1=HRX), so PR ROCm#602 stays mergeable and easy to sync with main while HRX remains opt-in. Squashes the prior HRX PR commits + the flag work into a single change on top of origin/main. - CMake: FLM_USE_HRX option + compile definition; runtime discovery/link/ bundle branch between XRT (pkg-config/manual/fetch, xrt_coreutil + aiebu) and HRX (find_package(hrx), hrx::hrx); flag-selected engine lib dir. - device_runtime.hpp: neutral flm_rt alias -> xrt or hrx; app-facing hrx:: replaced with flm_rt:: across models/runner/server/tests. - npu_utils.hpp split into _xrt (from main) / _hrx (PR) + dispatcher; buffer.hpp bo sync gated (xrt sync() vs hrx flush/invalidate). - main.cpp: XRT libxrt preload restored under !FLM_USE_HRX. - Engine libs reorganized into src/lib/xrt (refreshed from latest main) and src/lib/hrx (aiebu XRT-only); wrapper + src/test/common.mk backend-aware. - Folds in main updates: modelscope ctor arg (rest_handler), model_info.json install, refreshed gpt_oss/qwen3_6_moe XRT engine libs and xclbins. Verified on accel0 against latest main: default XRT build links libxrt_coreutil + lib/xrt and HRX build (-DFLM_USE_HRX=ON) links libhrx + lib/hrx; both answer "2 + 2 = 4". Co-authored-by: Cursor <cursoragent@cursor.com>
Abhishek-Varma
force-pushed
the
hrx_native_pr
branch
from
August 5, 2026 08:04
71c73ed to
1cde1b3
Compare
Abhishek-Varma
added a commit
to Abhishek-Varma/FastFlowLM_JT
that referenced
this pull request
Aug 5, 2026
Rebased onto latest main: reintroduces XRT as the default, selectable NPU dispatch backend alongside HRX behind a CMake flag (FLM_USE_HRX: 0=XRT default, 1=HRX), so PR ROCm#602 stays mergeable and easy to sync with main while HRX remains opt-in. Squashes the prior HRX PR commits + the flag work into a single change on top of origin/main. - CMake: FLM_USE_HRX option + compile definition; runtime discovery/link/ bundle branch between XRT (pkg-config/manual/fetch, xrt_coreutil + aiebu) and HRX (find_package(hrx), hrx::hrx); flag-selected engine lib dir. - device_runtime.hpp: neutral flm_rt alias -> xrt or hrx; app-facing hrx:: replaced with flm_rt:: across models/runner/server/tests. - npu_utils.hpp split into _xrt (from main) / _hrx (PR) + dispatcher; buffer.hpp bo sync gated (xrt sync() vs hrx flush/invalidate). - main.cpp: XRT libxrt preload restored under !FLM_USE_HRX. - Engine libs reorganized into src/lib/xrt (refreshed from latest main) and src/lib/hrx (aiebu XRT-only); wrapper + src/test/common.mk backend-aware. - Folds in main updates: modelscope ctor arg (rest_handler), model_info.json install, refreshed gpt_oss/qwen3_6_moe XRT engine libs and xclbins. Verified on accel0 against latest main: default XRT build links libxrt_coreutil + lib/xrt and HRX build (-DFLM_USE_HRX=ON) links libhrx + lib/hrx; both answer "2 + 2 = 4". Co-authored-by: Cursor <cursoragent@cursor.com>
Abhishek-Varma
force-pushed
the
hrx_native_pr
branch
from
August 5, 2026 08:10
1cde1b3 to
037c3fe
Compare
Abhishek-Varma
added a commit
to Abhishek-Varma/FastFlowLM_JT
that referenced
this pull request
Aug 5, 2026
Rebased onto latest main: reintroduces XRT as the default, selectable NPU dispatch backend alongside HRX behind a CMake flag (FLM_USE_HRX: 0=XRT default, 1=HRX), so PR ROCm#602 stays mergeable and easy to sync with main while HRX remains opt-in. Squashes the prior HRX PR commits + the flag work into a single change on top of origin/main. - CMake: FLM_USE_HRX option + compile definition; runtime discovery/link/ bundle branch between XRT (pkg-config/manual/fetch, xrt_coreutil + aiebu) and HRX (find_package(hrx), hrx::hrx); flag-selected engine lib dir. - device_runtime.hpp: neutral flm_rt alias -> xrt or hrx; app-facing hrx:: replaced with flm_rt:: across models/runner/server/tests. - npu_utils.hpp split into _xrt (from main) / _hrx (PR) + dispatcher; buffer.hpp bo sync gated (xrt sync() vs hrx flush/invalidate). - main.cpp: XRT libxrt preload restored under !FLM_USE_HRX. - Engine libs reorganized into src/lib/xrt (refreshed from latest main) and src/lib/hrx (aiebu XRT-only); wrapper + src/test/common.mk backend-aware. - Folds in main updates: modelscope ctor arg (rest_handler), model_info.json install, refreshed gpt_oss/qwen3_6_moe XRT engine libs and xclbins. Verified on accel0 against latest main: default XRT build links libxrt_coreutil + lib/xrt and HRX build (-DFLM_USE_HRX=ON) links libhrx + lib/hrx; both answer "2 + 2 = 4". Co-authored-by: Cursor <cursoragent@cursor.com>
Abhishek-Varma
force-pushed
the
hrx_native_pr
branch
from
August 5, 2026 08:22
037c3fe to
163672e
Compare
Rebased onto latest main: reintroduces XRT as the default, selectable NPU dispatch backend alongside HRX behind a CMake flag (FLM_USE_HRX: 0=XRT default, 1=HRX), so PR ROCm#602 stays mergeable and easy to sync with main while HRX remains opt-in. Squashes the prior HRX PR commits + the flag work into a single change on top of origin/main. - CMake: FLM_USE_HRX option + compile definition; runtime discovery/link/ bundle branch between XRT (pkg-config/manual/fetch, xrt_coreutil + aiebu) and HRX (find_package(hrx), hrx::hrx); flag-selected engine lib dir. - device_runtime.hpp: neutral flm_rt alias -> xrt or hrx; app-facing hrx:: replaced with flm_rt:: across models/runner/server/tests. - npu_utils.hpp split into _xrt (from main) / _hrx (PR) + dispatcher; buffer.hpp bo sync gated (xrt sync() vs hrx flush/invalidate). - main.cpp: XRT libxrt preload restored under !FLM_USE_HRX. - Engine libs reorganized into src/lib/xrt (refreshed from latest main) and src/lib/hrx (aiebu XRT-only); wrapper + src/test/common.mk backend-aware. - Folds in main updates: modelscope ctor arg (rest_handler), model_info.json install, refreshed gpt_oss/qwen3_6_moe XRT engine libs and xclbins. Verified on accel0 against latest main: default XRT build links libxrt_coreutil + lib/xrt and HRX build (-DFLM_USE_HRX=ON) links libhrx + lib/hrx; both answer "2 + 2 = 4". Co-authored-by: Cursor <cursoragent@cursor.com>
Abhishek-Varma
force-pushed
the
hrx_native_pr
branch
from
August 5, 2026 08:47
163672e to
eb21b01
Compare
Bump the HRX amdxdna pin to flm-hrx-amdxdna-v2026.07.30 (Linux + Windows tags/assets/sha256) and refresh src/lib/hrx/*.so with the 23 engine libraries rebuilt from FastFlowLM_IRON (branch hrx_native_flag) against that pin. Built with -DFLM_USE_HRX=ON; each lib links libhrx. Validated on the NPU: 2+2 smoke passes and greedy decode is an exact match vs the XRT reference for gemma3:1b, qwen3:0.6b, and lfm2.5-it:1.2b. Co-authored-by: Cursor <cursoragent@cursor.com>
…olchain Rebuild the 23 HRX engine .so with gcc-toolset-13 (Red Hat GCC 13.3.1) inside the manylinux_2_28 image so they match the rocm_npu_branch runtime environment exactly: GLIBC floor 2.22, GLIBCXX 3.4.22, CXXABI 1.3.11, libpthread/libgomp/libmvec deps, boost-free. Supersedes the Ubuntu 24.04 (glibc 2.39 / GLIBCXX 3.4.32) build, which would not load on EL8. 2+2 smoke re-verified on the NPU for gemma3:1b, qwen3:0.6b, lfm2.5-it:1.2b. Co-authored-by: Cursor <cursoragent@cursor.com>
…against HRX v2026.07.30 Regenerated the 23 FLM_DLL engine .dll/.lib on Windows (MSVC 14.44, VS2022) from FastFlowLM_IRON@hrx_native_flag with -DFLM_USE_HRX=ON. Verified each links hrx.dll (no xrt_coreutil/aiebu) and is boost-free (no boost_* dependency). flm.exe rebuilt against these; gemma3:1b, qwen3.5:0.8b, lfm2.5-it:1.2b all pass the 2+2 smoke on the amdxdna NPU. Co-authored-by: Cursor <cursoragent@cursor.com>
Collaborator
|
Hi! @Abhishek-Varma I just tested the portable artifacts, which I believe were built against XRT, on both Windows and Linux. The Windows portable build looks good from my testing and I didn't run into any issues. On the Linux portable build, however, both flm run I didn't see this issue with the latest release build, so it seems specific to this portable artifact. |
The portable XRT tarball shipped only libxrt_coreutil (flm's single link-time
NEEDED entry). XRT additionally dlopens libxrt_core.so.2 and the
libxrt_driver_xdna plugin at run time via a path it builds as
$XILINX_XRT/lib/x86_64-linux-gnu/<lib>, so those libraries were absent from the
self-contained tree and `flm run` / `flm serve` failed with:
Error: No such library '.../lib/x86_64-linux-gnu/libxrt_core.so.2'
Bundle the complete libxrt*.so* set from the discovered XRT lib dir, re-root
each real lib's RUNPATH at $ORIGIN so they resolve one another inside lib/, and
mirror them under lib/x86_64-linux-gnu so XRT's internal lookup succeeds
(mirrors the XRT bundling on main). libxrt is now excluded from the flm
dependency closure since this explicit block owns the full runtime set.
Reported on PR ROCm#602 (portable Linux artifact).
Co-authored-by: Cursor <cursoragent@cursor.com>
…dditive
Rework the Linux install/bundling so the XRT build uses the exact same
mechanisms it always has and every HRX-specific step is gated behind
FLM_USE_HRX, instead of a single backend-agnostic closure plus an XRT
add-on.
- Branch the non-Windows bundling by backend:
* XRT portable -> explicit libxrt*/boost/FFTW copy + $ORIGIN patchelf +
multiarch mirror (the XRT runtime is dlopen'd, so it cannot be captured
by a link-time dependency closure).
* XRT .deb -> plain install; deps resolved via ${shlibs:Depends} +
libxrt-npu2.
* HRX (portable + .deb) -> file(GET_RUNTIME_DEPENDENCIES) closure that
bundles libhrx and the engine .so private deps (libgomp/libmvec).
- Gate the engine .so RUNPATH patchelf under FLM_USE_HRX; XRT engine libs are
shipped unmodified (the portable wrapper already exposes ./lib via
LD_LIBRARY_PATH).
- CI is XRT-only: drop the unused HRX artifact fetch / CMAKE_PREFIX_PATH /
zstd from ubuntu-build, and the HRX fetch + hrx.dll bundling from
windows-build. ubuntu-build now matches upstream.
- Reword comments that compared against the "main" branch.
Verified on NPU (accel0): both backends configure, build, and run
qwen3:0.6b correctly (2 + 2 = 4).
Co-authored-by: Cursor <cursoragent@cursor.com>
Abhishek-Varma
marked this pull request as ready for review
August 6, 2026 13:12
Contributor
Author
|
Hi @zaneni6 - thanks for testing the patch. Can you try with the latest state of the PR? If everything looks okay, we can go ahead with merging it (CC: @tawei-amd ). |
Collaborator
Hi @Abhishek-Varma thanks for your update! Tried with the latest linux portable one, got a different lib error: |
…e builds
The rebase/squash dropped main's portable FFmpeg (and zlib) static-build
logic, so the portable XRT tarball dynamically linked the build host's
FFmpeg. On other machines flm-real then failed at startup with:
error while loading shared libraries: libavformat.so.61: cannot open
shared object file: No such file or directory
Restore main's behavior for the XRT path so the portable artifact is
self-contained again:
- Discovery: for FLM_PORTABLE_BUILD (non-HRX, Linux), fetch and build
FFmpeg (n7.1) and zlib (v1.3.1) as static archives; set
FFMPEG_BUILT_FROM_SOURCE / ZLIB_BUILT_FROM_SOURCE.
- Deps: only pkg-config the system FFmpeg when NOT building it statically;
skip the GPL readline/ncurses for portable builds (the CLI already has a
plain line-reader fallback under the FASTFLOWLM_USE_READLINE guard), so
the tarball carries no libavformat*/libreadline runtime dependency.
- Link: link the static libav*/libsw* archives + zlib for portable;
gate FASTFLOWLM_USE_READLINE + readline/ncurses on NOT FLM_PORTABLE_BUILD.
- Bundling: search the standard multiarch/system dirs for
libboost_program_options.so* independently of where XRT was found, so
Boost is bundled even when XRT lives under /opt/xilinx.
Verified on NPU (accel0): portable XRT build links no libav*/readline
(readelf NEEDED), the staged tree resolves all deps from bundled lib/,
and qwen3:0.6b answers "2 + 2 = 4".
Reported on PR ROCm#602 (portable Linux artifact).
Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
Author
|
Hi @zaneni6 - thank you for testing! Can you confirm if the issue stands resolved now? |
Collaborator
Sure! But it looks like the CI is currently down. I'll do it once the artifacts have been generated. |
Review follow-up: keep the XRT build byte-for-byte equivalent to main and confine every deviation to the HRX delta. - CMakeLists (non-portable/.deb): install engine .so directly into <prefix>/lib with flm RPATH $ORIGIN/../lib, exactly like main. The <prefix>/lib/flm split (engines separated from the bundled libhrx) is now gated behind FLM_USE_HRX, so only the HRX layout uses it. - debian/rules: drop the override_dh_shlibdeps block (now identical to main). It was only needed to locate the bundled libhrx during HRX .deb packaging; the XRT .deb resolves XRT from libxrt-npu2 and its engine libs from <prefix>/lib just as main does. - windows-build.yml: remove an unnecessary comment (now identical to main). - main.cpp: drop incidental blank-line changes; the only remaining delta vs main is the functional XRT/HRX preload gating. - test/common.mk: drop an incidental blank line. Verified: non-portable XRT install stages 23 engine libs into /opt/fastflowlm/lib with flm RPATH $ORIGIN/../lib (matches main); portable and non-portable both configure cleanly. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
Author
|
Hi @zaneni6 - the CI looks fine now, can you please try out the latest patch? I tried pulling in the Linux portable locally and the issue seems resolved. In case it persists, let me know and I can take a stab at it - else it'd be nice to get this in. :) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose: make the NPU dispatch backend selectable at build time via a single CMake flag (
FLM_USE_HRX: 0 = XRT, the default; 1 = HRXamdxdna), so the FastFlowLM application (theflmbinary and its runner/server/benchmark entry points) builds and runs against either the production XRT + aiebu runtime or the HRXamdxdnaruntime from one source tree. XRT is the default; HRX is opt-in. Prebuilt NPU engine libraries ship per backend for the full model matrix, and the HRX path is wired to a pinned, reproducible HRX release artifact.Motivation
FastFlowLM must keep building and running on the production XRT + aiebu runtime while also supporting the HRX (
amdxdna) runtime as a drop-in alternative. Rather than fork the tree per runtime, a build flag selects the backend behind a neutralflm_rtalias namespace, so device/buffer/kernel/run/dispatch code is written once and compiles against either runtime. This keeps XRT the default and unaffected (so the PR stays mergeable and easy to sync withmain) while HRX is available opt-in for validation and bring-up. HRX engines are compiled against a thinnamespace hrxlayer overlibhrx(hrx_cpp/hrx_cpp.hpp) that bypasses the aiebu assembler; XRT engines use the existing XRT + aiebu path.Technical Details
FLM_USE_HRXCMake option (defaultOFF= XRT) plus aFLM_USE_HRX=1compile definition on the HRX build. The flag drives runtime headers/namespace, discovery/link, the engine-lib directory, and portable bundling;FLM_RUNTIME_NAMEresolves toxrtorhrx.src/include/device_runtime.hpp: a neutralflm_rtalias that maps toxrtwhenFLM_USE_HRXis undefined and tohrxwhen defined. App-facing runtime references across models/runner/server/tests are written asflm_rt::(device, bo, ext::bo, kernel, hw_context, run, runlist, xclbin, info::device), so one source tree builds against either runtime.npu_utils.hppinto a thin dispatcher overnpu_utils_xrt.hpp(themainXRT implementation: ELF/module handling via aiebu) andnpu_utils_hrx.hpp(HRX: control code fromnpu_sequence::dump()into an amdxdna direct executable, with a host patch table replacing aiebu-emitted relocations).buffer.hppkeeps theFLM_DEVICE_BUFFERdevice-buffer path and gates buffer sync per backend — XRTbo::sync(XCL_BO_SYNC_BO_TO/FROM_DEVICE)vs HRXflush()/invalidate().main.cpppreloads the bundled XRT libraries (libxrt_core/libxrt_coreutil/libxrt_driver_xdna) only on the XRT + Linux build (#if !defined(FLM_USE_HRX) && !defined(_WIN32)); the HRX path has no equivalent preload requirement, and Windows retains its own DLL loading./opt/xilinx/xrtand an optional build-from-source, linkingxrt_coreutil(+aiebufor the ELF assembler used bynpu_utils_xrt); HRX is consumed from its public CMake package viafind_package(hrx CONFIG REQUIRED), linkinghrx::hrx.src/lib/xrt/(XRT engines + aiebu, XRT-only) andsrc/lib/hrx/(HRX engines) — for the full model matrix: Qwen3, Llama3, Qwen2, Phi4, Gemma3, Gemma3-Vision, Gemma4e, Gemma-Embedding, LFM2, Nanbeige, GPT-OSS, Qwen2VL, Qwen3VL, Qwen3.5VL, and Whisper. The flag selectslib/<backend>; shared third-party libs stay insrc/lib/. Windows.dll/.libfollow the samexrt/hrxlayout.FLM_PORTABLE_BUILD) is backend-aware throughflm-wrapper.sh.in(configured per backend): XRT setsXILINX_XRT, creates thelib/x86_64-linux-gnumultiarch symlinks the loader expects, and rewrites engine-lib RUNPATH to$ORIGINviapatchelf; HRX simply exportsLD_LIBRARY_PATH.Test Plan
-DFLM_USE_HRX=ON) from the same tree; confirm each links the expected runtime (libxrt_coreutil+lib/xrtvslibhrx+lib/hrx) and runs a smoke inference on the NPU.flmbenchmark path.Test Result
accel0: the default XRT build linkslibxrt_coreutil+src/lib/xrt, and the HRX build (-DFLM_USE_HRX=ON) linkslibhrx+src/lib/hrx; both answer "2 + 2 = 4".jtuyls/hrx@flm-hrx-amdxdna-v2026.07.20, assethrx-amdxdna-2026.07.20-amdxdna-hal-native-rel-fcaecde-linux-x86_64.tar.zst.hrx-amdxdna-2026.07.07-...asset. Benchmark input: 1k JSON withmax_length=1024anditerations=5(values are means). XRT baseline:FLM_xrt_main @ mainwith the RMSNorm pointer-hoisting fixes cherry-picked (gemma + gpt_oss); the XRT harness runs the same application entry points. No xclbin rebuilds on either side.gemma4-it:e4bverified at the engine level via e2b).embed-gemma:300membeddings are bit-identical across backends;whisper-v3:turbomatches within Whisper's own non-deterministic decode variance.1k Benchmark Results - XRT vs HRX - LINUX (Driver
2.23.0)1k Benchmark Results - XRT vs HRX - WINDOWS (Driver
32.0.203.314)nanbeige4.1:3bqwen3vl-it:4bgemma4-it:e2bgemma4-it:e4bqwen3.5:0.8bqwen3.5:2bqwen3.5:4bqwen3.5:9bqwen3.6-moe:35b-a3blfm2:1.2blfm2:2.6blfm2-trans:2.6blfm2.5-it:1.2blfm2.5-tk:1.2bphi4-mini-it:4bgemma3:1bgemma3:4btranslategemma:4bmedgemma:4bmedgemma1.5:4bllama3.2:1bllama3.2:3bllama3.1:8bdeepseek-r1:8bdeepseek-r1-0528:8bqwen3:0.6bqwen3:1.7bqwen3:4bqwen3:8bqwen3-tk:4bqwen3-it:4bgpt-oss:20bgpt-oss-sg:20bqwen2.5-it:3bqwen2.5vl-it:3bSubmission Checklist
flm(runner/server/benchmark + binary) builds and runs on both backends from one source tree viaFLM_USE_HRX(XRT default; HRX opt-in), with the runtime selected behind theflm_rtalias.src/lib/xrtandsrc/lib/hrxfor the full model matrix (Windows.dll/.libfollow the same layout); portable$ORIGINRUNPATH, no developer-absolute paths in committed binaries.hrx-integration/; missing-artifact failures are clear.gemma4-it:e4bengine-level via e2b).