gpl: opt-in GPU acceleration via Kokkos

[ApeachM]

Continuation of #10370, which was originally scoped to HPWL only. After ~3 weeks without review I completed the rest of the device-resident port so the win is actually visible; the original HPWL-only diff was net-neutral on wall time. If a smaller scope is preferable I'm happy to split this back up — see Splitting this PR below.

Summary

Opt-in GPU acceleration for the gpl Nesterov loop via Kokkos.

• Default (ENABLE_GPU=OFF) — byte-identical to master. No Kokkos / CUDA / HIP dependency, no CI impact, no Bazel impact.
• ENABLE_GPU=ON — compiles GPU kernels in. Backend is chosen per process at runtime via the ENABLE_GPU environment variable (0 forces CPU). Golden integration tests pin ENABLE_GPU=0 and stay green.
• Result — on a 274k-cell Nangate45 design at CPU 8T baseline, wall time drops 1:15 → 0:56 (−25%), RTX 5090. Convergence is preserved (±1 iter, HPWL within 1e-3 relative).

Per-iteration host↔device traffic shrinks to one reverse sync (~1.2 MB) plus two scalar reductions; everything else stays device-resident.

This is the path agreed in the #5352 discussion — integrate into src/gpl/ via Strategy interfaces, rather than maintain a parallel gpl2/ module.

Build matrix

	ENABLE_GPU=OFF (default)	ENABLE_GPU=ON
Extra deps	None	Kokkos + KokkosFFT + CUDA or HIP
GPU code compiled	No (src/gpl/src/gpu/ skipped)	Yes (CUDA/HIP TUs)
Runtime backend	CPU always	ENABLE_GPU=0 env → CPU, else GPU
CI / Bazel	Unaffected	Tests pin ENABLE_GPU=0 → all green
gpl test suite	All pass	All pass + fft_gpu_test

# Default build
cmake -S . -B build && ninja -C build
ctest --test-dir build -R gpl

# GPU build
cmake -S . -B build_gpu -DENABLE_GPU=ON \
  -DKokkos_ROOT=<path> -DKokkosFFT_DIR=<path>
ninja -C build_gpu
ctest --test-dir build_gpu -R gpl

Architecture

Each accelerated operation is a Strategy interface with CPU and GPU implementations. A factory function is the only #ifdef ENABLE_GPU in C++; consumer headers stay preprocessor-free.

Operation	Strategy	CPU	GPU
HPWL	HpwlBackend	OpenMP loop	Kokkos parallel_reduce
WL gradient	WirelengthGradientBackend	per-cell loop	5-kernel pipeline
Density gradient	DensityGradientBackend	per-cell FFT lookup	parallel_for gather
FFT / Poisson	FftBackend	Ooura DCT	KokkosFFT DCT

Device-resident state is split into two PIMPL objects:

• DeviceState (owned by NesterovBaseCommon) — inst/pin coords, pin-to-net CSR, bin grid Views.
• NesterovDeviceContext (owned by NesterovBase) — Nesterov vectors (coords, grads, sumGrads, clamp bounds) + 6 Kokkos kernels (coord update, gradient combine, step length, scatters).

Backend factories share a BackendContext POD. FftBackend::solve takes a BinGridSpan (data pointer + bin counts). Host→device coord sync goes through DeviceState::ensureCoordsFresh (master-thread only, no atomic). Smaller shared helpers: solverToGplField (axis swap + scale), mapNbcGrads (gradient gather for fillers vs inst cells), SlpSlot / SumGradSlot (typed slot kinds for the per-cell vector launchers).

All Kokkos types stay behind PIMPL. Only gpu/*.cpp files are CUDA TUs.

Numbers

12-core Intel host + RTX 5090, same binary, ENABLE_GPU env toggled, set_thread_count explicit. Nangate45.

CPU thread scaling (host OMP only):

Design	Cells	Iters	1T	2T	4T	8T
medium03	98k	486	2:02	1:40	1:31	1:28
large01	274k	554	1:56	1:27	1:16	1:15

GPU vs best CPU (both at 8 host threads; GPU runs use 8 host threads for the CPU-side density scatter + Nesterov loop overhead):

Design	Cells	CPU 8T	GPU	Δ
medium03	98k	1:28	1:29	parity
large01	274k	1:15	0:56	−25%

• medium03 (98k) — parity. The per-iter GPU overhead (~1.2 MB reverse sync for the host density scatter + two scalar reductions) cancels the compute saving at this size.
• large01 (274k) — GPU −25%. Overhead is constant, compute saving scales with cells, so GPU pulls ahead on larger designs.

Crossover sits around ~100k cells on this host. The largest single contributor on the GPU side is the WL gradient force kernel: 22 ms/call → 34 µs/call.

Determinism

Same convergence trajectory (±1 iter, HPWL within 1e-3 relative). Implemented via the sgizler pattern from #5352: integer arithmetic for HPWL bbox, serial per-net inner reduction, int64 cross-net sum.

Commits

#	Commit	Scope
1	9738393	Foundation + HPWL + FFT + WL gradient — build infra, ENABLE_GPU option, Strategy/Factory, DeviceState, three GPU backends
2	fc6059f	Density gradient — gather kernel, bin grid Views on DeviceState, device-resident FFT solve
3	49cd6d8	Nesterov loop body on GPU — NesterovDeviceContext + 6 kernels; eliminates the host↔device forward sync each iter
4	7ba5ded	Remove bench instrumentation; pin ENABLE_GPU=0 for all gpl tests on GPU builds
5	ff6fce9	Fix Poisson OOB + HPWL int overflow (Gemini bot review)
6	dcadb7f8	Address internal review pass — correctness fixes + helper-function refactor
7	885cbaa	Refactor GPU port surface for review — split VecSlot, unify factories via BackendContext, BinGridSpan POD, ensureCoordsFresh, shared solverToGplField + mapNbcGrads helpers

Each commit builds, tests, and is independently reviewable.

Splitting this PR

The diff is large because the wall-time win shows up only once the Nesterov loop body runs on the device — the foundation and per-operation backends are net-neutral by themselves. The commits above are already organized so any prefix can ship:

1. Commits 1+2 (build infra, HPWL, FFT, WL gradient, density gradient) — landable as a first PR, modest perf change but establishes the dispatch shape.
2. Commit 3 — adds the device-resident Nesterov loop; this is where the −25% appears.
3. Commits 4–7 — cleanup, review fixes, and design-pattern refactor.

Happy to break this into 2 or 3 sequential PRs if that's easier to review. Each prefix is independently buildable and CPU-parity-verified.

Follow-ups (out of scope for this PR)

• Cache the KokkosFFT::Plan (currently rebuilt on every call).
• Move the filler density gradient onto the device. Saves the ~1.2 MB upload per iteration.
• Move the density bin scatter (updateBinsGCellDensityArea) onto the device. The last per-iteration host round-trip.
• CPU↔GPU equivalence tests for HPWL / WL gradient / density gradient, matching fft_gpu_test.
• Split KokkosDeviceState by purpose (pin coords, WL gradient, density, Nesterov, bin grid) and drop the public kokkos() accessor.
• Pull the GPU branches out of updateGradients / getStepLength / nesterovUpdateCoordinates into their own functions. Today the slot kind is inferred from a pointer-identity check, which works but is fragile.

Related

• #5352 — Antmicro's gpl2/ DG-RePlAce Kokkos port. Same kernels; this PR takes the in-place integration path the discussion converged on.
• #10336 — Bazel vs CMake QoR divergence. The FP-contraction discipline in cmake/KokkosBackend.cmake aligns with that effort.
• #10370 — original HPWL-only PR.

[ApeachM]

Supersedes #10370 (HPWL-only, closed). Full device-resident port of the Nesterov inner loop.

[gemini-code-assist]

Code Review

This pull request introduces GPU acceleration via Kokkos for the OpenROAD global placer (gpl), implementing GPU backends for HPWL calculation, WA wirelength gradient computation, and density gradient gather/FFT Poisson solving. The review feedback identifies three valid issues: a critical out-of-bounds memory write in the Poisson solver's array initialization, performance inefficiencies from recreating KokkosFFT plans on every call, and a potential integer overflow in the HPWL backend's coordinate subtraction.

[gemini-code-assist]

The views d_expkMN1_ and d_expkMN2_ are allocated with a size of binCntX_ + binCntY_ (which is N + M). However, during initialization, the code writes to indices up to hid + M, which can be as large as 2 * M - 1 (where M = binCntY_). If binCntY_ > binCntX_, this results in a critical out-of-bounds memory write. Additionally, during the IDCT phase, the code accesses indices up to 1.5 * N (where N = binCntX_), which can also exceed N + M if N > 2 * M. Allocating 2 * (binCntX_ + binCntY_) ensures that both initialization and IDCT accesses are always safe.

  d_expkMN1_
      = Kokkos::View<Kokkos::complex<float>*>("d_expkMN1", 2 * (binCntX_ + binCntY_));
  d_expkMN2_
      = Kokkos::View<Kokkos::complex<float>*>("d_expkMN2", 2 * (binCntX_ + binCntY_));

[ApeachM]

Fixed in 29f029d (the OOB allocation sizing) and 7ff9ab5 (added a runtime kMaxBinAspectRatio=2 assertion in the PoissonSolver constructor that fails fast if a future caller pushes the aspect past where the expkMN1/2 index math is valid).

d_expkMN1_ / d_expkMN2_ are now allocated as 2 * std::max(binCntX_, binCntY_), which covers the worst-case hid + M (up to 2M-1) index and its symmetric counterpart. The aspect-ratio assertion protects the lower-bound case (negative indices in the IDCT post kernel) that emerges when M ≫ N or N ≫ M.

[gemini-code-assist]

Creating a new KokkosFFT::Plan on every call to dct_2d_fft and idct_2d_fft is highly inefficient because plan creation can involve expensive heuristics and memory allocations. Since the grid dimensions are constant for a given PoissonSolver instance, the forward and backward plans should be created once during initialization (e.g., in PoissonSolver::initBackend()) and stored as members of PoissonSolver to be reused across iterations.

[ApeachM]

Attempted plan caching with persistent host buffers + cached PlanT bound to those buffers; the change regressed large01 wall time from 1:34 to 1:52 (~12%, reproducible across 3 runs). Likely a host-buffer type-mismatch causing a slow deep_copy path between the unmanaged device View and the cached host View, but the root cause was not isolated.

Deferring to a follow-up — listed under "Follow-ups" in the PR description. The current per-call Plan construction is the known-working baseline; the optimization needs persistent buffers tied to the PoissonSolver lifetime, not file-scope thread_local caching.

[gemini-code-assist]

The subtraction ux - lx and uy - ly is performed using int arithmetic before being cast to int64_t. If the coordinates are extremely far apart, this subtraction can overflow the int range, leading to undefined behavior. Casting the operands to int64_t before subtraction prevents any potential signed integer overflow.

        acc += (static_cast<int64_t>(ux) - lx) + (static_cast<int64_t>(uy) - ly);

[ApeachM]

Fixed in 29f029d. The subtraction now casts the operands to int64_t before subtracting:

acc += (static_cast<int64_t>(ux) - lx) + (static_cast<int64_t>(uy) - ly);

No theoretical signed-int overflow path remains.

[maliberty]

How many threads are used for the CPU results?

[ApeachM]

Both numbers were single-threaded — OpenROAD defaults to 1 thread and the test TCLs don't call set_thread_count. Not a fair baseline for gpl, which has parallelizable OMP loops.

Re-ran the sweep on the same 12-core Intel host (RTX 5090, same binary, set_thread_count explicit).

CPU thread scaling (host OMP only):

Design	Cells	1T	2T	4T	8T
medium03	98k	2:02	1:40	1:31	1:28
large01	274k	1:56	1:27	1:16	1:15

GPU vs best CPU (both at 8 host threads; GPU runs use 8 host threads for the CPU-side density scatter + Nesterov loop overhead):

Design	Cells	CPU 8T	GPU	Δ
medium03	98k	1:28	1:29	parity
large01	274k	1:15	0:56	−25%

• medium03 (98k) — parity. The GPU per-iter overhead (~1.2 MB reverse sync for the host density scatter + two scalar reductions) cancels the compute saving at this size.
• large01 (274k) — GPU −25%. Overhead is constant, compute saving scales with cells, so GPU pulls ahead on larger designs.

Crossover sits around ~100k cells on this host. Updating the Numbers table in the PR body to reflect this.

---

Separately on the Gemini bot review: Critical (Poisson expkMN1/2 OOB) and Medium (HPWL ux-lx int overflow) are fixed. High (KokkosFFT Plan recreation) is deferred — a caching attempt regressed wall time, so the current per-call construction stays as the known-working baseline. See that thread and the Follow-ups section.