[BUG] Failing tests on GB300

[KyleFromNVIDIA]

Describe the bug
While testing cuml on GB300, I observed the following test failure:

=================================== FAILURES ===================================
_ test_sklearn_compatible_estimator[LinearSVR()-check_regressor_data_not_an_array] _
[gw1] linux -- Python 3.14.6 /opt/conda/envs/test/bin/python
Traceback (most recent call last):
  File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/runner.py", line 361, in from_call
    result: TResult | None = func()
                             ~~~~^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/runner.py", line 250, in <lambda>
    lambda: runtest_hook(item=item, **kwds),
            ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall
    raise exception
  File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/logging.py", line 865, in pytest_runtest_call
    yield
  File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call
    return (yield)
            ^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper
    return result.get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_result.py", line 103, in get_result
    raise exc.with_traceback(tb)
  File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper
    res = yield
          ^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call
    return (yield)
            ^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall
    res = hook_impl.function(*args)
  File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/runner.py", line 184, in pytest_runtest_call
    item.runtest()
    ~~~~~~~~~~~~^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/python.py", line 1707, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall
    raise exception
  File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall
    res = hook_impl.function(*args)
  File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/python.py", line 167, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/__w/cuml/cuml/python/cuml/tests/test_sklearn_compatibility.py", line 311, in test_sklearn_compatible_estimator
    check(estimator)
    ~~~~~^^^^^^^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/sklearn/utils/_testing.py", line 144, in wrapper
    return fn(*args, **kwargs)
  File "/opt/conda/envs/test/lib/python3.14/site-packages/sklearn/utils/estimator_checks.py", line 4010, in check_regressor_data_not_an_array
    check_estimators_data_not_an_array(name, estimator_orig, X, y, obj_type)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/sklearn/utils/_testing.py", line 144, in wrapper
    return fn(*args, **kwargs)
  File "/opt/conda/envs/test/lib/python3.14/site-packages/sklearn/utils/estimator_checks.py", line 4057, in check_estimators_data_not_an_array
    assert_allclose(pred1, pred2, atol=1e-2, err_msg=name)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/sklearn/utils/_testing.py", line 236, in assert_allclose
    np_assert_allclose(
    ~~~~~~~~~~~~~~~~~~^
        actual,
        ^^^^^^^
    ...<5 lines>...
        verbose=verbose,
        ^^^^^^^^^^^^^^^^
    )
    ^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/numpy/testing/_private/utils.py", line 1768, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                         verbose=verbose, header=header, equal_nan=equal_nan,
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                         strict=strict)
                         ^^^^^^^^^^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/numpy/testing/_private/utils.py", line 983, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=0.0001, atol=0.01
LinearSVR
Mismatched elements: 74 / 200 (37%)
First 5 mismatches are at indices:
 [1]: 21.620885848999023 (ACTUAL), 21.6062759915089 (DESIRED)
 [2]: 22.719371795654297 (ACTUAL), 22.682475870939612 (DESIRED)
 [6]: 13.214752197265625 (ACTUAL), 13.20281020279908 (DESIRED)
 [10]: -0.5637669563293457 (ACTUAL), -0.5808369146929113 (DESIRED)
 [11]: 79.20233917236328 (ACTUAL), 79.1652457127903 (DESIRED)
Max absolute difference among violations: 0.04352974
Max relative difference among violations: 0.11924252
 ACTUAL: array([ 2.811466e+01,  2.162089e+01,  2.271937e+01,  5.861282e-03,
       -2.898407e+01,  2.818293e+01,  1.321475e+01,  8.779308e+01,
        2.023422e+01,  3.124043e+01, -5.637670e-01,  7.920234e+01,...
 DESIRED: array([ 2.811139e+01,  2.160628e+01,  2.268248e+01,  4.357709e-03,
       -2.898866e+01,  2.818264e+01,  1.320281e+01,  8.780300e+01,
        2.024469e+01,  3.122895e+01, -5.808369e-01,  7.916525e+01,...

Steps/Code to reproduce bug
Run the tests on GB300

Expected behavior
All tests should pass

Environment details (please complete the following information):
GB300 CI runner

Additional context
https://github.com/rapidsai/cuml/actions/runs/30637312436/job/91186078101?pr=8375
https://github.com/rapidsai/cuml/actions/runs/30637312436/job/91185861124?pr=8375

[viclafargue]

Re-opening the issue as I do not have access to a machine with a GB300 to confirm that the issue is actually solved. Also if #8432 did solve the issue we should probably revert #8431.

@KyleFromNVIDIA Could you check if the issue has been solved on the GB300 machine?

[KyleFromNVIDIA]

Testing on #8375

[KyleFromNVIDIA]

The wheel tests are passing, and the single-GPU tests are failing for packaging reasons that are unrelated to this bug. Closing as complete.