[umbrella] velox_cudf_test: 12 failed CTest suites

[sperlingxx]

68% tests passed, 12 tests failed out of 37

Umbrella for GPU CTest failures recorded below. Parent harness: #66.

Tracking

One child issue / PR per failed CTest suite (or split further when a suite has unrelated inner failures). Check off when the suite's failed cases are fixed on dev.

• velox_cudf_adapter_operator_test (2 cases) — [bug] CudfWindow streaming disk-spill tests see 1 write instead of 3 #105, fix fix(cudf): Preserve GPU batch boundaries in window spill tests #106
  • AdapterOperatorTest.streamingFullPartitionCountSpillsActivePartition
  • AdapterOperatorTest.streamingRangeSumSpillsActivePeer
• velox_cudf_aggregation_test (6 cases)
  • AggregationTest.partialAggregationUsesBalancedRunMerges
  • FinalAggregationStreamingTest.finalAggregationStreamsOnAddInput
  • FinalAggregationStreamingTest.finalAggregationStreamingMixedAggs
  • FinalAggregationStreamingTest.finalAggregationStreamingMultiKey
  • FinalAggregationStreamingTest.finalAggregationLevelledRuns
  • FinalAggregationStreamingTest.companionFinalAggregationUsesLevelledRuns
• velox_cudf_aggregation_selection_test (2 cases) — [bug] cuDF aggregation selection accepts unsupported grouping keys and stale max(boolean) #107, fix fix(cudf): Restore Project expansion for agg eligibility #108
  • CudfAggregationSelectionTest.complexGroupbyClauseExpressions
  • CudfAggregationSelectionTest.invalidTypeCombinationsRejected
• velox_cudf_batch_concat_test (1 case)
  • CudfBatchConcatTest.concatFlushesAtByteThreshold
• velox_cudf_decimal_aggregation_test (1 case)
  • CudfDecimalTest.decimalFinalStreamingProbeReleasesMaterializedColumns
• velox_cudf_filter_project_test (5 cases)
  • CudfFilterProjectTest.zeroColumnValuesConstantProjection
  • CudfFilterProjectTest.rowConstructorWithUntypedNullField
  • CudfFilterProjectTest.substringWithIntegerBounds
  • CudfSimpleFilterProjectTest.castNumericToTimestamp
  • CudfSimpleFilterProjectTest.castDateToVarchar
• velox_cudf_nested_loop_join_test (2 cases) — [bug] CudfNestedLoopJoin drops build byte limit and rejects two-sided timestamp filters #111, fix draft: fix(cudf): Restore NLJ build byte limit and timestamp AST compares #112
  • CudfNestedLoopJoinTest.buildByteLimitAndCleanup
  • CudfNestedLoopJoinTest.innerJoinWithTimestampFilter
• velox_cudf_order_by_test (2 cases)
  • OrderByTest.boundedExternalSortManyRuns
  • OrderByTest.boundedExternalSortMapPayload
• velox_cudf_tocudf_selection_test (1 case) — same helper as [bug] cuDF aggregation selection accepts unsupported grouping keys and stale max(boolean) #107, fix fix(cudf): Restore Project expansion for agg eligibility #108
  • ToCudfSelectionTest.complexGroupingKeyExpressionsFallsBack
• velox_cudf_iceberg_read_test (process terminated)
  • CudfIcebergReadTest.combinedPositionalAndEqualityDeletes (SIGTERM / hang)
• velox_cudf_spark_aggregates_test (1 case)
  • AggregationTest.groupedCollectListOfRow
• velox_cudf_spark_filter_project_test (1 case) — [bug] Spark multi-branch SWITCH test uses Presto eq instead of equalto #109, fix fix(cudf): Use Spark equalto in multi-branch SWITCH test #110
  • CudfFilterProjectTest.multiBranchSwitchWithRegexpExtract

Run context

This issue records the GPU CTest result for the native harness tracked by #66.

• Date: 2026-08-21
• Velox commit: 46040b5a08f76f9d26b2ec867b0881d9cf2cda24
• CUDA architecture: 86-real
• CTest selection: every registered test carrying the exact cuda_driver label

ctest \
  --test-dir _build/release/velox/experimental/cudf/tests \
  --output-on-failure \
  --no-tests=error \
  --label-regex '^cuda_driver$' \
  --output-junit junit.xml

Complete CTest suite inventory

#	Registered CTest suite	Status
1	velox_cudf_adapter_operator_test	Failed
2	velox_cudf_aggregation_test	Failed
3	velox_cudf_aggregation_selection_test	Failed
4	velox_cudf_assign_unique_id_test	Succeed
5	velox_cudf_batch_concat_test	Failed
6	velox_cudf_config_test	Succeed
7	velox_cudf_decimal_aggregation_test	Failed
8	velox_cudf_vector_test	Succeed
9	velox_cudf_decimal_expression_test	Succeed
10	velox_cudf_enforce_single_row_test	Succeed
11	velox_cudf_expression_selection_test	Succeed
12	velox_cudf_filter_project_test	Failed
13	velox_cudf_function_registry_test	Succeed
14	velox_cudf_group_id_test	Succeed
15	velox_cudf_hash_join_test	Succeed
16	velox_cudf_interop_test	Succeed
17	velox_cudf_limit_test	Succeed
18	velox_cudf_logical_functions_test	Succeed
19	velox_cudf_local_partition_test	Succeed
20	velox_cudf_mark_distinct_test	Succeed
21	velox_cudf_nested_loop_join_test	Failed
22	velox_cudf_null_mask_test	Succeed
23	velox_cudf_order_by_test	Failed
24	velox_cudf_s3_read_test	Succeed
25	velox_cudf_step_aware_aggregation_registry_test	Succeed
26	velox_cudf_string_function_test	Succeed
27	velox_cudf_subfield_filter_ast_test	Succeed
28	velox_cudf_table_scan_test	Succeed
29	velox_cudf_executor_prefetch_test	Succeed
30	velox_cudf_tocudf_selection_test	Failed
31	velox_cudf_topn_test	Succeed
32	velox_cudf_topnrownumber_test	Succeed
33	velox_cudf_window_test	Succeed
34	velox_cudf_iceberg_read_test	Failed — subprocess terminated
35	velox_cudf_iceberg_deletion_vector_test	Succeed
36	velox_cudf_spark_aggregates_test	Failed
37	velox_cudf_spark_filter_project_test	Failed

Critical errors by failed CTest suite

1. velox_cudf_adapter_operator_test

Inner result: 16 passed, 2 failed.

• AdapterOperatorTest.streamingFullPartitionCountSpillsActivePartition: expected CudfWindow::testingStreamingSpillWrites() >= 3; actual value was 1 (AdapterOperatorTest.cpp:316).
• AdapterOperatorTest.streamingRangeSumSpillsActivePeer: expected CudfWindow::testingStreamingSpillWrites() >= 3; actual value was 1 (AdapterOperatorTest.cpp:765).

2. velox_cudf_aggregation_test

Inner result: 110 passed, 6 failed.

• AggregationTest.partialAggregationUsesBalancedRunMerges: runtime stat cudfIntermediateAggregationRunMerges was absent; stats.count(statName) was 0, expected 1 (AggregationTest.cpp:1066).
• FinalAggregationStreamingTest.finalAggregationStreamsOnAddInput, finalAggregationStreamingMixedAggs, and finalAggregationStreamingMultiKey: expected cudfFinalStreamingBatches > 1; actual value was 1 (AggregationTest.cpp:1204).
• FinalAggregationStreamingTest.finalAggregationLevelledRuns: finalStats.contains("cudfFinalAggregationRunMerges") was false, expected true (AggregationTest.cpp:1348).
• FinalAggregationStreamingTest.companionFinalAggregationUsesLevelledRuns: uncaught C++ exception: unordered_map::at.

3. velox_cudf_aggregation_selection_test

Inner result: 18 passed, 2 failed.

• CudfAggregationSelectionTest.complexGroupbyClauseExpressions: canBeEvaluatedByCudf(...) returned true, expected false (AggregationSelectionTest.cpp:246).
• CudfAggregationSelectionTest.invalidTypeCombinationsRejected: canAggregationBeEvaluatedByCudf(...) returned true, expected false (AggregationSelectionTest.cpp:662).

5. velox_cudf_batch_concat_test

Inner result: 9 passed, 1 failed.

• CudfBatchConcatTest.concatFlushesAtByteThreshold: expected outputVectors < inputVectors; both were 6 (BatchConcatTest.cpp:155).

7. velox_cudf_decimal_aggregation_test

Inner result: 48 passed, 1 failed.

• CudfDecimalTest.decimalFinalStreamingProbeReleasesMaterializedColumns: expected cudfFinalAggregationInputRuns > 1; actual value was 1 (DecimalAggregationTest.cpp:792).

12. velox_cudf_filter_project_test

Inner result: 120 passed, 5 failed.

• CudfFilterProjectTest.zeroColumnValuesConstantProjection: VeloxRuntimeError INVALID_STATE at ToCudf.cpp:393: Replacement with cuDF operator failed for operator: FilterProject; failed expression !isPureCpuOperator (ToCudf.cpp:393).
• CudfFilterProjectTest.rowConstructorWithUntypedNullField: VeloxRuntimeError INVALID_STATE: not a scalar type! kind: UNKNOWN at AstUtils.h:257 (AstUtils.h:257).
• CudfFilterProjectTest.substringWithIntegerBounds: VeloxRuntimeError INVALID_STATE at ToCudf.cpp:393: Replacement with cuDF operator failed for operator: FilterProject; failed expression !isPureCpuOperator (ToCudf.cpp:393).
• CudfSimpleFilterProjectTest.castNumericToTimestamp: VeloxUserError INVALID_ARGUMENT: Cannot cast DOUBLE '1767225600' to TIMESTAMP. Conversion to Timestamp is not supported (EvalCtx.cpp:208).
• CudfSimpleFilterProjectTest.castDateToVarchar: VeloxRuntimeError INVALID_STATE: No cuDF expression evaluator can handle: cast(ROW["c0"] as VARCHAR) at ExpressionEvaluator.cpp:3460 (ExpressionEvaluator.cpp:3460).

21. velox_cudf_nested_loop_join_test

Inner result: 56 passed, 2 failed.

• CudfNestedLoopJoinTest.buildByteLimitAndCleanup: expected an exception, but no exception was thrown (NestedLoopJoinTest.cpp:100).
• CudfNestedLoopJoinTest.innerJoinWithTimestampFilter: VeloxRuntimeError INVALID_STATE: Expression spans both join sides and cannot be precomputed: gt(ROW["p_ts"],ROW["b_ts"]); sideIdx was -2 (AstExpressionUtils.h:523).

23. velox_cudf_order_by_test

Inner result: 9 passed, 2 failed.

• OrderByTest.boundedExternalSortManyRuns: expected maxActiveRuns == 2, actual 1; expected sourceChunks >= 23, actual 3 (OrderByTest.cpp:523).
• OrderByTest.boundedExternalSortMapPayload: expected sourceChunks >= 7, actual 1 (OrderByTest.cpp:565).

30. velox_cudf_tocudf_selection_test

Inner result: 24 passed, 1 failed.

• ToCudfSelectionTest.complexGroupingKeyExpressionsFallsBack: wasCudfAggregationUsed(task) returned true, expected false (ToCudfSelectionTest.cpp:635).

34. velox_cudf_iceberg_read_test

The executable registered 89 inner tests and stalled while running CudfIcebergReadTest.combinedPositionalAndEqualityDeletes. It was externally terminated after 316.21 seconds. The captured terminal error was:

Signal 15 (SIGTERM)
(error retrieving stack trace)
Fatal signal handler. ThreadDebugInfo object not found.

No test assertion was captured for this suite because the process did not finish.

36. velox_cudf_spark_aggregates_test

Inner result: 1 passed, 1 failed.

• AggregationTest.groupedCollectListOfRow failed in SetUp() with VeloxRuntimeError INVALID_STATE: The periodic stats reporter has already started; failed expression instanceRef == nullptr at PeriodicStatsReporter.cpp:47 (PeriodicStatsReporter.cpp:47).

37. velox_cudf_spark_filter_project_test

Inner result: 56 passed, 1 failed.

• CudfFilterProjectTest.multiBranchSwitchWithRegexpExtract: VeloxUserError INVALID_ARGUMENT: Scalar function doesn't exist: eq in resolveScalarFunctionType at TypeResolver.cpp:106 (TypeResolver.cpp:106).

Aggregate result

• CTest suites: 25 succeeded, 12 failed, 37 total
• CTest pass rate: 68%
• Completed failed binaries: 491 inner GTests, 467 passed, 24 failed
• velox_cudf_iceberg_read_test is excluded from the completed-inner-test subtotal because its process was terminated before completion.

[sperlingxx]

Treating this as the umbrella for the 12 failed cuda_driver CTest suites.

First suite is in progress:

• Child issue: [bug] CudfWindow streaming disk-spill tests see 1 write instead of 3 #105
• Fix PR: fix(cudf): Preserve GPU batch boundaries in window spill tests #106 (velox_cudf_adapter_operator_test, 2 streaming CudfWindow disk-spill cases)

The tracking checklist is in the issue body.

[sperlingxx]

Selection suites in progress:

• Child issue: [bug] cuDF aggregation selection accepts unsupported grouping keys and stale max(boolean) #107
• Fix PR: fix(cudf): Restore Project expansion for agg eligibility #108 (velox_cudf_aggregation_selection_test, and ToCudfSelectionTest.complexGroupingKeyExpressionsFallsBack via the same grouping-key helper)

13b1de5e0 runtime projected-input remapping is kept. Eligibility no longer shares that remapper: grouping by a column produced by to_big_endian_64 stays on CPU HashAggregation.

[sperlingxx]

Spark filter-project suite in progress:

• Child issue: [bug] Spark multi-branch SWITCH test uses Presto eq instead of equalto #109
• Fix PR: fix(cudf): Use Spark equalto in multi-branch SWITCH test #110 (velox_cudf_spark_filter_project_test, CudfFilterProjectTest.multiBranchSwitchWithRegexpExtract)

The CASE WHEN predicates were introduced in #54 with SQL =. DuckParser emits Presto eq; the Spark fixture registers equalto. Test-only rewrite; SWITCH/regexp_extract production path unchanged.

[sperlingxx]

Nested-loop-join suite in progress:

• Child issue: [bug] CudfNestedLoopJoin drops build byte limit and rejects two-sided timestamp filters #111
• Fix PR: draft: fix(cudf): Restore NLJ build byte limit and timestamp AST compares #112 (velox_cudf_nested_loop_join_test, buildByteLimitAndCleanup + innerJoinWithTimestampFilter)

Build byte accounting was dropped from the cpp in #38 while the header/test remained. Two-sided timestamp compares are compiled as AST ops instead of a cross-side precompute. buildByteLimitAndCleanup also pins gpu_batch_size_rows=1 so Values pages stay distinct GPU batches.