Summary
On dev at de935de174, cuDF operator selection can insert CudfFromVelox before a GPU source plan node that has no input edge. Classifying that nonexistent boundary aborts native execution with:
[RUNTIME/INVALID_STATE] (0 vs. 0) GPU input boundary requires a source
This was observed in three FluxStrategyPlanSuite cases and is reproduced deterministically by writing a Spark DataFrame backed by ExistingRDD to Parquet with the Velox cuDF path enabled.
Reproduction
- Enable the Gluten/Velox cuDF execution path.
- Create a DataFrame from driver-local rows, so its input is
ExistingRDD.
- Write the DataFrame to Parquet through the native writer path.
- Observe the exception above while compiling the native operator pipeline, before the Parquet write completes.
A minimal Spark-side shape is:
rows = [(1, "a"), (2, "b")]
df = spark.createDataFrame(rows, ["id", "value"])
df.write.mode("overwrite").parquet(output_path)
The native plan exposes an already-GPU source leaf with sources().empty(). The failure depends on that boundary shape, not on input scale.
Root cause
Commit 07e146e88b deliberately treats operator index zero as having a non-GPU predecessor. That preserves conversion when an MPP/UCX fragment's first GPU operator consumes an external Velox RowVector:
previousOperatorIsNotGpu =
operatorIndex == 0 || !operatorProperties[operatorIndex - 1].producesGpuOutput;
The same condition also matches a genuine GPU source leaf. Because the source operator accepts GPU input, CompileState::compile() asks gpuInputBoundaryType() to classify an input that does not exist, producing the (0 vs. 0) invalid-state error.
The two cases must be distinguished by the plan edge:
- a first GPU operator with a real input source may require
CudfFromVelox;
- a GPU source plan node with zero sources has no upstream
RowVector to convert.
Expected behavior
- Do not insert
CudfFromVelox before an already-GPU source leaf with no input plan edge.
- Continue inserting the conversion for a CPU source feeding a GPU consumer.
- Preserve the external-fragment-input behavior introduced by
07e146e88b.
- ExistingRDD-to-Parquet execution completes without the GPU input-boundary exception.
Fix and validation
Fix available in #58. It requires a real input plan edge before inserting the boundary and adds focused tests for both sides of the distinction.
Validation reported by #58:
- focused boundary tests: 2/2 passed;
- all
FluxStrategyPlanSuite cases: 22/22 passed;
- ExistingRDD -> Parquet -> readback reproducer: passed;
- complete L3 TPC-H SF1 Q1-Q22 row and plan golden comparison: passed;
- native UCX suite: 78 run, 126 expected skips, 0 failures;
- forced multi-fragment MPP/UCX TPC-H Q3: passed with remote UCX exercised;
- full
velox_cudf_tocudf_selection_test: 24/25, with the remaining complexGroupingKeyExpressionsFallsBack failure independently reproduced on current dev and outside this boundary path.
Acceptance criteria
Summary
On
devatde935de174, cuDF operator selection can insertCudfFromVeloxbefore a GPU source plan node that has no input edge. Classifying that nonexistent boundary aborts native execution with:This was observed in three
FluxStrategyPlanSuitecases and is reproduced deterministically by writing a Spark DataFrame backed byExistingRDDto Parquet with the Velox cuDF path enabled.Reproduction
ExistingRDD.A minimal Spark-side shape is:
The native plan exposes an already-GPU source leaf with
sources().empty(). The failure depends on that boundary shape, not on input scale.Root cause
Commit
07e146e88bdeliberately treats operator index zero as having a non-GPU predecessor. That preserves conversion when an MPP/UCX fragment's first GPU operator consumes an external VeloxRowVector:previousOperatorIsNotGpu = operatorIndex == 0 || !operatorProperties[operatorIndex - 1].producesGpuOutput;The same condition also matches a genuine GPU source leaf. Because the source operator accepts GPU input,
CompileState::compile()asksgpuInputBoundaryType()to classify an input that does not exist, producing the(0 vs. 0)invalid-state error.The two cases must be distinguished by the plan edge:
CudfFromVelox;RowVectorto convert.Expected behavior
CudfFromVeloxbefore an already-GPU source leaf with no input plan edge.07e146e88b.Fix and validation
Fix available in #58. It requires a real input plan edge before inserting the boundary and adds focused tests for both sides of the distinction.
Validation reported by #58:
FluxStrategyPlanSuitecases: 22/22 passed;velox_cudf_tocudf_selection_test: 24/25, with the remainingcomplexGroupingKeyExpressionsFallsBackfailure independently reproduced on currentdevand outside this boundary path.Acceptance criteria
CudfFromVelox.CudfFromVelox.07e146e88bremains covered.