Skip to content

[cuDF] GPU source leaf triggers "GPU input boundary requires a source" during boundary selection #59

Description

@sperlingxx

Summary

On dev at de935de174, cuDF operator selection can insert CudfFromVelox before a GPU source plan node that has no input edge. Classifying that nonexistent boundary aborts native execution with:

[RUNTIME/INVALID_STATE] (0 vs. 0) GPU input boundary requires a source

This was observed in three FluxStrategyPlanSuite cases and is reproduced deterministically by writing a Spark DataFrame backed by ExistingRDD to Parquet with the Velox cuDF path enabled.

Reproduction

  1. Enable the Gluten/Velox cuDF execution path.
  2. Create a DataFrame from driver-local rows, so its input is ExistingRDD.
  3. Write the DataFrame to Parquet through the native writer path.
  4. Observe the exception above while compiling the native operator pipeline, before the Parquet write completes.

A minimal Spark-side shape is:

rows = [(1, "a"), (2, "b")]
df = spark.createDataFrame(rows, ["id", "value"])
df.write.mode("overwrite").parquet(output_path)

The native plan exposes an already-GPU source leaf with sources().empty(). The failure depends on that boundary shape, not on input scale.

Root cause

Commit 07e146e88b deliberately treats operator index zero as having a non-GPU predecessor. That preserves conversion when an MPP/UCX fragment's first GPU operator consumes an external Velox RowVector:

previousOperatorIsNotGpu =
    operatorIndex == 0 || !operatorProperties[operatorIndex - 1].producesGpuOutput;

The same condition also matches a genuine GPU source leaf. Because the source operator accepts GPU input, CompileState::compile() asks gpuInputBoundaryType() to classify an input that does not exist, producing the (0 vs. 0) invalid-state error.

The two cases must be distinguished by the plan edge:

  • a first GPU operator with a real input source may require CudfFromVelox;
  • a GPU source plan node with zero sources has no upstream RowVector to convert.

Expected behavior

  • Do not insert CudfFromVelox before an already-GPU source leaf with no input plan edge.
  • Continue inserting the conversion for a CPU source feeding a GPU consumer.
  • Preserve the external-fragment-input behavior introduced by 07e146e88b.
  • ExistingRDD-to-Parquet execution completes without the GPU input-boundary exception.

Fix and validation

Fix available in #58. It requires a real input plan edge before inserting the boundary and adds focused tests for both sides of the distinction.

Validation reported by #58:

  • focused boundary tests: 2/2 passed;
  • all FluxStrategyPlanSuite cases: 22/22 passed;
  • ExistingRDD -> Parquet -> readback reproducer: passed;
  • complete L3 TPC-H SF1 Q1-Q22 row and plan golden comparison: passed;
  • native UCX suite: 78 run, 126 expected skips, 0 failures;
  • forced multi-fragment MPP/UCX TPC-H Q3: passed with remote UCX exercised;
  • full velox_cudf_tocudf_selection_test: 24/25, with the remaining complexGroupingKeyExpressionsFallsBack failure independently reproduced on current dev and outside this boundary path.

Acceptance criteria

  • A zero-source GPU leaf does not receive CudfFromVelox.
  • A CPU source followed by a GPU consumer still receives CudfFromVelox.
  • The MPP/UCX external-input scenario from 07e146e88b remains covered.
  • ExistingRDD-to-Parquet and the affected Flux strategy cases complete without the exception.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcudf

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions