[SPARK-53959][CONNECT] Throw a client-side error when creating a dataframe from a pandas dataframe with an index but no data #52670
+20
−4
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
HyukjinKwon
reviewed
Oct 20, 2025
| spark_type = from_arrow_type(field_type) | ||
| struct.add(field.name, spark_type, nullable=field.nullable) | ||
| schema = struct | ||
| if len(schema) == 0: |
There was a problem hiding this comment.
qq, how's Spark Classic code path? I think we should keep it matched
There was a problem hiding this comment.
hey @HyukjinKwon!
In spark classic the code does not throw an error
df = pd.DataFrame(index=range(5))
spark.createDataFrame(df).collect()
this code results in an empty list []. We could add the same error for the classic spark, but it will be a breaking change technically. Do you think it makes sense to do that?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Spark Connect Python client does not throw a proper error when creating a dataframe from a pandas dataframe with a index and empty data.
Generally, spark connect client throws a client-side error
[CANNOT_INFER_EMPTY_SCHEMA] Can not infer schema from an empty dataset. when creating a dataframe without data, for example viaor
or
This does not happen when pandas dataframe has an index but no data, e.g.
What happens instead is that the dataframe is successfully converted to a LocalRelation on the client, is sent to the server, but the server then throws the following exception:
INTERNAL_ERROR: Input data for LocalRelation does not produce a schema. SQLSTATE: XX000. XX000 is an internal error sql state and the error is not actionable enough for the user.This PR fixes this problem by throwing
CANNOT_INFER_EMPTY_SCHEMAin case the dataframe has rows (because of the index), but does not have columns.Why are the changes needed?
Currently the error is thrown as a server-side internal error and the error message is not actionable enough for the user.
Does this PR introduce any user-facing change?
Creating a spark connect dataframe from an pandas dataframe with an index but no data will now throw a client-side error
[CANNOT_INFER_EMPTY_SCHEMA] Can not infer schema from an empty datasetinstead of the server-sideINTERNAL_ERROR: Input data for LocalRelation does not produce a schema. SQLSTATE: XX000.How was this patch tested?
New unit test.
Was this patch authored or co-authored using generative AI tooling?
No