[SPARK-54348][INFRA] Recover Python unit tests CI by installing zstandard==0.25.0

[SCHJonathan]

What changes were proposed in this pull request?

In #53024 (comment), PR CI Python unit tests failed due to

pyspark.errors.exceptions.base.PySparkImportError: [PACKAGE_NOT_INSTALLED] zstandard >= 0.25.0 must be installed; however, it was not found.

This PR add the required dependency to the pre-merge CI.

Why are the changes needed?

Recover Python unit tests CI

Does this PR introduce any user-facing change?

No

How was this patch tested?

PR #53024 Python CI back to healthy with this change

Was this patch authored or co-authored using generative AI tooling?

No

[LuciferYang]

@SCHJonathan Which Scala-side test cases will be affected by zstandard? We should also add some assume statements for these cases to ensure that they are skipped rather than failing when zstandard is not available.

[dongjoon-hyun]

To @sryza , the absolute path patch seems to break our CI on this PR somehow.

• [SPARK-54280][SDP] Require pipeline checkpoint storage dir to be absolute path #52999

[info] EndToEndAPISuite:
[info] - SQL Pipeline with mv, st, and flows *** FAILED *** (690 milliseconds)
[info]   java.lang.RuntimeException: Pipeline update process failed with exit code 1.
...
[info] pyspark.errors.exceptions.connect.SparkException: [PIPELINE_STORAGE_ROOT_INVALID] Pipeline storage root must be an absolute path with a URI scheme (e.g., file://, s3a://, hdfs://). Got: `/home/runner/work/spark/spark/target/tmp/spark-e693e257-1d24-4b6f-9265-2510010e5366/storage`. SQLSTATE: 42K03

[dongjoon-hyun]

cc @xi-db and @hvanhovell from the original PR, too.

The zstandard requirement was added by this.

• [SPARK-54194][CONNECT] Spark Connect Proto Plan Compression #52894

We have 3 fixes after that.

• [SPARK-54326][INFRA] Recover MacOS CIs by installing zstandard==0.25.0 #53025
• [SPARK-54194][PYTHON][FOLLOWUP] Fix connectutils.py to import pb2 conditionally #53037
• [SPARK-54348][INFRA] Recover Python unit tests CI by installing zstandard==0.25.0 #53058

[dongjoon-hyun]

Could you re-trigger the failed test pipeline, please, @SCHJonathan ?

[sryza]

Here's a PR to fix the EndToEndAPITest test failure: #53069. It fails reliably, I think it just must have somehow been skipped in the CI for my original PR.

[SCHJonathan]

To @sryza , the absolute path patch seems to break our CI on this PR somehow.

• [SPARK-54280][SDP] Require pipeline checkpoint storage dir to be absolute path #52999

Thanks @dongjoon-hyun , for this test, I have fixed it in #53024. I just re-trigger and the other failing test pipeline

[SCHJonathan]

@SCHJonathan Which Scala-side test cases will be affected by zstandard? We should also add some assume statements for these cases to ensure that they are skipped rather than failing when zstandard is not available.

In this case, it is the org.apache.spark.sql.connect.pipelines.PythonPipelineSuite

[dongjoon-hyun]

Thank you, @sryza . I merged your PR.

• [SPARK-54356][SDP] Fix EndToEndAPISuite caused by missing storage root schema #53069

[dongjoon-hyun]

+1, LGTM. Thank you, @SCHJonathan and @sryza .
Merged to master/4.1.