[MAINTENANCE] Run the GCS docs snippet tests - #12059
Merged
Merged
Conversation
The five GCS docs snippet fixtures were unreachable for two independent reasons in the test wiring, both of which had to be fixed for any of them to run. Removes GCS from the unconditional cloud-object-store skip in the docs snippet runner. S3 and Azure Blob stay in that block -- neither has storage nor a live credential yet -- but GCS now has both, so the blanket gate no longer applies to it. Gates GCS on its own --gcs flag instead of --bigquery, and requests that flag from docs-creds-needed. Sharing --bigquery made GCS coverage silently conditional on BigQuery's health, and turning --bigquery on here would also open a real BigQuery connection at collection time. The new flag is inert for build_test_backends_list, so it cannot abort a session the way the credential-validating flags can. The workflow half of this change -- writing the GCP credentials file for the docs-snippets job and pointing GOOGLE_APPLICATION_CREDENTIALS at an absolute path -- is in #12057 and has to land first. Because the workflow definition is read from the base branch, these tests cannot pass until it does.
✅ Deploy Preview for niobium-lead-7998 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
joshua-stauffer
added a commit
that referenced
this pull request
Aug 12, 2026
This branch changes both halves of a pull_request_target workflow: the workflow definition, which GitHub reads from the base branch, and the test code, which is checked out from the PR head. Only the code half takes effect in this PR's own CI run, so enabling the GCS docs snippet tests here made them run against a job whose credentials setup was still develop's. Leaving the workflow half alone here keeps this PR a no-op against develop -- it writes a credentials file no test currently reads -- which is what makes it safe to merge without a CI run that can exercise it. The test wiring moves to #12059, which can then be validated against a base branch that already has this.
The two GCS fixtures that also depend on Spark are reachable only from a leg that requests both flags. docs-creds-needed requests --gcs without --spark and docs-spark requested --spark without --gcs, so create_a_data_source_filesystem_gcs_spark and create_a_data_asset_filesystem_gcs_directory_asset were skipped in both and nothing reported the gap. The GCP credentials and bucket are job-level environment, so the docs-spark leg already has everything it needs to reach the bucket.
docs-spark requests --gcs but installed neither google-cloud-storage nor anything that depends on it, so the Spark GCS datasources failed to construct with a missing google dependency. docs-creds-needed was getting the client transitively from the bigquery requirements. That works today but reintroduces the coupling this change set set out to remove -- GCS coverage should not depend on BigQuery's dependency set staying where it is -- so the requirement is now declared explicitly.
The script still passed batching_regex to add_csv_asset and called build_batch_request on the asset. Both predate the move to batch definitions, so the asset now rejects batching_regex as an extra field. The pandas guide alongside it was migrated at the time; this one was not, and nothing noticed because the fixture had never run. Mirrors the pandas guide exactly: the regex moves to add_batch_definition_monthly, and the batch request comes from the batch definition.
Requesting --gcs from docs-spark got the fixtures past their Python-side
gates, but Spark reads GCS through gs:// URIs and Hadoop resolves that scheme
only with the GCS connector on the JVM classpath:
org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "gs"
The connector is a jar that has to be present when the JVM starts, so it
cannot come from a requirements file. Until the workflow stages it, requesting
the flag converts a skip into a failure and buys nothing, so docs-spark goes
back to requesting --spark alone.
The gap is now recorded in the GCS README and next to the flags themselves,
rather than being a silent consequence of which leg requests what.
The docs-creds-needed GCS requirement stays: that leg does request --gcs, and
the pandas fixtures it covers are passing.
Both GCS guides built a BatchRequest by hand and passed it back to the asset. BatchDefinition.get_batch is public API and does exactly that internally, so the intermediate request is machinery the reader never needs to see. Applied to the pandas guide as well as the spark one: they are sibling guides demonstrating the same flow, and leaving them on different patterns would be worse than either pattern on its own.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Un-skips the GCS docs snippet fixtures and gates them on a new
--gcspytest flag, whichdocs-creds-needednow requests.Two independent gates each made these fixtures unreachable, so both had to go:
_check_for_skipped_testsskipped anything backed by S3, GCS, or Azure Blob outright. S3 and Azure Blob stay in that block — neither has storage or a live credential yet — but GCS now has both, so the blanket gate no longer applies to it.--bigquerygate. GCS was gated on--bigquery, which made GCS coverage silently conditional on BigQuery's health. It now has its own--gcsflag, which is inert forbuild_test_backends_listand so cannot abort collection the way the credential-validating flags can.Two further fixes fell out of actually running them:
docs-creds-needednow declaresrequirements-dev-gcs.txtexplicitly. It was gettinggoogle-cloud-storagetransitively from the BigQuery requirements — the same GCS-on-BigQuery coupling this PR removes at the flag level, still present one layer down.batching_regexintoadd_csv_assetand calledbuild_batch_requeston the asset, both of which predate batch definitions, so the asset now rejectsbatching_regexas an extra field. The pandas guide beside it was migrated when that API changed; this one was not, and nothing noticed because the fixture had never run. It has been shipping instructions that raise.Why
The GCS docs snippets are user-facing documentation that has not been executed in CI for the duration of the CI transition. The bucket (#12056), the credentials (#12057), and the datasource-level tests (#12058) are all restored; this reconnects the docs snippets to them.
User impact
The Spark-on-GCS guide no longer documents an API call that raises. Everything else is test and CI wiring.
How to review
All pandas GCS fixtures pass. The bucket needs exactly three objects under
data/taxi_yellow_tripdata_samples/—yellow_tripdata_sample_2019-01/02/03.csv— becausepartitioned_on_datetime.pyasserts an exact batch count of 3.The Spark GCS fixtures remain skipped, deliberately. Requesting
--gcsfromdocs-sparkgets them past their Python-side gates and straight into:Spark resolves
gs://only with the GCS Hadoop connector on the JVM classpath. That jar must exist when the JVM starts, so no requirements file can supply it — it needsPYSPARK_SUBMIT_ARGSor a pre-staged jar plusfs.gs.impland service-account Hadoop config. Until that exists, requesting the flag converts a skip into a failure and buys nothing. The gap is documented intests/integration/test_definitions/gcs/README.mdand beside the flags intasks.py, rather than being a silent consequence of which leg requests what.Worth a close look:
--gcsflag is genuinely inert forbuild_test_backends_list— the concern is a flag that aborts collection rather than skipping.