[#38059] Fix GCS glob matching to support ** and / in object names by hilaryRope · Pull Request #38099 · apache/beam

hilaryRope · 2026-04-07T21:59:14Z

Fix GCS filesystem glob matching to handle `/` in object names and support `**`

Summary

The GCS filesystem's List() method was using filepath.Match() to filter objects, which incorrectly treats / as a path separator. Since GCS object names are flat (with / being just another character), patterns like gs://bucket/** failed to match objects containing /.

Problem

filepath.Match treats / as a path separator, so * cannot match across /
filepath.Match does not support ** for recursive matching
fileio.MatchFiles(scope, "gs://my-bucket/**") silently excluded objects like dir/subdir/file.txt

Solution

Replaced filepath.Match with a custom globToRegex() function:

* → matches any characters except / (single path segment)
** → matches any characters including / (recursive)
**/ → matches zero or more path segments

This aligns the Go SDK with the Python and Java SDKs.

Testing

Added unit tests for globToRegex() and List() with nested object names.

Mention the appropriate issue in your description (for example: addresses #123), if applicable.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

…ort **

gemini-code-assist · 2026-04-07T21:59:24Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses an issue where GCS filesystem glob patterns were incorrectly filtered due to the standard library's path-aware matching logic. By implementing a custom regex-based matching system, the GCS connector now properly handles object names containing slashes and supports recursive globbing, ensuring consistent behavior across different Apache Beam SDKs.

Highlights

Custom Glob Matching: Replaced the standard library's filepath.Match with a custom globToRegex function to correctly handle GCS object names, which are flat and treat '/' as a standard character.
Recursive Matching Support: Added support for the '**' glob pattern to enable recursive matching across GCS object paths, aligning the Go SDK with Python and Java implementations.
Comprehensive Testing: Introduced new unit tests for the glob-to-regex conversion logic and verified GCS list operations with nested object structures.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

utafrali

The core approach of replacing filepath.Match with a custom globToRegex is correct and well-motivated. The main issue is that ? is translated to . (matches /) while * is explicitly restricted to [^/]*, creating an inconsistency that will surprise callers. There are also minor concerns around the hand-maintained isRegexSpecial list (use regexp.QuoteMeta instead) and the silent swallowing of malformed [ patterns.

github-actions · 2026-04-07T22:47:31Z

Assigning reviewers:

R: @jrmccluskey for label go.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

hilaryRope · 2026-04-14T20:13:20Z

@utafrali all comments addressed, thanks!

github-actions · 2026-04-22T12:36:36Z

Reminder, please take a look at this pr: @jrmccluskey

jrmccluskey · 2026-04-22T14:59:26Z

@gemini-code-assist review

gemini-code-assist

Code Review

This pull request fixes GCS filesystem glob matching in the Go SDK to correctly handle slashes in object names and support recursive matching with **, aligning its behavior with the Python and Java SDKs. The implementation replaces filepath.Match with a custom globToRegex translator and updates the List operation to use regex-based filtering. Feedback was provided to optimize the generated regular expression by using non-capturing groups for recursive path segments.

jrmccluskey

Modify .github/trigger_files/beam_PostCommit_Go.json to trigger the post-commit, which will run all of the Go tests that interact with Dataflow / GCS.

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

hilaryRope · 2026-05-09T10:48:24Z

Modify .github/trigger_files/beam_PostCommit_Go.json to trigger the post-commit, which will run all of the Go tests that interact with Dataflow / GCS.

Thank you @jrmccluskey I did so - apologies for the delay in actioning it.

github-actions · 2026-05-16T12:35:18Z

Reminder, please take a look at this pr: @jrmccluskey

github-actions · 2026-05-20T13:13:15Z

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @lostluck for label go.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

github-actions · 2026-05-27T13:31:18Z

Reminder, please take a look at this pr: @lostluck

github-actions · 2026-06-01T14:20:13Z

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @shunping for label go.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

github-actions · 2026-06-09T13:15:54Z

Reminder, please take a look at this pr: @shunping

github-actions · 2026-06-11T13:40:06Z

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @lostluck for label go.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

damccorm · 2026-06-12T17:36:43Z

@jrmccluskey could you take another look at this one?

jrmccluskey · 2026-06-12T17:41:19Z

I don't know how I missed the follow-up ping, let me trigger the checks and we can get this moving

jrmccluskey · 2026-06-12T20:14:33Z

Thank you for your patience!

…mes (apache#38099) * fix GCS filesystem glob matching to handle / in object names and support ** * update CHANGES.md * update CHANGES.md * address comments * Update sdks/go/pkg/beam/io/filesystem/gcs/gcs.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * modify beam_PostCommit_Go.json as required --------- Co-authored-by: corda.ilaria@gmail.com <ilariac691@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com>

corda.ilaria@gmail.com added 3 commits April 7, 2026 23:48

fix GCS filesystem glob matching to handle / in object names and supp…

612d5b4

…ort **

update CHANGES.md

8476e8f

update CHANGES.md

61e625d

github-actions Bot added go io labels Apr 7, 2026

utafrali reviewed Apr 7, 2026

View reviewed changes

Comment thread sdks/go/pkg/beam/io/filesystem/gcs/gcs.go

Comment thread sdks/go/pkg/beam/io/filesystem/gcs/gcs.go

Comment thread sdks/go/pkg/beam/io/filesystem/gcs/gcs.go

Comment thread sdks/go/pkg/beam/io/filesystem/gcs/gcs_test.go

Comment thread sdks/go/pkg/beam/io/filesystem/gcs/gcs_test.go

github-actions Bot added the Next Action: Reviewers label Apr 7, 2026

address comments

5936119

github-actions Bot added the slow-review label Apr 22, 2026

github-actions Bot removed the slow-review label Apr 22, 2026

gemini-code-assist Bot reviewed Apr 22, 2026

View reviewed changes

Comment thread sdks/go/pkg/beam/io/filesystem/gcs/gcs.go Outdated

jrmccluskey reviewed Apr 22, 2026

View reviewed changes

jrmccluskey added Next Action: Author and removed Next Action: Reviewers labels Apr 23, 2026

Update sdks/go/pkg/beam/io/filesystem/gcs/gcs.go

f4ae220

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

github-actions Bot added Next Action: Reviewers and removed Next Action: Author labels May 9, 2026

modify beam_PostCommit_Go.json as required

0c42609

github-actions Bot added the slow-review label May 16, 2026

github-actions Bot added reassigned-reviewers and removed slow-review labels May 20, 2026

github-actions Bot added the slow-review label May 27, 2026

github-actions Bot removed the slow-review label Jun 1, 2026

github-actions Bot added the slow-review label Jun 9, 2026

github-actions Bot removed the slow-review label Jun 11, 2026

Merge branch 'master' into fix/gcs-glob-matching-slash-separator

bdd687c

jrmccluskey approved these changes Jun 12, 2026

View reviewed changes

Merge branch 'master' into fix/gcs-glob-matching-slash-separator

1364408

jrmccluskey merged commit 22ac580 into apache:master Jun 12, 2026
8 checks passed

Uh oh!

Conversation

hilaryRope commented Apr 7, 2026

Fix GCS filesystem glob matching to handle / in object names and support **

Summary

Problem

Solution

Testing

Uh oh!

gemini-code-assist Bot commented Apr 7, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

utafrali left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Apr 7, 2026

Uh oh!

hilaryRope commented Apr 14, 2026

Uh oh!

github-actions Bot commented Apr 22, 2026

Uh oh!

jrmccluskey commented Apr 22, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

jrmccluskey left a comment

Choose a reason for hiding this comment

Uh oh!

hilaryRope commented May 9, 2026

Uh oh!

github-actions Bot commented May 16, 2026

Uh oh!

github-actions Bot commented May 20, 2026

Uh oh!

github-actions Bot commented May 27, 2026

Uh oh!

github-actions Bot commented Jun 1, 2026

Uh oh!

github-actions Bot commented Jun 9, 2026

Uh oh!

github-actions Bot commented Jun 11, 2026

Uh oh!

damccorm commented Jun 12, 2026

Uh oh!

jrmccluskey commented Jun 12, 2026

Uh oh!

jrmccluskey commented Jun 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Fix GCS filesystem glob matching to handle `/` in object names and support `**`