[Java IO] Add ArrowFlight IO connector

[bvolpato]

Summary

Add a Java I/O connector for Apache Arrow Flight, providing bounded reads and batched writes over gRPC using Arrow record batches.

Fixes #20116

Changes

• ArrowFlightIO.Read: fetches the Arrow schema through getInfo, splits work by FlightEndpoint, and converts record batches to Beam Row values through ArrowConversion.
• Endpoint-level parallelism: creates one source per endpoint. An unsplit reader narrows its current source to the selected endpoint, preventing later resplits from duplicating other endpoints.
• ArrowFlightIO.Write: buffers Beam rows into configurable batches, converts them to Arrow vectors, streams them through doPut, and propagates server errors.
• Schema conversion: adds public Beam-to-Arrow schema conversion to ArrowConversion.ArrowSchemaTranslator for supported primitive types.
• Authentication: supports optional bearer tokens through withToken().
• Build integration: registers flight-core, adds the :sdks:java:io:arrow-flight module, and includes it in javaioPreCommit.
• Documentation: adds ArrowFlightIO to the 2.76.0 changelog and built-in connector table.

Java 17+ runtime requirement

Arrow requires --add-opens=java.base/java.nio=ALL-UNNAMED on JVMs executing the connector. Beam SDK containers enable this by default. Other environments must configure the local or runner JVM and SDK harnesses as documented in ArrowFlightIO Javadoc.

Testing

Tests use an in-process FlightServer and cover:

• reads and endpoint splitting
• unsplit-reader current-source behavior
• writes and bearer-token propagation
• server error propagation
• unsupported write schemas
• Beam-to-Arrow schema conversion

Validated locally on Java 17 with:

./gradlew --no-daemon :sdks:java:extensions:arrow:build :sdks:java:io:arrow-flight:build

Validated tests on Java 21 with:

./gradlew --no-daemon :sdks:java:extensions:arrow:test :sdks:java:io:arrow-flight:test

Did this cause any problems?

Rollback/revert this commit and redeploy the service.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description. This automatically links the pull request to the issue.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on making the review process smoother.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a new Java I/O connector for Apache Arrow Flight, significantly enhancing data transfer capabilities within Beam pipelines. The connector leverages Arrow Flight's high-performance RPC framework to facilitate efficient reading from and writing to Arrow Flight servers, minimizing serialization overhead and enabling faster data movement. This integration expands Beam's ecosystem by providing a robust solution for interacting with various Arrow Flight-compatible data systems.

Highlights

• ArrowFlightIO.Read: Implemented a BoundedSource-based read transform that fetches Arrow Flight schemas, splits work by FlightEndpoint, and converts record batches to Beam Rows using the existing ArrowConversion extension.
• ArrowFlightIO.Write: Developed a DoFn-based write transform that buffers Beam Rows into configurable batches, converts them to Arrow vectors, and streams them to a Flight server via doPut.
• Endpoint-level Split Parallelism: Enabled each FlightEndpoint returned by the server to become a separate sub-source for parallel reads, enhancing data ingestion efficiency.
• Bearer Token Authentication: Added optional support for bearer token authentication via withToken() for secure connections.
• Build Integration: Registered flight-core and flight-sql artifacts in BeamModulePlugin.groovy and added the new :sdks:java:io:arrow-flight module to settings.gradle.kts.
• Documentation: Updated CHANGES.md and the built-in connectors table in connectors.md to reflect the addition of the ArrowFlightIO connector.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[github-actions]

Assigning reviewers:

R: @chamikaramj for label java.
R: @damccorm for label build.
R: @shunping for label website.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[Abacn]

Thanks for the new IO! Had an initial comment to kick off

[bvolpato]

Run Java_Amazon-Web-Services2_IO_Direct PreCommit

[github-actions]

Reminder, please take a look at this pr: @chamikaramj @damccorm @shunping

[github-actions]

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @kennknowles for label java.
R: @Abacn for label build.
R: @kennknowles for label website.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

[github-actions]

Reminder, please take a look at this pr: @kennknowles @Abacn

[github-actions]

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @ahmedabu98 for label java.
R: @damccorm for label build.
R: @damccorm for label website.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

[damccorm]

waiting on author

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 56.99%. Comparing base (de21b1c) to head (447a459).
⚠️ Report is 87 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #37904      +/-   ##
============================================
+ Coverage     54.76%   56.99%   +2.23%     
- Complexity     1716     3634    +1918     
============================================
  Files          1066     1186     +120     
  Lines        169075   190661   +21586     
  Branches       1255     3774    +2519     
============================================
+ Hits          92589   108676   +16087     
- Misses        74269    78501    +4232     
- Partials       2217     3484    +1267     

Flag	Coverage Δ
java	72.44% <ø> (+4.82%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Reminder, please take a look at this pr: @ahmedabu98 @damccorm