Bump apache-beam[gcp] from 2.28.0 to 2.38.0 in /megalist_dataflow

[dependabot]

Bumps apache-beam[gcp] from 2.28.0 to 2.38.0.

Release notes

Sourced from apache-beam[gcp]'s releases.

Beam 2.38.0 release

We are happy to present the new 2.38.0 release of Beam. This release includes both improvements and new functionality. See the download page for this release.

For more information on changes in 2.38.0 check out the detailed release notes.

I/Os

• Introduce projection pushdown optimizer to the Java SDK (BEAM-12976). The optimizer currently only works on the BigQuery Storage API, but more I/Os will be added in future releases. If you encounter a bug with the optimizer, please file a JIRA and disable the optimizer using pipeline option --experiments=disable_projection_pushdown.
• A new IO for Neo4j graph databases was added. (BEAM-1857) It has the ability to update nodes and relationships using UNWIND statements and to read data using cypher statements with parameters.
• amazon-web-services2 has reached feature parity and is finally recommended over the earlier amazon-web-services and kinesis modules (Java). These will be deprecated in one of the next releases (BEAM-13174).
  • Long outstanding write support for Kinesis was added (BEAM-13175).
  • Configuration was simplified and made consistent across all IOs, including the usage of AwsOptions (BEAM-13563, BEAM-13663, BEAM-13587).
  • Additionally, there's a long list of recent improvements and fixes to S3 Filesystem (BEAM-13245, BEAM-13246, BEAM-13441, BEAM-13445, BEAM-14011), DynamoDB IO (BEAM-13209, BEAM-13209), SQS IO (BEAM-13631, BEAM-13510) and others.

New Features / Improvements

• Pipeline dependencies supplied through --requirements_file will now be staged to the runner using binary distributions (wheels) of the PyPI packages for linux_x86_64 platform (BEAM-4032). To restore the behavior to use source distributions, set pipeline option --requirements_cache_only_sources. To skip staging the packages at submission time, set pipeline option --requirements_cache=skip (Python).
• The Flink runner now supports Flink 1.14.x (BEAM-13106).
• Interactive Beam now supports remotely executing Flink pipelines on Dataproc (Python) (BEAM-14071).

Breaking Changes

• (Python) Previously DoFn.infer_output_types was expected to return Iterable[element_type] where element_type is the PCollection elemnt type. It is now expected to return element_type. Take care if you have overriden infer_output_type in a DoFn (this is not common). See BEAM-13860.
• (amazon-web-services2) The types of awsRegion / endpoint in AwsOptions changed from String to Region / URI (BEAM-13563).

Deprecations

• Beam 2.38.0 will be the last minor release to support Flink 1.11.
• (amazon-web-services2) Client providers (withXYZClientProvider()) as well as IO specific RetryConfigurations are deprecated, instead use withClientConfiguration() or AwsOptions to configure AWS IOs / clients. Custom implementations of client providers shall be replaced with a respective ClientBuilderFactory and configured through AwsOptions (BEAM-13563).

Bugfixes

• Fix S3 copy for large objects (Java) (BEAM-14011)
• Fix quadratic behavior of pipeline canonicalization (Go) (BEAM-14128)
  • This caused unnecessarily long pre-processing times before job submission for large complex pipelines.
• Fix pyarrow version parsing (Python)(BEAM-14235)

Known Issues

• See a full list of open issues that affect this version.

List of Contributors

... (truncated)

Changelog

Sourced from apache-beam[gcp]'s changelog.

[2.38.0] - 2022-04-20

I/Os

• Introduce projection pushdown optimizer to the Java SDK (BEAM-12976). The optimizer currently only works on the BigQuery Storage API, but more I/Os will be added in future releases. If you encounter a bug with the optimizer, please file a JIRA and disable the optimizer using pipeline option --experiments=disable_projection_pushdown.
• A new IO for Neo4j graph databases was added. (BEAM-1857) It has the ability to update nodes and relationships using UNWIND statements and to read data using cypher statements with parameters.
• amazon-web-services2 has reached feature parity and is finally recommended over the earlier amazon-web-services and kinesis modules (Java). These will be deprecated in one of the next releases (BEAM-13174).
  • Long outstanding write support for Kinesis was added (BEAM-13175).
  • Configuration was simplified and made consistent across all IOs, including the usage of AwsOptions (BEAM-13563, BEAM-13663, BEAM-13587).
  • Additionally, there's a long list of recent improvements and fixes to S3 Filesystem (BEAM-13245, BEAM-13246, BEAM-13441, BEAM-13445, BEAM-14011), DynamoDB IO (BEAM-13209, BEAM-13209), SQS IO (BEAM-13631, BEAM-13510) and others.

New Features / Improvements

• Pipeline dependencies supplied through --requirements_file will now be staged to the runner using binary distributions (wheels) of the PyPI packages for linux_x86_64 platform (BEAM-4032). To restore the behavior to use source distributions, set pipeline option --requirements_cache_only_sources. To skip staging the packages at submission time, set pipeline option --requirements_cache=skip (Python).
• The Flink runner now supports Flink 1.14.x (BEAM-13106).
• Interactive Beam now supports remotely executing Flink pipelines on Dataproc (Python) (BEAM-14071).

Breaking Changes

• (Python) Previously DoFn.infer_output_types was expected to return Iterable[element_type] where element_type is the PCollection elemnt type. It is now expected to return element_type. Take care if you have overriden infer_output_type in a DoFn (this is not common). See BEAM-13860.
• (amazon-web-services2) The types of awsRegion / endpoint in AwsOptions changed from String to Region / URI (BEAM-13563).

Deprecations

• Beam 2.38.0 will be the last minor release to support Flink 1.11.
• (amazon-web-services2) Client providers (withXYZClientProvider()) as well as IO specific RetryConfigurations are deprecated, instead use withClientConfiguration() or AwsOptions to configure AWS IOs / clients. Custom implementations of client providers shall be replaced with a respective ClientBuilderFactory and configured through AwsOptions (BEAM-13563).

Bugfixes

• Fix S3 copy for large objects (Java) (BEAM-14011)
• Fix quadratic behavior of pipeline canonicalization (Go) (BEAM-14128)
  • This caused unnecessarily long pre-processing times before job submission for large complex pipelines.
• Fix pyarrow version parsing (Python)(BEAM-14235)

[2.37.0] - 2022-03-04

Highlights

• Java 17 support for Dataflow (BEAM-12240).
  • Users using Dataflow Runner V2 may see issues with state cache due to inaccurate object sizes (BEAM-13695).
  • ZetaSql is currently unsupported (issue).
• Python 3.9 support in Apache Beam (BEAM-12000).

I/Os

• Go SDK now has wrappers for the following Cross Language Transforms from Java, along with automatic expansion service startup for each.
  • JDBCIO (BEAM-13293).
  • Debezium (BEAM-13761).

... (truncated)

Commits

• 93fd177 Set version for 2.38.0 RC1
• 22d91bd Merge pull request #17330 from chamikaramj/cherry_pick_pr_17329
• 12bc0e0 Merge pull request #17345: [cherry-pick][release-2.38.0][BEAM-13538][BEAM-142...
• 9211bb6 Merge pull request #17343: [BEAM-13538][BEAM-14296] Workaround to fix go-lice...
• 3da922c Merge pull request #17332: [cherry-pick][release-2.38.0][BEAM-14172] Update t...
• 9cc9692 [BEAM-14172] Update tox.ini for pydocs (#17176)
• e67cc36 Re-raise exceptions swallowed in several Python I/O connectors
• 93ac2a0 Merge pull request #17295: [cherry-pick][release-2.38.0][BEAM-13519] Solve ra...
• 4954909 [BEAM-13519] Solve race issues when the server responds with an error before ...
• 47a9d8f Merge pull request #17241: [cherry-pick][release-2.38.0][BEAM-14179] Fix poss...
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

[dependabot]

Superseded by #69.