Bump apache-beam[gcp] from 2.28.0 to 2.37.0 in /megalist_dataflow

[dependabot]

Bumps apache-beam[gcp] from 2.28.0 to 2.37.0.

Release notes

Sourced from apache-beam[gcp]'s releases.

Beam 2.37.0 release

We are happy to present the new 2.37.0 release of Beam. This release includes both improvements and new functionality. See the download page for this release.

For more information on changes in 2.37.0 check out the detailed release notes.

Highlights

• Java 17 support for Dataflow (BEAM-12240).
  • Users using Dataflow Runner V2 may see issues with state cache due to inaccurate object sizes (BEAM-13695).
  • ZetaSql is currently unsupported (issue).
• Python 3.9 support in Apache Beam (BEAM-12000).
  • Dataflow support for Python 3.9 is expected to be available with 2.37.0, but may not be fully available yet when the release is announced (BEAM-13864).
  • Users of Dataflow Runner V2 can run Python 3.9 pipelines with 2.37.0 release right away.

I/Os

• Go SDK now has wrappers for the following Cross Language Transforms from Java, along with automatic expansion service startup for each.
  • JDBCIO (BEAM-13293).
  • Debezium (BEAM-13761).
  • BeamSQL (BEAM-13683).
  • BiqQuery (BEAM-13732).
  • KafkaIO now also has automatic expansion service startup. (BEAM-13821).

New Features / Improvements

• DataFrame API now supports pandas 1.4.x (BEAM-13605).
• Go SDK DoFns can now observe trigger panes directly (BEAM-13757).

Known Issues

• See a full list of open issues that affect this version.

List of Contributors

According to git shortlog, the following people contributed to the 2.37.0 release. Thank you to all contributors!

Aizhamal Nurmamat kyzy Alexander Alexander Chermenin Alexandr Zhuravlev Alexey Romanenko Anand Inguva andoni-guzman andreukus Andy Ye Artur Khanin

... (truncated)

Changelog

Sourced from apache-beam[gcp]'s changelog.

[2.37.0] - 2022-03-04

Highlights

• Java 17 support for Dataflow (BEAM-12240).
  • Users using Dataflow Runner V2 may see issues with state cache due to inaccurate object sizes (BEAM-13695).
  • ZetaSql is currently unsupported (issue).
• Python 3.9 support in Apache Beam (BEAM-12000).
  • Dataflow support for Python 3.9 is expected to be available with 2.37.0, but may not be fully available yet when the release is announced (BEAM-13864).
  • Users of Dataflow Runner V2 can run Python 3.9 pipelines with 2.37.0 release right away.

I/Os

• Go SDK now has wrappers for the following Cross Language Transforms from Java, along with automatic expansion service startup for each.
  • JDBCIO (BEAM-13293).
  • Debezium (BEAM-13761).
  • BeamSQL (BEAM-13683).
  • BiqQuery (BEAM-13732).
  • KafkaIO now also has automatic expansion service startup. (BEAM-13821).

New Features / Improvements

• DataFrame API now supports pandas 1.4.x (BEAM-13605).
• Go SDK DoFns can now observe trigger panes directly (BEAM-13757).

Breaking Changes

Deprecations

Bugfixes

Known Issues

[2.36.0] - 2022-02-07

I/Os

• Support for stopReadTime on KafkaIO SDF (Java).(BEAM-13171).
• Added ability to register URI schemes to use the S3 protocol via FileIO using amazon-web-services2 (amazon-web-services already had this ability). (BEAM-12435, BEAM-13245).

New Features / Improvements

• Added support for cloudpickle as a pickling library for Python SDK (BEAM-8123). To use cloudpickle, set pipeline option: --pickler_lib=cloudpickle
• Added option to specify triggering frequency when streaming to BigQuery (Python) (BEAM-12865).
• Added option to enable caching uploaded artifacts across job runs for Python Dataflow jobs (BEAM-13459). To enable, set pipeline option: --enable_artifact_caching, this will be enabled by default in a future release.

Breaking Changes

• Updated the jedis from 3.x to 4.x to Java RedisIO. If you are using RedisIO and using jedis directly, please refer to this page to update it. (BEAM-12092).
• Datatype of timestamp fields in SqsMessage for AWS IOs for SDK v2 was changed from String to long, visibility of all fields was fixed from package private to public BEAM-13638.

... (truncated)

Commits

• d97c521 Set version for 2.37.0 RC3
• 86b7cfa [release-2.37.0][BEAM-13980] Re-add method gone missing in af2f8ee6 (#16918) ...
• 82a05e3 [release-2.37.0][BEAM-14005] Fix ignored exception in BatchSpannerRead (#16969)
• 304420a [release-2.37.0][BEAM-12000] Build wheels for Python 3.9 #16964 (#16968)
• 503268b [release-2.37.0] Fix BoundedQueueExecutor and StreamingDataflowWorker to actu...
• 39895d1 [cherry-pick][release-2.37.0][BEAM-12164]: fix bug when retrieving either str...
• 8487557 [release-2.37.0][BEAM-13959] Fix StorageApi writes when a field is named f (#...
• 90d8a6e [release-2.37.0][BEAM-13921] filter out debeziumIO test for spark runner (#16...
• 4e215fd [release-2.37.0][BEAM-13955] Fix pylint breakage from #16836 (#16868)
• 821df1f [cherry-pick][release-2.37.0][BEAM-13931] - make sure large rows cause BQIO t...
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

[dependabot]

Superseded by #66.