feat: Add LogElements transform to Java SDK by lalitium · Pull Request #38533 · apache/beam

lalitium · 2026-05-19T05:06:05Z

Please add a meaningful description for your change here

What does this PR do?

Added a Java SDK LogElements transform for logging each element of a PCollection while passing the elements through unchanged.

Supports SLF4J log levels plus optional prefix, timestamp, window, and pane info logging, mirroring the Python SDK’s LogElements convenience transform.

fixes #38528

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

gemini-code-assist · 2026-05-19T05:07:22Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request adds a convenient logging utility to the Apache Beam Java SDK, mirroring the existing functionality in the Python SDK. It enables developers to easily inspect PCollection elements during pipeline execution without altering the data flow, which is particularly useful for debugging and monitoring data processing pipelines.

Highlights

New LogElements Transform: Introduced a new LogElements PTransform in the Java SDK that allows logging elements of a PCollection while passing them through unchanged.
Flexible Logging Configuration: Supports SLF4J log levels and provides optional configuration for including prefixes, timestamps, windowing, and pane information.
Testing: Added comprehensive unit tests to verify element preservation, metadata formatting, and display data population.

New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds a new LogElements PTransform to the Beam Java core SDK that logs each element of a PCollection (optionally with prefix, timestamp, window, and pane info) and passes elements through unchanged, along with unit tests.

Changes:

Introduces LogElements<T> PTransform with fluent builders (trace/debug/info/warn/error/of, withPrefix, withTimestamp, withWindow, withPaneInfo) and display data.
Implements an internal LoggingFn DoFn that formats and logs each element at the configured SLF4J level, then outputs it unchanged.
Adds LogElementsTest covering element pass-through, format string composition, and display data.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/LogElements.java	New `PTransform` for logging PCollection elements at a configurable SLF4J level with optional metadata.
sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/LogElementsTest.java	Unit tests for element pass-through, log formatting, and display data.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+      default:
+        LOG.error("{}", message);


+  @Test
+  public void testDisplayData() {
+    DisplayData displayData =
+        DisplayData.from(
+            LogElements.of(Level.WARN).withPrefix("row: ").withTimestamp().withWindow());
+
+    assertThat(displayData, hasDisplayItem("level", "WARN"));
+    assertThat(displayData, hasDisplayItem("prefix", "row: "));
+    assertThat(displayData, hasDisplayItem("withTimestamp", true));
+    assertThat(displayData, hasDisplayItem("withWindow", true));
+  }


gemini-code-assist

Code Review

This pull request introduces the LogElements transform, which allows for logging elements in a PCollection at various log levels with optional metadata such as timestamps and window information. The review feedback highlights a performance concern where log messages are formatted regardless of whether the log level is enabled; it is recommended to wrap the logging logic in a level check to avoid unnecessary overhead. Additionally, the explicit call to setCoder should be removed to avoid potential runtime exceptions and rely on Beam's automatic coder propagation.

gemini-code-assist · 2026-05-19T05:09:25Z

+      log(
+          level,
+          formatForLogging(
+              element,
+              prefix,
+              withTimestamp,
+              withWindow,
+              withPaneInfo,
+              timestamp,
+              window,
+              paneInfo));
+      receiver.output(element);
+    }


The formatForLogging method is called for every element before checking if the log level is enabled. This involves string concatenation and toString() calls (potentially on large objects, windows, or timestamps) which can be very expensive in high-throughput pipelines. You should check if the log level is enabled before constructing the log message.

if (isLoggingEnabled(level)) { log( level, formatForLogging( element, prefix, withTimestamp, withWindow, withPaneInfo, timestamp, window, paneInfo)); } receiver.output(element); } private boolean isLoggingEnabled(Level level) { switch (level) { case TRACE: return LOG.isTraceEnabled(); case DEBUG: return LOG.isDebugEnabled(); case INFO: return LOG.isInfoEnabled(); case WARN: return LOG.isWarnEnabled(); case ERROR: default: return LOG.isErrorEnabled(); } }

gemini-code-assist · 2026-05-19T05:09:25Z

+
+  @Override
+  public PCollection<T> expand(PCollection<T> input) {
+    return input.apply("Log", ParDo.of(new LoggingFn<>(this))).setCoder(input.getCoder());


Calling setCoder(input.getCoder()) is generally redundant for a ParDo that returns the same type as its input, as Beam automatically propagates the coder. Additionally, input.getCoder() can throw an IllegalStateException if the coder hasn't been set yet and cannot be inferred at this stage of pipeline construction. It is safer to let the SDK handle coder propagation.

Suggested change

return input.apply("Log", ParDo.of(new LoggingFn<>(this))).setCoder(input.getCoder());

return input.apply("Log", ParDo.of(new LoggingFn<>(this)));

github-actions · 2026-05-19T08:15:06Z

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

lalitium · 2026-05-19T17:45:38Z

the failed checks seems unrelated to my commit.

lalitium · 2026-05-19T17:46:07Z

assign set of reviewers

github-actions · 2026-05-19T17:47:43Z

Assigning reviewers:

R: @chamikaramj for label java.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

lalitium · 2026-05-19T17:49:15Z

retest this please

github-actions · 2026-05-19T17:50:04Z

Reviewers are already assigned to this PR: @chamikaramj

lalitium · 2026-05-19T18:17:35Z

R: @ahmedabu98 please take a look. Thanks.

github-actions · 2026-05-19T18:19:42Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

ahmedabu98

The transform looks great, but I think testing could be better

ahmedabu98

LGTM

ahmedabu98 · 2026-05-21T16:31:55Z

Will merge when tests go green

lalitium · 2026-05-21T16:36:35Z

really appreciate the speedy review.

ahmedabu98 · 2026-05-21T17:59:10Z

Looks like the new test failed:

2026-05-21T17:04:06.3026397Z LogElementsTest > testLogElementsLogsAtConfiguredLevels FAILED
2026-05-21T17:04:06.3057465Z     java.lang.AssertionError at LogElementsTest.java:88

lalitium · 2026-05-21T20:15:29Z

so the issue is that the DirectRunner uses slf4j-simple which outputs to stderr, which ExpectedLogs can't capture(only works with JUL).
also slf4j-simple has trace and debug disabled by default.

solutions like changing slf4j-simple to slf4j-jdk14 has high blast radius.

the next most obvious path is to capture stderr(trace and debug still won't appear) but can be flaky. (bad idea)

Lmk if you have better idea.

chamikaramj · 2026-06-01T23:23:25Z

What are the next steps here ? Thanks!

lalitium · 2026-06-04T12:21:14Z

What are the next steps here ? Thanks!

Thanks for the ping @chamikaramj! I’ve move this forward by avoiding the runner-dependent test.

Given that ExpectedLogs only works reliably when SLF4J is bridged to JUL, and DirectRunner uses slf4j-simple, I don’t think changing the project-wide SLF4J binding or capturing stderr is worth the blast radius/flakiness.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

lalitium · 2026-06-04T15:14:34Z

+    @ProcessElement
+    public void processElement(
+        @Element T element,
+        @DoFn.Timestamp Instant timestamp,
+        BoundedWindow window,
+        PaneInfo paneInfo,
+        OutputReceiver<T> receiver) {


false positive

lalitium · 2026-06-04T15:16:35Z

+        @DoFn.Timestamp Instant timestamp,
+        BoundedWindow window,
+        PaneInfo paneInfo,


false positive

ahmedabu98

LGTM! Thanks for the quick solution @lalitx17

* feat: Add Java LogElements transform * Addressed LogElements review feedback * Add LogElements tests * Add LogElements level logging test * Refactor LogElements logging test

feat: Add Java LogElements transform

a27580b

Copilot AI review requested due to automatic review settings May 19, 2026 05:06

github-actions Bot added the java label May 19, 2026

Copilot AI reviewed May 19, 2026

View reviewed changes

gemini-code-assist Bot reviewed May 19, 2026

View reviewed changes

Addressed LogElements review feedback

c34d3ee

Add LogElements tests

cded2f4

github-actions Bot added the Next Action: Reviewers label May 19, 2026

ahmedabu98 reviewed May 20, 2026

View reviewed changes

Comment thread sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/LogElementsTest.java

Add LogElements level logging test

8a90b96

ahmedabu98 approved these changes May 21, 2026

View reviewed changes

Refactor LogElements logging test

2696c39

lalitium requested a review from Copilot June 4, 2026 14:17

Copilot AI reviewed Jun 4, 2026

View reviewed changes

ahmedabu98 approved these changes Jun 4, 2026

View reviewed changes

ahmedabu98 merged commit 3a99f94 into apache:master Jun 4, 2026
25 checks passed

	return input.apply("Log", ParDo.of(new LoggingFn<>(this))).setCoder(input.getCoder());
	return input.apply("Log", ParDo.of(new LoggingFn<>(this)));

Uh oh!

Conversation

lalitium commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

GitHub Actions Tests Status (on master branch)

Uh oh!

gemini-code-assist Bot commented May 19, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot May 19, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 19, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented May 19, 2026

Uh oh!

lalitium commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lalitium commented May 19, 2026

Uh oh!

github-actions Bot commented May 19, 2026

Uh oh!

lalitium commented May 19, 2026

Uh oh!

github-actions Bot commented May 19, 2026

Uh oh!

lalitium commented May 19, 2026

Uh oh!

github-actions Bot commented May 19, 2026

Uh oh!

ahmedabu98 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ahmedabu98 left a comment

Choose a reason for hiding this comment

Uh oh!

ahmedabu98 commented May 21, 2026

Uh oh!

lalitium commented May 21, 2026

Uh oh!

ahmedabu98 commented May 21, 2026

Uh oh!

lalitium commented May 21, 2026

Uh oh!

chamikaramj commented Jun 1, 2026

Uh oh!

lalitium commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

lalitium Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

lalitium Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

ahmedabu98 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

lalitium commented May 19, 2026 •

edited

Loading

lalitium commented May 19, 2026 •

edited

Loading

lalitium commented Jun 4, 2026 •

edited

Loading