Bug: Metric dimensions are persisting across Lambda invocations

[surajkumar]

Expected Behaviour

When adding custom dimensions to metrics, they should not persist across Lambda invocations.

Current Behaviour

• Metric dimensions are persisting across Lambda invocations which causes inaccurate metric data.
• This has been tested on version 2.1.0 and the latest 2.8.0

Code snippet

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.lambda.runtime.events.APIGatewayProxyRequestEvent;
import com.amazonaws.services.lambda.runtime.events.APIGatewayProxyResponseEvent;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import software.amazon.lambda.powertools.logging.CorrelationIdPaths;
import software.amazon.lambda.powertools.logging.Logging;
import software.amazon.lambda.powertools.metrics.FlushMetrics;
import software.amazon.lambda.powertools.metrics.Metrics;
import software.amazon.lambda.powertools.metrics.MetricsFactory;
import software.amazon.lambda.powertools.metrics.model.DimensionSet;

import java.util.Map;

public class LambdaHandler implements RequestHandler<APIGatewayProxyRequestEvent, APIGatewayProxyResponseEvent> {
    private static final Logger LOGGER = LogManager.getLogger(LambdaHandler.class);
    private static final Metrics METRICS = MetricsFactory.getMetricsInstance();

    private boolean shouldEmitMetric = true;

    public LambdaHandler() {}

    @Override
    @Logging(correlationIdPath = CorrelationIdPaths.API_GATEWAY_REST, clearState = true)
    @FlushMetrics(namespace = "test-aws-stack", captureColdStart = true)
    public APIGatewayProxyResponseEvent handleRequest(APIGatewayProxyRequestEvent input, Context context) {
        LOGGER.info("Entry LambdaHandler#handleRequest");

        try {
            if (shouldEmitMetric) {
                Map<String, String> dimensions = Map.of("EXAMPLE_KEY", "EXAMPLE_VALUE");
                DimensionSet dimensionSet = new DimensionSet();
                dimensions.forEach(dimensionSet::addDimension);
                METRICS.addDimension(dimensionSet);
                METRICS.addMetric("SERL", 1d);

                shouldEmitMetric = false;
                LOGGER.info("Emitted example `SERL` metric and EXAMPLE dimensions");
            } else {
                LOGGER.info("Metric has been emitted in a previous invocation so will not emit this time");
            }

            METRICS.addMetric("Expected", 1d);

        } catch (Exception e) {
            LOGGER.error("An error occurred", e);
        }

        LOGGER.info("Finished!");

        return response();
    }

    private static APIGatewayProxyResponseEvent response() {
        APIGatewayProxyResponseEvent response = new APIGatewayProxyResponseEvent();
        response.withHeaders(Map.of("Content-Type", "application/json"));
        response.setStatusCode(200);
        response.setBody("{}");
        return response;
    }
}

Possible Solution

Clear dimensions at the end of an invocation.

We have tried a few things like @FlushMetrics, manually calling metrics.flush() etc but no success.

Steps to Reproduce

Create a metric with a custom dimension e.g.

Map<String, String> dimensions = Map.of("EXAMPLE_KEY", "EXAMPLE_VALUE");
DimensionSet dimensionSet = new DimensionSet();
dimensions.forEach(dimensionSet::addDimension);
METRICS.addDimension(dimensionSet);
METRICS.addMetric("SERL", 1d);

Wrap it in a controllable if-statement so that it only executes once. See code in Code snippet section of this bug report.

Ensure to do this in a Java Lambda.

Assert logs still contain the "EXAMPLE_KEY" and "EXAMPLE_VALUE" in the log group e.g.

{
    "_aws": {
        "Timestamp": 1765297536692,
        "CloudWatchMetrics": [
            {
                "Namespace": "test-aws-stack",
                "Metrics": [
                    {
                        "Name": "Expected",
                        "Unit": "None"
                    }
                ],
                "Dimensions": [
                    [
                        "LogGroup",
                        "ServiceName",
                        "ServiceType",
                        "EXAMPLE_KEY"
                    ]
                ]
            }
        ]
    },
    "function_request_id": "....",
    "Expected": 1,
    "xray_trace_id": "...",
    "EXAMPLE_KEY": "EXAMPLE_VALUE",
    "LogGroup": "aws-stack-LambdaFunction-Q7sUiq8ZLlT1",
    "ServiceName": "aws-stack-LambdaFunction-Q7sUiq8ZLlT1",
    "ServiceType": "AWS::Lambda::Function"
}

Powertools for AWS Lambda (Java) version

latest

AWS Lambda function runtime

Java 21

Debugging logs

[phipag]

Hey @surajkumar,

thanks for reporting this. It is actually expected that a flush does not automatically reset the dimensions. The reason for this is because a user can decide to flush the metrics at any time during the invocation. Hence, resetting dimensions cannot be bound to flushing since flushing does not necessarily happen on a per invocation basis.

Regarding your issue: I see your use-cases and am thinking of a way to address your problem. Can you try to update your handler code with this? (Add finally block clearing dimensions)

    @Override
    @Logging(correlationIdPath = CorrelationIdPaths.API_GATEWAY_REST, clearState = true)
    @FlushMetrics(namespace = "test-aws-stack", captureColdStart = true)
    public APIGatewayProxyResponseEvent handleRequest(APIGatewayProxyRequestEvent input, Context context) {
        LOGGER.info("Entry LambdaHandler#handleRequest");

        try {
            if (shouldEmitMetric) {
                Map<String, String> dimensions = Map.of("EXAMPLE_KEY", "EXAMPLE_VALUE");
                DimensionSet dimensionSet = new DimensionSet();
                dimensions.forEach(dimensionSet::addDimension);
                METRICS.addDimension(dimensionSet);
                METRICS.addMetric("SERL", 1d);

                shouldEmitMetric = false;
                LOGGER.info("Emitted example `SERL` metric and EXAMPLE dimensions");
            } else {
                LOGGER.info("Metric has been emitted in a previous invocation so will not emit this time");
            }

            METRICS.addMetric("Expected", 1d);

        } catch (Exception e) {
            LOGGER.error("An error occurred", e);
        } finally {
            METRICS.clearDefaultDimensions();
        }

        LOGGER.info("Finished!");

        return response();
    }

Does this solve your problem?

[surajkumar]

Hey @phipag, this does solve the issue. Just strange that since upgrading to v2 this is happening as I believe it wasn't the case during 1.x. I can suffice with this work around though! Thanks for the support.

If automatic clearing of dimensions isn't a thing then happy for this issue to be closed.

[phipag]

This is a good point, in v1 we actually refresh the metrics context which clears the dimensions automatically (

powertools-lambda-java/powertools-metrics/src/main/java/software/amazon/lambda/powertools/metrics/internal/LambdaMetricsAspect.java

Line 106 in fb9d437

refreshMetricsContext(metrics);

).

Let me get a second opinion on this since I am unsure how the other Powertools runtimes handle this. Potentially, we should re-add the auto-clearing to Java. My concern is with manual flushing though because this would also clear the dimensions if we auto-enable this feature.

@aws-powertools/lambda-python-core @aws-powertools/lambda-typescript-core @aws-powertools/lambda-dotnet-core How do you handle this? Are you auto-resetting dimensions on every flush?

[svozza]

We clear dimensions after a flush:

  public publishStoredMetrics(): void {
    const hasMetrics = this.hasStoredMetrics();
    if (!this.shouldThrowOnEmptyMetrics && !hasMetrics) {
      this.#logger.warn(
        'No application metrics to publish. The cold-start metric may be published if enabled. ' +
          'If application metrics should never be empty, consider using `throwOnEmptyMetrics`'
      );
    }

    /* v8 ignore else -- @preserve */
    if (!this.disabled) {
      const emfOutput = this.serializeMetrics();
      hasMetrics && this.console.log(JSON.stringify(emfOutput));
    }

    this.clearMetrics();
    this.clearDimensions();
    this.clearMetadata();
  }

[surajkumar]

Bit of an edge case but even adding the clearing of dimensions after a flush is added, my concern would then be:

flushing does not necessarily happen on a per invocation basis

This might still cause the issue to bubble.

[phipag]

Yes, I agree. Potentially, we should make this an opt-in feature rather than always clearing when flushing. What do you think @surajkumar, @svozza ?

Something like

@FlushMetrics(clearDimensions = true)