Feature request: Add multiple dimensets to the same Metrics instance

[north-star-saj]

Edit history

03/20/2025: Updated EMF spec and code snippit for accuracy.

Use case

Overview

Enabling the AWS Powertools Python package to support multiple dimension sets for the same Metrics instance would significantly enhance the packages monitoring capabilities. This feature would allow users to gain more granular insights and comprehensive views of their applications by creating aggregating metrics across various dimensions.

Reference Code

This feature request is akin to the aws_embedded_metrics (code link) put_dimensions method which adds a dimension set to a common MetricsContext. The metrics then get serialized into Embedded Metric Format (EMF) with multiple dimension sets (code link).

Example Use Case

This is a simplified example that demonstrates my use case. In my usecase, one of these dimensions's values is not known in advance. Instead, it is dynamically retrieved.

I am monitoring a lambda that gets deployed to two environments (beta and production) across three regions (us-east-1, us-west-1, and eu-west-1). My lambda produces application-specific metrics such as SuccessfulRequests, FailedRequests, and RetryCount. By creating multiple dimension sets across these three dimensions, I can create aggregate metrics which enable a comprehensive view of my application.

The generated EMF log may look something like this:

03/20/2025 edit: fixed incorrect emf spec with repetitive dimensions / keys

{
  "_aws": {
    "Timestamp": 946684800000,
    "CloudWatchMetrics": [
      {
        "Namespace": "MyApplication",
        "Dimensions": [
          [
            "Environment",
            "Region"
          ],
          [
            "Enviornment"
          ],
          [
            "Region"
          ]
        ],
        "Metrics": [
          {
            "Name": "SuccessfulRequests",
            "Unit": "Count"
          },
          {
            "Name": "FailedRequests",
            "Unit": "Count"
          },
          {
            "Name": "RetryCount",
            "Unit": "Count"
          }
        ]
      }
    ]
  },
  "Environment": "Production",
  "Region": "us-west-2",
  "SuccessfulRequests": 20,
  "FailedRequests": 0,
  "RetryCount": 1
}

Benefits

• Improved granularity in monitoring and alerting: Supports better visibility into metrics for environments, regions, and other dynamic factors.
• Easier aggregation of metrics across various dimensions, making it simpler to analyze and report on complex data sets.
• Automation of dimension management reduces manual errors and maintenance overhead.

Solution/User Experience

The following is an example uses a new add_dimension_set method defined in the AmazonCloudWatchEMFProvider class.

Edit on 03/20/2025: added missing add_dimension_set call to code snippit

# logic which would produce the example EMF log from the ticket
@metrics.log_metrics
def lambda_handler(event: dict, context: LambdaContext):
    metrics.add_dimension_set({"environment": STAGE})
    metrics.add_dimension_set({"environment:" STAGE, "region": REGION})
    metrics.add_dimension_set({"region": REGION})
    metrics.add_metric(name="SuccessfulRequests", unit=MetricUnit.Count, value=20)
    metrics.add_metric(name="FailedRequests", unit=MetricUnit.Count, value=0)
    metrics.add_metric(name="RetryCount", unit=MetricUnit.Count, value=1)

Considerations:

From my point of view, these are a few considerations that will need to be accounted for as part of this feature request.

• Does add_dimension add the dimension to all dimension sets? How does it work when invoked before or after the add_dimension_set method?
• Does add_dimension_set handle duplicate dimensions? What about duplicate dimension keys, but differing values?
• Is the new method name clear to existing and future customers (e.g. add_dimension_set) ?

Alternatives

I've considered the following alternatives. Each of these solutions come up short compared to an easy-to-use method in powertools that lets me add multiple dimension set to the same metric value.

Alternative	Benefits	Weaknesses
Publish metrics using aws-embedded-metrics-python	Supports multi-dimension set metrics	Limited to EMF logging; adds unnecessary dependencies like async which is not required in Lambda.
Create multiple instances of EphemeralMetrics for each dimensionset	Enables aggregated metric logging	Creates excess EMF logs, error prone and repetitive
Create custom metric without using dimension sets	Simple to implement	Lacks granularity and flexibility; requires manual aggregation of metrics and fails to provide the same level of dynamic insight across multiple dimensions.
Use CloudWatch Metric Math	Allows mathematical operations across metrics	Complex to set up, especially for a large number of dimensions. As the number of dimensions increases, maintaining and scaling Metric Math queries becomes harder
Use CloudWatch SEARCH functionality	Enables querying and analyzing metrics data	Requires advanced querying skills, which may not be familiar to all users.Additionally, this approach may require additional resources for query optimization, performance tuning, and potentially more API calls to fetch and process large volumes of data.

Acknowledgment

• This feature request meets Powertools for AWS Lambda (Python) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Java, TypeScript, and .NET

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

[leandrodamascena]

Hey @north-star-saj! Thanks so much for opening this issue here! I'm working on finishing up a few items for the release we have on Friday, but I'll give you some feedback here by the end of the week.

[north-star-saj]

Hey @leandrodamascena , just checking in -- is there anything I can clear up in the feature request? I'd be open to providing a proof of concept / creating a PR once we align on the general idea if that helps.

[leandrodamascena]

Hey @leandrodamascena , just checking in -- is there anything I can clear up in the feature request? I'd be open to providing a proof of concept / creating a PR once we align on the general idea if that helps.

Hey hey @north-star-saj! I had some internal stuff to sort out, but I promise I'll give you feedback here tomorrow.

[leandrodamascena]

Hi @north-star-saj! Thank you so much for the detailed explanation you gave me during the meeting we had!

Yes, I think it makes sense to add support for adding multiple dimension sets for the same metric, especially since you can get better visualization in CloudWatch metrics without doing any trick with MATH or SEARCH. Also, DimensionSet in EMF Specification is an array of arrays, which means EMF expects customers to create multiple dimension sets. Some other popular EMF libraries, such as aws-embedded-metrics-python, also support this.

I think the experience could look like this:

from aws_lambda_powertools import Metrics
from aws_lambda_powertools.metrics import MetricUnit

metrics = Metrics(namespace="A")

metrics.add_dimension(name="dimension1", value="1") # This continue working AS IS

metrics.add_dimension_set({"dimension1": "1", "dimension2": "2"}) # This is a new method to add a new array to the Dimensions
metrics.add_dimension_set({"dimension1": "1", "dimension2": "2", "dimension3": "3"}) # Same here
metrics.add_metric(name="mymmmm", unit=MetricUnit.Bytes, value=1 )

metrics.flush_metrics()

And this will produce the following output. Note that dimension1 will be the only value in the first array because it was added by the add_dimension method. Also, add_dimension will not add dimensions to any other array than the first one, which means that add_dimension is intended to be the first dimensions you will have if you use this method.

{
   "_aws":{
      "Timestamp":1742404106448,
      "CloudWatchMetrics":[
         {
            "Namespace":"A",
            "Dimensions":[
               [
                  "dimension1"
               ],
               [
                  "dimension1",
                  "dimension2"
               ],
               [
                  "dimension1",
                  "dimension2",
                  "dimension3"
               ]
            ],
            "Metrics":[
               {
                  "Name":"mymmmm",
                  "Unit":"Bytes"
               }
            ]
         }
      ]
   },
   "dimension1":"1",
   "dimension2":"2",
   "dimension3":"3",
   "mymmmm":[
      1.0
   ]
}

We need to be aware that this may increase the costs for the customer in CloudWatch who uses this feature. But this is something we can make well documented if we go ahead with this implementation.

I would like to hear the opinion of other maintainers. Taggging @aws-powertools/lambda-typescript-core @aws-powertools/lambda-java-core and @aws-powertools/lambda-dotnet-core.

[dreamorosi]

Thanks for flagging this, I'll take a look and share my feedback before end of the week.