[examples] Add manual activities and custom metrics to ASP.NET Core example #4133

danelson · 2023-02-01T14:13:50Z

Fixes #4107.

Changes

Please provide a brief description of the changes here.

Add ActivitySource and add manual activity
Add Meter and add custom metric (counter)

For significant contributions please make sure you have completed the following items:

Appropriate CHANGELOG.md updated for non-trivial changes
Design discussion issue #
Changes in public API reviewed

…xample

linux-foundation-easycla · 2023-02-01T14:13:53Z

The committers listed above are authorized under a signed CLA.

✅ login: dnelson-relativity / name: Dan Nelson (232ab2b, 62779b2, 8da9a2d, 6d791de)

danelson · 2023-02-01T15:08:00Z

Sorry I thought the CLA was already signed. Working with legal for approval

CodeBlanch · 2023-02-01T18:38:05Z

examples/AspNetCore/Controllers/WeatherForecastController.cs


-    public WeatherForecastController(ILogger<WeatherForecastController> logger)
+    public WeatherForecastController(ILogger<WeatherForecastController> logger, ActivitySource activitySource, Meter meter)


@dnelson-relativity In the current shape of the .NET API I think it is an anti-pattern to use DI for ActivitySource & Meter. They are really intended to be used statically. If you really want to inject them, might be better to create a telemetry helper class which exposes source + meter for the app logic and can be safely injected.

/cc @noahfalk

Happy to change this to static but I am curious if you could provide more context as to the reason why. I think the DI pattern is a bit more natural for an AspNetCore developer so that is what I chose. Perhaps the examples/docs should speak to that reasoning?

In this case is there something fundamentally different between using it as a singleton and using it statically?

This example is pretty simple so I can just new up the ActivitySource/Meter in WeatherForecastController but I wouldn't want to make people think they should be doing that in every controller. Do you think that is ok or should I create these elsewhere?

In this case is there something fundamentally different between using it as a singleton and using it statically?

The main issue conceptually is it is perfectly acceptable and expected there will many ActivitySources and many Meters in a process so who wins when multiple things call AddSingleton? 😄 Here is what ASP.NET Core does so this code registering its own singleton is probably taking over ASP.NET Core's and causing some very odd telemetry!

This example is pretty simple so I can just new up the ActivitySource/Meter in WeatherForecastController but I wouldn't want to make people think they should be doing that in every controller. Do you think that is ok or should I create these elsewhere?

Probably best to keep it simple and just new them up in the controller. Best practice would probably be build something which plugs into MVC like an action filter (are those still a thing?) but that might be more code than it is worth for an example. It is very hard to show something useful without too much code in these docs/examples! If you are super passionate about it, we could have a more advanced example. Not sure how everyone would feel about that, but I'm not opposed to it.

I see what you mean. If I update the TracerProvider configuration to the following and disable the auto instrumentation

builder .AddSource("Microsoft.AspNetCore", "Examples.AspNetCore") .SetSampler(new AlwaysOnSampler()); //.AddAspNetCoreInstrumentation() -> Disabled

I get this unexpected result

Interestingly this doesn't happen with only the auto instrumentation which is why I never encountered this before.

I updated the example to use statics. On the fence about the "Examples.AspNetCore" magic string but it removes the need for some global constant or passing of additional configuration to the controller. Part of me thinks it is more explicit with the duplication.

I don't know that it is written down anywhere, but we had past discussions with @davidfowl on this topic and agreed that:

It was a mistake for ASP.NET to directly inject an instance of ActivitySource, but given back-compat concerns for now we are keeping the mistake as-is rather than changing it.

In general developers shouldn't add types to a DI container if their code doesn't define the type. If you want to access an instance of some common type, a better pattern is to define a type that will contain it and inject the containing type. For example instead of injecting ActivitySource, define class MyLibraryTelemetry { ActivitySource Source; } and inject MyLibraryTelemetry instead. This avoids the type collisions with other components that might also like to locate an ActivitySource via the DI container.

examples/AspNetCore/Controllers/WeatherForecastController.cs

davidfowl · 2023-02-01T23:59:18Z

examples/AspNetCore/Controllers/WeatherForecastController.cs

+    private static readonly ActivitySource MyActivitySource = new ActivitySource("Examples.AspNetCore");
+    private static readonly Meter MyMeter = new Meter("Examples.AspNetCore");


Why is anything static? I think we need a DI centric approach to counters here. This doesn't look like something we'd tell people to do.

Related threads:

[examples] Add manual activities and custom metrics to ASP.NET Core example #4133 (comment)

[examples] Add manual activities and custom metrics to ASP.NET Core example #4133 (comment)

Injecting the ActivitySource or Meter is bad but injecting a singleton service that has the instance of those types is good (that's what we should do here). With DI the type is the key so global singletons that the user doesn't own is a bad idea and easy to stomp on.

@davidfowl looks like @noahfalk captured it here #4133 (comment).

@CodeBlanch also hinted at this in his original comment. I am curious where you think the responsibility lies for creating the actual Instrument (in this case a Counter)

Short answer - If the Meter is shared via a DI container you probably want the Instruments also shared via the DI container. Most likely this means the code that created the Meter is creating the instruments.

Long answer - Meters hold a reference to all Instruments they create, so once created Instruments live as long as their parent Meter does. Assuming the Meter object has a long lifetime that will be reused for an unbounded number of requests, creating new instances of Instrument objects per-request would leak memory so we don't want to do that. The alternatives that don't leak require all code using the Instrument objects to be sharing the same bounded set of them. This means either that the instruments are singletons in the container or they are stored in fields of some other DI singleton, or some other mechanism is employed to share them such as a static cache.

More broadly folks who have looked into using Meters in DI have found them lacking. I would argue none of the requests I have seen represent an inability for Meters to be used with a DI container, but people's expectations change and they expect more functionality or alternate patterns in addition to just putting the objects in DI. I think the best pattern you can do today is to put the Meter + all the Instruments in a container type and let DI inject it. As we look into some of the requests folks are making I expect some new APIs will show up and best practice will shift.

I would recommend declaring singleton service that holds meter and activity source instances and flowing that to sample code.

@noahfalk @davidfowl thank you for the feedback. I pushed an update that I hope aligns with your desires. I named the type TelemetryService but maybe it should be TelemetryContainer, TelemetryWrapper, etc.

reyang · 2023-02-02T18:19:55Z

examples/AspNetCore/TelemetryService.cs

+
+    public TelemetryService(string version)
+    {
+        this.ActivitySource = new ActivitySource(InstrumentationScopeName, version);


Couple things for discussion:

Where do we dispose some of these properties?

Do we want to make sure that at maximum 1 instance can exist? (rather than assuming that folks who use DI pattern would ensure it is singleton)?

A few comments regarding your first question

TelemetryService or ITelemetryService (if the is registered like appBuilder.Services.AddSingleton<ITelemetryService> does the interface have to implement IDisposable?) could implement IDisposable and I believe it will be handled on application shutdown. Is there a concern regarding order of disposal between ActivitySource and TracerProvider or Meter and MeterProvider?

The examples that use statics don't dispose of ActivitySource and Meter. Is this a concern?

https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/examples/MicroserviceExample/Utils/Messaging/MessageReceiver.cs#L29

opentelemetry-dotnet/docs/metrics/getting-started/Program.cs

Lines 25 to 26 in c0f927f

private static readonly Meter MyMeter = new("MyCompany.MyProduct.MyLibrary", "1.0");

private static readonly Counter<long> MyFruitCounter = MyMeter.CreateCounter<long>("MyFruitCounter");

opentelemetry-dotnet/examples/Console/TestPrometheusExporter.cs

Lines 27 to 28 in c0f927f

private static readonly Meter MyMeter = new("MyMeter");

private static readonly Meter MyMeter2 = new("MyMeter2");

I'd recommend TelemetryService implement IDisposable (it is not necessary for ITelemetryService to extend IDisposable if we still have the interface at all).

I don't think there is any concern on the relative disposal order, any order should work.

I don't think we should try to enforce that only one instance of this object exists. For a developer unit testing their own telemetry generation it may be quite useful to have multiple instances existing even if OTel wouldn't support transmitting the telemetry sent from those multiple instances. Longer term we probably want OTel to handle situations where an app may have multiple identically named Meters existing in the process within different DI containers. These Meters might be serviced by different instances of the OTel telemetry pipeline per container, or they might be tagged in some way to allow them to be consilidated in a single pipeline.

The reason the statics don't worry about disposal is they assume the lifetime of the Meter is the same as the lifetime of the process and the OS will release all resource on process exit. However in a DI container there is a presumption that the container might be shorter lived than the process (for example in a unit test).

Disposal of ActivitySource and Meter has been added.

examples/AspNetCore/Program.cs

Co-authored-by: David Fowler <[email protected]>

noahfalk · 2023-02-03T00:23:04Z

examples/AspNetCore/ITelemetryService.cs

+using System.Diagnostics;
+using System.Diagnostics.Metrics;
+
+public interface ITelemetryService


@davidfowl may disagree, but creating an interface feels like unnecessary ceremony to me. Presumably this is an internal API boundary between two different components in the same library so I would not expect any alternate implementations to exist. I think the other usual argument is that someone might want to mock the interface for testing, but I doubt they could produce a useful mock unless the interface abstracted them from all the concrete types. If we need to delve into it I have some thoughts on how this can be tested without mocking.

If we do wind up keeping the interface I'd suggest not making it public.

We don’t need an interface, just an implementation would suffice. Though, I think naturally people would expect to see an interface (even though there's nothing really mockable here).

noahfalk · 2023-02-03T00:28:39Z

examples/AspNetCore/TelemetryService.cs

+using System.Diagnostics;
+using System.Diagnostics.Metrics;
+
+public class TelemetryService : ITelemetryService


I'd recommend not appending 'Service' on this type name. All types that get added to a DI container can be refered to as services, but it is not usually added to the type name.

Naming nit: how about 'Instrumentation' instead of 'Telemetry'?

noahfalk · 2023-02-03T01:14:12Z

examples/AspNetCore/TelemetryService.cs

+{
+    private const string InstrumentationScopeName = "Examples.AspNetCore";
+
+    public TelemetryService(string version)


I'd recommend not having a version parameter here. Its not clear to me we are gaining any value by parameterizing it. We can inline typeof(TelemetryService).Assembly.GetName().Version?.ToString() below.

I parameterized it so that it was guaranteed to match the version defined as part of the provider's resource. Not that I would necessarily expect either of these to change.

Slightly related but I dislike how the original code was setting "unknown" in the case of null since it is an optional parameter and already handled nicely.

noahfalk · 2023-02-03T01:42:52Z

examples/AspNetCore/TelemetryService.cs

+
+    public TelemetryService(string version)
+    {
+        this.ActivitySource = new ActivitySource(InstrumentationScopeName, version);


I'd recommend TelemetryService implement IDisposable (it is not necessary for ITelemetryService to extend IDisposable if we still have the interface at all).

I don't think there is any concern on the relative disposal order, any order should work.

I don't think we should try to enforce that only one instance of this object exists. For a developer unit testing their own telemetry generation it may be quite useful to have multiple instances existing even if OTel wouldn't support transmitting the telemetry sent from those multiple instances. Longer term we probably want OTel to handle situations where an app may have multiple identically named Meters existing in the process within different DI containers. These Meters might be serviced by different instances of the OTel telemetry pipeline per container, or they might be tagged in some way to allow them to be consilidated in a single pipeline.

The reason the statics don't worry about disposal is they assume the lifetime of the Meter is the same as the lifetime of the process and the OS will release all resource on process exit. However in a DI container there is a presumption that the container might be shorter lived than the process (for example in a unit test).

danelson · 2023-02-06T20:45:55Z

examples/AspNetCore/Instrumentation.cs

+
+public class Instrumentation : IDisposable
+{
+    internal const string ScopeName = "Examples.AspNetCore";


Exposing this since the ActivitySource and Meter registration is coupled to their names.

I'd suggest to rename "ScopeName" to something else.
Scope is used in many context, so it could confuse new users.
https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#get-a-tracer

Do you have a suggestion? In my opinion the instrumentation scope is the entity to which this data is associated. It also matches the underlying proto (granted most users probably won't know that)

okay to use ActivitySourceName, MeterName.
AddSource, AddMeter usage is shown later, which looks easier to follow with AddMeter(instrumentaion.MeterNamer), AddActivitySource(instrument.ActivitySourceName)

danelson · 2023-02-06T20:46:35Z

examples/AspNetCore/Instrumentation.cs

+
+    public ActivitySource ActivitySource { get; }
+
+    public Meter Meter { get; }


If we expect all Instruments to be created ahead of time is there any value with exposing Meter?

Nope, not really. You could remove it or make it private.

I made it private. I could also see one making the argument that it should be static now 😂.

Don't make it static :) Many devs who like DI patterns will immediately be worried if they see it.

codecov · 2023-02-06T23:50:30Z

Codecov Report

Merging #4133 (4352fc1) into main (da938d7) will decrease coverage by 0.01%.
The diff coverage is n/a.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    open-telemetry/opentelemetry-dotnet#4133      +/-   ##
==========================================
- Coverage   85.59%   85.58%   -0.01%     
==========================================
  Files         293      293              
  Lines       11371    11371              
==========================================
- Hits         9733     9732       -1     
- Misses       1638     1639       +1

Impacted Files	Coverage Δ
...ZPages/Implementation/ZPagesExporterEventSource.cs	`56.25% <0.00%> (-6.25%)`	⬇️
src/OpenTelemetry/BatchExportProcessor.cs	`82.24% <0.00%> (-1.87%)`	⬇️
...Telemetry/Internal/SelfDiagnosticsEventListener.cs	`97.65% <0.00%> (+0.78%)`	⬆️
...ter.ZPages/Implementation/ZPagesActivityTracker.cs	`100.00% <0.00%> (+2.85%)`	⬆️

cijothomas · 2023-02-07T15:42:02Z

examples/AspNetCore/Controllers/WeatherForecastController.cs

+
+        // Manually create an activity. This will become a child of
+        // the incoming request.
+        using var activity = this.instrumentation.ActivitySource.StartActivity("calculate forecast");


could we store the activitysource, and counter instead? so that this.Activitysource.StartActivity, this.FreezingDaysCounter style code can be used.

Do you have a preference on the null checks in the constructor?

I am just checking for Instrumentation

oh I see I just made the opposite advice ;p @cijothomas what do you like about the multiple fields?

examples/AspNetCore/Instrumentation.cs

examples/AspNetCore/Program.cs

examples/AspNetCore/Controllers/WeatherForecastController.cs

Co-authored-by: Cijo Thomas <[email protected]>

examples/AspNetCore/Instrumentation.cs

cijothomas

LGTM!
Thanks for doing this and congrats on 1st PR :)

examples/AspNetCore/Controllers/WeatherForecastController.cs

examples/AspNetCore/Instrumentation.cs

reyang · 2023-02-07T20:12:56Z

examples/AspNetCore/Program.cs

@@ -14,7 +14,7 @@
 // limitations under the License.
 // </copyright>

-using System.Reflection;
+using Examples.AspNetCore;


"using" or "namespace"?

I think using is correct. This allows me to reference Instrumentation here. As far as I know you cannot define a namespace with these top level statements

reyang

LGTM with minor suggestions (nothing blocking).

noahfalk

Thanks @dnelson-relativity! Its looking a lot better and sorry to keep nit-picking. This is the first sample I've seen in any official documentation that suggests best-practices for using these APIs from an ASP.NET Core app so its getting above average scrutiny.

noahfalk · 2023-02-07T20:14:45Z

examples/AspNetCore/Controllers/WeatherForecastController.cs

    {
        this.logger = logger ?? throw new ArgumentNullException(nameof(logger));
+
+        ArgumentNullException.ThrowIfNull(instrumentation);
+        this.activitySource = instrumentation.ActivitySource;


Nit: How about we store the Instrumentation object directly in a single field instead of one field per source and counter. It doesn't make a big difference for this sample, but for folks who have many counters it avoids adding lots of fields to their type if they apply the same pattern.

I suggested this pattern to simply code elsewhere, but I can see merits to your suggestion too.

this.instrumentation.activitySource.StartActivity, this.instrumentation.counter.Add
vs
this.activitySource.StartActivity, this.counter.Add

Considering this as non-blocking, so proceeding to merge.

Are you asking to revert this suggestion? I will let you and @cijothomas decide.

noahfalk · 2023-02-07T20:33:09Z

examples/AspNetCore/Controllers/WeatherForecastController.cs

@@ -54,6 +74,9 @@ public IEnumerable<WeatherForecast> Get()
        })
        .ToArray();

+        // Optional: Count the freezing days
+        this.freezingDaysCounter.Add(forecast.Count(f => f.TemperatureC < 0));


People might be confused if they think about the meaning of this counter. Presumably if a web service is issuing weather reports it is probably giving reports for the same days over and over which means this counter will have a value something like num_requested_reports * num_freezing_days_per_report. How about a counter that tracks the number of forecasts instead? That value both seems more plausibly useful and makes more sense to use a Counter to track it.

Good idea. We can add forecast as a dimension.
Lets address in a follow up.

I was trying to find a "real" enough use case but didn't want to change the logic of the example. Would you define a forecast here as single invocation of this endpoint? In that case this information is the same as number of requests which one can get from the histogram created by AddAspNetCoreInstrumentation.

Yeah, each invocation of this method would be Add(1) in my suggestion. I agree it is duplicative most of the time (they might not have those other metrics enabled) but that still seemed like an improvement. If we wanted something that is both reasonable and unique we'd probably need to change the sample at least a little. For example the forecast service could have a cache and the counter counts how many requests weren't found in the cache. Another option could be forecast requests are for a variable number of days and the counter counts how many 10 day forecasts were requested (maybe 10 day forecasts are more expensive to produce than shorter forecasts?).

I tried capturing this in a new issue (https://github.com/open-telemetry/opentelemetry-dotnet/issues/4166) so this does not get lost. If you feel extra clarity is needed please add it to the issue.

cijothomas · 2023-02-07T20:45:01Z

@dnelson-relativity Once last red CI check (sanitycheck/encoding one), and once its done, we can merge

danelson · 2023-02-07T20:56:13Z

@dnelson-relativity Once last red CI check (sanitycheck/encoding one), and once its done, we can merge

@cijothomas I think I fixed it. My Visual Studio keeps using CRLF for some reason. I know line endings are a contentious topic but is this something that should be added to the .editorconfig? Perhaps some other local weirdness is messing it up.

cijothomas · 2023-02-07T21:45:39Z

minor non-blocking comments could be a follow up. Merging to get this example ready by 1.4 release :)

cartermp · 2023-02-21T17:38:16Z

Is this example considered to be canonical?

If so, it'd be great to copy things over to the official OTel docs and demo app:

open-telemetry/opentelemetry.io#2024
https://github.com/open-telemetry/opentelemetry-demo/blob/main/src/cartservice/src/Program.cs

Otherwise most people who instrument with otel-dotnet may not be following best practices.

cijothomas · 2023-02-21T18:02:49Z

Yes. We expect all asp.net core examples to move to this model for manual span/metrics.

[examples] Add manual activities and custom metrics to ASP.NET Core e…

232ab2b

…xample

danelson requested a review from a team February 1, 2023 14:13

CodeBlanch reviewed Feb 1, 2023

View reviewed changes

danelson added 2 commits February 1, 2023 14:51

Use statics for custom ActivitySource and Meter

62779b2

Remove unused using statements

8da9a2d

reyang reviewed Feb 1, 2023

View reviewed changes

examples/AspNetCore/Controllers/WeatherForecastController.cs Outdated Show resolved Hide resolved

davidfowl reviewed Feb 1, 2023

View reviewed changes

Expose ActivitySource, Meters, and Instruments via a TelemetryService

6d791de

reyang reviewed Feb 2, 2023

View reviewed changes

davidfowl reviewed Feb 2, 2023

View reviewed changes

examples/AspNetCore/Program.cs Outdated Show resolved Hide resolved

More efficient assembly access

8973699

Co-authored-by: David Fowler <[email protected]>

noahfalk reviewed Feb 3, 2023

View reviewed changes

Remove interface, rename instrumentation container and dispose

3961147

danelson commented Feb 6, 2023

View reviewed changes

Make Meter private

5d71484

cijothomas reviewed Feb 7, 2023

View reviewed changes

examples/AspNetCore/Instrumentation.cs Show resolved Hide resolved

cijothomas reviewed Feb 7, 2023

View reviewed changes

examples/AspNetCore/Instrumentation.cs Outdated Show resolved Hide resolved

cijothomas reviewed Feb 7, 2023

View reviewed changes

examples/AspNetCore/Program.cs Outdated Show resolved Hide resolved

cijothomas reviewed Feb 7, 2023

View reviewed changes

examples/AspNetCore/Controllers/WeatherForecastController.cs Outdated Show resolved Hide resolved

danelson and others added 4 commits February 7, 2023 10:00

Use long instead of int for counter

660c721

Co-authored-by: Cijo Thomas <[email protected]>

Clarify comments

c013789

Co-authored-by: Cijo Thomas <[email protected]>

Clarify names for ActivitySource and Meter

4f7668d

Rework ActivitySource and Instrument usage

813df21

danelson force-pushed the manual-instrumentation-aspnetcore-example branch from eb9e94e to 813df21 Compare February 7, 2023 16:36

cijothomas reviewed Feb 7, 2023

View reviewed changes

examples/AspNetCore/Instrumentation.cs Show resolved Hide resolved

danelson added 2 commits February 7, 2023 12:39

Clarify comments for manual instrumentation

ce8c48c

Clarify purpose of Instrumentation class

dc516aa

cijothomas approved these changes Feb 7, 2023

View reviewed changes

Merge branch 'main' into manual-instrumentation-aspnetcore-example

d241178

reyang reviewed Feb 7, 2023

View reviewed changes

examples/AspNetCore/Controllers/WeatherForecastController.cs Show resolved Hide resolved

reyang reviewed Feb 7, 2023

View reviewed changes

examples/AspNetCore/Instrumentation.cs Show resolved Hide resolved

reyang reviewed Feb 7, 2023

View reviewed changes

reyang approved these changes Feb 7, 2023

View reviewed changes

noahfalk reviewed Feb 7, 2023

View reviewed changes

Fix line endings

4352fc1

cijothomas merged commit 1273e83 into open-telemetry:main Feb 7, 2023

danelson mentioned this pull request Feb 8, 2023

Update custom metric in AspNetCore example open-telemetry/opentelemetry-dotnet-contrib#1769

Open


		public WeatherForecastController(ILogger<WeatherForecastController> logger)
		public WeatherForecastController(ILogger<WeatherForecastController> logger, ActivitySource activitySource, Meter meter)

		private static readonly ActivitySource MyActivitySource = new ActivitySource("Examples.AspNetCore");
		private static readonly Meter MyMeter = new Meter("Examples.AspNetCore");

	private static readonly Meter MyMeter = new("MyCompany.MyProduct.MyLibrary", "1.0");
	private static readonly Counter<long> MyFruitCounter = MyMeter.CreateCounter<long>("MyFruitCounter");

	private static readonly Meter MyMeter = new("MyMeter");
	private static readonly Meter MyMeter2 = new("MyMeter2");


		public ActivitySource ActivitySource { get; }

		public Meter Meter { get; }

[examples] Add manual activities and custom metrics to ASP.NET Core example #4133

[examples] Add manual activities and custom metrics to ASP.NET Core example #4133

Conversation

danelson commented Feb 1, 2023

Changes

linux-foundation-easycla bot commented Feb 1, 2023 • edited Loading

danelson commented Feb 1, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danelson Feb 1, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidfowl Feb 2, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danelson Feb 2, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidfowl Feb 3, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danelson Feb 7, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Feb 6, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

noahfalk Feb 7, 2023 • edited Loading

Choose a reason for hiding this comment

cijothomas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reyang left a comment

Choose a reason for hiding this comment

noahfalk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cijothomas commented Feb 7, 2023

danelson commented Feb 7, 2023

cijothomas commented Feb 7, 2023

cartermp commented Feb 21, 2023

cijothomas commented Feb 21, 2023

linux-foundation-easycla bot commented Feb 1, 2023 •

edited

Loading

danelson Feb 1, 2023 •

edited

Loading

davidfowl Feb 2, 2023 •

edited

Loading

danelson Feb 2, 2023 •

edited

Loading

davidfowl Feb 3, 2023 •

edited

Loading

danelson Feb 7, 2023 •

edited

Loading

codecov bot commented Feb 6, 2023 •

edited

Loading

noahfalk Feb 7, 2023 •

edited

Loading