Post 1.0 Brainstorming

[austinlparker]

We made it to 1.0! 🎉 Let's not rest on our laurels, however -- there's still a lot of work to do. What work should we do, though?

This thread is intended to collect ideas ("big rocks") on feature enhancements, new functionality, and other items that the SIG should work on towards our next major release.

[austinlparker]

In general, here's a few things I think we need to implement:

Metrics Ubitquity
We need to create, collect, and display metrics from each service, runtime, and infrastructure method. This includes not only custom metrics for 'business logic', but also golden signals for API endpoints.

Log Collection and Schemas
OTLP Logging is stable, and while Log APIs and SDKs are still in-flight, we can set ourselves up by adding in log collection and forwarding through the collector. In addition, we should ensure our logs are structured and follow, generally, best practices.

Configurable Sampling/Observability Strategies
Currently, we're very proscriptive about 'how' the demo works from an observability POV. We can add different scenarios to demonstrate different options; For example, turning on trace sampling while having metrics provide golden signals vs. deriving those signals from spans. We could also introduce different sampling strategies, tail sampling, etc.

Benchmarks
A pretty frequent question I see is "what's the overhead of instrumentation" -- the demo can help answer that. We should figure out a way to create benchmark suites that demonstrate performance of the system at various levels of instrumentation and publish those results.

Native Clients
We should be able to add native clients (iOS/MacOS, Android, Windows) in some fashion to demonstrate integrations there. These don't necessarily have to be just the existing Web UI, we could create some kind of separate workflows for them (for example, something that mostly talks through the API to check stock levels, like an inventory application, or a POS application)

[joshleecreates]

I think it would be good to look into OpenFeature for feature flag stuff.

I'd like to see an example of propagating trace context through something like OTP messages in Erlang/Elixir.

[GaryPWhite]

It'd be interesting to have the demo be a bit more interactive. right now it feels a bit like a fish tank. Cool to look at, and lots of great examples. It might add even more interest to implement into the front-end in some admin tab buttons to "kill". Services would have endpoints to bring down apps/pods. Then it would be easy to trace an event that crashes an app -- and see it as a user in an example. Eventually being able to show failovers / multiple instances and stuff could come from that -- a fully comprehensive ops-oriented sample alongside OTel samples.

[austinlparker]

Would it make more sense to do this 'manually', or to use a chaos framework (i.e., https://github.com/chaos-mesh/chaos-mesh) for it?

[GaryPWhite]

that seems like a great approach! Plus we get to play with more CNCF stuff :)

[cartersocha]

Chaos engineering would be really interesting. I agree it's a bit sparse / empty feeling rn

[henrikrexed]

hi i have already started to create experiments for the OpenTelemtry Demo project. I have decided to use ChaosMesh ( a bit lighter), but litmus would be more powerful.

[julianocosta89]

Just for reference: https://github.com/litmuschaos/litmus

[dineshg13]

There is disconnect in the end user where & how each feature of opentelemetry can be used. Currently demo is good starter for basic features of opentelemetry. It is rather ambitious to have all the capabilities of opentelemetry fit into one application. We should aim for multiple smaller applications, each showcasing a good use case for opentelemetry may be a unique feature.

As an example, let's talk about span links. For a good use case of spanlinks, we need some fan-in or multiple requests triggering a target. We can add a simple application that showcases spanlinks capabilities , and how and where it would be helpful.

There are a few such cases that would benefit from small demos :

• Span Links
• Integrating traditional loggers to opentelemetry. How would trace / log correlation work for these ?
• Histograms/Exponential histograms /exemplars in metrics