Distributed Tracing for Entities

[sophiatev]

This PR adds additional information passing to enable distributed tracing for entities.

• DurableOrchestrationClient.signal_entity was modified to pass the current trace context in its HTTP headers (similar to how is done in the start_new call) such that the trace for signaling the entity created in WebJobs can correctly be linked as a child of the client trace that triggered the signal entity request.
• The start time for processing an entity invocation is also passed in the OperationResult such that the traces can accurately capture the time it takes for the request to be completed (using the OperationResult.duration field to calculate the end time).

Note that for a client signaling an entity via DurableOrchestrationClient.signal_entity, in order for this method to be able to obtain the current trace context, the user needs to actually start their own trace from their function app. An example is provided in this PR for a client starting an orchestration (the example code would be identical in this case, with client.start_new replaced by client.signal_entity):

Client Signaling an Entity

import azure.functions as func
import azure.durable_functions as df

from azure.monitor.opentelemetry import configure_azure_monitor
from opentelemetry import trace
from opentelemetry.trace import SpanKind
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

configure_azure_monitor()
tracer = trace.get_tracer(__name__)

myApp = df.DFApp(http_auth_level=func.AuthLevel.ANONYMOUS)

@myApp.route(route="entity_add_value")
@myApp.durable_client_input(client_name="client")
async def http_add(req: func.HttpRequest, client, context: func.Context):
    # Extract the traceparent and tracestate from the context
    parent = {
        "traceparent": context.trace_context.trace_parent,
        "tracestate": context.trace_context.trace_state
    }
    parent_context = TraceContextTextMapPropagator().extract(parent)

    # Create a new span with the extracted parent context
    with tracer.start_as_current_span(
            "myCustomSpan",
            context=parent_context,
            kind=SpanKind.CLIENT) as span:
        
        span.set_attribute("az.namespace", "app")

        entityId = df.EntityId("Counter", "myCounter")
        await client.signal_entity(entityId, "add", 1)
        response = func.HttpResponse("Done", status_code=200)
    return response

Which leads to the following trace

Orchestartion Calling/Signaling an Entity

Whereas starting the following orchestration

@myApp.route(route="orchestrator")
@myApp.durable_client_input(client_name="client")
async def http_start(req: func.HttpRequest, client, context: func.Context):
    # Extract the traceparent and tracestate from the context
    parent = {
        "traceparent": context.trace_context.trace_parent,
        "tracestate": context.trace_context.trace_state
    }
    parent_context = TraceContextTextMapPropagator().extract(parent)

    # Create a new span with the extracted parent context
    with tracer.start_as_current_span(
            "myCustomSpan",
            context=parent_context,
            kind=SpanKind.CLIENT) as span:
        
        span.set_attribute("az.namespace", "app")

        instance_id = await client.start_new('orchestrator')
        response = client.create_check_status_response(req, instance_id)
        return response

@myApp.orchestration_trigger(context_name="context")
def orchestrator(context: df.DurableOrchestrationContext):
    entityId = df.EntityId("Counter", "myCounter")
    context.signal_entity(entityId, "add", 3)
    state = yield context.call_entity(entityId, "get")
    return state

leads to the following trace

[bachuv]

Can you add a screenshot of a trace from Application Insights showing that a signal_entity call from a Python function creates the right trace?

Also, can you fix the linter issues that are showing up in the CI? Please reach out if you need help with this.

[andystaples]

lgtm