Spec adding span links for instrumentation of messaging systems (#616)

- Specify that span links should be used on "messaging" spans and
  on "messaging" transactions that cover a batch of messages, when
  messages include trace context.
- Drop special-casing of single-message SQS/SNS Lambda triggers.
- Specify a guard/limit of 1000 messages to check for trace-context.
- Starts documenting message metadata mechanisms to use for propagating
  trace-context, for those systems that provide a facility.

Closes: #606
Co-authored-by: Felix Barnsteiner <[email protected]>

Author: trentm

.gitignore

@@ -0,0 +1 @@
+.DS_Store

specs/agents/tracing-instrumentation-aws-lambda.md

@@ -133,31 +133,31 @@ In version 2.0, the `${event.requestContext.routeKey}` can have the format `GET
 If `use_path_as_transaction_name` is applicable and set to `true`, use `${event.requestContext.http.method} ${event.requestContext.http.path}` as the transaction name.
 
 ### SQS / SNS
-Lambda functions that are triggered by SQS (or SNS) accept an `event` input that may contain one or more SQS / SNS messages in the `event.records` array. All message-related context information (including the `traceparent`) is encoded in the individual message attributes (if at all). We cannot (automatically) wrap the processing of the individual messages that are sent as a batch of messages with a single `event`.
 
-Thus, in case that an SQS / SNS `event` contains **exactly one** SQS / SNS message, the agents must apply the following, messaging-specific retrieval of information. Otherwise, the agents should apply the [Generic Lambda Instrumentation](generic-lambda-instrumentation) as described above.
+Lambda functions that are triggered by SQS (or SNS) accept an `event` input that may contain one or more SQS / SNS messages in the `event.records` array. All message-related context information (including the `traceparent`) is encoded in the individual message attributes (if at all).
 
-With only one message in `event.records`, the agents can use the single SQS / SNS `record` to retrieve the `traceparent` and `tracestate` from message attributes and use it for starting the lambda transaction.
+#### SQS
 
-In addition the following fields should be set for Lambda functions triggered by SQS or SNS:
+Agents SHOULD check each record, [up to a maximum of 1000](tracing-instrumentation-messaging.md#receiving-trace-context),
+for a `traceparent` message attribute, and create a [span link](span-links.md)
+on the transaction for each message with trace-context.
+
+In addition to [the generic Lambda transaction fields](#generic-lambda-instrumentation)
+the following fields SHOULD be set. The use of `records[0]` below depends on the
+understanding from [AWS Lambda SQS docs](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html)
+that a trigger invocation can only include messages from *one* queue.
 
-#### SQS
 Field | Value | Description | Source
 ---   | ---   | ---         | ---
 `type` | `messaging`| Transaction type: constant value for SQS. | -
-`name` | e.g. `RECEIVE SomeQueue` | Transaction name: Follow the [messaging spec](./tracing-instrumentation-messaging.md) for transaction naming. | Simple queue name can be derived from the 6th segment of `record.eventSourceArn`.
+`name` | e.g. `RECEIVE SomeQueue` | Transaction name: Follow the [messaging spec](./tracing-instrumentation-messaging.md) for transaction naming. | Simple queue name can be derived from the 6th segment of `records[0].eventSourceArn`.
 `faas.trigger.type` | `pubsub` | Constant value for message based triggers | -
-`faas.trigger.request_id` | e.g. `someMessageId` | SQS message ID. | `record.messageId`
-`context.service.origin.name` | e.g. `my-queue` | SQS queue name | Simple queue name can be derived from the 6th segment of `record.eventSourceArn`.
-`context.service.origin.id` | e.g. `arn:aws:sqs:us-east-2:123456789012:my-queue` | SQS queue ARN. | `record.eventSourceArn`
+`context.service.origin.name` | e.g. `my-queue` | SQS queue name | Simple queue name can be derived from the 6th segment of `records[0].eventSourceArn`.
+`context.service.origin.id` | e.g. `arn:aws:sqs:us-east-2:123456789012:my-queue` | SQS queue ARN. | `records[0].eventSourceArn`
 `context.cloud.origin.service.name` | `sqs` | Fix value for SQS. | -
-`context.cloud.origin.region` | e.g. `us-east-1` | SQS queue region. | `record.awsRegion`
-`context.cloud.origin.account.id` | e.g. `12345678912` | Account ID of the SQS queue. | Parse account segment (5th) from `record.eventSourceArn`.
+`context.cloud.origin.region` | e.g. `us-east-1` | SQS queue region. | `records[0].awsRegion`
+`context.cloud.origin.account.id` | e.g. `12345678912` | Account ID of the SQS queue. | Parse account segment (5th) from `records[0].eventSourceArn`.
 `context.cloud.origin.provider` | `aws` | Use `aws` as fix value. | -
-`context.message.queue.name` | e.g. `my-queue` | The SQS queue name. | The 6th segment of `record.eventSourceArn`
-`context.message.age.ms` | e.g. `3298` | Age of the message in milliseconds. `current_time` - `SentTimestamp`, if SentTimestamp is available.  | Message attribute with key `SentTimestamp`.
-`context.message.body` | - | The message body. Should only be captured if body capturing is enabled in the configuration. | `record.body`
-`context.message.headers` | - | The message attributes. Should only be captured, if capturing headers is enabled in the configuration. | Use the `stringValue` of entries in `record.messageAttributes` with `dataType == "String" || dataType == "Number"`. [Other attribute types](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-metadata.html#message-attribute-data-types) are ignored.
 
 An example SQS event:
 
@@ -206,23 +206,28 @@ An example SQS event:
 
 #### SNS
 
+Agents SHOULD check each record, [up to a maximum of 1000](tracing-instrumentation-messaging.md#receiving-trace-context),
+for a `traceparent` message attribute (`Records.*.Sns.MessageAttributes`), and
+create a [span link](span-links.md) on the transaction for each message with
+trace-context.
+
+In addition to [the generic Lambda transaction fields](#generic-lambda-instrumentation)
+the following fields should be set. The use of `records[0]` is based on the
+understanding, from ["all notification messages will contain a single published
+message"](https://aws.amazon.com/sns/faqs/#Reliability), that an SNS trigger
+will only ever have a single record.
+
 Field | Value | Description | Source
 ---   | ---   | ---         | ---
 `type` | `messaging`| Transaction type: constant value for SNS. | -
-`name` | e.g. `RECEIVE SomeTopic` | Transaction name: Follow the [messaging spec](./tracing-instrumentation-messaging.md) for transaction naming. | Simple topic name can be derived from the 6th segment of `record.sns.topicArn`.
+`name` | e.g. `RECEIVE SomeTopic` | Transaction name: Follow the [messaging spec](./tracing-instrumentation-messaging.md) for transaction naming. | Simple topic name can be derived from the 6th segment of `records[0].sns.topicArn`.
 `faas.trigger.type` | `pubsub` | Constant value for message based triggers | -
-`faas.trigger.reuqest_id` | e.g. `someMessageId` | SNS message ID. | `record.sns.messageId`
-`context.service.origin.name` | e.g. `my-topic` | SNS topic name | Simple topic name can be derived from the 6th segment of `record.sns.topicArn`.
-`context.service.origin.id` | e.g. `arn:aws:sns:us-east-2:123456789012:my-topic` | SNS topic ARN. | `record.sns.topicArn`
-`context.service.origin.version` | e.g. `2.1` | SNS event version | `record.eventVersion`
+`context.service.origin.name` | e.g. `my-topic` | SNS topic name | Simple topic name can be derived from the 6th segment of `records[0].sns.topicArn`.
+`context.service.origin.id` | e.g. `arn:aws:sns:us-east-2:123456789012:my-topic` | SNS topic ARN. | `records[0].sns.topicArn`
 `context.cloud.origin.service.name` | `sns` | Fix value for SNS. | -
-`context.cloud.origin.region` | e.g. `us-east-1` | SNS topic region. | Parse region segment (4th) from `record.sns.topicArn`.
-`context.cloud.origin.account.id` | e.g. `12345678912` | Account ID of the SNS topic. | Parse account segment (5th) from `record.sns.topicArn`.
+`context.cloud.origin.region` | e.g. `us-east-1` | SNS topic region. | Parse region segment (4th) from `records[0].sns.topicArn`.
+`context.cloud.origin.account.id` | e.g. `12345678912` | Account ID of the SNS topic. | Parse account segment (5th) from `records[0].sns.topicArn`.
 `context.cloud.origin.provider` | `aws` | Use `aws` as fix value. | -
-`context.message.queue.name` | e.g. `my-topic` | The SNS topic name. | The 6th segment of `record.sns.topicArn`
-`context.message.age.ms` | e.g. `3298` | Age of the message in milliseconds. `current_time` - `snsTimestamp`.  | `record.sns.timestamp`
-`context.message.body` | - | The message body. Should only be captured if body capturing is enabled in the configuration. | `record.sns.message`
-`context.message.headers` | - | The message attributes. Should only be captured, if capturing headers is enabled in the configuration. | Use the `Value` of entries in `record.Sns.MessageAttributes` with `Type == "String"`. [Other attribute types](https://docs.aws.amazon.com/sns/latest/dg/sns-message-attributes.html#SNSMessageAttributes.DataTypes) are ignored. |
 
 An example SNS event:
 

specs/agents/tracing-instrumentation-aws.md

@@ -2,6 +2,7 @@
 
 We describe how to instrument some of AWS' services in this document.
 Some of the services can use existing specs. When there are differences or additions, they have been noted below.
+The spec for [instrumenting AWS Lambda](tracing-instrumentation-aws-lambda.md) is in a separate document.
 
 ### S3 (Simple Storage Service)
 

specs/agents/tracing-instrumentation-messaging.md

@@ -15,9 +15,9 @@ occurring within a traced transaction.
 
 ![publish](uml/publish.svg)
 
-### Trace Context
+### Sending Trace Context
 
-if the messaging system exposes an API for sending additional message properties/metadata, it
+If the messaging system exposes a mechanism for sending additional [message metadata](#message-metadata), it
 SHOULD be used to propagate the [Trace Context](https://www.w3.org/TR/trace-context/) of the
 `messaging` span, to continue the [distributed trace](tracing-distributed-tracing.md).
 
@@ -68,19 +68,28 @@ If creating a transaction for the processing of each message in a batch is not p
 the agent SHOULD create a single `messaging` transaction for the processing of the batch
 of messages.
 
-### Trace Context
+### Receiving Trace Context
 
-When message reception is captured as a `messaging` transaction, if the messaging
-system exposes an API for sending additional message properties/metadata, it
-SHOULD be checked for the presence of [Trace Context](https://www.w3.org/TR/trace-context/).
-If Trace Context is present, it SHOULD be propagated to the `messaging` transaction
+This section applies to messaging systems that support [message metadata](#message-metadata).
+The instrumentation of message reception SHOULD check message metadata for the
+presence of [Trace Context](https://www.w3.org/TR/trace-context/).
+
+When single message reception is captured as a `messaging` transaction,
+and a Trace Context is present, it SHOULD be used as the parent of the `messaging` transaction
 to continue the [distributed trace](tracing-distributed-tracing.md).
 
-If a batch of messages is processed in a single `messaging` transaction, it may be
-possible that each message in the batch has its own Trace Context. In this
-scenario, it is not currently possible to propagate a Trace Context to the `messaging`
-transaction, since there a multiple contexts present. It may be possible to capture
-these in future through [span links](https://github.com/elastic/apm/issues/122).
+Otherwise (a single message being captured as a `messaging` span, or a batch
+of messages is processed in a single `messaging` transaction or span), a
+[span link](span-links.md) SHOULD be added for each message with Trace Context.
+This includes the case where the size of the batch of received messages is one.
+
+The number of events processed for trace context SHOULD be limited to a maximum
+of 1000, as a guard on agent overhead for extremely large batches of events.
+(For example, SQS's maximum batch size is 10000 messages. The maximum number of
+span links that could be sent for a single transaction/span to APM server with
+the default configuration is approximately 4000: 307200 bytes
+[APM server `max_event_size` default](https://www.elastic.co/guide/en/apm/server/current/configuration-process.html#max_event_size)
+/ 77 bytes per serialized span link.)
 
 ### Examples
 
@@ -188,6 +197,25 @@ queues/topics/exchanges will be ignored.
 | Central config | `true` |
 
 
+### Message metadata
+
+To support distributed tracing with automatic instrumentation, the messaging
+system must provide a mechanism to add metadata/properties/attributes to
+individual messages, akin to HTTP headers. If an APM agent supports
+trace-context for a given messaging system, it MUST use the following mechanisms
+so that cross-language tracing works:
+
+| Messaging system       | Mechanism |
+| ---------------------- | --------- |
+| Azure Queue            | No mechanism |
+| Azure Service Bus      | Possibly `Diagnostic-Id` [application property](https://docs.microsoft.com/en-us/dotnet/api/azure.messaging.servicebus.servicebusmessage.applicationproperties). See [this doc](https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-end-to-end-tracing). |
+| Java Messaging Service | [Message Properties](https://docs.oracle.com/javaee/7/api/javax/jms/Message.html) |
+| Apache Kafka           | [Kafka Record headers](https://cwiki.apache.org/confluence/display/KAFKA/KIP-82%2B-%2BAdd%2BRecord%2BHeaders) using [binary trace context fields](tracing-distributed-tracing.md#binary-fields) |
+| RabbitMQ               | [Message Attributes](https://www.rabbitmq.com/tutorials/amqp-concepts.html#messages) (a.k.a. `AMQP.BasicProperties` in [Java API](https://www.rabbitmq.com/api-guide.html)) |
+| AWS SQS                | [SQS message attributes](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-metadata.html), if within message attribute limits. See [AWS instrumentation](tracing-instrumentation-aws.md). |
+| AWS SNS                | [SNS message attributes](https://docs.aws.amazon.com/sns/latest/dg/sns-message-attributes.html), if within message attribute limits. See [AWS instrumentation](tracing-instrumentation-aws.md). |
+
+
 ### AWS messaging systems
 
 The instrumentation of [SQS](tracing-instrumentation-aws.md#sqs-simple-queue-service) and [SNS](tracing-instrumentation-aws.md#sns-aws-simple-notification-service) services generally follow this spec, with some nuances specified in the linked specs.