Skip to content

An adapter between slipstream telemetry and honeycomb events

License

Notifications You must be signed in to change notification settings

NFIBrokerage/slipstream_honeycomb

Folders and files

NameName
Last commit message
Last commit date

Latest commit

53cc399 · Mar 18, 2024

History

20 Commits
Mar 18, 2024
Oct 19, 2023
Feb 4, 2021
Oct 19, 2023
Oct 19, 2023
Feb 3, 2021
Feb 3, 2021
Feb 3, 2021
May 11, 2022
Feb 3, 2021
Feb 4, 2021
Feb 3, 2021
May 11, 2022
Feb 3, 2021
May 11, 2022
May 11, 2022

Repository files navigation

SlipstreamHoneycomb

Actions CI

An adapter between slipstream telemetry and honeycomb events

Honeycomb

Honeycomb is an observability tool with a sweet query interface and distributed tracing support.

At NFIBrokerage, we ❤️ honeycomb because it enables us to debug issues in fractions of time (and with far more certainty) than log ingestion services, and lets us understand our system in real-time.

For example, it was gloriously fast to debug this issue: NFIBrokerage/slipstream#12 because of the telemetry support added in 0.3 and this exporter library.

Connection Telemetry

SlipstreamHoneycomb currently captures the only telemetry which Slipstream emits: connection messages.

These connection messages are fairly low-level and are very quick (usually a matter of microseconds). Below is an example screenshot of a trace in our Honeycomb:

example

The long purple bar is the connection, which was open for ~150s in this case. That came from the [:slipstream, :connection, :connect, :stop] event. The other spans of this connection are [:slipstream, :connection, :handle, :stop]. From top to bottom, we see from the example:

  • the initial handle_continue(:connect, ..
  • the gun connection coming up ({:gun_up, conn, :http})
  • the gun upgrade which is decoded into the %Slipstream.Events.ChannelConnected{} event
  • a JoinTopic command from the client
  • a :gun_ws message that decoded into a TopicJoinSucceeded event
  • a big push of data from the remote server (decoded to MessageReceived event)

and then much later (~30s), we see:

  • a SendHeartbeat command, followed by
  • the HeartbeatAcknowledged

(x4). And finally, the parent process exits with a ParentProcessExited event.

Installation

def deps do
  [
    {:slipstream_honeycomb, "~> 0.1"}
  ]
end

Check out the docs here: https://hexdocs.pm/slipstream_honeycomb