Sentry Structured Logging for Python

[AbhiPrasad]

Note: These instructions are now available in the Sentry docs - https://docs.sentry.io/platforms/python/logs/

Sentry is adding support for structured logs, so you can view and query your logs alongside your traces and errors—all in one place.

Note

To get early access to the Sentry logging product and to see how it works, see the announcement post.

To use the new Logging APIs to send application logs directly to Sentry, you'll have to upgrade to 2.25.1 of the Python SDK or higher.

pip install --upgrade sentry-sdk

Logging is gated by an experimental option, _experiments.enable_logs.

sentry_sdk.init(
    dsn=...,
    _experiments={
        "enable_logs": True,
    },
)

Then you can import and use methods from the logger namespace to send logs to Sentry.

from sentry_sdk import logger as sentry_logger

sentry_logger.trace('Starting database connection {database}', database="users");
sentry_logger.debug('Cache miss for user {user_id}', user_id=123);
sentry_logger.info('Updated global cache');
sentry_logger.warn('Rate limit reached for endpoint {endpoint}', endpoint='/api/results/' });
sentry_logger.error('Failed to process payment. Order: {order_id}. Amount: {amount}', order_id="or_2342", amount=99.99);
sentry_logger.fatal('Database {database} connection pool exhausted', database="users");

Sentry also automatically instruments Python loggers via the LoggingIntegration:

import logging

sentry_sdk.init(
  dsn="...",
  _experiments={
      "enable_logs": True
  }
)

# Your existing logging setup
my_logger = logging.Logger("some-logger")

my_logger.debug('In this example debug events will not be sent to Sentry logs. my_value=%s', my_value)
my_logger.info('But info events will be sent to Sentry logs. my_value=%s', my_value)

The logging integration automatically monkeypatches a handler into all loggers except for the root logger. If you'd like to manually configure the handler, you can do that like this:

import sentry_sdk
from sentry_sdk.integrations.logging import SentryLogsHandler
from sentry_sdk.integrations.logging import LoggingIntegration

sentry_sdk.init(
  dsn="...",
  _experiments={
      "enable_logs": True
  },
  integrations=[
    LoggingIntegration(sentry_logs_level=None),  # Do not monkeypatch the sentry handler
  ]
}

# Instead, configure the root logger to send INFO-level logs to Sentry 
logging.basicConfig(level=logging.INFO, handlers=[SentryLogsHandler(level=logging.INFO)])

To filter logs, or update them before they are sent to Sentry, you can use the _experiments.before_send_log option.

[antonpirker]

Just a note: this is yet experimental and right now is not optimized, so it will make a lot of http requests to Sentry at the moment. So please be careful and do not enable this in your production system yet! :-)

Edit: We added batching in 2.25.1 of the Python SDK, and have updated the instructions to reflect this. Please only use logging with Python SDK 2.25.1+

[AbhiPrasad]

We’ve released 2.26.0 of the Python SDK which contains some improvements around logging. We added a variety of new default attributes like server name and sdk name/version which should make debugging easier.

https://github.com/getsentry/sentry-python/releases/tag/2.26.0

Please upgrade and give it a try!

[lwpamihiranga]

@AbhiPrasad @antonpirker Is there a way I can send JSON formatted logs to sentry and use it as parameterized ones?

[AbhiPrasad]

Ah currently we don't parse json like that. We're exploring do this in the server-side, but I recommend you do something like so:

# With https://docs.python.org/3/library/logging.html
logger.info(log.message, extra=log)

# With sentry API https://docs.sentry.io/platforms/python/logs/
logger.info(log.message, attributes=log)

This has the added advantage that you can easily remove or adjust parts of the log in before_send_log.

[lwpamihiranga]

Got it. It does not actually fulfill our requirement at the moment. Hope the JSON serialization will be added in future. We will revisit this feature again later! :)

[AbhiPrasad]

It does not actually fulfill our requirement at the moment

Could you elaborate why this is the case? Are you not able to change what you pass into the logger atm?

[lwpamihiranga]

It does not actually fulfill our requirement at the moment

Could you elaborate why this is the case? Are you not able to change what you pass into the logger atm?

Yeah, we already have pre-configured JSON logs throughout our application, and we want to forward them to Sentry as they are, without modifying them. Making changes would require updates across our codebase, which we’d like to avoid at this point.

[AbhiPrasad]

Understood!

One solution here in the time being is to carry on calling logger.info(json.dumps(log)), but then use the before_send_log SDK option to grab the json serialized log, parse it for the attributes and add them accordingly.