-
Notifications
You must be signed in to change notification settings - Fork 30
Mapping properties to labels
According to Grafana Loki documentation:
Labels are key value pairs and can be defined as anything! We like to refer to them as metadata to describe a log stream. Labels in Loki perform a very important task: They define a stream. More specifically, the combination of every label key and value defines the stream. If just one label value changes, this creates a new stream.
Every unique combination of labels creates a new separate stream in Loki. Big count of streams can kill Loki and called high cardinality. In case of structured logging every log event property translates into label, which cause a lot of metadata for a simple log entry. For example, next labels are created for a standard ASP.NET Core Web API call: ActionId, ActionName, ConnectionId, Message, MessageTemplate, RequestId, RequestPath, SourceContext, Timestamp, Level. This is a lot of labels with a dynamic values.
First public version (v3) of this sink has only possibility to exclude some properties from mapping to the labels. This wasn't a good solution, cause you needed to add all exclusion rules explicitly, what becomes a hell with a huge amount of labels (especially in cases with external libs.
We have changed this behavior in v4. It brought us a filtrationMode and filtrationLabels, where you could select one of modes: Include/Exclude and define properties' names to map 'em to the labels. Or you could leave it blank and in this case you will face the high cardinality problem. We always recommended to use Include mode with some always present labels (such a app name). But in last discussion we have found that this behavior is straightforward, but has no sense.
From v8 we propose you a clear way - you define only properties that should be mapped into a labels (or none if you don't need it). For this case use a new propertiesAsLabels configuration parameters. The rest of properties would be passed to a formatted and should be included as a part of log entry.