Failsafe for failing logs

[tisonv]

Hello !

I had an issue in my custom LokiJsonTextFormatter where I was expecting a field that was not always in the logEvent.Properties.
The formatting crashes and the logEvent stays in the queue.
The sink then tries to format it every time and fails every time prohibiting any further logEvent to be published.

While I managed to track that particular issue in my formatter, I could get the same problem with other use-cases.

Would it be possible to push a message to Loki with the failed formated log ? after n retries ?
Unlike a network problem, I don't think a formatting issue will ever resolve itself in the next loop.

Thanks !

[mishamyte]

Hi @tisonv!

I would be very grateful if you will provide the sample app with the issue.

I think the main issue that Loki (probably?) rejects the logs, because they are formatted incorrectly. But that should not be a valid point (what is the reason for that formatting?).

And about the event log drop, please, check the next lines of code. You could see there that if we got the unsuccessful status code, we just drop some events for avoiding the bottleneck

[tisonv]

Hey !

Here you are.
If no EventId is present, it goes stuck.
There no rejection from Loki as it never gets to send it.
It crashes to there : https://github.com/serilog-contrib/serilog-sinks-grafana-loki/blob/master/src/Serilog.Sinks.Grafana.Loki/LokiSink.cs#L154

public sealed class LokiTextFormatter : LokiJsonTextFormatter
{
    public override void Format(LogEvent logEvent, TextWriter output)
    {
        if (logEvent == null)
        {
            throw new ArgumentNullException(nameof(logEvent));
        }

        if (output == null)
        {
            throw new ArgumentNullException(nameof(output));
        }

        output.Write("{\"Message\":");
        JsonValueFormatter.WriteQuotedJsonString(logEvent.MessageTemplate.Render(logEvent.Properties, CultureInfo.CurrentCulture), output);

        LogEventPropertyValue eventid = logEvent.Properties["EventId"];
        output.Write($",\"EventId\":{((StructureValue)eventid).Properties[0].Value}");
       ....
}

[mishamyte]

I'm sorry, but not sure I could help you there. The formatter should be always in a valid state by design.

I could propose next solutions:

1. If your formatter is derived from the base and you know that your logs could have that field as an optional, so you should work with it as with the optional, but not as with the mandatory
2. If some specific changes are needed in places, which where designed not in the way you need, you could fork and/or reuse parts from that solution and build a part, which is specific to your business-logic.

Thanks for the understanding!

[tisonv]

oh yes I added a test to check if the field is present and everything is fine. There is no more problem there.

What I mean is failure to format blocks the logging process entirely.

[mishamyte]

In my opinion, those parts should work correct and error while formatting will be a case for the investigation

[tisonv]

No worries ! This is what the Ideas discussion is for
Thanks for your help !