fix(NodeLogs): Fix labelmap regex missing User Journal Fields

[WesselAtWork]

Closes: #1517

[CLAassistant]

All committers have signed the CLA.

[WesselAtWork]

want some help.

---

originally I was just changing the regex to look like this:

    rule {
      action = "labelmap"
      regex = "__journal__?(.+)"
    }

---

but I then noticed that that would set the unit label from __journal_unit

labelmap: __journal_(unit) -> unit

currently we are setting the unit label to __journal_systemd_unit
if we wanted to keep this functionality we need to give the user a way to use the original __journal_unit without causing issues downstream.

so then I added this block

    rule {
      action = "replace"
      source_labels = ["__journal_unit"]
      target_label = "journal_unit"
    }

I then realised that we needed a label fill priority for unit
so I did

    rule {
      action = "replace"
      source_labels = [
        "__journal_unit",
        "__journal__systemd_unit",
      ]
      separator = ";"
      regex = "^;*([^;]+).*$"
      replacement = "$1"
      target_label = "unit"
    }

which should prioritise the __journal_unit if it is set.

---

but then I double checked my changes on my deployment, and was surprised to see that __journal_unit was never set.
this is weird, because it seemed like all the other fields (like __journal_priority, __journal_syslog_identifier) were set!

I took a closer look at the User Journal Fields and I am now more confused then before.

UNIT=, USER_UNIT=

    The name of a unit. Used by the system and user managers when logging about specific units.

    When --unit=name or --user-unit=name are used with [journalctl(1)](https://www.freedesktop.org/software/systemd/man/latest/journalctl.html#), a match pattern that includes "UNIT=name.service" or "USER_UNIT=name.service" will be generated.

    Added in version 251.

The name of a unit. Used by the system and user managers when logging about specific units.

What does this mean?

Is this like INVOCATION_ID where it is only set sometimes?

INVOCATION_ID=, USER_INVOCATION_ID=

    A randomized, unique 128-bit ID identifying each runtime cycle of the unit. This is different from _SYSTEMD_INVOCATION_ID in that it is only used for messages coming from systemd code (e.g. logs from the system/user manager or from forked processes performing systemd-related setup).

    Added in version 245

or is this some kind of client/request only field, that should never apear to us?

I don't know.

Question 1

How should I deal with __journal_unit and __journal_user_unit.

Should I:

Ignore them

no idea when (if ever) this field is set.

      source_labels = ["__journal__systemd_unit" ]

Always override/set the unit label to the Trusted Journal Field

De-Prioritise the fields

if the __journal__systemd_unit is empty, at least try the __journal_unit because it MIGHT be set.

      source_labels = [
        "__journal__systemd_unit",
        "__journal_unit",
      ]

we always prioritise the __journal__systemd_unit

Prioritise the fields

try the __journal_unit field, and THEN use the __journal__systemd_unit field

      source_labels = [
        "__journal_unit",
        "__journal__systemd_unit",
      ]

we always prioritise the __journal__unit

Question 2

Should I fold user_unit into the unit label?

Basically:

      source_labels = [
        "__journal__systemd_unit",
        "__journal__systemd_user_unit",
      ]
     target_label = "unit"

The documentation does not make it 100% clear when one would be set over the other.
The intent seems to make the unit label contain a value to identify the message. If that is the case we should try to fill it with as much fallback options as possible in my opinion.

[WesselAtWork]

Question 3

I just noticed you use ^(?:;*)?([^;]+).*$
Is there a particular reason for this ^(?:;*)? non-capture group?
^;*([^;]+).*$ should be fine too, no?

[WesselAtWork]

I'm going to wait for feedback before I start fixing up the issues