Automatically setting partition for a format

[Qgel]

Hello,

Is there a way to automatically set the partition (log_part) column for a specific format? I tried defining a trigger like so, but I think it runs before the rows in the formats table are actually available, and so does not actually set the partitions:

CREATE TRIGGER IF NOT EXISTS my_trigger
  AFTER INSERT ON lnav_events WHEN
    jget(NEW.content, '/$schema') =
      'https://lnav.org/event-file-format-detected-v1.schema.json' AND
    jget(NEW.content, '/format') = 'myformat' 
BEGIN
    UPDATE myformat SET log_part = run_number WHERE log_line IN (
      SELECT log_line FROM (
        SELECT log_line, run_number, lag(run_number) OVER (ORDER BY log_line) AS prev_run_number from myformat
      ) WHERE run_number != prev_run_number
    );
END;

[tstack]

but I think it runs before the rows in the formats table are actually available, and so does not actually set the partitions:

Yes, the event is triggered as the log messages are being processed, so they're not going to be available.

Can I ask what you're trying to do? There might be a different way to accomplish what you want.

[Qgel]

I have a log of an application that has a run_number column that occasionally changes. I.e. the lines are all of the form

2024-03-18T18:54:21.392 INFO 172883 a message
2024-03-18T18:54:22.492 WARN 172883 another message
[snip thousands of the above]
2024-03-18T18:59:33.123 INFO 172884 a message with a new run number

That number indicates a 'run' of the application, so having a partition marker where they change is convenient to navigate/get an overview of the log file.

I can run the body of the above SQL manually to create those partition markers, but I essentially always want to do it if I open a log file that matches that applications format. So it would be nice to be able to run it automatically.

[tstack]

Sorry, the partition functionality has been neglected for a bit. But, I've made some changes to the top-of-tree that should help to revive it. You can now add to a format a "partitions" section that can be used to automatically discover partitions in logs as they are indexed. Here are a couple of examples:

• The following will create partitions for tests run by gtest:

  lnav/src/formats/vmw_log.json

  Lines 244 to 254 in 554f0e2

  	"partitions": {
  	"test-partition": {
  	"description": "Partition for gtest sections",
  	"paths": [
  	{
  	"glob": "*/test.log"
  	}
  	],
  	"pattern": "^\\[ RUN \\] ([^\\n]+)"
  	}
  	},

• The following will create partitions when dhclient binds an IP:
  

  lnav/test/formats/partitions/format.json

  Lines 1 to 11 in 554f0e2

  	{
  	"$schema": "https://lnav.org/schemas/format-v1.schema.json",
  	"syslog_log": {
  	"partitions": {
  	"dhclient-binding": {
  	"description": "Set partition name to current IP",
  	"pattern": " bound to ([^ ]+) --"
  	}
  	}
  	}
  	}

I've also modified the breadcrumb bar to show the current partition as the first crumb after the view. In this screenshot, it shows the name of the test after the "⊑" symbol:

If you want to jump to a particular partition, you can focus on the breadcrumb bar by pressing backtick (`), press right-arrow to select the partition crumb, and then enter the name of the partition.

To jump to the previous/next partition, you can press the { and } hotkeys.

You can download the binaries from the following nightly build to try things out:

https://github.com/tstack/lnav/actions/runs/8377868163

[Qgel]

Wow, thanks for going above and beyond on my question. I tried the nightly build, and it works pretty well for my usecase.

I noticed a small issue in the nightly build:
The ticks in the bar on the right show up the first time you load a file. If you quit lnav, then open the same file again (without resetting the session with CTRL+R), there will not be any ticks in the bar. The partitions are available though (navigation works). If you reset the session, quit lnav, and re-open the file the ticks work again.