Ordering of matched files being watched

[dev-head]

The Request:

This is more of an enhancement request to add support for the matched files to be watched in the order of their timestamp oldest to newest and or add an option to define the some ordering behaviors.

The problem

Given: path => "/tmp/file.*.json"

Right now the ordering of watched files appear to be based on whatever order glob returns. This leads to situations where a newer file ends up being watched before an older one.

In my case I need logstash to respect this historical order of the matched files, otherwise it leads to the old data being considered the latest version. (i get around this by aggressively pruning old files that i "think" are done and when I run into this issue I end up having to reload the latest data manually).

---

Is anyone else running into this?

[jordansissel]

I can confirm that the order of each file being visited is not defined.

As a workaround, can you try generating or templating your logstash config based on the file list first?

Something like
ls -t /tmp/file.*.json will give you the list of files ordered by modification date, if that helps.

[splashx]

Related: https://github.com/jordansissel/ruby-filewatch/pull/60/files

jordansissel/ruby-filewatch#60

[PhaedrusTheGreek]

From @guyboertje 's blog post, this is a desired future capability:

Support for prioritising the order of the files that are activated. Choose between oldest, newest, biggest, smallest, most read, etc.

[guyboertje]

I suspect the OP was not doing file tailing but file read.

[guyboertje]

This is now supported.