[FIX] Add min-time to remote-out-of-order bool flag.

[amanycodes]

Fixes [TODO]

The remote-out-of-order flag now supports specifying a minimum time (min-time), so that out-of-order samples are only generated for timestamps no older than the defined time.

[bwplotka]

Thanks for helping!

This is a nice improvement, unfortunately looking closer the current implementation (yours and before) is not really introducing out-of-order as Prometheus data model understands this. My bad for not catching this during the initial PR #68

The current flow adds -60s, -5s or 0 to each sample timestamp. However in the current implementation we only send ONE sample per time series. While different samples might have different timestamps (up to 60s or remote-ooo.min-time duration in the past vs other) with our ooo option, it does NOT really exercise any OOO semantics as a single series samples are still very much in order.

With e.g. -3h offset we might exercise "too old" sample logic in the receiving backend (Prometheus simply rejects this), but it's not very interesting case to benchmark.

I think perhaps what @Vanshikav123 intended was to try shuffle order of samples within a single series, which would be interesting to exercise (Prometheus with OOO feature enabled packs out of order sample to different data structures). Unfortunately, for this, more complex implementation is needed e.g. we could gather metrics within some time frame (e.g. wait for at least two sample and shuffle that, even without min-time or offsets).

To sum, I think this feature is currently broken, so I would either remove it OR try to implement proper OOO and for that proper implementation a bool flag might be enough.

Sorry for confusion and thanks for making me realize this. Let me know if you want to either remove this and work on a proper solution 💪🏽

cc @saswatamcode for extra eyes.