Plots/stats/analyses for scaling dynamics

[athewsey]

AFAIK the plots and stats in LLMeter today all treat each Run as a bucket of IID data points to calculate statistics over - ignoring trends/dynamics during the run.

We're interested for LLMeter to provide more tools for analyzing auto-scaling responses over time to step increases in demand. For example:

1. Were there significant trends/shifts in TTFT or TTLT/TPOT or error rate over time during the Run, or was it homogeneous?
2. If so, how long did latency/error rate take to "stabilize", and could we characterize stats for that steady-state excluding the scale-up period?

Open to suggestions on good/practical ways to approach this, but it seems like a gap in the current tooling.

[acere]

would a separated companion library for advanced trend analysis address this need? or it also need to change the data acquisition, e.g.: test parameters update throughout the test run?

With a separated library, it's relatively straightforward to leverage the standard format for the Results class to establish default 2nd an higher order stats and companion visualization for single/multiple runs.

Dynamic update of test parameters it's much more ambitious, and probably would requires a completely new approach to testing.

[athewsey]

So far one key challenge I think we need to address in the main library is that InvocationResponse has no attached timestamp, which prevents doing any time-based post-analysis at all.

I started a quick sketch on feat/dynamics branch but haven't fully addressed the de/serialization of the added datetime yet (serialization seems pretty automatic, but something needs to deserialize them back to datetime objects when loading from file).