Generic statistics

[jsmariegaard]

New method statistics in generic module which calculates min, max and mean for any dfs file.

[ecomodeller]

@jsmariegaard still relevant?

[jsmariegaard]

@jsmariegaard still relevant?

Yes, I think having this functionality would be of great value. I am afraid I don't have the time to finish this myself though - and I think it would be easier to start a new branch and then copy the relevant parts from this PR over. Maybe a task for @otzi5300 ?

[ecomodeller]

It would be fantastic if we could support arbitrary aggregation using e.g. https://realpython.com/python-reduce-function/ , but I don't know if it is possible.

[jsmariegaard]

Yeah, that would be cool - reminds me about our array API ideas also

[ecomodeller]

E.g. exceedance (according to 🤖)

import numpy as np
from functools import reduce

# Example data: Each row represents a different time step (t=0, t=1, ...) (from `.ReadItemTimeStep`)
data = [
    np.array([10, 15, 22]),  # t=0
    np.array([12, 18, 25]),  # t=1
    np.array([20, 21, 23]),  # t=2
]

thresholds = np.array([10, 15, 20])

def reducer(acc, x):
    exceedance_counts, total_count = acc
    exceedance_counts += (x[:, None] > thresholds).sum(axis=0)
    total_count += len(x)
    return (exceedance_counts, total_count)

initial_counts = np.zeros(len(thresholds), dtype=int)
initial = (initial_counts, 0)

exceedance_counts, total_count = reduce(reducer, data, initial)
exceedance_probs = exceedance_counts / total_count

results = {t: prob for t, prob in zip(thresholds, exceedance_probs)}