Make benchmarks of initial commit and current main

[danlooo]

We need to benchmark the following for the mid term review:

• alocations of single time series
• timing for single time series
• timing for small testdata
• allocations for testdata
• accuracy, sensitivity, and specificity on this dataset
• code coverage with unit tests
• Usage: CPU and RAM

[felixcremer]

This is the result for a single time series at the current main:

julia> @benchmark RQADeforestation.rqatrend(pix_trend, ts, threshold)
BenchmarkTools.Trial: 10000 samples with 10 evaluations per sample.
 Range (min … max):  1.118 μs …  1.673 μs  ┊ GC (min … max): 0.00% … 0.00%
 Time  (median):     1.137 μs              ┊ GC (median):    0.00%
 Time  (mean ± σ):   1.142 μs ± 37.711 ns  ┊ GC (mean ± σ):  0.00% ± 0.00%

   ▂▇█▃                                                       
  ▄████▅▄▄▃▂▂▁▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▂▁▁▁▂▁▁▂▁▂▁▁▂▂ ▃
  1.12 μs        Histogram: frequency by time        1.45 μs <

 Memory estimate: 0 bytes, allocs estimate: 0.

And this is from the first commit:

BenchmarkTools.Trial: 10000 samples with 1 evaluation per sample.
 Range (min … max):  12.032 μs …  26.399 μs  ┊ GC (min … max): 0.00% … 0.00%
 Time  (median):     12.774 μs               ┊ GC (median):    0.00%
 Time  (mean ± σ):   13.057 μs ± 963.779 ns  ┊ GC (mean ± σ):  0.00% ± 0.00%

   ▂▃▄▄▅▇███▇▆▅▅▄▃▃▁▁▁▁      ▂▂▃▃▂▂▁             ▁▁▁▁▁         ▃
  █████████████████████▇▆▇▇▇▇████████▇▅▄▅▁▄▁▄▄▆▇███████▇▇▆▆▆▅▇ █
  12 μs         Histogram: log(frequency) by time      16.9 μs <

 Memory estimate: 3.75 KiB, allocs estimate: 8.

[felixcremer]

And this is the benchmark of running rqatrend on the main branch

BenchmarkTools.Trial: 16 samples with 1 evaluation per sample.
 Range (min … max):  305.049 ms … 357.478 ms  ┊ GC (min … max): 82.02% … 82.49%
 Time  (median):     330.254 ms               ┊ GC (median):    82.57%
 Time  (mean ± σ):   329.525 ms ±  15.942 ms  ┊ GC (mean ± σ):  82.14% ±  1.05%

  █▁   ▁                 ▁▁  ▁ █      ▁▁  ▁  ▁ ▁  ▁           ▁  
  ██▁▁▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁██▁▁█▁█▁▁▁▁▁▁██▁▁█▁▁█▁█▁▁█▁▁▁▁▁▁▁▁▁▁▁█ ▁
  305 ms           Histogram: frequency by time          357 ms <

[danlooo]

Looks good! Is this tested at version b63c875? Let's create the git tags before and after to mark the tested versions.

[felixcremer]

And this is the benchmark of the firstcommit for the rqatrend on the testdata cube.
I suspect, that something is wrong with the computation because it is a factor of three faster on the first commit.

BenchmarkTools.Trial: 16 samples with 1 evaluation per sample.
 Range (min … max):  305.049 ms … 357.478 ms  ┊ GC (min … max): 82.02% … 82.49%
 Time  (median):     330.254 ms               ┊ GC (median):    82.57%
 Time  (mean ± σ):   329.525 ms ±  15.942 ms  ┊ GC (mean ± σ):  82.14% ±  1.05%

  █▁   ▁                 ▁▁  ▁ █      ▁▁  ▁  ▁ ▁  ▁           ▁  
  ██▁▁▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁██▁▁█▁█▁▁▁▁▁▁██▁▁█▁▁█▁█▁▁█▁▁▁▁▁▁▁▁▁▁▁█ ▁
  305 ms           Histogram: frequency by time          357 ms <

 Memory estimate: 49.43 MiB, allocs estimate: 803624.

[felixcremer]

This is the benchmark of running the rqatrend on a two year cube with 73 time steps for a whole tile. This is on the latest commit.

BenchmarkTools.Trial: 1 sample with 1 evaluation per sample.
 Single result which took 1961.704 s (14.86% GC) to evaluate,
 with a memory estimate of 1847.31 GiB, over 33300195031 allocations.

[danlooo]

Benchmarks #89 (comment) and #89 (comment) are identical. Are they really measured from different commits?

[felixcremer]

No they are not, I clearly copy pasted the wrong one.

[felixcremer]

And this is running on a full tile from the first commit:

BenchmarkTools.Trial: 1 sample with 1 evaluation per sample.
 Single result which took 884.030 s (11.97% GC) to evaluate,
 with a memory estimate of 606.03 GiB, over 1498056002 allocations.

[felixcremer]

This is on current main with stackgroups=:lazyagg

BenchmarkTools.Trial: 1 sample with 1 evaluation per sample.                                                                                                                                  
 Single result which took 649.038 s (18.01% GC) to evaluate,                                                                                                                                  
 with a memory estimate of 223.29 GiB, over 887461 allocations.     

[danlooo]

Thank you! So we are 27% faster and use 63% less memory with 99.94% less allocations. Please link the data (e.g. how many pixels) and code used for this test below. A tile is 15000*15000 pixels.

[felixcremer]

The code is available at #97 and a tile is 15000x15000 pixels but not all of them are coverd by the Sentinel-1 orbit. But this is a reasonable test case, because we will always have these kind of data.

[danlooo]

integrated in EarthyScience/FAIRSenDD#28