Wrong predictions obtained for CHORUS datasets

[andreab1997]

When using the 0.11 version of eko to compute the FKs of the CHORUS datasets, some of the resulting predictions are wrong.

However one can get the correct predictions for that points cutting the grid and keeping only the relevant point (which suggests that the problem is the size of the eko. CHORUS ekos are in fact the biggest among the DIS datasets).

Using an older version of eko (like 0.10.2) the problem does not appear.

For example this is the diff between the comparison files obtained one with eko==0.11 and the other with eko=0.10.2 for all our DIS datasets

Click me

> 17    2.934063   2.934063   0.125    0.125  0.748299 -0.052954    -1070.766013',
 '< 24    5.750762   5.750762   0.175    0.175  0.378966  0.378953       -0.034068',
 '> 24    5.750762   5.750762   0.175    0.175  0.378966 -0.026678    -1070.397244',
 '< 60    2.070274   2.070274   0.045    0.045  0.709288  0.699541      -13.741605',
 '> 60    2.070274   2.070274   0.045    0.045  0.709288 -0.200298    -1282.393623',
 '< 72    5.750762   5.750762   0.125    0.125  0.512782  0.512776       -0.010860',
 '> 72    5.750762   5.750762   0.125    0.125  0.512782 -0.082526    -1160.938299',
 '< 76    5.750762   5.750762   0.175    0.175  0.627399  0.627385       -0.022504',
 '> 76    5.750762   5.750762   0.175    0.175  0.627399 -0.021528    -1034.313612',
 '< 200   3.304928   3.304928   0.080    0.080  1.064367  1.064360       -0.006361',
 '> 200   3.304928   3.304928   0.080    0.080  1.064367 -0.097083    -1091.212174',
 '< 424   2.065580   2.065580   0.020    0.020  1.188870  1.157950      -26.008048',
 '> 424   2.065580   2.065580   0.020    0.020  1.188870 -0.238061    -1200.240987',
 '< 435   3.304928   3.304928   0.080    0.080  1.476857  1.476848       -0.006577',
 '> 435   3.304928   3.304928   0.080    0.080  1.476857 -0.045593    -1030.871773',
 '< 502   2.929368   2.929368   0.020    0.020  1.188744  1.188873        0.108632',
 '> 502   2.929368   2.929368   0.020    0.020  1.188744 -0.280264    -1235.764703',
 '< 505   3.295539   3.295539   0.045    0.045  1.449915  1.449899       -0.010744',
 '> 505   3.295539   3.295539   0.045    0.045  1.449915 -0.120010    -1082.770466',
 '< 570   5.746068   5.746068   0.045    0.045  1.369287  1.369247       -0.029136',
 '> 570   5.746068   5.746068   0.045    0.045  1.369287 -0.177239    -1129.439143'

The points that actually differ between the two belong only to the two CHORUS datasets.
@alecandido
@felixhekhorn pls help me to make this readable :(

[alecandido]

@alecandido
@felixhekhorn pls help me to make this readable :(

Now it is barely readable. I would suggest loading the DataFrames in Pandas, and put side by side only the relevant columns, but I'm not going to do it myself.

And actually, I don't remember the columns by heart, so please provide labels for each of them, it is barely impossible to know which is what...
(even though I recognize the pattern of the PineAPPL output...)

[alecandido]

For example this is the diff between the comparison files obtained one with eko==0.11 and the other with eko=0.10.2 for all our DIS datasets

I'm not even sure EKO versions being the only change: you are actually changing even LHAPDF version (written on top), that on its own should be harmless, but it tells that you're definitely using two different environments.

Are you sure that you isolated the problem?

In any case: you should have the associated EKOs, they are files, so it is easier to reproduce the problem wrt the in-memory case (memory objects die at the end of execution, files don't).
So start taking the operators, take a dummy PDF set, and evolve one flavor at a time (iterating over the union of flavor and evolution bases), in order to understand the differences.

The moment we know more about discrepancies in EKO, we can start making actual hypothesis.
However, in principle the new EKO version have been unit tested, and manually benchmarked before releases. You (as well) should have done by yourself in #162.
So, most of it is expected to work, thus you should find where starts failing and why.

[andreab1997]

@alecandido
@felixhekhorn pls help me to make this readable :(

Now it is barely readable. I would suggest loading the DataFrames in Pandas, and put side by side only the relevant columns, but I'm not going to do it myself.

And actually, I don't remember the columns by heart, so please provide labels for each of them, it is barely impossible to know which is what... (even though I recognize the pattern of the PineAPPL output...)

I updated the format showing only one of the two CHORUS datasets (the other is just equivalent). Now it should be more readable.

[alecandido]

You screwed up formatting once more...

However, I fixed it, never mind 😄

[felixhekhorn]

the pre-tag is your friend ;-)

[andreab1997]

I'm not even sure EKO versions being the only change: you are actually changing even LHAPDF version (written on top), that on its own should be harmless, but it tells that you're definitely using two different environments.

Are you sure that you isolated the problem?

Indeed I am using two different environments: I need to use two different versions of eko at least. However I am not doing anything special. Other than the two different versions of eko I am using of course two different versions of pineko and two different version of lhapdf (which, as you said, it is most likely irrelevant). So I would say that it is not an isolated problem. However, since it would not take a lot of time, you can try to compute just one of the two CHORUS with the exact same versions of pineko and eko and check if you get the exact same results or not.

The moment we know more about discrepancies in EKO, we can start making actual hypothesis.
However, in principle the new EKO version have been unit tested, and manually benchmarked before releases. You (as well) should have done by yourself in #162.
So, most of it is expected to work, thus you should find where starts failing and why.

This of course I can do but I would say that the fact that they are unit tested does not say that every point we compute should be correct. Indeed we get most of the points correctly, only for two datasets we get some of the points wrongly and I believe our unit tests cannot account for such a subset of points.

[alecandido]

the pre-tag is your friend ;-)

Never used <pre> tag directly! It is more annoying, and in case you have some code there won't be any syntax highlight.

You can keep using Markdown inside, just leave a blank line before Markdown, and one after (blank lines should separate HTML from Markdown, so they should be external to the fenced block ```).

[alecandido]

This of course I can do but I would say that the fact that they are unit tested does not say that every point we compute should be correct. Indeed we get most of the points correctly, only for two datasets we get some of the points wrongly and I believe our unit tests cannot account for such a subset of points.

From the point of view of EKO, there are no "datasets", there is a continuum of scales, and some discrete options.
Either it is bugged for some specific values of the discrete options, and we are not testing it, or the unit tests are not properly testing the code.

[andreab1997]

This of course I can do but I would say that the fact that they are unit tested does not say that every point we compute should be correct. Indeed we get most of the points correctly, only for two datasets we get some of the points wrongly and I believe our unit tests cannot account for such a subset of points.

From the point of view of EKO, there are no "datasets", there is a continuum of scales, and some discrete options. Either it is bugged for some specific values of the discrete options, and we are not testing it, or the unit tests are not properly testing the code.

Yes indeed, this is what I meant. Unit tests passing is not equivalent to all the points we compute should be correct because unit tests are not testing every corner of the code.

[alecandido]

Yes indeed, this is what I meant. Unit tests passing is not equivalent to all the points we compute should be correct because unit tests are not testing every corner of the code.

Yes, but they should interpolate the continuum.

If instead the problem is with discrete options, where interpolation is not expected, then it would be useful to find which are the options bugged, in order to start investigating where the bug is in the code.

[giacomomagni]

@andreab1997 this is now solved in eko=0.12 ?

[andreab1997]

Yes, this can be closed