Why are error bars on identity line and not the points themselves in plot_reliability_diagram?

[karllandheer]

Hello, I believe my title explains my confusion. Shouldn't the points themselves display the error bars? I'm guessing I'm mis-understanding something subtle, could someone please explain?

[blucena]

There are two perspectives:

1. The "null hypothesis" is that the probabilities are well-calibrated. Under this null hypothesis, the bars measure how far we would expect the empirical probabilities to stray from the y=x axis based on random chance. From this perspective the confidence interval is centered around the axis.

2. The empirical probabilities are "estimates" of the "true parameter value" for the bin and we want a confidence interval that contains this "true value". From this perspective the interval is centered around the point.

The good news is the package lets you choose either perspective! The parameter ci_ref can be set to 'axis' (the default) for the first perspective or 'point' if you prefer the second perspective.

Hope this helps!