-
-
Notifications
You must be signed in to change notification settings - Fork 87
Open
Labels
Description
Currently bayesplot
offers are three functions for discrete PPCs: ppc_bars
, ppc_bars_grouped
, and ppc_rootogram
.
In our preprint, we argue why bar graphs are usually not a good PPC, especially for binary observations.
Calibration plots
I suggest adding the binned calibration plot and PAV-adjusted calibration plot under a function called ppc_calibration
. For ordinal or categorical data, it would possibly be clearer to have another function that produces calibration plots for 1-vs-others, or cumulative event probability.
Residual plots
Another missing function are residual plots, ppc_error_scatter_vs_x
, and ppc_error_binned
work on this field, but the scatter doesn't work for discrete observations, and ppc_error_binned
doesn't currently support covariates on the x-axis.
- The confidence bands in these need work, but at the very least adding options for binned residual plots and PAV-adjusted residuals would be an improvement.
Metadata
Metadata
Assignees
Labels
Type
Projects
Milestone
Relationships
Development
Select code repository
Activity
TeemuSailynoja commentedon Apr 14, 2025
This issue could work as a general discussion on functionality for discrete distributions. I already have bayesplot-like implementations of these from the article, but would appreciate input on the best details of arguments and features.
TeemuSailynoja commentedon Apr 14, 2025
The binned residual plots are also mentioned in #263, and binned calibration plots in #150.
jgabry commentedon Apr 15, 2025
This all sounds good, thanks Teemu! Let me know which specific feedback you want. I think adding all of this is a good idea.
TeemuSailynoja commentedon Apr 16, 2025
Had a conversation about the PAV-adjusted calibration plots with @avehtari, and wanted to continue it here:
For predicted event probabilities, p = p1, ..., pN, the original CORP reliability diagrams paper makes the consistency regions as follows:
In bayesplot, we would expect
y
andyrep
from the user, and can just computep = colMeans(yrep)
, and use the posterior draws for the consistency regions. This is what we did in the Recommendations for visual PCs article.In our conversation with Aki, I got confused with the possibility, that, especially from brms and rstanarm, we could receive posterior samples of the predicted event probability, p, itself. This would allow the consistency region computation to also include the uncertainty about
p
through changing the pool of availablep
between bootstrapping steps.jgabry commentedon Apr 17, 2025
It does sound nice to be able to include the uncertainty about
p
, but I also agree that it's good to avoid too much complexity in the functions. I suppose we could add an optional argument for providing the draws of event probabilities but this argument would never be required. Is that basically what you were suggesting?TeemuSailynoja commentedon Apr 24, 2025
Here is a quick demo of how this
ppc_calibration_overlay
would show the posterior uncertainty of the calibration.This is a raw version without proper theming, and could for example be made into a ribbon.
On the left, pava-calibration plots for two models,
on the right, posterior samples of calibration curves for the models.
jgabry commentedon Apr 25, 2025
Thanks for the demo. I think this would be very nice to have!
jgabry commentedon Apr 25, 2025
And just to confirm, is the idea to have two separate functions for these,
ppc_calibration
andppc_calibration_overlay
?TeemuSailynoja commentedon May 13, 2025
Yes, I was thinking the overlay would be a separate function. Perhaps the other could be just
ppc_calibration()
, and include options for showing the red calibration curve for the mean posterior predicted event probability.There could also be
ppc_loo_calibration()
, which would show LOO predictive probabilities.jgabry commentedon May 14, 2025
Sounds good!