Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DOC: V1 improve migration guide #582

Closed
wants to merge 27 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
c615457
v1 branch - initial commit (made of a bunch of wip commits squashed t…
Oct 9, 2024
f0d4778
DOC: reorganize documentation, add migration_guide in doc in .rst for…
Valentin-Laurent Nov 21, 2024
65fe7e9
DOC: add groups in .conformalize for CrossConformalRegression, improv…
Valentin-Laurent Nov 21, 2024
aa6e027
TEST & DOC : add integration tests for CrossConformalRegressor, finis…
Valentin-Laurent Nov 27, 2024
6e09d78
DOC: add decision trees (#546)
Valentin-Laurent Nov 29, 2024
cf4a536
REFACTOR & ENH: Split .fit into .fit_single_estimator and fit_multi_e…
jawadhussein462 Dec 2, 2024
7cc80e9
REFACTOR: make SplitConformalRegressor attributes private, improve v1…
Valentin-Laurent Dec 2, 2024
9de92af
CHORE: add -k option to launch specific v1 integration tests from MAK…
Valentin-Laurent Dec 3, 2024
514a8d0
ENH: implement checks for CrossConformalRegression, make utils.py non…
Valentin-Laurent Dec 4, 2024
80b2ced
TESTS: fix and improve v1 integration tests
Valentin-Laurent Dec 4, 2024
fb29fec
ENH: fix and finish CrossConformalRegressor implementation
Valentin-Laurent Dec 4, 2024
4b15977
DOCS: Add MAPIE v1 docstrings to the DOC (#554)
jawadhussein462 Dec 6, 2024
fa65e5f
CHORE: set-up mypy for v1 code, add it to CI, fix existing warnings
Valentin-Laurent Dec 5, 2024
98285c8
CHORE: set-up mypy for v1 code, add it to CI, fix existing warnings
Valentin-Laurent Dec 5, 2024
4b637b9
CHORE: remove type check v1 from CI
Valentin-Laurent Dec 6, 2024
1f9c2b8
TEST: add integration test for JackknifeAfterBootstrapRegressor (#556)
jawadhussein462 Dec 9, 2024
35f08c7
FIX: correct warnings and error messages to match the v1 API
Valentin-Laurent Dec 9, 2024
55bf216
ENH: implement JackknifeAfterBootstrap regressor, fix integration tes…
Valentin-Laurent Dec 12, 2024
4fedb5b
CHORE: install mapie from master for v1 integration tests, add checks…
Valentin-Laurent Dec 13, 2024
43617db
ENH: improve jackknife docstring (#561)
jawadhussein462 Dec 13, 2024
de4cfc0
TEST: Add integrations tests CQR (#563)
jawadhussein462 Dec 16, 2024
e42b265
FIX: correct ConformalizedQuantileRegressor integration tests warning…
jawadhussein462 Dec 17, 2024
c88bd51
ENH: V1 conformalized quantile regressor implementation (#579)
jawadhussein462 Dec 20, 2024
42db582
TEST: Improve v1 integration test for SplitConformalRegressor (#580)
jawadhussein462 Dec 27, 2024
ce8d5ca
DOC: fix docstrings including examples, test v1 docstring examples in…
jawadhussein462 Dec 29, 2024
10dabb7
DOC: V1 improve migration guide
jawadhussein462 Dec 23, 2024
98a5d08
add sample_weight and groups
jawadhussein462 Dec 23, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
DOCS: Add MAPIE v1 docstrings to the DOC (#554)
* DOCS: Add MAPIE v1 regression docstrings to ReadTheDoc, fix warnings
jawadhussein462 authored and Valentin-Laurent committed Dec 29, 2024
commit 4b15977e860afaa0a594088c7f6e87078138d4da
111 changes: 60 additions & 51 deletions doc/api.rst
Original file line number Diff line number Diff line change
@@ -2,18 +2,28 @@
MAPIE API
#########

.. currentmodule:: mapie
Regression V1 (from mapie_v1)
=============================

Regression
==========
.. autosummary::
:toctree: generated/
:template: class.rst

mapie_v1.regression.SplitConformalRegressor
mapie_v1.regression.CrossConformalRegressor
mapie_v1.regression.JackknifeAfterBootstrapRegressor
mapie_v1.regression.ConformalizedQuantileRegressor

Regression (from mapie)
=======================

.. autosummary::
:toctree: generated/
:template: class.rst

regression.MapieRegressor
regression.MapieQuantileRegressor
regression.MapieTimeSeriesRegressor
mapie.regression.MapieRegressor
mapie.regression.MapieQuantileRegressor
mapie.regression.MapieTimeSeriesRegressor

Classification
==============
@@ -22,7 +32,7 @@ Classification
:toctree: generated/
:template: class.rst

classification.MapieClassifier
mapie.classification.MapieClassifier

Multi-Label Classification
==========================
@@ -31,7 +41,7 @@ Multi-Label Classification
:toctree: generated/
:template: class.rst

multi_label_classification.MapieMultiLabelClassifier
mapie.multi_label_classification.MapieMultiLabelClassifier

Calibration
===========
@@ -40,64 +50,64 @@ Calibration
:toctree: generated/
:template: class.rst

calibration.MapieCalibrator
mapie.calibration.MapieCalibrator

Metrics
=======

.. autosummary::
:toctree: generated/
:template: function.rst
metrics.classification_coverage_score
metrics.classification_coverage_score_v2
metrics.classification_mean_width_score
metrics.classification_ssc
metrics.classification_ssc_score
metrics.cumulative_differences
metrics.expected_calibration_error
metrics.hsic
metrics.kolmogorov_smirnov_cdf
metrics.kolmogorov_smirnov_p_value
metrics.kolmogorov_smirnov_statistic
metrics.kuiper_cdf
metrics.kuiper_p_value
metrics.kuiper_statistic
metrics.length_scale
metrics.regression_coverage_score
metrics.regression_coverage_score_v2
metrics.regression_mean_width_score
metrics.regression_ssc
metrics.regression_ssc_score
metrics.spiegelhalter_p_value
metrics.spiegelhalter_statistic
metrics.top_label_ece

Conformity scores (regression)

mapie.metrics.classification_coverage_score
mapie.metrics.classification_coverage_score_v2
mapie.metrics.classification_mean_width_score
mapie.metrics.classification_ssc
mapie.metrics.classification_ssc_score
mapie.metrics.cumulative_differences
mapie.metrics.expected_calibration_error
mapie.metrics.hsic
mapie.metrics.kolmogorov_smirnov_cdf
mapie.metrics.kolmogorov_smirnov_p_value
mapie.metrics.kolmogorov_smirnov_statistic
mapie.metrics.kuiper_cdf
mapie.metrics.kuiper_p_value
mapie.metrics.kuiper_statistic
mapie.metrics.length_scale
mapie.metrics.regression_coverage_score
mapie.metrics.regression_coverage_score_v2
mapie.metrics.regression_mean_width_score
mapie.metrics.regression_ssc
mapie.metrics.regression_ssc_score
mapie.metrics.spiegelhalter_p_value
mapie.metrics.spiegelhalter_statistic
mapie.metrics.top_label_ece

Conformity Scores (Regression)
==============================

.. autosummary::
:toctree: generated/
:template: class.rst

conformity_scores.BaseRegressionScore
conformity_scores.AbsoluteConformityScore
conformity_scores.GammaConformityScore
conformity_scores.ResidualNormalisedScore
mapie.conformity_scores.BaseRegressionScore
mapie.conformity_scores.AbsoluteConformityScore
mapie.conformity_scores.GammaConformityScore
mapie.conformity_scores.ResidualNormalisedScore

Conformity scores (classification)
Conformity Scores (Classification)
==================================

.. autosummary::
:toctree: generated/
:template: class.rst

conformity_scores.BaseClassificationScore
conformity_scores.NaiveConformityScore
conformity_scores.LACConformityScore
conformity_scores.APSConformityScore
conformity_scores.RAPSConformityScore
conformity_scores.TopKConformityScore
mapie.conformity_scores.BaseClassificationScore
mapie.conformity_scores.NaiveConformityScore
mapie.conformity_scores.LACConformityScore
mapie.conformity_scores.APSConformityScore
mapie.conformity_scores.RAPSConformityScore
mapie.conformity_scores.TopKConformityScore

Resampling
==========
@@ -106,15 +116,14 @@ Resampling
:toctree: generated/
:template: class.rst

subsample.BlockBootstrap
subsample.Subsample

mapie.subsample.BlockBootstrap
mapie.subsample.Subsample

Mondrian
==========
========

.. autosummary::
:toctree: generated/
:template: class.rst

mondrian.MondrianCP
mapie.mondrian.MondrianCP
8 changes: 8 additions & 0 deletions mapie_v1/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
from . import classification, regression
from ._version import __version__

__all__ = [
"regression",
"classification",
"__version__"
]
1 change: 1 addition & 0 deletions mapie_v1/_version.py
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
__version__ = "1.0.0"
55 changes: 28 additions & 27 deletions mapie_v1/regression.py
Original file line number Diff line number Diff line change
@@ -22,9 +22,10 @@
class SplitConformalRegressor:
"""
A conformal regression model using split conformal prediction to generate
prediction intervals with statistical guarantees. This method involves
using a hold-out conformity set to determine prediction intervals around
point predictions from a base regressor.
prediction intervals.

This method involves using a hold-out conformity set to determine
prediction intervals around point predictions from a base regressor.

Parameters
----------
@@ -243,10 +244,11 @@ def predict(
class CrossConformalRegressor:
"""
A conformal regression model using cross-conformal prediction to generate
prediction intervals with statistical guarantees. This method involves
computing conformity scoring across multiple folds in a cross-validation
fashion to determine prediction intervals around point predictions from a
base regressor.
prediction intervals.

This method involves computing conformity scoring across multiple folds in
a cross-validation fashion to determine prediction intervals around point
predictions from a base regressor.

Parameters
----------
@@ -272,20 +274,20 @@ class CrossConformalRegressor:
The method used to compute prediction intervals. Options are:
- "base": Based on the conformity scores from each fold.
- "plus": Based on the conformity scores from each fold and
the test set predictions.
the test set predictions.
- "minmax": Based on the conformity scores from each fold and
the test set predictions, using the minimum and maximum among
each fold models.
the test set predictions, using the minimum and maximum among
each fold models.

cv : Union[int, BaseCrossValidator], default=5
The cross-validation strategy used to compute confomity scores.
Valid options:
- integer, to specify the number of folds
- any ``sklearn.model_selection.BaseCrossValidator`` suitable for
regression, or a custom cross-validator inheriting from it.
Main variants in the cross conformal setting are:
- ``sklearn.model_selection.KFold`` (vanilla cross conformal)
- ``sklearn.model_selection.LeaveOneOut`` (jackknife)
regression, or a custom cross-validator inheriting from it.
Main variants in the cross conformal setting are:
* ``sklearn.model_selection.KFold`` (vanilla cross conformal)
* ``sklearn.model_selection.LeaveOneOut`` (jackknife)

n_jobs : Optional[int], default=None
The number of jobs to run in parallel when applicable.
@@ -304,7 +306,7 @@ class CrossConformalRegressor:
An array containing the prediction intervals with shape:
- `(n_samples, 2)` if `confidence_level` is a single float
- `(n_samples, 2, n_confidence_levels)` if `confidence_level`
is a list of floats.
is a list of floats.

Examples
--------
@@ -492,7 +494,7 @@ def predict(
Generates point predictions for the input data `X`:
- using the model fitted on the entire dataset
- or if aggregation_method is provided, aggregating predictions from
the models fitted on each fold
the models fitted on each fold

Parameters
----------
@@ -524,9 +526,11 @@ def predict(
class JackknifeAfterBootstrapRegressor:
"""
A conformal regression model using the jackknife-after-bootstrap approach
to generate prediction intervals with statistical guarantees. This method
combines bootstrap sampling with the jackknife technique to produce robust
prediction intervals around point predictions from a base regressor.
to generate prediction intervals.

This method combines bootstrap sampling with the jackknife technique
to produce robust prediction intervals around point predictions from
a base regressor.

Parameters
----------
@@ -553,9 +557,9 @@ class JackknifeAfterBootstrapRegressor:
The method used for jackknife-after-bootstrap prediction. Options are:
- "base": Based on the conformity scores from each bootstrap sample.
- "plus": Based on the conformity scores from each bootstrap sample and
the testing prediction.
the testing prediction.
- "minmax": Based on the minimum and maximum conformity scores from
each bootstrap sample.
each bootstrap sample.

n_bootstraps : int, default=100
The number of bootstrap resamples to generate for the
@@ -718,8 +722,9 @@ def predict(
class ConformalizedQuantileRegressor:
"""
A conformal quantile regression model that generates prediction intervals
with statistical guarantees using quantile regression as the base
estimator. This approach provides prediction intervals by leveraging
using quantile regression as the base estimator.

This approach provides prediction intervals by leveraging
quantile predictions and applying conformal adjustments to ensure coverage.

Parameters
@@ -790,10 +795,6 @@ def __init__(
estimator: RegressorMixin = QuantileRegressor(),
confidence_level: Union[float, List[float]] = 0.9,
conformity_score: Union[str, BaseRegressionScore] = "absolute",
# n_jobs: Optional[int] = None
# Not yet available in MapieQuantileRegressor
# verbose: int = 0,
# Idem
random_state: Optional[Union[int, np.random.RandomState]] = None,
) -> None:
pass