Skip to content

[ENH] Adds a check for consistent output for predict and predict_proba #2824

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

[ENH] Adds a check for consistent output for predict and predict_proba #2824

wants to merge 1 commit into from

Conversation

notaryanramani
Copy link
Contributor

Reference Issues/PRs

Fixed #2802

What does this implement/fix? Explain your changes.

Adds a check to ensure that outputs produced by predict() and predict_proba() are consistent for the classification and clustering estimators.

Does your contribution introduce a new dependency? If yes, which one?

No

Any other comments?

PR checklist

For all contributions
  • I've added myself to the list of contributors. Alternatively, you can use the @all-contributors bot to do this for you after the PR has been merged.
  • The PR title starts with either [ENH], [MNT], [DOC], [BUG], [REF], [DEP] or [GOV] indicating whether the PR topic is related to enhancement, maintenance, documentation, bugs, refactoring, deprecation or governance.
For new estimators and functions
  • I've added the estimator/function to the online API documentation.
  • (OPTIONAL) I've added myself as a __maintainer__ at the top of relevant files and want to be contacted regarding its maintenance. Unmaintained files may be removed. This is for the full file, and you should not add yourself if you are just making minor changes or do not want to help maintain its contents.
For developers with write access
  • (OPTIONAL) I've updated aeon's CODEOWNERS to receive notifications about future changes to these files.

@aeon-actions-bot aeon-actions-bot bot added enhancement New feature, improvement request or other non-bug code enhancement testing Testing related issue or pull request labels May 19, 2025
@aeon-actions-bot
Copy link
Contributor

Thank you for contributing to aeon

I have added the following labels to this PR based on the title: [ $\color{#FEF1BE}{\textsf{enhancement}}$ ].
I have added the following labels to this PR based on the changes made: [ $\color{#2C2F20}{\textsf{testing}}$ ]. Feel free to change these if they do not properly represent the PR.

The Checks tab will show the status of our automated tests. You can click on individual test runs in the tab or "Details" in the panel below to see more information if there is a failure.

If our pre-commit code quality check fails, any trivial fixes will automatically be pushed to your PR unless it is a draft.

Don't hesitate to ask questions on the aeon Slack channel if you have any.

PR CI actions

These checkboxes will add labels to enable/disable CI functionality for this PR. This may not take effect immediately, and a new commit may be required to run the new configuration.

  • Run pre-commit checks for all files
  • Run mypy typecheck tests
  • Run all pytest tests and configurations
  • Run all notebook example tests
  • Run numba-disabled codecov tests
  • Stop automatic pre-commit fixes (always disabled for drafts)
  • Disable numba cache loading
  • Push an empty commit to re-run CI checks

MatthewMiddlehurst
MatthewMiddlehurst reviewed May 19, 2025
View reviewed changes
aeon/testing/estimator_checking/_yield_classification_checks.py
@@ -374,3 +374,10 @@ def check_classifier_output(estimator, datatype):
# check predict proba (all classifiers have predict_proba by default)
y_proba = estimator.predict_proba(FULL_TEST_DATA_DICT[datatype]["test"][0])
_assert_predict_probabilities(y_proba, datatype, n_classes=len(unique_labels))

y_pred_proba = np.argmax(y_proba, axis=1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The labels from predict will not necessarily be the argmax. Use the classes and/or class_dictionary attribute

aeon/testing/estimator_checking/_yield_classification_checks.py
@@ -374,3 +374,10 @@ def check_classifier_output(estimator, datatype):
# check predict proba (all classifiers have predict_proba by default)
y_proba = estimator.predict_proba(FULL_TEST_DATA_DICT[datatype]["test"][0])
_assert_predict_probabilities(y_proba, datatype, n_classes=len(unique_labels))

y_pred_proba = np.argmax(y_proba, axis=1)
_assert_predict_labels(y_pred_proba, datatype, unique_labels=unique_labels)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is line 379 necessary?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just added it as we were checking the y_pred and y_proba. But since we get y_pred_proba from y_proba, I don't think it is necessary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@MatthewMiddlehurst MatthewMiddlehurst MatthewMiddlehurst left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
enhancement New feature, improvement request or other non-bug code enhancement testing Testing related issue or pull request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[ENH] Ensure that predict and predict_proba produce comparable output
2 participants
@notaryanramani @MatthewMiddlehurst