What to do with the warnings

[MarieSacksick]

Today, we have warnings inside a skore function train_test_split. It takes as input the same inputs as the train_test_split in scikit-learn, plus a parameter project, and it returns the same output as in scikit-learn. The goal is to have methodological help to avoid common pitfalls to the user.

Verbatim in favor of this kind of feature

Warith

DS warnings is a great feature folks 👏

Vincent D. W.

One of the most productive things you can do with scikit-learn is to not ignore the warnings that it gives you.

I tend to celebrate warnings because it is much nicer to catch a bug early.

the idea of showing warnings … that makes complete sense.

(All these during the same conversation)

Linkedin users

When we did a post about this part of skore, the post had a great success compared to the others published around the same time.

One of the comments is

I like the warning but love ❤️ the suggestion that it gives

(source)

Downsides of the current UX

• the user has to know all the scikit-learn function that could be replaced
• we have little maneuver possible, as we impose ourselves to have (almost) the same inputs, and the same outputs
• the warnings are always displayed, whatever where the code is called, and there is nothing stored
• there is no way to say that a warning has been acknowledged and that it shouldn't be displayed anymore.

What pain point we want to solve

Data Science is a large field, even when being a expert and senior, we can't know everything, and it can happen to do mistakes or to not see mistakes in a review: errors can happen.

What value it brings to the user

1. Peace of mind
2. Gain of time because errors are detected earlier
   a. faster reviews
   b. fewer iterations
3. Positive impact on the project, either by increasing performances, or closing the perf in xp to prod. (to be measured)

What we want to provide

NB: not necessarily all in one place. Part of it can be in hub, and part can be in lib.

• a notification about this warning;
• a storage where the warnings can be found;
• a system to acknowledge the warning, and say why the code/data is kept as is despite the warning;
• maybe some resources to learn?

Suggestion of solution

The solution suggested should not block any element listed above. However, let's focus on the two first elements, and let's not try to do all at once.

Possible tracks

We have two main tracks possible. The linting and the embed analysis.

For the linting, we can't use the data and the content of the artefacts, such as models and pipelines. Furthermore, many linter already exists, and it's not consistant with the rest of the product.
For the embed analysis, the downside is that it's not exhaustive in coverage. It's limited to what skore can see.

What it would look like

We would have warnings accessible in a report:

report = EstimatorReport(estimator, X_train, y_train, X_test, y_test)
report.check() # to run something to create the warnings
report.warnings # renders a list, can be empty, of warnings
report.display_warnings() # renders a nice display of warnings

The question now is: those warnings, are they just a sentence in a dict, or.. an object?
I really like the idea to have an object, because it actually allows to have the two tracks: linting, and embed analysis. Furthermore, having an object would allow to do things such as warning.edit_justification("the reason why I did this").
This is the solution I advocate for.

[glemaitre]

An additional downside of train_test_split is that it has access to only the target and not the model. There are sometimes ambiguity to define something simple as the ML task at hand because we don't have access to the model. Having an accessor in the reporting is certainly something that I would like to have.

The question now is: those warnings, are they just a sentence in a dict, or.. an object?

I see something much more advance than that (I don't the UI yet in head). I would expect something like the "Associations" tab from the skrub.TableReport in which we can have tabs and more quantitative info explaining why we would raise the warning. So I would expect an HTML rendering with a back up using rich if not available.

[MarieSacksick]

The question I try to solve in my mind is more about the nature of the warnings than their display: if they are just a piece of text in a dict, we can display them in plenty of different way. However they cannot be annotated (for instance to say why they are not relevant for a given use case), and they cannot live outside the report. While we can do that if warnings are an object per se, and it doesn't prevent to display them like the associations tab.

[glemaitre]

Since I think of quantitative additional information, we might be limited with just a dict. But I would start somewhere and see what more we need.

[adrinjalali]

Those warnings can come from checkers / linters which have a very structured way of representing warnings, errors, and suggestions. They would have rules which can be disable/enabled or even extended by the user/community.

Those linters can be called directly by the user, or by the EstimatorReport like objects we have. Something along the lines of:

# checking a simple sklearn/xgboost/... estimator, with or w/o data
linter.check_model(estimator)
linter.check_model(estimator, X, y)
# they could also check the data
linter.check_data(X, y, ...)
linter.check_data(X_train, y_train, X_test, y_test)

The result of the above functions are all very structured, think a list of instances of classes, each representing a message about a particular case/rule.

The EstimatorReport can then call those optionally / on user inspection and store or view them when necessary.

[MarieSacksick]

It looks like we are on the same page, thanks for confirming and precising the API for the linter part!

[sylvaincom]

Some thoughts:

• in a UI, it would be nice to have the list of which checks were applied, and which passed and which failed ; then for the human DS to inspect further, for example:
  • he is doing train test split, we raise a warning about a feature being temporal, then the user can check by hand if that feature is indeed temporal or not
  • he is doing linear regression on a feature on which we say the distribution is not normal and should go under log transformation before, after the check failed, the user can click in "In-depth analysis" for that specific check and visualize the distribution of the data for himself to be convinced
• adding a justification to ignoring a warning is great
• indeed, train_test_split as currently is not great, but suggestions of API such as report.check() or linter.check_model(estimator) seem great
• in the long run, with the hub, the lead DS can decide, out of the full list of checks that skore does, which ones to discard because it does not correspond to his use case
• a warning could redirect to a doc where the methodological pitfalls are more clearly explained for beginners