How to use SlicedClassificationMetrics

[asteroth]

Hope it's the right place to ask the question like that but I couldn't find any reference example anywhere on the net.
I want to use the sliced metrics and for the life of mine I can't figure out how are they supposed to be used.

kfp.version >>>> '2.15.2'

Running in a pipeline

I expected the sliced metrics to use the same syntax as the normal metrics:

@dsl.component(base_image=DEFAULT_IMAGE, packages_to_install=["numpy"])
def evaluate(model_in: dsl.Input[dsl.Model]
         , slicer: dsl.Output[dsl.SlicedClassificationMetrics]):
    import numpy as np
    # simulate doing something with a model:
    with open(model_in.path, "rb") as fin:
        model = np.load(fin)
    model[1, 1] += 1

    # simulate per batch metric:
    for x in range(100):
        randf = np.random.random()
        slicer.log_roc_data_point("potato", threshold=1/(x+1), tpr=randf, fpr=np.cos(randf))
        slicer.log_roc_data_point("strawberry", threshold=1/(x+1), tpr=randf, fpr=np.sin(randf))

the pipeline fails with an attribute error.

I have a feeling I need to define the SlicedClassificationMetrics somewhere else first and make it both an input and an output to pass it from there but I just can't figure out how is it supposed to work exactly.

Trying to just use a dummy locally:

I looked at the test code in the pipelines wiki and it appears this should work:

import kfp
metrics = kfp.dsl.SlicedClassificationMetrics()
metrics.log_roc_data_point("slice1", threshold=0.12, tpr=27.5, fpr=86.32)
#  autocomplete suggests only metrics.log_roc_reading exists, but it also gives me attribute error

but it gives me AttributeError: 'SlicedClassificationMetrics' object has no attribute '_sliced_metrics' indicating the constructor does not build a complete object even though the arguments are all flagged as optional.

metrics._sliced_metrics = {}  # add an empty dict just to see what happens
# neither works:
metrics.log_roc_reading("slice1", threshold=1.0, tpr=26.5, fpr=88.4)
metrics.log_roc_data_point("potato", threshold=1.0, tpr=26.5, fpr=88.0)

[farfromexact]

The immediate problem is that SlicedClassificationMetrics has a different API from ClassificationMetrics. log_roc_data_point() is the method for the non-sliced artifact; a sliced artifact uses log_roc_reading() and includes the slice name.

Your component should look like this:

@dsl.component(base_image=DEFAULT_IMAGE, packages_to_install=["numpy"])
def evaluate(
    model_in: dsl.Input[dsl.Model],
    slicer: dsl.Output[dsl.SlicedClassificationMetrics],
):
    import numpy as np

    with open(model_in.path, "rb") as fin:
        model = np.load(fin)

    for x in range(100):
        threshold = 1 / (x + 1)
        randf = float(np.random.random())
        slicer.log_roc_reading(
            "potato", threshold=threshold, tpr=randf, fpr=float(np.cos(randf))
        )
        slicer.log_roc_reading(
            "strawberry", threshold=threshold, tpr=randf, fpr=float(np.sin(randf))
        )

The output artifact is injected by the component runtime. Do not instantiate SlicedClassificationMetrics() in a local script and do not set _sliced_metrics yourself; that is an internal implementation detail. If you test locally, the constructor may not be fully initialized because it is normally used as an injected output artifact.

The current SDK reference documents log_roc_reading(slice, threshold, tpr, fpr) here: https://kubeflow-pipelines.readthedocs.io/en/latest/source/dsl.html. Also make sure the rates are in the expected numeric range rather than percentages. log_roc_data_point() is the reason for the attribute error in the pipeline example.