Module Independency & Standalone Usability

[vlomonaco]

We should make sure the main avalanche modules are also usable as standalone libraries.

The Benchmarks and Extra modules already satisfy this requirement.

The Training one may be improved to work on any input data, making less assumptions as possible about the nature of the data processed.

The evaluation module and mostly the metrics are probably too difficult to use off-the-shelf at the moment as heavily integrated with the evaluation plugin. We should relax this!

What do you think?

[AntonioCarta]

The Training one may be improved to work on any input data, making less assumptions as possible about the nature of the data processed.

Benchmarks and metrics right now assume image classification problems. Datasets also implement slicing assuming image classification data. I don't think the base strategies make any specific assumptions. Replay and MultiHead plugins need an additional argument to specify the batch dimension but apart from these minor modifications they should work with any data.

Metrics could be used standalone if we keep the boilerplate to a minimum.

To proceed with this issue it would be useful to implement a CL scenario with non-image data. We (in Pisa) have some scenarios using sequences so we can start with them.

[vlomonaco]

@lrzpellegrini as you update the eval module, we should also provide an example for using the metrics "manually", thanks!

@AntonioCarta I think it's ok if for the alpha version we support only image classification scenarios. What I wanted with this issue was to make the features made available from each module as independent as possible, so that people can use only part of avalanche if they want, without the overhead of learning it all. This will help to make the avalanche learning curve less steep.

But, yes, if you have already such non-image scenarios, you can integrate them as soon as you like! :-)

[AntonioCarta]

@AntonioCarta I think it's ok if for the alpha version we support only image classification scenarios.

I agree, that's why I never opened an issue about it. What I'm saying is that we can't hope to support different input data if we only test avalanche on image classification scenarios.

[vlomonaco]

Yes, but also if we stick to only image classification, the training module at the moment needs to process a StepInfo.

This way people are obliged to learn and use the benchmarks module too somehow. I was wondering if we could support also simpler data format to help people use avalanche sub-module independently.

For example:

cl_model.train((x_tensor, y_tensor, label))

and we take care of the rest?

The same goes from the other avalanche modules...

[AntonioCarta]

Ok, now I get it. Is this really a problem? StepInfo is a simple object, and we already provide a couple of methods to create scenarios given tensors. Isn't it easier to convert the different data formats into scenarios instead of supporting every possible format inside the training module?

Maybe you want to add some other methods to make the transition between tensors and scenarios easier?

[vlomonaco]

Ok, now I get it. Is this really a problem? StepInfo is a simple object, and we already provide a couple of methods to create scenarios given tensors. Isn't it easier to convert the different data formats into scenarios instead of supporting every possible format inside the training module?

It's not really a problem. It's just that people may want to use just the training module without caring about the rest of Avalanche... This may help us make the learning curve of the tool less steep and stimulate early adoption!

[AntonioCarta]

Regarding the strategies, maybe there is a way to make plugins more independent from the rest of avalanche.
We could restrict the plugin callbacks. Instead of taking the strategy as an argument, we could explicitly pass the needed arguments. For example:

    def adapt_train_dataset(self, strategy, **kwargs):
        """
        Expands the current training set with datapoint from
        the external memory before training.
        """
        pass

becomes:

    def adapt_train_dataset(self, current_data, **kwargs):
        """
        Expands the current training set with datapoint from
        the external memory before training.
        """
        return current_data

Notice that this change would restrict the plugin's callbacks, because they would not have access to the entire strategy state at each point. However, it would make plugins relatively separated from the rest of avalanche. It seems a good tradeoff to me. Maybe it's even better to have explicit arguments than having the entire strategy, because you remove the risk of plugins interfering with the strategy's state. This will make it easier to implement new plugins.

Strategy classes (BaseStrategy, JointTraining) are basic avalanche components because they glue together plugins, data streams, and loggers. I don't see how we could make them standalone.

@vlomonaco what do you think? If you agree I can work on this.

[vlomonaco]

Hi @AntonioCarta I somehow missed your last comment... I think it makes sense, but I'd like to discuss it all together this in the next meeting. Please remind me of this in case I forget! :)