Development for multioutput regression (#803)

* PEP8 (#718)

* multioutput_regression

* multioutput_regression

* multioutput_regression

* multioutput regression

* multioutput regression

* multioutput regression

* multioutput regression

* multioutput regression

* #782 showcase pipeline components iteration

* Fixed flake-8 violations

* multi_output regression v1

* fix y_shape in multioutput regression

* fix xy_data_manager change due to merge

* automl.py missing import

* Release note 070 (#842)

* First version of 070 release notes

* Missed a bugfix

* Vim added unexpected space -- fix

* prepare new release (#846)

* Clip predict values to [0-1] in classification

* Fix for 3.5 python!

* Sensible default value of 'score_func' for SelectPercentileRegression (#843)

Currently default value of 'score_func' for SelectPercentileRegression
is "f_classif", which is an invalid value, and will surely be rejected and
will not work

* More robust tmp file naming (#854)

* More robust tmp file naming

* UUID approach

* 771 worst possible result (#845)

* Initial Commit

* Make worst result a function

* worst possible result in metric

* Fixing the name of the scorers

* Add exceptions to log file, not just stdout (#863)

* Add exceptions to log file, not just stdout

* Removing dummy pred as trys is not needed

* Add prediction with models trained with cross-validation (#864)

* add the possibility to predict with cross-validation

* fix unit tests

* test new feature, too

* 715 ml memory (#865)

* #715 Support for no ml memory limit

* API update

* Docs enhancement (#862)

* Improved docs

* Fixed example typos

* Beautify examples

* cleanup examples

* fixed rsa equal

* Move to minmax scaler (#866)

* Do not read predictions in memory, only after score (#870)

* Do not read predictions in memory, only after score

* Precission support for string/int

* Removal of competition manager (#869)

* Removal of competition manager

* Removed additional unused methods/files and moved metrics to estimator

* Fix meta data generation

* Make sure pytest is older newer than 4.6

* Unit tst fixing

* flake8 fixes in examples

* Fix metadata gen metrics

* Fix dataprocessing get params (#877)

* Fix dataprocessing get params

* Add clone-test to regression pipeline

* Allow 1-D threshold binary predictions (#879)

* fix single output regression not working

* regression need no _enusre_prediction_array_size_prediction_array_sizess

* #782 showcase pipeline components iteration

* Fixed flake-8 violations

* Release note 070 (#842)

* First version of 070 release notes

* Missed a bugfix

* Vim added unexpected space -- fix

* prepare new release (#846)

* Clip predict values to [0-1] in classification

* Fix for 3.5 python!

* Sensible default value of 'score_func' for SelectPercentileRegression (#843)

Currently default value of 'score_func' for SelectPercentileRegression
is "f_classif", which is an invalid value, and will surely be rejected and
will not work

* More robust tmp file naming (#854)

* More robust tmp file naming

* UUID approach

* 771 worst possible result (#845)

* Initial Commit

* Make worst result a function

* worst possible result in metric

* Fixing the name of the scorers

* Add exceptions to log file, not just stdout (#863)

* Add exceptions to log file, not just stdout

* Removing dummy pred as trys is not needed

* Add prediction with models trained with cross-validation (#864)

* add the possibility to predict with cross-validation

* fix unit tests

* test new feature, too

* 715 ml memory (#865)

* #715 Support for no ml memory limit

* API update

* Docs enhancement (#862)

* Improved docs

* Fixed example typos

* Beautify examples

* cleanup examples

* fixed rsa equal

* Move to minmax scaler (#866)

* Do not read predictions in memory, only after score (#870)

* Do not read predictions in memory, only after score

* Precission support for string/int

* Removal of competition manager (#869)

* Removal of competition manager

* Removed additional unused methods/files and moved metrics to estimator

* Fix meta data generation

* Make sure pytest is older newer than 4.6

* Unit tst fixing

* flake8 fixes in examples

* Fix metadata gen metrics

* Fix dataprocessing get params (#877)

* Fix dataprocessing get params

* Add clone-test to regression pipeline

* Allow 1-D threshold binary predictions (#879)

* multioutput_regression

* multioutput_regression

* multioutput_regression

* multioutput_regression

* multioutput_regression

* multioutput_regression

* multioutput regression

* multioutput regression

* multioutput regression

* multioutput regression

* multi_output regression v1

* fix y_shape in multioutput regression

* fix xy_data_manager change due to merge

* fix single output regression not working

* regression need no _enusre_prediction_array_size_prediction_array_sizess

* Add prediction with models trained with cross-validation (#864)

* add the possibility to predict with cross-validation

* fix unit tests

* test new feature, too

* multioutput_regression

* multioutput_regression

* multioutput_regression

* Removal of competition manager (#869)

* Removal of competition manager

* Removed additional unused methods/files and moved metrics to estimator

* Fix meta data generation

* Make sure pytest is older newer than 4.6

* Unit tst fixing

* flake8 fixes in examples

* Fix metadata gen metrics

* multioutput after rebased to 0.7.0

Problem:

Cause:

Solution:

* Regressor target y shape index out of range

* Revision for make tester

* Revision: Cancel Multiclass-MultiOuput

* Resolve automl.py metrics(__init__) reg_gb reg_svm

* Fix Flake8 errors

* Fix automl.py flake8

* Preprocess w/ mulitout reg,automl self._n_outputs

* test_estimator.py changed back

* cancel multioutput multiclass for multi reg

* Fix automl self._n_output update placement

* fix flake8

* Kernel pca cancelled mulitout reg

* Kernel PCA test skip python <3.8

* Add test unit for multioutput reg and fix.

* Fix flake8 error

* Kernel PCA multioutput regression

* default kernel to cosine, dodge sklearn=0.22 error

* Kernel PCA should be updated to 0.23

* Kernel PCA uses rbf kernel

* Kernel Pca

* Modify labels in reg, class, perpro in examples

* Kernel PCA

* Add missing supports to mincoal and truncateSVD

Co-authored-by: Matthias Feurer <[email protected]>
Co-authored-by: chico <[email protected]>
Co-authored-by: Francisco Rivera Valverde <[email protected]>
Co-authored-by: Xiaodong DENG <[email protected]>

Author: 4 people

autosklearn/automl.py

@@ -36,7 +36,7 @@
 from autosklearn.util.hash import hash_array_or_matrix
 from autosklearn.metrics import f1_macro, accuracy, r2
 from autosklearn.constants import MULTILABEL_CLASSIFICATION, MULTICLASS_CLASSIFICATION, \
-    REGRESSION_TASKS, REGRESSION, BINARY_CLASSIFICATION
+    REGRESSION_TASKS, REGRESSION, BINARY_CLASSIFICATION, MULTIOUTPUT_REGRESSION
 
 
 def _model_predict(model, X, batch_size, logger, task):
@@ -936,13 +936,16 @@ def _check_X(self, X):
 
     def _check_y(self, y):
         y = sklearn.utils.check_array(y, ensure_2d=False)
-
         y = np.atleast_1d(y)
-        if y.ndim == 2 and y.shape[1] == 1:
+
+        if y.ndim == 1:
+            return y
+        elif y.ndim == 2 and y.shape[1] == 1:
             warnings.warn("A column-vector y was passed when a 1d array was"
                           " expected. Will change shape via np.ravel().",
                           sklearn.utils.DataConversionWarning, stacklevel=2)
             y = np.ravel(y)
+            return y
 
         return y
 
@@ -1097,6 +1100,9 @@ def predict_proba(self, X, batch_size=None, n_jobs=1):
 class AutoMLRegressor(BaseAutoML):
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
+        self._task_mapping = {'continuous-multioutput': MULTIOUTPUT_REGRESSION,
+                              'continuous': REGRESSION,
+                              'multiclass': REGRESSION}
 
     def fit(
         self,
@@ -1110,17 +1116,20 @@ def fit(
         load_models: bool = True,
     ):
         X, y = super()._perform_input_checks(X, y)
-        _n_outputs = 1 if len(y.shape) == 1 else y.shape[1]
-        if _n_outputs > 1:
-            raise NotImplementedError(
-                'Multi-output regression is not implemented.')
+        y_task = type_of_target(y)
+        task = self._task_mapping.get(y_task)
+        if task is None:
+            raise ValueError('Cannot work on data of type %s' % y_task)
+
         if self._metric is None:
             self._metric = r2
+
+        self._n_outputs = 1 if len(y.shape) == 1 else y.shape[1]
         return super().fit(
             X, y,
             X_test=X_test,
             y_test=y_test,
-            task=REGRESSION,
+            task=task,
             feat_type=feat_type,
             dataset_name=dataset_name,
             only_return_configuration_space=only_return_configuration_space,

autosklearn/constants.py

@@ -4,8 +4,9 @@
 MULTICLASS_CLASSIFICATION = 2
 MULTILABEL_CLASSIFICATION = 3
 REGRESSION = 4
+MULTIOUTPUT_REGRESSION = 5
 
-REGRESSION_TASKS = [REGRESSION]
+REGRESSION_TASKS = [REGRESSION, MULTIOUTPUT_REGRESSION]
 CLASSIFICATION_TASKS = [BINARY_CLASSIFICATION, MULTICLASS_CLASSIFICATION,
                         MULTILABEL_CLASSIFICATION]
 
@@ -15,10 +16,12 @@
     {BINARY_CLASSIFICATION: 'binary.classification',
      MULTICLASS_CLASSIFICATION: 'multiclass.classification',
      MULTILABEL_CLASSIFICATION: 'multilabel.classification',
-     REGRESSION: 'regression'}
+     REGRESSION: 'regression',
+     MULTIOUTPUT_REGRESSION: 'multioutput.regression'}
 
 STRING_TO_TASK_TYPES = \
     {'binary.classification': BINARY_CLASSIFICATION,
      'multiclass.classification': MULTICLASS_CLASSIFICATION,
      'multilabel.classification': MULTILABEL_CLASSIFICATION,
-     'regression': REGRESSION}
+     'regression': REGRESSION,
+     'multioutput.regression': MULTIOUTPUT_REGRESSION}

autosklearn/data/xy_data_manager.py

@@ -4,7 +4,7 @@
 from scipy import sparse
 
 from autosklearn.constants import STRING_TO_TASK_TYPES, REGRESSION, BINARY_CLASSIFICATION, \
-    MULTICLASS_CLASSIFICATION, MULTILABEL_CLASSIFICATION
+    MULTICLASS_CLASSIFICATION, MULTILABEL_CLASSIFICATION, MULTIOUTPUT_REGRESSION
 from autosklearn.data.abstract_data_manager import AbstractDataManager
 
 
@@ -27,6 +27,7 @@ def __init__(self, X, y, X_test, y_test, task, feat_type, dataset_name):
         label_num = {
             REGRESSION: 1,
             BINARY_CLASSIFICATION: 2,
+            MULTIOUTPUT_REGRESSION: y.shape[-1],
             MULTICLASS_CLASSIFICATION: len(np.unique(y)),
             MULTILABEL_CLASSIFICATION: y.shape[-1]
         }

autosklearn/estimators.py

@@ -766,15 +766,15 @@ def fit(self, X, y,
         X : array-like or sparse matrix of shape = [n_samples, n_features]
             The training input samples.
 
-        y : array-like, shape = [n_samples]
+        y : array-like, shape = [n_samples] or [n_samples, n_targets]
             The regression target.
 
         X_test : array-like or sparse matrix of shape = [n_samples, n_features]
             Test data input samples. Will be used to save test predictions for
             all models. This allows to evaluate the performance of Auto-sklearn
             over time.
 
-        y_test : array-like, shape = [n_samples]
+        y_test : array-like, shape = [n_samples] or [n_samples, n_targets]
             The regression target. Will be used to calculate the test error
             of all models. This allows to evaluate the performance of
             Auto-sklearn over time.
@@ -799,7 +799,6 @@ def fit(self, X, y,
         target_type = type_of_target(y)
         if target_type in ['multiclass-multioutput',
                            'multilabel-indicator',
-                           'continuous-multioutput',
                            'unknown',
                            ]:
             raise ValueError("regression with data of type %s is not"

autosklearn/evaluation/abstract_evaluator.py

@@ -15,6 +15,7 @@
     REGRESSION_TASKS,
     MULTILABEL_CLASSIFICATION,
     MULTICLASS_CLASSIFICATION,
+    MULTIOUTPUT_REGRESSION
 )
 from autosklearn.pipeline.implementations.util import (
     convert_multioutput_multiclass_to_multilabel
@@ -204,12 +205,19 @@ def _get_model(self):
                                      random_state=self.seed,
                                      init_params=self._init_params)
         else:
-            dataset_properties = {
-                'task': self.task_type,
-                'sparse': self.datamanager.info['is_sparse'] == 1,
-                'multilabel': self.task_type == MULTILABEL_CLASSIFICATION,
-                'multiclass': self.task_type == MULTICLASS_CLASSIFICATION,
-            }
+            if self.task_type in REGRESSION_TASKS:
+                dataset_properties = {
+                    'task': self.task_type,
+                    'sparse': self.datamanager.info['is_sparse'] == 1,
+                    'multioutput': self.task_type == MULTIOUTPUT_REGRESSION,
+                }
+            else:
+                dataset_properties = {
+                    'task': self.task_type,
+                    'sparse': self.datamanager.info['is_sparse'] == 1,
+                    'multilabel': self.task_type == MULTILABEL_CLASSIFICATION,
+                    'multiclass': self.task_type == MULTICLASS_CLASSIFICATION,
+                }
             model = self.model_class(config=self.configuration,
                                      dataset_properties=dataset_properties,
                                      random_state=self.seed,

autosklearn/metrics/__init__.py

@@ -60,6 +60,8 @@ def __call__(self, y_true, y_pred, sample_weight=None):
         elif type_true == 'multilabel-indicator':
             y_pred[y_pred > 0.5] = 1.0
             y_pred[y_pred <= 0.5] = 0.0
+        elif type_true == 'continuous-multioutput':
+            pass
         else:
             raise ValueError(type_true)
 

autosklearn/pipeline/components/base.py

@@ -44,8 +44,8 @@ def add_component(self, obj):
         properties = set(classifier.get_properties())
         should_be_there = {'shortname', 'name', 'handles_regression',
                            'handles_classification', 'handles_multiclass',
-                           'handles_multilabel', 'is_deterministic',
-                           'input', 'output'}
+                           'handles_multilabel', 'handles_multioutput',
+                           'is_deterministic', 'input', 'output'}
         for property in properties:
             if property not in should_be_there:
                 raise ValueError('Property %s must not be specified for '
@@ -54,7 +54,8 @@ def add_component(self, obj):
                                  (property, name, str(should_be_there)))
         for property in should_be_there:
             if property not in properties:
-                raise ValueError('Property %s not specified for algorithm %s')
+                raise ValueError('Property %s not specified for algorithm %s' %
+                                 (property, name))
 
         self.components[name] = classifier
 
@@ -292,7 +293,7 @@ def predict(self, X):
 
         Returns
         -------
-        array, shape = (n_samples,)
+        array, shape = (n_samples,) or shape = (n_samples, n_targets)
             Returns the predicted values
 
         Notes

autosklearn/pipeline/components/classification/adaboost.py

@@ -57,6 +57,7 @@ def get_properties(dataset_properties=None):
                 'handles_classification': True,
                 'handles_multiclass': True,
                 'handles_multilabel': False,
+                'handles_multioutput': False,
                 'is_deterministic': True,
                 'input': (DENSE, SPARSE, UNSIGNED_DATA),
                 'output': (PREDICTIONS,)}

autosklearn/pipeline/components/classification/bernoulli_nb.py

@@ -53,6 +53,7 @@ def get_properties(dataset_properties=None):
                 'handles_classification': True,
                 'handles_multiclass': True,
                 'handles_multilabel': True,
+                'handles_multioutput': False,
                 'is_deterministic': True,
                 'input': (DENSE, SPARSE, UNSIGNED_DATA),
                 'output': (PREDICTIONS,)}

autosklearn/pipeline/components/classification/decision_tree.py

@@ -84,6 +84,7 @@ def get_properties(dataset_properties=None):
                 'handles_classification': True,
                 'handles_multiclass': True,
                 'handles_multilabel': True,
+                'handles_multioutput': False,
                 'is_deterministic': True,
                 'input': (DENSE, SPARSE, UNSIGNED_DATA),
                 'output': (PREDICTIONS,)}

autosklearn/pipeline/components/classification/extra_trees.py

@@ -115,6 +115,7 @@ def get_properties(dataset_properties=None):
                 'handles_classification': True,
                 'handles_multiclass': True,
                 'handles_multilabel': True,
+                'handles_multioutput': False,
                 'is_deterministic': True,
                 'input': (DENSE, SPARSE, UNSIGNED_DATA),
                 'output': (PREDICTIONS,)}

autosklearn/pipeline/components/classification/gaussian_nb.py

@@ -49,6 +49,7 @@ def get_properties(dataset_properties=None):
                 'handles_classification': True,
                 'handles_multiclass': True,
                 'handles_multilabel': True,
+                'handles_multioutput': False,
                 'is_deterministic': True,
                 'input': (DENSE, UNSIGNED_DATA),
                 'output': (PREDICTIONS,)}

autosklearn/pipeline/components/classification/gradient_boosting.py

@@ -154,6 +154,7 @@ def get_properties(dataset_properties=None):
                 'handles_classification': True,
                 'handles_multiclass': True,
                 'handles_multilabel': False,
+                'handles_multioutput': False,
                 'is_deterministic': True,
                 'input': (DENSE, UNSIGNED_DATA),
                 'output': (PREDICTIONS,)}

autosklearn/pipeline/components/classification/k_nearest_neighbors.py

@@ -48,6 +48,7 @@ def get_properties(dataset_properties=None):
                 'handles_classification': True,
                 'handles_multiclass': True,
                 'handles_multilabel': True,
+                'handles_multioutput': False,
                 'is_deterministic': True,
                 'input': (DENSE, SPARSE, UNSIGNED_DATA),
                 'output': (PREDICTIONS,)}

autosklearn/pipeline/components/classification/lda.py

@@ -70,6 +70,7 @@ def get_properties(dataset_properties=None):
                 'handles_classification': True,
                 'handles_multiclass': True,
                 'handles_multilabel': True,
+                'handles_multioutput': False,
                 'is_deterministic': True,
                 'input': (DENSE, UNSIGNED_DATA),
                 'output': (PREDICTIONS,)}

autosklearn/pipeline/components/classification/liblinear_svc.py

@@ -82,6 +82,7 @@ def get_properties(dataset_properties=None):
                 'handles_classification': True,
                 'handles_multiclass': True,
                 'handles_multilabel': True,
+                'handles_multioutput': False,
                 'is_deterministic': False,
                 'input': (SPARSE, DENSE, UNSIGNED_DATA),
                 'output': (PREDICTIONS,)}

autosklearn/pipeline/components/classification/libsvm_svc.py

@@ -115,6 +115,7 @@ def get_properties(dataset_properties=None):
             'handles_classification': True,
             'handles_multiclass': True,
             'handles_multilabel': False,
+            'handles_multioutput': False,
             'is_deterministic': True,
             'input': (DENSE, SPARSE, UNSIGNED_DATA),
             'output': (PREDICTIONS,)}

autosklearn/pipeline/components/classification/multinomial_nb.py

@@ -68,6 +68,7 @@ def get_properties(dataset_properties=None):
                 'handles_classification': True,
                 'handles_multiclass': True,
                 'handles_multilabel': True,
+                'handles_multioutput': False,
                 'is_deterministic': True,
                 'input': (DENSE, SPARSE, SIGNED_DATA),
                 'output': (PREDICTIONS,)}

autosklearn/pipeline/components/classification/passive_aggressive.py

@@ -139,6 +139,7 @@ def get_properties(dataset_properties=None):
                 'handles_classification': True,
                 'handles_multiclass': True,
                 'handles_multilabel': True,
+                'handles_multioutput': False,
                 'is_deterministic': True,
                 'input': (DENSE, SPARSE, UNSIGNED_DATA),
                 'output': (PREDICTIONS,)}

autosklearn/pipeline/components/classification/qda.py

@@ -64,6 +64,7 @@ def get_properties(dataset_properties=None):
                 'handles_classification': True,
                 'handles_multiclass': True,
                 'handles_multilabel': True,
+                'handles_multioutput': False,
                 'is_deterministic': True,
                 'input': (DENSE, UNSIGNED_DATA),
                 'output': (PREDICTIONS,)}

autosklearn/pipeline/components/classification/random_forest.py

@@ -123,6 +123,7 @@ def get_properties(dataset_properties=None):
                 'handles_classification': True,
                 'handles_multiclass': True,
                 'handles_multilabel': True,
+                'handles_multioutput': False,
                 'is_deterministic': True,
                 'input': (DENSE, SPARSE, UNSIGNED_DATA),
                 'output': (PREDICTIONS,)}

autosklearn/pipeline/components/classification/sgd.py

@@ -143,6 +143,7 @@ def get_properties(dataset_properties=None):
                 'handles_classification': True,
                 'handles_multiclass': True,
                 'handles_multilabel': False,
+                'handles_multioutput': False,
                 'is_deterministic': True,
                 'input': (DENSE, SPARSE, UNSIGNED_DATA),
                 'output': (PREDICTIONS,)}

autosklearn/pipeline/components/data_preprocessing/balancing/balancing.py

@@ -98,6 +98,7 @@ def get_properties(dataset_properties=None):
                 'handles_classification': True,
                 'handles_multiclass': True,
                 'handles_multilabel': True,
+                'handles_multioutput': False,
                 'is_deterministic': True,
                 'handles_sparse': True,
                 'handles_dense': True,

autosklearn/pipeline/components/data_preprocessing/categorical_encoding/no_encoding.py

@@ -22,6 +22,7 @@ def get_properties(dataset_properties=None):
                 'handles_classification': True,
                 'handles_multiclass': True,
                 'handles_multilabel': True,
+                'handles_multioutput': True,
                 'handles_sparse': True,
                 'handles_dense': True,
                 'input': (DENSE, SPARSE, UNSIGNED_DATA),

autosklearn/pipeline/components/data_preprocessing/categorical_encoding/one_hot_encoding.py

@@ -38,6 +38,7 @@ def get_properties(dataset_properties=None):
                 'handles_classification': True,
                 'handles_multiclass': True,
                 'handles_multilabel': True,
+                'handles_multioutput': True,
                 # TODO find out of this is right!
                 'handles_sparse': True,
                 'handles_dense': True,

autosklearn/pipeline/components/data_preprocessing/category_shift/category_shift.py

@@ -42,6 +42,7 @@ def get_properties(dataset_properties=None):
                 'handles_classification': True,
                 'handles_multiclass': True,
                 'handles_multilabel': True,
+                'handles_multioutput': True,
                 'is_deterministic': True,
                 # TODO find out of this is right!
                 'handles_sparse': True,

autosklearn/pipeline/components/data_preprocessing/imputation/categorical_imputation.py

@@ -41,6 +41,7 @@ def get_properties(dataset_properties=None):
                 'handles_classification': True,
                 'handles_multiclass': True,
                 'handles_multilabel': True,
+                'handles_multioutput': True,
                 'is_deterministic': True,
                 # TODO find out of this is right!
                 'handles_sparse': True,

autosklearn/pipeline/components/data_preprocessing/imputation/numerical_imputation.py

@@ -37,6 +37,7 @@ def get_properties(dataset_properties=None):
                 'handles_classification': True,
                 'handles_multiclass': True,
                 'handles_multilabel': True,
+                'handles_multioutput': True,
                 'is_deterministic': True,
                 # TODO find out if this is right!
                 'handles_sparse': True,

autosklearn/pipeline/components/data_preprocessing/minority_coalescense/minority_coalescer.py

@@ -38,6 +38,7 @@ def get_properties(dataset_properties=None):
                 'handles_classification': True,
                 'handles_multiclass': True,
                 'handles_multilabel': True,
+                'handles_multioutput': True,
                 # TODO find out of this is right!
                 'handles_sparse': True,
                 'handles_dense': True,

autosklearn/pipeline/components/data_preprocessing/minority_coalescense/no_coalescense.py

@@ -25,6 +25,7 @@ def get_properties(dataset_properties=None):
                 'handles_classification': True,
                 'handles_multiclass': True,
                 'handles_multilabel': True,
+                'handles_multioutput': True,
                 'handles_sparse': True,
                 'handles_dense': True,
                 'input': (DENSE, SPARSE, UNSIGNED_DATA),

autosklearn/pipeline/components/data_preprocessing/rescaling/minmax.py

@@ -22,6 +22,7 @@ def get_properties(dataset_properties=None):
                 'handles_classification': True,
                 'handles_multiclass': True,
                 'handles_multilabel': True,
+                'handles_multioutput': True,
                 'is_deterministic': True,
                 # TODO find out if this is right!
                 'handles_sparse': True,