This repository contains the implementation of FeNeC (Feature Neighborhood Classifier) and FeNeC-Log, as introduced in the paper FeNeC: Enhancing Continual Learning via Feature Clustering with Neighbor- or Logit-based Classification.
These methods address Class-Incremental Learning scenarios by leveraging feature-based representations, data clustering and distance metrics. FeNeC employs a nearest neighbor approach, while FeNeC-Log extends it with trainable parameters, enabling more flexible and adaptive classification.
Run the following command to install the required libraries listed in requirements.txt:
pip install -r requirements.txtPlace your dataset (feature representations from a model such as ResNet or ViT) into the data/ folder. Detailed guidelines on dataset structure and format are provided in the Data and Project Structure section.
Refer to the Jupyter notebooks for example usage and experiments:
cifar_vit.ipynb: Includes all evaluated methods on CIFAR datasets with ViT representations.cifar_resnet.ipynb: Focuses on FeNeC and FeNeC-Log with ResNet representations.
These notebooks walk through the usage and evaluation of our methods.
├── data/ # Place your dataset HDF5 files here
│ ├── dataset_name/
│ │ ├── task_0.hdf5
│ │ ├── task_1.hdf5
│ │ └── ...
│
├── FeNeC.py # FeNeC implementation
├── FeNeC_Log.py # FeNeC-Log implementation
├── MLPClassifier.py # Obsolete classifier
├── KMeans.py # Custom KMeans clustering implementation
├── Metrics.py # Distance metrics (Euclidean, Cosine, Mahalanobis)
├── utils.py # Helper functions (training, evaluation, visualization tools, etc.)
├── tests.py # Unit tests for core components
├── grid_searches/ # Optuna grid searches and experiments
│ ├── *.json # Search spaces and experiment configs
│ └── *.py # Scripts used to run Optuna experiments on a server
│
└── notebooks/ # Example Jupyter notebooks
├── cifar_vit.ipynb # All methods evaluated
├── cifar_resnet.ipynb # FeNeC and FeNeC-Log
├── 2d_embeddings.ipynb # Representations embedded with t-SNE and UMAP
├── kmeans.ipynb # Custom KMeans test
├── sample_mnist.ipynb # FeNeC with Mahalanobis tests
└── sample_moons.ipynb # Similar tests
Each task representation should be stored in separate .hdf5 file, for example: task_0.hdf5, task_1.hdf5, etc.
Each file must include:
X_train: training features (shape[num_samples, num_features])y_train: training labels (shape[num_samples])X_test: test features (shape[num_samples, num_features])y_test: test labels (shape[num_samples])
Note: These files are not provided in this repository.
An abstract class providing shared functionality for all classifiers.
- metric: Distance metric for classification (
Euclidean,Cosine,Mahalanobis). - data_normalization: Whether to normalize the data (important if the input features aren't already normalized).
- tukey_lambda: Lambda parameter for Tukey’s Ladder of Powers transformation. Controls the strength of the transformation and can be seen as a regularization force.
- kmeans: Optional custom KMeans object for clustering-based sample selection.
- device: Computation device (
'cpu'or'cuda'). - batch_size: Controls batch size for distance computations. Impacts speed and memory usage but does not affect outputs.
A nearest-neighbor classifier leveraging distance metrics for classification. No additional training is required beyond storing and accessing task data.
- n_neighbors: Number of neighbors considered when classifying new samples.
An extension of FeNeC that introduces trainable parameters and makes decisions based on calculated logits.
- optimizer: Optimization algorithm (e.g.,
'SGD'). - n_points: Number of representative points retained per class for future tasks.
- num_epochs: Number of training epochs.
- lr: Learning rate for the optimizer.
- early_stop_patience: Number of epochs to wait before early stopping if the validation loss does not improve.
- train_only_on_first_task: If
True, training is only performed on the first task with no further updates in future tasks. - mode:
0: Shared parameters across all classes (recommended and used in the paper).1: Separate parameters for each class (obsolete, generally worse results).
Additional hyperparameters are available for mode=1 (refer to the class documentation). Mode 0 is recommended.
A custom KMeans clustering algorithm, implemented in kmeans.py, a key component of both FeNeC and FeNeC-Log.
- n_clusters: Number of clusters to form.
- max_iter: Maximum number of iterations.
- tol: Tolerance for the stopping criterion; stops if centroids move less than this threshold.
- metric: Distance metric used (
Euclidean,Cosine,Mahalanobis). - seed: Random seed for reproducibility.
For examples and comparisons with scikit-learn's KMeans, refer to kmeans.ipynb.
Implemented in metrics.py, used in both FeNeC and FeNeC-Log.
- EuclideanMetric: Standard L2 distance.
- CosineMetric:
1 - cosine similarity. - MahalanobisMetric: Mahalanobis distance, requiring covariance matrix estimation.
Hyperparameters for Mahalanobis:
- shrinkage: Shrinkage type (
0: none,1: normal,2: double). - gamma_1: Diagonal shrinkage factor.
- gamma_2: Off-diagonal shrinkage factor.
- normalization: Whether to normalize the covariance matrix.
All hyperparameter search definitions and experiment configurations are located in the grid_searches/ folder.
*.json: Search spaces and experiment configurations for FeNeC and FeNeC-Log on various datasets.grid_search.py: Runs grid searches using Optuna.hyperparameter_importance.py: Analyzes hyperparameter importance from Optuna studies.multiple_runs.py: Executes multiple training runs for statistical analysis.
utils.py: Helper functions for training, evaluation, grid search, and visualization tools (e.g., plots, performance metrics).tests.py: Unit tests for core components to ensure correctness and stability.
For experimentation, visualization, and debugging:
cifar_resnet.ipynb: Experiments on CIFAR datasets with ResNet representations (FeNeC and FeNeC-Log).cifar_vit.ipynb: Experiments on CIFAR datasets with ViT representations (includes obsolete methods).2d_embeddings.ipynb: t-SNE and UMAP embeddings for visualization.kmeans.ipynb: Comparisons between the custom KMeans and scikit-learn’s implementation.sample_mnist.ipynb: Testing Mahalanobis distance and FeNeC on MNIST.sample_moons.ipynb: Experiments on synthetic datasets.
Note: If import issues arise in the notebooks, try moving them to the main directory.
Hubert Jastrzębski: Designed and implemented the core logic, including FeNeC and FeNeC-Log classifiers, distance metrics, and custom KMeans. Developed Jupyter notebooks, visualization tools, and unit tests.
Krzysztof Pniaczek: Led large-scale experiment execution, performance analysis, and hyperparameter optimization using Optuna. Integrated wandb for experiment tracking and visualization.
If you use our code, please cite the following paper:
@misc{ksiazek2025fenec,
title={FeNeC: Enhancing Continual Learning via Feature Clustering with Neighbor- or Logit-Based Classification},
author={Kamil Książek and Hubert Jastrzębski and Bartosz Trojan and Krzysztof Pniaczek and Michał Karp and Jacek Tabor},
year={2025},
eprint={2503.14301},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2503.14301},
}