LdaBoost Analysis

Overview

This repository accompanies the paper:

Does Gradient Boosting improve with Fisher’s Discriminant Analysis? Proposal of an integrated approach and empirical assessment of five popular datasets.

Authors: Giulio Enzo Donninelli, Pietro Giorgio Lovaglio — University of Milano‑Bicocca, Department of Statistics and Quantitative Methods & CRISP Research Center

The project investigates how dimensionality reduction techniques — Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA) — affect Gradient Boosting Machines (GBM). It introduces LdaBoost, an integrated boosting algorithm that iteratively computes LDA projections aligned with the current boosting residuals. The goal is to exploit LDA’s linear class‑separating power in tandem with GBM’s non‑linear modeling capacity.

What’s inside

• Standard GBM baselines and pipelines with PCA and LDA.
• LdaBoost: an iterative algorithm integrating LDA within the boosting loop.
• Empirical evaluations on five benchmark datasets (Iris, Sonar, Human Activity Recognition, Yeast, Rainfall).
• Controlled simulation studies exploring dimensionality, feature correlation, and the Hughes phenomenon.
• Result artifacts (plots, tables, summaries) generated by notebooks and scripts.

Key Ideas

• LDA features can consistently improve GBM, especially in multiclass and high‑dimensional binary setups.
• LdaBoost updates LDA projections at each boosting step to follow residuals, improving separability where it matters most.
• Simulations probe the effect of dimensionality and correlation, illustrating robustness and conditions where LdaBoost outperforms classical GBM.

Repository Map

Top‑level packages and folders:

• LdaBoost/

  • algorithm.py: Core implementation of the LdaBoost algorithm (integration of LDA into boosting steps).
  • __init__.py: Package exports.

• simulations/

  • LdaBoosting/: Alternate or exploratory implementation namespace (mirrors core logic used in synthetic studies).
    • algorithm.py, LdaBoost.py, __init__.py: Simulation‑focused implementation/entry points.
  • simulations_different_pipelines.ipynb: Synthetic experiments comparing GBM, PCA+GBM, LDA+GBM, and LdaBoost.
  • simulations_LdaBoost_Bayes_estimator.ipynb: Experiments with Bayes‑optimal references to contextualize achievable accuracy.
  • plots.ipynb: Utilities to aggregate and visualize simulation outputs.
  • optuna_lda_boost_synth/: Example hyperparameter tuning outputs for synthetic datasets (e.g., best trials, metrics).
  • output_lda_boost/: Aggregated CSV summaries (means/stds) from LdaBoost simulation runs.
  • output_pipeline_confront/: Raw and summarized results comparing pipelines (CSV/JSON).
  • output_bayes_optimal/: Bayes‑optimal summaries used as references.
  • img/, img_without_bayes/, img_pipeline_confront/: Pre‑rendered figures from simulations.

• real_datasets/

  • iris/, sonar/, har/ (Human Activity Recognition), yeast/, rain/:
    • pipelines.ipynb: End‑to‑end experiments per dataset (baseline GBM, PCA+GBM, LDA+GBM, LdaBoost).
    • tuning.ipynb (where present): Hyperparameter exploration for the corresponding dataset.
    • external.ipynb (where present): Data ingestion, feature preparation, or external checks.
    • Data/: Local copies of dataset CSVs used by the notebooks.

• common.txt: Shared notes, scratch, or checklist material used during analysis.

Getting Started

This repository is organized around notebooks. A typical workflow is:

1. Open a dataset notebook, e.g. real_datasets/har/pipelines.ipynb.
2. Run the baseline pipeline (GBM), then PCA+GBM and LDA+GBM.
3. Enable/execute the LdaBoost pipeline leveraging the implementation in LdaBoost/algorithm.py.
4. Optionally run tuning.ipynb for that dataset to explore hyperparameters.
5. For synthetic studies, run simulations/simulations_different_pipelines.ipynb and review generated artifacts in simulations/output_* and simulations/img*.

Notes

• Datasets: CSVs are stored under each dataset’s Data/ directory to keep notebooks self‑contained.
• Results: Notebooks write intermediate and final artifacts to the respective output_* and img* folders for later aggregation and plotting.
• Reproducibility: Many notebooks can be re‑run end‑to‑end; for heavy simulations, summaries and plots are already saved.

Dependencies

The notebooks rely on a standard scientific Python stack. At minimum, expect:

• numpy, pandas, scikit‑learn
• a boosting library (e.g., scikit‑learn’s GradientBoosting or compatible GBM implementation)
• matplotlib and/or seaborn
• optuna (for tuning notebooks and optuna_lda_boost_synth artifacts)

Please install via your environment manager of choice and run notebooks with the same interpreter.

How LdaBoost Works (High level)

• Initialize the boosting model and fit a base learner.
• At each iteration, compute residuals and derive LDA projections tailored to those residuals.
• Augment or transform features using the LDA directions, then fit the next weak learner on the enriched representation.
• Iterate until the desired number of boosting rounds is reached.

Results Overview

• Real datasets (Iris, Sonar, HAR, Yeast, Rainfall): LDA features typically improve accuracy over baseline GBM; LdaBoost often yields further gains, notably in multiclass tasks.
• Simulations: Increasing dimensionality and correlation stress standard GBM; LdaBoost shows robustness and can outperform baselines under these conditions. The Hughes phenomenon emerges clearly with very high dimensional feature spaces.

Citation

If you use this code or results, please cite the paper:

Giulio Enzo Donninelli, Pietro Giorgio Lovaglio. Does Gradient Boosting improve with Fisher’s Discriminant Analysis? Proposal of an integrated approach and empirical assessment of five popular datasets. University of Milano‑Bicocca, Department of Statistics and Quantitative Methods & CRISP Research Center.

Contact

For questions or collaborations, please open an issue or reach out to the authors.