Lighter, Better, Faster Multi-Source Domain Adaptation with Gaussian Mixture Models for optimal Transport

In this repository, we provide the source code for our paper "Lighter, Better, Faster Multi-Source Domain Adaptation with Gaussian Mixture Models for optimal Transport", Accepted at the European Conference on Machine Learning and Principles and Practices of Knowledge Discovery in Databases (ECML-PKDD'24), which proposes new tools for multi-source domain adaptation through Gaussian Mixture Model-based OT. Especially, here you will find our source code for reproducing the toy example in Section 4.1 of our main paper.

Abstract

In this paper, we tackle Multi-Source Domain Adaptation (MSDA), a task in transfer learning where one adapts multiple heterogeneous, labeled source probability measures towards a different, unlabeled target measure. We propose a novel framework for MSDA, based on Optimal Transport (OT) and Gaussian Mixture Models (GMMs). Our framework has two key advantages. First, OT between GMMs can be solved efficiently via linear programming. Second, it provides a convenient model for supervised learning, especially classification, as components in the GMM can be associated with existing classes. Based on the GMM-OT problem, we propose a novel technique for calculating barycenters of GMMs. Based on this novel algorithm, we propose two new strategies for MSDA: GMM-Wasserstein Barycenter Transport (WBT) and GMM-Dataset Dictionary Learning (DaDiL). We empirically evaluate our proposed methods on four benchmarks in image classification and fault diagnosis, showing that we improve over the prior art while being faster and involving fewer parameters.

Keyword. Domain Adaptation, Optimal Transport, Gaussian Mixture Models

Contributions

We summarize our contributions as follows,

1. We propose a novel strategy for mapping the parameters of GMMs using OT.
2. We propose a novel algorithm for computing mixture-Wasserstein barycenters of GMMs.
3. We propose an efficient parametric extension of the WBT and DaDiL algorithms based on GMMs.

Results

Office Home

Algorithm	Ar	Cl	Pr	Rw	Avg. $\uparrow$
ResNet101	72.90	62.20	83.70	85.00	75.95
M$^{3}$SDA	71.13	61.41	80.18	80.64	73.34
LtC-MSDA	74.52	60.56	85.52	83.63	76.05
KD3A	73.80	63.10	84.30	83.50	76.17
Co-MDA$^{\ddag}$	74.40	64.00	85.30	83.90	76.90
WJDOT	74.28	63.80	83.78	84.52	76.59
WBT	75.72	63.80	84.23	84.63	77.09
DaDiL-E	77.16	64.95	85.47	84.97	78.14
DaDiL-R	75.92	64.83	85.36	85.32	77.86
GMM-WBT	75.31	64.26	86.71	85.21	77.87
GMM-DaDiL	77.16	66.21	86.15	85.32	78.81

Office 31

Algorithm	A	D	W	Avg. $\uparrow$
ResNet50	67.50	95.00	96.83	86.40
M$^{3}$SDA	66.75	97.00	96.83	86.86
LtC-MSDA	66.82	100.00	97.12	87.98
KD3A	65.20	100.0	98.70	87.96
Co-MDA	64.80	99.83	98.70	87.83
WJDOT	67.77	97.32	95.32	86.80
WBT	67.94	98.21	97.66	87.93
DaDiL-E	70.55	100.0	98.83	89.79
DaDiL-R	70.90	100.0	98.83	89.91
GMM-WBT	70.13	99.11	96.49	88.54
GMM-DaDiL	72.47	100.0	99.41	90.63

CWRU

Algorithm	A	B	C	Avg. $\uparrow$
MLP$^{\star}$	70.90 $\pm$ 0.40	79.76 $\pm$ 0.11	72.26 $\pm$ 0.23	74.31
M3SDA	56.86 $\pm$ 7.31	69.81 $\pm$ 0.36	61.06 $\pm$ 6.35	62.57
LTC-MSDA$^{\star}$	82.21 $\pm$ 8.03	75.33 $\pm$ 5.91	81.04 $\pm$ 5.45	79.52
KD3A	81.02 $\pm$ 2.92	78.04 $\pm$ 4.05	74.64 $\pm$ 5.65	77.90
Co-MDA	62.66 $\pm$ 0.96	55.78 $\pm$ 0.85	76.35 $\pm$ 0.79	64.93
WJDOT	99.96 $\pm$ 0.02	98.86 $\pm$ 0.55	100.0 $\pm$ 0.00	99.60
WBT$^{\star}$	99.28 $\pm$ 0.18	79.91 $\pm$ 0.04	97.71 $\pm$ 0.76	92.30
DaDiL-R$^{\star}$	99.86 $\pm$ 0.21	99.85 $\pm$ 0.08	100.00 $\pm$ 0.00	99.90
DaDiL-E$^{\star}$	93.71 $\pm$ 6.50	83.63 $\pm$ 4.98	99.97 $\pm$ 0.05	92.33
GMM-WBT	100.00 $\pm$ 0.00	99.95 $\pm$ 0.07	100.00 $\pm$ 0.00	99.98
GMM-DaDiL	100.00 $\pm$ 0.00	99.95 $\pm$ 0.04	100.00 $\pm$ 0.00	99.98

TEP

Algorithm	Mode 1	Mode 2	Mode 3	Mode 4	Mode 5	Mode 6	Avg. $\uparrow$
CNN$^{\dag}$	80.82 $\pm$ 0.96	63.69 $\pm$ 1.71	87.47 $\pm$ 0.99	79.96 $\pm$ 1.07	74.44 $\pm$ 1.52	84.53 $\pm$ 1.12	78.48
M$^{3}$SDA$^{\dag}$	81.17 $\pm$ 2.00	61.61 $\pm$ 2.71	79.99 $\pm$ 2.71	79.12 $\pm$ 2.41	75.16 $\pm$ 3.01	78.91 $\pm$ 3.24	75.99
LtC-MSDA$^{1}$	-	-	-	-	-	-	-
KD3A	72.52 $\pm$ 3.04	18.96 $\pm$ 4.54	81.02 $\pm$ 2.40	74.42 $\pm$ 1.60	67.18 $\pm$ 2.37	78.22 $\pm$ 2.14	65.38
Co-MDA	64.56 $\pm$ 0.62	35.99 $\pm$ 1.21	79.66 $\pm$ 1.36	72.06 $\pm$ 1.66	66.33 $\pm$ 0.97	78.91 $\pm$ 1.87	66.34
WJDOT	89.06 $\pm$ 1.34	75.60 $\pm$ 1.84	89.99 $\pm$ 0.86	89.38 $\pm$ 0.77	85.32 $\pm$ 1.29	87.43 $\pm$ 1.23	86.13
WBT$^{\dag}$	92.38 $\pm$ 0.66	73.74 $\pm$ 1.07	88.89 $\pm$ 0.85	89.38 $\pm$ 1.26	85.53 $\pm$ 1.35	86.60 $\pm$ 1.63	86.09
DaDiL-R$^{\ddag}$	91.97 $\pm$ 1.22	77.15 $\pm$ 1.32	85.41 $\pm$ 1.69	89.39 $\pm$ 1.03	84.49 $\pm$ 1.95	88.44 $\pm$ 1.29	86.14
DaDiL-E$^{\ddag}$	90.45 $\pm$ 1.02	77.08 $\pm$ 1.21	86.79 $\pm$ 2.14	89.01 $\pm$ 1.35	84.04 $\pm$ 3.16	87.85 $\pm$ 1.06	85.87
GMM-WBT	92.23 $\pm$ 0.70	71.81 $\pm$ 1.78	84.72 $\pm$ 1.92	89.28 $\pm$ 1.55	87.51 $\pm$ 1.73	82.49 $\pm$ 1.81	84.67
GMM-DaDiL	91.72 $\pm$ 1.41	76.41 $\pm$ 1.89	89.68 $\pm$ 1.49	89.18 $\pm$ 1.17	86.05 $\pm$ 1.46	88.02 $\pm$ 1.12	86.85

Citation

@article{montesuma2024lighter,
  title={Lighter, Better, Faster Multi-Source Domain Adaptation with Gaussian Mixture Models and Optimal Transport},
  author={Montesuma, Eduardo Fernandes and Mboula, Fred Ngol{\`e} and Souloumiac, Antoine},
  journal={arXiv preprint arXiv:2404.10261},
  year={2024}
}