add PertEval-scFM: Benchmarking Single-Cell Foundation Models for Perturbation Effect Prediction

[szalata]

evaluation paper https://www.biorxiv.org/content/10.1101/2024.10.02.616248v1.abstract

[aaronwtr]

Hi @szalata,

First of all, thank you for maintaining this fantastic centralised resource and for including us! Here are some details about PertEval-scFM.

With PertEval-scFM, we introduce a benchmark to assess the zero-shot utility of single-cell foundation models for transcriptomic perturbation prediction. Using SPECTRA, we generate train-test splits with increasing dissimilarity to simulate distribution shifts. This enables us to evaluate robustness against distribution shift—a crucial aspect that we argue is often overlooked in current evaluations.

To evaluate the models, we use MSE and AUSPC, with the latter reflecting robustness under distribution shift. Additionally, we analyse predictions in more detail using E-distance and predicted transcriptomic distributions across the top 20 DEGs, which help reveal failure modes of the models.

Our findings suggest that single-cell foundation model embeddings can, at best, capture average perturbation effects. However, these embeddings generally lack robustness to distribution shift. Furthermore, in ongoing work (to be shared soon as an update to our manuscript on bioRxiv), we demonstrate that the domain-specific model GEARS outperforms foundation model embeddings within our evaluation setup. This highlights that masked-language modeling on gene expression data without domain-specific inductive biases is insufficient for accurate transcriptomic perturbation prediction.

If you have any questions or need assistance presenting our benchmark in the repository, feel free to reach out. I’m happy to help!

[Paulos2411]

Hi Aaron,

thanks for your summary. Could you perhaps summarize the information in a way as it has been done on the website for your specific case.

https://theislab.github.io/single-cell-transformer-papers/

I will then add them or feel free to create a PR.

Thanks :)

[aaronwtr]

Done! See PR #78