notebooks/getting-started.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8efc6d60-f207-4e54-92b0-a6070b0158b4",
   "metadata": {},
   "source": [
    "# Getting Started\n",
    "\n",
    "In this notebook we're going to demonstrate how to use `cev` to compare (a) two _different_ embeddings of the same data and (b) two aligned embeddings of _different_ data.\n",
    "\n",
    "The embeddings we're exploring in this notebook represent single-cell surface proteomic data. In other words, each data point represents a individual cell whose surface protein expression was measured. The cells were then clustered into cellular phenotypes based on their protein expression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47c31bea-24b3-4d16-a69a-a3ad3a746234",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "from cev.widgets import Embedding, EmbeddingComparisonWidget"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dea71d70-e467-49af-9165-6e278f953977",
   "metadata": {},
   "source": [
    "The notebook requires downloading the three embeddings from data of from [Mair et al., 2022](https://www.nature.com/articles/s41586-022-04718-w):\n",
    "- Tissue sample 138 (32 MB) embedded with [UMAP](https://umap-learn.readthedocs.io/en/latest/)\n",
    "- Tissue sample 138 (32 MB) embedded with [UMAP](https://umap-learn.readthedocs.io/en/latest/) after being transformd with [Ozette's Annotation Transformation](https://github.com/flekschas-ozette/ismb-biovis-2022)\n",
    "- Tumor sample 6 (82 MB) embedded with [UMAP](https://umap-learn.readthedocs.io/en/latest/) after being transformd with [Ozette's Annotation Transformation](https://github.com/flekschas-ozette/ismb-biovis-2022)\n",
    "\n",
    "All three embeddings are annotated with [Ozette's FAUST method](https://doi.org/10.1016/j.patter.2021.100372)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbf802bc-f709-4163-9b49-8fa5f6ce59ab",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# download the data\n",
    "!curl -sL https://figshare.com/ndownloader/articles/23063615/versions/1 -o data.zip\n",
    "!unzip data.zip -d data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e62390d2-1242-49a8-9780-be976d39fa42",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Comparing Two Embeddings of the same Data\n",
    "\n",
    "In the first example, we are going to use `cev` to compare two different embeddings methods that were run on the very same data (the tissue sample): standard UMAP and annotation transformation UMAP.\n",
    "\n",
    "Different embedding methods can produce very different embedding spaces and it's often hard to assess the difference wholelistically. `cev` enables us to quantify two properties based on shared point labels:\n",
    "\n",
    "1. Confusion: the degree to which two or more labels are visually intermixed\n",
    "2. Neighborhood: the degree to which the local neighborhood of a label has changed between the two embeddings\n",
    "\n",
    "Visualized as a heatmap, these two property can quickly guide us to point clusters that are better or less resolved in either one of the two embeddings. It can also help us find compositional changes between the two embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7874813c-810f-40e5-92ab-91f228046a5e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "tissue_umap_embedding = Embedding.from_ozette(\n",
    "    df=pd.read_parquet(\"./data/mair-2022-tissue-138-umap.pq\")\n",
    ")\n",
    "tissue_ozette_embedding = Embedding.from_ozette(\n",
    "    df=pd.read_parquet(\"./data/mair-2022-tissue-138-ozette.pq\")\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3d7e114-9fd3-4785-bdca-e3f4bbf37df8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "umap_vs_ozette = EmbeddingComparisonWidget(\n",
    "    tissue_umap_embedding,\n",
    "    tissue_ozette_embedding,\n",
    "    titles=[\"Standard UMAP (Tissue)\", \"Annotation-Transformed UMAP (Tissue)\"],\n",
    "    metric=\"confusion\",\n",
    "    selection=\"synced\",\n",
    "    auto_zoom=True,\n",
    "    row_height=320,\n",
    ")\n",
    "umap_vs_ozette"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a516d65a-351b-4365-a267-704cd93a9c0e",
   "metadata": {},
   "source": [
    "In this example, we can see that the point labels are much more intermixed in the standard UMAP embedding compared to the annotation transformation UMAP. This not surprising as the standard UMAP embedding is not optimized for Flow cytometry data in any way and is thus only resolving broad cell phenotypes based on a few markers. You can see this by holding down `SHIFT` and clicking on `CD8` under _Markers_, which reduces the label resolution and shows that under a reduced label resolution, the confusion is much lower in the standard UMAP embedding.\n",
    "\n",
    "When selecting _Neighborhood_ from the _Metric_ drop down menu, we switch to the neighborhood composition difference quantification. When only a few markers (e.g., `CD4` and `CD8`) are active, we can see that most of the neighborhood remain unchanged. When we gradually add more markers, we can see how the the local neighborhood composition difference slowly increases, which is due to the fact that the annotation transformation spaces out all point label clusters.\n",
    "\n",
    "To study certain clusters or labels in detail, you can either interactively select points in the embedding via [jupyter-scatter](https://github.com/flekschas/jupyter-scatter)'s lasso selection or you can programmatically select points by their label via the `select()`. For instance, the next call will select all CD4+ T cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba7a378f-4212-4953-be5b-7a273f8bc75e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "umap_vs_ozette.select([\"CD3+\", \"CD4+\", \"CD8-\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c439e4d-0679-4e64-a1c7-4be93cbbe039",
   "metadata": {},
   "source": [
    "## Size Differences Between _Non-Responder_ and _Responder_\n",
    "\n",
    "Instead of comparing identical data, let's take a look at two transformed and aligned embeddings: tissue vs tumor. The embeddings are both annotation-transformed and aligned, ensuring low confusion and high neighborhood similarity (check to confirm!). The abundance metric aids in identifying potential shifts in phenotype abundance, providing a comprehensive and visually intuitive method for analyzing complex cytometry data. Remember, our metric should be used as a exploratory tool guide exploration and quickly surface potentially interesting phenotypes, but robust statical methods must be applied to confirm whether any abundance differences exist."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "180f0945-d97c-4261-aa67-5368e3b560ad",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "tumor_ozette_embedding = Embedding.from_ozette(\n",
    "    df=pd.read_parquet(\"./data/mair-2022-tumor-006-ozette.pq\")\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f99361b-6e96-4a6d-ad65-0533c23bece7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "tissue_vs_tumor = EmbeddingComparisonWidget(\n",
    "    tissue_ozette_embedding,\n",
    "    tumor_ozette_embedding,\n",
    "    titles=[\"Tissue\", \"Tumor\"],\n",
    "    metric=\"abundance\",\n",
    "    selection=\"phenotype\",\n",
    "    auto_zoom=True,\n",
    "    row_height=320,\n",
    ")\n",
    "\n",
    "tissue_vs_tumor"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d632c95-dff8-4b90-b763-f3055c4e8047",
   "metadata": {
    "tags": []
   },
   "source": [
    "The following **CD8+ T cells** are more abundant in `tissue` (i.e., the relative abundance is higher on the left) compared to `tumor` (i.e., the relative abundance is lower on the right)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f7ebd73-32e7-48ed-8575-8d14d2edc73f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "tissue_vs_tumor.select(\n",
    "    \"CD4-CD8+CD3+CD45RA+CD27+CD19-CD103-CD28-CD69+PD1+HLADR-GranzymeB-CD25-ICOS-TCRgd-CD38-CD127-Tim3-\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eefac753-7920-4c87-99ef-d155f1ec5114",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}