ESPnet2/Course/CMU_SpeechProcessing_Spring2023/assignment3_spk.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k8GMfsH2mRND"
      },
      "source": [
        "# CMU 11492/11692 Spring 2023: Speaker Recognition\n",
        "\n",
        "In this demonstration, we will show you the procedure to conduct speaker recognition with the ASR functions of ESPnet.\n",
        "\n",
        "Main references:\n",
        "- [ESPnet repository](https://github.com/espnet/espnet)\n",
        "- [ESPnet documentation](https://espnet.github.io/espnet/)\n",
        "\n",
        "Author:\n",
        "- Jiatong Shi (jiatongs@andrew.cmu.edu)\n",
        "\n",
        "## Objectives\n",
        "\n",
        "After this demonstration, you are expected to understand the main procedure of using ESPnet ASR functions for speaker recognition.\n",
        "\n",
        "## ❗Important Notes❗\n",
        "- We are using Colab to show the demo. However, Colab has some constraints on the total GPU runtime. If you use too much GPU time, you may not be able to use GPU for some time.\n",
        "- There are multiple in-class checkpoints ✅ throughout this tutorial. **Your participation points are based on these tasks.** Please try your best to follow all the steps! If you encounter issues, please notify the TAs as soon as possible so that we can make an adjustment for you.\n",
        "- Please submit PDF files of your completed notebooks to Gradescope. You can print the notebook using `File -> Print` in the menu bar."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1Ftl74aZnBrH"
      },
      "source": [
        "## ESPnet installation\n",
        "\n",
        "We follow the ESPnet installation as the previous tutorials (takes around 15 minutes)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3p6lJ3YNmArH"
      },
      "outputs": [],
      "source": [
        "!git clone --depth 5 -b 2023spring_speaker_recognition https://github.com/espnet/espnet\n",
        "\n",
        "%cd /content/espnet/tools\n",
        "!./setup_anaconda.sh anaconda espnet 3.9\n",
        "\n",
        "# # It may take 12 minutes\n",
        "%cd /content/espnet/tools\n",
        "!make TH_VERSION=1.12.1 CUDA_VERSION=11.6\n",
        "\n",
        "!. ./activate_python.sh && installers/install_speechbrain.sh\n",
        "!. ./activate_python.sh && installers/install_rawnet.sh\n",
        "!. ./activate_python.sh && pip install ipykernel"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C0G94uP6qVTz"
      },
      "source": [
        "## Speaker Recognition\n",
        "\n",
        "Speaker recognition is a typical task that conduct utterance-level classification.  Specifically, we will map an utterance into a pre-defined category. Recall that the ASR is doing a sequence-to-sequence task, so we can easily utilize ASR by using a 1-length sequence (i.e., class). Following this concept, we can start to implement the speaker recognition system! Noted that following the definition of the lecture, today, we will focus on **speaker identification** (close-set classification) instead of **speaker verification**.\n",
        "\n",
        "## Dataset\n",
        "\n",
        "`mini_librispeech` is a tiny subset of `librispeech` dataset for development usage. Because of the free-license and cleaness of the data, `librispeech` has been one of the most widely used corpora in the speech community. For more details, please refer to its [original paper](http://www.danielpovey.com/files/2015_icassp_librispeech.pdf). In this demonstration, we will use the train set of `mini_librispeech` to train and test a simple speaker recognition model.\n",
        "\n",
        "First of all, let's get into the directory to check the structure."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0YTatDXCsWOD"
      },
      "outputs": [],
      "source": [
        "%cd /content/espnet/egs2/mini_librispeech/sid1\n",
        "!ls -l"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Data Preparation\n",
        "\n",
        "Similar to the previous tutorials, we will use the Kaldi-style format for the data preparation. The differences in this recipe is that we need to predict speaker ID instead of predicting transcription. Therefore, a straightforward process is to simply change the `text` into `utt2spk`.\n",
        "\n",
        "So final files after preparation should be:\n",
        "```\n",
        "wav.scp text utt2spk spk2utt\n",
        "```\n",
        "But on the other hand, we change the format of text into\n",
        "```\n",
        "utt_id1 spk_id0\n",
        "utt_id2 spk_id0\n",
        "utt_id3 spk_id1\n",
        "```\n",
        "where `spk_id0` and `spk_id1` refers to the speaker IDs"
      ],
      "metadata": {
        "id": "wXaFrm070-90"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!./run.sh --stage 1 --stop_stage 1"
      ],
      "metadata": {
        "id": "ek_XkPaD1qTF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Data Preprocessing\n",
        "\n",
        "For data preprocessing,  we follow the similar way in previous tutorials/assignments."
      ],
      "metadata": {
        "id": "Ek-AIkFt13B3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!./run.sh --stage 2 --stop_stage 5"
      ],
      "metadata": {
        "id": "fksm71Ht29vo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Question1  (✅ Checkpoint 1 (1 point))\n",
        "In previous tutorials, we usually use character as our modeling units. But for here, we use a speaker id, which is a sequence of character, representing one speaker. So, in our preprocessing, which tokenizer (e.g., char, bpe, phn, word) is actually used to achieve speaker prediction? Please also indicate your reason(s).\n",
        "\n",
        "To help you understand more, please check the documentation at https://espnet.github.io/espnet/search.html?q=tokenizer&check_keywords=yes&area=default\n",
        "\n",
        "(For question-based checkpoint: please directly answer it in the text box)\n",
        "\n",
        "[ANSWER HERE]\n"
      ],
      "metadata": {
        "id": "uRrO_b8t3J42"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Use Pre-trained speaker representation\n",
        "\n",
        "One feature in ESPnet is to adopt pre-trained speaker representation from other toolkits (including TDNN-based speaker embedding extraction from [speechbrain](https://github.com/speechbrain/speechbrain) and RawNet-based speaker embedding from [RawNet](https://github.com/Jungjee/RawNet). We can efficiently extract the speaker embedding with our supported scripts.\n",
        "\n",
        "The speaker embedding can be used for text-to-speech purpose to handle multi-speaker synthesis. In this demonstration, we directly use the extraction model for speaker recognition."
      ],
      "metadata": {
        "id": "OClNDbFb478r"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!cat ./local/extract_xvector.sh"
      ],
      "metadata": {
        "id": "OrfTBuyv64ZD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!./local/extract_xvector.sh"
      ],
      "metadata": {
        "id": "lMm4raIN7oXp"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "After calculating the xvectors, we also can analysis the embedding by [t-SNE algorithm](https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding). The t-sne image is located at the extracted xvector folder"
      ],
      "metadata": {
        "id": "_JPCIqqTjrXT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from IPython.display import Image, display\n",
        "display(Image('dump/extracted/train/tsne.png'))\n"
      ],
      "metadata": {
        "id": "7wYhC0v0j-iJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Extract speaker embedding from SpeechBrain\n",
        "\n",
        "Similarly, we can also extract speaker embedding from speechbrain."
      ],
      "metadata": {
        "id": "fHoH5c6-_UaE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!cat ./local/extract_xvector_speechbrain.sh\n",
        "!./local/extract_xvector_speechbrain.sh"
      ],
      "metadata": {
        "id": "smgjXsrq_sKf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Similar to the speechbrain-based embedding, we can visualize the embeddings from RawNet with t-SNE plot."
      ],
      "metadata": {
        "id": "v8N-BzIwjj-W"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from IPython.display import Image, display\n",
        "display(Image('dump/extracted_speechbrain/train/tsne.png'))"
      ],
      "metadata": {
        "id": "sgQVp2rFlBCM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Training for speaker recognition\n",
        "\n",
        "First, let's use xvector trained from TDNN (speech-brain model) to conduct speaker recognition."
      ],
      "metadata": {
        "id": "wtHZt5GjAD0L"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!cat ./run_xvector_speechbrain.sh\n",
        "!./run_xvector_speechbrain.sh"
      ],
      "metadata": {
        "id": "SbzOTy9TBIcq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Question2  (✅ Checkpoint 2 (0.5 point))\n",
        "We still use the ASR scoring scheme for our evaluation because it is already sufficient. Please briefly discuss which metric can be used for evaluation of the accuracy/error rate of speaker recognition results.\n",
        "\n",
        "(For question-based checkpoint: please directly answer it in the text box)\n",
        "\n",
        "[ANSWER HERE]\n"
      ],
      "metadata": {
        "id": "0WEW4vaOBXVE"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Then, let's use RawNet-based xvector to conduct speaker recognition"
      ],
      "metadata": {
        "id": "5dZcYJC0BuWr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!cat ./run_xvector.sh\n",
        "!./run_xvector.sh"
      ],
      "metadata": {
        "id": "pboFXwTABzZh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Question3  (✅ Checkpoint 3 (0.5 point))\n",
        "Clearly, we find some differences in the number between TDNN-based speaker embedding and RawNet-based speaker embedding. Could you briefly exaplin some possible reasons that why we could get such different results?\n",
        "\n",
        "References:\n",
        "- [RawNet](https://arxiv.org/abs/2203.08488)\n",
        "- [Xvector (TDNN-based)](https://www.danielpovey.com/files/2018_icassp_xvectors.pdf)\n",
        "\n",
        "(For question-based checkpoint: please directly answer it in the text box)\n",
        "\n",
        "[ANSWER HERE]\n"
      ],
      "metadata": {
        "id": "cHpF5W3ZB3tH"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can also use ESPnet ASR model directly for speaker recognition purpose by predicting the target as speaker ID."
      ],
      "metadata": {
        "id": "VES2lQ5LCT74"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!./run.sh --stage 10"
      ],
      "metadata": {
        "id": "a1ex27kLCSdK"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Question4  (✅ Checkpoint 4 (0.5 point))\n",
        "We could get reasonable performances with the ASR model. However, we could easily find that the training is much more time-consuming than those with speaker embeddings. Could you please explain why we have such differences?\n",
        "\n",
        "(For question-based checkpoint: please directly answer it in the text box)\n",
        "\n",
        "[ANSWER HERE]"
      ],
      "metadata": {
        "id": "JTPdygRQCmr4"
      }
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU",
    "gpuClass": "standard"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}