explanations/pytorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PyTorch\n",
    "\n",
    "In the last few lessons, we learned how to build and optimize neural network architectures.  This gave us a grounding in how data flows between layers, how parameters get adjusted, and how loss decreases.  So far, we've been using NumPy to build and optimize our networks.  In this lesson, we'll learn about PyTorch, a framework that makes building and applying neural networks much simpler.\n",
    "\n",
    "We'll start off by taking a look at how PyTorch represents data, and we'll move to building a complete neural network in PyTorch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tensors\n",
    "\n",
    "We'll first load in the same house prices dataset from the last lesson.  Each row in this dataset represents a single house.  The predictor columns are:\n",
    "\n",
    "- `interest`: The interest rate\n",
    "- `vacancy`: The vacancy rate\n",
    "- `cpi`: The consumer price index\n",
    "- `price`: The price of a house\n",
    "- `value`: The value of a house\n",
    "- `adj_price`: The price of a house, adjusted for inflation\n",
    "- `adj_value`: The value of a house, adjusted for inflation\n",
    "\n",
    "The predictor columns have all been scaled using the scikit-learn `StandardScaler`.  This gives each column a mean of 0 and a standard deviation of 1.  This makes it easier to activate our nonlinearities.\n",
    "\n",
    "The target column is `next_quarter`, which is the price of the house in three months.  `next_quarter` has been scaled so the minimum value is `0`, and it has been divided by `1000` and rounded to the nearest integer.  This makes the prediction task simpler for our network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys, os\n",
    "sys.path.append(os.path.abspath('../data'))\n",
    "from csv_data import HousePricesDatasetWrapper\n",
    "\n",
    "# Load in data from csv file\n",
    "wrapper = HousePricesDatasetWrapper()\n",
    "train_data, valid_data, test_data = wrapper.get_flat_datasets()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data is currently loaded into NumPy arrays.  We can instead load the data into torch tensors.  Tensors are n-dimensional data structures similar to NumPy arrays.  The primary difference is that torch tensors can be loaded onto different devices, like GPUs.  We'll discuss this more later.\n",
    "\n",
    "For now, we'll load our training set predictors and targets into torch tensors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import torch.  You can install it with pip install torch.\n",
    "import torch\n",
    "\n",
    "# Convert the numpy arrays to torch tensors\n",
    "train_x = torch.from_numpy(train_data[0])\n",
    "train_y = torch.from_numpy(train_data[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.9451,  1.3964, -1.5228,  ..., -0.1168, -0.1389,  0.8226],\n",
       "        [ 1.9325,  1.3964, -1.4935,  ..., -0.1168, -0.1560,  0.8022],\n",
       "        [ 1.9955,  1.3964, -1.4935,  ..., -0.1168, -0.0446,  0.8022],\n",
       "        ...,\n",
       "        [-0.2595, -0.6860,  0.5061,  ...,  0.3840,  0.4345,  0.3539],\n",
       "        [-0.2469, -0.6860,  0.5061,  ...,  0.3840,  0.4217,  0.3539],\n",
       "        [-0.1839, -0.6860,  0.5061,  ...,  0.3840,  0.6257,  0.3539]],\n",
       "       dtype=torch.float64)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tensors work very similarly to NumPy arrays.  For example, you can do operations using scalars:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 2.9451,  2.3964, -0.5228,  ...,  0.8832,  0.8611,  1.8226],\n",
       "        [ 2.9325,  2.3964, -0.4935,  ...,  0.8832,  0.8440,  1.8022],\n",
       "        [ 2.9955,  2.3964, -0.4935,  ...,  0.8832,  0.9554,  1.8022],\n",
       "        ...,\n",
       "        [ 0.7405,  0.3140,  1.5061,  ...,  1.3840,  1.4345,  1.3539],\n",
       "        [ 0.7531,  0.3140,  1.5061,  ...,  1.3840,  1.4217,  1.3539],\n",
       "        [ 0.8161,  0.3140,  1.5061,  ...,  1.3840,  1.6257,  1.3539]],\n",
       "       dtype=torch.float64)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_x + 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One important difference is that you want to make sure to use torch functions instead of NumPy methods.  This ensures that the operation is done on the appropriate device.  There are torch equivalents for most NumPy functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1.3947, 1.1817,    nan,  ...,    nan,    nan, 0.9070],\n",
       "        [1.3901, 1.1817,    nan,  ...,    nan,    nan, 0.8957],\n",
       "        [1.4126, 1.1817,    nan,  ...,    nan,    nan, 0.8957],\n",
       "        ...,\n",
       "        [   nan,    nan, 0.7114,  ..., 0.6196, 0.6591, 0.5949],\n",
       "        [   nan,    nan, 0.7114,  ..., 0.6196, 0.6494, 0.5949],\n",
       "        [   nan,    nan, 0.7114,  ..., 0.6196, 0.7910, 0.5949]],\n",
       "       dtype=torch.float64)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Take the square root of each value in the array.  Negative values have an undefined square root.\n",
    "torch.sqrt(train_x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Autograd\n",
    "\n",
    "One big advantage that Torch has over NumPy for deep learning is autograd.  Autograd will automatically calculate the gradient, without you having to write a backward pass!\n",
    "\n",
    "To do this, we first need to define that parameter that we want a gradient for, then set `requires_grad` to `True`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a matrix of weights\n",
    "# Torch.rand generates random numbers\n",
    "weights = torch.rand(train_x.shape[1], 1)\n",
    "# Set requires_grad to True so that autograd can work\n",
    "weights.requires_grad = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can load in our training data and multiply it by the weights.  You may have noticed above that our `train_x` tensor is in `float64`.  This is because the NumPy arrays were in `float64`.  `float64` means that each number is stored using `64` bits of data.  In PyTorch, the default tends to be `float32`, which uses `32` bits to store each number.\n",
    "\n",
    "The main difference is the range of possible values that the number can store."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.7976931348623157e+308"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Display the maximum value of float64\n",
    "np.finfo(\"float64\").max"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.4028235e+38"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display the maximum value of float32\n",
    "np.finfo(\"float32\").max"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`float32` can store large enough numbers that we rarely have issues when training deep learning models.  Thus, it's much more common to work with `float32` in PyTorch.  There are also times when you'll work with `float16` or `int8`, and we'll cover those in a later lesson.\n",
    "\n",
    "We'll convert our array to `float32`, which is just `torch.float`, since it's the default.  We can then multiply the weights and the `train_x` values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_x = train_x.to(torch.float)\n",
    "predictions = train_x @ weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can find the gradient by finding the loss (mean squared error derivative), then calling `loss.backward()`.  This will automatically backpropagate from `loss` to `weights`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss = (predictions - train_y).mean()\n",
    "loss.backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can display the weight gradient:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.2605],\n",
       "        [ 0.4172],\n",
       "        [-0.5335],\n",
       "        [-0.5425],\n",
       "        [-0.5502],\n",
       "        [-0.5209],\n",
       "        [-0.5167]])"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weights.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And make the gradient update with a `1e-5` learning rate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights = weights - 1e-5 * weights.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## nn.Module\n",
    "\n",
    "We can use the `nn.Module` class to organize our code and make it easier to keep track of parameters.  We can only write the forward pass of the network, and torch will automatically run backpropagation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch import nn\n",
    "import math\n",
    "\n",
    "class DenseLayer(nn.Module):\n",
    "    def __init__(self, input_units, output_units):\n",
    "        super().__init__()\n",
    "\n",
    "        # Initialize our weights and biases\n",
    "        # Scale by k to improve convergence\n",
    "        k = math.sqrt(1/input_units)\n",
    "        # Putting a tensor inside nn.Parameter will mark that tensor as needing a gradient\n",
    "        self.weight = nn.Parameter(torch.rand(input_units, output_units) * 2 * k - k)\n",
    "        self.bias = nn.Parameter(torch.rand(1, output_units) * 2 * k - k)\n",
    "\n",
    "    def forward(self, x):\n",
    "        # Simple forward pass!\n",
    "        return x @ self.weight + self.bias"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above, we write a complete neural network layer.  We initialize the weights and biases, then we write a forward pass.  We use `nn.Parameter` to mark parameters that need gradients.\n",
    "\n",
    "We can then initialize a multilayer network as another `nn.Module`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DenseNetwork(nn.Module):\n",
    "    def __init__(self, input_units, hidden_units, output_units, layers):\n",
    "        super().__init__()\n",
    "\n",
    "        torch.manual_seed(0)\n",
    "        modules = []\n",
    "        # Define multiple network layers\n",
    "        for i in range(layers):\n",
    "            in_size = out_size = hidden_units\n",
    "            if i == 0:\n",
    "                # The first layer has the same number of rows in the weight matrix as columns in the input data\n",
    "                in_size = input_units\n",
    "            elif i == layers - 1:\n",
    "                # The last layer has the same number of columns in the weight matrix as the target\n",
    "                out_size = output_units\n",
    "            modules.append(DenseLayer(in_size, out_size))\n",
    "\n",
    "        # A modulelist holds a list of parameters\n",
    "        self.module_list = nn.ModuleList(modules)\n",
    "\n",
    "    def forward(self, x):\n",
    "        # Loop through each module and apply it to the data sequentially\n",
    "        for module in self.module_list:\n",
    "            x = module(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above, we use `nn.ModuleList` to store a list of parameters/modules.  Most of the work is just setting the correct number of inputs and outputs in the layers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DataLoader\n",
    "\n",
    "Before we can train our network, we have to get our data into the right format for PyTorch.  To do that, we need to setup a `Dataset` and a `DataLoader`.  A `Dataset` is a wrapper around our data.  It behaves like a list, and returns a single training example when indexed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader, Dataset\n",
    "\n",
    "class PriceData(Dataset):\n",
    "    def __init__(self, x, y):\n",
    "        # Take in our x and y tensors (predictor, target)\n",
    "        self.x = x.float()\n",
    "        self.y = y.float()\n",
    "\n",
    "    def __len__(self):\n",
    "        # Return how many examples are in the dataset\n",
    "        return len(self.x)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        # Return a single training example\n",
    "        x = self.x[idx]\n",
    "        y = self.y[idx]\n",
    "        return x, y\n",
    "\n",
    "# Initialize the dataset\n",
    "train_ds = PriceData(train_x, train_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then have to wrap the dataset in a `DataLoader`.  A `DataLoader` makes it easy to work with batches of data, or distributed data across multiple devices.  We first set a batch size, and then initialize our `DataLoader` by passing in our `Dataset`.\n",
    "\n",
    "By default, a `DataLoader` will shuffle the input data every epoch.  We set `shuffle` to `False` since our data is time series, and we want to preserve temporal relationships:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 16\n",
    "train = DataLoader(train_ds, batch_size=batch_size, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Loop\n",
    "\n",
    "We can now write and run a full training loop using our network and DataLoader:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "from statistics import mean\n",
    "\n",
    "# Define our hyperparameters\n",
    "epochs = 50\n",
    "layers = 5\n",
    "hidden_size = 25\n",
    "lr = 5e-4\n",
    "\n",
    "# Initialize our network\n",
    "net = DenseNetwork(train_x.shape[1], hidden_size, 1, layers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PyTorch makes it easy to create optimizers.  You can define your own, or import premade optimizers.  We'll use the existing torch implementation of `SGD`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the optimizer\n",
    "optimizer = torch.optim.SGD(net.parameters(), lr=lr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whenever we want to update our parameters, we call `optimizer.step()`.  We call `optimizer.zero_grad()` to initialize our gradients to zero, just like we did in the [backpropagation lesson](https://github.com/VikParuchuri/zero_to_gpt/blob/master/explanations/comp_graph.ipynb).\n",
    "\n",
    "We can now write and run a training loop:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "49.68326367922127\n",
      "30.341548889130355\n",
      "23.66963948508104\n",
      "20.26992240929976\n",
      "18.193296501636507\n"
     ]
    }
   ],
   "source": [
    "def train_loop(net, optimizer, epochs):\n",
    "    # Use a predefined loss function\n",
    "    loss_fn = nn.MSELoss()\n",
    "\n",
    "    train_losses = []\n",
    "    for epoch in range(epochs):\n",
    "        for batch, (x, y) in enumerate(train):\n",
    "            # zero_grad will set all the gradients to zero\n",
    "            # We need this because gradients will accumulate in the backward pass\n",
    "            optimizer.zero_grad()\n",
    "            # Make a prediction using the network\n",
    "            pred = net(x)\n",
    "            # Calculate the loss\n",
    "            loss = loss_fn(pred, y)\n",
    "            # Call loss.backward to run backpropagation\n",
    "            loss.backward()\n",
    "            # Step the optimizer to update the parameters\n",
    "            optimizer.step()\n",
    "            train_losses.append(loss.item())\n",
    "\n",
    "        # Display loss information every few epochs\n",
    "        if (epoch + 1) % 10 == 0:\n",
    "            print(mean(train_losses))\n",
    "\n",
    "train_loop(net, optimizer, epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prebuilt modules\n",
    "\n",
    "We used prebuilt components for our optimizer and loss functions.  This is another one of the advantages of PyTorch - you don't have to code everything from scratch.  We can also use prebuilt components for our neural network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's an example of swapping our manual `DenseLayer` implementation for `nn.Linear`, which works very similarly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DenseNetwork(nn.Module):\n",
    "    def __init__(self, input_units, hidden_units, output_units, layers):\n",
    "        super().__init__()\n",
    "\n",
    "        torch.manual_seed(0)\n",
    "        modules = []\n",
    "        for i in range(layers):\n",
    "            in_size = out_size = hidden_units\n",
    "            if i == 0:\n",
    "                in_size = input_units\n",
    "            elif i == layers - 1:\n",
    "                out_size = output_units\n",
    "            # Use nn.Linear instead of our own implementation\n",
    "            modules.append(nn.Linear(in_size, out_size))\n",
    "        self.module_list = nn.ModuleList(modules)\n",
    "\n",
    "    def forward(self, x):\n",
    "        for module in self.module_list:\n",
    "            x = module(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "51.05724552236497\n",
      "30.849216412380336\n",
      "23.951016113410393\n",
      "20.450236316770315\n",
      "18.319719779416918\n"
     ]
    }
   ],
   "source": [
    "net = DenseNetwork(train_x.shape[1], hidden_size, 1, layers)\n",
    "optimizer = torch.optim.SGD(net.parameters(), lr=lr)\n",
    "train_loop(net, optimizer, epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PyTorch makes it easy to swap components in and out to make a more complex network.  You can pick from:\n",
    "\n",
    "- Layer types\n",
    "- Complete networks\n",
    "- Optimizers\n",
    "- Schedulers\n",
    "- Loss functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Portability\n",
    "\n",
    "PyTorch also makes your code portable across devices.  So far, we've run on CPU.  Running on the CPU is convenient, but it's also much slower than running on the GPU.  We'll dive into why in a later lesson.\n",
    "\n",
    "If you want to run on a different device, it's usually hard - you have to use an interface to the specific device.  Luckily, PyTorch makes it simple to swap between devices.  You just use the `.to()` method to send tensors to different devices.  If you call `.to()` on an `nn.Module` instance, all of the parameters used by the model will be sent to the device.\n",
    "\n",
    "We can first set our device appropriately, depending on our system:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "if torch.cuda.is_available():\n",
    "    device = \"cuda\"\n",
    "elif torch.backends.mps.is_available() and torch.backends.mps.is_built():\n",
    "    device = \"mps\"\n",
    "else:\n",
    "    device = \"cpu\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we need to make a small modification to our `Dataset`.  We'll now send our predictors and target to the device.  If you're going to use a device other than the cpu for computation, you need to make sure that all the tensors that you're using (the model, the inputs, etc) are on the same device.  Otherwise, you'll get torch errors.\n",
    "\n",
    "In this case, we'll send the data to the device when we need it, instead of all upfront.  This saves GPU RAM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "class PriceData(Dataset):\n",
    "    def __init__(self, x, y):\n",
    "        self.x = x.float()\n",
    "        self.y = y.float()\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.x)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        x = self.x[idx]\n",
    "        y = self.y[idx]\n",
    "        # Send x and y to the device\n",
    "        return x.to(device), y.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then initialize our new dataset and train the network.  We'll also need to send the network to the same device that the inputs are on:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "51.057245976105335\n",
      "30.84921663403511\n",
      "23.95101627384623\n",
      "20.450236380659042\n",
      "18.319719846621155\n"
     ]
    }
   ],
   "source": [
    "train_ds = PriceData(train_x, train_y)\n",
    "train = DataLoader(train_ds, batch_size=batch_size, shuffle=False)\n",
    "\n",
    "net = DenseNetwork(train_x.shape[1], hidden_size, 1, layers).to(device)\n",
    "optimizer = torch.optim.SGD(net.parameters(), lr=lr)\n",
    "\n",
    "train_loop(net, optimizer, epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to pull a value back from the device you sent it to (for example, if you're logging errors), you'll need to use `detach()`, and `cpu()`.  This will remove the tensor from the computational graph, so autograd doesn't try to use it in backpropagation.  It will also pull the tensor back to system RAM so the CPU can access it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.3907, 0.2057, 0.6909, 0.6334, 0.6904],\n",
       "        [0.4445, 0.4336, 0.4603, 0.6318, 0.1163],\n",
       "        [0.0340, 0.6871, 0.2262, 0.4579, 0.6386],\n",
       "        [0.5701, 0.8223, 0.5655, 0.6238, 0.4552],\n",
       "        [0.5738, 0.6833, 0.8411, 0.0262, 0.2917]])"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.rand(5,5).to(device)\n",
    "y = x.detach().cpu()\n",
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Disable autograd\n",
    "\n",
    "There are times when you'll need to disable autograd.  For example, when you're calculating error across a validation set.  In cases like this, you can use the `torch.no_grad` context manager:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "ename": "RuntimeError",
     "evalue": "element 0 of tensors does not require grad and does not have a grad_fn",
     "output_type": "error",
     "traceback": [
      "\u001B[0;31m---------------------------------------------------------------------------\u001B[0m",
      "\u001B[0;31mRuntimeError\u001B[0m                              Traceback (most recent call last)",
      "Cell \u001B[0;32mIn[57], line 7\u001B[0m\n\u001B[1;32m      4\u001B[0m \u001B[38;5;28;01mwith\u001B[39;00m torch\u001B[38;5;241m.\u001B[39mno_grad():\n\u001B[1;32m      5\u001B[0m     y \u001B[38;5;241m=\u001B[39m torch\u001B[38;5;241m.\u001B[39mmean(x \u001B[38;5;241m*\u001B[39m \u001B[38;5;241m2\u001B[39m)\n\u001B[0;32m----> 7\u001B[0m \u001B[43my\u001B[49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mbackward\u001B[49m\u001B[43m(\u001B[49m\u001B[43m)\u001B[49m\n",
      "File \u001B[0;32m~/.virtualenvs/nnets/lib/python3.10/site-packages/torch/_tensor.py:488\u001B[0m, in \u001B[0;36mTensor.backward\u001B[0;34m(self, gradient, retain_graph, create_graph, inputs)\u001B[0m\n\u001B[1;32m    478\u001B[0m \u001B[38;5;28;01mif\u001B[39;00m has_torch_function_unary(\u001B[38;5;28mself\u001B[39m):\n\u001B[1;32m    479\u001B[0m     \u001B[38;5;28;01mreturn\u001B[39;00m handle_torch_function(\n\u001B[1;32m    480\u001B[0m         Tensor\u001B[38;5;241m.\u001B[39mbackward,\n\u001B[1;32m    481\u001B[0m         (\u001B[38;5;28mself\u001B[39m,),\n\u001B[0;32m   (...)\u001B[0m\n\u001B[1;32m    486\u001B[0m         inputs\u001B[38;5;241m=\u001B[39minputs,\n\u001B[1;32m    487\u001B[0m     )\n\u001B[0;32m--> 488\u001B[0m \u001B[43mtorch\u001B[49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mautograd\u001B[49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mbackward\u001B[49m\u001B[43m(\u001B[49m\n\u001B[1;32m    489\u001B[0m \u001B[43m    \u001B[49m\u001B[38;5;28;43mself\u001B[39;49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[43mgradient\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[43mretain_graph\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[43mcreate_graph\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[43minputs\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43minputs\u001B[49m\n\u001B[1;32m    490\u001B[0m \u001B[43m\u001B[49m\u001B[43m)\u001B[49m\n",
      "File \u001B[0;32m~/.virtualenvs/nnets/lib/python3.10/site-packages/torch/autograd/__init__.py:197\u001B[0m, in \u001B[0;36mbackward\u001B[0;34m(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)\u001B[0m\n\u001B[1;32m    192\u001B[0m     retain_graph \u001B[38;5;241m=\u001B[39m create_graph\n\u001B[1;32m    194\u001B[0m \u001B[38;5;66;03m# The reason we repeat same the comment below is that\u001B[39;00m\n\u001B[1;32m    195\u001B[0m \u001B[38;5;66;03m# some Python versions print out the first line of a multi-line function\u001B[39;00m\n\u001B[1;32m    196\u001B[0m \u001B[38;5;66;03m# calls in the traceback and some print out the last line\u001B[39;00m\n\u001B[0;32m--> 197\u001B[0m \u001B[43mVariable\u001B[49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43m_execution_engine\u001B[49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mrun_backward\u001B[49m\u001B[43m(\u001B[49m\u001B[43m  \u001B[49m\u001B[38;5;66;43;03m# Calls into the C++ engine to run the backward pass\u001B[39;49;00m\n\u001B[1;32m    198\u001B[0m \u001B[43m    \u001B[49m\u001B[43mtensors\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[43mgrad_tensors_\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[43mretain_graph\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[43mcreate_graph\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[43minputs\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m    199\u001B[0m \u001B[43m    \u001B[49m\u001B[43mallow_unreachable\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[38;5;28;43;01mTrue\u001B[39;49;00m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[43maccumulate_grad\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[38;5;28;43;01mTrue\u001B[39;49;00m\u001B[43m)\u001B[49m\n",
      "\u001B[0;31mRuntimeError\u001B[0m: element 0 of tensors does not require grad and does not have a grad_fn"
     ]
    }
   ],
   "source": [
    "x = torch.rand(5,5).to(device)\n",
    "x.requires_grad = True\n",
    "\n",
    "with torch.no_grad():\n",
    "    y = torch.mean(x * 2)\n",
    "\n",
    "y.backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, torch will throw an error about not requiring a gradient.  Disabling the gradient calculation can save a lot of time and memory when doing inference and validation.  We don't need a gradient in these cases, since we're not updating the parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrap-up\n",
    "\n",
    "In this lesson, we learned about PyTorch, a deep learning framework.  PyTorch has prebuilt components, autograd, and support for different devices.  This makes it much simpler to build neural networks.  We'll learn more about PyTorch in subsequent lessons, but this should cover the main features.\n",
    "\n",
    "We're very close to implementing a transformer.  In the next lesson, we'll learn how to work with text data, then the lesson after that will be implementing transformers."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}