courses/udacity_deep_learning/2_fullyconnected.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "version": "0.3.2",
      "views": {},
      "default_view": {},
      "name": "2_fullyconnected.ipynb",
      "provenance": [],
      "toc_visible": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kR-4eNdK6lYS",
        "colab_type": "text"
      },
      "source": [
        "Deep Learning\n",
        "=============\n",
        "\n",
        "Assignment 2\n",
        "------------\n",
        "\n",
        "Previously in `1_notmnist.ipynb`, we created a pickle with formatted datasets for training, development and testing on the [notMNIST dataset](http://yaroslavvb.blogspot.com/2011/09/notmnist-dataset.html).\n",
        "\n",
        "The goal of this assignment is to progressively train deeper and more accurate models using TensorFlow."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JLpLa8Jt7Vu4",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        },
        "cellView": "both"
      },
      "source": [
        "# These are all the modules we'll be using later. Make sure you can import them\n",
        "# before proceeding further.\n",
        "from __future__ import print_function\n",
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "from six.moves import cPickle as pickle\n",
        "from six.moves import range"
      ],
      "outputs": [],
      "execution_count": 0
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1HrCK6e17WzV",
        "colab_type": "text"
      },
      "source": [
        "First reload the data we generated in `1_notmnist.ipynb`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y3-cj1bpmuxc",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 1
            }
          ]
        },
        "cellView": "both",
        "executionInfo": {
          "elapsed": 19456,
          "status": "ok",
          "timestamp": 1449847956073,
          "user": {
            "color": "",
            "displayName": "",
            "isAnonymous": false,
            "isMe": true,
            "permissionId": "",
            "photoUrl": "",
            "sessionId": "0",
            "userId": ""
          },
          "user_tz": 480
        },
        "outputId": "0ddb1607-1fc4-4ddb-de28-6c7ab7fb0c33"
      },
      "source": [
        "pickle_file = 'notMNIST.pickle'\n",
        "\n",
        "with open(pickle_file, 'rb') as f:\n",
        "  save = pickle.load(f)\n",
        "  train_dataset = save['train_dataset']\n",
        "  train_labels = save['train_labels']\n",
        "  valid_dataset = save['valid_dataset']\n",
        "  valid_labels = save['valid_labels']\n",
        "  test_dataset = save['test_dataset']\n",
        "  test_labels = save['test_labels']\n",
        "  del save  # hint to help gc free up memory\n",
        "  print('Training set', train_dataset.shape, train_labels.shape)\n",
        "  print('Validation set', valid_dataset.shape, valid_labels.shape)\n",
        "  print('Test set', test_dataset.shape, test_labels.shape)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Training set (200000, 28, 28) (200000,)\n",
            "Validation set (10000, 28, 28) (10000,)\n",
            "Test set (18724, 28, 28) (18724,)\n"
          ],
          "name": "stdout"
        }
      ],
      "execution_count": 0
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L7aHrm6nGDMB",
        "colab_type": "text"
      },
      "source": [
        "Reformat into a shape that's more adapted to the models we're going to train:\n",
        "- data as a flat matrix,\n",
        "- labels as float 1-hot encodings."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IRSyYiIIGIzS",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 1
            }
          ]
        },
        "cellView": "both",
        "executionInfo": {
          "elapsed": 19723,
          "status": "ok",
          "timestamp": 1449847956364,
          "user": {
            "color": "",
            "displayName": "",
            "isAnonymous": false,
            "isMe": true,
            "permissionId": "",
            "photoUrl": "",
            "sessionId": "0",
            "userId": ""
          },
          "user_tz": 480
        },
        "outputId": "2ba0fc75-1487-4ace-a562-cf81cae82793"
      },
      "source": [
        "image_size = 28\n",
        "num_labels = 10\n",
        "\n",
        "def reformat(dataset, labels):\n",
        "  dataset = dataset.reshape((-1, image_size * image_size)).astype(np.float32)\n",
        "  # Map 0 to [1.0, 0.0, 0.0 ...], 1 to [0.0, 1.0, 0.0 ...]\n",
        "  labels = (np.arange(num_labels) == labels[:,None]).astype(np.float32)\n",
        "  return dataset, labels\n",
        "train_dataset, train_labels = reformat(train_dataset, train_labels)\n",
        "valid_dataset, valid_labels = reformat(valid_dataset, valid_labels)\n",
        "test_dataset, test_labels = reformat(test_dataset, test_labels)\n",
        "print('Training set', train_dataset.shape, train_labels.shape)\n",
        "print('Validation set', valid_dataset.shape, valid_labels.shape)\n",
        "print('Test set', test_dataset.shape, test_labels.shape)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Training set (200000, 784) (200000, 10)\n",
            "Validation set (10000, 784) (10000, 10)\n",
            "Test set (18724, 784) (18724, 10)\n"
          ],
          "name": "stdout"
        }
      ],
      "execution_count": 0
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nCLVqyQ5vPPH",
        "colab_type": "text"
      },
      "source": [
        "We're first going to train a multinomial logistic regression using simple gradient descent.\n",
        "\n",
        "TensorFlow works like this:\n",
        "* First you describe the computation that you want to see performed: what the inputs, the variables, and the operations look like. These get created as nodes over a computation graph. This description is all contained within the block below:\n",
        "\n",
        "      with graph.as_default():\n",
        "          ...\n",
        "\n",
        "* Then you can run the operations on this graph as many times as you want by calling `session.run()`, providing it outputs to fetch from the graph that get returned. This runtime operation is all contained in the block below:\n",
        "\n",
        "      with tf.Session(graph=graph) as session:\n",
        "          ...\n",
        "\n",
        "Let's load all the data into TensorFlow and build the computation graph corresponding to our training:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Nfv39qvtvOl_",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        },
        "cellView": "both"
      },
      "source": [
        "# With gradient descent training, even this much data is prohibitive.\n",
        "# Subset the training data for faster turnaround.\n",
        "train_subset = 10000\n",
        "\n",
        "graph = tf.Graph()\n",
        "with graph.as_default():\n",
        "\n",
        "  # Input data.\n",
        "  # Load the training, validation and test data into constants that are\n",
        "  # attached to the graph.\n",
        "  tf_train_dataset = tf.constant(train_dataset[:train_subset, :])\n",
        "  tf_train_labels = tf.constant(train_labels[:train_subset])\n",
        "  tf_valid_dataset = tf.constant(valid_dataset)\n",
        "  tf_test_dataset = tf.constant(test_dataset)\n",
        "  \n",
        "  # Variables.\n",
        "  # These are the parameters that we are going to be training. The weight\n",
        "  # matrix will be initialized using random values following a (truncated)\n",
        "  # normal distribution. The biases get initialized to zero.\n",
        "  weights = tf.Variable(\n",
        "    tf.truncated_normal([image_size * image_size, num_labels]))\n",
        "  biases = tf.Variable(tf.zeros([num_labels]))\n",
        "  \n",
        "  # Training computation.\n",
        "  # We multiply the inputs with the weight matrix, and add biases. We compute\n",
        "  # the softmax and cross-entropy (it's one operation in TensorFlow, because\n",
        "  # it's very common, and it can be optimized). We take the average of this\n",
        "  # cross-entropy across all training examples: that's our loss.\n",
        "  logits = tf.matmul(tf_train_dataset, weights) + biases\n",
        "  loss = tf.reduce_mean(\n",
        "    tf.nn.softmax_cross_entropy_with_logits(labels=tf_train_labels, logits=logits))\n",
        "  \n",
        "  # Optimizer.\n",
        "  # We are going to find the minimum of this loss using gradient descent.\n",
        "  optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)\n",
        "  \n",
        "  # Predictions for the training, validation, and test data.\n",
        "  # These are not part of training, but merely here so that we can report\n",
        "  # accuracy figures as we train.\n",
        "  train_prediction = tf.nn.softmax(logits)\n",
        "  valid_prediction = tf.nn.softmax(\n",
        "    tf.matmul(tf_valid_dataset, weights) + biases)\n",
        "  test_prediction = tf.nn.softmax(tf.matmul(tf_test_dataset, weights) + biases)"
      ],
      "outputs": [],
      "execution_count": 0
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KQcL4uqISHjP",
        "colab_type": "text"
      },
      "source": [
        "Let's run this computation and iterate:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z2cjdenH869W",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 9
            }
          ]
        },
        "cellView": "both",
        "executionInfo": {
          "elapsed": 57454,
          "status": "ok",
          "timestamp": 1449847994134,
          "user": {
            "color": "",
            "displayName": "",
            "isAnonymous": false,
            "isMe": true,
            "permissionId": "",
            "photoUrl": "",
            "sessionId": "0",
            "userId": ""
          },
          "user_tz": 480
        },
        "outputId": "4c037ba1-b526-4d8e-e632-91e2a0333267"
      },
      "source": [
        "num_steps = 801\n",
        "\n",
        "def accuracy(predictions, labels):\n",
        "  return (100.0 * np.sum(np.argmax(predictions, 1) == np.argmax(labels, 1))\n",
        "          / predictions.shape[0])\n",
        "\n",
        "with tf.Session(graph=graph) as session:\n",
        "  # This is a one-time operation which ensures the parameters get initialized as\n",
        "  # we described in the graph: random weights for the matrix, zeros for the\n",
        "  # biases. \n",
        "  tf.global_variables_initializer().run()\n",
        "  print('Initialized')\n",
        "  for step in range(num_steps):\n",
        "    # Run the computations. We tell .run() that we want to run the optimizer,\n",
        "    # and get the loss value and the training predictions returned as numpy\n",
        "    # arrays.\n",
        "    _, l, predictions = session.run([optimizer, loss, train_prediction])\n",
        "    if (step % 100 == 0):\n",
        "      print('Loss at step %d: %f' % (step, l))\n",
        "      print('Training accuracy: %.1f%%' % accuracy(\n",
        "        predictions, train_labels[:train_subset, :]))\n",
        "      # Calling .eval() on valid_prediction is basically like calling run(), but\n",
        "      # just to get that one numpy array. Note that it recomputes all its graph\n",
        "      # dependencies.\n",
        "      print('Validation accuracy: %.1f%%' % accuracy(\n",
        "        valid_prediction.eval(), valid_labels))\n",
        "  print('Test accuracy: %.1f%%' % accuracy(test_prediction.eval(), test_labels))"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Initialized\n",
            "Loss at step 0 : 17.2939\n",
            "Training accuracy: 10.8%\n",
            "Validation accuracy: 13.8%\n",
            "Loss at step 100 : 2.26903\n",
            "Training accuracy: 72.3%\n",
            "Validation accuracy: 71.6%\n",
            "Loss at step 200 : 1.84895\n",
            "Training accuracy: 74.9%\n",
            "Validation accuracy: 73.9%\n",
            "Loss at step 300 : 1.60701\n",
            "Training accuracy: 76.0%\n",
            "Validation accuracy: 74.5%\n",
            "Loss at step 400 : 1.43912\n",
            "Training accuracy: 76.8%\n",
            "Validation accuracy: 74.8%\n",
            "Loss at step 500 : 1.31349\n",
            "Training accuracy: 77.5%\n",
            "Validation accuracy: 75.0%\n",
            "Loss at step 600 : 1.21501\n",
            "Training accuracy: 78.1%\n",
            "Validation accuracy: 75.4%\n",
            "Loss at step 700 : 1.13515\n",
            "Training accuracy: 78.6%\n",
            "Validation accuracy: 75.4%\n",
            "Loss at step 800 : 1.0687\n",
            "Training accuracy: 79.2%\n",
            "Validation accuracy: 75.6%\n",
            "Test accuracy: 82.9%\n"
          ],
          "name": "stdout"
        }
      ],
      "execution_count": 0
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x68f-hxRGm3H",
        "colab_type": "text"
      },
      "source": [
        "Let's now switch to stochastic gradient descent training instead, which is much faster.\n",
        "\n",
        "The graph will be similar, except that instead of holding all the training data into a constant node, we create a `Placeholder` node which will be fed actual data at every call of `session.run()`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qhPMzWYRGrzM",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        },
        "cellView": "both"
      },
      "source": [
        "batch_size = 128\n",
        "\n",
        "graph = tf.Graph()\n",
        "with graph.as_default():\n",
        "\n",
        "  # Input data. For the training data, we use a placeholder that will be fed\n",
        "  # at run time with a training minibatch.\n",
        "  tf_train_dataset = tf.placeholder(tf.float32,\n",
        "                                    shape=(batch_size, image_size * image_size))\n",
        "  tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, num_labels))\n",
        "  tf_valid_dataset = tf.constant(valid_dataset)\n",
        "  tf_test_dataset = tf.constant(test_dataset)\n",
        "  \n",
        "  # Variables.\n",
        "  weights = tf.Variable(\n",
        "    tf.truncated_normal([image_size * image_size, num_labels]))\n",
        "  biases = tf.Variable(tf.zeros([num_labels]))\n",
        "  \n",
        "  # Training computation.\n",
        "  logits = tf.matmul(tf_train_dataset, weights) + biases\n",
        "  loss = tf.reduce_mean(\n",
        "    tf.nn.softmax_cross_entropy_with_logits(labels=tf_train_labels, logits=logits))\n",
        "  \n",
        "  # Optimizer.\n",
        "  optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)\n",
        "  \n",
        "  # Predictions for the training, validation, and test data.\n",
        "  train_prediction = tf.nn.softmax(logits)\n",
        "  valid_prediction = tf.nn.softmax(\n",
        "    tf.matmul(tf_valid_dataset, weights) + biases)\n",
        "  test_prediction = tf.nn.softmax(tf.matmul(tf_test_dataset, weights) + biases)"
      ],
      "outputs": [],
      "execution_count": 0
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XmVZESmtG4JH",
        "colab_type": "text"
      },
      "source": [
        "Let's run it:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FoF91pknG_YW",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 6
            }
          ]
        },
        "cellView": "both",
        "executionInfo": {
          "elapsed": 66292,
          "status": "ok",
          "timestamp": 1449848003013,
          "user": {
            "color": "",
            "displayName": "",
            "isAnonymous": false,
            "isMe": true,
            "permissionId": "",
            "photoUrl": "",
            "sessionId": "0",
            "userId": ""
          },
          "user_tz": 480
        },
        "outputId": "d255c80e-954d-4183-ca1c-c7333ce91d0a"
      },
      "source": [
        "num_steps = 3001\n",
        "\n",
        "with tf.Session(graph=graph) as session:\n",
        "  tf.global_variables_initializer().run()\n",
        "  print(\"Initialized\")\n",
        "  for step in range(num_steps):\n",
        "    # Pick an offset within the training data, which has been randomized.\n",
        "    # Note: we could use better randomization across epochs.\n",
        "    offset = (step * batch_size) % (train_labels.shape[0] - batch_size)\n",
        "    # Generate a minibatch.\n",
        "    batch_data = train_dataset[offset:(offset + batch_size), :]\n",
        "    batch_labels = train_labels[offset:(offset + batch_size), :]\n",
        "    # Prepare a dictionary telling the session where to feed the minibatch.\n",
        "    # The key of the dictionary is the placeholder node of the graph to be fed,\n",
        "    # and the value is the numpy array to feed to it.\n",
        "    feed_dict = {tf_train_dataset : batch_data, tf_train_labels : batch_labels}\n",
        "    _, l, predictions = session.run(\n",
        "      [optimizer, loss, train_prediction], feed_dict=feed_dict)\n",
        "    if (step % 500 == 0):\n",
        "      print(\"Minibatch loss at step %d: %f\" % (step, l))\n",
        "      print(\"Minibatch accuracy: %.1f%%\" % accuracy(predictions, batch_labels))\n",
        "      print(\"Validation accuracy: %.1f%%\" % accuracy(\n",
        "        valid_prediction.eval(), valid_labels))\n",
        "  print(\"Test accuracy: %.1f%%\" % accuracy(test_prediction.eval(), test_labels))"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Initialized\n",
            "Minibatch loss at step 0 : 16.8091\n",
            "Minibatch accuracy: 12.5%\n",
            "Validation accuracy: 14.0%\n",
            "Minibatch loss at step 500 : 1.75256\n",
            "Minibatch accuracy: 77.3%\n",
            "Validation accuracy: 75.0%\n",
            "Minibatch loss at step 1000 : 1.32283\n",
            "Minibatch accuracy: 77.3%\n",
            "Validation accuracy: 76.6%\n",
            "Minibatch loss at step 1500 : 0.944533\n",
            "Minibatch accuracy: 83.6%\n",
            "Validation accuracy: 76.5%\n",
            "Minibatch loss at step 2000 : 1.03795\n",
            "Minibatch accuracy: 78.9%\n",
            "Validation accuracy: 77.8%\n",
            "Minibatch loss at step 2500 : 1.10219\n",
            "Minibatch accuracy: 80.5%\n",
            "Validation accuracy: 78.0%\n",
            "Minibatch loss at step 3000 : 0.758874\n",
            "Minibatch accuracy: 82.8%\n",
            "Validation accuracy: 78.8%\n",
            "Test accuracy: 86.1%\n"
          ],
          "name": "stdout"
        }
      ],
      "execution_count": 0
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7omWxtvLLxik",
        "colab_type": "text"
      },
      "source": [
        "---\n",
        "Problem\n",
        "-------\n",
        "\n",
        "Turn the logistic regression example with SGD into a 1-hidden layer neural network with rectified linear units [nn.relu()](https://www.tensorflow.org/versions/r0.7/api_docs/python/nn.html#relu) and 1024 hidden nodes. This model should improve your validation / test accuracy.\n",
        "\n",
        "---"
      ]
    }
  ]
}