python-examples/tf-tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow Tutorial\n",
    "\n",
    "We've used numpy to build neural networks but we will use a deep learning framework to build neural networks more easily. There are many machine learning frameworks like TensorFlow, PaddlePaddle, Torch, Caffe, Keras, and many others can speed up machine learning development significantly. \n",
    "\n",
    "In this notebook, we will use TensorFlow to do following: \n",
    "\n",
    "- Initialize variables\n",
    "- Start your own session\n",
    "- Train algorithms \n",
    "- Implement a Neural Network\n",
    "\n",
    "Programing frameworks can not only shorten your coding time, but sometimes also perform optimizations that speed up your code. \n",
    "\n",
    "## 1 - Exploring the Tensorflow Library\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "import numpy as np\n",
    "import h5py\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "from tensorflow.python.framework import ops\n",
    "from tf_utils import load_dataset, random_mini_batches, convert_to_one_hot, predict\n",
    "\n",
    "%matplotlib inline\n",
    "np.random.seed(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will compute for the loss of one training example. \n",
    "$$loss = \\mathcal{L}(\\hat{y}, y) = (\\hat y^{(i)} - y^{(i)})^2 \\tag{1}$$\n",
    "\n",
    "Writing and running programs in TensorFlow has the following steps:\n",
    "\n",
    "1. Create Tensors (variables) that are not yet executed/evaluated. \n",
    "2. Write operations between those Tensors.\n",
    "3. Initialize your Tensors. \n",
    "4. Create a Session. \n",
    "5. Run the Session. \n",
    "\n",
    "Therefore, when we created a variable for the loss, we simply defined the loss as a function of other quantities, but did not evaluate its value. To evaluate it, we had to run `init=tf.global_variables_initializer()`. That initialized the loss variable, and in the last line we were finally able to evaluate the value of `loss` and print its value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 - Initialize variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(\"Mul:0\", shape=(), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "a = tf.constant(2)\n",
    "b = tf.constant(10)\n",
    "c = tf.multiply(a,b)\n",
    "print(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We got a tensor that does not have the shape attribute, and is of type \"int32\". All we did was put in the 'computation graph', but we have not run this computation yet. In order to actually multiply the two numbers, we will have to create a session and run it.\n",
    "### 1.2 - Create a session and run the operations inside the session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20\n"
     ]
    }
   ],
   "source": [
    "sess=tf.Session()\n",
    "print(sess.run(c))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 - Placeholder\n",
    "\n",
    "A placeholder is an object whose value we can specify only later. \n",
    "To specify values for a placeholder, we pass in values by using a \"feed dictionary\" (`feed_dict` variable). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6\n"
     ]
    }
   ],
   "source": [
    "# Change the value of x in the feed_dict\n",
    "\n",
    "x = tf.placeholder(tf.int64, name = 'x')\n",
    "print(sess.run(2 * x, feed_dict = {x: 3}))\n",
    "sess.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we first defined `x` we did not have to specify a value for it. A placeholder is simply a variable that we will assign data to only later, when running the session. We say that we **feed data** to these placeholders when running the session. \n",
    "\n",
    "Here's what's happening: When we specify the operations needed for a computation, we are telling TensorFlow how to construct a computation graph. The computation graph can have some placeholders whose values you will specify only later. Finally, when we run the session, you are telling TensorFlow to execute the computation graph."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4 - Linear function\n",
    "\n",
    "We will compute the following equation: $Y = WX + b$, where $W$ and $X$ are random matrices and b is a random vector drawn from a random normal distribution. \n",
    "\n",
    "$W$ is of shape (4, 3), $X$ is (3,1) and $b$ is (4,1). \n",
    "\n",
    "The following functions are helpful: \n",
    "- tf.matmul(..., ...) to do a matrix multiplication\n",
    "- tf.add(..., ...) to do an addition\n",
    "- np.random.randn(...) to initialize random numbers from **n**ormal distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def linear_function():\n",
    "    \"\"\"\n",
    "    Implements a linear function: \n",
    "            Initializes W to be a random tensor of shape (4,3)\n",
    "            Initializes X to be a random tensor of shape (3,1)\n",
    "            Initializes b to be a random tensor of shape (4,1)\n",
    "    Returns: \n",
    "    result -- runs the session for Y = WX + b \n",
    "    \"\"\"\n",
    "    \n",
    "    np.random.seed(1)\n",
    "    \n",
    "    X = tf.constant(np.random.randn(3,1),name='X')\n",
    "    W = tf.constant(np.random.randn(4,3), name='W')\n",
    "    b = tf.constant(np.random.randn(4,1), name='b')\n",
    "    Y = tf.add(tf.matmul(W,X),b)\n",
    "    \n",
    "    # Create the session on the variable you want to calculate\n",
    "    sess = tf.Session()\n",
    "    result = sess.run(Y)\n",
    "    \n",
    "    # close the session \n",
    "    sess.close()\n",
    "\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "result = [[-2.15657382]\n",
      " [ 2.95891446]\n",
      " [-1.08926781]\n",
      " [-0.84538042]]\n"
     ]
    }
   ],
   "source": [
    "print( \"result = \" + str(linear_function()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.5 - Computing the sigmoid \n",
    "Tensorflow offers a variety of commonly used neural network functions like `tf.sigmoid` and `tf.softmax`. For this exercise lets compute the sigmoid function of an input. \n",
    "\n",
    "We will use a placeholder variable `x` and the feed dictionary to pass in the input `z`. \n",
    "\n",
    "1. create a placeholder `x`, \n",
    "2. define the operations needed to compute the sigmoid using `tf.sigmoid`, and then \n",
    "3. run the session. \n",
    "\n",
    "Following functions are useful \n",
    "\n",
    "- `tf.placeholder(tf.float32, name = \"...\")`\n",
    "- `tf.sigmoid(...)`\n",
    "- `sess.run(..., feed_dict = {x: z})`\n",
    "\n",
    "\n",
    "there are two typical ways to create and use sessions in tensorflow: \n",
    "\n",
    "**Method 1:**\n",
    "```python\n",
    "sess = tf.Session()\n",
    "# Run the variables initialization (if needed), run the operations\n",
    "result = sess.run(..., feed_dict = {...})\n",
    "sess.close() # Close the session\n",
    "```\n",
    "**Method 2:**\n",
    "```python\n",
    "with tf.Session() as sess: \n",
    "    # run the variables initialization (if needed), run the operations\n",
    "    result = sess.run(..., feed_dict = {...})\n",
    "    # This takes care of closing the session for you :)\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(z):\n",
    "    \"\"\"\n",
    "    Computes the sigmoid of z\n",
    "    \n",
    "    Arguments:\n",
    "    z -- input value, scalar or vector\n",
    "    \n",
    "    Returns: \n",
    "    results -- the sigmoid of z\n",
    "    \"\"\"\n",
    "    \n",
    "    # Create a placeholder for x. Name it 'x'.\n",
    "    x = tf.placeholder(tf.float32, name='x')\n",
    "\n",
    "    # compute sigmoid(x)\n",
    "    sigmoid = tf.sigmoid(x)\n",
    "\n",
    "    # You should use a feed_dict to pass z's value to x. \n",
    "    sess=tf.Session()\n",
    "    result = sess.run(sigmoid, feed_dict={x: z})\n",
    "\n",
    "    \n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sigmoid(0) = 0.5\n",
      "sigmoid(12) = 0.9999938\n"
     ]
    }
   ],
   "source": [
    "print (\"sigmoid(0) = \" + str(sigmoid(0)))\n",
    "print (\"sigmoid(12) = \" + str(sigmoid(12)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Summary**:\n",
    "1. Create placeholders\n",
    "2. Specify the computation graph corresponding to operations you want to compute\n",
    "3. Create the session\n",
    "4. Run the session, using a feed dictionary if necessary to specify placeholder variables' values. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.6 -  Computing the Cost\n",
    "\n",
    "We will use a built-in function to compute the cost of your neural network. So instead of needing to write code to compute this as a function of $a^{[2](i)}$ and $y^{(i)}$ for i=1...m: \n",
    "$$ J = - \\frac{1}{m}  \\sum_{i = 1}^m  \\large ( \\small y^{(i)} \\log a^{ [2] (i)} + (1-y^{(i)})\\log (1-a^{ [2] (i)} )\\large )\\small\\tag{2}$$\n",
    "\n",
    "we implement the cross entropy loss with: \n",
    "\n",
    "\n",
    "- `tf.nn.sigmoid_cross_entropy_with_logits(logits = ...,  labels = ...)`\n",
    "\n",
    "Input `z` is used to compute the sigmoid (to get `a`) and then compute the cross entropy cost $J$. The code `tf.nn.sigmoid_cross_entropy_with_logits` computes\n",
    "\n",
    "$$- \\frac{1}{m}  \\sum_{i = 1}^m  \\large ( \\small y^{(i)} \\log \\sigma(z^{[2](i)}) + (1-y^{(i)})\\log (1-\\sigma(z^{[2](i)})\\large )\\small\\tag{3}$$\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cost(logits, labels):\n",
    "    \"\"\"\n",
    "    Computes the cost using the sigmoid cross entropy\n",
    "    \n",
    "    Arguments:\n",
    "    logits -- vector containing z, output of the last linear unit (before the final sigmoid activation)\n",
    "    labels -- vector of labels y (1 or 0) \n",
    "    \n",
    "    Note: What we've been calling \"z\" and \"y\" in this class are respectively called \"logits\" and \"labels\" \n",
    "    in the TensorFlow documentation. So logits will feed into z, and labels into y. \n",
    "    \n",
    "    Returns:\n",
    "    cost -- runs the session of the cost (formula (2))\n",
    "    \"\"\"\n",
    "    \n",
    "    # Create the placeholders for \"logits\" (z) and \"labels\" (y)\n",
    "    z = tf.placeholder(tf.float32, name='z')\n",
    "    y = tf.placeholder(tf.float32, name='y')\n",
    "    \n",
    "    # Use the loss function\n",
    "    cost = tf.nn.sigmoid_cross_entropy_with_logits(logits=z, labels=y)\n",
    "    \n",
    "    # Create a session \n",
    "    sess = tf.Session()\n",
    "    \n",
    "    # Run the session\n",
    "    cost = sess.run(cost,feed_dict={z:logits, y:labels}) # use feed_dict since z and y are placeholder\n",
    "    \n",
    "    # Close the session \n",
    "    sess.close()\n",
    "\n",
    "    \n",
    "    return cost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cost = [1.0053872  1.0366409  0.4138543  0.39956614]\n"
     ]
    }
   ],
   "source": [
    "logits = sigmoid(np.array([0.2,0.4,0.7,0.9]))\n",
    "cost = cost(logits, np.array([0,0,1,1]))\n",
    "print (\"cost = \" + str(cost))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.7 - Using One Hot encodings\n",
    "\n",
    "Many times in deep learning you will have a y vector with numbers ranging from 0 to C-1, where C is the number of classes. If C is for example 4, then you might have the following y vector which you will need to convert as follows:\n",
    "\n",
    "\n",
    "<img src=\"images/onehot.png\" style=\"width:600px;height:150px;\">\n",
    "\n",
    "This is called a \"one hot\" encoding, because in the converted representation exactly one element of each column is \"hot\" (meaning set to 1). To do this conversion in numpy, you might have to write a few lines of code. In tensorflow, you can use one line of code: \n",
    "\n",
    "- tf.one_hot(labels, depth, axis) \n",
    "\n",
    "We will mplement the function below to take one vector of labels and the total number of classes $C$, and return the one hot encoding. Use `tf.one_hot()` to do this. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**In TensorFlow, what is the argument 'axis' in the function 'tf.one_hot'**\n",
    "\n",
    "Here's an example:\n",
    "\n",
    "```python\n",
    "x = tf.constant([0, 1, 2])\n",
    "```\n",
    "... is the input tensor and `N=4` (each index is transformed into 4D vector).\n",
    "\n",
    "`axis=-1`\n",
    "Computing `one_hot_1 = tf.one_hot(x, 4).eval()` yields a `(3, 4)` tensor:\n",
    "\n",
    "```\n",
    "[[ 1.  0.  0.  0.]\n",
    " [ 0.  1.  0.  0.]\n",
    " [ 0.  0.  1.  0.]]\n",
    " ```\n",
    "... where the last dimension is one-hot encoded (clearly visible). This corresponds to the default axis=-1, i.e. the last one.\n",
    "\n",
    "`axis=0`\n",
    "Now, computing `one_hot_2 = tf.one_hot(x, 4, axis=0).eval()` yields a `(4, 3)` tensor, which is not immediately recognizable as one-hot encoded:\n",
    "\n",
    "```\n",
    "[[ 1.  0.  0.]\n",
    " [ 0.  1.  0.]\n",
    " [ 0.  0.  1.]\n",
    " [ 0.  0.  0.]]\n",
    " ```\n",
    "This is because the one-hot encoding is done along the 0-axis and one has to transpose the matrix to see the previous encoding. The situation becomes more complicated, when the input is higher dimensional, but the idea is the same: the difference is in placement of the extra dimension used for one-hot encoding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "def one_hot_matrix(labels, C):\n",
    "    \"\"\"\n",
    "    Creates a matrix where the i-th row corresponds to the ith class number and the jth column\n",
    "                     corresponds to the jth training example. So if example j had a label i. Then entry (i,j) \n",
    "                     will be 1. \n",
    "                     \n",
    "    Arguments:\n",
    "    labels -- vector containing the labels \n",
    "    C -- number of classes, the depth of the one hot dimension\n",
    "    \n",
    "    Returns: \n",
    "    one_hot -- one hot matrix\n",
    "    \"\"\"\n",
    "    \n",
    "    # Create a tf.constant equal to C (depth), name it 'C'. \n",
    "    C = tf.constant(C, name='C')\n",
    "    \n",
    "    # Use tf.one_hot, be careful with the axis \n",
    "    one_hot_matrix = tf.one_hot(labels,C,axis=-1)\n",
    "    \n",
    "    # Create the session\n",
    "    sess = tf.Session()\n",
    "    \n",
    "    # Run the session\n",
    "    one_hot = sess.run(one_hot_matrix)\n",
    "    \n",
    "    # Close the session. \n",
    "    None\n",
    "    sess.close()\n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    return one_hot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "one_hot = [[0. 1. 0. 0.]\n",
      " [0. 0. 1. 0.]\n",
      " [0. 0. 0. 1.]\n",
      " [1. 0. 0. 0.]\n",
      " [0. 0. 1. 0.]\n",
      " [0. 1. 0. 0.]]\n"
     ]
    }
   ],
   "source": [
    "labels = np.array([1,2,3,0,2,1])\n",
    "one_hot = one_hot_matrix(labels, C = 4)\n",
    "print (\"one_hot = \" + str(one_hot))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.8 - Initialize with zeros and ones\n",
    "\n",
    "`tf.ones()` creates a tensor with all elements set to 1. This operation returns a tensor of type `dtype` with shape `shape` and all elements set to 1.\n",
    "\n",
    "To initialize with zeros you could use tf.zeros() instead. These functions take in a shape and return an array of dimension shape full of zeros and ones respectively. \n",
    "\n",
    "For example, `tf.ones([2, 3], tf.int32)`  returns \n",
    "```\n",
    "[[1, 1, 1], \n",
    "[1, 1, 1]]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ones(shape):\n",
    "    \"\"\"\n",
    "    Creates an array of ones of dimension shape\n",
    "    \n",
    "    Arguments:\n",
    "    shape -- shape of the array you want to create\n",
    "        \n",
    "    Returns: \n",
    "    ones -- array containing only ones\n",
    "    \"\"\"\n",
    "    \n",
    "    # Create \"ones\" tensor \n",
    "    ones = tf.ones(shape,tf.int32)\n",
    "    \n",
    "    sess = tf.Session()\n",
    "    ones = sess.run(ones)\n",
    "    sess.close\n",
    "\n",
    "    return ones"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ones = [[1 1 1 1]\n",
      " [1 1 1 1]\n",
      " [1 1 1 1]]\n"
     ]
    }
   ],
   "source": [
    "print (\"ones = \" + str(ones([3,4])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2 - Building neural network in tensorflow\n",
    "\n",
    "There are two parts to implement a tensorflow model:\n",
    "\n",
    "- Create the computation graph\n",
    "- Run the graph\n",
    "\n",
    "### 2.0 - Problem statement: SIGNS Dataset\n",
    "\n",
    "One afternoon, with some friends we decided to teach our computers to decipher sign language. We spent a few hours taking pictures in front of a white wall and came up with the following dataset. It's now your job to build an algorithm that would facilitate communications from a speech-impaired person to someone who doesn't understand sign language.\n",
    "\n",
    "- **Training set**: 1080 pictures (64 by 64 pixels) of signs representing numbers from 0 to 5 (180 pictures per number).\n",
    "- **Test set**: 120 pictures (64 by 64 pixels) of signs representing numbers from 0 to 5 (20 pictures per number).\n",
    "\n",
    "Note that this is a subset of the SIGNS dataset. The complete dataset contains many more signs.\n",
    "\n",
    "Here are examples for each number, and how an explanation of how we represent the labels. These are the original pictures, before we lowered the image resolutoion to 64 by 64 pixels.\n",
    "<img src=\"images/hands.png\" style=\"width:800px;height:350px;\"><caption><center> <u><font color='purple'> **Figure 1**</u><font color='purple'>: SIGNS dataset <br> <font color='black'> </center>\n",
    "\n",
    "\n",
    "#### 2.0.1 - Load the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loading the dataset\n",
    "X_train_orig, Y_train_orig, X_test_orig, Y_test_orig, classes = load_dataset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.0.2 - View the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y = 4\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP4AAAD8CAYAAABXXhlaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJztfWusXNd13rfmfd+Xb1GiKEoW9bATSw4IR45TR7HjxHWDGCiSIg8UaiFAf9LCQVPEdgu0SdEWzp8k/VEEEOo0+pHGdl6VYQRJDMWqGzeRTEcPS6Qp0jQlUiRFXt57ee/M3Hnv/pjh7LXWmb3vmUveuVTO+oCLu8+sffbZM3P2nLX2Wutb5JyDwWDIFnI7PQGDwTB52MI3GDIIW/gGQwZhC99gyCBs4RsMGYQtfIMhg7CFbzBkEDe18InoE0R0iojOENFnb9WkDAbD9oK2GsBDRHkAbwD4OIALAL4F4Beccydu3fQMBsN2oHAT534QwBnn3FkAIKIvAvgUgODCn5+bc/v37QEA5HJS2SAi39YycJlv59g5egxomew4+vXNQGP1vjlErhWbBf8Z3/ps+Zm3IrJzgp/bbYP0n1vanmm+lfPn38by8vKmH/jNLPy7AJxnxxcA/HDshP379uC3/ut/BACUy2UhK1f8cbFUErJioThsl0qsXVb9iv64UJBvLc9+THL5/LBNkR8PLQstxuTLNLKZHDMyfqRf7PfH8R9JMY3I+0yMMvoWi96gSijnGJ6wi8wxNCc9j639rMTejRzR8b6R02Q/F5apQWTX8E83Mctca+oOXQDAT33yn4YnyHAzNv6ozzvxsRDRU0R0nIiOr61Xb+JyBoPhVuFmnvgXANzNjg8BuKg7OeeeBvA0ANx/3xHX6XQAyKcuAOTa/Cksf4+4St/psid3pyv75bqsHTElXI8J9G9fOhU7/fMiBt0z3Zny2umekskxYkZB4N2N8aZdQBZ9hy58SPxoPPtsnM6jJxIbLvwgTz1k6AJxDUifNd77vJkn/rcAHCWie4moBODnAXzlJsYzGAwTwpaf+M65DhH9KwB/CSAP4Pecc6/fspkZDIZtw82o+nDO/TmAP79FczEYDBPCTS38ceGcQ6fTBpC0wfM5b+NrWS7vj2mwRzCyH7Pjk64+1uYuwYTtldIhxkRuLPtqK67E2D52dJvZN/XnITaq9ZZ82nnxDyG8rU/pNsVT79an3JFISLkdHN2uGMODGYqDiQ8RfgfE3Bz6O0t8xtEx47CQXYMhg7CFbzBkEBNX9dsDVZ1yyp2Xb7O2MgPy3Azw7a5y53W4udCVMvAxu8ydp1QkYT5Edb7RqiywWVBKSjeaUPl0t9Hqq4YTam54jhoUEwbGT46RMvBHBBzpoJTRwUgJjVdMVzs7t2IWhWVp/XL6e5Fev/ANE787+I0VuelSwJ74BkMGYQvfYMggbOEbDBnEDrjz+rZ3Pt8Tsi6zu7s9JWPHeWbgdZWxl+uFxyBm84tsv0giTi7iPonloIjxIwapi9hpFHc4sWY6J5i2K0luRET7BhGxK2XIbtrx5GHUrg/Og8KiqCEfnkfaJJ3IEPFtgsBUoinziTn2RvcLwJ74BkMGYQvfYMggJqrqA1596TmpmvDjblfJesw1xNSfnlKFYmPkcv64R6ytTQLiqniE6CMWKZVSo4yp88J1k1AN3eh+qrPUcsM55jE6gXj04vgqsJ5HLG9fqrqRrMmYJyuUJhhF2CWYNM8CZzk9BhOltDJSvy0+aMrvwZ74BkMGYQvfYMggJrurD6++6R3LXkRr5GopV82dMhdcwCToj5/OXOC78HpHXo4Zjp5LqINi/FEj3BiHjxEQ6OtFkmPSJhklkFbfjI0nPqqUc9p00NFwke9CdoxEE0YD/MbI2gleOmbWBeY/TjYSqf+bwJ74BkMGYQvfYMggbOEbDBnEZN15jtn4kdimhH0esN0TFMMxVx8bQ0T1qSw+Sa+tfxf9eTyJr5egQQ4fxSQuaGeGXXEJ+z/gSkzMQtjx4UwymQkYGSJKHDo6y26riGW3jcWiIQe9BaeF9xDC323k0pGNiOR3Md4bsCe+wZBB2MI3GDKICUfuOfQGqrXrKlU8ZZKOSNjpqTGEGaBdfcwN2GO/d+qnz8XGcLySiX9dR+A5wTenZfy8RHYME24tGyS1wjc+P0UsgBC9dlMKeeUiVuHI6bA1UdEnktAUeH2rSL7liHt2C5k5sXNi3kLpCh7jfVrknsFg2Ay28A2GDMIWvsGQQUyeiKPbJ9vMdztCVuj5qfR60sXWZS43nnXX01l8rHZeNyftI86z3+0xu7KnnG+8rF7KiFenyEHTmueaOoECXqloaG9kjFiWoOBvT23Dyn5rZ787bDfOvCZk3YK363cf+8iwXdm1LzhmL1HaPBJiG5xizJmadryUGXhbGyL5eYf2hNJzhYy967HpE5+Ifo+IrhDRa+y13UT0NSI6Pfi/a8zrGgyGHUQaVf/3AXxCvfZZAM85544CeG5wbDAY3iXYVNV3zn2DiI6olz8F4PFB+xkAzwP4zOZjeS78bjeszuvy192c59zvMLU6r1TsXJ7x5XWl8tMVEXn8dTnHtJFlsmyzVLw4N78myojlqaVWZ8WAEfIKNq+EiyrmNQpMpH75bXF88RvPDdu5Rk3I1jbqw3ZrxiuEhz/0uOgnyE0S6nHIqEmkKwb6bS0gLxkwF86GdIGDeCJdOFIyGoQYi8ScUAmtA865SwAw+L9/i+MYDIYdwLbv6hPRU0R0nIiOV2u1zU8wGAzbjq3u6r9DRAedc5eI6CCAK6GOzrmnATwNAIcP3eluRNe5nubV43x5cse/2/W/T11WLbdbkGW4eqxfT5kBPefNhx7f1Y94BjT1tth1j6j6HLqib9q4rJAKmZiIJhIJVKmNXiHSr9fxZtY7rx4Xsuba2rDdareF7FrVq/qL4N9TLFpRiVo+GrBTXR22c6WK6JefnWdjhMeP5D1FP4OYd0EaI+kMiyQpx2gDMBkpyeaRsEknk6TzFQBPDNpPAHh2i+MYDIYdQBp33h8C+FsADxLRBSJ6EsDnAXyciE4D+Pjg2GAwvEuQZlf/FwKij93iuRgMhglhwtl5hBs2jDZzRGZdzP7v8Sg+7RL09n++q2x8ZhP1iNnxKsKPBw0mzCjKj5TpstI8qy/JhRlx9QlX3NbYMMOkDnovIHIOe2GNufCunzstuvXY57+8Jjduq+wrXLjrMJtG2B3WazaEbOXvvzlsdy6e8/3KZdFv8diPDdvTBw9DIvA5Ru1slfUpeoXdojEr20U3GEaPkbTiw1mf4/otLVbfYMggbOEbDBnE5Ik4BjpJooRWpNJtjqmU+a5Xt3tK1e/1IjIW1dfjVWlVQhA3CUiKhKtM9ItU3NUqmXDJRJI14qQcKQncIqQfQkXVQ7DP/8p3vzNsN6rrol9zw7vwag2ppk8fumfYnt/rE3O0Gcevvfb2OSFbO3tq2M63vRm3fm1J9Nt44+Sw/Z477pbj3wJO/JiLLdgvWt5XI6XPcat8giNgT3yDIYOwhW8wZBC28A2GDOL2qZ0nauIpl0+AADNJ2MH2CfLSlswLsk3WVva5cPVpE4tdjxN79HLaLcdtvbBtTcpmC1lwSVLOsNsvlKiWurwzgGb1+rC9funNYbut+q01Nny/dkvI7nvf+4ftfMmTcuiy5F123urp1+UFOr5vdcOH715cXRPdjs7vHrZj7sKoDzOSySgIPNKGQWvJFszzyBRH9J1MyK7BYHgXwxa+wZBBTNid51WxXi+szif9SzyqLxzh51gGnnMqc4+dx9VN7YrLsci6HsnxqUsjZZq3zwXcfgOhv5ZTXH28HU7A20SYEuLjlmOsvOkj9FzNq/3lsrxdimX/GU9P7xWye97/yLAdM+PWLr81bNevSKKPIvtc1+veXVhY2C367TvyHn+tmH4cy8DbCgehQvqvIk7TEeqnIz23du0+7IlvMGQQtvANhgxix1T9RGRTyiq4MgEm7BnQu8eyhBZT63KasIOp8JHkG94vp34/01b0jRV9Fa8nXgjvQLvQ7nTi4/YvtOtVIVs69eqwzSnMGw1ZJqvFiDIOvO8HhKzCyDE6jMyjq3b/l077yMCOkq3X/PhrLIFn//seEf0qs7PDttOk5Vz3Z/dALi9v/a3HxKVLAoon34w+SkSERmG7+gaDYRPYwjcYMghb+AZDBjFxG/+G1TKW+ZIaYZegsLsxes+gfxyLuhudfZVwK3KyjQjLRcJtFDTTwqSfMURLNTPR8lvfE7L60uVhu8Oy4qobMgOvxiLrjt5zv5B1RNkzP0Zz5R3RL1ddHrYrU1NCtlr15B71or9VDzwk9xO6vOS64vevv/HysO2qPuKvdEjOt3KYHZN+Ho6uyQCMm4U3OCdtP528ye8dXU/hBsFNyrHtiW8wZBC28A2GDGKiqj6BkC/0L1lQ7pRCPj+yDQCFQo61vSynS2hx11yCL48rQSlVfe2m464i4VaU842580TEosq6EFMWRB+xCK6wuUARAolO27vKLp14RciaTe9+q9d9v+uMKx8A3MzCsL2LEW8AUr3vdbybrn5e8vYVWOJToyfrKfAqxAceft+wPbN7j+jXY9daOfGykDVO/L2/Frt3qufPi36HF/yYlUU5vphTjKAwyqXnIr1C50W+94S70Nx5BoNhE9jCNxgyCFv4BkMGMVkbn4DiDRu/qO14f5xXNfFyzObP83ZO9WNumGRpsdF2t7aNREStziDMsQw/1jGXMsS4fx630xSpKJ9/jCmDggdyHgjvNVRXrg3bq5cvyOHZHHlNvJaqM3jHPfcN21Ozc0LWEy48X1qxdfWC6udt/PWa3ENosssdec+DCKG+fHXYXj0lyTxyjBB0ecOThS615H7CPnbt8oK28VM74LbYa/R5Y4Xs3ursPCK6m4i+TkQnieh1Ivr04PXdRPQ1Ijo9+L9rs7EMBsPtgTSqfgfArzrnHgbwGIBfJqL3AvgsgOecc0cBPDc4NhgM7wKkqZ13CcClQXudiE4CuAvApwA8Puj2DIDnAXwmOhjR0AWny0cTO9ayfC6kwmv9JuwyEc68WIZcJLKOn8ZVfV0jQGT1KbKNfET9DpoZOR0uxq6l5yhKewXGA7B+1UfQFdX8d+9ZHLbLrB4BlPv04HuOsinJ8TsNrzqvnPbZfq4jeRKrNR8NuKZU/fzuO4btyoJ3HXaaKpvwhC/fTRtyjA6L6ltmZB6lu4+IftPMRRgrY51QvlNHUcZOSekSTF1ibXOMtblHREcAfADACwAODH4Ubvw47L+pmRgMhokh9cInolkAfwLgV5xza5v1Z+c9RUTHieh4tVrb/ASDwbDtSLXwiaiI/qL/A+fcnw5efoeIDg7kBwFcGXWuc+5p59wx59yx2dmZWzFng8Fwk9jUxqe+T+ELAE46536Lib4C4AkAnx/8f3bTsQDkB/ZqPhcOt81pnnrWHh14e+M4bD+nZSOMFZ0OjRBn2VEhu2ndgFIg5yiIdcL2vyg97pRtvXRp2C4p9yn/biqsJPXCorxdFvd5G7zbke6xq2e8Xd+44sNjXa4i+q1c98rjumL4uffue4dtfrtUr14S/dbPf3/Yni8Vhaze9OHCraLn93/4Rz4i+hXKPjNQ71eIvZI4fU5EEKm/F4zYjfhxY/dcCqTx438YwD8H8B0iuhEI/e/QX/BfJqInAbwF4OfGurLBYNgxpNnV/xuEf5M+dmunYzAYJoHJE3HciE7TpZ8iZaeFjP0G5RL9eHZeuHR1Wh+MJrIcf4RRamNEJeOqeaTUtnD+aEIGYWX4g25bFsCqXvHqckFFQLZYVBsn1EBBqtEFZga06rKEdv1tr37v3eVju+obUp13jkX4deX7nN93wB90/fyvn/2u6Ecsq+9aXZKFLF33rr/F9/7QsL3nrsOiX9QU3JLnLP1JwksXIdsQ7uoQM2tKWKy+wZBB2MI3GDKICSfpEPIDdTyneM1yEVWfc6DxHX/Su/80uq3HjJkVwTpWevzAKfqVBB8IhWUSfIc/UhE3lp3BdMiG4s5vr60O20X1XbQ7nGfPV8TF/LTolyv426e2dFHISs6r5nl4kyCvogS5R2F217yQTTH3b+3iuWG73Lgu+hVn/byuri0JWWvRl/a6/4d/1AvUe47y3kfMAJIuFtaMkGhEbjkxj8iufpIPhAKC0bAnvsGQQdjCNxgyCFv4BkMGMXF33o0IPR2dJyL3Utr/CXebIOJIZ/8n+rHfQj2+sMYiGwryUMkioYcuZOxpl2B4COXO8/Z0fVnavr2mz2LrMrccADSYy63GMub2Hn5AX23Yql6Q3PxFloW3Uff7BI2GdLfVGVHGvh+U4zfXVvwY504N2yW1T1Br+eg8V5GRgQ996MeH7em5BSYJRzxqTxlFPnFp/4dtcEG2GeZHVQjvAiWyMoM9R8Oe+AZDBmEL32DIIHYgcq+vvkSj4qJRfUwV14k+rJ8m8wglASWuxU0QTYARUO+TLsGw6+aWEDKITJywrMc466+pMlnlkv/qF+alG222wggrGM/gdFFerM4SfRqX31bzYGYGS77hSTMA0Kr4EteFaVlC653vvDhsVxg3/0ZPJhxdZ9z/+T2S3392j4/+41GIhWK4THbse4mr1BGeRN4rcQFOIJOSQzHq4t0c9sQ3GDIIW/gGQwZhC99gyCAmbuMHs6Ai5IEkzO6wOy9md6cN2RX7BBEZRdx54rrjWF8hQzO1+0e6l+rXfQnq1TfPiH6LRe/2mtbEp4ywojjnw2a79WXRr3bKk1wWncy667Fbq834+Fc2ZJZg6bAn22hdPCtkM10/5jqz4zfakvRjvevnf/geyb/POUZ7rJy503sj/CNIxsOydrqA6WgtOyUKftWRWye8D5HOsWdPfIMhg7CFbzBkEBMuk+1V3/HcD6N7R71oW43c45mACVV/dGRgapNgxLEUsrYg5dDdYpFkXoddOvuGF9Rkdl5l0XPnk+LLc8xdVsp7tb+hVOwSebW9MidLaLXafh7dlh8vtzgr+uXbPqpvkWRUX2XKE3802Rwvrqr3cvdDw/bMLln+ipdji5ZRZ593pyZJpNtXvasyPyNdn6W9d/ohUn+3kW4UcwnyGzdckyEN7IlvMGQQtvANhgxi8rv6N35qImQEMcTV9BgBxuirxUyCpC3hmzHvQnwnPxIFJupf8WZ6ta7FCDeunnpt2M4reu1mw6vY5GSSTo+p1cT4+DTxSbfpd9pnFqSqX2H0fOWyP6+hdvWp5qv2FuYk0Uebcf8JrsU5qc7fcf97/Rh5XUGZXYu9rsk16pc9BfjKS/9PjlHzpCVtZvoAwP5/9Mlhu7LXRwkmKDTSbvKnrJI1rmqvYU98gyGDsIVvMGQQtvANhgxisjY+0dBdRopsQ0bd6RNDdvEYIW1BeswkZYJvhmUuZJghXkIr6s4LvAEXqdvkVAjaO2dODNut1avD9owi27i27l1Whbrky5+e8lF9eeay099Zh3HdJ8qe5f2tlWPXnlMRfh32ebQa+n368Vfrvr1w7/tEr0LR2/WdliqTXfJz7rR9u3bx+6LftePeru+uS3dek+01rHfkHsVs3ReCLUUM79itGc6zi5TQCtxGaU3/TZ/4RFQhoheJ6BUiep2IfmPw+r1E9AIRnSaiLxFRabOxDAbD7YE0qn4TwEedc48AeBTAJ4joMQC/CeC3nXNHAawAeHL7pmkwGG4l0tTOcwBu+IiKgz8H4KMAfnHw+jMAfh3A7242Xj7AuUdBVRxpNJyBLB3HPGevcOq3L2ZKiGq2TMXuKVU8x3w3ifJXPGtE/+wK3rco0/uwVb++IiTnX/XkFTli71N93lPzPvlmbb0mZI26J70oFv0kxdwBtLteBb6qot3yzK02XfFuukJOmhWlgjcrWg1J0lFd9260jYqPNCwrl93aqncJdmakSxDMjVm94JOA1k68LLoVGl6FbzakOr/MOAOn771fyKYY0QdPAtJITZvBRNrEi0b1jcnEkWpzj4jyg0q5VwB8DcD3AKw6X/jsAoC7xru0wWDYKaRa+M65rnPuUQCHAHwQwMOjuo06l4ieIqLjRHR8fb06qovBYJgwxnLnOedWATwP4DEAi0R0w1Q4BOBi4JynnXPHnHPH5uZmR3UxGAwTxqY2PhHtA9B2zq0S0RSAn0B/Y+/rAH4WwBcBPAHg2U3Hgg/7jNasS40oJUX4SHDbh21wbbEJrgZmbzll24kj7bXkxIo9TeqQjtyj0/YusTPf+oaQ1VeuDNszjGO+m5MhuxzT85KLvszcbzwEVtv4nQ7LwOvIz4B7Gfn7bLblGGtVrwUuL8uaeOsbPluv8sDBYXuqKvcTOmy+ua60z4tLPhSXVi57QVu6FWtt//ms83qBALqLu4ftBz/yE0JWKPlrSzeu6KYyLHWdBObKjoXsRrZ9xo3gTePHPwjgGSLKo38bf9k591UiOgHgi0T0nwG8BOALY17bYDDsENLs6r8K4AMjXj+Lvr1vMBjeZZhwdh4hh9GqfowHL5jtptUp4W7TfV2gX8xlp82F0ep9jC+hp9TjUI2AxIms2e1KNf3sSy8M28tnXxOyg/t2DdtTJR9TpedRq/sIt5Yi2Ggx+8Qxz5mahih/VV2XEXNt5prrdf21N1Tk2/WqV6vXanKM6b2+xPUU49zPtWW/mZ43FwrXVoWMmM2RL3pXYkvdYxdWmFt0breQfeDxn/JzWtglZD12j8gMvPQFFfh9lRPuPNlPlmvQg4yn7FusvsGQQdjCNxgyiAkn6XhVN0ZekSTRCBLmpb+22Fbl+lT0JHXIefAo1AsUi84To2u/weh5XXlLJpS88YLfyT+4ICPhZplKnGdRcvUNyWfX5NVsN1RZq3W/493h6r2KJKtX/Zgrq3KnvVbzMl7qrNmW9kKdVeatVGS6xwIj8Ni17st13ZWX0XmzLCFouSN369eb/oNswb/PK4q3r3y3j8h76LEfE7LF/WGCDWEOivsxfGNRsobWyLN0BbfgdfuDhjuPgD3xDYYMwha+wZBB2MI3GDKIyZNtpnDnxVkGA20/9OanRbg2pCzs6qPIINz+16ZX3BLzNn+n7d1eJ178pujVbfpsukJOup4aTe+aK5WZzZyXewE9Zv93IF1sGy1vJ19nbrr1NRnRtrbu7fjlNeli45F8BWbjzyo7/vAdnjjz/nsOCtnCot+vmGKnzZfkbdvpMjIP+VZwaWXdz5/tL+x+zw+Ifg8c+7Cf47wkDtUl1wVEsB67P5I7Vb5ftE4CGy9SykuvnxuZfGm9evbENxgyCFv4BkMGsQOq/mhdRCSvRM6Ok23ELhMQRnT9uBnA2yo6L1b+KqAaaqyv+kiy5YtvCdluRlrPue0AgApe5pg6X61Lso2Vaz4hpqoIMK6vevU4z3j1V9alqv/2ku+3e1Gqx3fs9pmYh/d7df7OfTLybd+iL0lVLEpzpMXMnQ5z0603pEtwjfHxXViWbsWllv8CjjzyI8P2PQ/9oOhXnmKRgUq158faChXqvYup87EEHuHEGzm2Hj8xyHYQcRgMhn9YsIVvMGQQtvANhgxisja+8/ZwPCtOhbK6QHjsWK64QDuR5BQWhjP3NKEGk8TsuYirb+mSL83s2qp89IK3n4slyZdfLPtwVl792nXlxeZn/RilkrTxdzHCyqlpT8rZo/Oi35133TFsP3CfpFycLvhnysKsH29+RrIw5di0mk2ZJdhkbsXVmp/jcm1d9Fvm5KAH7hGy9z/iM8d37d3n+5XkfgInB83rfRMRnh0ONY/dmzFXXGqXYCTSfMyIXXviGwxZhC18gyGDmLA7z3lVXxFDuFxIjZZkBzl2niPF88ZYIyLVr+Kcezw6L5adFyHsEO68McIL+TjXLnm1erokf58rRa/elwoyEo6rfJ2WV4G19VQqsjJZKqqvxcLflla8e6yrSDR+4MG7h+09u2aErFhi7jE+p66cSJH8d9ZT5shG08//Msumu9aRt+2Bh7w6f+d7HhCyGWbScLccd1MCUtXX7rxY+fUww0b4e0/cmzxoNULYEbws+H2WTue3J77BkEHYwjcYMojJR+7dUPWh1XnfTuyEpyxdRRH1m5/HSxMl+QxiEVasHxP2tB7NDqMJHgrttldta9d8pdtKSanzbNe5p0pSdcDMHS4jqdq2mn7HfKMhI/I4rfU7K57DrlSQY8xOMVprSJQL7NZiHxypeayu+ghCzuEHAEs8QSjvTYlDjzwm+i3u8dx8BRX9x9V0uXMv55HLpVP1E6abC+jpYySQUcBE0B6ElMOlgj3xDYYMwha+wZBB2MI3GDKIidv4Q6dDNCsuFtU3+pz+4Wg7PjlGpEx2JIJQmHCOZWwlU7HY+HKOOYTtxQYrJ9Ws+uy82YK2R/3Xxl1eAFBvsj2Qjh+/oRgqWiysr6pkK4zfnpfCPrhnUfSbmvYuQafcdLxkNOfj17b1BiPHuLC0JOfR9p/V/AOP+uvOzot+nMyTEpl1+U3bAJDL8zHCJBqJe070SuuLUwgQdkYzUW8SqZ/4g1LZLxHRVwfH9xLRC0R0moi+RESlzcYwGAy3B8ZR9T8N4CQ7/k0Av+2cOwpgBcCTt3JiBoNh+5BK1SeiQwD+CYD/AuDfUN+/8VEAvzjo8gyAXwfwu7FxHJgqPUbpKohoPeZGy8l+FOgHqMjACBGHGD8SkRd3HfJzIGUR9W31iueO72x4tT+nyotz12dP8dR3eWkvFgnX6cgEmDYjuaiq6rB1FvFXYu6xfbsl2QYxdZm7IgEI3sEuM5nayjRpMOL+taY0OdyeI8P2zG7vssspc4FH4eXz8lnGj7k6nyDbYOXMgiXbRmK0GZrIw4nJUl4pRvSRfpQ+0j7xfwfAr8F7qPcAWHXO3bibLgC4a9SJBoPh9sOmC5+IfhrAFefct/nLI7qO/MkhoqeI6DgRHV9fr47qYjAYJow0qv6HAfwMEX0SQAXAPPoawCIRFQZP/UMALo462Tn3NICnAeC+I4e3b5vSYDCkxqYL3zn3OQCfAwAiehzAv3XO/RIR/RGAnwXwRQBPAHg2zQW9Oy9xId/WmXvE7Shmwyb4OmhkP92Xj66r13F3m3aniDLZ7Mwkxzl3K4aVql5P2ucXv/fdYTvf67BkZxQOAAAWDklEQVR+cpZVFlKb198gm3K75cdoNWRNuRoL062qunrVupc9eI8n25guS8dNS2T/yffCySxazP6vq/2E61XvOuyUZYbf/MF7h+1YSG2MRIP3zbHvKadcdiQ8drHnU3g/Jxeppxi1wcXF2cuJ+4/dcwmSju2x8UfhM+hv9J1B3+b/wk2MZTAYJoixAnicc88DeH7QPgvgg7H+BoPh9sRtk50X4snon5KOAEOS6WlVn7tC2BjKrADxqL7I8EyFj46R0MD8ebW160KydP7csF1kKirnlwcAx7TqHEkVGx1ujvj33FCqPlex15SqXyl7F95djAefnLyWYy7CvIou7LGIP06+sVaTpbaWq57vvzd3p5AVyj4ykFs7PfV5d0WUoHJvsuNuN8/aKioz5+eb60Ui97SEa+kUjiAU0YA6q5QbnGyMJGFHOIJwu9x5BoPhHxBs4RsMGcTk6bUHapriYxDKVELRooA0ou0krYDRwsTuP1fDtNcgwAsYI2dIqmv+hetLV4WotuYTc2ZKjIsOMuquy3bQqSdV+BIvocVe19F51QZX7+UkDzD67jmWiNNqSDWdE3PkdfkrtuPfYBF56w3lQWBq9fSuO4SswyIPc23/GbR15F7ej0+kVGxBaBfede902HmRarY66k6W12KkH8r0KTBvg05Ukggn+sQY/caFPfENhgzCFr7BkEHYwjcYMoiJ8+oniCkHIBFVFeE1p9Gv918Yn5Q8EQEljHI9V0ZkidH2PiB55BPWGHNFVddWhajNCDG4JZzr6Qw8dqA+gyqzp7kra70u9wKazN1WVvbonQd8Jhx3lfES3ABQYCSgmthig5XeXq/5/YWVqtwn6Ezt9mOUpoWMz7/D7OKCcsV1WIZfPq+zEEdn5+m7o8fuuRhPpr7neASgsN0T/fycc4lozgBvf7xevIRT/zeBPfENhgzCFr7BkEFMVNV3zqE3iPaisqzyKpIpErxpzAzgar9KtOBkCrmISkZx52FQIk2EsNooKp4q1YsTYKwqd16bqaw8OK1Tk2q6i8yfR7jx8ZqKiIObJ/t2Sw67+Yq/LRwzCfJFFZ0X4kIE0GTuvDUWnbeyIYk4KnsPDNtdlYxEQoVnBCmqHz/WUX38WJgO+h4TJmg4gSdH+lnJ71v2ctTVHNbHe9zkTUj5fZU+unAU7IlvMGQQtvANhgzCFr7BkEFM1Mbv9Xpo1Pv0W7OzkrixWAiHNOby6Wx8YYtF9glI7BnoLKow6aIgZBQuRh1ayUJDlUewyujHls6/KYXsPG7vatJPJ+xW5drq8kw136+gSCjn53wZ60N7JJlnmdnTOebq6yrDtcky/kgRn1aZ+3Dpun/PG3l5rRJ5F2FTEXEKU57Ce0AkvhdNosmPwi7YArv/kvcOc/VFQnbT7lMl3NB8XynijosRtSaG3AT2xDcYMghb+AZDBjFZVb/bQf36tX6blTYGgEJhmrVlhJhU0/3rySgqLpPXppCanjAXImaAUO/D6iW/ti5B/b3XXxm215YuCVmh4Mcp5HmUoJxjo8lILpT7ih/zWc2o7LndFe9OnatI1yp3VbbbjBO/IwlB2jnvHmt3pZq+vOZdeFfrzE25uFv04xF+RV1tXJSgjqn6o9v6OEb2wlEoKt6+yLX5MTdRY+ZI2qy7ZAmtcH7esFwF0sGe+AZDBmEL32DIICabpNPtoLPWV/Vra8tCtLDoud2KujpsPrSbLoen2JZoYBc+MUYgSrAvC5E1qN3upt/RPvP6d4Ts9PFv+oOOVI/zbGd5quITYDgHHgA0Wl4nrm4oIo66H5NvtC9MTYl++xe8V6VSkLTZ7ZZXzWtNn1SjI+YKRW8irFYlwcbbq4y+O++r7BbVLcf5BHVprLRci1Ey7AA5S1KJZsk2uTCJBt/918cxVT9Ko0GjZdrEix3FKcGTsCe+wZBB2MI3GDIIW/gGQwYxWRufgNzADq9euyJEnTt8sd2ZGUnIEIx6Stj0kfrUfBoUOtB2fNidx89rqRLRb545PWyfevH/Clm+7d1ceeU24mFmBU5kmYhk9O9tqiTHKHHXEyOynFPZkFOsX6ch3XTrLIKuzUhANH9Eo+PHv7wiC6IudVk5rBlv4+vPlNvCBVX+irs0uYutqFyT/LioyULYebxdUvsaBTZGch7s2gkbf/T3FCvRlUCIPyZmtqv7e9zIvVQLn4jOAVgH0AXQcc4dI6LdAL4E4AiAcwD+mXNuJTSGwWC4fTCOqv/jzrlHnXPHBsefBfCcc+4ogOcGxwaD4V2Am1H1PwXg8UH7GfRr6n0meoYD3CCJxDVrQlRd9qr/wsKikOWY+iZcdrpKbYRLT7hGIrx9kuIi7DbipA5vn/u+6PfyN54bthvX3hGy+RmvcueUqs+D8BqMyEK77Fosmi5B6c+SdKaZOqstnyarpNvsSDddq8sJQfyJDeXOq3X8vC5Jbx5as/47LDE1ulSSqniZVeAtqWq8XDbFymlVKhXZj3H/6fH5cZmZO2XVr8iOC5oTv8DdecqUYH1jbuJo1B07zkUJNcLEJ+Mi7RPfAfgrIvo2ET01eO2Ac+5SfxLuEoD9NzcVg8EwKaR94n/YOXeRiPYD+BoRfXfTMwYY/FA8BQC7Zqc36W0wGCaBVE9859zFwf8rAP4M/fLY7xDRQQAY/L8SOPdp59wx59yxmanyqC4Gg2HC2PSJT0QzAHLOufVB+ycB/CcAXwHwBIDPD/4/m2KsoUulrDjrN1imWn2ftBoWdvtMPm5HOTVGWhryqH0UKc3XZYSV5753Zth+5W+eF/2q77w9bJeL8rfVyfRCIePEkA1GZNFoSaJMXlOu05Kc+2W2b1CZ8bZvU7kc13p+TKfKQnNyjwZ7z1X1wS07f62N6QNCVmI2ObfVKxVtx5eZTD4YuC3PZWW9F8Bs/KSMufp4W+2vcDddQZGKxkhiRNZdJIxbhuJKhO5NvU8QlY1JrJ9G1T8A4M8GC64A4H855/6CiL4F4MtE9CSAtwD8XKorGgyGHcemC985dxbAIyNevwbgY9sxKYPBsL2YaORejoDpgRqVV66hbsO795YvXxSymXnP+15kal3ClRW7eCCSL6GQcZdaXZZ7euPkyWH7xIs+y669fFn0E94gFcEl+PN6uhQU59LjrhvlmmTnVZRqy11PNVYae6OtuOgZZz03HQD5mbRZZuRGUaribtfBYbs0I7n5ucotVP2ydMVxFb6iogsrgTFKpbCqr915/H4p8Si+mMsuUYabZYdGku6iQXdRLj0+XiSF8BbCYvUNhgzCFr7BkEHYwjcYMoiJ2vgEQnnAbtJVrjjHCOhr16TNfH3Zu/N27WNuo4RLg4kSBe1Gh1NqVpn1tbVh++SrrwjZ2Ve/7Yer+Xwk7bLrRahe2pwHP1Hnzbd5lmC+IMcXrC/K6Gwz9xt3xdVqDdWP15tT+y18T6HimXtm9x0U/SpzPiw3Xwy76WRYbjikNhZuyzPwStoVJzL3YtlzLBNQZdlxN13MZZcoqx5IrYv1ikF4THUGHq/XMM4mwgjYE99gyCBs4RsMGcRkiTjg4AYRY6QIDUWvluSiv8Cy36jkVc/pGUkgyVW5GPc6V5OuviMjjV/51t8N25dPS6LMEuOOL5WYKq6uJfjbtf+HHSsvmhinyFVPZRK02yyzrqui+ljfGi9V3ZIZfg3uOlSRe+VZ75rbe+i+YXt2YY/ol2PZf/lE1tpogspSUavzfoykms5dbLmR7X6/GGkJd8WlLMWeuHfSZd2JEldRzTtt6Khmgo2dZmWyDQbDJrCFbzBkEJON3MvlMDXdT82t1qU6z/nsu0q1ff3EKT/GnOffv/vuu0Q/vnus3xhX366vrg7bL/3dN0W/K2deG7YrkFx0MvKL7xArPnimdvWUfsZVcR6pB8jd9TZr68g67jXQlXSbbCd/gyX31FTkXoep9/N7ZYLNnUfuH7ZLU547T5tnuXw4eYV/JqKt+0V207manhftcD9d9owf5aI1GXxbc9SLxLDEdno6wjx+XpIkho/AM3HUlUKXAkDaLNgE9sQ3GDIIW/gGQwZhC99gyCAmauN3ez3Uqn3+dR051WUGzNmLS0L2xjlPWHnk4XV/TldFnDGSSG3D1es+++/l4y8O25feeE30m2J2fbms7dZ0JZFdom6aR57NK6ci5prMTdfu8uw5OQa36ztKWG/5+dcYSQepz3vfvjuH7bvue0DOkbnpHHs2EKmINh5dqOz/fG60fZ7Lyy+GH2suemGT5yL2OSdZTfDNx1xxcpQgIswtTjpvI2Owps4UFV7AQLqfPkwM0dt0Chz2xDcYMghb+AZDBjHhyD0AQxeQ1ElWGXf8375+Rsim5haG7TYrq9zrSjW31/Xq5roi0TjFylWfe/X4sF3qqTLT5dFJHYCMGOP8bU65HzeY+q2j83iSTku587irjyfRtLthk6Cu+PiqDX/Mk2323HGn6Hf4voeG7ZxKsOlx15Bwj+myUExNT6jw3ESIRMwF1Pl+X95v9Dl6Hkl1frTum+wWVtm3xIehzQPhL9Rd3chuiX78fW5lTgz2xDcYMghb+AZDBmEL32DIICZq4zvn0Gr2CSFaKsTwr1/yxXmW12VdvaN3eAIIbgNxex8Arl/3objfP3VSyC6c8KQahba3/6cqKluMhZdWFDGECA1lr3cT/iUvbSl3Gw+/5aQZANBg7rdWmxNqyDH48UZTk2j49q79PhT33gceFv24y06HSHNyDyfIH2K2tZTlRSZcuB9txW5NW3JadZXmc4T3vhceJMZ1L0dTbkXWcascGmL+AaaPtPsR9sQ3GDIIW/gGQwYxWVUfQGegjPyfV08L2Rtvei79ubk5IeNkDdV1H7l36qRU598+68fcuHJeyCrk1eMZpt4XVWZdMZJJJlXMSISViFqTMu4qc06q8FzlbjEXXlOp+m12TMo9tmuX5yc8+vD7hu2i4qzvsPF1RpvO+PMXU4fcxRYhr8hFVOWQKp6QRcLWYmOIsyJciDJFLpxZl/xoRvvfYip3WnVcZ9zFysBvC+ceES0S0R8T0XeJ6CQRfYiIdhPR14jo9OD/rs1HMhgMtwPSqvr/DcBfOOceQr+c1kkAnwXwnHPuKIDnBscGg+FdgDTVcucBfATAvwAA51wLQIuIPgXg8UG3ZwA8D+AzsbFqG028cLLPn/f6WamKc3WwrOikW1VPeX3uxKte0JS7/2UWhTdbkjr2NKN1rrB2TquejFxC70CHuPS0aiwjsSgo0+fx/Xku0ao4T0SpqNJV9zP1vsyqzbYTmT58TioCUkyLc9Zp2mnfzqsPMi847Ear/X3Z6PH6x1y1xch2H5y8IqzyqmJmagQ2kYQ+H7m6KHkVZspIehFGj8979RL02rcOaZ749wG4CuB/EtFLRPQ/BuWyDzjnLgHA4P/+2CAGg+H2QZqFXwDwQwB+1zn3AQA1jKHWE9FTRHSciI7zGHODwbBzSLPwLwC44Jx7YXD8x+j/ELxDRAcBYPD/yqiTnXNPO+eOOeeOlYuTzwkyGAxJbLoSnXOXieg8ET3onDsF4GMATgz+ngDw+cH/Zzcbq9NpY2nAY39oTmaEcQNvtiTtr4WWj8irOJYhV5BWz9QU42hXmXXlkn+rFdbWbiiWFCfIQQAZxdbpdVhb9uNJg5oshEfrdRURB7f5+bwcyX5U9Lb7PfdLEo0Cy7RrNHzZLD0PSdgpRHBgJCPMH1lIcNFzIg6dWeeP88K1FybbSOy3BDym2taNPb1I7sz417Udz0q4uS1b09ztF7PPw664+JUDHP5bQNpH8L8G8AdEVAJwFsC/RP/z/jIRPQngLQA/d1MzMRgME0Oqhe+cexnAsRGij93a6RgMhklgokZ3MUc4MH3jkvLSvOLslKqoWhBlkLy6U1Buvwpz4ZUKuhwTL0/FVH1lEuRYtJQmwGgx1Zwn0XRVhBXX4DUnHnertTqK655dj3Pi19We6B33HBq2CyVpMq0z1ycfTyficLUxF+HLL5KP+NN8dly916QlQhZw7QGS50O78yQnPleHw2F35JRZxKskMzNRq+IiMSduO6jxR80iifTUHlvj/ks3tofF6hsMGYQtfIMhg7CFbzBkEBO18fNEmKv07clkqWN/XFYEGNz4E6Wki9I2LRdH12vrDzF6DJ3d1mO2Xk/ZS9xOFll20qwUdvxGU5KF8L2BVjtMsFFnssrCXtGvVPHlwdfWqmr+fjK9SAipLGMt558TNQP869plV4hx4qcN2eUzTHDij5bFbPxYih/ns9fuMCdimFWIdLA+nrwPInyaqblDku+N9ePZkMFndjpXpD3xDYYMwha+wZBBULLs7zZejOgqgDcB7AWwtEn37cbtMAfA5qFh85AYdx73OOf2bdZpogt/eFGi4865UQFBmZqDzcPmsVPzMFXfYMggbOEbDBnETi38p3fouhy3wxwAm4eGzUNiW+axIza+wWDYWZiqbzBkEBNd+ET0CSI6RURniGhirLxE9HtEdIWIXmOvTZwenIjuJqKvDyjKXyeiT+/EXIioQkQvEtErg3n8xuD1e4nohcE8vjTgX9h2EFF+wOf41Z2aBxGdI6LvENHLRHR88NpO3CMTobKf2MInojyA/w7gHwN4L4BfIKL3Tujyvw/gE+q1naAH7wD4VefcwwAeA/DLg89g0nNpAvioc+4RAI8C+AQRPQbgNwH89mAeKwCe3OZ53MCn0adsv4GdmsePO+ceZe6znbhHJkNl75ybyB+ADwH4S3b8OQCfm+D1jwB4jR2fAnBw0D4I4NSk5sLm8CyAj+/kXABMA/h7AD+MfqBIYdT3tY3XPzS4mT8K4KvoB5vvxDzOAdirXpvo9wJgHsD3Mdh72855TFLVvwsAJ9O/MHhtp7Cj9OBEdATABwC8sBNzGajXL6NPkvo1AN8DsOqcu5FFNKnv53cA/Bo80f2eHZqHA/BXRPRtInpq8Nqkv5eJUdlPcuGPShvKpEuBiGYB/AmAX3HOrW3WfzvgnOs65x5F/4n7QQAPj+q2nXMgop8GcMU5923+8qTnMcCHnXM/hL4p+stE9JEJXFPjpqjsx8EkF/4FAHez40MALgb6TgKp6MFvNYioiP6i/wPn3J/u5FwAwDm3in4VpMcALBLRjSTdSXw/HwbwM0R0DsAX0Vf3f2cH5gHn3MXB/ysA/gz9H8NJfy83RWU/Dia58L8F4Ohgx7YE4OcBfGWC19f4Cvq04EBKevCbBfVJAb4A4KRz7rd2ai5EtI+IFgftKQA/gf4m0tcB/Oyk5uGc+5xz7pBz7gj698NfO+d+adLzIKIZIpq70QbwkwBew4S/F+fcZQDniejBwUs3qOxv/Ty2e9NEbVJ8EsAb6NuT/36C1/1DAJcAtNH/VX0SfVvyOQCnB/93T2AeP4q+2voqgJcHf5+c9FwAvB/AS4N5vAbgPwxevw/AiwDOAPgjAOUJfkePA/jqTsxjcL1XBn+v37g3d+geeRTA8cF3878B7NqOeVjknsGQQVjknsGQQdjCNxgyCFv4BkMGYQvfYMggbOEbDBmELXyDIYOwhW8wZBC28A2GDOL/A/xA0q3yRwv9AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Example of a picture\n",
    "index = 5\n",
    "plt.imshow(X_train_orig[index])\n",
    "print (\"y = \" + str(np.squeeze(Y_train_orig[:, index])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As usual you flatten the image dataset, then normalize it by dividing by 255. On top of that, you will convert each label to a one-hot vector as shown in Figure 1. Run the cell below to do so."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of training examples = 1080\n",
      "number of test examples = 120\n",
      "X_train shape: (12288, 1080)\n",
      "Y_train shape: (6, 1080)\n",
      "X_test shape: (12288, 120)\n",
      "Y_test shape: (6, 120)\n"
     ]
    }
   ],
   "source": [
    "# Flatten the training and test images\n",
    "X_train_flatten = X_train_orig.reshape(X_train_orig.shape[0], -1).T\n",
    "X_test_flatten = X_test_orig.reshape(X_test_orig.shape[0], -1).T\n",
    "# Normalize image vectors\n",
    "X_train = X_train_flatten/255.\n",
    "X_test = X_test_flatten/255.\n",
    "# Convert training and test labels to one hot matrices\n",
    "Y_train = convert_to_one_hot(Y_train_orig, 6)\n",
    "Y_test = convert_to_one_hot(Y_test_orig, 6)\n",
    "\n",
    "print (\"number of training examples = \" + str(X_train.shape[1]))\n",
    "print (\"number of test examples = \" + str(X_test.shape[1]))\n",
    "print (\"X_train shape: \" + str(X_train.shape))\n",
    "print (\"Y_train shape: \" + str(Y_train.shape))\n",
    "print (\"X_test shape: \" + str(X_test.shape))\n",
    "print (\"Y_test shape: \" + str(Y_test.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note** that 12288 comes from $64 \\times 64 \\times 3$. Each image is square, 64 by 64 pixels, and 3 is for the RGB colors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Goal** is to build an algorithm capable of recognizing a sign with high accuracy. \n",
    "\n",
    "**The model** is *LINEAR -> RELU -> LINEAR -> RELU -> LINEAR -> SOFTMAX*. The SIGMOID output layer has been converted to a SOFTMAX. A SOFTMAX layer generalizes SIGMOID to when there are more than two classes. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 - Create placeholders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_placeholders(n_x, n_y):\n",
    "    \"\"\"\n",
    "    Creates the placeholders for the tensorflow session.\n",
    "    \n",
    "    Arguments:\n",
    "    n_x -- scalar, size of an image vector (num_px * num_px = 64 * 64 * 3 = 12288)\n",
    "    n_y -- scalar, number of classes (from 0 to 5, so -> 6)\n",
    "    \n",
    "    Returns:\n",
    "    X -- placeholder for the data input, of shape [n_x, None] and dtype \"float\"\n",
    "    Y -- placeholder for the input labels, of shape [n_y, None] and dtype \"float\"\n",
    "    \n",
    "    \"\"\"\n",
    "\n",
    "    X = tf.placeholder(tf.float32,[n_x,None],name='X')\n",
    "    Y = tf.placeholder(tf.float32,[n_y,None], name='Y')\n",
    "    \n",
    "    return X, Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X = Tensor(\"X_26:0\", shape=(12288, ?), dtype=float32)\n",
      "Y = Tensor(\"Y_8:0\", shape=(6, ?), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "X, Y = create_placeholders(12288, 6)\n",
    "print (\"X = \" + str(X))\n",
    "print (\"Y = \" + str(Y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 - Initializing the parameters\n",
    "\n",
    "The function below will initialize the parameters in tensorflow. We will use Xavier Initialization for weights and Zero Initialization for biases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "def initialize_parameters():\n",
    "    \"\"\"\n",
    "    Initializes parameters to build a neural network with tensorflow. The shapes are:\n",
    "                        W1 : [25, 12288]\n",
    "                        b1 : [25, 1]\n",
    "                        W2 : [12, 25]\n",
    "                        b2 : [12, 1]\n",
    "                        W3 : [6, 12]\n",
    "                        b3 : [6, 1]\n",
    "    \n",
    "    Returns:\n",
    "    parameters -- a dictionary of tensors containing W1, b1, W2, b2, W3, b3\n",
    "    \"\"\"\n",
    "    \n",
    "    tf.set_random_seed(1)                   \n",
    "    W1 = tf.get_variable(\"W1\", [25,12288], initializer = tf.contrib.layers.xavier_initializer(seed = 1))\n",
    "    b1 = tf.get_variable(\"b1\", [25,1], initializer = tf.zeros_initializer())\n",
    "    W2 = tf.get_variable(\"W2\", [12,25], initializer = tf.contrib.layers.xavier_initializer(seed = 1))\n",
    "    b2 = tf.get_variable(\"b2\", [12,1], initializer = tf.zeros_initializer())\n",
    "    W3 = tf.get_variable(\"W3\", [6,12], initializer = tf.contrib.layers.xavier_initializer(seed = 1))\n",
    "    b3 = tf.get_variable(\"b3\", [6,1], initializer = tf.zeros_initializer())\n",
    "\n",
    "    parameters = {\"W1\": W1,\n",
    "                  \"b1\": b1,\n",
    "                  \"W2\": W2,\n",
    "                  \"b2\": b2,\n",
    "                  \"W3\": W3,\n",
    "                  \"b3\": b3}\n",
    "    \n",
    "    return parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "W1 = <tf.Variable 'W1:0' shape=(25, 12288) dtype=float32_ref>\n",
      "b1 = <tf.Variable 'b1:0' shape=(25, 1) dtype=float32_ref>\n",
      "W2 = <tf.Variable 'W2:0' shape=(12, 25) dtype=float32_ref>\n",
      "b2 = <tf.Variable 'b2:0' shape=(12, 1) dtype=float32_ref>\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "with tf.Session() as sess:\n",
    "    parameters = initialize_parameters()\n",
    "    print(\"W1 = \" + str(parameters[\"W1\"]))\n",
    "    print(\"b1 = \" + str(parameters[\"b1\"]))\n",
    "    print(\"W2 = \" + str(parameters[\"W2\"]))\n",
    "    print(\"b2 = \" + str(parameters[\"b2\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 - Forward propagation in tensorflow \n",
    "\n",
    "We will implement the forward propagation module in tensorflow. The function will take in a dictionary of parameters and it will complete the forward pass. \n",
    "\n",
    "It is important to note that the forward propagation stops at `z3`. The reason is that in tensorflow the last linear layer output is given as input to the function computing the loss. Therefore, you don't need `a3`!\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "def forward_propagation(X, parameters):\n",
    "    \"\"\"\n",
    "    Implements the forward propagation for the model: LINEAR -> RELU -> LINEAR -> RELU -> LINEAR -> SOFTMAX\n",
    "    \n",
    "    Arguments:\n",
    "    X -- input dataset placeholder, of shape (input size, number of examples)\n",
    "    parameters -- python dictionary containing your parameters \"W1\", \"b1\", \"W2\", \"b2\", \"W3\", \"b3\"\n",
    "                  the shapes are given in initialize_parameters\n",
    "\n",
    "    Returns:\n",
    "    Z3 -- the output of the last LINEAR unit\n",
    "    \"\"\"\n",
    "    \n",
    "    # Retrieve the parameters from the dictionary \"parameters\" \n",
    "    W1 = parameters['W1']\n",
    "    b1 = parameters['b1']\n",
    "    W2 = parameters['W2']\n",
    "    b2 = parameters['b2']\n",
    "    W3 = parameters['W3']\n",
    "    b3 = parameters['b3']\n",
    "    \n",
    "                                                           # Numpy Equivalents:\n",
    "    Z1 = tf.add(tf.matmul(W1, X),b1)                       # Z1 = np.dot(W1, X) + b1\n",
    "    A1 = tf.nn.relu(Z1)                                    # A1 = relu(Z1)\n",
    "    Z2 = tf.add(tf.matmul(W2, A1),b2)                      # Z2 = np.dot(W2, a1) + b2\n",
    "    A2 = tf.nn.relu(Z2)                                    # A2 = relu(Z2)\n",
    "    Z3 = tf.add(tf.matmul(W3, A2),b3)                      # Z3 = np.dot(W3,Z2) + b3\n",
    "    \n",
    "    return Z3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Z3 = Tensor(\"Add_2:0\", shape=(6, ?), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    X, Y = create_placeholders(12288, 6)\n",
    "    parameters = initialize_parameters()\n",
    "    Z3 = forward_propagation(X, parameters)\n",
    "    print(\"Z3 = \" + str(Z3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Noticed that the forward propagation doesn't output any cache."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 Compute cost\n",
    "\n",
    "Cost is computeed as follow:\n",
    "```python\n",
    "tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(logits = ..., labels = ...))\n",
    "```\n",
    "\n",
    "Note that I am using the v2 of tf.nn.softmax_cross_entropy_with_logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "# GRADED FUNCTION: compute_cost \n",
    "\n",
    "def compute_cost(Z3, Y):\n",
    "    \"\"\"\n",
    "    Computes the cost\n",
    "    \n",
    "    Arguments:\n",
    "    Z3 -- output of forward propagation (output of the last LINEAR unit), of shape (6, number of examples)\n",
    "    Y -- \"true\" labels vector placeholder, same shape as Z3\n",
    "    \n",
    "    Returns:\n",
    "    cost - Tensor of the cost function\n",
    "    \"\"\"\n",
    "    \n",
    "    # to fit the tensorflow requirement for tf.nn.softmax_cross_entropy_with_logits(...,...)\n",
    "    logits = tf.transpose(Z3)\n",
    "    labels = tf.transpose(Y)\n",
    "    \n",
    "    ### START CODE HERE ### (1 line of code)\n",
    "    cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(logits = logits, labels = labels))\n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    return cost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cost = Tensor(\"Mean:0\", shape=(), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    X, Y = create_placeholders(12288, 6)\n",
    "    parameters = initialize_parameters()\n",
    "    Z3 = forward_propagation(X, parameters)\n",
    "    cost = compute_cost(Z3, Y)\n",
    "    print(\"cost = \" + str(cost))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.5 - Backward propagation & parameter updates\n",
    "\n",
    "All the backpropagation and the parameters update is taken care of in 1 line of code. It is very easy to incorporate this line in the model.\n",
    "\n",
    "After the cost function computation, we will create an \"`optimizer`\" object. We have to call this object along with the cost when running the tf.session. When called, it will perform an optimization on the given cost with the chosen method and learning rate.\n",
    "\n",
    "For instance, for gradient descent the optimizer would be:\n",
    "```python\n",
    "optimizer = tf.train.GradientDescentOptimizer(learning_rate = learning_rate).minimize(cost)\n",
    "```\n",
    "\n",
    "To make the optimization we would do:\n",
    "```python\n",
    "_ , c = sess.run([optimizer, cost], feed_dict={X: minibatch_X, Y: minibatch_Y})\n",
    "```\n",
    "\n",
    "This computes the backpropagation by passing through the tensorflow graph in the reverse order. From cost to inputs.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.6 - Building the model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "def model(X_train, Y_train, X_test, Y_test, learning_rate = 0.0001,\n",
    "          num_epochs = 1500, minibatch_size = 32, print_cost = True):\n",
    "    \"\"\"\n",
    "    Implements a three-layer tensorflow neural network: LINEAR->RELU->LINEAR->RELU->LINEAR->SOFTMAX.\n",
    "    \n",
    "    Arguments:\n",
    "    X_train -- training set, of shape (input size = 12288, number of training examples = 1080)\n",
    "    Y_train -- test set, of shape (output size = 6, number of training examples = 1080)\n",
    "    X_test -- training set, of shape (input size = 12288, number of training examples = 120)\n",
    "    Y_test -- test set, of shape (output size = 6, number of test examples = 120)\n",
    "    learning_rate -- learning rate of the optimization\n",
    "    num_epochs -- number of epochs of the optimization loop\n",
    "    minibatch_size -- size of a minibatch\n",
    "    print_cost -- True to print the cost every 100 epochs\n",
    "    \n",
    "    Returns:\n",
    "    parameters -- parameters learnt by the model. They can then be used to predict.\n",
    "    \"\"\"\n",
    "    \n",
    "    ops.reset_default_graph()                         # to be able to rerun the model without overwriting tf variables\n",
    "    tf.set_random_seed(1)                             # to keep consistent results\n",
    "    seed = 3                                          # to keep consistent results\n",
    "    (n_x, m) = X_train.shape                          # (n_x: input size, m : number of examples in the train set)\n",
    "    n_y = Y_train.shape[0]                            # n_y : output size\n",
    "    costs = []                                        # To keep track of the cost\n",
    "    \n",
    "    # Create Placeholders of shape (n_x, n_y)\n",
    "    X, Y = create_placeholders(n_x, n_y)\n",
    "\n",
    "    # Initialize parameters\n",
    "    parameters = initialize_parameters()\n",
    "    \n",
    "    # Forward propagation: Build the forward propagation in the tensorflow graph\n",
    "    Z3 = forward_propagation(X, parameters)\n",
    "    \n",
    "    # Cost function: Add cost function to tensorflow graph\n",
    "    cost = compute_cost(Z3, Y)\n",
    "\n",
    "    \n",
    "    # Backpropagation: Define the tensorflow optimizer. Use an AdamOptimizer.\n",
    "    optimizer = tf.train.AdamOptimizer(learning_rate = learning_rate).minimize(cost)\n",
    "    \n",
    "    # Initialize all the variables\n",
    "    init = tf.global_variables_initializer()\n",
    "\n",
    "    # Start the session to compute the tensorflow graph\n",
    "    with tf.Session() as sess:\n",
    "        \n",
    "        # Run the initialization\n",
    "        sess.run(init)\n",
    "        \n",
    "        # Do the training loop\n",
    "        for epoch in range(num_epochs):\n",
    "\n",
    "            epoch_cost = 0.                       # Defines a cost related to an epoch\n",
    "            num_minibatches = int(m / minibatch_size) # number of minibatches of size minibatch_size in the train set\n",
    "            seed = seed + 1\n",
    "            minibatches = random_mini_batches(X_train, Y_train, minibatch_size, seed)\n",
    "\n",
    "            for minibatch in minibatches:\n",
    "\n",
    "                # Select a minibatch\n",
    "                (minibatch_X, minibatch_Y) = minibatch\n",
    "                \n",
    "                # IMPORTANT: The line that runs the graph on a minibatch.\n",
    "                # Run the session to execute the \"optimizer\" and the \"cost\"\n",
    "                _ , minibatch_cost = sess.run([optimizer, cost], feed_dict={X: minibatch_X, Y: minibatch_Y})\n",
    "\n",
    "                \n",
    "                epoch_cost += minibatch_cost / num_minibatches\n",
    "\n",
    "            # Print the cost every epoch\n",
    "            if print_cost == True and epoch % 100 == 0:\n",
    "                print (\"Cost after epoch %i: %f\" % (epoch, epoch_cost))\n",
    "            if print_cost == True and epoch % 5 == 0:\n",
    "                costs.append(epoch_cost)\n",
    "                \n",
    "        # plot the cost\n",
    "        plt.plot(np.squeeze(costs))\n",
    "        plt.ylabel('cost')\n",
    "        plt.xlabel('iterations (per tens)')\n",
    "        plt.title(\"Learning rate =\" + str(learning_rate))\n",
    "        plt.show()\n",
    "\n",
    "        # lets save the parameters in a variable\n",
    "        parameters = sess.run(parameters)\n",
    "        print (\"Parameters have been trained!\")\n",
    "\n",
    "        # Calculate the correct predictions\n",
    "        correct_prediction = tf.equal(tf.argmax(Z3), tf.argmax(Y))\n",
    "\n",
    "        # Calculate accuracy on the test set\n",
    "        accuracy = tf.reduce_mean(tf.cast(correct_prediction, \"float\"))\n",
    "\n",
    "        print (\"Train Accuracy:\", accuracy.eval({X: X_train, Y: Y_train}))\n",
    "        print (\"Test Accuracy:\", accuracy.eval({X: X_test, Y: Y_test}))\n",
    "        \n",
    "        return parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cost after epoch 0: 1.855702\n",
      "Cost after epoch 100: 1.016458\n",
      "Cost after epoch 200: 0.733102\n",
      "Cost after epoch 300: 0.572939\n",
      "Cost after epoch 400: 0.468774\n",
      "Cost after epoch 500: 0.381015\n",
      "Cost after epoch 600: 0.313829\n",
      "Cost after epoch 700: 0.254169\n",
      "Cost after epoch 800: 0.203800\n",
      "Cost after epoch 900: 0.166474\n",
      "Cost after epoch 1000: 0.141408\n",
      "Cost after epoch 1100: 0.107606\n",
      "Cost after epoch 1200: 0.086730\n",
      "Cost after epoch 1300: 0.061016\n",
      "Cost after epoch 1400: 0.050911\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xl8VdXV8PHfyjzPIQQSEmYQBcTI4IA4o1Wpta1SrbbaUq12sE8HbX2r1ce39unjW1uHWgecRa1T0TpXBQdAAjJPQiAkjCETgczJev84J3iJCblAbs7Nzfp+PveTe/fZ59y1c+Gu7LPP2VtUFWOMMaYrYV4HYIwxpnewhGGMMcYvljCMMcb4xRKGMcYYv1jCMMYY4xdLGMYYY/xiCcOEPBF5U0Su8joOY3o7SxgmYERki4ic5XUcqnqeqj7hdRwAIvKhiPygB94nWkRmi8heEdkpIr/oov6Nbr1qd79on235IvKBiNSKyLr2n2kX+94hIitFpFlEbuv2hpoeZQnD9GoiEuF1DG2CKRbgNmA4kAecDvxaRKZ3VFFEzgVuAs4E8oEhwB98qswBPgfSgd8BL4pIpp/7bgR+Dfy7W1plPGUJw3hCRC4QkWUiUiUin4rIWJ9tN4nIJhGpEZE1InKxz7bvicgnIvIXEakAbnPLPhaR/xWRShHZLCLn+exz4K96P+oOFpH57nu/JyL3i8jTnbRhmoiUishvRGQn8JiIpIrI6yJS5h7/dRHJcevfCZwK3Cci+0TkPrd8lIi8KyIVIrJeRL7dDb/iK4E7VLVSVdcCDwPf66TuVcCjqrpaVSuBO9rqisgIYAJwq6rWqepLwErgkq72BVDVJ1T1TaCmG9pkPGYJw/Q4EZkAzAZ+hPNX6z+AuT6nMjbhfLEm4/y1+rSIZPscYhJQBPQD7vQpWw9kAP8DPCoi0kkIh6r7LPCZG9dtwHe7aE5/IA3nL/lZOP+nHnNfDwLqgPsAVPV3wEfADaqaoKo3iEg88K77vv2AmcADIjKmozcTkQfcJNvRY4VbJxUYACz32XU50OEx3fL2dbNEJN3dVqSqNe22j/FjXxNiLGEYL/wQ+IeqLlLVFnd8oQGYDKCq/1TV7araqqrPA18AE332366q96pqs6rWuWXFqvqwqrYATwDZQFYn799hXREZBJwI/F5VG1X1Y2BuF21pxfnru8H9C7xcVV9S1Vr3S/ZO4LRD7H8BsEVVH3PbsxR4CfhmR5VV9ceqmtLJo62XluD+rPbZtRpI7CSGhA7q4tZvv639sQ61rwkxljCMF/KA//L96xjIxfmrGBG50ud0VRVwLE5voE1JB8fc2fZEVWvdpwkd1DtU3QFAhU9ZZ+/lq0xV69teiEiciPxDRIpFZC8wH0gRkfBO9s8DJrX7XVyO03M5Uvvcn0k+ZUl0flpoXwd1ceu339b+WIfa14QYSxjGCyXAne3+Oo5T1Tkikodzvv0GIF1VU4BVgO/ppUBNsbwDSBOROJ+y3C72aR/LfwEjgUmqmgRMdculk/olwLx2v4sEVb2uozcTkQfd8Y+OHqsB3LGEHcA4n13HAas7acPqDuruUtVyd9sQEUlst321H/uaEGMJwwRapIjE+DwicBLCtSIySRzxIvI190spHudLtQxARL6P08MIOFUtBgpxBtKjRGQKcOFhHiYRZ9yiSkTSgFvbbd+FcyVRm9eBESLyXRGJdB8nisjoTmK81k0oHT18xyieBG5xB+FH4ZwGfLyTmJ8ErhGRY9zxj1va6qrqBmAZcKv7+V0MjMU5bXbIfQHc9sTgfNdEuMforLdlgpwlDBNob+B8gbY9blPVQpwvsPuASpxLL78HoKprgLuBBThfrscBn/RgvJcDU4By4L+B53HGV/x1DxAL7AEWAm+12/5X4JvuFVR/c8c5zgEuA7bjnC77ExDN0bkV5+KBYmAe8GdVfQtARAa5PZJBAG75/wAfuPWLOTjRXQYU4HxWdwHfVNUyP/d9GOdzn4lzSW4dXV9IYIKU2AJKxnRORJ4H1qlq+56CMX2O9TCM8eGeDhoqImHi3Og2A3jV67iMCQbBdGeqMcGgP/Ayzn0YpcB1qvq5tyEZExzslJQxxhi/2CkpY4wxfgmpU1IZGRman5/vdRjGGNNrLFmyZI+qZvpTN6QSRn5+PoWFhV6HYYwxvYaIFPtb105JGWOM8YslDGOMMX6xhGGMMcYvljCMMcb4xRKGMcYYv1jCMMYY4xdLGMYYY/zS5xNGfVMLD88v4tONe7wOxRhjglqfTxgRYcJDHxUx+5PNXodijDFBzRJGeBiXTMjhg/Vl7K6p73oHY4zpo/p8wgD4VkEOLa3KK0u3eR2KMcYELUsYwNDMBMbnpjB3+XavQzHGmKBlCcN1wdhsVm/fy+Y9+70OxRhjgpIlDNf5x2UD8O8V1sswxpiOWMJwDUiJ5biBycz/wi6vNcaYjljC8HHS0HSWba2irrHF61CMMSboWMLwMWVoOo0trSwprvQ6FGOMCToBSxgiMltEdovIqk62/0pElrmPVSLSIiJp7rYtIrLS3dZjS+idmJ9GRJjwySY7LWWMMe0FsofxODC9s42q+mdVHa+q44GbgXmqWuFT5XR3e0EAYzxIfHQEE/JSmbe+rKfe0hhjeo2AJQxVnQ9UdFnRMROYE6hYDscZo/qxZsdedlbbXd/GGOPL8zEMEYnD6Ym85FOswDsiskREZnWx/ywRKRSRwrKyo+8ZnD6yHwAfrN991McyxphQ4nnCAC4EPml3OupkVZ0AnAdcLyJTO9tZVR9S1QJVLcjMzDzqYEZkJTAgOYb5G+y0lDHG+AqGhHEZ7U5Hqep29+du4BVgYk8FIyJMHpLOZ5srUNWeeltjjAl6niYMEUkGTgP+5VMWLyKJbc+Bc4AOr7QKlElD0ijf38imMpsmxBhj2kQE6sAiMgeYBmSISClwKxAJoKoPutUuBt5RVd9v5izgFRFpi+9ZVX0rUHF2ZOLgdAAWbS5nWL+EnnxrY4wJWgFLGKo60486j+NcfutbVgSMC0xU/slPjyMzMZrPNldw+aQ8L0MxxpigEQxjGEFHRJg0OI1FRTaOYYwxbSxhdGLS4DR27q2npKLO61CMMSYoWMLoxKQhzjjGws3lHkdijDHBwRJGJ4ZlJpAaF8lnm/29Wd0YY0KbJYxOhIUJJ+anWcIwxhiXJYxDmDQkna0VteyotnEMY4yxhHEIkwanAVgvwxhjsIRxSKOzk0iMjmCRJQxjjLGEcSjhYcLY3GRWbav2OhRjjPGcJYwujOqfxPqdNTS3tHodijHGeMoSRhdGZyfR0NzKlvJar0MxxhhPWcLowujsRADW7tjrcSTGGOMtSxhdGNYvgYgwYd1OSxjGmL7NEkYXoiPCGdYvgRWlNvBtjOnbLGH4YfKQdBZvqaC+qcXrUIwxxjOWMPxwyrAM6ptaWVpc6XUoxhjjGUsYfpg8NJ3wMOHjjXu8DsUYYzxjCcMPCdERjM9NYUGRTXVujOm7ApYwRGS2iOwWkVWdbJ8mItUissx9/N5n23QRWS8iG0XkpkDFeDgK8lNZta3axjGMMX1WIHsYjwPTu6jzkaqOdx+3A4hIOHA/cB5wDDBTRI4JYJx+OTEvjaYWZXlJldehGGOMJwKWMFR1PnAks/ZNBDaqapGqNgLPATO6NbgjcEJeKgCFNvBtjOmjvB7DmCIiy0XkTREZ45YNBEp86pS6ZR0SkVkiUigihWVlZQELNDU+imH9EmzmWmNMn+VlwlgK5KnqOOBe4FW3XDqoq50dRFUfUtUCVS3IzMwMQJhfmjYik4WbytnX0BzQ9zHGmGDkWcJQ1b2qus99/gYQKSIZOD2KXJ+qOcB2D0L8inPG9KexpZUP1+/2OhRjjOlxniUMEekvIuI+n+jGUg4sBoaLyGARiQIuA+Z6FaevE/JSSY+P4p3Vu7wOxRhjelxEoA4sInOAaUCGiJQCtwKRAKr6IPBN4DoRaQbqgMtUVYFmEbkBeBsIB2ar6upAxXk4wsOEk4ZlULjFxjGMMX1PwBKGqs7sYvt9wH2dbHsDeCMQcR2t8bkpvLZ8O7v31tMvKcbrcIwxpsd4fZVUrzM+NxmAZXY/hjGmj7GEcZjGDEgmIkxYXmoJwxjTt1jCOEwxkeGMyk60HoYxps+xhHEExuemsKKkmtbWTm8PMcaYkGMJ4wiMy0mhpqGZoj37vA7FGGN6jCWMIzA+NwWAZSW2bKsxpu+whHEEhmYmkBAdwbISm4jQGNN3WMI4AmFhwrEDk1i1ba/XoRhjTI+xhHGEBmfEU1pZ63UYxhjTYyxhHKGc1Dj27GukrtFW4DPG9A2WMI5QTmosANuqrJdhjOkbLGEcobaEUVJZ53EkxhjTMyxhHKGc1DgASi1hGGP6CEsYRygzIZqo8DAb+DbG9BmWMI5QWJgwMDXWehjGmD7DEsZRyEmNpaTCehjGmL7BEsZRyEuPY/Oe/TgLBRpjTGizhHEUhmQkUFPfzJ59jV6HYowxARewhCEis0Vkt4is6mT75SKywn18KiLjfLZtEZGVIrJMRAoDFePRGpIZD0BRmc1aa4wJfYHsYTwOTD/E9s3Aaao6FrgDeKjd9tNVdbyqFgQovqM2NDMBgKI9+z2OxBhjAi8iUAdW1fkikn+I7Z/6vFwI5AQqlkAZkBJLVESY9TCMMX1CsIxhXAO86fNagXdEZImIzDrUjiIyS0QKRaSwrKwsoEG2Fx4mDE6Pp6jMehjGmNAXsB6Gv0TkdJyEcYpP8cmqul1E+gHvisg6VZ3f0f6q+hDu6ayCgoIev1xpSGY863bW9PTbGmNMj/O0hyEiY4FHgBmqWt5Wrqrb3Z+7gVeAid5E2LXBGfFsrailqaXV61CMMSagPEsYIjIIeBn4rqpu8CmPF5HEtufAOUCHV1oFgyGZCbS0KlvtBj5jTIgL2CkpEZkDTAMyRKQUuBWIBFDVB4HfA+nAAyIC0OxeEZUFvOKWRQDPqupbgYrzaH15ae3+A1dNGWNMKArkVVIzu9j+A+AHHZQXAeO+ukdwGprhXlpbtg8n1xljTGgKlqukeq3kuEjS46PsSiljTMizhNENhmTGU7TH7sUwxoQ2SxjdYERWIut21NDaapMQGmNClyWMbjA2J5mahma2lNtpKWNM6LKE0Q3G5qQAsHJbtceRGGNM4FjC6AbD+yUQExnG8hJLGMaY0GUJoxtEhIcxZkAyK0qrvA7FGGMCxhJGNzk+N4UV26qpb2rxOhRjjAkISxjdZNKQdBqbW1leYr0MY0xosoTRTSbmpyECizZXeB2KMcYEhCWMbpIcF8mo/kks2lzedWVjjOmFLGF0o0mD01hSXEljs011bowJPZYwutHkIWnUN7Xa1VLGmJBkCaMbTRycDtg4hjEmNFnC6EZp8VGMzEpkYZGNYxhjQo8ljG42aYgzjmFLthpjQo0ljG42aXA6tY0tNq+UMSbkWMLoZhMHpwGwqMjGMYwxoSWgCUNEZovIbhFZ1cl2EZG/ichGEVkhIhN8tl0lIl+4j6sCGWd3ykyMZmhmvN2PYYwJOX4lDBH5lj9lHXgcmH6I7ecBw93HLODv7rHTgFuBScBE4FYRSfUn1mBw8rAMFhaVU13X5HUoxhjTbfztYdzsZ9lBVHU+cKhzMzOAJ9WxEEgRkWzgXOBdVa1Q1UrgXQ6deILKtwtyqW9q5eWlpV6HYowx3SbiUBtF5DzgfGCgiPzNZ1MS0NwN7z8QKPF5XeqWdVbeUYyzcHonDBo0qBtCOnrHDkxmfG4KTy0o5qop+YSFidchGWPMUeuqh7EdKATqgSU+j7k4vYCj1dE3qR6i/KuFqg+paoGqFmRmZnZDSN3j6lMGU7RnP++s2eV1KMYY0y0OmTBUdbmqPgEMU9Un3OdzgY3uqaKjVQrk+rzOwUlSnZX3Gucf259BaXE8OG+T16EYY0y38HcM410RSXIHo5cDj4nI/+uG958LXOleLTUZqFbVHcDbwDkikuoOdp/jlvUaEeFhXD5pEMtKqtheVed1OMYYc9T8TRjJqroX+AbwmKqeAJzV1U4iMgdYAIwUkVIRuUZErhWRa90qbwBFwEbgYeDHAKpaAdwBLHYft7tlvcppI51TZLfNXc1P53yOaodn1Ywxplc45KC3bz336qVvA7/z9+CqOrOL7Qpc38m22cBsf98rGI3MSqRfYvSBcYybzx9FdnKsx1EZY8yR8beHcTvOKaFNqrpYRIYAXwQurNAgIkwd8eVA/PISmy7EGNN7+ZUwVPWfqjpWVa9zXxep6iWBDS00/PKckcz+XgERYcLKbbZOhjGm9/L3Tu8cEXnFneZjl4i8JCI5gQ4uFPRPjuGMUVmM7J/IilLrYRhjei9/T0k9hnNF0wCcG+hec8uMn8bmJLNsaxU7q+u9DsUYY46IvwkjU1UfU9Vm9/E4EDx3yfUCl0/KQ4HvPLyQmnqbY8oY0/v4mzD2iMgVIhLuPq4AbDrWw3DswGQevaqALeX7uW3uGq/DMcaYw+Zvwrga55LancAO4JvA9wMVVKiaNCSdWVOH8tLSUrbs2e91OMYYc1j8TRh3AFepaqaq9sNJILcFLKoQ9t0peQC8vqJXzXRijDF+J4yxvnNHuXddHx+YkELbwJRYTsxPZe7y7XbntzGmV/E3YYT5LmDkzinl713ipp1vTMhhw659/O0/G/nDa6tpamn1OiRjjOmSv1/6dwOfisiLONOMfxu4M2BRhbhLJuTw4LxN/OW9DQCcNDSDs4/J8jgqY4w5NH/v9H4SuATYBZQB31DVpwIZWCiLigjjzq8fx6nDM0iPj7KV+YwxvYKE0nn0goICLSws9DqMw3L7a2t47NPNDMtM4MXrTiI5NtLrkIwxfYiILFHVAn/q+juGYQLkR6cN4YpJeXyxex9vr9rpdTjGGNMpSxgey0qK4fYZY8hLj+M1u9TWGBPELGEEARHhwrED+GTjHnbvtbmmjDHByRJGkLjkhBxaFZ5fXOJ1KMYY0yFLGEFicEY8pw7P4MmFxTw4bxN1jS28s3qn3dxnjAkaAU0YIjJdRNaLyEYRuamD7X8RkWXuY4OIVPlsa/HZNjeQcQaLWVOHUFXbyF1vruNb//iUWU8tYf4Xe7wOyxhjgAAmDBEJB+4HzgOOAWaKyDG+dVT1RlUdr6rjgXuBl30217VtU9WLAhVnMDl1eCYb/vs8xuUks2rbXgDmrS/zOCpjjHEEsocxEdjoLufaCDwHzDhE/ZnAnADG0yuICL88dySJ0REMyYhn/heWMIwxwSGQCWMg4DuCW+qWfYWI5AGDgfd9imNEpFBEForI1zt7ExGZ5dYrLCsLjS/XU4dnsvIP5zJz4iA27t7Hf72w3FbqM8Z4LpAJQzoo62wE9zLgRVVt8Skb5N59+B3gHhEZ2tGOqvqQqhaoakFmZmgtAnjBuGymDEnnjZU7+M7DC9leVed1SMaYPiyQCaMUyPV5nQN0dmfaZbQ7HaWq292fRcCH9MHp1LOTY5kzazJPXTORspoGZtz/CctKqrre0RhjAiCQCWMxMFxEBotIFE5S+MrVTiIyEkgFFviUpYpItPs8AzgZ6LPrmhbkp/Hyj08iJjKMS/+xgM+3Vna9kzHGdLOAJQxVbQZuAN4G1gIvqOpqEbldRHyvepoJPKcH33AwGigUkeXAB8BdqtpnEwbA8KxEXv3xyaTGRXHzyytptjU0jDE9zGar7WXeWrWDa59eyon5qcyaOpSzRvdDpKPhImOM6ZrNVhvCph+bzd3fGsfG3fv44ZOFPPHpFq9DMsb0EZYweqFLTsjhs9+dxcT8NP4+bxMNzS1d72SMMUfJEkYvFRkexk/OHMauvQ18/7HFXPf0Etbu2Ot1WMaYEGYJoxc7ZVgGt88Yw6pt1byzZhd3v7Pe65CMMSHMEkYvJiJcOSWf5beew/WnD+O9tbvZuHuf12EZY0KUJYwQ4CSOPOKjwrn55RVU7m+ktVV5csEWSipqvQ7PGBMiIrwOwHSPjIRo7rz4OH7+/DKOv+NdBmfEs3nPfs4YVcbWilqumpLHd6fkex2mMaYXs4QRQr5+/EDS4qMoLK7k0Y+KyE6O4f11uwH417LtljCMMUfFEkaImToik6kjMvnpGcNYv6uGr/3tYwA+L6lib30TidERdqOfMeaI2BhGiIoID2PMgGR+de5IbvnaaFpalbG3vcNf3vvC69CMMb2UJYwQd/3pw7hySj5JMU5n8uH5RVTsb/Q4KmNMb2QJow+Iigjjg19O498/PYX65hbOuPtDnlywhU837qGo7MvLcJtbWmltDZ25xYwx3cvGMPqI9IRo0hOiuefS8Tz+6RbueH0NLa1Kbloc79w4lXARvvngAgalxfG3mX1u6RFjjB8sYfQxM8YP5KShGZx594ckREdQXF7Lb15cQWp8FMtKqlizfS819U0kxkR6HaoxJshYwuiDMhOjeefG00iKjeB/397AUwu30NSijOqfyLqdNfxn7W6+fnyHy68bY/owWw/DUF3bxN76JrKTYzjlTx8won8iT1490euwjDE9wNbDMIclOS6S3LQ4IsLDuPKkPOZvKOPTTXsObF9SXMmSYlsW1pi+zk5JmYNcffJgnlm4lasfX8yZo7Oorm3i4417iI0M5z//dRoDUmK9DtEY45GA9jBEZLqIrBeRjSJyUwfbvyciZSKyzH38wGfbVSLyhfu4KpBxmi/FRIbz9A8mcfHxOSwtrmRrRS03nD4MRbni0UU8s6jY6xCNMR4JWA9DRMKB+4GzgVJgsYjMVdU17ao+r6o3tNs3DbgVKAAUWOLua+dFesDgjHj++I3jDiob2T+Re97bwP95dRVTh2eSmxbnUXTGGK8EsocxEdioqkWq2gg8B8zwc99zgXdVtcJNEu8C0wMUp/HDheMG8MwPJhMmwgMfbiKULpYwxvgnkAljIFDi87rULWvvEhFZISIvikjuYe6LiMwSkUIRKSwrK+uOuE0n+ifH8K2CXOZ8tpWZDy9kZWk1CzaVex2WMaaHBDJhdDQlavs/S18D8lV1LPAe8MRh7OsUqj6kqgWqWpCZmXnEwRr/3DFjDP/99WNZvKWSC+/7mO88spA3Vu7wOixjTA8I5FVSpUCuz+scYLtvBVX1/fP0YeBPPvtOa7fvh90eoTlsEeFhXDE5j4yEKBZvqWRZSRU/nfM55fsbuWjsAOqbW8hKivE6TGNMAATsxj0RiQA2AGcC24DFwHdUdbVPnWxV3eE+vxj4japOdge9lwAT3KpLgRNUteJQ72k37vW8vfVNXP/MUj76wrlvI0zgGxNyGJeTzBWT82ztDWOC3OHcuBewHoaqNovIDcDbQDgwW1VXi8jtQKGqzgV+KiIXAc1ABfA9d98KEbkDJ8kA3N5VsjDeSIqJ5InvT+SdNTtZu6OGHdV1zF2+nReXlJIcF8VF4wZ4HaIxppvY1CCm27W0KjPu/5hdexuY88NJDOuX6HVIxphO2NQgxlPhYcKfvzkOVbjw3k+49qklzHxoIQuLynngw400NLd4HaIx5gjY1CAmIEZnJ/HaT07mj2+sY0FROfsbmrnsoYUA7Kiq5/YZY2x8w5hexhKGCZjs5NgDizEtLCpn9sebSYyJ5KmFxSwpriQnNZbs5Bh+cc5IkmNt/Q1jgp0lDNMjJg9JZ/KQdFpaleMHpfDq59vYWlHLe2t3UVPfzK+mjyQ7OZbaxmb+9p+NzJo6hLT4KK/DNsb4sIRhelR4mHDF5DyumJwHwJ/fXsf9H2zi5c+38ZdLx7GvoYUH520iKTaCH08b5nG0xhhfljCMp248awTHDUzhofmb+P2rq0lPcHoVb63aaQnDmCBjV0kZT0WEhzH92P789bLjSYmPZEt5LcP6JbCitJrHP9nM3vomr0M0xrjsPgwTNGobm5m3vowxA5K54N6P2FvfTFZSNH+46FjOHZNlV1UZEwCHcx+GJQwTlFpblc9Lqrjl1VWs3bGXjIQoBqbEMiEvle9MHMTwLLsZ0JjuYAnDhIymllZe/XwbC4sq2F5Vx5KtlTS1tHJiXho/O2s4Ly0p5YJx2ZwxKsvrUI3plSxhmJBVsb+RpxcW89xnW9leXQ9AQnQEz/9oMmMGJHscnTG9jyUME/J2Vtdz4/PLOGV4Bo9+vJnK2kZOG5HJ2h17yU2Nc6ZbHzeAG88e4XWoxgQ1SximTynf18ATn27hmUVbyc+Ip7i8ltrGZlTh05vOILXdDYBt/+ZtEN0YSxheh2GCwPqdNZx7z3xOG5HJ6SMz2VJey6C0OC6fPIjrnl5KdEQYD1w+wZKG6fOCYj0MY7w0sn8iN541gkc/LmLehjKiIsJobG7lgQ83sWdfAwBPLijmqpPyvQ3UmF7EEoYJWT87azjXnz6U6romEmMimb+hjNtfX8NZo/vR0qr84bXVVOxv5MJxAxiaGc8DH25if0Mzv54+yuvQjQlKdkrK9CmqiirUNbXwwycL+XRTOcmxkZw1OouXlpYC8OTVE6mpbyY/I86uvDIhz8YwjPGDqrJuZw0XP/AJ9U2tfO+kfF4oLKG20VngKTYynHm/nka/xBiPIzUmcIJmDENEpgN/xVnT+xFVvavd9l8AP8BZ07sMuFpVi91tLcBKt+pWVb0okLGavkdEGJ2dxN8vP4GyfQ18uyCXASkxfLKxnCsm53Hd00u46aWVnHNMFifkpdrd5abPC1gPQ0TCgQ3A2UApsBiYqaprfOqcDixS1VoRuQ6YpqqXutv2qWrC4byn9TBMd2qbeh2cadlvOH0YA1JiWLmtmkFpcVwyIYf0hOgD9eubWoiJDPcqXGOOSFCckhKRKcBtqnqu+/pmAFX9Yyf1jwfuU9WT3deWMIzn9jc0U76vkbveWssbK3cCkBgdQU1DM9ERYVx6Yi63fO0YHpq/ib9/uInXfnIKQzIP65+tMZ4KllNSA4ESn9elwKRD1L8GeNPndYyIFOKcrrpLVV/t/hCNObT46AjioyN44PIT2FpeS1VdI8cNTGZT2X4e+aiIJxcUs2pbNctKqmhV+O0rK8lIiOba04Zy7MBkSipq+dNb67h9xrGkxEYSFmb3fZjeK5AJo6P/GR12Z0TkCqAAOM2neJCqbheRIcD7IrJSVTe+2rKeAAATHUlEQVR1sO8sYBbAoEGDjj5qYzoxKD2OQcQBMKxfAnddMpbctDjuff8Lzh3Tn4ToCP65pBQReHPVTu65dDxvrNzBm6t2Ul3XxJLiSi6fNIhfTx9FZLgtRWN6H89PSYnIWcC9wGmquruTYz0OvK6qLx7qPe2UlPFSdV0Tb63awekj+3HDs5/z2ZYKAOKjwtnf2EJSTAR765v52ZnD+f7J+cz+ZAtXTB5EVHgYKXG2frnxRrCMYUTgDHqfCWzDGfT+jqqu9qlzPPAiMF1Vv/ApTwVqVbVBRDKABcAM3wHzjljCMMGitrGZFxaXsGdfI6cOz+CXLy7nnkvH89SCYl5fsYP+yTGUVtYxqn8iG3bV8IuzR1Bd18SlJw5iWD8bAzE9JygShhvI+cA9OJfVzlbVO0XkdqBQVeeKyHvAccAOd5etqnqRiJwE/ANoxVlG9h5VfbSr97OEYYKVqiIiVNU28ofX1lBcvp/slFj+vWIHYQKt7n/D0dlJfOuEHNLio8hNi+Xlpdv4xoQcTshL5ZlFxZyQl8qo/kneNsaElKBJGD3NEobpTRqaW3h20VbG5qRw4/PLODE/7cDd5r6ykqL5/QVjuP7ZpRw/KIWfnjmc/PR4BmfEexC1CTWWMIzppRYWlTMwJZZ9Dc18sXsfcZHhXPv0EppblYgwodntiiRGR3D6qH70T47h2wW5vLlyB4kxEVx1Uv6BGXgLt1QQHiYcPyjVyyaZIGcJw5gQsqykiqcWFHPB2GzufGMto7OT2LW3nu1Vdeze24AINDS3AnD2MVn84aIxLC+p4idzPic1PopPfnMGURF2VZbpmCUMY0JU21hIm5KKWq6c/RkjsxI5IS+V/31nPc2tSkurMiA5hu3V9cycmMuQjATGDExiQHIs+XYqq9d65KMidlbXc8sFx3TbMYPlxj1jTDdrv+BTbloc7/3iNMLE2Tb92P7M+WwrKXGRXDkln/P++hFzPis5aJ8ByTEMz0pkWL8EPtm4h/OOzWbayEzG5aYcqPP+ul08v7iEv152vE13EkQ+WL+bkoq6bk0Yh8MShjG9XLjP3eO5aXEHrefxyFUFVO5vJCsphpKKWhZvqWT9rr28tWon8zaUkZsWy1/e28C973/BS9edxMj+iVTWNnLTSyvZXdPAc59t5XsnD/aiWaYDVbVNVNU2evb+ljCMCWFDMxMg03memxbHScMyAPjXsm2s21nDr84ZSfn+Rmbc9zEz7v/koH0HZ8Rz7/sb2d/YwoZdNVw5JY8P1pUxKjuR84/NPjDNSVlNAy8UlvD9k/OJi7KvlECqqm1ib30zLa160B8KPcU+XWP6oBnjBzLDfZ6ZGM0TV0/klc+3OXNnRYUzPCuR9IQornm8kD+/vR6Afy3bfmD/aSNLGZ+bwpzPthIbGc6W8lpiI8O5+pSDeyMtrcp7a3cxdXgmsVF2autoVdc1AVBT3+TJ7ACWMIwxDM9K7HBp2rd+firrdtbQ0NTKnMVbuWn6KN5ft5u73lzHh+vLGNYvga3ltWQnx/D0wmKmjshk9fZqistrmTF+AHM+K+HBeZv43kn53HbRGADqGlu48fllXHPqYE7MT+vppvZajc2t7GtoBpyehhcJw66SMsYctr31TRSV7WdcTjJNLcq/V27nxueXd1g3PT6K6romJg1J4+zRWZTvb+Te9zdyYn4q/7z2pAP19uxrICU2kgibmLFDZTUNnHjnewC8ev3JjPe5SOFo2FVSxpiASoqJPPCFFRUhzBg3kLioCPY3NDOyfyLJsZG8UFjKkIx4pgxN57qnl7B7bwO3veZMB5ccG8niLZVc/+xSmppbSYuP4uWl2xg/KIW7vnEc+enxVNc18e7aXYzNSbbpUIDqukaf502exGAJwxhz1MLChHPH9D+o7Bdnjzjw/OUfn4yqsnRrJZ9sLOdrY7P5ybOfs2xrFbFR4by/bjfHD0phaXElZ9w970CvpLlVSY+P4t6ZxzNmQDJ1TS18sbuG3NS4A/eTNDS3EBEWRniY0NTSSnOLhuR4SWXtl0nCqyulLGEYY3qEiHBCXhon5DnjFm/87NQD29qu+tm4u4YlxZUs2FROv6QYJuanceMLy/jOI4sOmqRRBHJT42hqaaWspoGspBiunJLHc4tLqK5r4qzR/UiNi+I300eFzKJVVT4Jw3oYxpg+q+0S0WH9EhnWL5FLT/xyMbT3fnEaa7bvZXlpFbGR4YzNSWFhUTnF5fuJDA8jPSGad9bs5I9vrmNgSiw5qbH8a9l2GppbeXftLhJjIrl26hD6JUUTHRFOZHgYJRW1nDYyE4FeM2bi26uorrWEYYwxX5GVFENWUgynj+p3oGzK0PSD6vzq3JFU1jYeGDRvbVXuemsd89aXsaemgeueWfqV40aGC9ER4Vw+aRClVXUUle1ndP9ESiprmTV1KBHhwo6qes5wJ3n8z9pd3Dp3NfdcOp6C/DTqGlv8OvXV0qrc8uoqvj5+AJOGpHdZvzNtvYowgSrrYRhjzJEJDxMyEqIPvA4LE357/mh+e/5o6ptaWLWtmn0NzVTWNlK5v4mspBiWbq1kw64a/jG/iIyEaHJSY/n3yh2kxkXxwycLDzr2CXmpLC+poqG5lZ8/v4wJg1J5bcV2ZowbQEVtE9eeNoSThmZQ39TCxt37GJ6VQHSEk0zeXr2TOZ9tZdW2aubecPJXpndpPz/Ytqo6YiKcnpOvytpGwsOEfonRB52e6kmWMIwxIS0mMpyCDu73+NrYbADqm1oOmi9rf0Mzn2zcQ0pcFClxkbzy+TbmbyjjvGP7M/3Y/vzmpZW8u8a5GfHVZduJjghj4aZyjhmQxLqde6lvauWY7CRy02JpVVi9rZrIcGHltmqeX1xCbFQ4m8r2M6xfAhPz0/juo4toaVV+euZwTh/Zjwvv/ZiMhCje/NnUg+7mrqptIiU2kpS4KM/GMOw+DGOMOUK79tYTHRHG3e9sYMOuGkZnJ5GfHseD84qIiwonKiKMhuZWbjx7BPe8u4GiPfsP2l8EIsPCGJ6VwOrtexmUFsfWiloALhw3AAGaW1upa2zhg/Vl5KXHkZ0cQ2NzKy9dd9JXeitHwqY3N8aYINPY3MqizeWkxkUxIiuRwuIKXlu+g9NGZHDW6CzufX8j763dxVmjs1hQVM7S4koGpMQSFRFGlDswP3VEJtuq6nht+XbiosKJjgijvqmVYwcm8cKPphxRArGEYYwxvZiq0qp0OMHgzup63l27i6KyfdQ3tbpJo4W7Lhl7RO8VNHd6i8h04K9AOPCIqt7Vbns08CRwAlAOXKqqW9xtNwPXAC3AT1X17UDGaowxwUJECO+ks9A/OYbvTs7r2YBcAbsAWUTCgfuB84BjgJki0n7Vj2uASlUdBvwF+JO77zHAZcAYYDrwgHs8Y4wxHgnkHSsTgY2qWqSqjcBzcGBG5TYzgCfc5y8CZ4pzEm4G8JyqNqjqZmCjezxjjDEeCWTCGAj4rg1Z6pZ1WEdVm4FqIN3PfQEQkVkiUigihWVlZd0UujHGmPYCmTA6OgPXfoS9szr+7OsUqj6kqgWqWpCZmXmYIRpjjPFXIBNGKZDr8zoH2N5ZHRGJAJKBCj/3NcYY04MCmTAWA8NFZLCIROEMYs9tV2cucJX7/JvA++pc5zsXuExEokVkMDAc+CyAsRpjjOlCwC6rVdVmEbkBeBvnstrZqrpaRG4HClV1LvAo8JSIbMTpWVzm7rtaRF4A1gDNwPWq2hKoWI0xxnTNbtwzxpg+rM/e6S0iZUDxEe6eAezpxnC8ZG0JPqHSDrC2BKsjbUueqvp1xVBIJYyjISKF/mbZYGdtCT6h0g6wtgSrnmhL71hqyhhjjOcsYRhjjPGLJYwvPeR1AN3I2hJ8QqUdYG0JVgFvi41hGGOM8Yv1MIwxxvjFEoYxxhi/9PmEISLTRWS9iGwUkZu8judwicgWEVkpIstEpNAtSxORd0XkC/dnqtdxdkREZovIbhFZ5VPWYezi+Jv7Oa0QkQneRf5VnbTlNhHZ5n42y0TkfJ9tN7ttWS8i53oTdcdEJFdEPhCRtSKyWkR+5pb3us/mEG3pdZ+NiMSIyGcistxtyx/c8sEissj9XJ53p2LCnVrpebcti0Qk/6iDUNU++8CZsmQTMASIApYDx3gd12G2YQuQ0a7sf4Cb3Oc3AX/yOs5OYp8KTABWdRU7cD7wJs5MxpOBRV7H70dbbgN+2UHdY9x/a9HAYPffYLjXbfCJLxuY4D5PBDa4Mfe6z+YQbel1n437+01wn0cCi9zf9wvAZW75g8B17vMfAw+6zy8Dnj/aGPp6D8OfRZ56I9+FqZ4Avu5hLJ1S1fk4c4j56iz2GcCT6lgIpIhIds9E2rVO2tKZoF4gTFV3qOpS93kNsBZnPZpe99kcoi2dCdrPxv397nNfRroPBc7AWYAOvvq5dLRA3RHr6wnD74WagpgC74jIEhGZ5ZZlqeoOcP7DAP08i+7wdRZ7b/2sbnBP08z2OTXYa9rinsY4Huev2V792bRrC/TCz0ZEwkVkGbAbeBenB1SlzgJ0cHC8nS1Qd8T6esLwe6GmIHayqk7AWTv9ehGZ6nVAAdIbP6u/A0OB8cAO4G63vFe0RUQSgJeAn6vq3kNV7aAsqNrTQVt65Wejqi2qOh5njaCJwOiOqrk/u70tfT1h9PqFmlR1u/tzN/AKzj+iXW2nBNyfu72L8LB1Fnuv+6xUdZf7H7wVeJgvT20EfVtEJBLnC/YZVX3ZLe6Vn01HbenNnw2AqlYBH+KMYaSIswAdHBxvZwvUHbG+njD8WeQpaIlIvIgktj0HzgFWcfDCVFcB//ImwiPSWexzgSvdK3ImA9Vtp0eCVbvz+BfjfDYQ5AuEuee5HwXWqur/89nU6z6bztrSGz8bEckUkRT3eSxwFs6YzAc4C9DBVz+XjhaoO3Jej/x7/cC5wmMDzrnA33kdz2HGPgTnio7lwOq2+HHOU/4H+ML9meZ1rJ3EPwfndEATzl9D13QWO073+n73c1oJFHgdvx9tecqNdYX7nzfbp/7v3LasB87zOv52bTkF59TFCmCZ+zi/N342h2hLr/tsgLHA527Mq4Dfu+VDcJLaRuCfQLRbHuO+3uhuH3K0MdjUIMYYY/zS109JGWOM8ZMlDGOMMX6xhGGMMcYvljCMMcb4xRKGMcYYv1jCMEFPRD51f+aLyHe6+di/7ei9AkVEvi4ivw/QsX/bda3DPuZxIvJ4dx/X9E52Wa3pNURkGs4Moxccxj7hqtpyiO37VDWhO+LzM55PgYtUdc9RHucr7QpUW0TkPeBqVd3a3cc2vYv1MEzQE5G2GTrvAk511y+40Z2I7c8istidRO5Hbv1p7hoIz+LcnIWIvOpO0Li6bZJGEbkLiHWP94zve7l3Lf9ZRFaJs97IpT7H/lBEXhSRdSLyTNsMoCJyl4iscWP53w7aMQJoaEsWIvK4iDwoIh+JyAYRucAt97tdPsfuqC1XiLN+wjIR+YeIhLe1UUTuFGddhYUikuWWf8tt73IRme9z+NdwZkEwfZ3Xdy/awx5dPYB97s9pwOs+5bOAW9zn0UAhzhoG04D9wGCfum13Jcfi3CWb7nvsDt7rEpzZQMOBLGArztoK03Bm/czB+YNrAc7dxGk4dwa39dpTOmjH94G7fV4/DrzlHmc4zh3iMYfTro5id5+Pxvmij3RfPwBc6T5X4EL3+f/4vNdKYGD7+IGTgde8/ndgD+8fbRNWGdMbnQOMFZG2eXSScb54G4HP1FnPoM1PReRi93muW6/8EMc+BZijzmmfXSIyDzgR2OseuxRAnKmm84GFQD3wiIj8G3i9g2NmA2Xtyl5QZwK8L0SkCBh1mO3qzJnACcBitwMUy5eTBTb6xLcEONt9/gnwuIi8ALz85aHYDQzw4z1NiLOEYXozAX6iqm8fVOiMdexv9/osYIqq1orIhzh/yXd17M40+DxvASJUtVlEJuJ8UV8G3ICzsI2vOpwvf1/tBxEVP9vVBQGeUNWbO9jWpKpt79uC+z2gqteKyCTga8AyERmvquU4v6s6P9/XhDAbwzC9SQ3OMptt3gauE2f6akRkhDtrb3vJQKWbLEbhTAndpqlt/3bmA5e64wmZOEuwdjprqTjrLSSr6hvAz3HWWWhvLTCsXdm3RCRMRIbiTCK3/jDa1Z5vW/4DfFNE+rnHSBORvEPtLCJDVXWRqv4e2MOX03yP4MvZXE0fZj0M05usAJpFZDnO+f+/4pwOWuoOPJfR8XK0bwHXisgKnC/khT7bHgJWiMhSVb3cp/wVYArOTMAK/FpVd7oJpyOJwL9EJAbnr/sbO6gzH7hbRMTnL/z1wDyccZJrVbVeRB7xs13tHdQWEbkFZzXGMJxZdK8Hig+x/59FZLgb/3/ctgOcDvzbj/c3Ic4uqzWmB4nIX3EGkN9z7294XVVf7GI3z4hINE5CO0W/XAbU9FF2SsqYnvV/gTivgzgMg4CbLFkYsB6GMcYYP1kPwxhjjF8sYRhjjPGLJQxjjDF+sYRhjDHGL5YwjDHG+OX/A15QPLXQFTNvAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parameters have been trained!\n",
      "Train Accuracy: 0.9990741\n",
      "Test Accuracy: 0.725\n"
     ]
    }
   ],
   "source": [
    "parameters = model(X_train, Y_train, X_test, Y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Insights**:\n",
    "- Our model seems big enough to fit the training set well. However, given the difference between train and test accuracy, you could try to add L2 or dropout regularization to reduce overfitting. \n",
    "- Think about the session as a block of code to train the model. Each time you run the session on a minibatch, it trains the parameters. In total you have run the session a large number of times (1500 epochs) until you obtained well trained parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def l2model(X_train, Y_train, X_test, Y_test, learning_rate = 0.0001,\n",
    "          num_epochs = 1500, minibatch_size = 32, print_cost = True):\n",
    "    \"\"\"\n",
    "    Implements a three-layer tensorflow neural network: LINEAR->RELU->LINEAR->RELU->LINEAR->SOFTMAX.\n",
    "    \n",
    "    Arguments:\n",
    "    X_train -- training set, of shape (input size = 12288, number of training examples = 1080)\n",
    "    Y_train -- test set, of shape (output size = 6, number of training examples = 1080)\n",
    "    X_test -- training set, of shape (input size = 12288, number of training examples = 120)\n",
    "    Y_test -- test set, of shape (output size = 6, number of test examples = 120)\n",
    "    learning_rate -- learning rate of the optimization\n",
    "    num_epochs -- number of epochs of the optimization loop\n",
    "    minibatch_size -- size of a minibatch\n",
    "    print_cost -- True to print the cost every 100 epochs\n",
    "    \n",
    "    Returns:\n",
    "    parameters -- parameters learnt by the model. They can then be used to predict.\n",
    "    \"\"\"\n",
    "    \n",
    "    ops.reset_default_graph()                         # to be able to rerun the model without overwriting tf variables\n",
    "    tf.set_random_seed(1)                             # to keep consistent results\n",
    "    seed = 3                                          # to keep consistent results\n",
    "    (n_x, m) = X_train.shape                          # (n_x: input size, m : number of examples in the train set)\n",
    "    n_y = Y_train.shape[0]                            # n_y : output size\n",
    "    costs = []                                        # To keep track of the cost\n",
    "    \n",
    "    # Create Placeholders of shape (n_x, n_y)\n",
    "    X, Y = create_placeholders(n_x, n_y)\n",
    "\n",
    "    # Initialize parameters\n",
    "    parameters = initialize_parameters()\n",
    "    \n",
    "    # Forward propagation: Build the forward propagation in the tensorflow graph\n",
    "    Z3 = forward_propagation(X, parameters)\n",
    "    \n",
    "    # L2 Regularization\n",
    "    #regularizer = tf.nn.l2_loss(weights)\n",
    "    \n",
    "    # Cost function: Add cost function to tensorflow graph\n",
    "    cost = compute_cost(Z3, Y)\n",
    "\n",
    "    \n",
    "    # Backpropagation: Define the tensorflow optimizer. Use an AdamOptimizer.\n",
    "    optimizer = tf.train.AdamOptimizer(learning_rate = learning_rate).minimize(cost)\n",
    "    \n",
    "    # Initialize all the variables\n",
    "    init = tf.global_variables_initializer()\n",
    "\n",
    "    # Start the session to compute the tensorflow graph\n",
    "    with tf.Session() as sess:\n",
    "        \n",
    "        # Run the initialization\n",
    "        sess.run(init)\n",
    "        \n",
    "        # Do the training loop\n",
    "        for epoch in range(num_epochs):\n",
    "\n",
    "            epoch_cost = 0.                       # Defines a cost related to an epoch\n",
    "            num_minibatches = int(m / minibatch_size) # number of minibatches of size minibatch_size in the train set\n",
    "            seed = seed + 1\n",
    "            minibatches = random_mini_batches(X_train, Y_train, minibatch_size, seed)\n",
    "\n",
    "            for minibatch in minibatches:\n",
    "\n",
    "                # Select a minibatch\n",
    "                (minibatch_X, minibatch_Y) = minibatch\n",
    "                \n",
    "                # IMPORTANT: The line that runs the graph on a minibatch.\n",
    "                # Run the session to execute the \"optimizer\" and the \"cost\"\n",
    "                _ , minibatch_cost = sess.run([optimizer, cost], feed_dict={X: minibatch_X, Y: minibatch_Y})\n",
    "\n",
    "                \n",
    "                epoch_cost += minibatch_cost / num_minibatches\n",
    "\n",
    "            # Print the cost every epoch\n",
    "            if print_cost == True and epoch % 100 == 0:\n",
    "                print (\"Cost after epoch %i: %f\" % (epoch, epoch_cost))\n",
    "            if print_cost == True and epoch % 5 == 0:\n",
    "                costs.append(epoch_cost)\n",
    "                \n",
    "        # plot the cost\n",
    "        plt.plot(np.squeeze(costs))\n",
    "        plt.ylabel('cost')\n",
    "        plt.xlabel('iterations (per tens)')\n",
    "        plt.title(\"Learning rate =\" + str(learning_rate))\n",
    "        plt.show()\n",
    "\n",
    "        # lets save the parameters in a variable\n",
    "        parameters = sess.run(parameters)\n",
    "        print (\"Parameters have been trained!\")\n",
    "\n",
    "        # Calculate the correct predictions\n",
    "        correct_prediction = tf.equal(tf.argmax(Z3), tf.argmax(Y))\n",
    "\n",
    "        # Calculate accuracy on the test set\n",
    "        accuracy = tf.reduce_mean(tf.cast(correct_prediction, \"float\"))\n",
    "\n",
    "        print (\"Train Accuracy:\", accuracy.eval({X: X_train, Y: Y_train}))\n",
    "        print (\"Test Accuracy:\", accuracy.eval({X: X_test, Y: Y_test}))\n",
    "        \n",
    "        return parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.7 - Test with your own image\n",
    "\n",
    "We can now take a picture of your hand and see the output of your model. To do that:\n",
    "    1. Click on \"File\" in the upper bar of this notebook, then click \"Open\".\n",
    "    2. Add your image to this Jupyter Notebook's directory, in the \"images\" folder\n",
    "    3. Write your image's name in the following code\n",
    "    4. Run the code and check if the algorithm is right"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import scipy\n",
    "from PIL import Image\n",
    "from scipy import ndimage\n",
    "\n",
    "## START CODE HERE ## (PUT YOUR IMAGE NAME) \n",
    "my_image = \"thumbs_up.jpg\"\n",
    "## END CODE HERE ##\n",
    "\n",
    "# We preprocess your image to fit your algorithm.\n",
    "fname = \"images/\" + my_image\n",
    "image = np.array(ndimage.imread(fname, flatten=False))\n",
    "my_image = scipy.misc.imresize(image, size=(64,64)).reshape((1, 64*64*3)).T\n",
    "my_image_prediction = predict(my_image, parameters)\n",
    "\n",
    "plt.imshow(image)\n",
    "print(\"Your algorithm predicts: y = \" + str(np.squeeze(my_image_prediction)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Summary**:\n",
    "\n",
    "- Tensorflow is a programming framework used in deep learning\n",
    "- The two main object classes in tensorflow are Tensors and Operators. \n",
    "- When you code in tensorflow you have to take the following steps:\n",
    "    - Create a graph containing Tensors (Variables, Placeholders ...) and Operations (tf.matmul, tf.add, ...)\n",
    "    - Create a session\n",
    "    - Initialize the session\n",
    "    - Run the session to execute the graph\n",
    "- You can execute the graph multiple times as you've seen in model()\n",
    "- The backpropagation and optimization is automatically done when running the session on the \"optimizer\" object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}