python-examples/build-image-recog-algorithm-lr-nn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build a simple image recognition algorithm with logistic regression with an optimization algorithm\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Objectives\n",
    "\n",
    "### Build a simple image-recognition algorithm that can correctly classify pictures as cat or non-cat.\n",
    "\n",
    "- Build the general architecture of a learning algorithm, including:\n",
    "    - Initializing parameters\n",
    "    - Calculating the cost function and its gradient\n",
    "    - Using an optimization algorithm (gradient descent) \n",
    "- Gather all three functions above into a main model function, in the right order.\n",
    "\n",
    "\n",
    "## TOC\n",
    "* [1 - Load Packages](1)\n",
    "* [2 - Preprocessing](2)\n",
    "    * [2.1 - Load the data](2.1)\n",
    "    * [2.2 - Preprocessing](2.2)\n",
    "        * [A. Confirm the shapes](A)\n",
    "        * [B. Flattening ](B)\n",
    "        * [C. Normalize ](C)\n",
    "* [3 - General Architecture of the learning algorithm](3)\n",
    "* [4 - Building the parts of our algorithm](4)\n",
    "    * [4.1 - Sigmoid function](4.1)\n",
    "    * [4.2 - Initializing parameters](4.2)\n",
    "    * [4.3 - Forward and Backward propagation](4.3)\n",
    "    * [4.4 - Optimization](4.4)\n",
    "    * [4.5 - Prediction](4.5)\n",
    "    * [4.6 - Merge all functions into a model](4.6)    \n",
    "* [5 - Train a model](5)\n",
    "* [6 - Further analysis ](6)\n",
    "* [7 - Test with your own image](7)\n",
    "* [8- Summary](8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1 - Load Packages <a name=\"1\"></a>\n",
    "\n",
    "- [numpy](www.numpy.org) is the fundamental package for scientific computing with Python.\n",
    "- [h5py](http://www.h5py.org) is a common package to interact with a dataset that is stored on an H5 file.\n",
    "- [matplotlib](http://matplotlib.org) is a famous library to plot graphs in Python.\n",
    "- [PIL](http://www.pythonware.com/products/pil/) and [scipy](https://www.scipy.org/) are used here to test your model with your own picture at the end."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import h5py\n",
    "import scipy\n",
    "from PIL import Image\n",
    "from scipy import ndimage\n",
    "from lr_utils import load_dataset\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2 - Preprocessing <a name=\"2\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2-1. Load the data <a name=\"2.1\"></a>\n",
    "A dataset (\"data.h5\") containing:\n",
    "    - a training set of m_train images labeled as cat (y=1) or non-cat (y=0)\n",
    "    - a test set of m_test images labeled as cat or non-cat\n",
    "    - each image is of shape (`height`, `width`, 3) where 3 is for the 3 channels (RGB). Each image is square (height = `num_px`) and (width = `num_px`). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Loading the data (cat/non-cat)\n",
    "train_set_x_orig, train_set_y, test_set_x_orig, test_set_y, classes = load_dataset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(209, 64, 64, 3)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_set_x_orig.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each line of the train_set_x_orig and test_set_x_orig is an array representing an image. Let's visualize an example by running the following code. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y = [1], it's a 'cat' picture.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP4AAAD8CAYAAABXXhlaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJztvWmQZNl1Hvad3Guv6q5ep2emZyMwwIAYgCNwCMAKLAQNUTTxh7RFyQ7YnogJKWgHFaZDAOwIh+SwI8g/Iv1DQceEQQlhUwQpUTAQCFEiYowRLQscYLByFmCWnp7eq7qra819uf5R2Xm+c7IyO6uXrBnm+SIq6r6899138753851zzznfkZQSAoHAZCFz0AMIBALjRyz8QGACEQs/EJhAxMIPBCYQsfADgQlELPxAYAIRCz8QmEDc1sIXkU+LyE9E5HUR+fydGlQgELi7kFt14BGRLIBXAXwKwAUA3wHwaymll+/c8AKBwN1A7jbO/RCA11NKZwBARL4M4DMABi78TEZSJiM37dj/FtljPT+TyZp22SyX86au02nvWU6p466lFxOxY83m5nrlZqtE/fkfzyZ12DI1mUybyrZOoGOxP8jJtRsN5iw/qTJaLzyOvldE2rPYd71MZrBgafp3Y8zSDc0Xir1yo14z7fiRymbtI83nlWYWtDw9Y9qVitpuc+2qqVtf12N+doahb3ZlcC0/P3Y+RrpUH1JKN725t7Pw7wFwno4vAPjZYSdkMoLZ2VyvbKHHjYb9xq2W1nVSoVeenp037Zbm9QGbXzxu6irl9V65VtnUa9Wr7lq6GLO5gqlbWP5or3xp/b3aX7Vp2qF1qVeUdM1UTZU2euXZKVuXzVR65XaL+7Q/Tpn+ZUbQumabfkjcjxPPf9+9oKaNlj7o3B8A8BrwP378AJdKNI/uUo26zne7ZftfXNT7e+LeB3vlc2/+2LQrZfRahxYPm7rj9/1Ur/yuD326V370g0+adg+dfrhX/tP/65+Yuq/8yf/eK++UNzAIGfox9S8N8+Pn6qo1vde1Ks1H288pH+29tkeV4G9n4e915b6risjTAJ7eLd/G1QKBwB3D7Sz8CwDupeNTAC75RimlZwA8AwC5XCbdWPx973vRX8ScewMlEulTR0XsRmvKtCsUtI+jx06aumY60StfPvtir9xp2be1sOjfsXVb179H1zpF47BvmU5HxciUtkxdGiqoy55FOMmND/3bn3/wudx2bwLzNnHSK0sHLXrLt1q2j0Ei6m4fWq6mRq+czVqxv9XUi5dKJVN3z70P9MrbWyqxpbZVkVhsn11YMnWHj9+v46Uxtp2EIh2WNqzk0WyqauHv3qjSOEsAqe8NOP434u3s6n8HwCMi8oCIFAD8LQBfuzPDCgQCdxO3/MZPKbVE5L8B8G8BZAH8fkrppTs2skAgcNdwO6I+Ukr/GsC/vkNjCQQCY8JtLfxbww19xmoZrNZ7Hb8jquO3RYfc7lidsNVWffHQYburf/j+R6l/1cwuvPGiaVct645/q1k3dfWm6pk5+WavvLz486bdtTXVaVPLm/PIXOh0O7sTzPNjdc7h2Fvv9jvEHd7xdz2w7t5ucR+DTZ/D0GnQPoHTb4sl3ae5/4FHTF2W2q5fX9VzcvbZKZHJLpezZtzi1HSvXMjpeHOtsmlXWb/cK19bvWDq2vRc9ZvpSHfnj/uaycBasy/j+7e98Fl79j+qCTBcdgOBCUQs/EBgAjF2Ub/nVDTEgpER52wiKioKVHTutCumXbsz2yvPTFlT3xEy71Ueen+vXNux5raNlbO9crXiRGyyUVXr6rs0Nf0t02x5Uc1Qtbr1LsxBj/utOpk967xjIJvKvI8Wi98ssnecqM9muj5HEe5/iGfdMMgAETiXt4/cyft0ro4evcfUvfnGX+oBmVan3b2dnVaPykNH7zV1LXIKunj21V75kYesWrF9VcX7lZXzpq6TyNMTg8HfOeNurlXrhjz8abA4z8f9PjH7c/OLN34gMIGIhR8ITCBi4QcCE4ix6vgpAZ2uayS76AJABqwTOtMTRbgZl1dyBQWAekP77DRt3aF5jczqnFb9rrZ93V6roWaejPNlLVKghTRU5yxvvmraLR3Wuofuf7epq9RUx9+8vmPqktG7tY9O8nsNrHf7wBnug8odb87Tctu54rIqLzLYSMV6ptcwWd9lN93lI8dMu4fu1+CblRVrRtva1HtTor2B6YINnipNq8nu0BG7T1CjoK7ZWXXtvfL6j0y719av9MqXLr1l6nivxJtgB8G7Ztu58u/bYXr93UG88QOBCUQs/EBgAjFeUR8Jra7pyEn6RsDpOE+11gCTkmStV1yVIr3KFWumW17Q2O7pKRW3169a0XB786FeuVi0noGVDfUey1Ec/5VrNq5+fe1sr7y4YE1PR49pVF8us2zqmnX1QGuv63dhzzHAmemcCM/mq1Z7sCnOCPBDTENMhiHOozLDdQM8yQBgcelQr/zgQ++yfVCfW+urpo7NaIW8iuziyDaQ1znO5IumamlGTX0nlhe1j4pV8S6u6/NSdibeIRwaAzHMc0/SYDWA0a8+jXLF0VSFeOMHAhOIWPiBwARivJ57Sb3EslkvejK9lj2NCRR4hz/jdvXbLRW/m84brZBVEbjWUWKFVmXdtFs+qcQNq21Ly1WvbvfKUyR6zpZtwMf1HT2+dOmsqWPReW5uwdbNq+dhKzFt1pppV62qNaDtSCnYC2+YSsBiY6FoA1tYoiwU9XsWi1aMniYCjFbTjiNX0D5PnFD1Zm7O0qVduXi2V06Ozy5LFoUOqTCew292Tsk3Ot67kJ6JRln7OHH8tGlWaepSaP27PzV1bGWSEeXvfta7ASQrfeDt/8HqmadL2y9pbrzxA4EJRCz8QGACEQs/EJhAjNmcp7pmZ7Az2h4nDmCQTJZfvdVQff3yZRthdfXSm73yoWU1Ly3Mzdl2Z8/2yl4vLkxp2+q2mvDmnd5aId73Zt1GEK5eOdcri9xn6o4c1+PlY2pm9HrlFJFLbKxbDnjm6q/TOHy0WLvDOrPrn0gv2Yw2MzNr2h07rhGPDUeNPUPU5xm6Z6tX7H1p1HS/opN8ngEdc44GWZqy41g+rnsItardl1lf1fnOnlTC1cc+8DHT7uUfazqIVsvuHdlB+ai7uwe/fXCrPPt7Id74gcAEIhZ+IDCBGLs5r2elGia3eHGKfp7YWyyT9SYN5cg7d+GsqTv3hopyizOaBWduyXrPtV57pVeemT9q6pYWj/TKVy8QF13diq/sJbiybr3AGg0V/Tc3XSYd4o6bntE+jpAoCwC1qvbRcKpElUTdRkNF1kLB3upaXeu8eYwtRXkS9efmF0275WUNuMkUrbqzualq11ki1PBecfef1O+2vW1Nq3m+v+Stl5my12owcUjH8iQeP6nq00c+9Z/ode+3atYf/p+qCjaHifqeL29EK53h3Ot79genS7N9DLlAeO4FAoGbIRZ+IDCBiIUfCEwgDoBXf1cH6U+FPSTTaI4IK4ljHy5NNqs3W9tWl3zpx8qff+yYmvPK61avzBH5Rr1miTIWT6g771EyPW1fXzHtClkl0ViYnTZ1OzXVH5su3fPOlnL6synunnsfNu1mptXVt7plo8xaTU1fmCUd2bt0lopMZmHnu1BQc16e5v7QoSOm3b33aa6CluOzL1d07srs0uw9asms2HFRiAXqM5vRRzWXs/edXZ87Lavjv+d9H+yVP/rhn+mVf/Td75p25869oUN0pmb7PDrX5wGavX+GrcfuaO/bW8xsPhJuOgIR+X0RWRWRF+mzQyLyDRF5rft/aVgfgUDg7YVRfnr+GYBPu88+D+DZlNIjAJ7tHgcCgXcIbirqp5T+XEROu48/A+Bj3fKXADwH4HOjXPCG+NLx6Z2HcIYLmZvSEG4+Pmy1rUnmlVfVnPfTj72vV778pk2hJU0V06eKNv11lkxbxTk1beVnrZmrfE1FW29c4T4yjo1ka2uN2qm4feSknZCjZEarVbZN3caG9lEq6RyUCpYQZHtHz/OcfkaUpi+QcyQXh8nTsFyzEYrthpoV88SR13ZpyWqUprzpPCWbpP61OLeCG8c88SmmjvX+O3RY7+HGtY1e+T88a1M+Xr2qKbT6WTQGHthmQ2RxTo3drx7sX4bvv9Z4ovOOpZQuA0D3/9GbtA8EAm8j3PXNPRF5GsDTd/s6gUBgdNzqwl8RkRMppcsicgLA6qCGKaVnADwDACKS0qBdfSpn+gQRIqVggoq2F29YnLJ16+u6+71TURH4/nc/adpdvXSmV647iu42kXa0SCydP3TCtNu8roEznbbduS/maZffecyVt1QUnVtUcXaeaKEBYIropI8es5yBjYpaBs6d0+/i+QPrNR1XO9nddNDx4UMqzJ2690HbjCwgG9ftI1AnTkIm7Nhct/Nx/bp6L3acyJslb708BebMzlnV6sI5CsA6ZNWzHboXf/7cn+k5V6wlhgOy7sZuuiSifh8ilY+eLXc/5/XjVkX9rwH4bLf8WQBfvcV+AoHAAWAUc94fAvgWgHeJyAUReQrAbwH4lIi8BuBT3eNAIPAOwSi7+r82oOqTd3gsgUBgTDgAz70BUUSctsnJIcJKF3HnZ4aY/bxe1iQz0vmLGk33+BMfNe3ml5Vc4rVXvmfq1tdUL8xNqZ55/JTVfa9RmuVqzUbPtSjdcwY2FRQLYC2KrFu9YlM6HTuuqaBnHAkI121u655Bu+W84or+2oqlBdWTD1PKq8Ul66fFkYYXzv7Y1K2Tzl+tqKmvzwxFt3ZmxpKPTpMuf2hRoyiPnTht2knSOT111PYxQ7kXnvvWv++Vl0+/z7SbmvkPvfJOecPUJbN3NCL6bNJMIOOjSgd2YtsNSVm2X4SvfiAwgYiFHwhMIMYv6g+QUdIQWZ8JN3JD5B0+LevJJej41deUGOLKuSdMuw99/Jd75dK8FRu//f+qOahDfPYV5z03T+QejYb1VGOPOc+9zkQXOSLlqFSsulAnk910yXrktZY0kGZqWk1gTSfqL7C3ntjHYGpBRfrj92hKsZLj3HvzjZ/0yquO45AJNzrkTZdzwTzFaVVV5hetKS5H9316Rk2CGbGehtMlnasTx6wv2bFjamq9935N3/XWVWt+bLLpto8nQ/Yq9sFU9cnvQ0xxhkfS1AwcxxCNaSTEGz8QmEDEwg8EJhCx8AOBCcTYdXzVTQYrKcmR7rM+w5FjfemGSY/v1/G17ca6RrC9TuSaAPDx//hv9soPPfiAqTv36vFe+a1zZ3tlJq4AgOlZ1ZFnm077yqt+3mpYDvh2W793Lqd9llzOurVralZszFq9e2FJ9eT77leijI3rln+/Seeltp3vQ0f0ey4tq45cmLauspWy7j00XR+st+aIMCWbtWbEedrXEJcHcH1DzWoZImBZWNg07ZbvV4KUxcPWhXn+mJKYnDil5CD/37f+qWlXJ7NrP0mMlofwa4z0OTC6Pu7H4c3Xt9Tpjb721zwQCPxVQCz8QGACMX5zXk9csbJJm1Mk90VHDeLjGyyTdTpe9NTzOM30W+fPmHY/eP65XnnxmI26m8lr/wXD+2bHMU+ebyXnjbZBfPOduuX0y5Kpa2uLUnJPWd6+IqW19uLg1JSavd792Ad65Zd+aDnmKts6P7OLNrfAIRLvsyXtr1yzpskORU3mp2wEYY5MnByp5/MAZIh8RJwaUKuqx1+5rObBrbKdt3JVx3X5ymVTlyvouC4Sv39ly+Y0qDv+Q4ZhvR8SVTpqFJ/nPxyY4tprw2zJdnnVhpkZ90K88QOBCUQs/EBgAjH+Xf3uf09hzEEkXmwZLOr7dvsdBZDJ2Cl489zFXnnq4jlTl++omFqkrLQ1t6OdzaloPud25Gcpi+zGpqXGzmZVhJ9d0p31LUffXZpRq0Eu78TjhgasHD2m7ebnrLpQr+j3XnbBN/mS9pnI8tBwvHpC6lkpbz3yOMyF1S6vgpVJbM/l7Hdh0pUyBRxtEq8gANSban05cvK0qWvX1ALAJCsrV6+Ydk3i/utjxBu2rc/jNSc5i5Ph3BttC34QdffuxXzW3tQ/hiGIN34gMIGIhR8ITCBi4QcCE4ix6/g31KXkGAeZN9Op/5AMc+6Pqsh7kwmV6QJO5cTMgka37ay8ZuoeuEfNXKmgnm8vn7FEGeWyRnrlcnaK5ygCbXbe6tZ1Mpc1a8zNb78zE30uutTVO7RvcH1DvfUOE0EHAGxRuq7ZJWu25Ig/5puvOZPX9Q29VnnHRijWa7o30GrpfkjbpcnKDNH/C7R/wX1UnTmvRR5/GXff33xdCUJ+RGnUtsp2v4Lh59vU9QXM7U3ScWuJsP05Pl3XsMbj4dUPBALvYMTCDwQmEOMX9WVvXn02XXSGuEfdKsV5Ml59+vn5SxdMuzdfV++uE4etR9tPffDjvfIa8fSvbDdNu0svv6QHLlhoi0Tx+UUrprc72nZ7Uz3LNtetl9ksEYQcv+eUqUv0W75DfZx+6F2mHWemvecBm423Svz+uXV9RC6fecO02yJRf3vLBs60WnZOeuNz95bF+6lpSypSpFRZnOar07Z9J+LcW1uzc3VlTT0lq3VVCYolG1hVq1KQTsflGRjdJU9PcVVDTXPchTnH9TEko3RfOrmbIN74gcAEIhZ+IDCBiIUfCEwgDsBlV24URofhIrwV054fhPaxsmpdN1dXLvXKbRdJdu68mu2YQNLvSQgxJlSr1szVYPNV0xJx5ErqVst7AU1H2Hnx7Ku98szMnO2DCCo3VlVXf/inHjPtFg4pX37OGVC31pWIcuXy2V75qifU3Fb92evFfGeyWY1k9OY8JkwpOPdjJsBksk027QHA2pqaLV9943VTd2mV3KLzuocgGXtfkvEhH2wKHp6eenD+Pd5T6TgX70F7WMO09v3q9B6jpNC6V0S+KSKviMhLIvIb3c8Picg3ROS17v+lm/UVCATeHhhF1G8B+M2U0qMAngTw6yLyHgCfB/BsSukRAM92jwOBwDsAo+TOuwzgcre8LSKvALgHwGcAfKzb7EsAngPwuZv1d0NU8iKTEXFG9ELy7UYV/bnVDvG/A8DrZLKqLNv0VN/+5ld65cfe/7O98vET1qT2xllN21x1/Wcpiq1P4qP0YPxdWk6MbpCI/fIP/8LUnTypHnoVuvb5tyzhyDyRbVSbVnQuV1UFWV9TsX9n26aWqlOarz7TE42fxfl5l+I6S56NWeflmChdd4ZSZheK1uxX3lZT4rVrlluw1dFxzC2qV2a1YU2CnB7Ne+4Nfa4M5z6bjN09a+lctZrepDnoeR89LbYSzYy2dva1uScipwF8AMDzAI51fxRu/DgcHXxmIBB4O2HkzT0RmQXwJwD+fkppa+S3q8jTAJ6+teEFAoG7gZHe+CKSx+6i/4OU0r/qfrwiIie69ScArO51bkrpmZTSEymlJ/aqDwQC48dN3/iy+2r/IoBXUkr/mKq+BuCzAH6r+/+rN72akPdjvz8iFX0a4dtNCjwY3gx15colqrNmtFkynS2R2a9VsgaNQlHdQadnrbltivq4fvWSqZulXHdLy8rA03bur1LQfYJKxUaZrRNbD0fTvfTD50279//Mh3vlztwhU9ckMlImyqw17Tg4Ks6HORaJIDRH5s0F56Y8Padu0W3XB+cP5ByEXseH6D1stmwEId/fDJkVvUuxUOLFPjJMNtPZKw9MdddyensiE2SrNTgHAZuCpY+Uc9ABsF9n9lFE/Y8A+C8A/KWI/KD72f+A3QX/xyLyFIBzAH51X1cOBAIHhlF29f89Bv+cfPLODicQCIwDY/XcEww2jfDnt+uVdDtgL7wWsqbuekXFwwvX1PNr/pBNY3X4sJqNPIFkh0TRY8vWELJTUbG6RKLyzLQlysyTh1vbsZZyJB+bpc6ff9O0O35CU0099KgVvzcp6o7TjTWdxxyboXzaszyJ6RmOjHRpstp0fPye+03d1LySlrRIbSkU7WO7clkjLNvthqnLkVrAHoQzLg9AntKg1WvDSDoGH/OtaDtbbcfUOXOe8dwb0XevL4A1ovMCgcBNEAs/EJhAHEC23Buke4M997w2cBc39fcQmRS5vN09zhdVPFzfVnGwtGDF1x3yJIOzGkzPKYlGecN6mR07ruI3E0NkXZrUWkU98kousKVC3nRNCgzxfHln39RglnkK2AFsoFK5ouNoOzG9QUE0eTdG3sXOFXWMVcd1V2uoDFwo2MfxaE7VheVjJ/VaWauCFfOqCjGBCQCAx0F5C3w25aFc98PYMQyXY9qzDPhneMhDx5catg588ong3AsEAjdDLPxAYAIRCz8QmEAcGBHHwICk3Ua3hGEefoPr7MXY5LN82JrbpimyrEXRVrMz1py3s0kkkW1LtsEK3aFjp03N1Kz201nRHH6tpvUgbJI3XSZrc9axR1qtrmPMuMi3teuqC7/0oxdM3ca27iFwTjnvccaZmv0bhD3+uJx1Cm2nofsQ2ztW/1+gPYQK7TX4HH45IuWsVOx8z8ypp2Qhr3PgyTYLZH6s1ywBi3lG3IPbsSwxVHbPFU1Qxu0vNGkfqI9o9i4h3viBwAQiFn4gMIEYu6h/uxgxY/Ho8FYRKvv0VxkmWiBRbmPLElQIkUYcPn7S1K1TYM7hY/eZuhzJg+W6ive1hvVG2yZRvOQCVmz+ABXNOQ05YM106xs2Xbfx0BNVfcSJqKlN3pbOg7BFYjqLr4WcNcV1oNfqF3K1zwaZI8uOw39qRglT5g8dMXVtSvOdzaraki/Y72LutXuwDKlGH/kLVZEakHEPFqdj7/ueFGTE09ivnt6JB747njvWUyAQeMcgFn4gMIGIhR8ITCDGr+OPpKYM9ou8db1+QNSTeF1Mj9fXLpu6rU11sT15SvPNTU/bSC/Wn7cd2SZH1mUzVt+9Svp/k0x4GTjixobqu32kEawk0p5Exn1PJrbwXPesaJqoSReBxy68zWTfIaUCuRJTxJx3lc0V1Yy2uGQJTfK0H7BIeQwPHbZ6/OZ1JX9qOtNnjvo4tEREKjN2n+DsGebEdya79uC6TGZvc543VzOpxqiWbE/6eScJaeKNHwhMIGLhBwITiAMw590QV7wYMxoBgeEn25fcv3efuZzto0gias6Znlg8ZtNbvWo9vcyvqSNdOEx89rNzlrf/woWzvfL1ayS+1q34mqXvXa1bU58xB3G0WHIEGCQStxyXXo64/7M0B36+WWz3HmdturbQjDSdWXF6Rq9V3rZprXbITLpxWdOXvecDHzbtiiW9ZwWxHnkZMkdWt1VFWrnqVCSeN6f5tFusGo4WxdenLrjvPejaBnfOeteHeOMHAhOIWPiBwATiAIg4BsgvafDO6Ujn3yGwCO933acosIO94mouqKPdUm+xzTVLDDFNfWSL1uMvS+I4ayCJCCkAIM8efjVLsMHedTxXGf8TT3X5giXzYCsFZ+qtlHdMO97Vb9SsOpLAgTk6V9NFH1Sk5124+Kqpy5Gs+9BDuqt/8pClA18jT761TWtFKdd1zO223s+NDTveWk3H69NfGZXJPX7Gc4+f4Y73/iNLie1idA6NUVPpjoB44wcCE4hY+IHABCIWfiAwgTgAc97e6Xw9OeHtYqiXEylm4jQujtKqVKxOyyQdfFqtbM1Q2zuqu5ed554QgWfJcbtPkRdbq8Hef9bLLGuiBJ0ZjdM/k64qfb/xel7REXbOEKd/jbwQve7bIhNVH998W8dRKOi8ZZ35NEOmwwSr/588ovfi7z2lqRdXMjYtefaSEpiUG7b/MhFzmJToHWvezJk5cFGIGGRvc2ZAeij6H2dOEWdrRk/vfudyT9z0jS8iJRH5toj8UEReEpF/1P38ARF5XkReE5E/EpHCzfoKBAJvD4wi6tcBfCKl9H4AjwP4tIg8CeC3AfxOSukRAOsAnrp7wwwEAncSo+TOSwBuyLz57l8C8AkAf7v7+ZcA/EMAvzdCf7v/Pfcae+QNOKe/dnRxh80wYsr2ak3yYmu1vWirdeVtFec9f9vSITU93X//w6ZOMipivvXqD0zd2roGAXXItDczY8k2GttqihKfkqqpY+bxdxypSL6oc1cq2vEvLKq5jFWV5OaD00Rl3StkpqjiPd/bqgv0KZGnXcaN49R9GlRz6v3v65UvvWSFywaNS3JWXeDUWJzHquK8LTnzrycVYXgxfdCjuZ/cEOb5Hib1U0CQOHPhfgN4RtrcE5FsN1PuKoBvAHgDwEZSP9ALAO4ZdH4gEHh7YaSFn1Jqp5QeB3AKwIcAPLpXs73OFZGnReQFEXlhTASigUDgJtiXOS+ltAHgOQBPAlgUkRvy4ykAlwac80xK6YmU0hN32ekuEAiMiJvq+CJyBEAzpbQhIlMAfh67G3vfBPArAL4M4LMAvnp7Q2GX3WEDGlY1uHLQj44nkMxmtSFz5wPA4pISQBRKqncXS1YHZ1LO5Tmrt9auaZRZWv+2qXv1LTUfrtV0HNNFq9Nyirn5GavTVsiDt0JeqX6/gl1xOx37PUuc4poj8Jx+niiMbbZkH6U5SmVdoX0HmbFkGymnJs1Mzs7je993ulde39FIxvUNy6u/uaXm1GrZmk+bddXlmUS0VHJpsrN7RyQCLuX1qJmrvcnuVvamvHswbVR51+Fe4xHF6lHs+CcAfElEstiVEP44pfR1EXkZwJdF5H8B8H0AXxzpioFA4MAxyq7+jwB8YI/Pz2BX3w8EAu8wjN1z74bZoV9SYW+6ESFWTGdeuT5RaKDJxBNIqFiXceLx4pKm1Lrv9Lt75eKUMy9RpFrRmagefEzPe/jD1uPvJ1/U6LRViiTbaVmTHepqSpx3nnD5Kb2lPDubVftd2AuPI9MAQIh3sE0mTO8lyOm7Z12K61mak5//qHraPfDTT5h2v/+nF3rlaWe2/I8+/Fiv/NYlNe1lCjZasUNkG/WGJRVp0dyxplKpWnWBn4OpaXvPOvQctFte3dHyiBwdfSbkQaf1+VoOu0C3zpOIDEL46gcCE4hY+IHABOIAU2jJwCO/A8/HWcp4mnHtMhREk3EeXB0S+ZhAIpO1ndTIo6tTsCIfB860yLtrtmh3iAt5DXKZnrekEa2Mnrf8rsdN3cOP6u7663+xphUZK77OFI/1ygs+fwhkAAAfW0lEQVTVVVNXpx3o3LSKxLWmVReEBMycIxxJpAbkKYgm5ywgLboxnp/wwfvUn+vv/r2/1iuffv+nTbvTj75Og7KibL74UK+8el3n5rojBLlG/IT1uhXhM+RSWKRnJ+8CkxKNv1V2c8Ueee5VaZz8hknig6tG3u83loE+sV+654+2qx9v/EBgAhELPxCYQMTCDwQmEAen43v9nD7wene+oHrm9Izq07m8NeuwF16xaOvYU4158JueU550PU+2WSViyzoRbNZq06ZdoaBmqUrD6lw5UhJfu7hg6n7pFz7YKz//2ne1f8er/3d/9cle+UTnvKn753/4/V75yo5+t1zW6fE03y2XhptNVHxf8k6PZ0r/lLF7KotHHuiVv/UdPW+tYz27f+av/Y1e+a3z1uvupRfP9spXrqkJ860L50y78iYRn5QtaUmOzIzGHOlIRRpVMmm66Dz2Xmw1XeTeLcWfDEkRdyvd3cJ58cYPBCYQsfADgQnEAWTL3RVKMs4Wx95MxZIV06dm1GtrekrLpelZWHB2WFvTJtMcB2RUqtY0ZDOeWrGuRrJts6JmowKlxQKAw0s6rpXLF0xdmQgqqmVrLvzoR9RT7Tf/a52DtQ0rAv/n/5l6UK++/j1Td8+31Btw6y3yiitblaZc0+OaE4/XrmqW4FpN50ecXMuEFVKw6o5MH+6VVyoq9r/1TXut+878Za9cnF40dW9cuNIrX7igKs321rppV8qzyc6a6SrEGZgo30HZ8SS2OAMxLFoUZHQnQstlgClut//RLuAD0u4KEUcgEPirhVj4gcAEIhZ+IDCBGK+OL9Ljpvf52rI5SnXs6qZKqjOzKy7r6gCQJ0LJtjPXZPg3rtThCoM2mfd2dqwe2CYCTBEiZ3Qc7VtbqhfPzFqTXbWsrri5qtXTvv+K6qOPvedneuWfe9LuZfzwFc3H9xf/zt7C7JK6uS4Sn+TlazZPX5sizuouUu3SBSUL6bQ4Nbg153GegYZzCd4mks6O6BiZYx8Azp09q/3PWPfmSl3vRYm4/r0+Wy/rdyt50hKK5Fu5rPezXB6cM8FH+N2RlA8yoAzckklwvzq9R7zxA4EJRCz8QGACMVZRX0R6KapKU1Z8zTvCCgPynErEppAcBxxMWmjHm0aiOYtunnstFYiLPlnxtVVTkXhlVU1NuYL9LoePqHmvULTkEu0Opbi2DnO4cOFir1yj1E+vzM7ZhmRGa6V5U7WwTL/lBS2//MpLph2TlnScrFkn0xanDS85FaxChCM7LlXYxYtne+Xle9Scl8naPr73/J/3yo1k30MlUpMOEdf/wtJh025u+nivXHWmyTal+cpk9XH30YQdikL04aGtxt7Pjsftit/jRLzxA4EJRCz8QGACMXZRP9+los45oowMBa+0Hdcdu0sxMUQHtl2dqJTzeas6cAAPE3bMzFpRmYN52m0bHMOOfAXyEBOXTfU6eb7lXSBRhiwPkrdzsHVdd/zPnntTx1S11oVDC+rhVszZ3+5GRb3aThxV8bjjZFTmy/NejkUaV4mow1PTzsd0UdtV61ZvuXjujPaf0TkolKxadH1dLRR1ZxmYpkCo5UNKbT7lUpYxccj2hvXq297WudugtGdtx2NYLA7O+Wq9TH2W573P6ZP6zfGdTzChnq9BxBEIBAYgFn4gMIGIhR8ITCDGq+NnMmTecnz2beJvb1vFqdXWYZZIPWq7FNEcLZZ33Ou8N5AnvZiJPQCgQTr4yoqNrGuRjlu8rNFiPk32wqKal8R5u3HbrQ3rTXf9uuq765uqq9Z2XPTcmu4FTGftXB2d0+tdS0Q+0rL7IeyFNz1t52CO0nxzFFitYvcaOLWX405Bg3TolcvqCbhwxCZVXj6mnPv8/QGblnz9+kqvXHS5BCo7NI+OWH5xWb/LVkXnMV+0+ytCptu6zzNAl/NRpcx1b6zLnq8jcfn2zX6em3+/GPmN302V/X0R+Xr3+AEReV5EXhORPxKRwbsjgUDgbYX9iPq/AeAVOv5tAL+TUnoEwDqAp+7kwAKBwN3DSKK+iJwC8DcB/K8A/jvZlTM+AeBvd5t8CcA/BPB7N+9tV8xpdQab7DhgB7ABFCwktZvWhJQnE2HGmUy4D+bqy+esStAms5f08c1rHZscmYsfABZmVcRuOO+/Kcoi68XG82/9pFdeW1Ox99Dho6Zdu6NjbjiT5uqmXnvjrHoCik9/RarP9KzNYLu4oMctUsHabesxV6urh9/8ovUuzBNJSoUCYhYWbSDO4tH7aEzWyxHQuZubUy++urvvBfK+XF29aOraRMSxQ+Qb3oTJ99aL4kyc4SXsToe9Rak/p8qyNXWQCXCcGPWN/7sA/gFUczkMYCOl3lN9AcA9e50YCATefrjpwheRXwKwmlL6Ln+8R9M9dyxE5GkReUFEXvCbdoFA4GAwiqj/EQC/LCK/CKAEYB67EsCiiOS6b/1TAC7tdXJK6RkAzwBAvlh450QxBAJ/hXHThZ9S+gKALwCAiHwMwH+fUvo7IvIvAPwKgC8D+CyAr970aimhdYPoYshPgLOADTZdeH2Lyp6Ig9RF5Mjd1qd+5i6zbiCZjE5Xkcg7m473vkJRfPNzlkCyShFtK1esPtpgfntjqrRjvOdejXY7fNjq3ZUtIvooaQThxrYl2+Ael5aPm7r5edWn2YV5umQJNdeuas66rbLd53jfuzVHQHlbzWjLR0+adnMUaefdistEhMJ7EnwfAOD6Nfqeay6XIJkxN8glOi8+nbbOfcdHfQ6B0esTRzzahzND363j9xcOIKrvdhx4Pofdjb7Xsavzf/HODCkQCNxt7MuBJ6X0HIDnuuUzAD5054cUCATuNsbquZdSUjOYk244XZUXtRpkNsoO86JK7EVlzWh8HvPqdVyUYIEIQZjPHwCa9b251+t16+l1/Zp6mTVdeqrFJfUkazeseNwmsZrTOM86c1uOzF6eGCJf1DEvLOucll/8vmk3Qxx2c04dWT52b69cLOk4Lp+3qavYc7LhUpG1yOQ2v6jfGe7eZklWPuzyEzSJz75GeQySM5GeOaNm0J0d6w3J6lOH5ruUt89OnVJj9ZvzBnvJcVvr4efamVzbgyP89qFl3BbCVz8QmEDEwg8EJhBjT6F1g7Si33NPj8X9HLVItGuSzO7png3ldd5+NablzpKoldw4UtLzPElHlTy/TAZbnw6MdqC3XWBLflpF8Zzj45smjrnD5K03u2C93djs0fG/3fQ9K0Rk4YXVakW96a5es9aF+x5Qq8HsvKoZzdYbpt1OWcXvxUWrLhxe1O9SmNY+co58JE9EH5UtO1fs/ba6qtbizfU10271itZl844CnKaHHwmfJZlFbPFzSpM3uhowurrA0r2h/LgjvN57I974gcAEIhZ+IDCBiIUfCEwgxq/jd3Ukr1s3yfzmyTYLRY1GY778nEuhxXsDWWdPMdzxxK/uU2G3yWTnNWNO+1Vl77wlp4PT72nFpWpq1M/qtZ3tpkimxNKU7i/ML9novDylhU5Od8zQnDAh6LFj1jvv4gU1zVU2rc5cI478FunC005/XpjXMeYKloykXiPTZ0bnoNW27bap3fraVVN3/aqaRZlsY2vLEnZ0aH8o49ziCjTmRDkNWn6Licp+j4nhdXXW+Yc74A2rNJsIe/bdd+zv+z6JOeKNHwhMIGLhBwITiPGK+imh1eVi6zhZKxmx2orALN4XyfxTcIQdLO14Mg8W75lHzgtgzPNWcJz484eP9crVHRVfK2UbANOo63HTZV5ll66FBRtgw+a8Bnm+ZVy6Jw4Gybjf7jqZD8vEI3/0mKVLWCd+u/vuf8jULRgiDr0XDz7ybtPuyqrmDzh30Xr1NSgNV41SUNWcl2OLboAP0tnZUt5BFvWnpyxHIJs7G1WrWuVFn7OdbfL4y7hsykOy2bJZzauG/AClAeXdY6NMmDoOBmvT89fnxJeGmQT3Z/qLN34gMIGIhR8ITCBi4QcCE4gDi87zJjsmtvQprgtkKuIoPnGuspwPLeuir0pMxEnntRw3f4si5Eo+r15T9SjWVWvOZGf491367xbtbVR3rL47O6/6+elHHuuVy1s24qxIhBjVLafTskszaYkdZ96cm1UyzPe893FT9zDp8lfXVM8+dsTuSaxTnroVikgEgCUycZ5/62yvfOWS3QtgF2Y/31NTOnflHXKD3rZ5BpgjXxyv/jaZXevkZt1q+eePcwnauRI69nVM5MLkL6mPZk77z7p8h01Owz2E9NOo8V6l3+crPN74gcAEIhZ+IDCBGLvn3g1vNS/GZEh2yWQHi/pMUOE58Vhs92mQWxk1kzBPuh9HllJodVwfdTbbsWhYsNNYyHD0n+2DcwHUa5akQ4gjf/WCppluVq3n3jylya5uXbfXphC0I4dV3PakJZvrarLrI9FoqZjKEYnnnalsY1NVEDYBAsDcrIrwPAfNZs20y3ZUbRFnKitkdR7rVfUmZE5DAJgSNfFmnQxcLuv1WpyG23m6MTdGGz41Gx+YKvO88HPqn78mXbtRt3XejDnoWsOwX67+eOMHAhOIWPiBwARivLv6ULEp43aZs7TrXnA74VkSX9sU2FJwGXFLdF67L+MueUSRqJV3wSUs+mezdnqm6Hq1qtZlp2ZtOwoMubbmAkqonCvYICNOSbW+ppTRLjYGJdrF3tywATapRVYJUkFOnrSeexwk9fLLL5q6DbIUNInMo+xUkzNnXuuVeQceABo0/1vk5ZibsjyGS8uqxoiz9GxuqpqRo+djdtZ67rFYXXbpzAbtmPd5542a7MWJ36323qpbfzAPl10nmb1leunz/ts7mOdWEG/8QGACEQs/EJhAxMIPBCYQYzfn3YDXgYTNed5zirz1WM3x3n95Mq3Aca/XmWCDbB85x6vP+qInZGTPL/bWq7m0zZhTnX+e0jsDQLNE+xBNO35OKmrzDNh2fG1PaJIlIg4TD+ZIS2bn1ST4/R9azv3rG+oZV6A5Xd+0HnOXLitJ55M/9xFTt0ltOa11y5mu+N5Wq9bUt7mtJrws7cXMT1sdn02OnbafUy3XiWPf7wHdMkbVtZlW3+0vyADmj36yzVFJP26OkRa+iJwFsA2gDaCVUnpCRA4B+CMApwGcBfCfppTWB/URCATePtiPqP/xlNLjKaUnusefB/BsSukRAM92jwOBwDsAtyPqfwbAx7rlL2E3p97nhp0gUDHHmzSYf86nv2LO/UJexbxOy4rYdfI4azqRr0oEFUXis/dcZWZUjhOvVFJzHtHeoda22XK3tvTaRW9ynNJr57OWLGRrS8XjHHnxSdMG82Q6KhI3nZrRIdF5fV0962bmtky7hx9+tFe+dOmCqVsjko4M2ZTYSw0AHn3ve3vlQy7j7g6pI8vHNSXX5qYVCnfI1Dc7a019Obr29paOv1r3Ho+kIrkxFqeIr5E8PRsutVmLVLxh2Wv7+fH3JscYxqM/KnwQmiEB8ba+btWoKsCob/wE4M9E5Lsi8nT3s2Mppcu7F0uXARwdeHYgEHhbYdQ3/kdSSpdE5CiAb4jIj0e9QPeH4und8i2MMBAI3HGM9MZPKV3q/l8F8BXspsdeEZETAND9vzrg3GdSSk90NwTvzKgDgcBt4aZvfBGZAZBJKW13y78A4H8G8DUAnwXwW93/Xx3pit3F78k2iuTyWSxZvZg17yaREaLtospIb+szxRF/+zS52HrXYTaPNRpWd+efSWMGdKahSln10R23TzA7p2QTeeeLy9ferum1my4ajQlBp1wK7SxFL1Zp/OubVsdfJILKj370k6bu3FnNkff6GS3n83bfZJrSiNf9fBsdWn/w8zl7b4+eONUr+9x5jbZOeJsm3+cj4Ig/aQxLnc4pre19L5D7tO9/YPScgxi927/k2BQ36gvQmT4H7CcAQLrh9tse2MRgFFH/GICvdN/WOQD/PKX0b0TkOwD+WESeAnAOwK+OdslAIHDQuOnCTymdAfD+PT5fA/DJ/jMCgcDbHeNPoXVD1Hdi7tS0it9TjjddSBxvkwmv07LiJZto2s7Ul8sxV59+7j3fOIqv4KLnOBowZzj8rNjIpBftlhUbqxQ9lstbjrkSfe8O9Z+cSrNZ1j58ZCDnApinFFcVl677wopy4s+4yLoqqUVtMiWKWPPjdeLcyxWnTd32jl5vZVUjDeHmm9NmbznPwOsUeZinSMm8M9mVijoHnlTEmu3YZOw490iK9qQlbFbrE+A55RXnaxjClzfMXGiu6642yhZZGpG9I3z1A4EJRCz8QGACEQs/EJhAjFfHF+mRVOYcUSYz8Igz9U2Ri63hrJ+yemWOdO3Usfoo56JrGSJEqxNxaunUp2MNyu/niEOpWaFo9wny9D3rFRuNttNS99Vshk1gphmKzPOes/o587xz3jvvQ8HzuOHy2a1Sumqeg3LFkm3OUFTcTsUy31QqaoKskTnyyNGTph3nCEjbtv/jpzSn31Xi408uq9xORcefzdjJ4v0Xa6bzurBQO2dGo7nzufMGmeb6c+dxef8RfX4cg88JHT8QCAxALPxAYAIxdnPeDdHIe+6xLNR26Y0MHzqh48QsJs5sOVNfp67HTOAhTuUoldSktu1SY6GgasAspXvacimumMyj3fHmPI6ss9+LJTn2LGu4MebJfNXy5ivyGmSPv9lFG0NVvqoe1px6HLCic53SfHvu9k5Hx7hK5kEAWLuqx236nhcvnDXt6jWt88Shs3NKFjJHnoY7W5bANE/3s1azagvLyyz2pz6RfRjJBXv8uYi5ASmvvDg/sglviDjPdbfr/h5v/EBgAhELPxCYQIxd1L/hFcWpsACbDstLMcyXl6ctbqcsGBHNqwfcZ4dSbfk0WRnyEEvO+48z6bKolXOeZCxGNhtWFOeMqjmXNZV54Fg0nJ623nlF4+1mU2hNER9dlghM1lYvmXa8Q18sWk/JLHk58nxzoBMArFzRPr1q1WhQ6iqa45Ljy2tQny0XFLW2quoCW0o6ncFibl9aNbp2q0W8js7bknf8h4npfSI2Ne0My7VlvPpcFX/Au/IZ32zwOG4cj6oAxBs/EJhAxMIPBCYQsfADgQnEWHX8jGQw3c1v50ko2VOtX/9i8x6ZNHwKMqrz/VcpxTN7DXrCjkKGo+6sxlQl81g+q2PMu3Y5qmu4QWaNJ5kdf472F2ZmrF7PuHRRdd9cwe6VLC6qTlul6MKdbRv5xtwhtYLVrYsUOVmgfYjZGWv2g2i7esN7L+r8b9Mcl7dtlCDrt335FLI6r2tXV3rlRUoTDgANMtX6FOuJ9jlYR261fCrswaa4Qe0Aq2vz87Iv3nuTE4+vNfzag8YxCuKNHwhMIGLhBwITiDEH6QCdASJJk0SvbF9wDP8+EYeaS5NlQjCcHO3F9t45Xt6ma2ddeq1Mm01xKmKXnLmtwmQbOatKNCltVs6RkfgAkxvwBBVMtsCmyd0xax/MRV9zprJZUiWWlo+YuulpDZxpUzDP9KwlDhlmArtyRck3mtfV5Oi9/4w3nbtFGVKZmsRBuLVl+QOnKW12pWz5CfnZadNc+edhFFPZXrDedKbG9a9l/8wlSXu260uTzaf5IXW/5qgaRrzxA4EJRCz8QGACEQs/EJhAjJ9ss6vTDXZo7I9oYxMY8+p7wsQimbaazoW0UNTIPZNK2efOS4PJKzqsW9OOQqlkCUHy5FKbKvabsnsvu8buXptILygysOlMT8Wifk9PTMpfbYu49HOOzaNA5CZZl0Kb+UFzBW3Xdq6y7GK848x0NSLi4HY+NTjr8fmCHSPrwky42nD5AvNNnQ9PzlIn1+Es3U9Pspoz92IwJ/4wmMfFe/aaR9re9w6T4XO0n99+ygwZxz7zZscbPxCYQMTCDwQmEGMV9VNSMdunKS4RGYR4cwd5Xwnlp+7jPxeO8BtcVyDxNedMdixS5l1dk8VGStedcdF5BRL1C67OiKyOc5/H3CKZ3afyZhF4dt56sV1bUQ+3FhOauLmqVFUU91zsTE4yN6tmv3zBqhysSmxtWjKSMnkKmjwD7lpMupJzov6oXnE8j1MzLlfBzGG9FkdlNqzZLzXp2KkBmSFmun5z8I0xerdSMhe6Kv6exqTpGhrRP+PrbuSf33M4fRjpjS8iiyLyL0XkxyLyioj8nIgcEpFviMhr3f9LN+8pEAi8HTCqqP+/Afg3KaV3Yzed1isAPg/g2ZTSIwCe7R4HAoF3AEbJljsP4K8D+C8BIKXUANAQkc8A+Fi32ZcAPAfgc8N7S73giGzWXTrDorgNsBGSX5howYviLMr5HWIWk4w1wAWGcPBG23nF8e50lXjpvLpQIN6+YtGKlNUaqTheZKXdahYVM1krXvL8NBzHHKfo4gu0Gva71Gj8ydFJd+h7TtN3qTvxeI14+3zwDXtOMr+iJ8pg3sFm3ap/vOPP8PTrQhyE2bnjpm5qdlnHRHParFnvv0ZZv0trx3L6GbWgb6s97VEaDnEi/ECv0j5CkMHD2C8F3yhv/AcBXAXwT0Xk+yLyf3TTZR9LKV3eHVC6DODosE4CgcDbB6Ms/ByADwL4vZTSBwCUsQ+xXkSeFpEXROSFQRshgUBgvBhl4V8AcCGl9Hz3+F9i94dgRUROAED3/+peJ6eUnkkpPZFSesLH2QcCgYPBTXX8lNIVETkvIu9KKf0EwCcBvNz9+yyA3+r+/+ooFxTZe/G3ydOu5upYJyokIspw+wTG+0qsHtgmMogsjcHvNWRJr9zesrz6bHbhqDVP6sBWuozzimsn0q2dUsgSEat9nlyCo/o2KFV1t9c9x+vNS6zj12s2cq9jPPJ0DnzacCYx8enATRQl7anIEO8zr9MyU4mJgss6otYpNSgV506Yupkl1fk53Vij5jwNt9XsVyucN3WNLSUVbVes2bLT2Tvnw1CV25uaB1TJkAhCv6EwclquLka14/+3AP5AdhOknwHwX2FXWvhjEXkKwDkAv7qvKwcCgQPDSAs/pfQDAE/sUfXJOzucQCAwDow9SKdnh+gzYTC5hA2wYVkoEQEGp8ICXOZS7zhFoj+TVSRnsmsQ97o35zEnHHsQtp2UW6BUXo2cNU0yr17LEYnwFgh76+VcDgLDHecCVkYFB0L1e0pqmdUPn67LekcOEzUHB54Y77whBBjs8YiCzRCcm5rrlYvT1o9sak5JRvJkmmw0bLtcST0gswUb6MNemtXMWVOXdjSzsPf4Y8iQ+WHPSdaEM0PIPLwJdmRbYq/vQCAwcYiFHwhMIGLhBwITiDFH5yV0ukqeV0mM26Iz+bHux9ag1PY6lX6dfNHqxU0im+y0SY/P2D6alC+v6fTnjGj/haLq7s7ahlaLxuvMhTZVs+dop3Z0nnddbZBrqyeoNPshw0w8gy1DI2O/JqS9wCYrcX4exrTFBCYlG4GXK6mOn/X6f0ndeQtF1d2zedsuiZoqvck5Q7kWkpvwKpnz2mVy9e1jmuEbM9iRzQYC+vnoDGi4/3sRb/xAYAIRCz8QmEDInRDXRr6YyFUAbwFYBnDtJs3vNt4OYwBiHB4xDov9juP+lNKRmzUa68LvXVTkhZTSXg5BEzWGGEeM46DGEaJ+IDCBiIUfCEwgDmrhP3NA12W8HcYAxDg8YhwWd2UcB6LjBwKBg0WI+oHABGKsC19EPi0iPxGR10VkbKy8IvL7IrIqIi/SZ2OnBxeRe0Xkm12K8pdE5DcOYiwiUhKRb4vID7vj+Efdzx8Qkee74/ijLv/CXYeIZLt8jl8/qHGIyFkR+UsR+YGIvND97CCekbFQ2Y9t4ctuRot/AuBvAHgPgF8TkfeM6fL/DMCn3WcHQQ/eAvCbKaVHATwJ4Ne7czDusdQBfCKl9H4AjwP4tIg8CeC3AfxOdxzrAJ66y+O4gd/ALmX7DRzUOD6eUnqczGcH8YyMh8o+pTSWPwA/B+Df0vEXAHxhjNc/DeBFOv4JgBPd8gkAPxnXWGgMXwXwqYMcC4BpAN8D8LPYdRTJ7XW/7uL1T3Uf5k8A+Dp2vdAPYhxnASy7z8Z6XwDMA3gT3b23uzmOcYr69wBgMrML3c8OCgdKDy4ipwF8AMDzBzGWrnj9A+ySpH4DwBsANlLqsYOM6/78LoB/APTSDx8+oHEkAH8mIt8Vkae7n437voyNyn6cC38v/sGJNCmIyCyAPwHw91NKWzdrfzeQUmqnlB7H7hv3QwAe3avZ3RyDiPwSgNWU0nf543GPo4uPpJQ+iF1V9NdF5K+P4Zoet0Vlvx+Mc+FfAHAvHZ8CcGlA23FgJHrwOw0RyWN30f9BSulfHeRYACCltIHdLEhPAlgU6cUej+P+fATAL4vIWQBfxq64/7sHMA6klC51/68C+Ap2fwzHfV9ui8p+Pxjnwv8OgEe6O7YFAH8LwNfGeH2Pr2GXFhzYBz347UB2SeS+COCVlNI/PqixiMgREVnslqcA/Dx2N5G+CeBXxjWOlNIXUkqnUkqnsfs8/D8ppb8z7nGIyIyIzN0oA/gFAC9izPclpXQFwHkReVf3oxtU9nd+HHd708RtUvwigFexq0/+j2O87h8CuAygid1f1aewq0s+C+C17v9DYxjHR7Ertv4IwA+6f7847rEA+GkA3++O40UA/1P38wcBfBvA6wD+BYDiGO/RxwB8/SDG0b3eD7t/L914Ng/oGXkcwAvde/N/A1i6G+MIz71AYAIRnnuBwAQiFn4gMIGIhR8ITCBi4QcCE4hY+IHABCIWfiAwgYiFHwhMIGLhBwITiP8fjgdmcWXGD6AAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Example of a picture\n",
    "index = 25\n",
    "plt.imshow(train_set_x_orig[index])\n",
    "print (\"y = \" + str(train_set_y[:, index]) + \", it's a '\" + classes[np.squeeze(train_set_y[:, index])].decode(\"utf-8\") +  \"' picture.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Many software bugs in deep learning come from having matrix/vector dimensions that don't fit. If you can keep your matrix/vector dimensions straight you will go a long way toward eliminating many bugs. \n",
    "\n",
    "## 2-2 Preprocessing <a name=\"2.2\"></a>\n",
    "\n",
    "Common steps for pre-processing a new dataset are:\n",
    "- Figure out the dimensions and shapes of the problem (m_train, m_test, num_px, ...)\n",
    "- Reshape the datasets such that each example is now a vector of size (num_px \\* num_px \\* 3, 1)\n",
    "- \"Standardize\" the data\n",
    "\n",
    "### A. Confirm the shapes <a name=\"A\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of training examples: m_train = 209\n",
      "Number of testing examples: m_test = 50\n",
      "Height/Width of each image: num_px = 64\n",
      "Width of each image: num_px = 64\n",
      "Each image is of size: (64, 64, 3)\n",
      "train_set_x shape: (209, 64, 64, 3)\n",
      "train_set_y shape: (1, 209)\n",
      "test_set_x shape: (50, 64, 64, 3)\n",
      "test_set_y shape: (1, 50)\n"
     ]
    }
   ],
   "source": [
    "m_train = train_set_x_orig.shape[0]\n",
    "m_test = test_set_x_orig.shape[0]\n",
    "num_px = train_set_x_orig.shape[1]\n",
    "Width = train_set_x_orig.shape[2]\n",
    "print (\"Number of training examples: m_train = \" + str(m_train))\n",
    "print (\"Number of testing examples: m_test = \" + str(m_test))\n",
    "print (\"Height/Width of each image: num_px = \" + str(num_px))\n",
    "print (\"Width of each image: num_px = \" + str(Width))\n",
    "print (\"Each image is of size: (\" + str(num_px) + \", \" + str(num_px) + \", 3)\")\n",
    "print (\"train_set_x shape: \" + str(train_set_x_orig.shape))\n",
    "print (\"train_set_y shape: \" + str(train_set_y.shape))\n",
    "print (\"test_set_x shape: \" + str(test_set_x_orig.shape))\n",
    "print (\"test_set_y shape: \" + str(test_set_y.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### B. Flattening <a name=\"B\"></a>\n",
    "\n",
    "Reshape the training and test data sets so that images of size (num_px, num_px, 3) are flattened into single vectors of shape (num\\_px $*$ num\\_px $*$ 3, 1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train_set_x_flatten shape: (12288, 209)\n",
      "train_set_y shape: (1, 209)\n",
      "test_set_x_flatten shape: (12288, 50)\n",
      "test_set_y shape: (1, 50)\n",
      "sanity check after reshaping: [17 71 49 38 70]\n"
     ]
    }
   ],
   "source": [
    "# Reshape the training and test examples\n",
    "\n",
    "train_set_x_flatten = train_set_x_orig.reshape(num_px*num_px*3, m_train)\n",
    "test_set_x_flatten =test_set_x_orig.reshape(num_px*num_px*3, m_test)\n",
    "\n",
    "print (\"train_set_x_flatten shape: \" + str(train_set_x_flatten.shape))\n",
    "print (\"train_set_y shape: \" + str(train_set_y.shape))\n",
    "print (\"test_set_x_flatten shape: \" + str(test_set_x_flatten.shape))\n",
    "print (\"test_set_y shape: \" + str(test_set_y.shape))\n",
    "print (\"sanity check after reshaping: \" + str(train_set_x_flatten[0:5,0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### C. Normalize <a name=\"C\"></a>\n",
    "\n",
    "\n",
    "To represent color images, the red, green and blue channels (RGB) must be specified for each pixel, and so the pixel value is actually a vector of three numbers ranging from 0 to 255.\n",
    "\n",
    "One common preprocessing step in machine learning is to center and standardize your dataset, meaning that you substract the mean of the whole numpy array from each example, and then divide each example by the standard deviation of the whole numpy array. But for picture datasets, it is simpler and more convenient and works almost as well to just divide every row of the dataset by 255 (the maximum value of a pixel channel).\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train_set_x = train_set_x_flatten/255.\n",
    "test_set_x = test_set_x_flatten/255."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3 - General Architecture of the learning algorithm  <a name=\"3\"></a>\n",
    "\n",
    "To design a simple algorithm to distinguish cat images from non-cat images, we will build a Logistic Regression, using a Neural Network mindset. The following Figure explains why **Logistic Regression is actually a very simple Neural Network!**\n",
    "\n",
    "![](images/LogReg_kiank.png)\n",
    "\n",
    "**Mathematical expression of the algorithm**:\n",
    "\n",
    "For one example $x^{(i)}$:\n",
    "$$z^{(i)} = w^T x^{(i)} + b \\tag{1}$$\n",
    "$$\\hat{y}^{(i)} = a^{(i)} = sigmoid(z^{(i)})\\tag{2}$$ \n",
    "$$ \\mathcal{L}(a^{(i)}, y^{(i)}) =  - y^{(i)}  \\log(a^{(i)}) - (1-y^{(i)} )  \\log(1-a^{(i)})\\tag{3}$$\n",
    "\n",
    "The cost is then computed by summing over all training examples:\n",
    "$$ J = \\frac{1}{m} \\sum_{i=1}^m \\mathcal{L}(a^{(i)}, y^{(i)})\\tag{6}$$\n",
    "\n",
    "**Key steps**:\n",
    " \n",
    "    - Initialize the parameters of the model\n",
    "    - Learn the parameters for the model by minimizing the cost  \n",
    "    - Use the learned parameters to make predictions (on the test set)\n",
    "    - Analyse the results and conclude"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4 - Building the parts of our algorithm   <a name=\"4\"></a>\n",
    "\n",
    "The main steps for building a Neural Network are:\n",
    "1. Define the model structure (such as number of input features) \n",
    "2. Initialize the model's parameters\n",
    "3. Loop:\n",
    "    - Calculate current loss (forward propagation)\n",
    "    - Calculate current gradient (backward propagation)\n",
    "    - Update parameters (gradient descent)\n",
    "\n",
    "You often build 1-3 separately and integrate them into one function we call `model()`.\n",
    "\n",
    "Compute $sigmoid( w^T x + b) = \\frac{1}{1 + e^{-(w^T x + b)}}$ to make predictions. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.1 - Sigmoid function <a name=\"4.1\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def sigmoid(z):\n",
    "    \"\"\"\n",
    "    Compute the sigmoid of z\n",
    "\n",
    "    Arguments:\n",
    "    z -- A scalar or numpy array of any size.\n",
    "\n",
    "    Return:\n",
    "    s -- sigmoid(z)\n",
    "    \"\"\"\n",
    "\n",
    "    s = 1/(1+np.exp(-z))\n",
    "    \n",
    "    return s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Example**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sigmoid([0, 2]) = [0.5        0.88079708]\n"
     ]
    }
   ],
   "source": [
    "print (\"sigmoid([0, 2]) = \" + str(sigmoid(np.array([0,2]))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.2 - Initializing parameters <a name=\"4.2\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def initialize_with_zeros(dim):\n",
    "    \"\"\"\n",
    "    This function creates a vector of zeros of shape (dim, 1) for w and initializes b to 0.\n",
    "    \n",
    "    Argument:\n",
    "    dim -- size of the w vector we want (or number of parameters in this case)\n",
    "    \n",
    "    Returns:\n",
    "    w -- initialized vector of shape (dim, 1)\n",
    "    b -- initialized scalar (corresponds to the bias)\n",
    "    \"\"\"\n",
    "\n",
    "    w = np.zeros((dim,1))\n",
    "    b = 0\n",
    " \n",
    "    assert(w.shape == (dim, 1))\n",
    "    assert(isinstance(b, float) or isinstance(b, int))\n",
    "    \n",
    "    return w, b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Example**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "w = [[0.]\n",
      " [0.]]\n",
      "b = 0\n"
     ]
    }
   ],
   "source": [
    "dim = 2\n",
    "w, b = initialize_with_zeros(dim)\n",
    "print (\"w = \" + str(w))\n",
    "print (\"b = \" + str(b))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.3 - Forward and Backward propagation <a name=\"4.3\"></a>\n",
    "\n",
    "Now that your parameters are initialized, you can do the \"forward\" and \"backward\" propagation steps for learning the parameters.\n",
    "\n",
    "Next step is to implement a function `propagate()` that computes the cost function and its gradient.\n",
    "\n",
    "\n",
    "Forward Propagation:\n",
    "- You get X\n",
    "- You compute $A = \\sigma(w^T X + b) = (a^{(1)}, a^{(2)}, ..., a^{(m-1)}, a^{(m)})$\n",
    "- You calculate the cost function: $J = -\\frac{1}{m}\\sum_{i=1}^{m}y^{(i)}\\log(a^{(i)})+(1-y^{(i)})\\log(1-a^{(i)})$\n",
    "\n",
    "Here are the two formulas for gradient: \n",
    "\n",
    "$$ \\frac{\\partial J}{\\partial w} = \\frac{1}{m}X(A-Y)^T\\tag{7}$$\n",
    "\n",
    "$$ \\frac{\\partial J}{\\partial b} = \\frac{1}{m} \\sum_{i=1}^m (a^{(i)}-y^{(i)})\\tag{8}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def propagate(w, b, X, Y):\n",
    "    \"\"\"\n",
    "    Implement the cost function and its gradient for the propagation explained above\n",
    "\n",
    "    Arguments:\n",
    "    w -- weights, a numpy array of size (num_px * num_px * 3, 1)\n",
    "    b -- bias, a scalar\n",
    "    X -- data of size (num_px * num_px * 3, number of examples)\n",
    "    Y -- true \"label\" vector (containing 0 if non-cat, 1 if cat) of size (1, number of examples)\n",
    "\n",
    "    Return:\n",
    "    cost -- negative log-likelihood cost for logistic regression\n",
    "    dw -- gradient of the loss with respect to w, thus same shape as w\n",
    "    db -- gradient of the loss with respect to b, thus same shape as b\n",
    "\n",
    "    \"\"\"\n",
    "    \n",
    "    m = X.shape[1]\n",
    "    \n",
    "    # FORWARD PROPAGATION (FROM X TO COST)\n",
    "    A = sigmoid(np.dot(w.T,X)+b)                # compute activation\n",
    "    cost = -(1/m)*(np.dot(Y, np.log(A).T)+np.dot((1-Y),np.log(1-A).T))                             # compute cost\n",
    "    \n",
    "    # BACKWARD PROPAGATION (TO FIND GRAD)\n",
    "    dw = 1/m * (np.dot(X,(A-Y).T))\n",
    "    db =  1/m * (np.sum((A-Y)))\n",
    "\n",
    "\n",
    "    assert(dw.shape == w.shape)\n",
    "    assert(db.dtype == float)\n",
    "    cost = np.squeeze(cost)\n",
    "    assert(cost.shape == ())\n",
    "    \n",
    "    grads = {\"dw\": dw,\n",
    "             \"db\": db}\n",
    "    \n",
    "    return grads, cost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Example**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dw = [[0.99845601]\n",
      " [2.39507239]]\n",
      "db = 0.001455578136784208\n",
      "cost = 5.801545319394553\n"
     ]
    }
   ],
   "source": [
    "w, b, X, Y = np.array([[1.],[2.]]), 2., np.array([[1.,2.,-1.],[3.,4.,-3.2]]), np.array([[1,0,1]])\n",
    "grads, cost = propagate(w, b, X, Y)\n",
    "print (\"dw = \" + str(grads[\"dw\"]))\n",
    "print (\"db = \" + str(grads[\"db\"]))\n",
    "print (\"cost = \" + str(cost))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.4 - Optimization <a name=\"4.4\"></a>\n",
    "\n",
    "\n",
    "Next is to implement an optimization function. The goal is to learn $w$ and $b$ by minimizing the cost function $J$. \n",
    "\n",
    "For a parameter $\\theta$, the update rule is $ \\theta = \\theta - \\alpha \\text{ } d\\theta$, where $\\alpha$ is the learning rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def optimize(w, b, X, Y, num_iterations, learning_rate, print_cost = False):\n",
    "    \"\"\"\n",
    "    This function optimizes w and b by running a gradient descent algorithm\n",
    "    \n",
    "    Arguments:\n",
    "    w -- weights, a numpy array of size (num_px * num_px * 3, 1)\n",
    "    b -- bias, a scalar\n",
    "    X -- data of shape (num_px * num_px * 3, number of examples)\n",
    "    Y -- true \"label\" vector (containing 0 if non-cat, 1 if cat), of shape (1, number of examples)\n",
    "    num_iterations -- number of iterations of the optimization loop\n",
    "    learning_rate -- learning rate of the gradient descent update rule\n",
    "    print_cost -- True to print the loss every 100 steps\n",
    "    \n",
    "    Returns:\n",
    "    params -- dictionary containing the weights w and bias b\n",
    "    grads -- dictionary containing the gradients of the weights and bias with respect to the cost function\n",
    "    costs -- list of all the costs computed during the optimization, this will be used to plot the learning curve.\n",
    "    \n",
    "    \"\"\"\n",
    "    \n",
    "    costs = []\n",
    "    \n",
    "    for i in range(num_iterations):\n",
    "        \n",
    "        \n",
    "        # Cost and gradient calculation \n",
    "        grads, cost =propagate(w, b, X, Y)\n",
    "        \n",
    "        # Retrieve derivatives from grads\n",
    "        dw = grads[\"dw\"]\n",
    "        db = grads[\"db\"]\n",
    "        \n",
    "        # update rule \n",
    "        w = w-learning_rate*dw\n",
    "        b = b-learning_rate*db\n",
    "        \n",
    "        # Record the costs\n",
    "        if i % 100 == 0:\n",
    "            costs.append(cost)\n",
    "        \n",
    "        # Print the cost every 100 training iterations\n",
    "        if print_cost and i % 100 == 0:\n",
    "            print (\"Cost after iteration %i: %f\" %(i, cost))\n",
    "    \n",
    "    params = {\"w\": w,\n",
    "              \"b\": b}\n",
    "    \n",
    "    grads = {\"dw\": dw,\n",
    "             \"db\": db}\n",
    "    \n",
    "    return params, grads, costs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Example**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "w = [[0.19033591]\n",
      " [0.12259159]]\n",
      "b = 1.9253598300845747\n",
      "dw = [[0.67752042]\n",
      " [1.41625495]]\n",
      "db = 0.21919450454067652\n"
     ]
    }
   ],
   "source": [
    "params, grads, costs = optimize(w, b, X, Y, num_iterations= 100, learning_rate = 0.009, print_cost = False)\n",
    "\n",
    "print (\"w = \" + str(params[\"w\"]))\n",
    "print (\"b = \" + str(params[\"b\"]))\n",
    "print (\"dw = \" + str(grads[\"dw\"]))\n",
    "print (\"db = \" + str(grads[\"db\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.5 - Prediction <a name=\"4.5\"></a>\n",
    "There are two steps to computing predictions:\n",
    "\n",
    "1. Calculate $\\hat{Y} = A = \\sigma(w^T X + b)$\n",
    "\n",
    "2. Convert the entries of a into 0 (if activation <= 0.5) or 1 (if activation > 0.5), stores the predictions in a vector `Y_prediction`. I am going to use `numpy.where(condition[, x, y])` instead of for loop for more efficiency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def predict(w, b, X):\n",
    "    '''\n",
    "    Predict whether the label is 0 or 1 using learned logistic regression parameters (w, b)\n",
    "    \n",
    "    Arguments:\n",
    "    w -- weights, a numpy array of size (num_px * num_px * 3, 1)\n",
    "    b -- bias, a scalar\n",
    "    X -- data of size (num_px * num_px * 3, number of examples)\n",
    "    \n",
    "    Returns:\n",
    "    Y_prediction -- a numpy array (vector) containing all predictions (0/1) for the examples in X\n",
    "    '''\n",
    "    \n",
    "    m = X.shape[1]\n",
    "    Y_prediction = np.zeros((1,m))\n",
    "    w = w.reshape(X.shape[0], 1)\n",
    "    \n",
    "    # Compute vector \"A\" predicting the probabilities of a cat being present in the picture\n",
    "\n",
    "    A = sigmoid(np.dot(w.T,X)+b)\n",
    "\n",
    "    #vectorization approach - see np.where for more details\n",
    "    Y_prediction=np.where(A<=0.5, A,1)                         # if true, return A, else 1.\n",
    "    Y_prediction=np.where(Y_prediction<0.5, 0, Y_prediction)   # if true, return 0, else Y_prediction.\n",
    "    \n",
    "    assert(Y_prediction.shape == (1, m))\n",
    "    \n",
    "    return Y_prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "predictions = [[1. 1. 0.]]\n"
     ]
    }
   ],
   "source": [
    "w = np.array([[0.1124579],[0.23106775]])\n",
    "b = -0.3\n",
    "X = np.array([[1.,-1.1,-3.2],[1.2,2.,0.1]])\n",
    "print (\"predictions = \" + str(predict(w, b, X)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.6 - Merge all functions into a model <a name=\"4.6\"></a>\n",
    "\n",
    "Last step is to implement the model function. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def model(X_train, Y_train, X_test, Y_test, num_iterations = 2000, learning_rate = 0.5, print_cost = False):\n",
    "    \"\"\"\n",
    "    Builds the logistic regression model by calling the function from above\n",
    "    \n",
    "    Arguments:\n",
    "    X_train -- training set represented by a numpy array of shape (num_px * num_px * 3, m_train)\n",
    "    Y_train -- training labels represented by a numpy array (vector) of shape (1, m_train)\n",
    "    X_test -- test set represented by a numpy array of shape (num_px * num_px * 3, m_test)\n",
    "    Y_test -- test labels represented by a numpy array (vector) of shape (1, m_test)\n",
    "    num_iterations -- hyperparameter representing the number of iterations to optimize the parameters\n",
    "    learning_rate -- hyperparameter representing the learning rate used in the update rule of optimize()\n",
    "    print_cost -- Set to true to print the cost every 100 iterations\n",
    "    \n",
    "    Returns:\n",
    "    d -- dictionary containing information about the model.\n",
    "    \"\"\"\n",
    "    \n",
    "    # initialize parameters with zeros \n",
    "    w, b = initialize_with_zeros(X_train.shape[0])\n",
    "\n",
    "    # Gradient descent \n",
    "    parameters, grads, costs = optimize(w, b, X_train, Y_train, num_iterations, learning_rate, print_cost)\n",
    "    \n",
    "    # Retrieve parameters w and b from dictionary \"parameters\"\n",
    "    w = parameters[\"w\"]\n",
    "    b = parameters[\"b\"]\n",
    "    \n",
    "    # Predict test/train set examples\n",
    "    Y_prediction_test = predict(w, b, X_test)\n",
    "    Y_prediction_train = predict(w, b, X_train)\n",
    "\n",
    "\n",
    "    # Print train/test Errors\n",
    "    print(\"train accuracy: {} %\".format(100 - np.mean(np.abs(Y_prediction_train - Y_train)) * 100))\n",
    "    print(\"test accuracy: {} %\".format(100 - np.mean(np.abs(Y_prediction_test - Y_test)) * 100))\n",
    "\n",
    "    \n",
    "    d = {\"costs\": costs,\n",
    "         \"Y_prediction_test\": Y_prediction_test, \n",
    "         \"Y_prediction_train\" : Y_prediction_train, \n",
    "         \"w\" : w, \n",
    "         \"b\" : b,\n",
    "         \"learning_rate\" : learning_rate,\n",
    "         \"num_iterations\": num_iterations}\n",
    "    \n",
    "    return d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5 - Train a model <a name=\"5\"></a>\n",
    "\n",
    "Run the following cell to train your model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cost after iteration 0: 0.693147\n",
      "Cost after iteration 100: 0.709726\n",
      "Cost after iteration 200: 0.657712\n",
      "Cost after iteration 300: 0.614611\n",
      "Cost after iteration 400: 0.578001\n",
      "Cost after iteration 500: 0.546372\n",
      "Cost after iteration 600: 0.518331\n",
      "Cost after iteration 700: 0.492852\n",
      "Cost after iteration 800: 0.469259\n",
      "Cost after iteration 900: 0.447139\n",
      "Cost after iteration 1000: 0.426262\n",
      "Cost after iteration 1100: 0.406617\n",
      "Cost after iteration 1200: 0.388723\n",
      "Cost after iteration 1300: 0.374678\n",
      "Cost after iteration 1400: 0.365826\n",
      "Cost after iteration 1500: 0.358532\n",
      "Cost after iteration 1600: 0.351612\n",
      "Cost after iteration 1700: 0.345012\n",
      "Cost after iteration 1800: 0.338704\n",
      "Cost after iteration 1900: 0.332664\n",
      "train accuracy: 91.38755980861244 %\n",
      "test accuracy: 34.0 %\n"
     ]
    }
   ],
   "source": [
    "d = model(train_set_x, train_set_y, test_set_x, test_set_y, num_iterations = 2000, learning_rate = 0.005, print_cost = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Example of a picture that was wrongly classified.\n",
    "index = 1\n",
    "plt.imshow(test_set_x[:,index].reshape((num_px, num_px, 3)))\n",
    "print (\"y = \" + str(test_set_y[0,index]) + \", you predicted that it is a \\\"\" + classes[d[\"Y_prediction_test\"][0,index]].decode(\"utf-8\") +  \"\\\" picture.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also plot the cost function and the gradients."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Plot learning curve (with costs)\n",
    "costs = np.squeeze(d['costs'])\n",
    "plt.plot(costs)\n",
    "plt.ylabel('cost')\n",
    "plt.xlabel('iterations (per hundreds)')\n",
    "plt.title(\"Learning rate =\" + str(d[\"learning_rate\"]))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6 - Further analysis  <a name=\"6\"></a>\n",
    "\n",
    "Let's analyze it further, and examine possible choices for the learning rate $\\alpha$. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Choice of learning rate ####\n",
    "\n",
    "**Reminder**:\n",
    "In order for Gradient Descent to work you must choose the learning rate wisely. The learning rate $\\alpha$  determines how rapidly we update the parameters. If the learning rate is too large we may \"overshoot\" the optimal value. Similarly, if it is too small we will need too many iterations to converge to the best values. That's why it is crucial to use a well-tuned learning rate.\n",
    "\n",
    "Let's compare the learning curve of our model with several choices of learning rates. Run the cell below. This should take about 1 minute. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "learning_rates = [0.01, 0.001, 0.0001]\n",
    "models = {}\n",
    "for i in learning_rates:\n",
    "    print (\"learning rate is: \" + str(i))\n",
    "    models[str(i)] = model(train_set_x, train_set_y, test_set_x, test_set_y, num_iterations = 1500, learning_rate = i, print_cost = False)\n",
    "    print ('\\n' + \"-------------------------------------------------------\" + '\\n')\n",
    "\n",
    "for i in learning_rates:\n",
    "    plt.plot(np.squeeze(models[str(i)][\"costs\"]), label= str(models[str(i)][\"learning_rate\"]))\n",
    "\n",
    "plt.ylabel('cost')\n",
    "plt.xlabel('iterations (hundreds)')\n",
    "\n",
    "legend = plt.legend(loc='upper center', shadow=True)\n",
    "frame = legend.get_frame()\n",
    "frame.set_facecolor('0.90')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Interpretation**: \n",
    "- Different learning rates give different costs and thus different predictions results.\n",
    "- If the learning rate is too large (0.01), the cost may oscillate up and down. It may even diverge (though in this example, using 0.01 still eventually ends up at a good value for the cost). \n",
    "- A lower cost doesn't mean a better model. You have to check if there is possibly overfitting. It happens when the training accuracy is a lot higher than the test accuracy.\n",
    "- In deep learning, we usually recommend that you: \n",
    "    - Choose the learning rate that better minimizes the cost function.\n",
    "    - If your model overfits, use other techniques to reduce overfitting. (We'll talk about this in later videos.) \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7 - Test with your own image <a name=\"7\"></a>\n",
    "Now you can use your own image and see the output of your model. To do that:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "## START CODE HERE ## (PUT YOUR IMAGE NAME) \n",
    "my_image = \"my_image.jpg\"   # change this to the name of your image file \n",
    "## END CODE HERE ##\n",
    "\n",
    "# We preprocess the image to fit your algorithm.\n",
    "fname = \"images/\" + my_image\n",
    "image = np.array(ndimage.imread(fname, flatten=False))\n",
    "my_image = scipy.misc.imresize(image, size=(num_px,num_px)).reshape((1, num_px*num_px*3)).T\n",
    "my_predicted_image = predict(d[\"w\"], d[\"b\"], my_image)\n",
    "\n",
    "plt.imshow(image)\n",
    "print(\"y = \" + str(np.squeeze(my_predicted_image)) + \", your algorithm predicts a \\\"\" + classes[int(np.squeeze(my_predicted_image)),].decode(\"utf-8\") +  \"\\\" picture.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 8- Summary <a name=\"8\"></a>\n",
    "\n",
    "1. Preprocessing the dataset is important.\n",
    "2. You implemented each function separately: initialize(), propagate(), optimize(). Then you built a model().\n",
    "3. Tuning the learning rate (which is an example of a \"hyperparameter\") can make a big difference to the algorithm. You will see more examples of this later in this course!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bibliography:\n",
    "- http://www.wildml.com/2015/09/implementing-a-neural-network-from-scratch/\n",
    "- https://stats.stackexchange.com/questions/211436/why-do-we-normalize-images-by-subtracting-the-datasets-image-mean-and-not-the-c"
   ]
  }
 ],
 "metadata": {
  "coursera": {
   "course_slug": "neural-networks-deep-learning",
   "graded_item_id": "XaIWT",
   "launcher_item_id": "zAgPl"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}