Initialization, Regularization, Optimization, Mini-Batch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Networks with Numpy: Initialization, Regularization, Optimization Methods and Minibatch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal of this exercise is to show different initialization methods like he, xavier; regularization methods such as l2 regularization and drop out; optimization methods such as gradient descent, momentum, rmsprop and adam together with minibatch. This exercise will be very simple. Using MNIST, I will be predicting weather the image contains the number 1 or not.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import math\n",
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "MNIST = pd.read_csv('G:\\\\MNIST\\\\MNIST_train.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(42000, 785)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "MNIST.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Right now we can see that there are 10 classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    4684\n",
       "7    4401\n",
       "3    4351\n",
       "9    4188\n",
       "2    4177\n",
       "6    4137\n",
       "0    4132\n",
       "4    4072\n",
       "8    4063\n",
       "5    3795\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "MNIST.label.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### I will convert all the digits beside the number one into the number zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def convert_to_zero(data):\n",
    "    if data['label'] != 1:\n",
    "        label = 0\n",
    "    else:\n",
    "        label = 1\n",
    "    return label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "MNIST['label'] = MNIST.apply(convert_to_zero,axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def ones_only(data):\n",
    "    if data['label'] == 1:\n",
    "        label1 = 1\n",
    "    else:\n",
    "        label1 = np.random.uniform()\n",
    "    return label1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "MNIST['label1'] = MNIST.apply(ones_only,axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "MNIST = MNIST.sort_values(by='label1',ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "MNIST = MNIST[:9368]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "MNIST = MNIST.drop(['label1'],axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Now we only have the classes one and zero"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    4684\n",
       "0    4684\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "MNIST.label.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9368, 785)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "MNIST.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### I will now divide the data into train and test sets. I will not make a dev set since this is not the focus of this exercise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "MNIST['permutation'] = np.random.permutation(9368)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    471\n",
       "0    465\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "MNIST[MNIST.permutation<936].label.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test = MNIST[MNIST.permutation<936]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train = MNIST[MNIST.permutation>936]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_x = train.drop('label', 1)\n",
    "train_x = train_x.drop('permutation', 1)\n",
    "train_x = train_x/255\n",
    "train_x = train_x.T\n",
    "\n",
    "train_y = train['label']\n",
    "train_y = train_y.values.reshape((1,train_y.shape[0]))\n",
    "\n",
    "test_x = test.drop('label', 1)\n",
    "test_x = test_x.drop('permutation', 1)\n",
    "test_x = test_x/255\n",
    "test_x = test_x.T\n",
    "test_y = test['label']\n",
    "test_y = test_y.values.reshape((1,test_y.shape[0]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>19215</th>\n",
       "      <th>19083</th>\n",
       "      <th>19094</th>\n",
       "      <th>19099</th>\n",
       "      <th>19106</th>\n",
       "      <th>3735</th>\n",
       "      <th>19117</th>\n",
       "      <th>19128</th>\n",
       "      <th>19133</th>\n",
       "      <th>...</th>\n",
       "      <th>41845</th>\n",
       "      <th>19957</th>\n",
       "      <th>4852</th>\n",
       "      <th>30128</th>\n",
       "      <th>38171</th>\n",
       "      <th>21353</th>\n",
       "      <th>1512</th>\n",
       "      <th>5495</th>\n",
       "      <th>25700</th>\n",
       "      <th>13190</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>pixel0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pixel1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pixel2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pixel3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pixel4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 8431 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        0      19215  19083  19094  19099  19106  3735   19117  19128  19133  \\\n",
       "pixel0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "pixel1    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "pixel2    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "pixel3    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "pixel4    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "\n",
       "        ...    41845  19957  4852   30128  38171  21353  1512   5495   25700  \\\n",
       "pixel0  ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "pixel1  ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "pixel2  ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "pixel3  ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "pixel4  ...      0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0   \n",
       "\n",
       "        13190  \n",
       "pixel0    0.0  \n",
       "pixel1    0.0  \n",
       "pixel2    0.0  \n",
       "pixel3    0.0  \n",
       "pixel4    0.0  \n",
       "\n",
       "[5 rows x 8431 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_x.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 1, 1, ..., 0, 0, 0]], dtype=int64)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sample of the Number 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.colorbar.Colorbar at 0x2158f4a9208>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAS4AAAD8CAYAAADJwUnTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFElJREFUeJzt3XusVeWZx/HvjwM6TjHWDgwhXAacMDNhplN0zqiJdgbo\nhUszQ5tOGeikXtIGNWovaRpJ/1ATM4kz1ba0YyVHJGJiddpqFQ1Tx1oV29F6wFDlMrSEQoVSLrXx\ngqEIPPPHXjibs89ee5+zL2u9h98n2Tl7rWetdz1nh/Pwvu+6bEUEZmYpGVV0AmZmQ+XCZWbJceEy\ns+S4cJlZcly4zCw5LlxmlhwXLjPrGEmrJR2QtLlOXJK+IWmHpJckXdBMuy5cZtZJ9wDzc+ILgBnZ\naxlwZzONunCZWcdExHrg1ZxNFgH3RsXzwLslTWzU7uh2JdiMcePGxbRp07p5SLPTyq5duzh06JBa\naUPSUG6n2QIcqVrui4i+Iew/CXilanlPtm5f3k4tFS5J84EVQA+wKiJuzdt+2rRpbNiwoZVDmlmO\n3t7ebh/ySER0/aDDHipK6gHuoDJGnQkslTSzXYmZWXEkNfVqg73AlKrlydm6XK3McV0I7IiInRFx\nFHiAynjVzBI3atSopl5tsBa4LDu7eDHwWkTkDhOhtaHiYGPTiwZuJGkZlbMFTJ06tYXDmVk3SGq6\nKB0/frxRW/cDs4FxkvYANwFjACJiJbAOWAjsAN4CrmzmuB2fnM8m6voAent7/QwdswS0aRhIRCxt\nEA/g2qG220rhGtbY1MzKr12Fq1NaGaT2AzMkTZd0BrCEynjVzBLXxcn5YRl2jysijkm6DnicyuUQ\nqyNiS9syM7PClL3H1dIcV0SsozK5ZmYjhCR6enqKTiNXV6+cN7M0jOgel5mNTC5cZpaUoifem+HC\nZWY1XLjMLDmenDezpHioaGZJcuEys+S4cJlZcly4zCw5LlxmlhTf8mNmSXKPy8yS48JlZknxdVxm\nliQXLjNLjifnzSwpHiqaWZJcuMwsOS5cZpacNn1Ldce4cJnZKTzHZVagxYsX141997vfzd33Rz/6\nUW58zpw5w8opFT6raGbJcY/LzJIiyXNcZpYe97jMLDkuXGaWFA8VzSxJPqtoZsnxUNGsQz7+8Y/n\nxh999NG6sUZDobL/4XbSiB8qStoFvAEcB45FRG87kjKzYpW9cLejrM6JiFkuWmYjx8nbfhq9mmxr\nvqTtknZIWj5I/BxJj0r6maQtkq5s1KaHimZ2inZ+y4+kHuAO4EPAHqBf0tqI2Fq12bXA1oj4B0nj\nge2S7ouIo/XabbXHFcAPJW2UtKxO4sskbZC04eDBgy0ezsy6YdSoUU29mnAhsCMidmaF6AFg0YBt\nAjhblS7cWOBV4Fheo632uC6NiL2S/hh4QtL/RsT6UzKK6AP6AHp7e6PF45lZFwxhjmucpA1Vy33Z\n3/xJk4BXqpb3ABcNaOM/gLXAr4GzgX+OiBN5B22pcEXE3uznAUnfp1Jd1+fvZWZlNsTH2hxqw/z2\nPGATMBf4UyqdoGcj4vV6Owx7qCjpXZLOPvke+DCwebjtmVl5tHGouBeYUrU8OVtX7UrgoajYAfwS\n+Iu8RlvpcU0Avp9V5tHAtyPiBy20Z3aKVatW5cbXrVuXGz9+/Hjd2DXXXJO77yWXXJIbH+naeDlE\nPzBD0nQqBWsJ8MkB2/wK+ADwrKQJwJ8DO/MaHXbhioidwPuGu7+ZlVM7zypGxDFJ1wGPAz3A6ojY\nIunqLL4SuAW4R9LLgIAbIuJQXru+HMLMarTzAtSIWAesG7BuZdX7X1OZamqaC5eZ1RjRt/yY2cjj\nL8swsyS5x2VmyXGPy6yO/v7+3PhnP/vZ3PjRo3VvZQPg4osvrhu7/fbbc/cdM2ZMbnwka+dZxU5x\n4TKzGu5xmVlyXLjMLCk+q2hmSXLhMrPk+HIIM0uOe1xmlpQR/y0/Zo28/nrdZ8HxhS98IXff3//+\n97nx8ePH58a/+c1v1o2deeaZufue7tzjMrPkuHCZWXJcuMwsKZ7jMrMkucdlZslx4TKz5LhwmVlS\nfK+ijXi7d+/OjS9ZsqRu7IUXXmjp2N/73vdy4xdccEFL7Z/OXLjMLDk+q2hmyXGPy8yS4jkuM0uS\nC5eZJceFy8yS48l5M0uK57gseU8//XRufO7cubnxvD+Ac889N3ffT3ziE7nx3t7e3LgNX9kLV8P+\noKTVkg5I2ly17j2SnpD0i+xn/r9AM0vKyV5Xo1dRmhnI3gPMH7BuOfBkRMwAnsyWzWyESL5wRcR6\n4NUBqxcBa7L3a4CPtjkvMytQ2QvXcOe4JkTEvuz9b4AJ9TaUtAxYBjB16tRhHs7MuiWFBwm2nF1E\nBBA58b6I6I2I3kZfbmBm5TBq1KimXoXlN8z99kuaCJD9PNC+lMysaO0cKkqaL2m7pB2SBp0PlzRb\n0iZJWyQ906jN4RautcDl2fvLgUeG2Y6ZlUyzRauZwiWpB7gDWADMBJZKmjlgm3cD3wL+MSL+Esi/\nDoYm5rgk3Q/MBsZJ2gPcBNwKfEfSp4HdwOKGv4GV0uHDh3Pjy5d37oTxFVdckRu/7bbbOnZsy9fG\nifcLgR0RsTNr9wEqJ/e2Vm3zSeChiPgVQEQ0HME1LFwRsbRO6AON9jWzNA1h/mqcpA1Vy30R0Ve1\nPAl4pWp5D3DRgDb+DBgj6WngbGBFRNybd1BfOW9mNYbQ4zoUEa3ewjAa+BsqnaGzgOckPR8RP8/b\nwczsHW2+RmsvMKVqeXK2rtoe4LcRcRg4LGk98D6gbuEq98UaZlaINp5V7AdmSJou6QxgCZWTe9Ue\nAS6VNFrSH1IZSm7La9Q9LjOr0a4eV0Qck3Qd8DjQA6yOiC2Srs7iKyNim6QfAC8BJ4BVEbG5fqsu\nXGY2iHbezhMR64B1A9atHLD8FeArzbbpwjXCHTlyJDf+wQ9+MDfe39/f0vHPOeecurHFi30VTRlJ\noqenp+g0crlwmVmNsj+Py4XLzGq4cJlZcly4zCwpRT9rqxkuXGZWw4XLzJJT9gcJunCZWQ33uKxQ\nb7/9dm78hRde6Ojx9+3bVzd25plndvTYNjwpPLrZhcvMarjHZWbJceEys6R4qGhmSXKPy8yS48Jl\nZslx4TKz5LhwWce99dZbdWMf+chHcvetfBH58M2bNy83XvbnOlkt36toZkkq+384LlxmVsM9LjNL\nioeKZpYkX4BqZslxj8vMkuPCZWZJ8b2K1hVf+tKX6sZ+8pOf5O7b6H/WBQsW5MYffvjh3Pjo0f4n\nlqKyF66G2UlaLemApM1V626WtFfSpuy1sLNpmlm3nDyr2MyrKM2U1XuA+YOs/1pEzMpe6waJm1mi\nyl64GvbjI2K9pGmdT8XMyqLsk/OtDGSvl/RSNpQ8t95GkpZJ2iBpw8GDB1s4nJl1gyR6enqaehVl\nuIXrTuA8YBawD7i93oYR0RcRvRHRO378+GEezsy6Kfmh4mAiYv/J95LuAh5rW0ZmVrgROVSUNLFq\n8WPA5nrbmll6ku9xSbofmA2Mk7QHuAmYLWkWEMAu4KoO5njay3veFsC2bduG3fYZZ5yRG7/lllty\n475Oa+QZERegRsTSQVbf3YFczKwkyj5U9H+XZlaj7D2ucmdnZl13cqjYzKvJ9uZL2i5ph6TlOdv9\nraRjkv6pUZsuXGZWo12T85J6gDuABcBMYKmkmXW2+zfgv5vJz4XLzGq08azihcCOiNgZEUeBB4BF\ng2x3PfAgcKCZRl24zKzGEArXuJN3xmSvZQOamgS8UrW8J1tXfaxJVC6rurPZ/Dw5XwKHDx/OjV95\n5ZW58WeeeaZu7Kyzzsrd97HH8q8dPv/883PjNjIN4azioYjobfFwXwduiIgTzR7XhcvMTnHyXsU2\n2QtMqVqenK2r1gs8cLIHByyUdCwi6j7szYXLzGq08TqufmCGpOlUCtYS4JPVG0TE9Krj3gM8lle0\nwIXLzAbRrsIVEcckXQc8DvQAqyNii6Srs/jK4bTrwmVmp2j3LT/Zg0bXDVg3aMGKiCuaadOFy8xq\n+JYfM0uOC5eZJceFyxp66qmncuMPPvjgsNueN29ebnz27NnDbttGpqKftdUMFy4zq+HCZWbJceEy\ns+S4cJlZcly4zCwpnpw3syS5cJlZcly4jGeffTY3ftlll7XU/sKFC+vG1qxZ01Lbdnpy4TKzpIyI\n71U0s9NP2Xtc5S6rZmaDcI/LzGqUvcflwmVmNVy4zCw5LlxmlpQRcVZR0hTgXmACEEBfRKyQ9B7g\nP4FpwC5gcUT8rnOplteRI0dy41dddVVu/LXXXmvp+DfeeGPd2NixY1tq205PZe9xNVNWjwFfjIiZ\nwMXAtZJmAsuBJyNiBvBktmxmI8AQvsm6EA0LV0Tsi4gXs/dvANuofIX2IuDkZdlrgI92Kkkz667k\nC1c1SdOA84GfAhMiYl8W+g2VoaSZWcc1PTkvaSzwIPD5iHi9utpGREiKOvstA5YBTJ06tbVszazj\niu5NNaOpHpekMVSK1n0R8VC2er+kiVl8InBgsH0joi8ieiOid/z48e3I2cw6bNSoUU29Csuv0Qaq\nlN67gW0R8dWq0Frg8uz95cAj7U/PzIpQ9jmuZoaKlwCfAl6WtClb92XgVuA7kj4N7AYWdybF8nvu\nuedy49u3b+/o8d98882Otm+nn7IPFRsWroj4MVDvt/hAe9Mxs6IV3ZtqRrkvjzUzG4Rv+TGzGmW/\n5afc2ZmZDcI9LjOrUfY5LhcuM6vhwmVmSUnhrKILVxuMHp3/MTaa6Dxx4kRuvKenJze+efPmurE5\nc+bk7mvWaZLmAyuAHmBVRNw6IP4vwA1ULrt6A7gmIn6W16YLl5nVaNdZRUk9wB3Ah4A9QL+ktRGx\ntWqzXwJ/HxG/k7QA6AMuys2vLdmZmQ3uQmBHROyMiKPAA1QeifWOiPifqoeQPg9MbtSoe1xmVmMI\nc1zjJG2oWu6LiL6q5UnAK1XLe8jvTX0a+K9GB3XhMrMaQyhchyKit03HnEOlcF3aaFsXLjM7RZvP\nKu4FplQtT87WDTzmXwOrgAUR8dtGjXqOy8w6qR+YIWm6pDOAJVQeifUOSVOBh4BPRcTPm2nUPS4z\nq9Gus4oRcUzSdcDjVC6HWB0RWyRdncVXAjcCfwR8K+vpHWs0/HThaoP3v//9ufH3vve9ufG33347\nN75ixYrc+Ny5c3PjZkPVzgtQI2IdsG7AupVV7z8DfGYobXqoaGbJcY/LzGr4lh8zS4rvVTSzJLlw\nmVlyXLjMLDllL1w+q2hmyXGPqwtefPHFolMwG5Ky97hcuMzsFD6raGZJcuEys+SUvXB5ct7MkuMe\nl5nVcI/LzKzN3OMys1OkcFaxYY9L0hRJT0naKmmLpM9l62+WtFfSpuy1sPPpmlk3nCxejV5FaabH\ndQz4YkS8KOlsYKOkJ7LY1yLits6lZ2ZWq2Hhioh9wL7s/RuStlH5yiEzG6GSHypWkzQNOB/4abbq\nekkvSVot6dw6+yyTtEHShoMHD7aUrJkZDKFwSRoLPAh8PiJeB+4EzgNmUemR3T7YfhHRFxG9EdE7\nfvz4NqRsZp02Eua4kDSGStG6LyIeAoiI/VXxu4DHOpKhmXVd8kNFVX6Du4FtEfHVqvUTqzb7GLC5\n/emZmdVqpsd1CfAp4GVJm7J1XwaWSpoFBLALuKojGZpZVxU9DGxGM2cVfwwM9lusG2SdmVnH+ZYf\nM0uOb/kxsxrJDxXN7PRT9sLloaKZJcc9LjOr4R6XmVmbucdlZjXc4zIzazP3uMzsFCPiynkzO/2U\nvXB5qGhmyXHhMrMa7Xwel6T5krZL2iFp+SBxSfpGFn9J0gWN2nThMrOOkdQD3AEsAGZSearMzAGb\nLQBmZK9lVB5SmsuFy8xqtLHHdSGwIyJ2RsRR4AFg0YBtFgH3RsXzwLsHPO+vRlcn5zdu3HhI0u6q\nVeOAQ93MYQjKmltZ8wLnNlztzO1PWm1g48aNj0sa1+TmfyBpQ9VyX0T0VS1PAl6pWt4DXDSgjcG2\nmUT2JT2D6WrhiohTHjovaUNE9HYzh2aVNbey5gXObbjKlltEzC86h0Y8VDSzTtoLTKlanpytG+o2\np3DhMrNO6gdmSJou6QxgCbB2wDZrgcuys4sXA69l3+daV9EXoPY13qQwZc2trHmBcxuuMufWkog4\nJuk64HGgB1gdEVskXZ3FV1J5DPxCYAfwFnBlo3YVEZ3L2sysAzxUNLPkuHCZWXIKKVyNbgEokqRd\nkl6WtGnA9SlF5LJa0gFJm6vWvUfSE5J+kf08t0S53Sxpb/bZbZK0sKDcpkh6StJWSVskfS5bX+hn\nl5NXKT63lHR9jiu7BeDnwIeoXGjWDyyNiK1dTaQOSbuA3ogo/GJFSX8HvEnlquK/ytb9O/BqRNya\nFf1zI+KGkuR2M/BmRNzW7XwG5DYRmBgRL0o6G9gIfBS4ggI/u5y8FlOCzy0lRfS4mrkFwICIWA+8\nOmD1ImBN9n4NlX/4XVcnt1KIiH0R8WL2/g1gG5UrsQv97HLysiEqonDVu7y/LAL4oaSNkpYVncwg\nJlRd4/IbYEKRyQzi+uwO/9VFDWOrSZoGnA/8lBJ9dgPygpJ9bmXnyflal0bELCp3rF+bDYlKKSrj\n/DJdz3IncB4wi8p9ZrcXmYykscCDwOcj4vXqWJGf3SB5lepzS0ERhWvIl/d3U0TszX4eAL5PZWhb\nJvtP3jmf/TxQcD7viIj9EXE8Ik4Ad1HgZydpDJXicF9EPJStLvyzGyyvMn1uqSiicDVzC0AhJL0r\nmzRF0ruADwOb8/fqurXA5dn7y4FHCszlFAMeRfIxCvrsVHneyt3Atoj4alWo0M+uXl5l+dxSUsiV\n89np3q/z/7cA/GvXkxiEpPOo9LKgcjvUt4vMTdL9wGwqjz3ZD9wEPAx8B5gK7AYWR0TXJ8nr5Dab\nynAngF3AVY3uOetQbpcCzwIvAyey1V+mMp9U2GeXk9dSSvC5pcS3/JhZcjw5b2bJceEys+S4cJlZ\ncly4zCw5LlxmlhwXLjNLjguXmSXn/wDtEZUMCRz4MwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2158e44c2e8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "grid_data = train_x.iloc[:,0].as_matrix().reshape(28,28)\n",
    "plt.imshow(grid_data, interpolation = \"none\", cmap = \"Greys\")\n",
    "plt.colorbar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sample of Not the Number 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.colorbar.Colorbar at 0x2158f58ebe0>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAS4AAAD8CAYAAADJwUnTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFblJREFUeJzt3X2MVfWdx/H3hxGrW6m1C4uGB8ENrmE3lnZnsYnUtWta\nwcalNusDNZUauxQrZkmaRmOTFbNZw661K1gLGZUoidU10SI2WFfNrrRBGwa1yIO2UwoVSoUpVemD\ntcB3/7gHe5k799w7c5/Ob/y8kpu553zP+Z3vXIevv9/vPFxFBGZmKRnV6QTMzIbKhcvMkuPCZWbJ\nceEys+S4cJlZcly4zCw5Llxm1jKSVknaJ2lLlbgkLZfUJ2mzpI/W064Ll5m10n3A7Jz4HGBa9loA\nrKinURcuM2uZiFgPHMjZZC6wOkqeBz4o6bRa7R7XrATrMXbs2JgyZUo7D2n2nrJz5076+/vVSBuS\nhnI7zVbg7bLlnojoGcL+E4DXypZ3Z+v25u3UUOGSNBtYBnQB90TE0rztp0yZQm9vbyOHNLMc3d3d\n7T7k2xHR9oMOe6goqQu4i9IYdTowT9L0ZiVmZp0jqa5XE+wBJpUtT8zW5Wpkjmsm0BcROyLiHeAh\nSuNVM0vcqFGj6no1wVrgquzs4seANyMid5gIjQ0VBxubnjNwI0kLKJ0tYPLkyQ0czszaQVLdRenw\n4cO12noQOB8YK2k3cDMwGiAiVgLrgIuAPuB3wNX1HLflk/PZRF0PQHd3t5+hY5aAJg0DiYh5NeIB\nXDfUdhspXMMam5pZ8TWrcLVKI4PUjcA0SVMlHQ9cQWm8amaJa+Pk/LAMu8cVEYckLQKepHQ5xKqI\n2Nq0zMysY4re42pojisi1lGaXDOzEUISXV1dnU4jV1uvnDezNIzoHpeZjUwuXGaWlE5PvNfDhcvM\nKrhwmVlyPDlvZknxUNHMkuTCZWbJceEys+S4cJlZcly4zCwpvuXHzJLkHpeZJceFy8yS4uu4zCxJ\nLlxmlhxPzptZUjxUNLMkuXCZWXJcuMwsOU36luqWceEys2N4jsvMkuSzimaWHPe4zCwpkjzHZWbp\ncY/LzJLjwmVmSfFQ0cyS5LOKZpYcDxWtYU888URufPHixVVjfX19DR37yJEjufGPf/zjufF169ZV\njZ100knDyslaa8QPFSXtBA4Ch4FDEdHdjKTMrLOK3uNqRln9RETMcNEyGzmO3vZT61VnW7MlvSqp\nT9KNg8RPlvS4pB9J2irp6lpteqhoZsdo5rf8SOoC7gI+CewGNkpaGxHbyja7DtgWERdLGge8KumB\niHinWruN9rgCeFrSJkkLqiS+QFKvpN79+/c3eDgza4dRo0bV9arDTKAvInZkheghYO6AbQIYo1IX\n7iTgAHAor9FGe1yzImKPpL8AnpL0SkSsPyajiB6gB6C7uzsaPJ6ZtcEQ5rjGSuotW+7J/s0fNQF4\nrWx5N3DOgDa+CawFfgGMAS6PiNyzQg0VrojYk/3cJ+k7lKrr+vy9zKzIhvhYm/4mzG9fCLwE/APw\nl5Q6Qd+PiLeq7TDsoaKk90sac/Q98Clgy3DbM7PiaOJQcQ8wqWx5Yrau3NXAo1HSB/wMOCuv0UZ6\nXOOB72SV+Tjg2xHxvQbaS9Yf//jH3Hhvb29u/Morr8yN79kz8L/zsQ4fPlw11uhp7Vp/nBs2bMiN\nf/nLX64aW7169bBystZr4uUQG4FpkqZSKlhXAJ8bsM3PgQuA70saD/wVsCOv0WEXrojYAXx4uPub\nWTE186xiRByStAh4EugCVkXEVkkLs/hK4N+A+yS9DAi4ISL689r15RBmVqGZF6BGxDpg3YB1K8ve\n/4LSVFPdXLjMrMKIvuXHzEYef1mGmSXJPS4zS457XCPE3r17q8buuOOO3H1vv/323HhE/g0Ftf6I\nzjln4IXIf/K1r30td9/p06fnxmvlVuuxNqecckpu3IqnmWcVW8WFy8wquMdlZslx4TKzpPisopkl\nyYXLzJLjyyHMLDnucZlZUkb8t/yMJDt25D5Fg3PPPbdqrNFHUp988sm58QcffDA3ft5551WNnXji\nicPKqV633HJLbvzCCy9s6fGtNdzjMrPkuHCZWXJcuMwsKZ7jMrMkucdlZslx4TKz5LhwmVlSfK9i\nQpYvX54bb+Rarauvvjo3/tWvfjU3fuaZZw772K12zTXXdDoFawEXLjNLjs8qmlly3OMys6R4jsvM\nkuTCZWbJceEys+R4ct7MkuI5roSsXbs2N17r+wXzrFq1Kjfe39+fG7/88stz4xMnTqwamzVrVu6+\nZoMpeuGq2R+UtErSPklbytZ9SNJTkn6S/fS3fpqNIEd7XbVenVLPQPY+YPaAdTcCz0TENOCZbNnM\nRojkC1dErAcODFg9F7g/e38/8Jkm52VmHVT0wjXcOa7xEbE3e/9LYHy1DSUtABYATJ48eZiHM7N2\nSeFBgg1nF6VZ66oz1xHRExHdEdE9bty4Rg9nZm0watSoul4dy2+Y+70u6TSA7Oe+5qVkZp3WzKGi\npNmSXpXUJ2nQ+XBJ50t6SdJWSc/WanO4hWstMD97Px94bJjtmFnB1Fu06ilckrqAu4A5wHRgnqTp\nA7b5IPAt4B8j4q+BS2u1W3OOS9KDwPnAWEm7gZuBpcDDkq4BdgGX1fwNCm7NmjW58dmzB55Y/ZN9\n+xrrcNa6huzxxx/Pjb/vfe+rGjv11FOHldNRF198cW681jVmed54443c+KJFi3LjtZ5jtnDhwiHn\nZCVNnHifCfRFxI6s3YcondzbVrbN54BHI+LnABFR8x9UzcIVEfOqhC6ota+ZpWkI81djJfWWLfdE\nRE/Z8gTgtbLl3cA5A9o4Exgt6f+AMcCyiFidd1BfOW9mFYbQ4+qPiO4GD3cc8LeUOkMnAs9Jej4i\nfpy3g5nZu5p8jdYeYFLZ8sRsXbndwK8i4rfAbyWtBz4MVC1cxb5Yw8w6oolnFTcC0yRNlXQ8cAWl\nk3vlHgNmSTpO0p9RGkpuz2vUPS4zq9CsHldEHJK0CHgS6AJWRcRWSQuz+MqI2C7pe8Bm4AhwT0Rs\nqd6qC5eZDaKZt/NExDpg3YB1Kwcs3wbcVm+bLlyZs88+Ozf+wgsvVI2tWLFi2PtC7Ufm1Poj2rBh\nQ9XYrl27cvet5c4778yNf/Ob32yo/Ua8+eabHTv2SCaJrq6uTqeRy4XLzCoU/XlcLlxmVsGFy8yS\n48JlZknp9LO26uHCZWYVXLjMLDlFf5CgC5eZVXCPa4TIezzMLbfc0sZMKr3yyitVY7UeHfPwww/n\nxpctWzasnNrhrbfeyo0fOXKkaqzoPYpOSuHRzS5cZlbBPS4zS44Ll5klxUNFM0uSe1xmlhwXLjNL\njguXmSXHhcta7qyzzqoae/vtt3P3ffbZmt+9WVhLly7Njc+cObNqbO7cuc1OZ8TwvYpmliQ/SNDM\nkuMel5klxUNFM0uSL0A1s+S4x2VmyXHhMrOk+F5Fa4uDBw9Wjc2ZMyd33+eeey43nvdMK6g9F9LX\n11c19uKLL+bue+mll+bGa+X22c9+dlh5AUydOjU3PtIVvXDVzE7SKkn7JG0pW7dE0h5JL2Wvi1qb\nppm1y9GzivW8OqWesnofMHuQ9f8VETOy17pB4maWqKIXrppDxYhYL2lK61Mxs6Io+uR8IwPZ6yVt\nzoaSp1TbSNICSb2Sevfv39/A4cysHSTR1dVV16tThlu4VgBnADOAvcDt1TaMiJ6I6I6I7nHjxg3z\ncGbWTskPFQcTEa8ffS/pbuC7TcvIzDpuRA4VJZ1WtngJsKXatmaWnuR7XJIeBM4HxkraDdwMnC9p\nBhDATuBLLczRarj22murxp5//vncfWv98dW6nmf+/Pm58cmTJ1eNnX766bn71vpOyE9/+tO58Q0b\nNlSN5X1mUPv7Jj/wgQ/kxlM2Ii5AjYh5g6y+twW5mFlBFH2o6CvnzaxC0Xtcxc7OzNru6FCxnled\n7c2W9KqkPkk35mz3d5IOSfqnWm26cJlZhWZNzkvqAu4C5gDTgXmSplfZ7j+A/6knPxcuM6vQxLOK\nM4G+iNgREe8ADwGDfVPJ9cAjwL56GnXhMrMKQyhcY4/eGZO9FgxoagLwWtny7mxd+bEmULqsakW9\n+XlyfgQ4cOBAy9qeNGlSbnzJkiW58UZuCxkzZkxu/NZbb82NX3jhhVVjTz/9dO6+V111VW58zZo1\nufHUDeGsYn9EdDd4uDuAGyLiSL3HdeEys2McvVexSfYA5f/3m5itK9cNPHS0BwdcJOlQRFT9v4ML\nl5lVaOJ1XBuBaZKmUipYVwCfK98gIt59aqOk+4Dv5hUtcOEys0E0q3BFxCFJi4AngS5gVURslbQw\ni68cTrsuXGZ2jGbf8pM9aHTdgHWDFqyI+EI9bbpwmVkF3/JjZslx4TKz5LhwWdJuvLHqrWVA/mNr\nWm3WrFm58VNPPbVqbNeuXbn7btu2bVg5jQSdftZWPVy4zKyCC5eZJceFy8yS48JlZslx4TKzpHhy\n3syS5MJlZslx4bKkLVy4sNMpWAe4cJlZUkbE9yqa2XtP0XtcxS6rZmaDcI/LzCoUvcflwmVmFVy4\nzCw5LlxmlpQRcVZR0iRgNTAeCKAnIpZJ+hDw38AUYCdwWUT8unWpWjURMazYSNfI53L48OFmp5OU\nove46imrh4CvRMR04GPAdZKmAzcCz0TENOCZbNnMRoAhfJN1R9QsXBGxNyJeyN4fBLZT+grtucD9\n2Wb3A59pVZJm1l7JF65ykqYAHwF+CIyPiL1Z6JeUhpJmZi1X9+S8pJOAR4DFEfFWebWNiJA06KSB\npAXAAujs88nNrD6d7k3Vo64el6TRlIrWAxHxaLb6dUmnZfHTgH2D7RsRPRHRHRHd48aNa0bOZtZi\no0aNquvVsfxqbaBS6b0X2B4R3ygLrQXmZ+/nA481Pz0z64Siz3HVM1Q8F/g88LKkl7J1NwFLgYcl\nXQPsAi5rTYpWS94fUKN/XIsXL86N33bbbbnx0aNHN3T8PP39/bnx3//+91VjtT6Xrq6uYeU0UhR9\nqFizcEXED4Bqv8UFzU3HzDqt072pehT78lgzs0H4lh8zq1D0W36KnZ2Z2SDc4zKzCkWf43LhMrMK\nLlxmlpQUziq6cI0At956a9XYH/7wh9x9n3322dz4nXfemRvfvHlzbvyEE07IjTfipz/9aW58//79\nVWMnn3xy7r4rVqwYVk5WSdJsYBnQBdwTEUsHxK8EbqB02dVB4NqI+FFemy5cZlahWWcVJXUBdwGf\nBHYDGyWtjYhtZZv9DPj7iPi1pDlAD3BObn5Nyc7MbHAzgb6I2BER7wAPUXok1rsiYkPZQ0ifBybW\natQ9LjOrMIQ5rrGSesuWeyKip2x5AvBa2fJu8ntT1wBP1DqoC5eZVRhC4eqPiO4mHfMTlArXrFrb\nunCZ2TGafFZxDzCpbHlitm7gMc8G7gHmRMSvajXqOS4za6WNwDRJUyUdD1xB6ZFY75I0GXgU+HxE\n/LieRt3jMrMKzTqrGBGHJC0CnqR0OcSqiNgqaWEWXwn8K/DnwLeynt6hWsNPF64RYMaMGVVja9as\nyd13ypQpufE33ngjN75+/frceCvV+oqx008/vWrs7rvvzt33ggve209sauYFqBGxDlg3YN3Ksvdf\nBL44lDY9VDSz5LjHZWYVfMuPmSXF9yqaWZJcuMwsOS5cZpacohcun1U0s+S4xzXCjRkzJjf+3HPP\n5cZffPHFho6/ZMmSqrEDBw7k7rt8+fKGjn3JJZdUjR1//PENtT3SFb3H5cJlZsfwWUUzS5ILl5kl\np+iFy5PzZpYc97jMrIJ7XGZmTeYel5kdY0ScVZQ0CVgNjAeC0sPwl0laAvwzcPTL627KnrtjCTnz\nzDMbitdy+eWXN7S/dUbyhQs4BHwlIl6QNAbYJOmpLPZfEfH11qVnZlapZuGKiL3A3uz9QUnbKX3l\nkJmNUEXvcQ1pcl7SFOAjwA+zVddL2ixplaRTquyzQFKvpN68r0Q3M6tX3YVL0knAI8DiiHgLWAGc\nAcyg1CO7fbD9IqInIrojonvcuHFNSNnMWu3oBH2tV6fUdVZR0mhKReuBiHgUICJeL4vfDXy3JRma\nWdslP1RU6Te4F9geEd8oW39a2WaXAFuan56ZWaV6elznAp8HXpb0UrbuJmCepBmULpHYCXypJRma\nWVt1ehhYj3rOKv4AGOy38DVbZtYRvuXHzJLjW37MrELyQ0Uze+8peuHyUNHMkuMel5lVcI/LzKzJ\n3OMyswrucZmZNZl7XGZ2jBFx5byZvfcUvXB5qGhmyXHhMrMKzXwel6TZkl6V1CfpxkHikrQ8i2+W\n9NFabbpwmVnLSOoC7gLmANMpPVVm+oDN5gDTstcCSg8pzeXCZWYVmtjjmgn0RcSOiHgHeAiYO2Cb\nucDqKHke+OCA5/1VaOvk/KZNm/ol7SpbNRbob2cOQ1DU3IqaFzi34Wpmbqc32sCmTZuelDS2zs1P\nkNRbttwTET1lyxOA18qWdwPnDGhjsG0mkH1Jz2DaWrgi4piHzkvqjYjuduZQr6LmVtS8wLkNV9Fy\ni4jZnc6hFg8VzayV9gCTypYnZuuGus0xXLjMrJU2AtMkTZV0PHAFsHbANmuBq7Kzix8D3sy+z7Wq\nTl+A2lN7k44pam5FzQuc23AVObeGRMQhSYuAJ4EuYFVEbJW0MIuvpPQY+IuAPuB3wNW12lVEtC5r\nM7MW8FDRzJLjwmVmyelI4ap1C0AnSdop6WVJLw24PqUTuayStE/SlrJ1H5L0lKSfZD9PKVBuSyTt\nyT67lyRd1KHcJkn6X0nbJG2V9C/Z+o5+djl5FeJzS0nb57iyWwB+DHyS0oVmG4F5EbGtrYlUIWkn\n0B0RHb9YUdJ5wG8oXVX8N9m6/wQORMTSrOifEhE3FCS3JcBvIuLr7c5nQG6nAadFxAuSxgCbgM8A\nX6CDn11OXpdRgM8tJZ3ocdVzC4ABEbEeODBg9Vzg/uz9/ZT+8NuuSm6FEBF7I+KF7P1BYDulK7E7\n+tnl5GVD1InCVe3y/qII4GlJmyQt6HQygxhfdo3LL4HxnUxmENdnd/iv6tQwtpykKcBHgB9SoM9u\nQF5QsM+t6Dw5X2lWRMygdMf6ddmQqJCiNM4v0vUsK4AzgBmU7jO7vZPJSDoJeARYHBFvlcc6+dkN\nklehPrcUdKJwDfny/naKiD3Zz33AdygNbYvk9aN3zmc/93U4n3dFxOsRcTgijgB308HPTtJoSsXh\ngYh4NFvd8c9usLyK9LmlohOFq55bADpC0vuzSVMkvR/4FLAlf6+2WwvMz97PBx7rYC7HGPAokkvo\n0Gen0vNW7gW2R8Q3ykId/eyq5VWUzy0lHblyPjvdewd/ugXg39uexCAknUGplwWl26G+3cncJD0I\nnE/psSevAzcDa4CHgcnALuCyiGj7JHmV3M6nNNwJYCfwpVr3nLUot1nA94GXgSPZ6psozSd17LPL\nyWseBfjcUuJbfswsOZ6cN7PkuHCZWXJcuMwsOS5cZpYcFy4zS44Ll5klx4XLzJLz/zhPF0z6uXj9\nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2158e44cbe0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "grid_data = train_x.iloc[:,5000].as_matrix().reshape(28,28)\n",
    "plt.imshow(grid_data, interpolation = \"none\", cmap = \"Greys\")\n",
    "plt.colorbar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def initialize_parameters(layer_dims,initialization,keep_prob):\n",
    "    \n",
    "    parameters = {}\n",
    "    L = len(layer_dims)           \n",
    "\n",
    "    for l in range(1, L):\n",
    "        \n",
    "        if initialization == \"random\":\n",
    "            parameters['W' + str(l)] = np.random.randn(layer_dims[l],layer_dims[l-1])*0.01\n",
    "            parameters['b' + str(l)] = np.zeros((layer_dims[l],1))\n",
    "            \n",
    "        elif initialization == \"he\":\n",
    "            parameters['W' + str(l)] = np.random.randn(layer_dims[l],layer_dims[l-1])*np.sqrt(2/layer_dims[l-1])*.01\n",
    "            parameters['b' + str(l)] = np.zeros((layer_dims[l],1))\n",
    "            \n",
    "        elif initialization == \"xavier\":\n",
    "            parameters['W' + str(l)] = np.random.randn(layer_dims[l],layer_dims[l-1])*np.sqrt(1/layer_dims[l-1])*.01\n",
    "            parameters['b' + str(l)] = np.zeros((layer_dims[l],1))          \n",
    "        \n",
    "        drop_out = np.random.rand(layer_dims[l],layer_dims[l-1])\n",
    "        parameters['D' + str(l)] = drop_out < keep_prob \n",
    "        parameters['W' + str(l)] = parameters['W' + str(l)]*parameters['D' + str(l)]                           \n",
    "        parameters['W' + str(l)] = parameters['W' + str(l)]/keep_prob\n",
    "        \n",
    "        assert(parameters['W' + str(l)].shape == (layer_dims[l], layer_dims[l-1]))\n",
    "        assert(parameters['b' + str(l)].shape == (layer_dims[l], 1))\n",
    "        \n",
    "    return parameters\n",
    "    \n",
    "def linear_forward(A, W, b):\n",
    "    Z = np.dot(W,A)+b\n",
    "    cache = (A, W, b)\n",
    "    assert(Z.shape == (W.shape[0], A.shape[1]))\n",
    "    return Z, cache\n",
    "\n",
    "def relu(z):\n",
    "    r = z * (z > 0)\n",
    "    return r,z \n",
    "\n",
    "def sigmoid(z):\n",
    "    s = 1/(1+np.exp(-z))\n",
    "    return s,z\n",
    "\n",
    "def tanh(z):\n",
    "    t = (2/(1+np.exp(-z))) - 1\n",
    "    return t,z\n",
    "\n",
    "def linear_activation_forward(A_prev, W, b, activation):\n",
    "    Z, linear_cache = linear_forward(A_prev, W, b)\n",
    "    if activation == \"sigmoid\":\n",
    "        A, activation_cache = sigmoid(Z)\n",
    "    \n",
    "    elif activation == \"relu\":\n",
    "        A, activation_cache = relu(Z)\n",
    "        \n",
    "    elif activation == \"tanh\":\n",
    "        A, activation_cache = tanh(Z)\n",
    "    \n",
    "    else:\n",
    "        print(\"Error in Activation\")\n",
    "    \n",
    "    cache = (linear_cache, activation_cache)\n",
    "    assert (A.shape == (W.shape[0], A_prev.shape[1]))\n",
    "    return A, cache\n",
    "\n",
    "def L_model_forward(X, parameters,activation):\n",
    "    caches = []\n",
    "    A = X\n",
    "    L = len(activation) \n",
    "    for l in range(1, L):\n",
    "        A_prev = A \n",
    "        A, cache = linear_activation_forward(A_prev,parameters['W' + str(l)], parameters['b' + str(l)], activation=activation[l-1])\n",
    "        caches.append(cache)\n",
    "    AL, cache = linear_activation_forward(A,parameters['W' + str(L)], parameters['b' + str(L)], activation='sigmoid')\n",
    "    caches.append(cache)\n",
    "    \n",
    "    assert(AL.shape == (1,X.shape[1]))\n",
    "    assert (L == len(activation))\n",
    "    return AL, caches\n",
    "\n",
    "def compute_cost_L2(AL, Y,lambd,layer_dims,parameters):\n",
    "    m = Y.shape[1]\n",
    "    L = len(layer_dims)    \n",
    "    sum_W=0\n",
    "    \n",
    "    if lambd == 0:\n",
    "        cost = -(1/m)*np.sum((Y*np.log(AL))+((1-Y)*np.log(1-AL)))\n",
    "    else:\n",
    "        for l in range(1, L):\n",
    "            sum_W = sum_W + np.sum(np.square(parameters['W' + str(l)])) \n",
    "        L2_regularization_cost = (1/m)*(lambd/2)*sum_W\n",
    "        cost = -(1/m)*np.sum((Y*np.log(AL))+((1-Y)*np.log(1-AL)))+L2_regularization_cost\n",
    "    \n",
    "    cost = np.squeeze(cost)\n",
    "    assert(cost.shape == ())\n",
    "    return cost\n",
    "\n",
    "def relu_backward(dA, cache):\n",
    "    Z = cache\n",
    "    dZ = np.array(dA, copy=True) \n",
    "    dZ[Z <= 0] = 0 \n",
    "    \n",
    "    assert (dZ.shape == Z.shape)\n",
    "    return dZ\n",
    "\n",
    "def sigmoid_backward(dA, cache):\n",
    "    Z = cache\n",
    "    s = 1/(1+np.exp(-Z))\n",
    "    dZ = dA * s * (1-s)\n",
    "    \n",
    "    assert (dZ.shape == Z.shape)\n",
    "    return dZ\n",
    "\n",
    "def tanh_backward(dA, cache):\n",
    "    Z = cache\n",
    "    s = np.tanh(-Z)\n",
    "    dZ = 1 - (s**2)\n",
    "    \n",
    "    assert (dZ.shape == Z.shape)\n",
    "    return dZ\n",
    "\n",
    "def linear_backward(dZ, cache,lambd):\n",
    "    A_prev, W, b = cache\n",
    "    m = A_prev.shape[1]\n",
    "    dW = (1/m)*np.dot(dZ,A_prev.T)\n",
    "    \n",
    "    if lambd != 0:\n",
    "        dW = dW + (lambd*W/m)\n",
    "    \n",
    "    db = (1/m)*np.sum(dZ,axis=1, keepdims=True)\n",
    "    dA_prev = np.dot(W.T,dZ)\n",
    "    \n",
    "    assert (dA_prev.shape == A_prev.shape)\n",
    "    assert (dW.shape == W.shape)\n",
    "    assert (db.shape == b.shape)\n",
    "    return dA_prev, dW, db\n",
    "\n",
    "def linear_activation_backward(dA, cache, lambd,activation):\n",
    "    linear_cache, activation_cache = cache\n",
    "    \n",
    "    if activation == \"relu\":\n",
    "        dZ = relu_backward(dA,activation_cache)\n",
    "        dA_prev, dW, db =  linear_backward(dZ, linear_cache,lambd)\n",
    "        \n",
    "    elif activation == \"sigmoid\":\n",
    "        dZ = sigmoid_backward(dA,activation_cache)\n",
    "        dA_prev, dW, db =  linear_backward(dZ, linear_cache,lambd)\n",
    "        \n",
    "    elif activation == \"tanh\":\n",
    "        dZ = tanh_backward(dA,activation_cache)\n",
    "        dA_prev, dW, db =  linear_backward(dZ, linear_cache,lambd)\n",
    "    \n",
    "    return dA_prev, dW, db\n",
    "\n",
    "def model_backward(AL, Y, caches,lambd,activation,parameters,keep_prob):\n",
    "\n",
    "    grads = {}\n",
    "    L = len(caches) \n",
    "    m = AL.shape[1]\n",
    "    Y = Y.reshape(AL.shape)\n",
    "    \n",
    "    dAL = - (np.divide(Y, AL) - np.divide(1 - Y, 1 - AL)) \n",
    "    \n",
    "    current_cache = caches[L-1]\n",
    "    grads[\"dA\" + str(L)], grads[\"dW\" + str(L)], grads[\"db\" + str(L)] = linear_activation_backward(dAL, current_cache,lambd=lambd, activation = 'sigmoid')\n",
    "    grads[\"dW\" + str(L)] = grads[\"dW\" + str(L)]*parameters['D' + str(L)] \n",
    "    grads[\"dW\" + str(L)] = grads[\"dW\" + str(L)]/keep_prob \n",
    "\n",
    "    for l in reversed(range(L-1)):\n",
    "        current_cache = caches[l]\n",
    "        dA_prev_temp, dW_temp, db_temp = linear_activation_backward(grads[\"dA\" + str(l + 2)], current_cache,lambd=lambd, activation = activation[l])\n",
    "        grads[\"dA\" + str(l + 1)] = dA_prev_temp\n",
    "        grads[\"dW\" + str(l + 1)] = dW_temp\n",
    "        grads[\"dW\" + str(l + 1)] = grads[\"dW\" + str(l + 1)]*parameters['D' + str(l + 1)] \n",
    "        grads[\"dW\" + str(l + 1)] = grads[\"dW\" + str(l + 1)]/keep_prob \n",
    "        grads[\"db\" + str(l + 1)] = db_temp\n",
    "        \n",
    "    return grads\n",
    "\n",
    "def update_parameters(parameters, grads, learning_rate):\n",
    "    L = len(activation) \n",
    "    \n",
    "    for l in range(L):\n",
    "        parameters[\"W\" + str(l+1)] = parameters['W' + str(l+1)] - learning_rate* grads[\"dW\" + str(l + 1)] \n",
    "        parameters[\"b\" + str(l+1)] = parameters['b' + str(l+1)] - learning_rate* grads[\"db\" + str(l + 1)] \n",
    "        \n",
    "    return parameters\n",
    "\n",
    "def initialize_velocity(parameters,activation):\n",
    "    L = len(activation) \n",
    "    v = {}\n",
    "\n",
    "    for l in range(L):\n",
    "        v[\"dW\" + str(l+1)] = np.zeros(parameters[\"W\" + str(l+1)].shape)\n",
    "        v[\"db\" + str(l+1)] = np.zeros(parameters[\"b\" + str(l+1)].shape)\n",
    "        \n",
    "    return v\n",
    "\n",
    "def update_parameters_with_momentum(parameters, grads, v, beta, learning_rate,activation):\n",
    "\n",
    "    L = len(activation) \n",
    "\n",
    "    for l in range(L):\n",
    "\n",
    "        v[\"dW\" + str(l+1)] = beta*v['dW' + str(l+1)] + (1-beta)*grads['dW' + str(l+1)]\n",
    "        v[\"db\" + str(l+1)] = beta*v['db' + str(l+1)] + (1-beta)*grads['db' + str(l+1)]\n",
    "\n",
    "        parameters[\"W\" + str(l+1)] = parameters[\"W\" + str(l+1)] - learning_rate*v[\"dW\" + str(l+1)] \n",
    "        parameters[\"b\" + str(l+1)] = parameters[\"b\" + str(l+1)] - learning_rate*v[\"db\" + str(l+1)] \n",
    "\n",
    "    return parameters, v\n",
    "\n",
    "def initialize_adam(parameters) :\n",
    "\n",
    "    L = len(activation) \n",
    "    v = {}\n",
    "    s = {}\n",
    "\n",
    "    for l in range(L):\n",
    "\n",
    "        v[\"dW\" + str(l+1)] = np.zeros(parameters['W' + str(l+1)].shape)\n",
    "        v[\"db\" + str(l+1)] = np.zeros(parameters['b' + str(l+1)].shape)\n",
    "        s[\"dW\" + str(l+1)] = np.zeros(parameters['W' + str(l+1)].shape)\n",
    "        s[\"db\" + str(l+1)] = np.zeros(parameters['b' + str(l+1)].shape)\n",
    "    \n",
    "    return v, s\n",
    "\n",
    "def update_parameters_with_adam(parameters, grads, v, s, t, learning_rate,beta1, beta2, epsilon):\n",
    "    L = len(activation) \n",
    "    v_corrected = {}                     \n",
    "    s_corrected = {}                    \n",
    "\n",
    "    for l in range(L):\n",
    "\n",
    "        v[\"dW\" + str(l+1)] = beta1 * v[\"dW\" + str(l+1)] + (1-beta1)*grads['dW' + str(l+1)]\n",
    "        v[\"db\" + str(l+1)] = beta1 * v[\"db\" + str(l+1)] + (1-beta1)*grads['db' + str(l+1)]\n",
    "\n",
    "        v_corrected[\"dW\" + str(l+1)] = v[\"dW\" + str(l+1)]/(1-beta1**t)\n",
    "        v_corrected[\"db\" + str(l+1)] = v[\"db\" + str(l+1)]/(1-beta1**t)\n",
    "\n",
    "        s[\"dW\" + str(l+1)] = beta2 * s[\"dW\" + str(l+1)] + (1-beta2)*grads['dW' + str(l+1)]**2\n",
    "        s[\"db\" + str(l+1)] = beta2 * s[\"db\" + str(l+1)] + (1-beta2)*grads['db' + str(l+1)]**2\n",
    "\n",
    "        s_corrected[\"dW\" + str(l+1)] = s[\"dW\" + str(l+1)]/(1-beta2**t)\n",
    "        s_corrected[\"db\" + str(l+1)] = s[\"db\" + str(l+1)]/(1-beta2**t)\n",
    "\n",
    "        parameters[\"W\" + str(l+1)] = parameters[\"W\" + str(l+1)] - learning_rate*(v_corrected[\"dW\" + str(l+1)]/(np.sqrt(s_corrected[\"dW\" + str(l+1)]+epsilon)))\n",
    "        parameters[\"b\" + str(l+1)] = parameters[\"b\" + str(l+1)] - learning_rate*(v_corrected[\"db\" + str(l+1)]/(np.sqrt(s_corrected[\"db\" + str(l+1)]+epsilon)))\n",
    "\n",
    "    return parameters, v, s\n",
    "\n",
    "\n",
    "def random_mini_batches(X, Y, mini_batch_size,seed = 0):    \n",
    "    \n",
    "    np.random.seed(seed)\n",
    "    m = X.shape[1]            \n",
    "    mini_batches = []\n",
    " \n",
    "    permutation = list(np.random.permutation(m))\n",
    "    shuffled_X = X.iloc[:, permutation]\n",
    "    shuffled_Y = Y[:, permutation].reshape((1,m))\n",
    "\n",
    "    num_complete_minibatches = math.floor(m/mini_batch_size)\n",
    "    for k in range(0, num_complete_minibatches):\n",
    "\n",
    "        mini_batch_X = shuffled_X.iloc[:,(k*mini_batch_size):((k+1)*mini_batch_size)]\n",
    "        mini_batch_Y = shuffled_Y[:,(k*mini_batch_size):((k+1)*mini_batch_size)]\n",
    "\n",
    "        mini_batch = (mini_batch_X, mini_batch_Y)\n",
    "        mini_batches.append(mini_batch)\n",
    "    \n",
    "\n",
    "    if m % mini_batch_size != 0:\n",
    "\n",
    "        mini_batch_X = shuffled_X.iloc[:, -( m - (mini_batch_size * math.floor(m/mini_batch_size) ) ) ]\n",
    "        mini_batch_Y = shuffled_Y[:, -( m - (mini_batch_size * math.floor(m/mini_batch_size) ) ) ]\n",
    "\n",
    "        mini_batch = (mini_batch_X, mini_batch_Y)\n",
    "        mini_batches.append(mini_batch)\n",
    "    \n",
    "    return mini_batches\n",
    "\n",
    "def L_layer_model_L2(X, Y, layer_dims,activation, initialization,keep_prob,mini_batch_size,\n",
    "                     test_x, test_y,\n",
    "                     lambd=0,learning_rate = 0.0075, \n",
    "                     num_iterations = 3000,print_cost=False,beta=.9,optimizer=\"gd\",beta1 = 0.9, beta2 = 0.999,  epsilon = 1e-8):\n",
    "    \n",
    "    costs = []    \n",
    "    train_accuracies = []\n",
    "    test_accuracies = []\n",
    "    t = 0\n",
    "    seed = 0\n",
    "    parameters = initialize_parameters(layer_dims,initialization,keep_prob)\n",
    "    v = initialize_velocity(parameters,activation)\n",
    "    v,s = initialize_adam(parameters)\n",
    "    num_batches = X.shape[1] // mini_batch_size\n",
    "\n",
    "    mini_batches = random_mini_batches(X, Y, mini_batch_size)\n",
    "\n",
    "    for i in range(0, num_iterations):\n",
    "        random_shuffle = np.arange(0, num_batches)\n",
    "        random.shuffle(random_shuffle)\n",
    "        for j in random_shuffle:\n",
    "\n",
    "            AL, caches = L_model_forward(mini_batches[j][0], parameters,activation)\n",
    "\n",
    "            cost = compute_cost_L2(AL, mini_batches[j][1], lambd,layer_dims,parameters)\n",
    "\n",
    "            grads = model_backward(AL, mini_batches[j][1], caches,lambd,activation,parameters,keep_prob)\n",
    "            \n",
    "            if optimizer == 'gd':\n",
    "                parameters = update_parameters(parameters, grads, learning_rate)\n",
    "            elif optimizer == 'momentum':\n",
    "                parameters,v =  update_parameters_with_momentum(parameters, grads, v, beta, learning_rate,activation)\n",
    "\n",
    "            elif optimizer == 'adam':\n",
    "                t = t + 1\n",
    "                parameters,v,s = update_parameters_with_adam(parameters, grads, v, s, t, learning_rate,\n",
    "                                beta1, beta2, epsilon)\n",
    "            \n",
    "            if print_cost and i % 10 == 0:\n",
    "                print (\"Cost after iteration %i: %f\" %(i, cost))\n",
    "                \n",
    "                m = X.shape[1]\n",
    "                n = len(parameters) // 2 \n",
    "                p_train = np.zeros((1,m))\n",
    "                p_test = np.zeros((1,test_x.shape[1]))\n",
    "                probas_train, caches = L_model_forward(train_x, parameters,activation)\n",
    "                probas_test, caches = L_model_forward(test_x, parameters,activation)\n",
    "                \n",
    "                for i in range(0, probas_train.shape[1]):\n",
    "                    if probas_train[0,i] > 0.5:\n",
    "                        p_train[0,i] = 1\n",
    "                    else:\n",
    "                        p_train[0,i] = 0\n",
    "                        \n",
    "                for i in range(0, probas_test.shape[1]):\n",
    "                    if probas_test[0,i] > 0.5:\n",
    "                        p_test[0,i] = 1\n",
    "                    else:\n",
    "                        p_test[0,i] = 0\n",
    "                        \n",
    "                train_accuracy = (np.sum((p_train == train_y)/m))\n",
    "                test_accuracy = (np.sum((p_test == test_y)/test_x.shape[1]))\n",
    "        \n",
    "                costs.append(cost)\n",
    "                train_accuracies.append(train_accuracy)\n",
    "                test_accuracies.append(test_accuracy)\n",
    "                \n",
    "    plt.plot(np.squeeze(costs),label = \"cost\")\n",
    "    plt.plot(np.squeeze(train_accuracies), label = \"train accuracy\")\n",
    "    plt.plot(np.squeeze(train_accuracies), label = \"test accuracy\")\n",
    "    plt.ylabel('cost')\n",
    "    plt.xlabel('iterations (per tens)')\n",
    "    plt.title(\"Learning rate =\" + str(learning_rate))\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "    \n",
    "    return parameters\n",
    "\n",
    "def predict(X, y, parameters,activation):\n",
    "\n",
    "    m = X.shape[1]\n",
    "    n = len(parameters) // 2 \n",
    "    p = np.zeros((1,m))\n",
    "    \n",
    "    probas, caches = L_model_forward(X, parameters,activation)\n",
    "    \n",
    "    for i in range(0, probas.shape[1]):\n",
    "        if probas[0,i] > 0.5:\n",
    "            p[0,i] = 1\n",
    "        \n",
    "        else:\n",
    "            p[0,i] = 0\n",
    "\n",
    "    print(\"Number of Ones = \"+str(np.sum(p)))\n",
    "    print(\"Accuracy: \"  + str(np.sum((p == y)/m)))\n",
    "    return"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plain DNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cost after iteration 0: 0.693175\n",
      "Cost after iteration 10: 0.693023\n",
      "Cost after iteration 20: 0.692863\n",
      "Cost after iteration 30: 0.692677\n",
      "Cost after iteration 40: 0.692436\n",
      "Cost after iteration 50: 0.692099\n",
      "Cost after iteration 60: 0.691600\n",
      "Cost after iteration 70: 0.690813\n",
      "Cost after iteration 80: 0.689491\n",
      "Cost after iteration 90: 0.687101\n",
      "Cost after iteration 100: 0.682363\n",
      "Cost after iteration 110: 0.671790\n",
      "Cost after iteration 120: 0.644793\n",
      "Cost after iteration 130: 0.573433\n",
      "Cost after iteration 140: 0.452636\n",
      "Cost after iteration 150: 0.350813\n",
      "Cost after iteration 160: 0.266873\n",
      "Cost after iteration 170: 0.189001\n",
      "Cost after iteration 180: 0.134610\n",
      "Cost after iteration 190: 0.106186\n",
      "Cost after iteration 200: 0.091394\n",
      "Cost after iteration 210: 0.082853\n",
      "Cost after iteration 220: 0.077389\n",
      "Cost after iteration 230: 0.073587\n",
      "Cost after iteration 240: 0.070766\n",
      "Cost after iteration 250: 0.068566\n",
      "Cost after iteration 260: 0.066783\n",
      "Cost after iteration 270: 0.065291\n",
      "Cost after iteration 280: 0.064011\n",
      "Cost after iteration 290: 0.062889\n",
      "Cost after iteration 300: 0.061888\n",
      "Cost after iteration 310: 0.060983\n",
      "Cost after iteration 320: 0.060155\n",
      "Cost after iteration 330: 0.059393\n",
      "Cost after iteration 340: 0.058688\n",
      "Cost after iteration 350: 0.058031\n",
      "Cost after iteration 360: 0.057416\n",
      "Cost after iteration 370: 0.056837\n",
      "Cost after iteration 380: 0.056292\n",
      "Cost after iteration 390: 0.055776\n",
      "Cost after iteration 400: 0.055287\n",
      "Cost after iteration 410: 0.054822\n",
      "Cost after iteration 420: 0.054380\n",
      "Cost after iteration 430: 0.053958\n",
      "Cost after iteration 440: 0.053554\n",
      "Cost after iteration 450: 0.053168\n",
      "Cost after iteration 460: 0.052798\n",
      "Cost after iteration 470: 0.052443\n",
      "Cost after iteration 480: 0.052101\n",
      "Cost after iteration 490: 0.051772\n",
      "Cost after iteration 500: 0.051454\n",
      "Cost after iteration 510: 0.051148\n",
      "Cost after iteration 520: 0.050852\n",
      "Cost after iteration 530: 0.050566\n",
      "Cost after iteration 540: 0.050288\n",
      "Cost after iteration 550: 0.050020\n",
      "Cost after iteration 560: 0.049758\n",
      "Cost after iteration 570: 0.049504\n",
      "Cost after iteration 580: 0.049257\n",
      "Cost after iteration 590: 0.049017\n",
      "Cost after iteration 600: 0.048782\n",
      "Cost after iteration 610: 0.048554\n",
      "Cost after iteration 620: 0.048330\n",
      "Cost after iteration 630: 0.048112\n",
      "Cost after iteration 640: 0.047898\n",
      "Cost after iteration 650: 0.047689\n",
      "Cost after iteration 660: 0.047484\n",
      "Cost after iteration 670: 0.047283\n",
      "Cost after iteration 680: 0.047086\n",
      "Cost after iteration 690: 0.046892\n",
      "Cost after iteration 700: 0.046702\n",
      "Cost after iteration 710: 0.046514\n",
      "Cost after iteration 720: 0.046330\n",
      "Cost after iteration 730: 0.046148\n",
      "Cost after iteration 740: 0.045969\n",
      "Cost after iteration 750: 0.045793\n",
      "Cost after iteration 760: 0.045619\n",
      "Cost after iteration 770: 0.045447\n",
      "Cost after iteration 780: 0.045278\n",
      "Cost after iteration 790: 0.045110\n",
      "Cost after iteration 800: 0.044944\n",
      "Cost after iteration 810: 0.044780\n",
      "Cost after iteration 820: 0.044618\n",
      "Cost after iteration 830: 0.044458\n",
      "Cost after iteration 840: 0.044299\n",
      "Cost after iteration 850: 0.044141\n",
      "Cost after iteration 860: 0.043985\n",
      "Cost after iteration 870: 0.043830\n",
      "Cost after iteration 880: 0.043677\n",
      "Cost after iteration 890: 0.043525\n",
      "Cost after iteration 900: 0.043374\n",
      "Cost after iteration 910: 0.043224\n",
      "Cost after iteration 920: 0.043075\n",
      "Cost after iteration 930: 0.042927\n",
      "Cost after iteration 940: 0.042780\n",
      "Cost after iteration 950: 0.042634\n",
      "Cost after iteration 960: 0.042489\n",
      "Cost after iteration 970: 0.042344\n",
      "Cost after iteration 980: 0.042201\n",
      "Cost after iteration 990: 0.042058\n",
      "Cost after iteration 1000: 0.041916\n",
      "Cost after iteration 1010: 0.041775\n",
      "Cost after iteration 1020: 0.041634\n",
      "Cost after iteration 1030: 0.041494\n",
      "Cost after iteration 1040: 0.041354\n",
      "Cost after iteration 1050: 0.041215\n",
      "Cost after iteration 1060: 0.041076\n",
      "Cost after iteration 1070: 0.040938\n",
      "Cost after iteration 1080: 0.040800\n",
      "Cost after iteration 1090: 0.040663\n",
      "Cost after iteration 1100: 0.040525\n",
      "Cost after iteration 1110: 0.040388\n",
      "Cost after iteration 1120: 0.040251\n",
      "Cost after iteration 1130: 0.040114\n",
      "Cost after iteration 1140: 0.039978\n",
      "Cost after iteration 1150: 0.039841\n",
      "Cost after iteration 1160: 0.039705\n",
      "Cost after iteration 1170: 0.039569\n",
      "Cost after iteration 1180: 0.039433\n",
      "Cost after iteration 1190: 0.039297\n",
      "Cost after iteration 1200: 0.039161\n",
      "Cost after iteration 1210: 0.039025\n",
      "Cost after iteration 1220: 0.038889\n",
      "Cost after iteration 1230: 0.038753\n",
      "Cost after iteration 1240: 0.038617\n",
      "Cost after iteration 1250: 0.038480\n",
      "Cost after iteration 1260: 0.038344\n",
      "Cost after iteration 1270: 0.038207\n",
      "Cost after iteration 1280: 0.038071\n",
      "Cost after iteration 1290: 0.037933\n",
      "Cost after iteration 1300: 0.037795\n",
      "Cost after iteration 1310: 0.037656\n",
      "Cost after iteration 1320: 0.037518\n",
      "Cost after iteration 1330: 0.037379\n",
      "Cost after iteration 1340: 0.037239\n",
      "Cost after iteration 1350: 0.037099\n",
      "Cost after iteration 1360: 0.036958\n",
      "Cost after iteration 1370: 0.036818\n",
      "Cost after iteration 1380: 0.036677\n",
      "Cost after iteration 1390: 0.036535\n",
      "Cost after iteration 1400: 0.036392\n",
      "Cost after iteration 1410: 0.036249\n",
      "Cost after iteration 1420: 0.036104\n",
      "Cost after iteration 1430: 0.035959\n",
      "Cost after iteration 1440: 0.035813\n",
      "Cost after iteration 1450: 0.035667\n",
      "Cost after iteration 1460: 0.035521\n",
      "Cost after iteration 1470: 0.035374\n",
      "Cost after iteration 1480: 0.035226\n",
      "Cost after iteration 1490: 0.035078\n",
      "Cost after iteration 1500: 0.034928\n",
      "Cost after iteration 1510: 0.034779\n",
      "Cost after iteration 1520: 0.034628\n",
      "Cost after iteration 1530: 0.034476\n",
      "Cost after iteration 1540: 0.034324\n",
      "Cost after iteration 1550: 0.034171\n",
      "Cost after iteration 1560: 0.034017\n",
      "Cost after iteration 1570: 0.033863\n",
      "Cost after iteration 1580: 0.033708\n",
      "Cost after iteration 1590: 0.033552\n",
      "Cost after iteration 1600: 0.033396\n",
      "Cost after iteration 1610: 0.033239\n",
      "Cost after iteration 1620: 0.033080\n",
      "Cost after iteration 1630: 0.032921\n",
      "Cost after iteration 1640: 0.032761\n",
      "Cost after iteration 1650: 0.032600\n",
      "Cost after iteration 1660: 0.032438\n",
      "Cost after iteration 1670: 0.032275\n",
      "Cost after iteration 1680: 0.032111\n",
      "Cost after iteration 1690: 0.031946\n",
      "Cost after iteration 1700: 0.031779\n",
      "Cost after iteration 1710: 0.031612\n",
      "Cost after iteration 1720: 0.031444\n",
      "Cost after iteration 1730: 0.031276\n",
      "Cost after iteration 1740: 0.031106\n",
      "Cost after iteration 1750: 0.030935\n",
      "Cost after iteration 1760: 0.030763\n",
      "Cost after iteration 1770: 0.030591\n",
      "Cost after iteration 1780: 0.030417\n",
      "Cost after iteration 1790: 0.030242\n",
      "Cost after iteration 1800: 0.030067\n",
      "Cost after iteration 1810: 0.029891\n",
      "Cost after iteration 1820: 0.029714\n",
      "Cost after iteration 1830: 0.029535\n",
      "Cost after iteration 1840: 0.029356\n",
      "Cost after iteration 1850: 0.029177\n",
      "Cost after iteration 1860: 0.028996\n",
      "Cost after iteration 1870: 0.028815\n",
      "Cost after iteration 1880: 0.028633\n",
      "Cost after iteration 1890: 0.028450\n",
      "Cost after iteration 1900: 0.028266\n",
      "Cost after iteration 1910: 0.028082\n",
      "Cost after iteration 1920: 0.027897\n",
      "Cost after iteration 1930: 0.027712\n",
      "Cost after iteration 1940: 0.027526\n",
      "Cost after iteration 1950: 0.027340\n",
      "Cost after iteration 1960: 0.027153\n",
      "Cost after iteration 1970: 0.026966\n",
      "Cost after iteration 1980: 0.026778\n",
      "Cost after iteration 1990: 0.026589\n",
      "Cost after iteration 2000: 0.026400\n",
      "Cost after iteration 2010: 0.026210\n",
      "Cost after iteration 2020: 0.026021\n",
      "Cost after iteration 2030: 0.025830\n",
      "Cost after iteration 2040: 0.025640\n",
      "Cost after iteration 2050: 0.025450\n",
      "Cost after iteration 2060: 0.025259\n",
      "Cost after iteration 2070: 0.025069\n",
      "Cost after iteration 2080: 0.024879\n",
      "Cost after iteration 2090: 0.024689\n",
      "Cost after iteration 2100: 0.024499\n",
      "Cost after iteration 2110: 0.024309\n",
      "Cost after iteration 2120: 0.024120\n",
      "Cost after iteration 2130: 0.023931\n",
      "Cost after iteration 2140: 0.023743\n",
      "Cost after iteration 2150: 0.023555\n",
      "Cost after iteration 2160: 0.023367\n",
      "Cost after iteration 2170: 0.023180\n",
      "Cost after iteration 2180: 0.022993\n",
      "Cost after iteration 2190: 0.022807\n",
      "Cost after iteration 2200: 0.022621\n",
      "Cost after iteration 2210: 0.022436\n",
      "Cost after iteration 2220: 0.022251\n",
      "Cost after iteration 2230: 0.022068\n",
      "Cost after iteration 2240: 0.021885\n",
      "Cost after iteration 2250: 0.021704\n",
      "Cost after iteration 2260: 0.021523\n",
      "Cost after iteration 2270: 0.021343\n",
      "Cost after iteration 2280: 0.021164\n",
      "Cost after iteration 2290: 0.020986\n",
      "Cost after iteration 2300: 0.020809\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cost after iteration 2310: 0.020634\n",
      "Cost after iteration 2320: 0.020459\n",
      "Cost after iteration 2330: 0.020286\n",
      "Cost after iteration 2340: 0.020115\n",
      "Cost after iteration 2350: 0.019944\n",
      "Cost after iteration 2360: 0.019775\n",
      "Cost after iteration 2370: 0.019607\n",
      "Cost after iteration 2380: 0.019440\n",
      "Cost after iteration 2390: 0.019275\n",
      "Cost after iteration 2400: 0.019111\n",
      "Cost after iteration 2410: 0.018949\n",
      "Cost after iteration 2420: 0.018787\n",
      "Cost after iteration 2430: 0.018628\n",
      "Cost after iteration 2440: 0.018469\n",
      "Cost after iteration 2450: 0.018313\n",
      "Cost after iteration 2460: 0.018157\n",
      "Cost after iteration 2470: 0.018003\n",
      "Cost after iteration 2480: 0.017851\n",
      "Cost after iteration 2490: 0.017700\n",
      "Cost after iteration 2500: 0.017551\n",
      "Cost after iteration 2510: 0.017403\n",
      "Cost after iteration 2520: 0.017257\n",
      "Cost after iteration 2530: 0.017112\n",
      "Cost after iteration 2540: 0.016969\n",
      "Cost after iteration 2550: 0.016827\n",
      "Cost after iteration 2560: 0.016687\n",
      "Cost after iteration 2570: 0.016548\n",
      "Cost after iteration 2580: 0.016410\n",
      "Cost after iteration 2590: 0.016274\n",
      "Cost after iteration 2600: 0.016139\n",
      "Cost after iteration 2610: 0.016006\n",
      "Cost after iteration 2620: 0.015873\n",
      "Cost after iteration 2630: 0.015743\n",
      "Cost after iteration 2640: 0.015613\n",
      "Cost after iteration 2650: 0.015485\n",
      "Cost after iteration 2660: 0.015358\n",
      "Cost after iteration 2670: 0.015232\n",
      "Cost after iteration 2680: 0.015108\n",
      "Cost after iteration 2690: 0.014984\n",
      "Cost after iteration 2700: 0.014862\n",
      "Cost after iteration 2710: 0.014741\n",
      "Cost after iteration 2720: 0.014621\n",
      "Cost after iteration 2730: 0.014502\n",
      "Cost after iteration 2740: 0.014383\n",
      "Cost after iteration 2750: 0.014266\n",
      "Cost after iteration 2760: 0.014150\n",
      "Cost after iteration 2770: 0.014035\n",
      "Cost after iteration 2780: 0.013920\n",
      "Cost after iteration 2790: 0.013807\n",
      "Cost after iteration 2800: 0.013695\n",
      "Cost after iteration 2810: 0.013584\n",
      "Cost after iteration 2820: 0.013474\n",
      "Cost after iteration 2830: 0.013365\n",
      "Cost after iteration 2840: 0.013257\n",
      "Cost after iteration 2850: 0.013150\n",
      "Cost after iteration 2860: 0.013043\n",
      "Cost after iteration 2870: 0.012938\n",
      "Cost after iteration 2880: 0.012834\n",
      "Cost after iteration 2890: 0.012731\n",
      "Cost after iteration 2900: 0.012629\n",
      "Cost after iteration 2910: 0.012528\n",
      "Cost after iteration 2920: 0.012428\n",
      "Cost after iteration 2930: 0.012329\n",
      "Cost after iteration 2940: 0.012231\n",
      "Cost after iteration 2950: 0.012134\n",
      "Cost after iteration 2960: 0.012037\n",
      "Cost after iteration 2970: 0.011942\n",
      "Cost after iteration 2980: 0.011847\n",
      "Cost after iteration 2990: 0.011753\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcFPWZ+PHP09cczAEDAyKHEKMi14BcxhMFFE2CMYaY\neLNRYjaa/W12/Umia4iJv5eJZtc1alzM4pF1IxpvxTNKNPFiIIqCIKjIDQMDc8AcfTy/P6q6p2eY\nowemp2a6n3dela761reqn5qWeupbx7dEVTHGGGMAfF4HYIwxpuewpGCMMSbBkoIxxpgESwrGGGMS\nLCkYY4xJsKRgjDEmwZKCyUoi8oKIXO51HMb0NJYUTLcSkY0iMtPrOFT1HFV90Os4AERkmYhc2Q3f\nkyMii0WkWkR2iMiP26k7WESeEZFtIqIiMiLd8ZmewZKCyTgiEvA6hrieFAuwEDgGOAo4A/i/IjK7\njbox4EXggu4JzfQUlhRMjyEiXxOR90Vkn4i8JSLjk+YtEJFPRaRGRNaIyPlJ864Qkb+JyH+IyB5g\noVv2VxG5XUT2isjnInJO0jKJo/MU6o4UkTfc735VRO4Wkf9pYxumi8gWEbleRHYA94tIPxF5TkQq\n3PU/JyJD3fq3AKcCd4lIrYjc5ZaPEpFXRKRSRNaJyLe74E98OfALVd2rqh8Di4ArWquoqjtV9R5g\neRd8r+lFLCmYHkFEJgKLge8D/YH/Ap4RkRy3yqc4O89i4OfA/4jI4KRVTAM+AwYBtySVrQMGAL8G\n/ltEpI0Q2qv7v8B7blwLgUs72JwjgBKcI/L5OP/O7nenhwN1wF0AqnoD8CZwjaoWqOo1ItIHeMX9\n3oHAd4B7RGR0a18mIve4ibS1YZVbpx8wGPggadEPgDEdbIvJMpYUTE8xH/gvVX1XVaPu+f4G4EQA\nVX1MVbepakxVlwDrgalJy29T1d+qakRV69yyL1T1PlWNAg/i7BQHtfH9rdYVkeHAFOAmVW1U1b8C\nz3SwLTHgZ6raoKp1qrpHVR9X1QOqWoOTtE5vZ/mvARtV9X53e/4OPA7Mba2yqv6jqvZtY4i3tgrc\nz6qkRauBwg62xWQZSwqmpzgK+Jfko1xgGHAkgIhclnRqaR8wFueoPm5zK+vcER9R1QPuaEEr9dqr\neyRQmVTW1nclq1DV+viEiOSLyH+JyBciUg28AfQVEX8byx8FTGvxt7gYpwVyqGrdz6KksmKg5jDW\naTKQJQXTU2wGbmlxlJuvqn8UkaOA+4BrgP6q2hf4CEg+FZSu7n63AyUikp9UNqyDZVrG8i/AccA0\nVS0CTnPLpY36m4G/tPhbFKjqD1r7MhG5170e0dqwGkBV97rbUpa0aBmwuoNtMVnGkoLxQlBEcpOG\nAM5O/2oRmSaOPiLyVREpBPrg7DgrAERkHk5LIe1U9QugHOfidUhEvgJ8vZOrKcS5jrBPREqAn7WY\nvxP4UtL0c8CxInKpiATdYYqIHN9GjFe7SaO1IfmawUPAje6F7+OBq4AH2gpaRHKB+DWdHHfaZDhL\nCsYLS3F2kvFhoaqW4+yk7gL2Ahtw74xR1TXAb4C3cXag44C/dWO8FwNfAfYAvwSW4FzvSNUdQB6w\nG3gH51bPZP8JfMu9M+lO97rDWTgXmLfhnNr6FU076EP1M5wL9l8Ay4Bfq2oiFrdlcWpS/TqaTjut\ndadNhhN7yY4xnSMiS4C1qtryiN+YXs9aCsZ0wD11c7SI+MR52Os84Cmv4zImHXrS05bG9FRHAE/g\nPKewBfiBe5uoMRnHTh8ZY4xJsNNHxhhjEnrd6aMBAwboiBEjvA7DGGN6lRUrVuxW1dKO6vW6pDBi\nxAjKy8u9DsMYY3oVEfkilXp2+sgYY0yCJQVjjDEJlhSMMcYkWFIwxhiTkLakIM67YHeJyEdtzBcR\nuVNENojIKhE5IV2xGGOMSU06WwoPAG29/xXgHJz3xR6D84KV36UxFmOMMSlIW1JQ1TeAynaqnAc8\npI53cF46Mrid+sYYY9LMy+cUhtD8DVZb3LLt3oRjjOntYtEI0VgjsWiYaCx80Gc02ugODUSiTdOR\nWCPRaJhoNNw0HgsT0xiqUaKxKDGNorEYUY2gGiOmMaKxqDseTZQ5y7if8f/Fx1WbPpPGY8RANak+\nxDQGiWUgRoxJQ0/jpCk/TOvfsFc8vCYi83FOMTF8+HCPozHGOxqLEYtFnJ1auI5ItIFItIFopIFI\npJ5ItJ5INEwkUu/s7KINRGJhIpFGorFGIlF3iDk7wJhGicQixGJRohoh6o5H3PGoO+7s5JSYRomp\nEo3vBNHEjjCmMSIaJRrfYSbPdz+j7o4vqkqUpDpuWQwlEp+HEgG3rhIFIu5nVHDmxQeBiEh7f7pe\nT1SJxqIZnRS20vy1hkPdsoOo6iJgEcDkyZOtB79OikUjxGIRNBYlps4/eiWGujsA5wjI+UceDtcR\njhxwP+sIR+oTQ2PkgPvpDtF6orGIe9TTdGQUP8KJuTsSknYc8aMmZwcT33lE3fE2digtpqPueqLu\nEVbzHYxDk95wGT/ScspJGtdEWXw63j+ktvj/ZnVo/v5MbaVeDJxtdHdiMcD5qzs7MRWnTGn6PGhc\nWpT3oJ2eTxUfzvlnn5IY98cHjU8L4pY5dcT5FMGPM/gQ/OJ8BsVHrgg+fATEGfzu0DTuxy9+AuJ3\npn1+/Pjw+wL4xYfPLfeJH7/Pj88dD/gCTh1fgIAv2HzcH8TvlgX8ocS4zxfAJz7E58cvAcTnd9br\nDyLijPt8Pny+ID7xu/MD+Hx+EEHw4fP5EfEhIuDGEp8WfIgv4IyLzxlwvi8xLT7E1303inqZFJ4B\nrhGRR4BpQJWqZsSpo0i4nv0HdlFbu4PaAxXUHthNbd0equsqqa7fQ1X9Puqi9YkdYCSxU4wRcXeS\nUbc87JZFcOeh7rQSThxVKRH3qCriHkElj/eknUlr4juYph1J006meXnyDqZp5xJw5znzhfjmils/\nLj4lkBgTcP6xxutI0zKStEyz5aX5ulvWceaDD5+zsxMffnzOpzvtw9kJ+HCmnTicT19ihyGJeuLu\nKH3iRBXw+Qn4gokdXXw84Au5O7YgAX/IKXd3cgF/TqLc7w4BfwifBPD7A/j9Ofh8AWeHGMhx6+fg\n8wfx+0LODs8XwOfvFScYzCFK268rIn8EpgMDRGQLzqsAgwCqei/OKxnPxXnt4gFgXrpi6UpV+zay\nZcdKfOJnw7blVNRuo6axmo0HdrAuXMVuUep8He+Ec2LadFSFs+OLj/sQgkAAISBCAB8BEfz4CImP\nfAkQFD9+EQLuEZMz+JxPX8A5ivI1jfvEjy9+NOLuiJrtfMQHCEF/iKA/RMifS8AfJBTII+jPIeAP\nEQrkEQrmEQzmEwr2IRTMx+/uVOJHRj5fwFmfz4+4R0Q+n985GnKPmnwSAJ+4ywW69SjIGNO+tCUF\nVf1uB/MVSO/JsS727Os38vONT9HQYqfvV2VYTDg+1JcjckooCBZQECqgIKeYwpx+9MnrR2Fefwry\nB9K3eDgFBYMJBO0d6MaYnsfagSnatOmv3PDFU0ySXC45Zi7RaJijBk1g2JFTycsfYEe7xpiMYEkh\nRY8v/w98wK/OeYCBg8Z6HY4xxqSFHd6mIBKu56nqdZzuL7aEYIzJaJYUUvDFpjep9Akzh073OhRj\njEkrSwop2LDtHQC+fORUjyMxxpj0sqSQgk93r8Gnysjhp3kdijHGpJUlhRRsqN3C0JiQm9fP61CM\nMSatLCmk4NNwFUcHirwOwxhj0s6SQgfCDfvZ5Ivx5YIhXodijDFpZ0mhA5u3vUtEhJH9jvU6FGOM\nSTtLCh3YtPPvABw1cJzHkRhjTPpZUujAF3vWATDcbkc1xmQBSwod2FyzmaKY0rffSK9DMcaYtLOk\n0IEvGvYw3Onx2xhjMp4lhQ5sjtYxPFTsdRjGGNMtrJfUFjQW46nXrmdd5VrmjP8e233K1/MHex2W\nMcZ0C0sKLbz29q+5aeuL+FV5+N1/Q4CpI2Z6HZYxxnQLO33UwoPrH2NIFB475XbKYkFuHXkBUyZ+\nz+uwjDGmW1hLIcmnn77C36WR6wefzDFfns3/fHm21yEZY0y3spZCkhWfPg/A6WMv9zgSY4zxhiWF\nJB/sXkX/qDJ0yDSvQzHGGE9YUkjyfn0FZcFixGd/FmNMdrK9n2vP7k/Y5IcJ/UZ5HYoxxnjGkoLr\n489eAmDs0JM9jsQYY7xjScG1budKAI4dac8kGGOylyUF17qqzxgcVYqLh3sdijHGeMaSgmt9416O\ns1duGmOynCUFoKG+is99MY4pGOp1KMYY4ylLCsDnX/yFqAjHDhjrdSjGGOMpSwrAtj1rARhWOsbj\nSIwxxluWFICd1ZsAGGRJwRiT5SwpADv2byegSkm/L3sdijHGeCqtSUFEZovIOhHZICILWplfLCLP\nisgHIrJaROalM5627KyvZFBM8Pmt01hjTHZLW1IQET9wN3AOMBr4roiMblHth8AaVS0DpgO/EZFQ\numJqy85wDYN83f61xhjT46SzpTAV2KCqn6lqI/AIcF6LOgoUiogABUAlEEljTK3aEWtgUKCgu7/W\nGGN6nHQmhSHA5qTpLW5ZsruA44FtwIfAP6lqrOWKRGS+iJSLSHlFRUWXBqmxGDt9yhG5JV26XmOM\n6Y28vtB8NvA+cCQwAbhLRA56rFhVF6nqZFWdXFpa2qUBVFZuICzCoD6Du3S9xhjTG6UzKWwFhiVN\nD3XLks0DnlDHBuBzoFv7rt65ew0AgwqHdVDTGGMyXzqTwnLgGBEZ6V48/g7wTIs6m4AZACIyCDgO\n+CyNMR1kb80WAPoXWVIwxpi03YOpqhERuQZ4CfADi1V1tYhc7c6/F/gF8ICIfAgIcL2q7k5XTK2p\nOuBcoygutNNHxhiT1hvzVXUpsLRF2b1J49uAs9IZQ0eq6vYAUFTY8hq4McZkH68vNHuuqmEfgL1H\nwRhjsKRAVWM1fWJKMJjvdSjGGOO5rE8K1eFailW8DsMYY3qErE8KVZE6isXvdRjGGNMjWFKINVDk\nC3odhjHG9AiWFGIRiv25XodhjDE9giUFiVEc6ON1GMYY0yNkdVLQWIwqgeKg9ZBqjDGQ5Ulh//6d\nREUozunrdSjGGNMjZHVSqKp2evYuzrNus40xBrI9KdRsA6A4b4DHkRhjTM+Q3Ulh/w4AivsM8jgS\nY4zpGbI8KewCoLjAekg1xhjI8qRQ7faQWmw9pBpjDJDmrrN7ko+3V/Pk37eSG/STH3KGLyq2A/B5\ndQH+Po2U9Al5HKUxxngra5LCxt37+cPbX1AXjibKTjuigtxi5du//wj4iC8N6MNF04Zz2VdGEApk\ndSPKGJOlRFW9jqFTJk+erOXl5Ye8fCym1EeiHGiMcvtT51LeuJMFJ/6Zz3bX8uqaXby3sZJTjxnA\noksnkxeyjvKMMZlBRFao6uSO6mXd4bDPJ+SHAgwoyGF/rI6+4mfm6EHMP+1oHr36K/zqgnG8uX43\n973Zra+KNsaYHiHrkkKy6lgDxdK8h9QLpwxn5vEDWfy3z9nfEPEoMmOM8UZWJ4WqWLjVHlJ/MP1o\n9h0I8+JHOzyIyhhjvJPdSYEYRYGDX8M5YVg/8kN+Ptxa5UFUxhjjnaxNChqLUeWD4lDhQfP8PmHU\nEYWs2V7tQWTGGOOdrE0KdfWVhEUoDhW3On/0kUV8vK2a3nZ3ljHGHI6sTQrVVW4Pqbn9Wp0/enAx\nNQ0Rtuyt686wjDHGU1mbFKpqtgJt95A6+sgiAFZvs1NIxpjskb1JIdFD6sBW5w/rlwfAjiprKRhj\nskcWJwW3h9TCI1udX5znPL+wry7cbTEZY4zXsjcp1O0G2u4hNeD3UZgbYN8BSwrGmOyRtUlhX30l\nAMVFw9qs0zc/yL4Djd0VkjHGeC5rekltqbqhipyYkpvX+t1HAH3zQnb6yBggHA6zZcsW6uvrvQ7F\ndCA3N5ehQ4cSDAY7rtyKrE0KVeFaijt4BMFpKVhSMGbLli0UFhYyYsQIRMTrcEwbVJU9e/awZcsW\nRo4ceUjrSOvpIxGZLSLrRGSDiCxoo850EXlfRFaLyF/SGU+yqsgBiqT9rrH75oeospaCMdTX19O/\nf39LCD2ciNC/f//DatGlraUgIn7gbmAWsAVYLiLPqOqapDp9gXuA2aq6SURavz80DaqiB/eQ2lLf\nPLumYEycJYTe4XB/p3S2FKYCG1T1M1VtBB4BzmtR5yLgCVXdBKCqu9IYTzNVGqbYn9Nunb75Qarq\nwsRi1tWFMSY7pDMpDAE2J01vccuSHQv0E5FlIrJCRC5rbUUiMl9EykWkvKKiokuCq9IofVvpITVZ\ncV6QmEJNvb1XwZje7v3332fp0qVeh9HjeX1LagCYBHwVOBv4NxE5tmUlVV2kqpNVdXJpaWmXfHG1\nQHGwoN06/fJDAOyrs1NIxvR2lhRSk86ksBVIfghgqFuWbAvwkqruV9XdwBtAWRpjAqC+bi/1PqEo\np/UeUuP65rtPNdsdSMb0CA899BDjx4+nrKyMSy+9lI0bN3LmmWcyfvx4ZsyYwaZNmwB47LHHGDt2\nLGVlZZx22mk0NjZy0003sWTJEiZMmMCSJUs83pKeK523pC4HjhGRkTjJ4Ds41xCSPQ3cJSIBIARM\nA/4jjTEBUF3tdoaX0/YzCpCUFOwOJGMSfv7satZ0cUeRo48s4mdfH9NundWrV/PLX/6St956iwED\nBlBZWcnll1+eGBYvXsyPfvQjnnrqKW6++WZeeuklhgwZwr59+wiFQtx8882Ul5dz1113dWnsmSZt\nLQVVjQDXAC8BHwOPqupqEblaRK5263wMvAisAt4Dfq+qH6Urpriqmi0AFOf1b7decZ57+sjuQDLG\nc6+99hpz585lwACnZ+OSkhLefvttLrrIOda89NJL+etf/wrAySefzBVXXMF9991HNBr1LObeKK0P\nr6nqUmBpi7J7W0zfBtyWzjha2le7DWi7h9S4ojznz1NtF5qNSejoiL4nuPfee3n33Xd5/vnnmTRp\nEitWrPA6pF7D6wvNnqh036XQr51+jwDyQ05SqGu0pGCM184880wee+wx9uzZA0BlZSUnnXQSjzzy\nCAAPP/wwp556KgCffvop06ZN4+abb6a0tJTNmzdTWFhITU2NZ/H3FlnZzUVFrZMUBvYf1W69vKDz\nxPOBRmt+GuO1MWPGcMMNN3D66afj9/uZOHEiv/3tb5k3bx633XYbpaWl3H///QBcd911rF+/HlVl\nxowZlJWVMXz4cG699VYmTJjAT37yEy688EKPt6hnys6kcKCCoCrFxUe1W8/vE3ICPuosKRjTI8Qv\nKid77bXXDqr3xBNPHFRWUlLC8uXL0xZbpsjK00cV9XspjQni63jz80N+6sKWFIwx2SE7k0KklgEd\n9HsUlxf02+kjY0zWSCkpiMjcVMp6i4poPQM76OIiLi/kt9NHxpiskWpL4ScplvUKFRJjQKgopbr5\noQAH7O4jY0yWaPdCs4icA5wLDBGRO5NmFQG9ck9ZX7eXap8wMG9ASvXz7JqCMSaLdHT30TagHJgD\nJD/9UQP8c7qCSqfde9YBMKDPESnVzw/52bvfnmg2xmSHdk8fqeoHqvog8GVVfdAdfwbnPQl7uyXC\nLlZRuQGAgUXDU6pvF5qN8d6+ffu45557DmnZc889l3379nVxRJkr1WsKr4hIkYiUACuB+0Qk7R3X\npcO6be8CMHRgap2x5oUsKRjjtfaSQiTS/pnspUuX0rdv33SEdVhUlVgs5nUYB0k1KRSrajXwTeAh\nVZ0GzEhfWOmzbOd7DI/C8GEnp1Q/P+Sn3q4pGOOpBQsW8OmnnzJhwgSuu+46li1bxqmnnsqcOXMY\nPXo0AN/4xjeYNGkSY8aMYdGiRYllR4wYwe7du9m4cSPHH388V111FWPGjOGss86irq7uoO969tln\nmTZtGhMnTmTmzJns3LkTgNraWubNm8e4ceMYP348jz/+OAAvvvgiJ5xwAmVlZcyY4ewWFy5cyO23\n355Y59ixY9m4cSMbN27kuOOO47LLLmPs2LFs3ryZH/zgB0yePJkxY8bws5/9LLHM8uXLOemkkygr\nK2Pq1KnU1NRw2mmn8f777yfqnHLKKXzwwQdd+JdO/YnmgIgMBr4N3NClEXSj/bU7eE/3893Co1N6\ncA3idx9ZUjAm4YUFsOPDrl3nEePgnFvbnH3rrbfy0UcfJXaIy5YtY+XKlXz00UeMHDkSgMWLF1NS\nUkJdXR1TpkzhggsuoH//5j0hr1+/nj/+8Y/cd999fPvb3+bxxx/nkksuaVbnlFNO4Z133kFE+P3v\nf8+vf/1rfvOb3/CLX/yC4uJiPvzQ2fa9e/dSUVHBVVddxRtvvMHIkSOprKzscFPXr1/Pgw8+yIkn\nngjALbfcQklJCdFolBkzZrBq1SpGjRrFhRdeyJIlS5gyZQrV1dXk5eXxve99jwceeIA77riDTz75\nhPr6esrKuvYVNKkmhZtxusD+m6ouF5EvAeu7NJJusGL1EsIinHb0V1NeJjfo3H0Uiyk+n7243Jie\nYurUqYmEAHDnnXfy5JNPArB582bWr19/UFIYOXIkEyZMAGDSpEls3LjxoPVu2bKFCy+8kO3bt9PY\n2Jj4jldffTXR+R5Av379ePbZZznttNMSdUpKSjqM+6ijjkokBIBHH32URYsWEYlE2L59O2vWrEFE\nGDx4MFOmTAGgqMi5hX7u3Ln84he/4LbbbmPx4sVcccUVHX5fZ6WUFFT1MeCxpOnPgAu6PJo0213j\nvDJ62BEnpLxMfsjpFK8+Ek30mmpMVmvniL479enTJzG+bNkyXn31Vd5++23y8/OZPn069fX1By2T\nk5OTGPf7/a2ePrr22mv58Y9/zJw5c1i2bBkLFy7sdGyBQKDZ9YLkWJLj/vzzz7n99ttZvnw5/fr1\n44orrmg17rj8/HxmzZrF008/zaOPPpqWLsFTfaJ5qIg8KSK73OFxERna5dGk2d46p8vdvineeQRN\nScFOIRnjnY66va6qqqJfv37k5+ezdu1a3nnnnUP+rqqqKoYMGQLAgw8+mCifNWsWd999d2J67969\nnHjiibzxxht8/vnnAInTRyNGjGDlypUArFy5MjG/perqavr06UNxcTE7d+7khRdeAOC4445j+/bt\niQ78ampqEhfUr7zySn70ox8xZcoU+vVr/+2RhyLVC83349yKeqQ7POuW9SpVDfsIqZKXn9qDa9DU\nfbZ1dWGMd/r378/JJ5/M2LFjue666w6aP3v2bCKRCMcffzwLFixodnqmsxYuXMjcuXOZNGlS4i1v\nADfeeCN79+5NvPv59ddfp7S0lEWLFvHNb36TsrKyRHfcF1xwAZWVlYwZM4a77rqLY489ttXvKisr\nY+LEiYwaNYqLLrqIk092boAJhUIsWbKEa6+9lrKyMmbNmpVoQUyaNImioiLmzZt3yNvYHlHVjiuJ\nvK+qEzoq6w6TJ0/W8vLyQ1r23/53Jm/V7+DP/5D6Gz+fW7WNa/7377z8z6dx7KDCQ/peY3q7jz/+\nmOOPP97rMAywbds2pk+fztq1a/G1ccNMa7+XiKxQ1ckdrT/VlsIeEblERPzucAmwJ8Vle4x9kQP0\nFX+nlrHTR8aYnuKhhx5i2rRp3HLLLW0mhMOV6lr/Aed21B3AduBbwBVpiSiNqmIN9E2xy+y4vKBz\ncdk6xTPGeO2yyy5j8+bNzJ2bvk6qU00KNwOXq2qpqg7ESRI/T1tUabI3FqbYn9epZRJ3H9kDbMaY\nLJBqUhif3NeRqlYCE9MTUvpUSYx+wT4dV0xip4+MMdkk1aTgE5HEvU9uH0i96qb9WDRClUBxiu9R\niMsNWlIwxmSPVHfsvwHeFpH4A2xzgVvSE1J61NRsISZCv9zO3dfbJ8f5E9ktqcaYbJBSS0FVH8Lp\nDG+nO3xTVf+QzsC62r4q52nmvim+XCcufvqotsEuNBvjlcPpOhvgjjvu4MCBA10YUeZK+Z4mVV2j\nqne5w5p0BpUO+2q2AlCcP7BTy+UEfAR8YncfGeOhTEgKHXXx3VOk50bXHmhf7TYA+hUc2anlRIT8\nkJ/9DXb6yBivtOw6G+C2225jypQpjB8/PtHl9P79+/nqV79KWVkZY8eOZcmSJdx5551s27aNM844\ngzPOOOOgdd98881MmTKFsWPHMn/+fOIP9G7YsIGZM2dSVlbGCSecwKeffgrAr371K8aNG0dZWRkL\nFiwAYPr06cQfqt29ezcjRowA4IEHHmDOnDmceeaZzJgxg9raWmbMmMEJJ5zAuHHjePrppxNxPPTQ\nQ4wfP56ysjIuvfRSampqGDlyJOFwGHC6xEieTpdedbH4cAT8IY6J+Sjpe1Snly3ICdjpI2Ncv3rv\nV6ytXNul6xxVMorrp17f5vyWXWe//PLLrF+/nvfeew9VZc6cObzxxhtUVFRw5JFH8vzzzwNOP0bF\nxcX8+7//O6+//nqzbivirrnmGm666SYALr30Up577jm+/vWvc/HFF7NgwQLOP/986uvricVivPDC\nCzz99NO8++675Ofnp9RV9sqVK1m1ahUlJSVEIhGefPJJioqK2L17NyeeeCJz5sxhzZo1/PKXv+St\nt95iwIABVFZWUlhYyPTp03n++ef5xje+wSOPPMI3v/lNgsHOPWvVWVnTUjh5yjU8Me8DhgyZ2ull\n++QE2G9JwZge4+WXX+bll19m4sSJnHDCCaxdu5b169czbtw4XnnlFa6//nrefPNNiouLO1zX66+/\nzrRp0xg3bhyvvfYaq1evpqamhq1bt3L++ecDkJubS35+Pq+++irz5s0jPz8fSK2r7FmzZiXqqSo/\n/elPGT9+PDNnzmTr1q3s3LmT1157jblz5yaSVrz+lVdeyf33O93M3X///Wnr7yhZ1rQUDkd+ToD9\ndveRMQDtHtF3F1XlJz/5Cd///vcPmrdy5UqWLl3KjTfeyIwZMxKtgNbU19fzj//4j5SXlzNs2DAW\nLlzYbtfVbUnuKrvl8sldZT/88MNUVFSwYsUKgsEgI0aMaPf7Tj75ZDZu3MiyZcuIRqOMHTu207F1\nVta0FA4+ugOoAAAU3UlEQVRHQY7fWgrGeKhl19lnn302ixcvpra2FoCtW7eya9cutm3bRn5+Ppdc\ncgnXXXddovvqtrreju+QBwwYQG1tLX/6058S9YcOHcpTTz0FQENDAwcOHGDWrFncf//9iYvWyV1l\nx99tEF9Ha6qqqhg4cCDBYJDXX3+dL774AoAzzzyTxx57jD179jRbLzhdW1x00UXd0kqANCcFEZkt\nIutEZIOILGin3hQRiYjIt9IZz6HqE7LTR8Z4qWXX2WeddRYXXXQRX/nKVxg3bhzf+ta3qKmp4cMP\nP2Tq1KlMmDCBn//859x4440AzJ8/n9mzZx90oblv375cddVVjB07lrPPPjvxpjOAP/zhD9x5552M\nHz+ek046iR07djB79mzmzJnD5MmTmTBhQuI9zP/6r//K7373OyZOnMju3bvb3I6LL76Y8vJyxo0b\nx0MPPcSoUaMAGDNmDDfccAOnn346ZWVl/PjHP262zN69e/nud7/bZX/P9qTUdfYhrVjED3wCzAK2\nAMuB77a8ndWt9wpQDyxW1bbTLIfXdfah+vGS93lvYyV/vf7Mbv1eY3oK6zrbO3/60594+umn+cMf\nUn807HC6zk7nNYWpwAb31Z2IyCPAeUDLZxyuBR4HptBD5dvpI2OMB6699lpeeOEFli5d2m3fmc6k\nMATYnDS9BZiWXEFEhgDnA2fQTlIQkfnAfIDhw1N/lWZX6WMXmo0xHvjtb3/b7d/p9YXmO4DrVTXW\nXiVVXaSqk1V1cmlpaTeF1qQgFKAxEiMcbTdMYzJauk41m651uL9TOlsKW4FhSdND3bJkk4FHRARg\nAHCuiERU9ak0xtVp8U7x9jdE6Jsf8jgaY7pfbm4ue/bsoX///rj/Xk0PpKrs2bOH3NzcQ15HOpPC\ncuAYERmJkwy+A1yUXEFVR8bHReQB4LmelhAA+uQ0dYpnScFko6FDh7JlyxYqKiq8DsV0IDc3l6FD\nhx7y8mlLCqoaEZFrgJcAP86dRatF5Gp3/r3p+u6uFm8p2DsVTLYKBoOMHDmy44qm10vrE82quhRY\n2qKs1WSgqlekM5bDEU8K1v+RMSbTeX2huVfoE2q6pmCMMZnMkkIK4tcULCkYYzKdJYUUFCROH9k1\nBWNMZrOkkILkW1KNMSaTWVJIQYFdaDbGZAlLCinIDfrJCfiorkvva/CMMcZrlhRSVJQXpLrekoIx\nJrNZUkhRUW6A6jo7fWSMyWyWFFJkLQVjTDawpJCiotygXVMwxmQ8SwopcloKdvrIGJPZLCmkyLmm\nYC0FY0xms6SQovg1BXvRiDEmk1lSSFFRbpBwVKkP29vXjDGZy5JCiorynKeaq+wUkjEmg1lSSFFR\nbhDAbks1xmQ0SwopKspzk4K1FIwxGcySQoqKcp3TR9ZSMMZkMksKKSpOtBTsWQVjTOaypJCixOkj\naykYYzKYJYUUFcZPH9k1BWNMBrOkkKKcgJ/coM+6ujDGZDRLCp1gneIZYzKdJYVOsO6zjTGZzpJC\nJ9iLdowxmc6SQidYS8EYk+ksKXSCXVMwxmQ6SwqdUJQXsLuPjDEZzZJCJ8RbCvZOBWNMprKk0AlF\neUEiMaUuHPU6FGOMSQtLCp2Q6D7b7kAyxmSotCYFEZktIutEZIOILGhl/sUiskpEPhSRt0SkLJ3x\nHC570Y4xJtOlLSmIiB+4GzgHGA18V0RGt6j2OXC6qo4DfgEsSlc8XaHYOsUzxmS4dLYUpgIbVPUz\nVW0EHgHOS66gqm+p6l538h1gaBrjOWxNp48sKRhjMlM6k8IQYHPS9Ba3rC3fA15obYaIzBeRchEp\nr6io6MIQO8e6zzbGZLoecaFZRM7ASQrXtzZfVRep6mRVnVxaWtq9wSVJvH3NLjQbYzJUII3r3goM\nS5oe6pY1IyLjgd8D56jqnjTGc9jiLYV9B6ylYIzJTOlsKSwHjhGRkSISAr4DPJNcQUSGA08Al6rq\nJ2mMpUsE/T765gfZXdvgdSjGGJMWaWspqGpERK4BXgL8wGJVXS0iV7vz7wVuAvoD94gIQERVJ6cr\npq5QWpBDRY0lBWNMZkrn6SNUdSmwtEXZvUnjVwJXpjOGrlZamEOFtRSMMRmqR1xo7k1KC62lYIzJ\nXJYUOslOHxljMpklhU4aUJhDXTjK/ga7LdUYk3ksKXRSaUEOgLUWjDEZyZJCJ5UWuknBLjYbYzKQ\nJYVOSiQFaykYYzKQJYVOsqRgjMlklhQ6qV9+iFDAx5a9B7wOxRhjupwlhU7y+4RjBhawdkeN16EY\nY0yXs6RwCEYdUcQ6SwrGmAxkSeEQjDqikF01DeyxO5CMMRnGksIhGDW4EMBaC8aYjGNJ4RCMOqII\ngDXbqz2OxBhjupYlhUNQWpjDyAF9eH3dLq9DMcaYLmVJ4RB9bfxg3v50jz2vYIzJKJYUDtHXxh9J\nTOGZD7Z5HYoxxnQZSwqH6LgjCpk6soS7X99Alb2z2RiTISwpHIaFXx/DvgONXP/4KqIx9TocY4w5\nbJYUDsPoI4v46bnH8+LqHfzgf1ZYi8EY0+tZUjhMV576Jf7ta6N5be0uTvn1a/zHK5/YQ23GmF5L\nVHvXaY/JkydreXm512Ec5OPt1dzx6ie8tHonPoGpI0s4/diBTDqqH+OHFpMb9HsdojEmi4nIClWd\n3GE9Swpda/3OGp75YBsvfrSD9btqAQj4hC8PLODogQUcXVrAlwcWMKJ/PkcU5zKgTw4+n3gctTEm\n01lS6AH21Dbw9037WLlpL+t21LChopbNlQdIviYd8vsYVJzD4KI8BvfNZVBRLv3yQ/TvE6JfnxAl\nfYL0yw9R0idEUW7QEogx5pCkmhQC3RFMtupfkMPM0YOYOXpQoqw+HGXjnv1srqxje1Ud2/bVs6Oq\njm1V9azctJdd1Q00RGKtrs8nUJwXpDA3SEFOgILcAIU5AQpznfGCnKAznuMMfXIC5If85If85Ab9\n5IX85AXdIeQnJ+BDxJKMMaaJJYVulhv0M+qIokT/Sa2pa4xSeaCRytpGKg80snd/I5X7G9l7wBn2\nN0SpqQ9TUx9hR3U9Gyoi1NZHqGmI0NhGQmmNCIkk0TJp5Ib85AV95Aad5JET8BMK+BLjOUEfIb+P\nnKA77c4LJc1P1G1lOWvxGNMzWVLogfJCfoaE8hjSN6/TyzZEouxviLpJIkx9OMqBxih1jVHqwlHq\nw/HxGHWNEerCTnldY8yt65RV1YXZURWhIRKjMRKjIRKjIRylIRIj0gXPZAT90moyCTWbdj5Dbr1Q\ny8QTaEpMTZ/+pOUOrt98nh+/JSdjmrGkkGGcnZ+fkj6htH1HNKZuooi6ySJGYzRKfdhNHm55y2SS\nmG6xXEPScvE6jZEYNfVOy6cx6qzD+YzREI11qkXUHr9PDk5C/tYSVGuJqXkya56g/E0tKX9HSc9Z\nxk7lmZ7AkoLpNL9PnFNNIe9us1VVGqOxZkmk6dNJLk1J6OCEE08w8aR08LqiiWVr6iPsiTQ660gs\n1/TZVU+zN52eayvROJ/NWj5BX+L0X9OQXOZrKg84v1lu0JcYt+tKpiVLCqZXEpFEq6jQ41gi0baT\nSrtJq1mCidKQNN1Uv2k99eEY1XWRZuuvDzsttPpIlEO9kTAn4HOSRVKiyE1ca2oaz2kl4TSVN9Vv\nNSEF/eQGfAT89rxsT2dJwZjDFPA7O7v89J2x65CqJk7J1Uec60b1ESdhxMcbwvHrSk4yiY83lbv1\n3fGGcIzdtY1OecS57tTgjoejh5aBgn4hN+DcyBBvsSRuZnBvWshtcfNCTrApUbW8aaG1mxpari++\njLWIUmNJwZgMICKJI/Jigmn/vkg0Rr3bUqlrjNIQaZ5QmieX1ufFk1FjJOYmLaclFF9X8rWnw2kJ\nxbVMMvHTc6GAj6DfR9AvBP1OHWfaGUIBIRSfdueF3LqhxLSPYECSxn1Ny/gl8V2JdfjEPZgQgj7n\nM+CTHpG40poURGQ28J+AH/i9qt7aYr64888FDgBXqOrKdMZkjDl8Ab+PAr+PgpzuOa5UVcJRbXat\nJ55sksvqw1GSb25oVhaJ39TgLNcYjRGOxAhHY4Sjzs0TNeEIldHmZY3x6YhbFu2amxxaE/BJ80Th\nb55ALpo6nCtP/VLavh/SmBRExA/cDcwCtgDLReQZVV2TVO0c4Bh3mAb8zv00xpgEEXGO2AM+z68h\nxRNU2E0WjUkJJOxeDwq3LEskmvh8JRJ1bu+Oj4djyWUxIlElEos1m19amJP27Utnmp8KbFDVzwBE\n5BHgPCA5KZwHPKROXxvviEhfERmsqtvTGJcxxhyy5ASVidK5VUOAzUnTW9yyztZBROaLSLmIlFdU\nVHR5oMYYYxy9ItWp6iJVnayqk0tLS70OxxhjMlY6k8JWYFjS9FC3rLN1jDHGdJN0JoXlwDEiMlJE\nQsB3gGda1HkGuEwcJwJVdj3BGGO8k7YLzaoaEZFrgJdwbkldrKqrReRqd/69wFKc21E34NySOi9d\n8RhjjOlYWm8yVtWlODv+5LJ7k8YV+GE6YzDGGJO6XnGh2RhjTPewpGCMMSah172jWUQqgC8OcfEB\nwO4uDMdLti09k21Lz2TbAkepaof39Pe6pHA4RKQ8lRdX9wa2LT2TbUvPZNuSOjt9ZIwxJsGSgjHG\nmIRsSwqLvA6gC9m29Ey2LT2TbUuKsuqagjHGmPZlW0vBGGNMOywpGGOMSciapCAis0VknYhsEJEF\nXsfTWSKyUUQ+FJH3RaTcLSsRkVdEZL372c/rOFsjIotFZJeIfJRU1mbsIvIT93daJyJnexN169rY\nloUistX9bd4XkXOT5vXIbRGRYSLyuoisEZHVIvJPbnmv+13a2Zbe+Lvkish7IvKBuy0/d8u773dR\n1YwfcDrk+xT4EhACPgBGex1XJ7dhIzCgRdmvgQXu+ALgV17H2UbspwEnAB91FDsw2v19coCR7u/m\n93obOtiWhcC/tlK3x24LMBg4wR0vBD5x4+11v0s729IbfxcBCtzxIPAucGJ3/i7Z0lJIvBpUVRuB\n+KtBe7vzgAfd8QeBb3gYS5tU9Q2gskVxW7GfBzyiqg2q+jlOD7pTuyXQFLSxLW3psduiqttVdaU7\nXgN8jPPWw173u7SzLW3pyduiqlrrTgbdQenG3yVbkkJKr/3s4RR4VURWiMh8t2yQNr1/YgcwyJvQ\nDklbsffW3+paEVnlnl6KN+17xbaIyAhgIs5Raa/+XVpsC/TC30VE/CLyPrALeEVVu/V3yZakkAlO\nUdUJwDnAD0XktOSZ6rQle+X9xb05dtfvcE5NTgC2A7/xNpzUiUgB8Djwf1S1Onleb/tdWtmWXvm7\nqGrU/bc+FJgqImNbzE/r75ItSaHXv/ZTVbe6n7uAJ3GaiDtFZDCA+7nLuwg7ra3Ye91vpao73X/I\nMeA+mprvPXpbRCSIsxN9WFWfcIt75e/S2rb01t8lTlX3Aa8Ds+nG3yVbkkIqrwbtsUSkj4gUxseB\ns4CPcLbhcrfa5cDT3kR4SNqK/RngOyKSIyIjgWOA9zyIL2Xxf6yu83F+G+jB2yIiAvw38LGq/nvS\nrF73u7S1Lb30dykVkb7ueB4wC1hLd/4uXl9t764B57Wfn+Bcnb/B63g6GfuXcO4w+ABYHY8f6A/8\nGVgPvAqUeB1rG/H/Eaf5HsY55/m99mIHbnB/p3XAOV7Hn8K2/AH4EFjl/iMd3NO3BTgF5xTEKuB9\ndzi3N/4u7WxLb/xdxgN/d2P+CLjJLe+238W6uTDGGJOQLaePjDHGpMCSgjHGmARLCsYYYxIsKRhj\njEmwpGCMMSbBkoLpMUTkLfdzhIhc1MXr/mlr35UuIvINEbkpTev+ace1Or3OcSLyQFev1/Q+dkuq\n6XFEZDpO75Zf68QyAVWNtDO/VlULuiK+FON5C5ijqrsPcz0HbVe6tkVEXgX+QVU3dfW6Te9hLQXT\nY4hIvHfIW4FT3T7w/9ntIOw2EVnudm72fbf+dBF5U0SeAda4ZU+5nQaujnccKCK3Annu+h5O/i5x\n3CYiH4nzvooLk9a9TET+JCJrReRh98lZRORWcfruXyUit7eyHccCDfGEICIPiMi9IlIuIp+IyNfc\n8pS3K2ndrW3LJeL0wf++iPyXiPjj2ygit4jTN/87IjLILZ/rbu8HIvJG0uqfxXna32Qzr5/gs8GG\n+ADUup/TgeeSyucDN7rjOUA5Tt/x04H9wMikuiXuZx7OE6H9k9fdynddALyC886NQcAmnP75pwNV\nOH3J+IC3cZ6c7Y/z5Gi8ld23le2YB/wmafoB4EV3PcfgPAmd25ntai12d/x4nJ150J2+B7jMHVfg\n6+74r5O+60NgSMv4gZOBZ73+78AGb4dAqsnDGA+dBYwXkW+508U4O9dG4D11+pGP+5GInO+OD3Pr\n7Wln3acAf1TVKE6nY38BpgDV7rq3AIjTlfEI4B2gHvhvEXkOeK6VdQ4GKlqUPapOx2zrReQzYFQn\nt6stM4BJwHK3IZNHU2dpjUnxrcDpRwfgb8ADIvIo8ETTqtgFHJnCd5oMZknB9AYCXKuqLzUrdK49\n7G8xPRP4iqoeEJFlOEfkh6ohaTwKBFQ1IiJTcXbG3wKuAc5ssVwdzg4+WcuLd0qK29UBAR5U1Z+0\nMi+sqvHvjeL+e1fVq0VkGvBVYIWITFLVPTh/q7oUv9dkKLumYHqiGpzXKsa9BPxAnO6REZFj3d5i\nWyoG9roJYRTOawzjwvHlW3gTuNA9v1+K87rNNnuZFKfP/mJVXQr8M1DWSrWPgS+3KJsrIj4RORqn\ng8N1ndiulpK35c/At0RkoLuOEhE5qr2FReRoVX1XVW/CadHEu14+lqaeRE2WspaC6YlWAVER+QDn\nfPx/4py6Wele7K2g9VePvghcLSIf4+x030matwhYJSIrVfXipPInga/g9ECrwP9V1R1uUmlNIfC0\niOTiHKX/uJU6bwC/ERFJOlLfhJNsioCrVbVeRH6f4na11GxbRORG4GUR8eH03vpD4It2lr9NRI5x\n4/+zu+0AZwDPp/D9JoPZLanGpIGI/CfORdtX3fv/n1PVP3kcVptEJAf4C84b/tq8tddkPjt9ZEx6\n/D8g3+sgOmE4sMASgrGWgjHGmARrKRhjjEmwpGCMMSbBkoIxxpgESwrGGGMSLCkYY4xJ+P+Dq66m\n9hPX3gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2158f42fe48>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 6min\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "layer_dims = [train_x.shape[0],128,128,1]\n",
    "activation = [\"relu\",\"relu\",\"sigmoid\"]\n",
    "\n",
    "parameters = L_layer_model_L2(X=train_x,Y=train_y, \n",
    "                              layer_dims=layer_dims,\n",
    "                              activation=activation,\n",
    "                              initialization='random',\n",
    "                              keep_prob=1,\n",
    "                              mini_batch_size = 8431,\n",
    "                              test_x=test_x, test_y=test_y,\n",
    "                              learning_rate = .1, lambd=0, \n",
    "                              num_iterations = 3000, print_cost = True,\n",
    "                              optimizer='gd')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Ones = 4212.0\n",
      "Accuracy: 0.997034752698\n"
     ]
    }
   ],
   "source": [
    "predict(train_x, train_y, parameters,activation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Ones = 475.0\n",
      "Accuracy: 0.987179487179\n"
     ]
    }
   ],
   "source": [
    "predict(test_x, test_y, parameters,activation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### This took almost 6.5 minutes. However, there is a one percent discrepancy between the training accuracy and testing accuracy. There is a sign of slight overfitting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ALL METHODS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cost after iteration 0: 0.693158\n",
      "Cost after iteration 10: 0.049765\n",
      "Cost after iteration 20: 0.039508\n",
      "Cost after iteration 30: 0.016118\n",
      "Cost after iteration 40: 0.016076\n",
      "Cost after iteration 50: 0.015660\n",
      "Cost after iteration 60: 0.012820\n",
      "Cost after iteration 70: 0.013739\n",
      "Cost after iteration 80: 0.010329\n",
      "Cost after iteration 90: 0.009763\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VPW5+PHPM5ONQFiSoCJrSlSQJSxhkaBFEQV6S2uV\ntq7F61Lrdn/XW6+09Wetrb9XW9v7q7u11q31V/e6ErcqpYooAZFNlIAgAZGEJQRClsk8vz/OmWGy\nkYWcnEnmeb9e48w553vOPGci85xzvme+j6gqxhhjDEDA7wCMMcbED0sKxhhjoiwpGGOMibKkYIwx\nJsqSgjHGmChLCsYYY6IsKZhuSUQKReQHfsdhTFdjScF0KBHZIiJn+h2Hqs5R1cf8jgNARBaLyOWd\n8D6pIvKwiOwXkZ0ickML7S8Qka0iclBEXhCRzNZuS0QeFJFPRSQsIgs82iXjA0sKpssRkSS/Y4iI\np1iAW4ETgKHA6cB/i8jsphqKyCjgj8DFwLFAJXBfG7b1MXA1sLJD98D4zpKC6TQi8m8iskpE9onI\nUhEZG7NsoYhsEpEKEVkvIufELFsgIu+JyP8Vkd3Are68d0XkdyKyV0Q+F5E5MetEj85b0TZHRJa4\n7/2WiNwrIn9tZh9miEiJiNwkIjuBR0Skn4i8IiKl7vZfEZFBbvvbgVOBe0TkgIjc484fISJvisge\n94j7ux3wEf8A+KWq7lXVT4AHgQXNtL0QeFlVl6jqAeB/A98RkYzWbEtV71XVfwBVHRC3iSOWFEyn\nEJHxwMPAD4EsnKPUl0Qk1W2yCefLsw/wC+CvIjIgZhNTgM04R7W3x8z7FMgGfgv8WUSkmRCO1Pb/\nAR+6cd2Kc/R8JMcBmThH0Vfi/Dt6xJ0eAhwC7gFQ1Z8B/wKuVdVeqnqtiPQE3nTf9xjg+8B9InJy\nU28mIve5ibSpx2q3TT9gAM4RfMTHwKhm9mFUbFtV3QRUAye2Y1umG7GkYDrLlcAfVfUDVa1zr/dX\nA1MBVPUZVd2hqmFVfQrYCEyOWX+Hqt6tqiFVPeTO26qqf1LVOuAxnC+yY5t5/ybbisgQYBJwi6rW\nqOq7wEst7EsY+LmqVqvqIVXdrarPqWqlqlbgJK2vH2H9fwO2qOoj7v58BDwHzG+qsaperap9m3lE\nzrZ6uc/lMavuBzJoWq8GbWPbt3VbphuxpGA6y1Dgv2KPcoHBwPEAInJJzKWlfcBonKP6iG1NbHNn\n5IWqVrovezXR7khtjwf2xMxr7r1ilapq9LKJiKSLyB/dTtv9wBKgr4gEm1l/KDClwWdxIc4ZSHsd\ncJ97x8zrA1QcoX3vBvMi7du6LdONWFIwnWUbcHuDo9x0Vf2biAwF/gRcC2Spal9gLRB7Kcir4Xy/\nBDJFJD1m3uAW1mkYy38BJwFTVLU3cJo7X5ppvw34Z4PPopeq/qipNxORB9z+iKYe6wBUda+7L3kx\nq+YB65rZh3WxbUVkOJACfNaObZluxJKC8UKyiKTFPJJwvvSvEpEp4ugpIt9wOzZ74nxxlgKIyKU4\nZwqeU9WtQBFO53WKiJwCfLONm8nA6UfYJ85tnT9vsPwr4Gsx06/gXLu/WESS3cckERnZTIxXuUmj\nqUfsdf7HgZvdju+RwBXAo83E/ATwTRE51e3j+CXwvHv5q8VtuZ9VGk7ii/y97fukG7A/ovHCIpwv\nycjjVlUtwvliuQfYCxTj3s2iquuB3wPv43yBjgHe68R4LwROAXYDvwKewunvaK0/AD2AMmAZ8FqD\n5XcC57l3Jt3lfvGehdPBvAPn0tZvgFSOzs9xOuy3AouB36pqNBb3zOJUAFVdB1yFkxx24STmq1u7\nLeANnL/tNJw7kw5x+AzJdGFiRXaMqU9EngI2qGrDI35juj07UzAJz710M1xEAuL8QOtbwAt+x2WM\nH+Lp15jG+OU44Hmc3ymUAD9ybxM1JuHY5SNjjDFRdvnIGGNMVJe7fJSdna3Dhg3zOwxjjOlSVqxY\nUaaq/Vtq1+WSwrBhwygqKvI7DGOM6VJEZGtr2tnlI2OMMVGWFIwxxkRZUjDGGBNlScEYY0yUJQVj\njDFRniUFcYp+7xKRtc0sFxG5S0SKRWS1iEzwKhZjjDGt4+WZwqNAk0XDXXNwCoOfgFOV634PYzHG\nGNMKnv1OQVWXiMiwIzT5FvC4OuNsLBORviIyQFW/9ComY2JpOExdXQ2h0CFCoSpUw6iGnWWRujga\nJjIUTMNlGg6jhN12DdpEpjm8zUZt0Jj2h9dTrSMcriOsIfe5Dg3XURcOucvC1GktGg4T1jrC4ZDz\nrGH3ddhpp2FnnUi7SBsNH96uhqlz52tkmTtd73M4HH5kqt5+1vtcmphX/zNtYp1m38d/2kx9p+aH\nCGpb+6bmNveeEwadyrRJ1zTzvh3Dzx+vDaR+2cMSd16jpCAiV+KcTTBkyJBOCc60X6i2iurqfVTX\nHKC6qpyqmgpqag5QVXOAmtqDVNUepLrmINWhQ1SHKqmtqyEUriUUDlHrPtd7aF30uVbrnGmtI6Rh\n9+G+RglpmFrUfe0+o4SAkOA8A7UCIZEW9sQY/0gTSeTfw6FunRRaTVUfxCnkQX5+fpwdR/iv4RFv\n7KM2VEWortqZrqsmFKpxnuuqqQkdorq2kuraSqpCldSEqqgKVVFTV0VVXTXVoWqqwzXuI0R1uJaq\ncIgaraNa66gmTLWGqUKpEagCqgXqOujLNkmVZHX+J00CkqKvxXmI85yMkCQBkhDSAskkSdCZloD7\nOkiy+5wUSDr8HPNIDiQjbtziVtGsP93UPOe54Tya2E7T23TaNlwWkCQCEnAegSABCRKQACIBgoEk\nxJ2OLgsED68TCBIIOK8lujzJ3UaQQDDZnRcgEHBeSyBIMJDstHfbSiAYE9fhq8z1X0f2JeYqdCDQ\naJ7E/P8QWb/ediJtA9J4Xpxorqhcm+cH4mu/muJnUthO/Vq4g9x53dpnxYU8WXQnNZEjYPdRG2dH\nvKJKmjpFe1MV0hBSJEAaAVIkQK9AMlmSTGogSGoghdRAEqnBVNKCKaQEU0kLppGSlEpaUg9Sk9JJ\nTepBakpPUpPTSU3uRVpKL1JSepGa0ouU5J4kJaXFPHoQDKZ0iX9AxnQ3fiaFl4BrReRJYApQngj9\nCfe+/yuW1JXTP+zdEa/zOtl5BJNJCqQ4z8EUkoMpznRSKsnBNFJTepGWmkFqivtI7U1aal+SknvY\nl7IxCcizpCAifwNmANkiUoJT8zUZQFUfwKnjOxenVm8lcKlXscSL/eXb+FddOd9Pz+Gm777sdzjG\nGNOIl3cfnd/CcgW87TGJM/8ouptaEeaOusTvUIwxpkl2faATFZYsZnAdjB55rt+hGGNMkywpdJKy\n0k/4QCuZ3XekXas3xsQt+3bqJK8X3U1YhLl5l/sdijHGNKtL/E6hOyjcuYwTCZA7/Cy/QzHGmGbZ\nmUInKClZxseBWuZkj/c7FGOMOSJLCp3gtZXOWH9zJlztcyTGGHNklhQ6waLdqxinyQwcONnvUIwx\n5ogsKXhsY/FrbAyEmXPcKX6HYowxLbKk4LHC1X8moMpZ+df5HYoxxrTIkoKHNBymcN8Gpko62dkj\n/A7HGGNaZEnBQ2s+eYaSIMwZdLrfoRhjTKtYUvDQonV/JUWVmZOu9zsUY4xpFUsKHqkL1fD6gc85\nNdiHjN4D/Q7HGGNaxZKCR5Z//AhlQWHusDl+h2KMMa1mScEjiz57hp5h5bT8hBod3BjTxVlS8EBN\ndQVvVe3kjJRjSOvRz+9wjDGm1SwpeODdFfdTERDmnnCO36EYY0ybWFLwQOHmV+gXVqaMv8zvUIwx\npk0sKXSwygO7WFy7h7N6DCY5Od3vcIwxpk0sKXSwt4vupiogzB15xBLVxhgTlywpdLDCL97iuDpl\n3OgL/A7FGGPazJJCB9q393OWhiuY0/sEAkErameM6XosKXSgN5bfSUiEOWMW+B2KMca0ix3OdqDC\nHe+SExZGnPBNv0Mxxph2sTOFDrJz5ypWUMWczDFIwD5WY0zXZN9eHeT1FfeiIswdf5XfoRhjTLvZ\n5aMOsmjXckYRZOjQU/0OxRhj2s3OFDrAli3/ZH2gjjnHTPY7FGOMOSqWFDpA4ao/IqrMzr/W71CM\nMeaoWFI4ShoOs2jPWvJJ49hjx/odjjHGHBVLCkdpw8aX2RJU5hxvfQnGmK7P06QgIrNF5FMRKRaR\nhU0s7yMiL4vIxyKyTkQu9TIeLyxa8yhJqsyyOszGmG7As6QgIkHgXmAOcDJwvoic3KDZNcB6Vc0D\nZgC/F5EUr2LqaOG6EIX7N1IQyKBvvxy/wzHGmKPm5ZnCZKBYVTerag3wJPCtBm0UyBARAXoBe4CQ\nhzF1qI/WPsFXQWHOkFl+h2KMMR3Cy6QwENgWM13izot1DzAS2AGsAf5DVcMNNyQiV4pIkYgUlZaW\nehVvmxV+8iRpYeV0u+vIGNNN+N3RfDawCjgeGAfcIyK9GzZS1QdVNV9V8/v379/ZMTaptraS1w9t\nY0ZyJum9jvE7HGOM6RBeJoXtwOCY6UHuvFiXAs+roxj4HBjhYUwdZtlHD7EvIMwdPs/vUIwxpsN4\nmRSWAyeISI7befx94KUGbb4AZgKIyLHAScBmD2PqMIUbXyAjrBRM+KHfoRhjTIfxbOwjVQ2JyLXA\n60AQeFhV14nIVe7yB4BfAo+KyBpAgJtUtcyrmDrKoco9/KNmF7PTBpCSmuF3OMYY02E8HRBPVRcB\nixrMeyDm9Q7gLC9j8MKSFfdSGRDmnjTf71CMMaZD+d3R3CUVbnmN/nVK/tgFfodijDEdypJCG+0v\n38aSunLO7pVDMKnL/M7OGGNaxZJCG/2j6G5qRZg76hK/QzHGmA5nSaGNCksWM7gORo881+9QjDGm\nw1lSaIOysg18oJXM7jvS6jAbY7ol+2Zrg9eX30VYhLl5l/sdijHGeMJqNLdB4c5lnEiA3OFd7i5a\nY4xpFTtTaKWSkmV8HKhlTvZ4v0MxxhjPWFJopddW3g/AnAlX+xyJMcZ4x5JCKy3avYpxmszAgZP9\nDsUYYzxjSaEVNha/xsZAmDnHneJ3KMYY4ylLCq1QuPrPBFQ5K/86v0MxxhhPWVJogYbDFO7bwFRJ\nJzu7S5R6MMaYdrOk0II1nzxDSRDmDDrd71CMMcZzCZMUyg/V8trandSFtU3rFa77KymqzJx0vUeR\nGWNM/EiYpPDOhl1c9dcVrN+xv9Xr1IVqeO3A55wa7ENG74EeRmeMMfEhYZLCtOFZALy3qfWF3ZZ/\n/AhlQWHusDlehWWMMXElYZLCMb3TOOGYXrxX3PqkUPjZs/QMK6flX+NhZMYYEz8SJikAFORms3zL\nHqpq61psW1NdwZtVX3JGyjGk9ejXCdEZY4z/Ei4pVNWGWfnF3hbbvrvifioCwtwTzumEyIwxJj4k\nVFKY8rVMggFhafHuFtsWbn6FfmFlyvjLOiEyY4yJDwmVFHqnJTN2UJ8WO5srD+xice0ezuoxmOTk\n9E6Kzhhj/JdQSQGgYHg2H2/bx/6q2mbbvF10N1UBYe7I8zsxMmOM8V/iJYXcbMIKH2ze02ybwi/e\n4rg6ZdzoCzoxMmOM8V/CJYUJQ/uSlhxo9tbUfXs/Z2m4gjm9TyAQtMJ0xpjEknBJITUpyKRhmc0m\nhTeW30lIhDljFnRuYMYYEwcSLimAcwlp464D7Npf1WhZ4Y53yakTRpzwTR8iM8YYfyVmUhieDTQe\n8mLnzlWsoIo5mWOQQEJ+NMaYBJeQ33wnH9+bvunJvNfg9wqvr7gXFWHu+Kt8iswYY/yVkD2pwYBw\nyteyeK+4DFVFRABYtGs5owgydOipPkdoTHypra2lpKSEqqrGl1xNfElLS2PQoEEkJye3a/2ETArg\n9CsUrt3J52UH+Vr/Xmzd+i/WB+r4cbbVYTamoZKSEjIyMhg2bFj0IMrEH1Vl9+7dlJSUkJOT065t\neHr5SERmi8inIlIsIgubaTNDRFaJyDoR+aeX8cQqyI30KziXkBZ99ACiyuz8azsrBGO6jKqqKrKy\nsiwhxDkRISsr66jO6DxLCiISBO4F5gAnA+eLyMkN2vQF7gPmqeooYL5X8TQ0LCud4/uk8d7GMjQc\nZtGeNeSTxrHHju2sEIzpUiwhdA1H+3fy8kxhMlCsqptVtQZ4EvhWgzYXAM+r6hcAqrrLw3jqEREK\ncrN5f/Nu1n/6EluCypzjrS/BGJPYvEwKA4FtMdMl7rxYJwL9RGSxiKwQkUua2pCIXCkiRSJSVFpa\n2mEBFuRmU36oluc/epgkVWZZHWZjuq1Vq1axaNEiv8OIe37fkpoETAS+AZwN/G8RObFhI1V9UFXz\nVTW/f//+Hfbm03KzEEIsrtpMQSCDvv3a1zFjjIl/lhRax8uksB0YHDM9yJ0XqwR4XVUPqmoZsATI\n8zCmeo7JSGPGccvZFRTmDJnVWW9rjGmnxx9/nLFjx5KXl8fFF1/Mli1bOOOMMxg7diwzZ87kiy++\nAOCZZ55h9OjR5OXlcdppp1FTU8Mtt9zCU089xbhx43jqqad83pP45eUtqcuBE0QkBycZfB+nDyHW\ni8A9IpIEpABTgP/rYUyNZPRcSmpYOWX8jzrzbY3psn7x8jrW79jfods8+fje/Pybo47YZt26dfzq\nV79i6dKlZGdns2fPHn7wgx9EHw8//DDXX389L7zwArfddhuvv/46AwcOZN++faSkpHDbbbdRVFTE\nPffc06GxdzeenSmoagi4Fngd+AR4WlXXichVInKV2+YT4DVgNfAh8JCqrvUqpoZqaytZGfyKEw/2\nYENZ+37oYYzpHG+//Tbz588nO9u5nTwzM5P333+fCy5wjjUvvvhi3n33XQAKCgpYsGABf/rTn6ir\na7kmuznM0x+vqeoiYFGDeQ80mL4DuMPLOJqz7KOHKA8GqNk/kfeKy5jmjolkjGleS0f08eCBBx7g\ngw8+4NVXX2XixImsWLHC75C6DL87mn1VuPEFMsIKfc5tNA6SMSa+nHHGGTzzzDPs3u38W92zZw/T\npk3jySefBOCJJ57g1FOd28o3bdrElClTuO222+jfvz/btm0jIyODiooK3+LvKhI2KVQd2ss/anYx\nK20Ap5wwhNUlRy7RaYzx16hRo/jZz37G17/+dfLy8rjhhhu4++67eeSRRxg7dix/+ctfuPPOOwG4\n8cYbGTNmDKNHj2batGnk5eVx+umns379eutobkHCjn30z6J7qAwIc0+aj/bN5u63i1m2aTdnjTrO\n79CMMc2IdCrHevvttxu1e/755xvNy8zMZPny5Z7F1l0k7JlC4ZbX6F+n5I9dwPghTonOpZvsEpIx\nJrElZFLYX76Nf9WVc3avHIJJKdESne82U6LTGGMSRauSgog0GqiuqXldxT+K7qZGhLmjDo+qMT03\nm+JdB/iqiRKdxhiTKFp7pvCTVs7rEgpLFjO4DkaPPDc6LzKU9tJNdrZgjElcR+xoFpE5wFxgoIjc\nFbOoNxDyMjCvlJVt4AOt5LK+J9erw3zyAKdE57sbd3PO+EE+RmiMMf5p6e6jHUARMA+I/fVHBfCf\nXgXlpTeK7iYswty8y+vNDwSEacOzWLqpfolOY4xJJEe8fKSqH6vqY0Cuqj7mvn4Jp07C3k6JsIMt\n+vJ9TgwHyB1+VqNl04Zn82V5FZvLDvoQmTGmOfv27eO+++5r17pz585l3759HRxR99XaPoU3RaS3\niGQCK4E/iUinDlzXEbZv/5CPA7XMyR7f5PLpkX4FuwvJmLhypKQQCh35SvaiRYvo27evF2EdFVUl\nHA77HUYjrU0KfVR1P/Ad4HFVnQLM9C4sbxSudP6nmjPh6iaXD81KZ2DfHjbkhTFxZuHChWzatIlx\n48Zx4403snjxYk499VTmzZvHySc7VX6//e1vM3HiREaNGsWDDz4YXXfYsGGUlZWxZcsWRo4cyRVX\nXMGoUaM466yzOHToUKP3evnll5kyZQrjx4/nzDPP5KuvvgLgwIEDXHrppYwZM4axY8fy3HPPAfDa\na68xYcIE8vLymDnT+Vq89dZb+d3vfhfd5ujRo9myZQtbtmzhpJNO4pJLLmH06NFs27aNH/3oR+Tn\n5zNq1Ch+/vOfR9dZvnx59NfYkydPpqKigtNOO41Vq1ZF20yfPp2PP/64Az/p1v+iOUlEBgDfBX7W\noRF0okVlHzFOkhk4cHKTy50SnVm8tnYndWElGLB+BWMaKVwIO9d07DaPGwNzft3s4l//+tesXbs2\n+oW4ePFiVq5cydq1a8nJcYpjPfzww2RmZnLo0CEmTZrEueeeS1ZWVr3tbNy4kb/97W/86U9/4rvf\n/S7PPfccF110Ub0206dPZ9myZYgIDz30EL/97W/5/e9/zy9/+Uv69OnDmjXOvu/du5fS0lKuuOIK\nlixZQk5ODnv27GlxVzdu3Mhjjz3G1KlTAbj99tvJzMykrq6OmTNnsnr1akaMGMH3vvc9nnrqKSZN\nmsT+/fvp0aMHl112GY8++ih/+MMf+Oyzz6iqqiIvr2NL0LT2TOE2nCGwN6nqchH5GrCxQyPxWPGm\nN9gYCDPnuFOO2K4gN5v9VSHW7SjvpMiMMe0xefLkaEIAuOuuu8jLy2Pq1Kls27aNjRsbf0Xl5OQw\nbtw4ACZOnMiWLVsatSkpKeHss89mzJgx3HHHHaxbtw6At956i2uuuSbarl+/fixbtozTTjstGkdm\nZmaLcQ8dOjSaEACefvppJkyYwPjx41m3bh3r16/n008/ZcCAAUyaNAmA3r17k5SUxPz583nllVeo\nra3l4YcfZsGCBS1/UG3UqjMFVX0GeCZmejNwbvNrxJ/tpWvpX6eclX/dEdudMtw5sni3uIyxg+Lv\nOqQxvjvCEX1n6tmzZ/T14sWLeeutt3j//fdJT09nxowZVFU1/iFqampq9HUwGGzy8tF1113HDTfc\nwLx581i8eDG33nprm2NLSkqq118QG0ts3J9//jm/+93vWL58Of369WPBggVNxh2Rnp7OrFmzePHF\nF3n66ac9GRK8tb9oHiQifxeRXe7jORHpUjfzf33qDbz1g1VkZ484YrtjMtI46dgMllq/gjFxo6Vh\nr8vLy+nXrx/p6els2LCBZcuWtfu9ysvLGThwIACPPfZYdP6sWbO49957o9N79+5l6tSpLFmyhM8/\n/xwgevlo2LBhrFy5EoCVK1dGlze0f/9+evbsSZ8+ffjqq68oLCwE4KSTTuLLL7+MDuBXUVER7VC/\n/PLLuf7665k0aRL9+vVr9342p7WXjx7BuRX1ePfxsjuvSwkEW9eFMi03i+Vb9lBVaxWbjIkHWVlZ\nFBQUMHr0aG688cZGy2fPnk0oFGLkyJEsXLiw3uWZtrr11luZP38+EydOjFZ5A7j55pvZu3dvtPbz\nO++8Q//+/XnwwQf5zne+Q15eHt/73vcAOPfcc9mzZw+jRo3innvu4cQTT2zyvfLy8hg/fjwjRozg\nggsuoKCgAICUlBSeeuoprrvuOvLy8pg1a1b0DGLixIn07t2bSy+9tN37eCSiqi03ElmlquNamtcZ\n8vPztaioyNP3+McnX3HZY0X8v8unMC3XqrEZ88knnzBy5Ei/wzDAjh07mDFjBhs2bCAQaPq4vqm/\nl4isUNX8lrbf2jOF3SJykYgE3cdFQLe9vjI5J5NgQHjPxkEyxsSRxx9/nClTpnD77bc3mxCOVmu3\n+u84t6PuBL4EzgMWeBJRHMhISyZvUB/etX4FY0wcueSSS9i2bRvz53s3SHVbbkn9gar2V9VjcJLE\nLzyLKg5Mz81mTck+yg9ZiU5jTOJobVIYGzvWkaruAZoeK6KbmJabTVjhg812tmCMSRytTQoBEYne\n++SOgdSt6zuPH9KXHslB3rNxkIwxCaS1X+y/B94XkcgP2OYDt3sTUnxITQoyKSeT96xuszEmgbTq\nTEFVH8cZDO8r9/EdVf2Ll4HFg4LhWRTvOsDOcivRaYyfjmbobIA//OEPVFZWdmBE3Ver72lS1fWq\neo/7WO9lUPHCSnQaEx+6Q1JoaYjveOHNja7dxMkDetMvPdmG0jbGZw2Hzga44447mDRpEmPHjo0O\nOX3w4EG+8Y1vkJeXx+jRo3nqqae466672LFjB6effjqnn356o23fdtttTJo0idGjR3PllVcS+UFv\ncXExZ555Jnl5eUyYMIFNmzYB8Jvf/IYxY8aQl5fHwoULAZgxYwaRH9WWlZUxbNgwAB599FHmzZvH\nGWecwcyZMzlw4AAzZ85kwoQJjBkzhhdffDEax+OPP87YsWPJy8vj4osvpqKigpycHGprnTsg9+/f\nX2/aK926s/hoOSU6s3mv2Ep0GhPxmw9/w4Y9Gzp0myMyR3DT5JuaXd5w6Ow33niDjRs38uGHH6Kq\nzJs3jyVLllBaWsrxxx/Pq6++CjjjGPXp04f/+Z//4Z133qk3bEXEtddeyy233ALAxRdfzCuvvMI3\nv/lNLrzwQhYuXMg555xDVVUV4XCYwsJCXnzxRT744APS09NbNVT2ypUrWb16NZmZmYRCIf7+97/T\nu3dvysrKmDp1KvPmzWP9+vX86le/YunSpWRnZ7Nnzx4yMjKYMWMGr776Kt/+9rd58skn+c53vkNy\ncnJ7PuJWszOFFkzLzWLnfivRaUw8eeONN3jjjTcYP348EyZMYMOGDWzcuJExY8bw5ptvctNNN/Gv\nf/2LPn36tLitd955hylTpjBmzBjefvtt1q1bR0VFBdu3b+ecc84BIC0tjfT0dN566y0uvfRS0tPT\ngdYNlT1r1qxoO1Xlpz/9KWPHjuXMM89k+/btfPXVV7z99tvMnz8/mrQi7S+//HIeecQZZu6RRx7x\nbLyjWHam0IKC4c4f6b3iMob37+VzNMb470hH9J1FVfnJT37CD3/4w0bLVq5cyaJFi7j55puZOXNm\n9CygKVVVVVx99dUUFRUxePBgbr311iMOXd2c2KGyG64fO1T2E088QWlpKStWrCA5OZlhw4Yd8f0K\nCgrYsmULixcvpq6ujtGjR7c5trayM4UWHC7RaZ3Nxvil4dDZZ599Ng8//DAHDhwAYPv27ezatYsd\nO3aQnp5ML/tuAAATeUlEQVTORRddxI033hgdvrq5obcjX8jZ2dkcOHCAZ599Ntp+0KBBvPDCCwBU\nV1dTWVnJrFmzeOSRR6Kd1rFDZUdqG0S20ZTy8nKOOeYYkpOTeeedd9i6dSsAZ5xxBs888wy7d++u\nt11whra44IILOuUsATxOCiIyW0Q+FZFiEVl4hHaTRCQkIud5GU97REp0vr9pN3XhlkeUNcZ0vIZD\nZ5911llccMEFnHLKKYwZM4bzzjuPiooK1qxZw+TJkxk3bhy/+MUvuPnmmwG48sormT17dqOO5r59\n+3LFFVcwevRozj777GilM4C//OUv3HXXXYwdO5Zp06axc+dOZs+ezbx588jPz2fcuHHROsw//vGP\nuf/++xk/fjxlZc0fQF544YUUFRUxZswYHn/8cUaMcOq7jBo1ip/97Gd8/etfJy8vjxtuuKHeOnv3\n7uX888/vsM/zSFo1dHa7NiwSBD4DZgElwHLg/Ia3s7rt3gSqgIdVtfk0S+cMnd3Qi6u28x9PruLF\nawrIG2zV2EzisaGz/fPss8/y4osv8pe/tP6nYUczdLaXfQqTgWK3dCci8iTwLaDhbxyuA54DJhGn\npkX6FTaVWVIwxnSa6667jsLCQhYtWtRp7+nl5aOBwLaY6RJ3XpSIDATOAe4/0oZE5EoRKRKRotLS\n0g4PtCX9M1I56dgM61cwxnSqu+++m+Li4mYrt3nB747mPwA3qWr4SI1U9UFVzVfV/P79+3dSaPUV\n5GZTtGWvleg0CcurS82mYx3t38nLpLAdGBwzPcidFysfeFJEtuAU7rlPRL7tYUztVpCbRXUozMqt\ne1tubEw3k5aWxu7duy0xxDlVZffu3aSlpbV7G172KSwHThCRHJxk8H3ggtgGqpoTeS0ijwKvqOoL\nHsbUblO+lkUwILxbXGZ1m03CGTRoECUlJfhx+da0TVpaGoMGDWr3+p4lBVUNici1wOtAEOfOonUi\ncpW7/AGv3tsLvVKTGDe4rw2lbRJScnIyOTk5LTc0XZ6nv2hW1UXAogbzmkwGqrrAy1g6QsHwLO55\np5jyQ7X06eHt+CPGGOMHvzuau5QCt0TnMivRaYzppiwptMH4If3okRxkqd2aaozppiwptEFKUoDJ\nOZm8a0nBGNNNWVJoo4LcLDaVHrQSncaYbsmSQhtFhrywEp3GmO7IkkIbRUp02iUkY0x3ZEmhjSIl\nOpcW2687jTHdjyWFdijIzWbn/io2lVqJTmNM92JJoR0KcrMA61cwxnQ/lhTaYUhmOoP6WYlOY0z3\nY0mhHUSEguHZVqLTGNPtWFJop2m5WeyvCrF2e7nfoRhjTIexpNBOkd8r2K2pxpjuxJJCO/XPSGXE\ncRnW2WyM6VYsKRyFgtxslluJTmNMN2JJ4SgU5GZREwqzwkp0GmO6CUsKR2FyThZJAbFbU40x3YYl\nhaMQLdFpScEY001YUjhK03KzWbO9nPJDtX6HYowxR82SwlGabiU6jTHdiCWFozRucF96JAftEpIx\npluwpHCUIiU6LSkYY7oDSwodYHputpXoNMZ0C5YUOsA0dyhtO1swxnR1lhQ6wMjjepPZM4X3bMgL\nY0wXZ0mhAwQCwinDs3ivuMxKdBpjujRLCh2kYHg2X+2vthKdxpguzZJCB5me6wylbf0KxpiuzJJC\nBxmSZSU6jTFdnyWFDjQ9N5v3N1uJTmNM12VJoQNNy82moirEGivRaYzpojxNCiIyW0Q+FZFiEVnY\nxPILRWS1iKwRkaUikudlPF6bNtx+r2CM6do8SwoiEgTuBeYAJwPni8jJDZp9DnxdVccAvwQe9Cqe\nzpDdyynRaUnBGNNVeXmmMBkoVtXNqloDPAl8K7aBqi5V1UjZsmXAIA/j6RQFudkUbbUSncaYrsnL\npDAQ2BYzXeLOa85lQGFTC0TkShEpEpGi0tLSDgyx403PzbYSncaYLisuOppF5HScpHBTU8tV9UFV\nzVfV/P79+3ducG00OSeTpIDwrl1CMsZ0QUkebns7MDhmepA7rx4RGQs8BMxR1S5fqaanW6JzqSUF\nY0wX5OWZwnLgBBHJEZEU4PvAS7ENRGQI8Dxwsap+5mEsnaogN5vV28spr7QSncaYrsWzpKCqIeBa\n4HXgE+BpVV0nIleJyFVus1uALOA+EVklIkVexdOZCnKzUYX3rUSnMaaL8fLyEaq6CFjUYN4DMa8v\nBy73MgY/jBvcl/SUIEs3lTF79HF+h2OMMa0WFx3N3U2kRKd1NhtjuhpLCh4pGJ7N5tKDfFl+yO9Q\njDGm1SwpeKQgOpS29SsYY7oOSwoeGXFcBpk9U+zWVGNMl2JJwSOBgDBteBbvbbISncaYrsOSgocK\nciMlOg/4HYoxxrSKJQUPFQy3fgVjTNdiScFDQ7LSGZzZw25NNcZ0GZYUPFYwPJtlm3cTqgv7HYox\nxrTIkoLHCtwSnWt37Pc7FGOMaZElBY9ZiU5jTFdiScFjWVai0xjThVhS6ATTrUSnMaaLsKTQCQrc\nEp1FW6xEpzEmvllS6ASREp3vbbJLSMaY+GZJoRP0TE1i/JC+1q9gjIl7lhQ6SUFuNmusRKcxJs5Z\nUugkh0t02tmCMSZ+WVLoJHmDnBKdNg6SMSaeWVLoJClJAabkZFpnszEmrllS6EQFuVai0xgT3ywp\ndCIr0WmMiXeWFDrRScdmkNUzxW5NNcbErSS/A0gkgYBwyvAsXlu7ky/L3ycjLZneaclkpCXRu0cy\nvdOS6k1nxExnpCWTkmQ53BjjLUsKneyy6TlU1YbZf6iWbXsqqagKsf9QLRXVoRbXTUsONEgaTiLJ\nSEumdw8ngcRON0w6PVOCiEgn7KUxpquypNDJxg/px0M/yG80vy6sHKgOUVFVy/5D7nNVZLrWSR5V\n9Z/LK2so2VPJfnedmhYK+QQEMtIOn4Fk9UphWFZPhmalMyQznWHZPRmSmU5actCr3TfGxDlLCnEi\nGBD69EimT49k6Ne+bVTV1tVPHvWSSf1ks/9QLbsqqnlh23YqquqfpRzXO40hWekMjUkUQ7PSGZrV\n04nPGNNtWVLoRtKSg6QlB+mfkdrqdVSVfZW1bN1TydbdB9m6u9J9HGTxZ6WUriip175vejJDM50E\nEXuGMTQznf4ZqXZ5ypguzpJCghMR+vVMoV/PFMYN7tto+cHqEF/scRLFF3sOsmV3JV/sruSjbXt5\nZfUOwnq4bY/kYKNLUUOz0hmW1ZMBfdJIClpHuTHxzpKCOaKeqUmMHNCbkQN6N1pWEwqzfd+hemcY\nX+w5yOYy5yyjJnS4jyMpIAzq16P+GYb7uncHXJLqkPMTiTwJIvVmISIxr5029daR+m0ji6SJbUbf\nLmY7kfVFhIBgZ1zGN5YUTLulJAXIye5JTnbPRsvCYeWriiq2lDmJInpZas9BVm7d26q7rRJZUkAI\nBoSkgJAUDESnk4OB6Pxgo2WRdQKNppOCErNeIGZZ09sPiqAoYQVVCKtzSqhaf546M515qDv/8GvV\nZubhru8uI7pu/faCsx9JQWc/UpIC0ZiTI89BJ+6koJDs7mtSMEBK8PC+JwcPr5cSadtgeaNtBCQh\nk7OnSUFEZgN3AkHgIVX9dYPl4i6fC1QCC1R1pZcxmc4RCAgD+vRgQJ8enDI8q94yVWVvZS1bdx/k\niz2VHKw+ujKl7lfT0W1DI9s6PKExyzRmXr22HF5Wfztar13s/Nh2kfUj02GFOlXqwmFCdUoorNSF\nlVCD6dq6sDu/8XRlTajZZYe3EY5OR7YfbufHGDnLCYhEz35EDp8JRc58mjobCgjgtg80aB9RW+fE\nWlvnfAaR6c6QFJMonUid/0T2I/aMMnI2KRJtGbPPjc9Ao2eU0viMMnZ7sSel508ewuWnfs3LXfYu\nKYhIELgXmAWUAMtF5CVVXR/TbA5wgvuYAtzvPptuTETI7JlCZs8Uxg9p561WpsOFw/WTUORLPhDz\n5RVwv6iiCcCnI2nVw0muNhymNlQ/cYTCYWrdBFJbp4SaSCy1YXd+o2048yPLa+vC0bOjesm+wYFC\n9EwoGqOzVGPWJdo2Zp7WP8DQem1iDlAUsnu1/iaS9vLyTGEyUKyqmwFE5EngW0BsUvgW8Lg6e75M\nRPqKyABV/dLDuIwxTQgEhJRA5Es+vn+rIiLuZSPoEeexdjVe3g4yENgWM13izmtrG0TkShEpEpGi\n0tLSDg/UGGOMo0vcI6iqD6pqvqrm9+/f3+9wjDGm2/IyKWwHBsdMD3LntbWNMcaYTuJlUlgOnCAi\nOSKSAnwfeKlBm5eAS8QxFSi3/gRjjPGPZx3NqhoSkWuB13F6rR5W1XUicpW7/AFgEc7tqMU4t6Re\n6lU8xhhjWubp7xRUdRHOF3/svAdiXitwjZcxGGOMab0u0dFsjDGmc1hSMMYYEyWxP9HvCkSkFNja\nztWzASuQfJh9HvXZ53GYfRb1dYfPY6iqtnhPf5dLCkdDRIpUtXHZswRln0d99nkcZp9FfYn0edjl\nI2OMMVGWFIwxxkQlWlJ40O8A4ox9HvXZ53GYfRb1JcznkVB9CsYYY44s0c4UjDHGHIElBWOMMVEJ\nkxREZLaIfCoixSKy0O94/CQig0XkHRFZLyLrROQ//I7JbyISFJGPROQVv2Pxm1vs6lkR2SAin4jI\nKX7H5BcR+U/338haEfmbiKT5HZPXEiIpxJQGnQOcDJwvIif7G5WvQsB/qerJwFTgmgT/PAD+A/jE\n7yDixJ3Aa6o6AsgjQT8XERkIXA/kq+ponIE9v+9vVN5LiKRATGlQVa0BIqVBE5KqfqmqK93XFTj/\n6BtVvEsUIjII+AbwkN+x+E1E+gCnAX8GUNUaVd3nb1S+SgJ6iEgSkA7s8DkezyVKUmhV2c9EJCLD\ngPHAB/5G4qs/AP8NhP0OJA7kAKXAI+7ltIdEpKffQflBVbcDvwO+AL7Eqffyhr9ReS9RkoJpgoj0\nAp4D/peq7vc7Hj+IyL8Bu1R1hd+xxIkkYAJwv6qOBw4CCdkHJyL9cK4o5ADHAz1F5CJ/o/JeoiQF\nK/vZgIgk4ySEJ1T1eb/j8VEBME9EtuBcVjxDRP7qb0i+KgFKVDVy5vgsTpJIRGcCn6tqqarWAs8D\n03yOyXOJkhRaUxo0YYiI4Fwz/kRV/8fvePykqj9R1UGqOgzn/4u3VbXbHw02R1V3AttE5CR31kxg\nvY8h+ekLYKqIpLv/ZmaSAJ3unlZeixfNlQb1OSw/FQAXA2tEZJU776dupTxjrgOecA+gNpOgZXJV\n9QMReRZYiXPH3kckwHAXNsyFMcaYqES5fGSMMaYVLCkYY4yJsqRgjDEmypKCMcaYKEsKxhhjoiwp\nmLghIkvd52EickEHb/unTb2XV0Tk2yJyi0fb/mnLrdq8zTEi8mhHb9d0PXZLqok7IjID+LGq/lsb\n1klS1dARlh9Q1V4dEV8r41kKzFPVsqPcTqP98mpfROQt4N9V9YuO3rbpOuxMwcQNETngvvw1cKqI\nrHLHsw+KyB0islxEVovID932M0TkXyLyEu6vbkXkBRFZ4Y6Bf6U779c4I12uEpEnYt9LHHe44+Wv\nEZHvxWx7cUxdgSfcX7UiIr92a1GsFpHfNbEfJwLVkYQgIo+KyAMiUiQin7njLUVqOLRqv2K23dS+\nXCQiH7rz/ugOFY+IHBCR20XkYxFZJiLHuvPnu/v7sYgsidn8yyTA0NCmBapqD3vExQM44D7PAF6J\nmX8lcLP7OhUowhmkbAbOgG05MW0z3ecewFogK3bbTbzXucCbOL90PxZnaIMB7rbLccbJCgDvA9OB\nLOBTDp9l921iPy4Ffh8z/SjwmrudE3DGF0pry341Fbv7eiTOl3myO30fcIn7WoFvuq9/G/Nea4CB\nDePH+aX7y37/f2APfx8JMcyF6fLOAsaKyHnudB+cL9ca4ENV/Tym7fUico77erDbbvcRtj0d+Juq\n1gFficg/gUnAfnfbJQDucCDDgGVAFfBncaq0NVWpbQDO8NOxnlbVMLBRRDYDI9q4X82ZCUwElrsn\nMj2AXe6ympj4VgCz3NfvAY+KyNM4g7xF7MIZDdQkMEsKpisQ4DpVfb3eTKfv4WCD6TOBU1S1UkQW\n4xyRt1d1zOs6IEmdcbQm43wZnwdcC5zRYL1DOF/wsRp23imt3K8WCPCYqv6kiWW1qhp53zrcf++q\nepWITMEpLLRCRCaq6m6cz+pQK9/XdFPWp2DiUQWQETP9OvAjd7hvROTEZgq/9AH2uglhBE6p0Yja\nyPoN/Av4nnt9vz9O1bEPmwtMnBoUfdQZPPA/ccpVNvQJkNtg3nwRCYjIcOBrOJegWrtfDcXuyz+A\n80TkGHcbmSIy9Egri8hwVf1AVW/BOaOJDCt/Is4lN5PA7EzBxKPVQJ2IfIxzPf5OnEs3K93O3lLg\n202s9xpwlYh8gvOluyxm2YPAahFZqaoXxsz/O3AK8DHO0ft/q+pON6k0JQN4UZwC7gLc0ESbJcDv\nRURijtS/wEk2vYGrVLVKRB5q5X41VG9fRORm4A0RCQC1wDXA1iOsf4eInODG/w933wFOB15txfub\nbsxuSTXGAyJyJ06n7Vvu/f+vqOqzPofVLBFJBf4JTNcj3Npruj+7fGSMN/4PTqH3rmIIsNASgrEz\nBWOMMVF2pmCMMSbKkoIxxpgoSwrGGGOiLCkYY4yJsqRgjDEm6v8DQFcZOySpkUwAAAAASUVORK5C\nYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2158f96fdd8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 21.5 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "layer_dims = [train_x.shape[0],128,128,1]\n",
    "activation = [\"relu\",\"relu\",\"sigmoid\"]\n",
    "\n",
    "parameters = L_layer_model_L2(X=train_x,Y=train_y, \n",
    "                              layer_dims=layer_dims,\n",
    "                              activation=activation,\n",
    "                              initialization='he',\n",
    "                              keep_prob=.7,\n",
    "                              mini_batch_size = 512,\n",
    "                              test_x=test_x, test_y=test_y,\n",
    "                              learning_rate = .001, lambd=.15, \n",
    "                              num_iterations = 100, print_cost = True,\n",
    "                              optimizer='adam')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Ones = 4211.0\n",
      "Accuracy: 0.999762780216\n"
     ]
    }
   ],
   "source": [
    "predict(train_x, train_y, parameters,activation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Ones = 474.0\n",
      "Accuracy: 0.990384615385\n"
     ]
    }
   ],
   "source": [
    "predict(test_x, test_y, parameters,activation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### In this model, it only took 20 seconds which is a small fraction of the 6.5 minutes of the earlier model. It also has a higher training accuracy at 99.99% and a higher testing accuracy at 99.03%. Training and testing accuracy are very close. We can safeley say that we are not overfitting. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}