t81_558_class_04_2_multi_class.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_04_2_multi_class.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 4: Training for Tabular Data**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 4.2: Keras Multiclass Classification for Deep Neural Networks with ROC and AUC\n",
    "\n",
    "* **Binary Classification** - Classification between two possibilities (positive and negative).  Common in medical testing, does the person have the disease (positive) or not (negative).\n",
    "* **Classification** - Classification between more than 2.  The iris dataset (3-way classification).\n",
    "* **Regression** - Numeric prediction.  How many MPG does a car get? (covered in next video)\n",
    "\n",
    "In this class session we will look at some visualizations for all three.\n",
    "\n",
    "\n",
    "It is important to evaluate the level of error in the results produced by a neural network.  In this part we will look at how to evaluate error for both classification and regression neural networks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Binary classification is used to create a model that classifies between only two classes.  These two classes are often called \"positive\" and \"negative\".  Consider the following program that uses the [wcbreast_wdbc dataset](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/datasets_wcbc.ipynb) to classify if a breast tumor is cancerous (malignant) or not (benign).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>diagnosis</th>\n",
       "      <th>mean_radius</th>\n",
       "      <th>mean_texture</th>\n",
       "      <th>mean_perimeter</th>\n",
       "      <th>mean_area</th>\n",
       "      <th>mean_smoothness</th>\n",
       "      <th>mean_compactness</th>\n",
       "      <th>mean_concavity</th>\n",
       "      <th>mean_concave_points</th>\n",
       "      <th>...</th>\n",
       "      <th>worst_radius</th>\n",
       "      <th>worst_texture</th>\n",
       "      <th>worst_perimeter</th>\n",
       "      <th>worst_area</th>\n",
       "      <th>worst_smoothness</th>\n",
       "      <th>worst_compactness</th>\n",
       "      <th>worst_concavity</th>\n",
       "      <th>worst_concave_points</th>\n",
       "      <th>worst_symmetry</th>\n",
       "      <th>worst_fractal_dimension</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>842302</td>\n",
       "      <td>M</td>\n",
       "      <td>17.99</td>\n",
       "      <td>10.38</td>\n",
       "      <td>122.80</td>\n",
       "      <td>1001.0</td>\n",
       "      <td>0.11840</td>\n",
       "      <td>0.27760</td>\n",
       "      <td>0.3001</td>\n",
       "      <td>0.14710</td>\n",
       "      <td>...</td>\n",
       "      <td>25.38</td>\n",
       "      <td>17.33</td>\n",
       "      <td>184.60</td>\n",
       "      <td>2019.0</td>\n",
       "      <td>0.1622</td>\n",
       "      <td>0.6656</td>\n",
       "      <td>0.7119</td>\n",
       "      <td>0.2654</td>\n",
       "      <td>0.4601</td>\n",
       "      <td>0.11890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>842517</td>\n",
       "      <td>M</td>\n",
       "      <td>20.57</td>\n",
       "      <td>17.77</td>\n",
       "      <td>132.90</td>\n",
       "      <td>1326.0</td>\n",
       "      <td>0.08474</td>\n",
       "      <td>0.07864</td>\n",
       "      <td>0.0869</td>\n",
       "      <td>0.07017</td>\n",
       "      <td>...</td>\n",
       "      <td>24.99</td>\n",
       "      <td>23.41</td>\n",
       "      <td>158.80</td>\n",
       "      <td>1956.0</td>\n",
       "      <td>0.1238</td>\n",
       "      <td>0.1866</td>\n",
       "      <td>0.2416</td>\n",
       "      <td>0.1860</td>\n",
       "      <td>0.2750</td>\n",
       "      <td>0.08902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>84300903</td>\n",
       "      <td>M</td>\n",
       "      <td>19.69</td>\n",
       "      <td>21.25</td>\n",
       "      <td>130.00</td>\n",
       "      <td>1203.0</td>\n",
       "      <td>0.10960</td>\n",
       "      <td>0.15990</td>\n",
       "      <td>0.1974</td>\n",
       "      <td>0.12790</td>\n",
       "      <td>...</td>\n",
       "      <td>23.57</td>\n",
       "      <td>25.53</td>\n",
       "      <td>152.50</td>\n",
       "      <td>1709.0</td>\n",
       "      <td>0.1444</td>\n",
       "      <td>0.4245</td>\n",
       "      <td>0.4504</td>\n",
       "      <td>0.2430</td>\n",
       "      <td>0.3613</td>\n",
       "      <td>0.08758</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>84348301</td>\n",
       "      <td>M</td>\n",
       "      <td>11.42</td>\n",
       "      <td>20.38</td>\n",
       "      <td>77.58</td>\n",
       "      <td>386.1</td>\n",
       "      <td>0.14250</td>\n",
       "      <td>0.28390</td>\n",
       "      <td>0.2414</td>\n",
       "      <td>0.10520</td>\n",
       "      <td>...</td>\n",
       "      <td>14.91</td>\n",
       "      <td>26.50</td>\n",
       "      <td>98.87</td>\n",
       "      <td>567.7</td>\n",
       "      <td>0.2098</td>\n",
       "      <td>0.8663</td>\n",
       "      <td>0.6869</td>\n",
       "      <td>0.2575</td>\n",
       "      <td>0.6638</td>\n",
       "      <td>0.17300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>84358402</td>\n",
       "      <td>M</td>\n",
       "      <td>20.29</td>\n",
       "      <td>14.34</td>\n",
       "      <td>135.10</td>\n",
       "      <td>1297.0</td>\n",
       "      <td>0.10030</td>\n",
       "      <td>0.13280</td>\n",
       "      <td>0.1980</td>\n",
       "      <td>0.10430</td>\n",
       "      <td>...</td>\n",
       "      <td>22.54</td>\n",
       "      <td>16.67</td>\n",
       "      <td>152.20</td>\n",
       "      <td>1575.0</td>\n",
       "      <td>0.1374</td>\n",
       "      <td>0.2050</td>\n",
       "      <td>0.4000</td>\n",
       "      <td>0.1625</td>\n",
       "      <td>0.2364</td>\n",
       "      <td>0.07678</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 32 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         id diagnosis  mean_radius  mean_texture  mean_perimeter  mean_area  \\\n",
       "0    842302         M        17.99         10.38          122.80     1001.0   \n",
       "1    842517         M        20.57         17.77          132.90     1326.0   \n",
       "2  84300903         M        19.69         21.25          130.00     1203.0   \n",
       "3  84348301         M        11.42         20.38           77.58      386.1   \n",
       "4  84358402         M        20.29         14.34          135.10     1297.0   \n",
       "\n",
       "   mean_smoothness  mean_compactness  mean_concavity  mean_concave_points  \\\n",
       "0          0.11840           0.27760          0.3001              0.14710   \n",
       "1          0.08474           0.07864          0.0869              0.07017   \n",
       "2          0.10960           0.15990          0.1974              0.12790   \n",
       "3          0.14250           0.28390          0.2414              0.10520   \n",
       "4          0.10030           0.13280          0.1980              0.10430   \n",
       "\n",
       "   ...  worst_radius  worst_texture  worst_perimeter  worst_area  \\\n",
       "0  ...         25.38          17.33           184.60      2019.0   \n",
       "1  ...         24.99          23.41           158.80      1956.0   \n",
       "2  ...         23.57          25.53           152.50      1709.0   \n",
       "3  ...         14.91          26.50            98.87       567.7   \n",
       "4  ...         22.54          16.67           152.20      1575.0   \n",
       "\n",
       "   worst_smoothness  worst_compactness  worst_concavity  worst_concave_points  \\\n",
       "0            0.1622             0.6656           0.7119                0.2654   \n",
       "1            0.1238             0.1866           0.2416                0.1860   \n",
       "2            0.1444             0.4245           0.4504                0.2430   \n",
       "3            0.2098             0.8663           0.6869                0.2575   \n",
       "4            0.1374             0.2050           0.4000                0.1625   \n",
       "\n",
       "   worst_symmetry  worst_fractal_dimension  \n",
       "0          0.4601                  0.11890  \n",
       "1          0.2750                  0.08902  \n",
       "2          0.3613                  0.08758  \n",
       "3          0.6638                  0.17300  \n",
       "4          0.2364                  0.07678  \n",
       "\n",
       "[5 rows x 32 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/wcbreast_wdbc.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "display(df[0:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ROC Curves\n",
    "\n",
    "ROC curves can be a bit confusing.  However, they are very common.  It is important to know how to read them.  Even their name is confusing.  Do not worry about their name, it comes from electrical engineering (EE).\n",
    "\n",
    "Binary classification is common in medical testing.  Often you want to diagnose if someone has a disease.  This can lead to two types of errors, know as false positives and false negatives:\n",
    "\n",
    "* **False Positive** - Your test (neural network) indicated that the patient had the disease; however, the patient did not have the disease.\n",
    "* **False Negative** - Your test (neural network) indicated that the patient did not have the disease; however, the patient did have the disease.\n",
    "* **True Positive** - Your test (neural network) correctly identified that the patient had the disease.\n",
    "* **True Negative** - Your test (neural network) correctly identified that the patient did not have the disease.\n",
    "\n",
    "Types of errors:\n",
    "\n",
    "![Type of Error](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_4_errors.png \"Type of Error\")\n",
    "\n",
    "Neural networks classify in terms of probability of it being positive. However, at what probability do you give a positive result?  Is the cutoff 50%? 90%?  Where you set this cutoff is called the threshold.  Anything above the cutoff is positive, anything below is negative.  Setting this cutoff allows the model to be more sensitive or specific:\n",
    "\n",
    "More info on Sensitivity vs Specificity: [Khan Academy](https://www.youtube.com/watch?v=Z5TtopYX1Gc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAEJCAYAAACqmv3eAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3hU1dbH8e9Op4ReJQgiRZIAMTQVEKQ3kZ5QpCjFclW8YC8vXq/dq4hIERBFgkmkqXQQUekEpAWQXgIivUPqef/Yk1AkkISZ2VPW53nyZBhmzvnlMKzs2bPPOsqyLIQQQjifj+kAQgjhraQACyGEIVKAhRDCECnAQghhiBRgIYQwRAqwEEIYIgXYCZRSrymlEpVSm5RSG5RS9Z2wz1ev+/MKR+8zJ5RS6bZjsFEptV4p9UAOnnPeGdlusN/hSqlhJvZ9M458PSml5iqlithuP6uU2qaUilFKdVBKvWyv/QhNyTpgx1JK3Q98AjSxLCtZKVUCCLAs67CD93vesqyCjtxHXlydSynVCnjVsqzGOX2OMymlhgPnLcv62Nn7zo4zX09Kqe1AG8uy9tp720KTEbDjlQWOW5aVDGBZ1nHLsg4rpWorpX5VSq1TSi1QSpUFUEotVUp9oJRao5TaoZRqZLs/zHbfBtvIp4rt/lm2bSQqpQbZ7nsfyGd7bIztvvO273FKqbaZ4ZRSXyuluiilfJVSHyml1tq2P9gJx6YQcMqWo6BS6mfbqHizUuqR6x+c3WOUUhVtI7XxtuOwUCmVz/Z3lZVSi68acd9tu/+Fq37Wt67ax2tKqT+VUouBak44BrmV3etp31WvmzVKqcoASqmSSqnptp91rVKqge3+gkqpSbbjuEkp1cV2/z6lVAml1FigEvCjUup5pVQ/pdQo22NKK6Vm2o7pxpy8ixHZsCxLvhz4BRQENgA7gNFAY8AfWAGUtD0mCvjKdnsp8D/b7bbAYtvtz4FettsBQD7b7WK27/mALUBx25/PX5fjvO17J+Cbq7Zz0PbcQcDrtvsDgQTgLgccj3Tb8dgOnAFq2+73AwrZbpcAdnHlHdr5mz0GqAikARG2v4sHetturwY62W4HAfmBlsCXtuf6ALOBB4HawGbbYwrZtj/M9GvoVq8n2/37gNdst/sAs223pwINbbfvBLbZbn8AjLhqu0Wv2k6JG9zuB4yy3Y4Dhthu+wKFTR8Xd/3yQziUZVnnlVK1gUbAQ+gX73+BcGCRUgr0i/ivq542w/Z9Hbq4AKwEXlNKhQAzLMvaabv/WaVUJ9vt8kAV4MRNIs0DRiqlAoHWwG+WZV1SSrUEaiqlutoeV9i2LXu//bxkWVYEZL2dnqyUCkcXw3eVUg8CGUA5oDRw5KrnZvcYgL2WZW2w3V4HVFRKBQPlLMuaCWBZ1mXbfluii/AftscXtP2swcBMy7Iu2h73o51/9tt2o9fTVXOz3131/VPb7eZAqO11BlDIdlyaA9FXbfdULmI0RRd5LMtKR/8iFXkgBdgJbC/SpcBSpdRm4Gkg0bKs+7N5SrLtezq2fyPLsqYqpVYD7YAFSqkB6CLUHLjfsqyLSqml6FHezbJctj2uFXrknfmfVgHPWJa1IE8/ZB5YlrXSNodZEj3aL4keEacqpfbxz5+l100ek3zV49LRo3rFjSngPcuyxl1zp1JDAJf/UOQGr6e+mX919cNs333Qr49LV29D6Yrs8j+rp5M5YAdTSlXLnK+1iQC2ASVtI0CUUv5KqbBbbKcSsMeyrJHAj0BN9Cj1lK343gPcd9VTUpVS/tlsLhbojx5FZRbcBcCTmc9RSlVVShXIzc+aW7bMvugRe2HgqK2wPgRUuMFTcvKYLJZlnQWSlFIdbfsLVErlR/+sjymlMj8MLKeUKgX8BnRSSuWzjRIfts9Paj/ZvJ72225HXfV9pe32QuBfVz0/Ipv7i+Yixs/Ak7bn+SqlCuXiueIqUoAdryDwjVJqq1JqExAKvAl0BT5QSm1Ez+nd6oOMKGCLUmoDcA8wGZgP+Nm2+zaw6qrHfwlsUrYP4a6zED3nudiyrBTbfROArcB6pdQWYByOeYeU+eHgBvR0TF/biC4GqKOUSkCPdLff4Lk5ecz1HkVP02xCz7uXsSxrIXpudKVtBDkNCLYsa70t0wZgOvD77fygDnKj19Nw298F2t4lPQc8b7vvWfQx26SU2go8Ybv/v0BRpdQW22vwoVxkeA54yHbs1gE3HTyI7MkyNCE8gG06po5lWcdNZxE5JyNgIYQwREbAQghhiIyAhRDCECnAQghhiBRgIYQwRAqwEEIYIgVYCCEMydVC+xIlSlgVK1Z0UBSRG3/++ScA1aq5YsMu55JjcYUcC9e0bt2645Zllbz+/lwV4IoVK5KQkGC/VCLPmjRpAsDSpUuN5nAFciyukGPhmpRS+290v0xBCCGEIVKAhRDCECnAQghhiPQDFkJkSU1NJSkpicuXL5uO4paCgoIICQnB3z+7TrDXkgIshMiSlJREcHAwFStW5KqraIgcsCyLEydOkJSUxF133ZWj58gUhBAiy+XLlylevLgU3zxQSlG8ePFcvXuQAiyEuIYU37zL7bGTAiyEcClKKYYOHZr1548//pjhw4fbfT/vvvvuNX9+4IFbXZTG/qQACyFcSmBgIDNmzOD4ccde3OP6ArxixQqH7u9GpAALIVyKn58fgwYN4tNPP/3H3x07dowuXbpQt25d6taty/Lly7Pub9GiBZGRkQwePJgKFSpkFfCOHTtSu3ZtwsLC+PLLLwF4+eWXuXTpEhEREfTq1QuAggULAhAVFcXcuXOz9tmvXz+mT59Oeno6L7zwAnXr1qVmzZqMG3fNRbXzRAqwEMLlPP3008TExHDmzJlr7n/uued4/vnnWbt2LdOnT2fAgAEAvPXWWzRt2pT169fTqVMnDhw4kPWcr776inXr1pGQkMDIkSM5ceIE77//Pvny5WPDhg3ExFx73dro6Gji4uIASElJ4eeff6Zt27ZMnDiRwoULs3btWtauXcv48ePZu3fvbf2csgxNCHFDQ+YPYcORDXbdZkSZCEa0HnHLxxUqVIg+ffowcuRI8uXLl3X/4sWL2bp1a9afz549y7lz51i2bBkzZ84EoHXr1hQtWjTrMSNHjsz6u4MHD7Jz506KFy+e7b7btGnDs88+S3JyMvPnz+fBBx8kX758LFy4kE2bNjFt2jQAzpw5w86dO3O85OxGpAALIVzSkCFDiIyMpH///ln3ZWRksHLlymuKMug1uDeydOlSFi9ezMqVK8mfPz9NmjS55TKxoKAgmjRpwoIFC4iLi6NHjx5Z+/j8889p1arVbf5kV0gBFkLcUE5Gqo5UrFgxunfvzsSJE3nssccAaNmyJaNGjeKFF14AYMOGDURERNCwYUPi4+N56aWXWLhwIadOnQL0KLVo0aLkz5+f7du3s2rVqqzt+/v7k5qaesOz1qKjo5kwYQIJCQl8/fXXALRq1YoxY8bQtGlT/P392bFjB+XKlaNAgQJ5/hllDlgI4bKGDh16zWqIkSNHkpCQQM2aNQkNDWXs2LEA/N///R8LFy4kMjKSefPmUbZsWYKDg2ndujVpaWnUrFmTN954g/vuuy9rW4MGDaJmzZpZH8JdrWXLlvz22280b96cgIAAAAYMGEBoaCiRkZGEh4czePBg0tLSbuvny9Vl6evUqWNJP2DXIH1fr5BjccXtHott27ZRvXp1+wVykuTkZHx9ffHz82PlypU8+eSTbNhg3/nrnLrRMVRKrbMsq871j5UpCCGE2ztw4ADdu3cnIyODgIAAxo8fbzpSjkgBFkK4vSpVqvDHH3+YjpFrMgcshBCGSAEWQghDpAALIYQhUoCFEMIQKcBCCJfi6+tLREQE4eHhdOvWjYsXL+Z6GwMGDMg6ZdkV2k5mRwqwEMKlZDbJ2bJlCwEBAVknW+TGhAkTCA0NBVyj7WR2pAALIVxWo0aN2LVrFwCffPIJ4eHhhIeHM2KEPk36woULtGvXjlq1ahEeHp7VxaxJkyYkJCS4TNvJ7Mg6YCGES0pLS2PevHm0bt2adevWMWnSJFavXo1lWdSvX5/GjRuzZ88e7rjjDubMmQPwj/aV77//PqNGjbrhWXGZbSfbtm2b1XZyzJgx17SdTE5OpkGDBrRs2fK2up5lRwqwEOLGhgwBe5/OGxEBI27e5CdzxAp6BPz4448zZswYOnXqlNX4pnPnzvz++++0bt2aYcOG8dJLL9G+fXsaNWqU4yjObDuZHSnAQgiXkjkHfLXsetZUrVqVdevWMXfuXF555RVatmzJm2++maP9OLPtZHakAAshbuwWI1VnevDBB+nXrx8vv/wylmUxc+ZMvv32Ww4fPkyxYsXo3bs3BQsWzGodeTVXaDuZHSnAQgiXFxkZSb9+/ahXrx6gl5nde++9LFiwgBdeeAEfHx/8/f0ZM2bMP56b2XYyMjLyH5cfatmyJX369KFDhw7XtJ3ct28fkZGRWJZFyZIlmTVrlkN+LmlH6aakBeMVciyu8NZ2lK4kN+0oZRmaEEIYIgVYCCEMkQIshBCGSAEWQlwjN58LiWvl9thJARZCZAkKCuLEiRNShPPAsixOnDhBUFBQjp8jy9CEEFlCQkJISkri2LFjpqO4paCgIEJCQnL8eCnAQogs/v7+DjnlVtyYTEEIIYQhUoCFEMIQKcBCCGGIFGAhhDBECrAQQhgiBVgIIQyRAiyEEIZIARZCCEOkAAshhCFSgIUQwhApwEIIYYgUYCGEMEQKsBBCGCIF2M2cunSK+bvms+/0Pnaf3E3sllj2n94v/Vu93fHjMHs27N0Le/bA9Olw+LDpVOIWpB2lm0g6m8TTc5/mxz9/1HecAYWix/QeAFQvUZ1x7cfRqEIjgymF0+3aBYMHw5IlV+5TCrp21bdr14Yvv4TISDP5xE3JCNjFWZbFxPUTCRsdxqLdi3it0Wv83OdnGpZvSKM7G5EwMIHP23zO5bTLNP66Mc/Oe5YLKRdMxxaOlp4On34KNWvCunXw9tvw66/QqBE0bAirVsH//qdHwfXqweuvQ3Ky6dTiOjICdmHpGen0mtGLuMQ4GldozMQOE7m72N0A/MfnPwDUvqM2te+oTb+Ifrz686t8vuZzFuxewNK+SykbXNZkfOEoly/DI4/AwoXQvj2MHQvlyum/87GNqerX11/9+8Pzz8M778DcufDzz1C0qLns4hoyAnZRlmUxePZg4hLjeKfpOyzpuySr+N5IwYCCjGwzkiV9lnDo7CFaTWnFqUunnJhYOEVaGvTsqYvvmDHw449Xiu+NFC0KX38Ns2ZBYqIu2BcvOi2uuDkpwC7q5cUvM/GPibze6HVebfQqPipn/1QP3fUQP0T/wJ8n/qT9d+1lOsKTWBY88QTMnAmffaZvK5Wz5z7yCEydqqcmunaFlBTHZhU5IgXYBX268lM+XPEhT9Z5kv889J9cP79ZpWZ81+U7ViWtotv33ciwMhyQUjjd66/DxInwxhvw7LO5f36XLnq6Yt48eOwxXdCFUVKAXcwff/3Bi4tfpNM9nfi8zeeonI5wrtO5eme+aPsF83bN47NVn9k5pXC6X36Bd9+FAQPgrbfyvp2BA+E//4GYGJgyxX75RJ5IAXYhyWnJ9JnVh5L5SzKhwwR8fXxva3uDaw+mQ7UOvPLzK2w/vt1OKYXTnT2rP0yrUkVPPeTxl3KWV1+FBg3gmWcgKck+GUWeSAF2IW/9+hZbjm5h/MPjKZav2G1vTynFuPbjKBBQgL6z+pKWkWaHlMLphg2Dgwf1h2n589/+9nx99bZSU/WIWqYijJEC7CJWJ63mg+Uf8FjEY7Sr2s5u2y1TsAyj245mzaE1fLj8Q7ttVzjJ/Pkwfrwuwg88YL/tVq4MH34ICxbAhAn2267IFSnALiA9I52BPw2kXHA5Pmn1id23HxUeRfew7gxfOpzdJ3fbffvCQS5dgkGDIDT09uZ9s/Pkk9CsGQwdCn//bf/ti1uSAuwCJm+czOajm/mk1ScUDirskH2MaDUCf19/XlvymkO2Lxzg88/11MMXX0BQkP237+Ojt33xoj6TTjidFGDDLqZe5I1f3qB+ufp0qd7FYfspG1yWofcPJS4xjrWH1jpsP8JOTpzQqx7atYMmTRy3n2rV9Ch73DjYudNx+xE3JAXYsJGrR3Lo3CE+bPFhnpec5dQLD7xAyfwleXHxi9I9zdW9+y6cOwfvv+/4ff3f/0FgoF4dIZxKCrBBxy8e571l7/Fw1Yd5sMKDDt9fcGAwbzZ+k6X7ljJv1zyH70/k0b59MGoU9O0L4eGO31/p0vpDvmnTYPVqx+9PZJECbNA7v73D+ZTzvN/cCaMcm0G1B1G5WGVeWvwS6RnpTtuvyIU33tDzs//J/VmQeTZ0qC7EL74oy9KcSAqwIX+f/5sxCWPoU6sPoSVDnbbfAN8A/vvQf9lydAszt8902n5FDu3cqc9Se+YZCAlx3n6Dg3Xh/+03/SWcQgqwIZ+v+ZyU9BRebvCy0/fdNbQrlYtV5sPlH8pcsKv53/8gIAD+/W/n7/uxx6BUKfjgA+fv20tJATbgXPI5vlj7BR3v6Ui1EtWcvn9fH1+G3T+MtYfXsnTfUqfvX2TjyBF9hlrfvlCmjPP3ny+fbvIzbx5s2uT8/XshKcAGjF8/ntOXT/NSg5eMZegb0ZfSBUrzwXIZ7biMkSN1m8hhw8xleOopKFBAnyUnHE4KsJOlpKfw6apPaVyhMfVD6hvLEeQXxHP1n2PB7gVsOLLBWA5hc/YsjB6tW0ZWqWIuR9Giel1wbCzs328uh5eQAuxk323+jqSzSUZHv5merPskwQHB0iPCFXz5JZw5Ay+Zf13w/PO649on9j8tXlxLCrATWZbFxys/pkapGrSu3Np0HIoEFWFwbX3Zo/2nZbRjTGoqjBgBTZtCnTqm00D58tCrl27Sc/Kk6TQeTQqwE/22/ze2HN3CkPuGOPyst5x6pv4zAIxbN85wEi/2ww9w6BAMGWI6yRXPP697RHzzjekkHk0KsBONSRhDkaAiRIdHm46S5c7Cd9K+ansm/jGR5DS5bLkRY8ZAhQrQtq3pJFfUqgX336+zZcglrRxFCrCTHDl/hOnbptM/oj/5/e3QVNuOnqrzFEcvHGXGthmmo3if7dthyRIYPFg3SnclTz2lTwxZssR0Eo8lBdhJJqyfQFpGGk/UecJ0lH9ocXcL7i56N6MTRpuO4n3GjgV/f3j8cdNJ/qlrVyhRQq/OEA4hBdgJ0jLS+HLdlzSv1JyqxauajvMPPsqHJ+o8wbIDy9j892bTcbzHhQv6xIuuXfUZaK4mKEifHffjj3LtOAeRAuwEc3bM4eDZgzxV5ynTUbLVP6I/gb6BjEkYYzqK94iN1UvPnnzSdJLsDR6s54DHjzedxCNJAXaC0QmjKRdcjoerPWw6SraK5y9OVHgU3276lrPJZ03H8Q6jR+t2kw0bmk6SvUqVoE0bvU45NdV0Go8jBdjB9p/ez6LdixgQOQA/Hz/TcW7qidpPcD7lPPGJ8aajeL4//oD16/UI00WWJGbriSd0n4q5c00n8ThSgB3sm43fYGHRL6Kf6Si3dF/IfVQrXo1JGyaZjuL5Jk3SXc969jSd5NbatNG9gifJ68LepAA7UIaVwdcbvqbpXU2pWKSi6Ti3pJSif0R/VhxcwZ/H/zQdx3MlJ+uevx07QrFiptPcmp8fPPoozJkDR4+aTuNRpAA70G/7f2Pv6b30j+hvOkqOPVrrUXyUD99slDOgHOann/Qpvv3d53VB//6QlgZTpphO4lGkADvQpA2TKBRYiM7VO5uOkmN3BN9B68qtmbxxslyyyFEmTYJy5aBFC9NJci40FOrV09mlib/dSAF2kHPJ55i2dRpRYVEud+bbrfSP6M+hc4dYtGeR6Sie5/BhmD8f+vRxvTPfbqV/f9iyBdatM53EY0gBdpD4xHgupl50q+mHTA9XfZhi+YrJh3GO8O23el1tv36mk+RedLQ+OUM+jLMbKcAOMmnDJKoVr8Z9IfeZjpJrgX6B9K7Rm1nbZ3HykrQjtBvL0me+NWwIVV3vjMhbKlIEOneGqVPh8mXTaTyCFGAH2HNqD8sPLqdvrb4u03Yyt/pF9CMlPYXvE783HcVzJCTo5jt9+5pOknf9+sHp03pFhLhtUoAdIGZTDAC9avYynCTvIspEEFoylCmb5VNvu5kyBQIDde8Hd9W0qb5gqKyGsAspwHZmWRYxm2N4sMKD3Fn4TtNx8kwpRa8avVh2YBn7Tu8zHcf9paXp3g/t2+u38u7K1xd69NBnxcnVMm6bFGA7W/fXOv488Se9a/Q2HeW29ayhz9Kaunmq4SQeYPFifRJDb/d/XdC7t75687RpppO4PSnAdhazKYYA3wC6hrrx20ybikUq0vDOhkzZNAVL1n7enpgYPfJt08Z0ktt3771wzz36ZxK3RQqwHaVlpBGbGEu7Ku0omq+o6Th20atGL7Yd3yaXrr8dFy7AzJnQrZueA3Z3SumLdv72Gxw4YDqNW5MCbEdL9i7hyPkj9K7pAW8zbbqFdsPfx5+YzTLaybMfftBF2BOmHzJlNhGaKtNTt0MKsB3FbI6hcGBh2lZxoYsr3qbi+YvTpkobpm6eKqcm51VMjL7Uuyv3/c2tSpXggQf0agiZnsozKcB2cjH1IjO2zaBraFeC/IJMx7Gr3jV689f5v1i6b6npKO7n2DFYsECPGH087L9b796QmAibNplO4rY87BVhztydczmfcj5r5YAnaV+1PQUDChK7JdZ0FPczfTqkp7tH39/c6tpVL0uLizOdxG1JAbaT2C2xlC5QmsYVGpuOYnf5/PPR8Z6OTN82nZT0FNNx3EtsrO4kVqOG6ST2V7IkNG+uf0aZhsgTKcB2cDb5LLN3zKZ7WHd8fdysw1UORYdFc+ryKRbtlg5pOXbokF4pEB3t+pcdyqvoaNi7F9asMZ3ELUkBtoMftv9Acnoy0eHRpqM4TIu7W1A0qCixiTINkWPx8XpkGBVlOonjdOyoL60UK6+LvJACbAexibFUKFyB+0PuNx3FYQJ8A+hSvQuzts/iUuol03HcQ2wsREa6Z+eznCpSBNq21fPA6bJKJrekAN+mExdPsHD3QqLCoty281lORYdHcz7lPHN3ytVxb2nPHv22PNpz3xVliY6Gv/6CZctMJ3E7UoBv04xtM0jLSPPo6YdMTSo2oXSB0jINkROZKwO6dzebwxnat4f8+WUaIg+kAN+m2MRYqhavSkSZCNNRHM7Xx5duod2YvWM255LPmY7j2mJj9YkKFSqYTuJ4BQpAhw66OU9qquk0bkUK8G04cv4IS/ct9Yrph0w9avTgctplfvjzB9NRXNe2bfrkBG+YfsgUHQ3Hj8PPP5tO4lakAN+GaVunkWFlEBXmwZ9yX+e+kPsIKRRCfGK86SiuKy5OLztz58brudWqFRQqpFd+iByTAnwb4hPjCSsZRlipMNNRnMZH+dA9tDvzd83n9OXTpuO4HsvSRahxYyhb1nQa5wkKgkce0V3fUuRknZySApxHh84eYtmBZV41+s0UFR5FakYqP2yXaYh/2LJFT0F48trf7ERF6evFLZKTdXJKCnAeTds6DQuL7mFe8Cn3dereUZeKRSoSlyg9AP4hPl433enc2XQS52vRQq8LlmmIHJMCnEdxiXHUKl2LaiWqmY7idEopuod2Z9GeRZy4eMJ0HNdhWXr+t2lTKFXKdBrnCwiATp1g1iy5bH0OSQHOgwNnDrAyaaVXjn4zdQ/rTlpGGrO2zzIdxXVs3Ag7d3rH2t/sdO8OZ8/CwoWmk7gFKcB58H3i9wBeXYAjy0Zyd9G7ZRrianFxuj2jN04/ZGrWDIoVkxaVOSQFOA/it8YTWTaSysUqm45ijFKK7mHdWbJ3CccuHDMdx7zM1Q/Nm0Px4qbTmOPvr38B/fgjXJKeIbciBTiX9p7ay5pDa+ge6r2j30xRYVGkW+nM2DbDdBTz1q3T/R+8efohU1QUnD8P8+aZTuLypADnUuYJCN48/ZCpZumaVC1eVaYhQL/l9vfXH0J5uyZNdLN2mYa4JSnAuRS/NZ565epxV9G7TEcxTilFVFgUv+7/lSPnj5iOY07m9EPLllC0qOk05vn5QZcuMHu2vhq0yJYU4FzYdXIX6/9a75UnX2QnKiyKDCuD6Vunm45izurVcOCAd558kZ2oKLh4EebMMZ3EpUkBzoXM6Yduod0MJ3EdYaXCCC0ZSvxWL158Hx+v18B26GA6ieto1AjKlJGTMm5BCnAuxCXG8UD5ByhfuLzpKC4lKiyK3/f/zuFzh01Hcb6MDF1k2rSBwoVNp3Edvr66GdGcOXBOWpdmRwpwDm0/vp1Nf2+S1Q830D2sOxZW1vpor7Jihb74pqx++KeoKH1G3E8/mU7isqQA51B8YjwKRddQL2oxmEP3lLiHmqVreuc0RHy87gT28MOmk7ieBx6AcuVkGuImpADnUFxiHA3vbEi5QuVMR3FJ3UO7s+LgCg6eOWg6ivOkp8P33+uLUgYHm07jenx8oFs3vR74zBnTaVySFOAc2HJ0C1uPbZXVDzcRFa6PjVc1av/9dzhyRFY/3ExUlO4P/IO0Lr0RKcA5ELclDh/lI9MPN1G5WGVql63tXSdlxMXp66G1a2c6ieuqX19fF09OyrghKcC3YFkWsYmxNL2rKaULljYdx6VFh0ez9vBadp/cbTqK46Wm6otQduigi7C4MaX0KHjhQjghrUuvJwX4Ftb/tZ5dJ3cRHeZFF1jMo8zTs71iFLxkib4IpTddeDOvoqMhLQ1mSM+Q60kBvoXYLbH4+/jTqbqc438rdxa+kwblGxC7JdZ0FMeLjdXrflu1Mp3E9UVEQNWq+piJa0gBvokMK4O4xDhaVW5FsXzFTMdxC9Hh0Ww+upnEo4mmozhOcrIezXXuDIGBptO4PqX0KPiXX+Cvv0yncSlSgG9i5cGVHDx7UKYfcqFraFd8lI9nT0PMn6+v+iDTDzkXFaWbFk2bZjqJS5ECfBOxW2IJ8guiQzU5xz+nyhQsw0MVHyIuMQ7LskzHcYzYWChRQl/7TeRMaCjUrCnTENeRApyNtHNeEdoAABkpSURBVIw04rfG065KO4IDZZF9bkSFRbHjxA42HNlgOor9Xbigr/bQtatuuyhyLjpan7q9f7/pJC5DCnA2lu5bytELR4kOl7eZudW5emf8fPyYunmq6Sj299NPus2iTD/kXuYJKzIKziIFOBsxm2MIDgimXRVZZJ9bxfMXp3Xl1sQmxpJhZZiOY18xMRASotstitypVEmfmDHVA38x55EU4Bu4lHqJ6Vun0yW0C/n885mO45Z61ehF0tkkftv/m+ko9nP8uP4ArkcP3edA5F6vXrBpE2zZYjqJS5BX0Q3M2TmHcynn6FWjl+kobqtDtQ4U8C9AzKYY01Hs5/vv9QkFveR1kWfdu+tewTIKBqQA31DM5pisT/NF3uT3z0+n6p2Ytm0ayWnJpuPYx9SpVz7NF3lTujQ0b66PZYaHTU/lgRTg65y6dIq5O+cSHRaNr4+v6ThurVeNXpy+fJp5uzzg8uT798OyZXr0q5TpNO6tVy99PFeuNJ3EOCnA15m+bTop6Sn0qilvM29X80rNKVWgFDGbPWAaIvMtc48eZnN4go4dIV8+/YGml5MCfJ2YzTFULV6V2mVrm47i9vx8/IgKi+KnP3/izGU3bshtWbpYPPAA3HWX6TTuLzgYHnlEXykjNdV0GqOkAF8l6WwSv+77lZ7hPVHyNtMuetboSXJ6MjO2uXEnrM2bITERevY0ncRz9Oyp21MuWGA6iVFSgK8SsykGC0umH+yofrn6VC5WmW83fWs6St5NnqzPepMLb9pPq1b6dO5v3fh1YQdSgG0sy+Kbjd/QoHwDKherbDqOx1BK0admH37Z9wv7T7vhKahpaTBlir7qRcmSptN4joAAPZ/+ww9w6pTpNMZIAbZJOJzAtuPb6FOrj+koHufRWo8CuOcoeOFC+Ptv6NvXdBLP06ePbu3pxVdNlgJsM3njZAJ9A7Ou6iDsp2KRijSu0JjJGye7X4e0yZOhWDF95WNhX7Vr63XVkyebTmKMFGAgJT2F77Z8R8d7OlIkqIjpOB6pb62+7Dy5k1VJq0xHybnTp2HWLP1WWRqv259S+p3FihWwc6fpNEZIAQbm7JjDiUsn6FtL3mY6StfQruTzy8c3G78xHSXn4uP1W2SZfnCc3r11Xw0vHQVLAQYmb5pMmYJlaHF3C9NRPFZwYDCdq3cmLjGOy2mXTcfJmcmToXp1qFPHdBLPdccd+tTkb7/1ylOTvb4AH794nDk75tCrRi/8fKTBtiP1rdWX05dP89OfP5mOcmu7dsHy5Xr0K2vCHatvX31q8m8e1Dkvh7y+AE/ZNIXUjFSZfnCCpnc1JaRQCF9t+Mp0lFubNEm/Ne7d23QSz9exIxQqBF+5wevCzry6AFuWxYT1E6hXrh41StcwHcfj+fr40j+iPwt2LeDAmQOm42QvLU0X4DZtoFw502k8X/78+sy477/XH3x6Ea8uwKuSVpF4LJGBkQNNR/Eaj937GACT/phkOMlNzJunL58+UF4XTjNwIFy+7HUNery6AE9YP4EC/gWICosyHcVrVCxSkRZ3t+CrDV+RnpFuOs6NTZgAZcrI2l9nioyEe++F8eN18yMv4bUF+GzyWWITY+kR3kOueuxkAyMHcuDMARbtWWQ6yj8dPgxz5kC/fuDvbzqNdxk4EDZuhPXrTSdxGq8twLFbYrmYepEBkQNMR/E6Hap1oET+EkxYP8F0lH/6+mtIT4fHHzedxPv06KH7BI8fbzqJ03htAZ6wfgLhpcKpV66e6SheJ8A3gL61+vLDnz9w9MJR03GuyMiAiRPhoYegsjRkcroiRaBbN938/sIF02mcwisL8MYjG1l7eC0DIwdK319DBkQOIC0jjW82uNCZcb/8Anv2wAB5V2TMwIFw7pzXNOjxygI8eu1ogvyC6F1T1niack+Je3iwwoOMXTeWDMtFzoAaPRqKF4dOnUwn8V4NGugGPaNHm07iFF5XgE9fPs2UzVPoGd6TYvmKmY7j1Z6u+zR7Tu1h/q75pqPAwYO6N+3jj+t5SGGGUvDUU5CQAGvWmE7jcF5XgL/e8DUXUy/ydL2nTUfxep3u6UTZgmUZtWaU6SgwbpyeA37ySdNJRJ8++rpxo1zgdeFgXlWAM6wMRq8dzf0h9xNZNtJ0HK/n7+vP4NqDmbdrHrtO7jIXJDlZf/L+8MNQsaK5HEILDtZFOC4Ojh0zncahvKoAL9q9iJ0nd/Kvev8yHUXYDKo9CD8fP8asHWMuxLRpcPQoPC3vilzG009DSoo+KcaDeVUB/mLtF5QqUIou1buYjiJsygaXpUv1Lny14SsupBhaevTFF1C1qm6LKFxD9erQtCmMHat7c3gorynAe0/tZfaO2QyKHESgn1zdwJX8q96/OH35NFM3T3X+ztevh5Ur9YjLx2v+O7iHf/0LDhyA2bNNJ3EYr3nFjVw9El8fXwbXGWw6irhOg/INiCgTwYjVI5y/JO3TT6FgQbnqhSt6+GG48079b+ShvKIAn7p0ivHrxxMdHk1IoRDTccR1lFIMvX8oW49tde6StIMHITZWL/4vXNh5+xU54+cHQ4boRu0euiTNKwrwuHXjuJB6gWH3DzMdRWQjKiyKkEIhfLTiI+ft9LPPdOet555z3j5F7gwYoH85fvyx6SQO4fEFODktmc9Wf0aLSi2oVaaW6TgiG/6+/gypP4Sl+5aScDjB8Ts8cwa+/BKioqBCBcfvT+RNcDA88QRMn65PE/cwHl+Ap26eypHzR3jhgRdMRxG3MLD2QAoFFuLjFU4Y7Ywbp3sODJN3RS7v2WfB1xc++cR0Ervz6AKcYWXw8cqPqVW6Fs0ryRIjV1cosBBP1H6C77d+z95Tex23o5QUPf3QrJluAi5c2x136GvzffUVHD9uOo1deXQBnrtzLluPbWXYA8Ok65mbeLb+s/gqXz5Z6cDRTkyMbrwuo1/3MXQoXLqk12x7EI8twJZl8Z9f/0PFIhXlkkNupFyhcvSp1Yfx68dz+Nxh++8gLQ3eeQciIqBVK/tvXzhGWBh06AAjRuj5ew/hsQV43q55rD28ltcbvY6/r1xaxp282uhV0q10Plj2gf03HhMDu3fD8OG685ZwH//3f/qqySNHmk5iNx5ZgC3LYvjS4VQsUpE+tfqYjiNyqVLRSvSt1Zdx68bZdxSclgZvv61Hvx062G+7wjkiI/W/2yefeMwo2CMLsIx+3Z9DRsEy+nV/HjYK9rgCLKNfz2D3UbCMfj2Dh42CPa4Az9k5R0a/HiJzFPze7+/d/sa+/VZGv54icxT82Wemk9w2jyrAaRlpvLT4JSoXqyyjXw9QqWglHot4jLHrxrLzxM68b+jiRXjzTahTR0a/niAyUl+376OP4O+/Tae5LR5VgCf9MYmtx7byQfMPZPTrIYY3GU6gbyCv/PxK3jcyYgQkJel+AjL69Qzvvw+XL+t3NG7MYwrw+ZTzvPHLGzQo34BO98hVbT1F2eCyvNjgRaZvm87yA8tzv4G//4b33oNHHoHGje0fUJhRtaq+ft/48bB1q+k0eeYxBfij5R/x94W/+V/L/8lZbx5m6P1DuSP4DoYuHIplWbl78ltv6ZHShx86Jpww5803oUABeOkl00nyzCMK8KGzh/hoxUdEhUVRP6S+6TjCzgoEFOC/D/2X1YdW8/3W73P+xG3bdMezJ57QIybhWUqUgNde01fMWLLEdJo88YgC/OLiF0m30nm32bumowgH6VOrD7VK12LYwmGcTzl/6ydk9vktUECPlIRnevZZ3U70uecgNdV0mlxz+wK8cPdCpm6eyisNX6FS0Uqm4wgH8fXx5Yu2X3Dw7EGGLx1+6yd89x0sWqT7PpQs6fB8wpCgIH1SxpYtbtmu0q0L8KXUSzw15ymqFKvCyw1fNh1HOFiDOxswMHIgI1aNYOORjdk/8NQpeP55qFtXf1AjPFuHDtCxo57v3+vANqYO4NYF+J3f32H3qd2MbT+WIL8g03GEE7zf/H2K5SvG4NmDSc9Iv/GDXnlF940dN0438haeb+RI/W/91FN6+slNuG0B3npsKx8u/5A+tfrQ9K6mpuMIJymWrxiftvqU1YdWM27duH8+YMUKXXiHDJFm696kfHn4739h/nz4Phcf1BrmlgU4NT2VvrP6EhwYzMctPPNifSJ7PWv0pHml5ry0+CV2n9x95S8uXID+/fV/xrfeMhdQmPGvf0Ht2vD003DkiOk0OeKWBXj40uEkHE7gy/ZfUrKAfMDibZRSTOwwEV/lS++ZvbGwveX8979h50745hsoWNBsSOF8vr4weTKcP69/EbvBVITbFeDf9//Oe8veo39Ef7qEdjEdRxhyZ+E7Gdt+LKuSVrH/9H495/vll/oyQw89ZDqeMCU0VPeImD8fRo82neaW3KoAn7l8hkdnPkqlopX4rLX7d0IStyc6PJreNXvz14n9WH9u160m337bdCxh2tNPQ5s2+pexi5+m7DYF2LIsBvw0gKSzSUzpPIXgwGDTkYQLGNViBKEnFFZ6OqcmfgGBgaYjCdOU0ldQLlgQoqL0lISLcpsC/N6y95i2dRrvN3+f+0LuMx1HuIjCb/yXwpcsdhWHLptfJzXd/c6GEg5QpgxMnapHwP36uex8sFsU4Nk7ZvP6ktfpVaMXQ+8fajqOcBWTJ+tWk+XKUahCNX7Z9wvDFsql5oVNixa6CdP06fqMSBfkZzrArWw/vp2e03tyb9l7Gf/weOl0JrS1a2HQIP2BW3o6ZZSidf3WfLb6MyLKRND/3v6mEwpX8O9/w4YN8MYbULOmyzXkd+kRcNLZJFpPaU2QXxAzo2aSzz+f6UjCFezYAe3aQdmyEB+f1WT945Yf0+yuZgyePZh5O+cZDilcglJ6dUzt2tCjB6xcaTrRNVy2AB+7cIwW37bg5KWTzOs1jzsL32k6knAFBw5A8+b69oIFuiWhjZ+PH9O7Tye8VDhd4rvw+/7fDYUULiVfPt2y8o47oG1b2HiTPiJO5pIF+MzlM7SOac2+0/uY3XM2te+obTqScAV//63n9c6ehYULb9jjt3BQYeb3ns+dhe+k/XftWf/XegNBhcspUwYWL9YrI1q21CfsuACXK8AnLp6g1ZRWbPp7E9O7T+fBCg+ajiRcQVISNG2qv8+Zo9f8ZqNUgVIsenQRRYKK0PLblqw5tMaJQYXLqlBBtyi1LP3ZgQusEXapAnzwzEEaTWrEhiMbmNZtGm2rtDUdSbiC7duhQQM4eFAX3wYNbvmU8oXLs6TPEgoHFabpN01ZuHuhE4IKl3fPPfDzz5CeDo0aGZ8TdpkCvPXYVh746gEOnzvMwkcX8sg9j5iOJFzBmjXQsKG+rtuvv0KTJjl+6t3F7mb5Y8upUrwK7aa2Y+rmqY7LKdxHjRq6a16xYtCsGcydayyKSxTgaVuncd+E+0jLSOPXfr/KtIPQJk6EBx+EQoVg+fI8tZcsU7AMS/supUH5BvSa0YsXF71IWkaaA8IKt3LXXfo1Vb06PPwwvPsuZGQ4PYbRApyansq/F/ybbt93I7RkKGsGrKFWmVomIwlXcOkSPP44DBig3yauXg2VK+d5c4WDCrOg9wKerPMkH634iGaTm3HkvHu0KxQOVKqUflcVFaUv7vnII/pqKk5krAAnHk2k4aSGfLrqU56p9wy/9f+N8oXLm4ojXMXatVCvnj6X//XXdVcrO1zTLdAvkNHtRjO542TWHlpLxNgIZmybYYfAwq0VLAgxMTBqlF7WGBGhV9g4idMLcEp6Cm8tfYt7x93L7pO7iesax8g2IwnwDXB2FOFKLl7U3avuuw9OntTzcm+/bfdLCj1a61FWD1hN2eCydInvQtf4rjIa9nZK6Q5qv/+u1wy3aqX7CZ886fBdO60AW5bFzG0zqTmmJsN/HU63sG5se3ob3cO6OyuCcEUZGTBliu7j+r//wcCBenlQmzYO22WN0jVYM2AN7zV7j9k7ZnPPqHv4YNkHXEq95LB9CjdQv74+bfnVV+Hbb/WKiVGjICXFYbt0eAG2LIsle5fwwFcP0Dm+M0opZveYTUznGLmahTfLyNBnJ0VGwqOP6k+kf/kFxo6FwoUdvnt/X39ebvgyG5/YSKMKjXj555ep8nkVxq8bT3JassP3L1xUUJBu3JOQAGFh8Mwz+oO6mBhIs/+Htw4rwMlpyUzeOJnILyNpNrkZB88cZMLDE9j85GbaVW3nqN0KV3fxor5oZliY/vT53Dn47jv9gs/FEjN7qVaiGj/1+Ilf+/1K+cLlGTR7EBVGVODtX9/m2IVjTs8jXEREBCxZAvPmQXAw9O4NlSrpq22cPm233di1AFuWxZpDa3hm7jOEfBpC31l9SUlPYfzD49n5zE4ej3wcPx+Xb8Am7C0jQ8+vDRoE5crBE09AgQJ66mHbNoiOBh+zKyIfrPAgKx5bwcLeC4ksG8mbS98k5NMQusZ3Zdb2WaSkO+5tqHBRSkHr1rB+PfzwA9x9N7z4on4N9+qlPyC+zVHxbVfDy2mX+XXfr8zdOZfZO2ez59QeAn0D6VCtAwMiB9CiUgtpIemNLlzQI4i5c/XZawcP6qLbufOV5WUu9rpQStHi7ha0uLsF245tY2zCWGITY5m+bTpFg4rSunJr2lVpR6vKrSiRv8StNyg8g4+PbmPZoQP88Yd+Bxcfrxu+lyypG/y0bat7TBQpkqtNKysXneLr1Kljzfh5BhuObGDlwZUsP7icNYfWkJyeTJBfEE3vakrnezrTNbQrhYMcP4/nzZrY3q4vXbrUaA5An1u/b59+ca5YoRe4r1sHqam66LZoAV27QseO+s925shjkZaRxuI9i4ndEsu8XfM4euEoCkVYqTAalm9IgzsbcG+Ze6lWoppLvLtzqdeFJ0tO1tMT8fF6JHzqlC7UtWrpU+UbNNAnDlWuDL6+KKXWWZZV5/rN5KoA+4X4WekD0/VtHz9ql61Ng/INaFapGQ9VfEj69TqR0/+jpafDkSN6JHvgAOzerTtK7dgBmzfrDmUAAQFQt65+AbZooUe6Dr5Om7OORYaVwbrD61iwewHLDixjZdJKzibrnzvQN5CwUmFUK16NKsWqUKV4FSoUrkBIoRDKFSrntGWWUoANSEvTJwstXAjLlsGqVfqzDoD8+SE8HLVmzQ0LcK5+ZRfNV5S32r5FrdK1uLfsveT3z2+fH0DkXVqanmNNT//nV1qa/kpN1UtpUlN1T4XkZP390iX9deGCvnDhuXO6kJ46pb9OnoRjx+DoUf09Pf3afZcpA1Wq6A8oatW68hUUZOZYOJiP8qFuubrULVcXgPSMdLYe28qGIxvY+PdGNv29iZVJK4ndEovFtQOb4vmKU6pAKUoVKEXx/MUpGlSUokFFKRRYiODAYIIDgikQUIB8fvnI75+fIL8gAv0CCfQNJMA3gADfAPx9/fHz8cv68lW++Pr44qN88FX6u4WFwrWmdjyen9+VUS/o/3ObN+u+wxs3wqZN2T4111MQCQkJtxvXvR09qifjM+Xk+GX3mKvvz7x99ffrb1/11cT2tKW5iJ4j/v5QtOiVr1KlrnyFhED58vrr7rv1WUQuwNVGfclpyew9vZcDZw5w8MxBDp49yNELR7O+Tl46yanLpzh16RSX0uy89ngSFAgowPkdrnslYG9klykIpdQxYL89g+VBCeC44QyuQo7FFXIsrpBjcYWrHIsKlmX948SHXBVgV6CUSrjRbxJvJMfiCjkWV8ixuMLVj4VLtKMUQghvJAVYCCEMcccC/KXpAC5EjsUVciyukGNxhUsfC7ebAxZCCE/hjiNgIYTwCG5dgJVSw5RSllLKa0/MV0p9pJTarpTapJSaqZTK3cnoHkAp1Vop9adSapdS6mXTeUxRSpVXSv2ilNqmlEpUSj1nOpNJSilfpdQfSqnZprNkx20LsFKqPNACOGA6i2GLgHDLsmoCO4BXDOdxKqWUL/AF0AYIBXoopULNpjImDRhqWVZ14D7gaS8+FgDPAdtMh7gZty3AwKfAi4BXT2JblrXQsqzMnnirgBCTeQyoB+yyLGuPZVkpQCzwiOFMRliW9ZdlWettt8+hi085s6nMUEqFAO2ACaaz3IxbFmClVAfgkGVZG01ncTGPAfNMh3CycsDBq/6chJcWnasppSoC9wKrzSYxZgR6gOb8a83ngvn+edlQSi0Gytzgr14DXgVaOjeROTc7FpZl/WB7zGvot6AxzszmAm7Uecar3xUppQoC04EhlmWdNZ3H2ZRS7YGjlmWtU0o1MZ3nZly2AFuW1fxG9yulagB3ARttjd5DgPVKqXqWZXnk5W2zOxaZlFJ9gfZAM8v71hUmAeWv+nMIcNhQFuOUUv7o4htjWdYM03kMaQB0UEq1BYKAQkqpKZZl9Tac6x/cfh2wUmofUMeyLFdouOF0SqnWwCdAY8uyvO4iZkopP/SHj82AQ8BaoKdlWYlGgxmg9IjkG+CkZVlDTOdxBbYR8DDLstqbznIjbjkHLK4xCggGFimlNiilxpoO5Ey2DyD/BSxAf+gU743F16YB8CjQ1PZa2GAbBQoX5fYjYCGEcFcyAhZCCEOkAAshhCFSgIUQwhApwEIIYYgUYCGEMEQKsBBCGCIFWAghDJECLIQQhvw/kwuSLoUJnlcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import scipy.stats as stats\n",
    "import math\n",
    "\n",
    "mu1 = -2\n",
    "mu2 = 2\n",
    "variance = 1\n",
    "sigma = math.sqrt(variance)\n",
    "x1 = np.linspace(mu1 - 5*sigma, mu1 + 4*sigma, 100)\n",
    "x2 = np.linspace(mu2 - 5*sigma, mu2 + 4*sigma, 100)\n",
    "plt.plot(x1, stats.norm.pdf(x1, mu1, sigma)/1,color=\"green\")\n",
    "plt.plot(x2, stats.norm.pdf(x2, mu2, sigma)/1,color=\"red\")\n",
    "plt.axvline(x=-2,color=\"black\")\n",
    "plt.axvline(x=0,color=\"black\")\n",
    "plt.axvline(x=+2,color=\"black\")\n",
    "plt.text(-2.7,0.55,\"Sensative\")\n",
    "plt.text(-0.7,0.55,\"Balanced\")\n",
    "plt.text(1.7,0.55,\"Specific\")\n",
    "plt.ylim([0,0.53])\n",
    "plt.xlim([-5,5])\n",
    "plt.legend(['Negative','Positive'])\n",
    "plt.yticks([])\n",
    "#plt.set_yticklabels([])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import zscore\n",
    "\n",
    "# Prepare data - apply z-score to ALL x columns\n",
    "# Only do this if you have no categoricals (and are sure you want to use z-score across the board)\n",
    "x_columns = df.columns.drop('diagnosis').drop('id')\n",
    "for col in x_columns:\n",
    "    df[col] = zscore(df[col])\n",
    "\n",
    "# Convert to numpy - Regression\n",
    "x = df[x_columns].values\n",
    "y = df['diagnosis'].map({'M':1,\"B\":0}).values # Binary classification, M is 1 and B is 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import roc_curve, auc\n",
    "\n",
    "# Plot a confusion matrix.\n",
    "# cm is the confusion matrix, names are the names of the classes.\n",
    "def plot_confusion_matrix(cm, names, title='Confusion matrix', cmap=plt.cm.Blues):\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(names))\n",
    "    plt.xticks(tick_marks, names, rotation=45)\n",
    "    plt.yticks(tick_marks, names)\n",
    "    plt.tight_layout()\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')\n",
    "    \n",
    "\n",
    "# Plot an ROC. pred - the predictions, y - the expected output.\n",
    "def plot_roc(pred,y):\n",
    "    fpr, tpr, _ = roc_curve(y, pred)\n",
    "    roc_auc = auc(fpr, tpr)\n",
    "\n",
    "    plt.figure()\n",
    "    plt.plot(fpr, tpr, label='ROC curve (area = %0.2f)' % roc_auc)\n",
    "    plt.plot([0, 1], [0, 1], 'k--')\n",
    "    plt.xlim([0.0, 1.0])\n",
    "    plt.ylim([0.0, 1.05])\n",
    "    plt.xlabel('False Positive Rate')\n",
    "    plt.ylabel('True Positive Rate')\n",
    "    plt.title('Receiver Operating Characteristic (ROC)')\n",
    "    plt.legend(loc=\"lower right\")\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ROC Chart Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 426 samples, validate on 143 samples\n",
      "Epoch 1/1000\n",
      "426/426 - 2s - loss: 5.7210 - accuracy: 0.6291 - val_loss: 5.8248 - val_accuracy: 0.6224\n",
      "Epoch 2/1000\n",
      "426/426 - 0s - loss: 5.7210 - accuracy: 0.6291 - val_loss: 5.8248 - val_accuracy: 0.6224\n",
      "Epoch 3/1000\n",
      "426/426 - 0s - loss: 5.7210 - accuracy: 0.6291 - val_loss: 5.8248 - val_accuracy: 0.6224\n",
      "Epoch 4/1000\n",
      "426/426 - 0s - loss: 5.7210 - accuracy: 0.6291 - val_loss: 5.8248 - val_accuracy: 0.6224\n",
      "Epoch 5/1000\n",
      "426/426 - 0s - loss: 5.7210 - accuracy: 0.6291 - val_loss: 5.8248 - val_accuracy: 0.6224\n",
      "Epoch 6/1000\n",
      "Restoring model weights from the end of the best epoch.\n",
      "426/426 - 0s - loss: 5.7210 - accuracy: 0.6291 - val_loss: 5.8248 - val_accuracy: 0.6224\n",
      "Epoch 00006: early stopping\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x245f31bd848>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Classification neural network\n",
    "import numpy as np\n",
    "import tensorflow.keras\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Split into train/test\n",
    "x_train, x_test, y_train, y_test = train_test_split(    \n",
    "    x, y, test_size=0.25, random_state=42)\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(100, input_dim=x.shape[1], activation='relu',kernel_initializer='random_normal'))\n",
    "model.add(Dense(50,activation='relu',kernel_initializer='random_normal'))\n",
    "model.add(Dense(25,activation='relu',kernel_initializer='random_normal'))\n",
    "model.add(Dense(1,activation='linear',kernel_initializer='random_normal'))\n",
    "model.compile(loss='binary_crossentropy', \n",
    "              optimizer=tensorflow.keras.optimizers.Adam(),\n",
    "              metrics =['accuracy'])\n",
    "monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, \n",
    "    patience=5, verbose=1, mode='auto', restore_best_weights=True)\n",
    "\n",
    "model.fit(x_train,y_train,validation_data=(x_test,y_test),callbacks=[monitor],verbose=2,epochs=1000)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZxN9f/A8dfbNraxa0VjH7uYVFT0LUuFZK8oRbKrkCyJUJTiK0Q/lUoaaUMKKaUsiexLCDGWryV7xjLevz/OmekaM3fujLn3zvJ+Ph734Z79fY8z930/n885n4+oKsYYY0xisgQ7AGOMMWmbJQpjjDFeWaIwxhjjlSUKY4wxXlmiMMYY45UlCmOMMV5ZosjgRORREVkY7DjSEhE5LSKlgnDcMBFREckW6GP7g4hsEpF6KdguxdekiDQQka9Ssm1KiUiIiGwVkWsCedy0xBJFAInIbhE5635RHRSRaSKS15/HVNWPVbWBP4/hSURqi8gPInJKRE6IyFwRqRio4ycQz48i0slznqrmVdWdfjpeORGZJSJH3M+/XkSeE5Gs/jheSrkJq8zV7ENVK6nqj0kc54rkeJXX5CvAKI/9q4iccf+m9onIm/HPtYg0FpGV7npHReRjESkWb53rReRdETngXrtbRWSYiORR1XPAe0D/FMac7lmiCLwmqpoXqA7cDAwIcjwpktCvYhG5HVgIzAZuAEoC64Cl/vgFn9Z+mYtIaeBXYC9QRVXzA62ACCA0lY8VtM8erGOLyC1AflVdEW9RNfdvqi7QBnjSY5uWwAzgv0ARoBJwDvhFRAq66xQClgO5gNtVNRSoDxQASru7mgE8LiIhfvp4aZuq2itAL2A3cK/H9GvAPI/pEGAMsAf4HzAZyOWx/EFgLXAS+BNo5M7PD7wLHAD2ASOArO6yDsAv7vvJwJh4Mc0GnnPf3wB8DhwGdgG9PNYbCnwGTHeP3ymBz/czMCmB+d8CH7rv6wFRwEDgiHtOHvXlHHhs2x84CHwEFAS+dmM+5r4v5q4/EogBooHTwAR3vgJl3PfTgInAPOAUzhd9aY94GgB/ACeAScBPCX12d93pnv+fCSwPc4/9uPv5jgCDPJbXwvnCOu7+X04AcngsV6A7sB3Y5c77L05iOgmsBu70WD+re57/dD/baqA4sMTd1xn3vLRx12+Mc30dB5YBVeNdu/2B9ThftNnwuJ7d2Fe5cfwPeNOdv8c91mn3dTse16S7TiXgO+Bvd9uBiZy/IcDUePPi/i/d6U+Bie57Af4Cno+3TRZgI/CyOz0C2ABkSeLvdztQN9jfI8F4BT2AzPSK94dVzL04/+uxfBwwByiE8wt0LvCqu6yW+2VV373QbwTC3WVfAVOAPMA1wErgaXdZ3B8lcJf7pSLudEHgLE6CyOJ+kQwBcgClgJ1AQ3fdocAFoJm7bq54ny03zpfy3Ql87ieAA+77esBF4E2cpFAX5wurvA/nIHbb0e62uYDCQAv3+KHALOArj2P/SLwvdq5MFH+75zcb8DEQ6S4rgvPF19xd1ts9B4klioPAE17+/8PcY/+fG3s1nC/dCu7ymsBt7rHCgC3AM/Hi/s49N7HJs517DrIBfdwYcrrL+uFcY+VxvjSrAYXjnwN3ugZwCLgVJ8E8jnO9hnhcu2txEk0uj3mx1/NyoL37Pi9wW7zPnM3jWB3495oMxUmKfYCc7vStiZy/WUA/L/+X4e6+nvWYVqBkAvsaBix3368Ahvnw9zsHjx9PmekV9AAy08v9wzqN8+tOge+BAu4ywfnC9Pw1ezv//nKcAoxNYJ/Xul82niWPh4HF7nvPP0rB+YV3lzv9FPCD+/5WYE+8fQ8A3nffDwWWePlsxdzPFJ7AskbABfd9PZwv+zweyz8FXvThHNQDzuN+ESYSR3XgmMf0jySdKKZ6LLsf2Oq+fyz2y8Tj/O2Nvz+P5RdwS3mJLA9zj13MY95KoG0i6z8DfBkv7v8kcY0dw6mKAack9GAi68VPFG8Dw+Ot8wfuL2j32n0yges5NlEswfnyLZLIZ04sUTwMrPHx7+c7oEsCn+Oke90o8An/Jrc73HlXXC9AF2C7+357/P0mcvyPgSG+xJrRXtZGEXjN1KkDrYfzi6eIO78ozq/i1SJyXESOA/Pd+eD8kvszgf3dBGQHDnhsNwWnZHEZda72SJw/ToBHcC7+2P3cELsPdz8DcRJRrL1ePtcx4BJwfQLLrsepZolbV1XPeEz/hVOqSeocABxW1ejYCRHJLSJTROQvETmJ84VVIJmNxwc93v+D84sYN6a4z+yevygv+zlKwp/fp+O5DeFfuzc6nMRpuC0Sb9vL/g9EpI+IbHEbzo/jVEPGbpPYNZOQm4A+8f7/i+OcgwSPHU9HoBywVUR+E5HGPh43OTEeI+G2nho457ANzg+ePO782GsuqWvS1/+3UJxquUzHEkWQqOpPOL9mx7izjuBUA1VS1QLuK786jXTg/JGWvnJP7MUpURTx2C6fqlZK5NCfAC1F5CacP6rPPfazy2MfBVQ1VFXv9wzby+c5g1P90CqBxa1xSk+xCopIHo/pEsB+H85BQjH0walauVVV8+FUr4Hz699rzD44gFNScnYoIp7TCViEUw2WUm8DW4Gy7mcZyL+fI1bc5xGRO3HaDVoDBVW1AE71ZOw2iV0zCdkLjIz3/59bVT9J6Njxqep2VX0Y5wfKaOAz9/84qfOfnBjX4ySjhI6vqvopzjU4xJ39B05iv+yaFJEsOP9PsdfkIuAhd743FXBuzsh0LFEE1zigvohUV9VLOHXXY2Pv1xaRG0Wkobvuu8ATInKPiGRxl4Wr6gGcO43eEJF87rLSIlI3oQOq6hqcht+pwAJVjf2FtBI4KSL9RSSXiGQVkcrunSa+egHnzpBeIhIqIgVFZARO9dGweOsOE5Ec7pddY2CWD+cgIaE4yeW4e/fKS/GW/w+nvSUl5gFVRKSZe6dPd+A6L+u/BNQWkddF5Do3/jIiMl1ECvhwvFCcapTTIhIOdPVh/Ys4/5/ZRGQIkM9j+VRguIiUFUdVESnsLot/Xv4P6CIit7rr5hGRB0TEp7u1RKSdiBR1/w9jr6kYN7ZLJP5/8DVwnYg8I87zCqEicmsi636D06blzSigs4hc55YA+wKDReQR97q+Due85APGutu86U5/4P6Air3u3hSRqrHTOG1D8e+4yhQsUQSRqh4GPsSpnwfn1+EOYIVb9bAI59cyqroSp1F4LM6vxp9wqgvAqUvPAWzGKZ5/hvei9CfAvTi3/MXGEgM0wanj34Xz634qTlWGr5/nF6AhTuPvAZwqpZuBO1R1u8eqB9049+NUfXVR1a1JnYNEjMNpGD6C80c8P97y/+KUoI6JyHhfP4v7eY7g/Bp9Dad6oiLOnT3nEln/T5ykGAZsEpETOCW2VTjtUknpi1MdeArni3tmEusvwLmjbBvOuY7m8uqhN3HafxbiJKB3cc4VOG1OH7jVTK1VdRVOm9UEnP+bHThtCb5qhPOZT+Oc87aqGq2q/+DcfbbUPdZtnhup6imcGzSa4FwX24G7EzqAqv4OnPCSSFDVDTh/G/3c6ZlAe+BZnGtks3sO6qjqUXedv4HaOG1Mv4rIKZzSxgn3PIDz//KBOs9UZDqxd78YExDiPMk7XVW9VeGkSW7VRBTO7byLgx1PZiQiDYBuqtosgMcMwalyuktVDwXquGlJmnpgyZi0xq32+hWneqsfTv1/pqx+SAtUdSFOCSmQxzyHc+NJpmVVT8Z4dzvOXTlHcKpHmqnq2eCGZExgWdWTMcYYr6xEYYwxxqt010ZRpEgRDQsLC3YYxhiTrqxevfqIqhZNes0rpbtEERYWxqpVq4IdhjHGpCsi8ldKt7WqJ2OMMV5ZojDGGOOVJQpjjDFeWaIwxhjjlSUKY4wxXlmiMMYY45XfEoWIvCcih0RkYyLLRUTGi8gOEVkvIjX8FYsxxpiU82eJYhpO18OJuQ8o67464wzaYowxJo3xW6JQ1SU4g9Yn5kHgQ3dkqhU4w1f6MhyhMcYYH6gqX3zxBdXubnJV+wlmG8WNXD7ISpQ77woi0llEVonIqsOHDwckOGOMSc92795NkyZNaNGiBX9t35r0Bl4EM1HEHwsYEhlfV1XfUdUIVY0oWjRFXZUYY0ymoaq0aNGCH3/8kTfeeIP6A9+7qv0FM1FEAcU9povhDI1pjDEmBZYtW8apU6cQEaZOncqWLVt47rnnyJL16rr1C2aimAM85t79dBtwQlUPBDEeY4xJl44ePcpTTz1FnTp1eOONNwC4+eabKV68eBJb+sZvvceKyCdAPaCIiEQBLwHZAVR1MvANcD/O4OX/AE/4KxZjjMmIVJUPP/yQvn37cuzYMfr160e/fv1S/Th+SxSq+nASyxXo7q/jG2NMRte/f39ef/11ateuzeTJk6lSpYpfjpPuxqMwxpjM7OzZs5w5c4YiRYrQsWNHypYtS8eOHcmSxX8tCdaFhzHGpBPz58+ncuXKPP300wCUL1+ep556yq9JAixRGGNMmrd//35at27NfffdR/bs2enRo0dAj29VT8YYk4Z9//33PPTQQ5w/f57hw4fTr18/QkJCAhqDJQpjjEmDLly4QPbs2alWrRr3338/I0aMoEyZMj5tO+PXPcxeuy9uevOBk1cVi1U9GWNMGnLy5El69+7NnXfeSUxMDEWKFCEyMtLnJAEwe+2+y5JDxevzXVVMVqIwxpg0QFX57LPP6N27NwcPHqRbt26cO3eO3Llzp2h/Fa/Px8ynb4+b/rRLymOzRGGMMUF2+PBhHn/8cb799ltuvvlmZs+ezS233BLssOJYojDGmCDLly8fR44cYdy4cXTv3p1s2ZL31ZxQm8TVVjd5sjYKY4wJgiVLltCwYUNOnz5NSEgIK1asoHfv3slOEpBwm8SD1RMctSFFrERhjDEBdOTIEfr168e0adMICwtj9+7dVK5c+aofmovfJpGarERhjDEBoKq89957lC9fnunTpzNgwAA2bdpE5cqVgx1akqxEYYwxATJ9+nQqVqzI5MmTqVSpUrDD8ZmVKIwxxk/++ecfBg8eTFRUFCLC559/zk8//ZSukgRYojDGGL/45ptvqFSpEiNHjmTu3LkAFCxY0O8d+PlD+ovYGGPSsKioKFq2bMkDDzxArly5+Omnn+jatWuww7oq1kZhjDGpaOTIkcybN49XXnmFPn36kCNHjlTZb/xnJTyl9nMT8VmJwhhjrtLKlSvZsGEDACNGjGDTpk0MGDAg1ZIEXPmshKfUfm4iPitRGGNMCp04cYKBAwfy9ttv07hxY+bMmUPhwoUpXLiwX47nz2clvLEShTHGJJOqEhkZSXh4OJMnT6Znz55Mnz492GH5jZUojDEmmaZPn85jjz1GREQEX3/9NTVr1gx2SH5licIYY3xw7tw5du7cSYUKFWjdujUXL17kscceI2vWrMEOze+s6skYY5KwePFiqlWrRsOGDTl37hwhISE88cQTmSJJgJUojDEmUYcOHaJv37589NFHlCpVinfeeSdg41X7u+vw5LBEYYwxCdixYwe1atXi9OnTDBo0iEGDBpErV66AHT/2dtjY5ODvW2C9sURhjDEeTp48Sb58+ShdujQdO3bkySefpEKFCkGJJVi3w8ZnbRTGGAOcOXOG/v37ExYWFteJ3+uvvx60JJGWWInCGJPpzZ07lx49erBnzx46duxI7ty5/XIcb91wxBfMNon4rERhjMm0Ll68SPPmzWnatCmhoaH8/PPPTJ06lUKFCvnleN664YgvmG0S8VmJwhiT6agqIkK2bNm4/vrrGTVqFM8++2yq9s2UmLTS7pAcVqIwxmQqK1asICIigt9//x2AiRMn0r9//4AkifTKShTGmEzh2LFjDBw4kClTpnDDDTdw7Nixq95nctocIG21OySHX0sUItJIRP4QkR0i8kICy0uIyGIRWSMi60Xkfn/GY4zJnGbOnEl4eDjvvPMOzzzzDFu2bOGee+656v0mp80B0la7Q3L4rUQhIlmBiUB9IAr4TUTmqOpmj9UGA5+q6tsiUhH4BgjzV0zGmMxp69athIWFMX/+fG6++eZU3Xd6bHNILn+WKGoBO1R1p6qeByKBB+Oto0BsOSw/sN+P8RhjMono6GiGDRsWN1b1wIEDWbZsWaoniczCn20UNwJ7PaajgFvjrTMUWCgiPYE8wL0J7UhEOgOdAUqUKJHqgRpjMo5FixbRrVs3tm/fTp8+fWjSpAnZs2dP8f6COQRpWuHPEoUkME/jTT8MTFPVYsD9wEcickVMqvqOqkaoakTRokX9EKoxJr373//+x6OPPkr9+vVRVRYuXMiYMWOuer/BHII0rfBniSIKKO4xXYwrq5Y6Ao0AVHW5iOQEigCH/BiXMSYD+u677/jss88YMmQIAwYMIGfOnKm278zQDuGNPxPFb0BZESkJ7APaAo/EW2cPcA8wTUQqADmBw36MyRiTgaxbt47t27fTsmVLHn30UerUqUPJkiWDHVaG47eqJ1W9CPQAFgBbcO5u2iQiL4tIU3e1PsBTIrIO+ATooKrxq6eMMeYyp0+fpk+fPtSsWZMXXniBixcvIiKWJPzErw/cqeo3OLe8es4b4vF+M1DHnzEYYzKWr776ip49exIVFUXnzp159dVXyZbNnh32Jzu7xph0Y8OGDTz00ENUqVKFmTNnUrt27WCHlClYX0/GmDTtwoUL/PDDDwBUqVKFefPmsXr1aksSAWQlCmNMmrVs2TK6dOnCpk2b+OOPPyhTpgz335+6Pf0k1V9TZnlWwhsrURhj0py///6bzp07U6dOHY4fP84XX3xBmTJl/HKspPpryizPSnhjJQpjTJoSHR1N9erV2b9/P3369GHo0KHkzZvXr8fM7M9JJMUShTEmTYiKiqJYsWLkzJmT4cOHU716dapVqxbssAxW9WSMCbKzZ88yZMgQSpcuHdeJ3+OPP25JIg3xqUQhIjmAEqq6w8/xGGMykYULF9KtWzf+/PNP2rVrR61atYIdkklAkiUKEXkA2AB8505XF5Ev/R2YMSZj69mzJw0bNiRLliwsWrSIjz76iGuvvTbYYZkE+FKieBmne/DFAKq6VkT8c/uBMSZDm75sF7PX7SNLlqz8daYwlRp3JLxhO97ZEcI7O5YHJSa7/TVpviSKC6p6XOSyXsOtPyZjTLL8/vvv9H6kHSEV6nFns/bcdGvDYIcE2O2vvvAlUWwRkdZAFrcn2N7ACv+GZYzJKE6dOsWQIUMYP348OfIWoHzxG+1W1HTGl0TRAxgCXAK+wOkNdoA/gzLGZAwLFy7kySefZP/+/XTp0oUDZZuRI3dosMMyyeRLomioqv2B/rEzRKQ5TtIwxphELdlxjDNZ8vCf59/haMlK7Dhwkoq5gx2VSS5fnqMYnMC8QakdiDEm/btw4QKjR49m0CDnK2J71hLc+MQ4CpesBFh7QHqVaIlCRBriDFN6o4i86bEoH041lDHGxPnll1/iOvBr1aoVly45XxOVbihgbRLpnLcSxSFgIxANbPJ4LQTu839oxpj04OjRo3Tq1Ik777yTU6dOMXfuXD799FOyZLGOHzKKREsUqroGWCMiH6tqdABjMsakI0ePHiUyMpLnn3+e8vc9wUd/HOOjKc4zEfaMQsbgS8q/UUQiRWS9iGyLffk9MmNMmrVlyxaGDRsGQLly5dizZw+jR49mwR/HLuuy29okMgZf7nqaBowAxuBUOT2BtVEYkyn9888/jBw5ktdff528efPSsWNHihUrRqFCheLWsS67Mx5fShS5VXUBgKr+qaqDgbv9G5YxJq2ZP38+lStX5pVXXuGRRx7hjz/+oFixYsEOywSALyWKc+L03/GniHQB9gHX+DcsY0xacvr0adq3b0/hwoVZvHgx9erVC3ZIJoB8KVE8C+QFegF1gKeAJ/0ZlDEm+GJiYpg+fToxMTHkzZuXRYsWsW7dOksSmVCSJQpV/dV9ewpoDyAiVt40JgNbvXo1Tz/9NKtXryZXrly0aNHCBhLKxLyWKETkFhFpJiJF3OlKIvIh1imgMRnSiRMn6NWrF7Vq1WLfvn1ERkbSvHnzYIdlgszbk9mvAi2AdcBgd7Ci3sBooEtgwjPGBFKLFi344Ycf6N69OyNGjCB//vxXrDPj1z3MXrsvwe3tuYmMyVvV04NANVU9KyKFgP3u9B+BCc0YEwg7d+6kaNGihIaGMnLkSLJkycItt9yS6Pqz1+5LNCHYcxMZk7dEEa2qZwFU9W8R2WpJwpiM4/z584wZM4bhw4fTq1cvRo8eza233urTtvasRObiLVGUEpHYrsQFCPOYRlWt4tKYdGrJkiV06dKFLVu20LJlS3r16pXouvGrmqx6KfPxlihaxJue4M9AjDGBMXbsWJ577jnCwsKYN28e999/v9f141c1WfVS5uOtU8DvAxmIMcZ/Ll26xJkzZwgNDeWBBx7g8OHDDB48mNy5fRtFyKqaMjfrB9iYDG7Tpk3UrVuXDh06AE4nfq+88orPScIYX7rwSDERaQT8F8gKTFXVUQms0xoYCiiwTlUf8WdMxmRECd2yevF8NJvnvc8f380ge668VGvRndaTl+H0yOM7a5MwPicKEQlR1XPJWD8rMBGoD0QBv4nIHFXd7LFOWWAAUEdVj4mI9SFlTArEb0c4tucPlk0ZyJmjBwir/QDVmncnJG+BFO3b2iRMkolCRGoB7wL5gRIiUg3opKo9k9i0FrBDVXe6+4nEeTZjs8c6TwETVfUYgKoeSv5HMMaA84Ue2fk2RISjR8vR/Jf3GT48krvuuivYoZl0zpc2ivFAY+AogKquw7duxm8E9npMR7nzPJUDyonIUhFZ4VZVGWOS6VLMRbZ9P5N77rmHmJgYChcuzE8//WRJwqQKX6qesqjqX/HqNWN82C6hilBN4PhlgXpAMeBnEamsqscv25FIZ6AzQIkSJXw4tDHpj7euMbw5umszSz94leiDf3Lfffdx8uRJChYs6IcITWblS4lir1v9pCKSVUSeAXwZCjUKKO4xXQynG5D468xW1Ququgv4AydxXEZV31HVCFWNKFq0qA+HNib9iW1n8NWF6H9Y/ckYvn/tKeTsCXq9Mol58+ZZkjCpzpcSRVec6qcSwP+ARe68pPwGlBWRkjiDHbUF4t/R9BXwMDDN7aG2HLDTt9CNyXiS87zCuXPnqPFOL3r17Mnw4cPJl8/uTDL+4UuiuKiqbZO7Y1W9KCI9gAU4t8e+p6qbRORlYJWqznGXNRCRzTjVWf1U9Whyj2VMZrFjxw5efvllJk6cSGhoKKtXryZnzpzBDstkcL4kit9E5A9gJvCFqp7ydeeq+g3wTbx5QzzeK/Cc+zIm0/Fsl/D2vMK5c+d47bXXGDlyJDly5OCpp57izjvvtCRhAiLJNgpVLQ2MAGoCG0TkKxFJdgnDGHMlz3aJxJ5XWLx4MdWqVWPIkCE0a9aMrVu3cueddwY6VJOJ+fTAnaouA5aJyFBgHPAxEOnHuIzJNLy1S6gqI0eO5MKFC8yfP5+GDRsGODpjfHvgLi/Og3JtgQrAbKC2n+MyJtO6dOkS7777Lo0aNaJ48eJ89NFHFChQgFy5cgU7NJNJ+VKi2AjMBV5T1Z/9HI8x6U5Kn3+AK9sl1q9fT5cuXVi+fDlDhgxh2LBhXH/99akVqjEp4kuiKKWql/weiTHplLehQZMS2y5x+vRphg0bxtixYylYsCDTpk3jscce80O0xiRfoolCRN5Q1T7A5yIS/4lqG+HOGA9XO15D3759eeONN+jUqROjRo2icOHCqRidMVfHW4lipvuvjWxnjB/s3buXM2fOEB4ezgsvvECzZs244447gh2WMVfwNsLdSvdtBVW9LFm4D9LZCHgmQ0pum0Nyq50uXrzI+PHjGTJkCDVr1uSnn36iSJEiliRMmuVLX09PJjCvY2oHYkxakdw+l5IzXsOKFSuIiIigT58+1KtXjw8++CClYRoTMN7aKNrg3BJbUkS+8FgUChxPeCtjMgZ/jBE9b948mjRpwg033MAXX3xBs2bNkj3anDHB4K2NYiXOGBTFcEaqi3UKWOPPoIzJKFSV/fv3c+ONN3Lvvffy8ssv07t3b0JDQ4MdmjE+89ZGsQvYhdNbrDEmmbZt20a3bt3Ytm0bmzdvJm/evAwePDjYYRmTbIm2UYjIT+6/x0Tkb4/XMRH5O3AhGpO+REdHM3ToUKpUqcKqVasYMGCAPVVt0jVvVU+xw50WCUQgxmQEBw8e5K677mL79u08/PDDvPnmm1x33XXBDsuYq5JoicLjaeziQFZVjQFuB54G8gQgNmPSjQsXLgBw7bXXctddd7Fw4UJmzJhhScJkCL7cHvsVzjCopYEPcToGnOHXqIxJJy5dusTkyZMpXbo0UVFRiAhTp06lfv36wQ7NmFTjS6K4pKoXgObAOFXtCfh207gxGdi6deuoXbs2Xbt2pWzZsnGlCmMyGl8SxUURaQW0B75252X3X0jGpG2qSt++falZsyY7d+7ko48+YtGiRZQsWTLYoRnjF770Hvsk0A2nm/GdIlIS+MS/YRnjX9666UiqSw4R4dixY3Ts2JFRo0ZRsGBBf4VpTJrgy1CoG4FewCoRCQf2qupIv0dmjB9566YjoS45/vrrL5o1a8bvv/8OwP/93/8xZcoUSxImU/BlhLs7gY+AfYAA14lIe1Vd6u/gjPEnX7rpuHDhAmPHjmXYsGEAtGnThho1apAliy+1tsZkDL5UPY0F7lfVzQAiUgEncUT4MzBjgm3ZsmU8/fTTbNy4kQcffJDx48dTokSJYIdlTMD5kihyxCYJAFXdIiI5/BiTMakufpuEL12DL1q0iBMnTvDVV1/x4IMP+jtEY9IsX8rPv4vIFBG5w329jXUKaNKZ+G0SCbVDqCoffvgh3377LQD9+/dn8+bNliRMpudLiaILTmP28zhtFEuAt/wZlDH+4K1NYuvWrXTt2pUff/yRVq1acd999xESEkJISEiAozQm7fGaKESkClAa+FJVXwtMSMYEztmzZ3nllVcYPXo0efLkYcqUKXTq1CnYYRmTpngbuGggzkh2vwO3iMjLqvpewCIzJp7kDmwYDOAAAB/BSURBVFHqKbE2iblz5zJixAjatWvHmDFjuPbaa682TGMyHG8likeBqqp6RkSKAt8AlihM0MS2MyRnfOpYnm0SBw8eZO3atTRq1IhWrVoRFhZGrVq1UjtcYzIMb4ninKqeAVDVwyJiN46boLuaIUpjYmKYNGkSAwYMIEeOHOzZs4dcuXJZkjAmCd4SRSmPsbIFKO05draqNvdrZMakot9//50uXbrw22+/ce+99zJp0iQbTMgYH3lLFC3iTU/wZyDGJNUGkdJqp127dlGrVi2KFCnCjBkzaNu2LSJyNaEak6l4GzP7+0AGYkxSbRAJPfuQGFVlw4YNVK1alZIlS/L+++/TpEkTChQokJohG5Mp+PIchTEBczVtELF27dpFjx49mD9/PmvWrKFq1aq0b98+lSI0JvPxawO1iDQSkT9EZIeIvOBlvZYioiJi/UdlMjN+3UObKctpM2V5or25+ur8+fOMGjWKSpUq8dNPPzFmzBgqVqyYSpEak3n5XKIQkRBVPZeM9bMCE4H6QBTwm4jM8ew3yl0vFOfJ71993bfJODyrm5JTtRRfTEwMtWvXZvXq1TRv3pxx48ZRvHjxVI7WmMzJl27GawHvAvmBEiJSDejkDonqTS1gh6rudPcTCTwIbI633nDgNaBvMmM3GcTVVDedPHmSfPnykTVrVp588kmGDh1K48aNUzlCYzI3X6qexgONgaMAqroOuNuH7W4E9npMRxFvrG0RuRkorqpf44WIdBaRVSKy6vDhwz4c2mR0qsq0adMoVaoUs2fPBqBbt26WJIzxA1+qnrKo6l/xbieM8WG7hO4/1LiFzgN8Y4EOSe1IVd8B3gGIiIjQJFY3aVhKuvuOb/PmzXTt2pUlS5ZQp04dSpcundphGmM8+FKi2OtWP6mIZBWRZ4BtPmwXBXhWEhcD9ntMhwKVgR9FZDdwGzDHGrQzNl+6+/bmtddeo1q1amzcuJGpU6eyZMkSKleu7I9QjTEuX0oUXXGqn0oA/wMWufOS8htQVkRK4gyj2hZ4JHahqp4AisROi8iPQF9VXeVr8CZ9SkmbhKoiIlx33XU8+uijvP766xQtWtRPERpjPCVZolDVQ6raVlWLuK+2qnrEh+0uAj2ABcAW4FNV3SQiL4tI06sP3WQG+/fvp1WrVrz1ljMEymOPPca0adMsSRgTQL7c9fR/eLQtxFLVzkltq6rf4PQ66zlvSCLr1ktqfybziO3Ab9CgQVy4cIHatWsHOyRjMi1fqp4WebzPCTzE5XczGZOq1q5dS6dOnVi9ejUNGjRg0qRJ1mBtTBAlmShUdabntIh8BHznt4hMpnfixAn279/PzJkzadWqlXXgZ0yQpaSvp5LATakdiMm8VJVZs2axfft2Bg0aRN26ddm5cyc5c+YMdmjGGHxrozjGv20UWYC/gUT7bTLGU1LPTfz5559xHfjdcsstPP/882TPnt2ShDFpiNe7nsQp81cDirqvgqpaSlU/DURwJv1L7LmJc+fOMXLkSCpXrszSpUv573//y7Jly8iePXsQozXGJMRriUJVVUS+VNWagQrIZDwJPTexY8cOhg8fTpMmTRg3bhw33piyzgCNMf7ny5PZK0Wkht8jMRne4cOHmTDBGSixTJkybN68mVmzZlmSMCaNE9WEu04SkWyqelFENgAVgD+BMzh9OKmqBiV5RERE6KpV9vB2WpVQm0SFa/PSMMdWnn/+eU6dOsWGDRsoX758EKM0JvMRkdWqmqIukrxVPa0EagDNUhSVyZTiD2d646XDrJ/yIrPW/cadd97J5MmTLUkYk854SxQCoKp/BigWk0HEtkmcP3+esLAWnD9/nvfee48OHTrYMxHGpEPeEkVREXkusYWq+qYf4jEZwP+2riImphY5cuTg008/JTw8nCJFiiS9oTEmTfKWKLICeUl4XAljrhAVFcXSKQPZt+ZHPqyalyeeeII77rgj2GEZY66St0RxQFVfDlgkJt26ePEiEyZM4MUXX+TsuQtUadaFRx99NNhhGWNSSZJtFMYkpX379kRGRnLfffchdTqSt8gN5MiRI9hhGWNSibfnKO4JWBQm3Tl+/DinT58GoHv37syaNYt58+aRt8gNQY7MGJPaEk0Uqvp3IAMx6YOqEhkZSYUKFXjxxRcBuOOOO2jZsqXd0WRMBuXLk9nGAE63Gw0bNuThhx+mWLFitGvXLtghGWMCwBKF8cmMGTOoXLkyv/76KxMmTGDFihXUrGldgBmTGaRkPAqTiVy4cIHs2bMTERFBy5Ytee211/hx70UemboywfXjdyNujEn/rERhEnTo0CHat29PmzZtAChXrhzTp0/nhhtuuKLrcE+x3YgbYzIOK1GYy1y6dImpU6fSv39/zpw5Q//+/YmJiSFr1qyXrZdQ1+HGmIzJEoWJs3PnTtq1a8fy5cupV68eb7/9NuHh4cEOyxgTZJYoTJz8+fNz/PhxPvjgA9q3b2+3uxpjAGujyPTmzJlD8+bNiYmJoXDhwmzcuJHHHnvMkoQxJo4likxqz549NGvWjAcffJBt27Zx4MABALJksUvCGHM5+1bIZC5evMiYMWOoUKECCxcuZPTo0axZs4ZixYoFOzRjTBplbRSZTExMDFOnTuU///kPb731FmFhYcEOyRiTxlmJIhM4duwY/fv359SpU4SEhLB06VLmzJljScIY4xNLFBmYqvLxxx8THh7OG2+8weLFiwEoXLiwNVYbY3xmiSKD2rZtG/Xr16ddu3aEhYWxatUqmjZtGuywjDHpkLVRZFDPPPMMq1atYtKkSXTu3PmKJ6uNMcZXligykO+++47w8HCKFy/O22+/TUhICNddd12wwzLGpHN+rXoSkUYi8oeI7BCRFxJY/pyIbBaR9SLyvYjc5M94MqqDBw/yyCOP0KBBA0aPHg3ATTfdZEnCGJMq/JYoRCQrMBG4D6gIPCwiFeOttgaIUNWqwGfAa/6KJyO6dOkSkydPJjw8nM8//5yXXnqJMWPGBDssY0wG488SRS1gh6ruVNXzQCTwoOcKqrpYVf9xJ1cA9tRXMrz66qt07dqVmjVrsn79eoYOHUrOnDmDHZYxJoPxZxvFjcBej+ko4FYv63cEvk1ogYh0BjoDlChRIrXiS5dOnTrFkSNHKFmyJF26dKFkyZI8/PDDdrurMcZv/FmiSOibSxNcUaQdEAG8ntByVX1HVSNUNaJo0aKpGGL6oap8+eWXVKxYkTZt2qCqFC5cmEceecSShDHGr/xZoogCintMFwP2x19JRO4FBgF1VfWcH+NJt/766y969OjB119/TdWqVRk/fnySyWHGr3uYvXafX+Kx4U6NyVz8WaL4DSgrIiVFJAfQFpjjuYKI3AxMAZqq6iE/xpJuLV++nIoVK/LDDz8wZswYVq9ezW233Zbkdt6GK71aNtypMZmL30oUqnpRRHoAC4CswHuquklEXgZWqeocnKqmvMAs9xfyHlW1x4eBkydPki9fPmrUqMGTTz5Jv379kt0+Y8OVGmNSg18fuFPVb4Bv4s0b4vH+Xn8ePz06evQoL7zwAgsXLmTTpk3kzZuXt956K9hhGWMyMXsy2898bStQVf76dT7rPnuL8/+cotw9bXn8vZVkC8mVouNaO4IxJrVYovCz2LYCb1/a58+eZunbL3B42+8ULlWZmo88T4FiZa7quNaOYIxJLZYoAiCxtgJVRURQVVovLUWDvl3o2LGjDUdqjElT7BspSBYsWECNGjWIiopCRJg1axZPPfWUJQljTJpjJYpU4K0dIn6104EDB3j22WeZOXMm5cqV49ChQzZetTEmTbOfr6nA2zMLnm0FEydOJDw8nK+++ophw4axfv16atSoEchQjTEm2axEkUp8eWZh9erV3HrrrUycOJGyZcsGKDJjjLk6VqLwo5MnT/LMM8+wevVqACZNmsSCBQssSRhj0hUrUaRA/DaJ+O0Qqsrnn39O7969OXDgACVKlKBmzZrWBbgxJl2yEkUKxG+T8GyH2LVrF40bN6ZVq1Zcc801LF++nOeeey5YoRpjzFWzEkUKJdYm8fHHH7NkyRLGjh1Ljx49yJbNTrExJn2zb7FU8PPPP3Pu3Dnuvfde+vXrR4cOHeyWV2NMhmGJwgeJtUkcOXKE559/nvfff58777yTe++9l5CQEEsSxpgMxRKFD+L311ThulAKRC0lPLwJJ06coH///rz44otBjtKkNRcuXCAqKoro6Ohgh2IykZw5c1KsWDGyZ8+eavu0ROEjzzaJefPm0bhrP+rUqcPkyZOpXLlykKMzaVFUVBShoaGEhYXZcLUmIFSVo0ePEhUVRcmSJVNtv3bXk48uno9m6dKlANx///3Mnj2bJUuWWJIwiYqOjqZw4cKWJEzAiAiFCxdO9VKslSh8cGDjcn6PfIPvzp1iz549FChQgKZNbSA+kzRLEibQ/HHNWYnCi3379tGqVSt+ntCHLNmyM3fuXAoUKBDssIwxJqAsUSTi0KFDVKxYka+//prKTTvTYPCH1K1bN9hhGZMsWbNmpXr16lSuXJkmTZpw/PjxuGWbNm3iP//5D+XKlaNs2bIMHz4cVY1b/u233xIREUGFChUIDw+nb9++wfgIXq1Zs4ZOnToFOwyvXn31VcqUKUP58uVZsGCB13V79uxJ3rx546b37NnD3Xffzc0330zVqlX55htnZOkNGzbQoUMHf4Z9GUsU8ezbt48Zv+6h55d/UqrRk/xn0Idwc3OyZku9OwiMCZRcuXKxdu1aNm7cSKFChZg4cSIAZ8+epWnTprzwwgts27aNdevWsWzZMiZNmgTAxo0b6dGjB9OnT2fLli1s3LiRUqVKpWpsFy9evOp9vPLKK/Ts2TOgx0yOzZs3ExkZyaZNm5g/fz7dunUjJiYmwXVXrVp1WSIHGDFiBK1bt2bNmjVERkbSrVs3AKpUqUJUVBR79uzx+2cAa6OIc+LECQYPHsyUKVOo1+8dDmS/nor/aQ1ARbBhRc1VGTZ3E5v3J9wVfUpVvCEfLzWp5PP6t99+O+vXrwdgxowZ1KlThwYNGgCQO3duJkyYQL169ejevTuvvfYagwYNIjw8HIBs2bLFfUl5On36ND179mTVqlWICC+99BItWrQgb968nD59GoDPPvuMr7/+mmnTptGhQwcKFSrEmjVrqF69Ol9++SVr166Nq9ItU6YMS5cuJUuWLHTp0iXui3DcuHHUqVPnsmOfOnWK9evXU61aNQBWrlzJM888w9mzZ8mVKxfvv/8+5cuXZ9q0acybN4/o6GjOnDnDDz/8wOuvv86nn37KuXPneOihhxg2bBgAzZo1Y+/evURHR9O7d286d+7s8/lNyOzZs2nbti0hISGULFmSMmXKsHLlSm6//fJeHWJiYujXrx8zZszgyy+/jJsvIpw86Vw3J06c4IYbbohb1qRJEyIjI3n++eevKkZfZPpEoarMmjWLZ555hoMHD9KjRw/2Fr2RirnyJtltuDHpRUxMDN9//z0dO3YEnGqnmjVrXrZO6dKlOX36NCdPnmTjxo306dMnyf0OHz6c/Pnzs2HDBgCOHTuW5Dbbtm1j0aJFZM2alUuXLvHll1/yxBNP8OuvvxIWFsa1117LI488wrPPPssdd9zBnj17aNiwIVu2bLlsP6tWrbrsrsPw8HCWLFlCtmzZWLRoEQMHDuTzzz8HYPny5axfv55ChQqxcOFCtm/fzsqVK1FVmjZtypIlS7jrrrt47733KFSoEGfPnuWWW26hRYsWFC5c+LLjPvvssyxevPiKz9W2bVteeOGFy+bt27eP2267LW66WLFi7Nt35SBnEyZMoGnTplx//fWXzR86dCgNGjTgrbfe4syZMyxatChuWUREBKNGjbJE4W+qSvPmzfnqq6+oUaMGc+bMISIigjZTlgc7NJPBJOeXf2o6e/Ys1atXZ/fu3dSsWZP69esD/47XnpDk3DWzaNEiIiMj46YLFiyY5DatWrUia9asALRp04aXX36ZJ554gsjISNq0aRO3382bN8dtc/LkSU6dOkVoaGjcvAMHDlC0aNG46RMnTvD444+zfft2RIQLFy7ELatfvz6FChUCYOHChSxcuJCbb74ZcEpF27dv56677mL8+PFxv+j37t3L9u3br0gUY8eO9e3kwGVtPrHin9/9+/cza9YsfvzxxyvW/eSTT+jQoQN9+vRh+fLltG/fno0bN5IlSxauueYa9u/f73MsVyPTJArPbjguxVwkS1bno+/KWoyb2zxLqbrNeX31BVi9/Ipuw41Jr2LbKE6cOEHjxo2ZOHEivXr1olKlSixZsuSydXfu3EnevHkJDQ2lUqVKrF69Oq5aJzGJJRzPefHv6c+TJ0/c+9tvv50dO3Zw+PBhvvrqKwYPHgzApUuXWL58Obly5fL62Tz3/eKLL3L33Xfz5Zdfsnv3burVq5fgMVWVAQMG8PTTT1+2vx9//JFFixaxfPlycufOTb169RJ8HiE5JYpixYqxd+/euOmoqKjLqo/AaZDfsWMHZcqUAeCff/6hTJky7Nixg3fffZf58+fHnavo6GiOHDnCNddcQ3R0tNfzk5oyTWN2bDcch/74nQXD27NvrfNHUr7+I5S9uxVZsmSNW9ez23BjMoL8+fMzfvx4xowZw4ULF3j00Uf55Zdf4qoyzp49S69eveKqMfr168crr7zCtm3bAOeL+80337xivw0aNGDChAlx07FVT9deey1btmyJq1pKjIjw0EMP8dxzz1GhQoW4X+/x97t27dortq1QoQI7duyImz5x4gQ33uj83U6bNi3RYzZs2JD33nsvrg1l3759HDp0iBMnTlCwYEFy587N1q1bWbFiRYLbjx07lrVr117xip8kAJo2bUpkZCTnzp1j165dbN++nVq1al22zgMPPMDBgwfZvXs3u3fvJnfu3HGfq0SJEnz//fcAbNmyhejo6LhS1LZt2wL3wK+qpqtXzZo1NSWavj5Pb7rtPgW0ZMmS+v3336doP8b4avPmzcEOQfPkyXPZdOPGjfXDDz9UVdX169dr3bp1tVy5clq6dGkdOnSoXrp0KW7duXPnao0aNTQ8PFwrVKigffv2vWL/p06d0scee0wrVaqkVatW1c8//1xVVWfNmqWlSpXSunXravfu3fXxxx9XVdXHH39cZ82addk+fvvtNwV02rRpcfMOHz6srVu31ipVqmiFChX06aefTvDzVa5cWU+ePKmqqsuWLdOyZctq7dq1dfDgwXrTTTepqur777+v3bt3v2y7cePGaeXKlbVy5cp622236Y4dOzQ6OlobNWqkVapU0ZYtW2rdunV18eLFSZzhpI0YMUJLlSql5cqV02+++SZu/n333af79u27Yn3P/7NNmzZp7dq1tWrVqlqtWjVdsGBB3LLu3bvrnDlzEjxmQtcesEpT+L0rmkAdWloWERGhq1atStY2n3zyCR06Pc3Fc2d5of/zDBo0iNy5c/spQmMcW7ZsoUKFCsEOI0MbO3YsoaGhaf5ZitR27tw56tatyy+//JLgmDcJXXsislpVI1JyvExR9XTx4kXy3VCK+oM+YOTIkZYkjMkgunbtSkhISLDDCLg9e/YwatSogA2MliEbs8+cOcPw4cMpUaIE3bp1o127dsw9U9r63TEmg8mZMyft27cPdhgBV7ZsWcqWLRuw42W4EsXXX39NpUqVGD16dFxDnIhYkjBBkd6qdk36549rLsMkiqioKJo3b06TJk3IkycPS5YsYdy4ccEOy2RiOXPm5OjRo5YsTMCoOx5Fzpw5U3W/GabqaefOnXzz7XyqNOtCuXsfZsLmbEzY/O+Dc/ZshAm0YsWKERUVxeHDh4MdislEYke4S03pOlGsXLmS5cuX07t3b+666y4ajfiCP09Jgh342bMRJtCyZ8+eqqOMGRMsfk0UItII+C+QFZiqqqPiLQ8BPgRqAkeBNqq6O6n9Hj9+nIEDBzJ58mSKFy9Op06dyJMnDyF581MxL9ZHkzHGpCK/tVGISFZgInAfTgesD4tIxXirdQSOqWoZYCwwOqn9/v3334SHhzNlyhR69erFhg0bLns83xhjTOryZ4miFrBDVXcCiEgk8CCw2WOdB4Gh7vvPgAkiIuql9W/Xrt0UvKk897wwigMlyvPUJ5villk7hDHGpD6/PZktIi2BRqrayZ1uD9yqqj081tnorhPlTv/prnMk3r46A7Edw1cGNvol6PSnCHAkybUyBzsX/7Jz8S87F/8qr6qhSa92JX+WKBJ6cCF+VvJlHVT1HeAdABFZldLH0DMaOxf/snPxLzsX/7Jz8S8RSV7fRx78+RxFFFDcY7oYEL/z9Lh1RCQbkB/4248xGWOMSSZ/JorfgLIiUlJEcgBtgTnx1pkDPO6+bwn84K19whhjTOD5repJVS+KSA9gAc7tse+p6iYReRmnu9s5wLvARyKyA6ck0daHXb/jr5jTITsX/7Jz8S87F/+yc/GvFJ+LdNfNuDHGmMDKMH09GWOM8Q9LFMYYY7xKs4lCRBqJyB8iskNErhiMVkRCRGSmu/xXEQkLfJSB4cO5eE5ENovIehH5XkRuCkacgZDUufBYr6WIqIhk2FsjfTkXItLavTY2iciMQMcYKD78jZQQkcUissb9O7k/GHH6m4i8JyKH3GfUElouIjLePU/rRaSGTztO6Riq/nzhNH7/CZQCcgDrgIrx1ukGTHbftwVmBjvuIJ6Lu4Hc7vuumflcuOuFAkuAFUBEsOMO4nVRFlgDFHSnrwl23EE8F+8AXd33FYHdwY7bT+fiLqAGsDGR5fcD3+I8w3Yb8Ksv+02rJYq47j9U9TwQ2/2HpweBD9z3nwH3SMYcnSjJc6Gqi1X1H3dyBc4zKxmRL9cFwHDgNSA6kMEFmC/n4ilgoqoeA1DVQwGOMVB8ORcKxPbvk58rn+nKEFR1Cd6fRXsQ+FAdK4ACInJ9UvtNq4niRmCvx3SUOy/BdVT1InACKByQ6ALLl3PhqSPOL4aMKMlzISI3A8VV9etABhYEvlwX5YByIrJURFa4vTlnRL6ci6FAOxGJAr4BegYmtDQnud8nQNodjyLVuv/IAHz+nCLSDogA6vo1ouDxei5EJAtOL8QdAhVQEPlyXWTDqX6qh1PK/FlEKqvqcT/HFmi+nIuHgWmq+oaI3I7z/FZlVb3k//DSlBR9b6bVEoV1//EvX84FInIvMAhoqqrnAhRboCV1LkJxOo38UUR249TBzsmgDdq+/o3MVtULqroL+AMncWQ0vpyLjsCnAKq6HMiJ02FgZuPT90l8aTVRWPcf/0ryXLjVLVNwkkRGrYeGJM6Fqp5Q1SKqGqaqYTjtNU1VNcWdoaVhvvyNfIVzowMiUgSnKmpnQKMMDF/OxR7gHgARqYCTKDLjGLVzgMfcu59uA06o6oGkNkqTVU/qv+4/0h0fz8XrQF5gltuev0dVmwYtaD/x8VxkCj6eiwVAAxHZDMQA/VT1aPCi9g8fz0Uf4P9E5FmcqpYOGfGHpYh8glPVWMRtj3kJyA6gqpNx2mfuB3YA/wBP+LTfDHiujDHGpKK0WvVkjDEmjbBEYYwxxitLFMYYY7yyRGGMMcYrSxTGGGO8skRh0hwRiRGRtR6vMC/rhiXWU2Yyj/mj2/voOrfLi/Ip2EcXEXnMfd9BRG7wWDZVRCqmcpy/iUh1H7Z5RkRyX+2xTeZlicKkRWdVtbrHa3eAjvuoqlbD6Wzy9eRurKqTVfVDd7IDcIPHsk6qujlVovw3zkn4FuczgCUKk2KWKEy64JYcfhaR391X7QTWqSQiK91SyHoRKevOb+cxf4qIZE3icEuAMu6297hjGGxw+/oPceePkn/HABnjzhsqIn1FpCVOn1sfu8fM5ZYEIkSkq4i85hFzBxF5K4VxLsejQzcReVtEVokz9sQwd14vnIS1WEQWu/MaiMhy9zzOEpG8SRzHZHKWKExalMuj2ulLd94hoL6q1gDaAOMT2K4L8F9VrY7zRR3ldtfQBqjjzo8BHk3i+E2ADSKSE5gGtFHVKjg9GXQVkULAQ0AlVa0KjPDcWFU/A1bh/PKvrqpnPRZ/BjT3mG4DzExhnI1wuumINUhVI4CqQF0Rqaqq43H68rlbVe92u/IYDNzrnstVwHNJHMdkcmmyCw+T6Z11vyw9ZQcmuHXyMTj9FsW3HBgkIsWAL1R1u4jcA9QEfnO7N8mFk3QS8rGInAV243RDXR7Yparb3OUfAN2BCThjXUwVkXmAz12aq+phEdnp9rOz3T3GUne/yYkzD053FZ4jlLUWkc44f9fX4wzQsz7etre585e6x8mBc96MSZQlCpNePAv8D6iGUxK+YlAiVZ0hIr8CDwALRKQTTrfKH6jqAB+O8ahnB4IikuD4Jm7fQrVwOplrC/QA/pOMzzITaA1sBb5UVRXnW9vnOHFGcRsFTASai0hJoC9wi6oeE5FpOB3fxSfAd6r6cDLiNZmcVT2Z9CI/cMAdP6A9zq/py4hIKWCnW90yB6cK5nugpYhc465TSHwfU3wrECYiZdzp9sBPbp1+flX9BqehOKE7j07hdHuekC+AZjhjJMx05yUrTlW9gFOFdJtbbZUPOAOcEJFrgfsSiWUFUCf2M4lIbhFJqHRmTBxLFCa9mAQ8LiIrcKqdziSwThtgo4isBcJxhnzcjPOFulBE1gPf4VTLJElVo3F615wlIhuAS8BknC/dr939/YRT2olvGjA5tjE73n6PAZuBm1R1pTsv2XG6bR9vAH1VdR3O+NibgPdwqrNivQN8KyKLVfUwzh1Zn7jHWYFzroxJlPUea4wxxisrURhjjPHKEoUxxhivLFEYY4zxyhKFMcYYryxRGGOM8coShTHGGK8sURhjjPHq/wHKENJZAGK05wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pred = model.predict(x_test)\n",
    "plot_roc(pred,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multiclass Classification Error Metrics\n",
    "\n",
    "The following sections will examine several metrics for evaluating classification error. The following classification neural network will be used to evaluate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from scipy.stats import zscore\n",
    "\n",
    "# Read the data set\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "# Generate dummies for job\n",
    "df = pd.concat([df,pd.get_dummies(df['job'],prefix=\"job\")],axis=1)\n",
    "df.drop('job', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for area\n",
    "df = pd.concat([df,pd.get_dummies(df['area'],prefix=\"area\")],axis=1)\n",
    "df.drop('area', axis=1, inplace=True)\n",
    "\n",
    "# Missing values for income\n",
    "med = df['income'].median()\n",
    "df['income'] = df['income'].fillna(med)\n",
    "\n",
    "# Standardize ranges\n",
    "df['income'] = zscore(df['income'])\n",
    "df['aspect'] = zscore(df['aspect'])\n",
    "df['save_rate'] = zscore(df['save_rate'])\n",
    "df['age'] = zscore(df['age'])\n",
    "df['subscriptions'] = zscore(df['subscriptions'])\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x_columns = df.columns.drop('product').drop('id')\n",
    "x = df[x_columns].values\n",
    "dummies = pd.get_dummies(df['product']) # Classification\n",
    "products = dummies.columns\n",
    "y = dummies.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 1500 samples, validate on 500 samples\n",
      "Epoch 1/1000\n",
      "1500/1500 - 1s - loss: 1.4479 - accuracy: 0.4067 - val_loss: 1.1188 - val_accuracy: 0.5040\n",
      "Epoch 2/1000\n",
      "1500/1500 - 0s - loss: 1.1299 - accuracy: 0.4627 - val_loss: 1.0855 - val_accuracy: 0.4720\n",
      "Epoch 3/1000\n",
      "1500/1500 - 0s - loss: 1.0783 - accuracy: 0.5047 - val_loss: 1.0324 - val_accuracy: 0.5200\n",
      "Epoch 4/1000\n",
      "1500/1500 - 0s - loss: 0.9767 - accuracy: 0.5733 - val_loss: 0.9607 - val_accuracy: 0.5440\n",
      "Epoch 5/1000\n",
      "1500/1500 - 0s - loss: 0.8638 - accuracy: 0.6340 - val_loss: 0.8446 - val_accuracy: 0.6780\n",
      "Epoch 6/1000\n",
      "1500/1500 - 0s - loss: 0.7972 - accuracy: 0.6820 - val_loss: 0.8344 - val_accuracy: 0.6500\n",
      "Epoch 7/1000\n",
      "1500/1500 - 0s - loss: 0.7667 - accuracy: 0.6753 - val_loss: 0.7984 - val_accuracy: 0.7020\n",
      "Epoch 8/1000\n",
      "1500/1500 - 0s - loss: 0.7495 - accuracy: 0.6780 - val_loss: 0.8347 - val_accuracy: 0.6400\n",
      "Epoch 9/1000\n",
      "1500/1500 - 0s - loss: 0.7376 - accuracy: 0.6860 - val_loss: 0.7714 - val_accuracy: 0.6680\n",
      "Epoch 10/1000\n",
      "1500/1500 - 0s - loss: 0.7160 - accuracy: 0.6980 - val_loss: 0.7557 - val_accuracy: 0.6960\n",
      "Epoch 11/1000\n",
      "1500/1500 - 0s - loss: 0.7165 - accuracy: 0.6973 - val_loss: 0.7754 - val_accuracy: 0.6680\n",
      "Epoch 12/1000\n",
      "1500/1500 - 0s - loss: 0.7025 - accuracy: 0.7040 - val_loss: 0.7528 - val_accuracy: 0.6780\n",
      "Epoch 13/1000\n",
      "1500/1500 - 0s - loss: 0.6995 - accuracy: 0.7000 - val_loss: 0.7592 - val_accuracy: 0.6700\n",
      "Epoch 14/1000\n",
      "1500/1500 - 0s - loss: 0.6858 - accuracy: 0.7053 - val_loss: 0.7415 - val_accuracy: 0.6980\n",
      "Epoch 15/1000\n",
      "1500/1500 - 0s - loss: 0.6777 - accuracy: 0.7060 - val_loss: 0.7365 - val_accuracy: 0.6980\n",
      "Epoch 16/1000\n",
      "1500/1500 - 0s - loss: 0.6750 - accuracy: 0.7007 - val_loss: 0.7390 - val_accuracy: 0.6860\n",
      "Epoch 17/1000\n",
      "1500/1500 - 0s - loss: 0.6732 - accuracy: 0.7060 - val_loss: 0.7349 - val_accuracy: 0.7000\n",
      "Epoch 18/1000\n",
      "1500/1500 - 0s - loss: 0.6758 - accuracy: 0.7067 - val_loss: 0.7332 - val_accuracy: 0.7020\n",
      "Epoch 19/1000\n",
      "1500/1500 - 0s - loss: 0.6641 - accuracy: 0.7127 - val_loss: 0.7764 - val_accuracy: 0.6980\n",
      "Epoch 20/1000\n",
      "1500/1500 - 0s - loss: 0.6688 - accuracy: 0.7073 - val_loss: 0.7311 - val_accuracy: 0.6980\n",
      "Epoch 21/1000\n",
      "1500/1500 - 0s - loss: 0.6602 - accuracy: 0.7133 - val_loss: 0.7372 - val_accuracy: 0.7060\n",
      "Epoch 22/1000\n",
      "1500/1500 - 0s - loss: 0.6628 - accuracy: 0.7220 - val_loss: 0.7402 - val_accuracy: 0.6740\n",
      "Epoch 23/1000\n",
      "1500/1500 - 0s - loss: 0.6635 - accuracy: 0.7167 - val_loss: 0.7313 - val_accuracy: 0.7060\n",
      "Epoch 24/1000\n",
      "1500/1500 - 0s - loss: 0.6515 - accuracy: 0.7100 - val_loss: 0.7301 - val_accuracy: 0.7020\n",
      "Epoch 25/1000\n",
      "Restoring model weights from the end of the best epoch.\n",
      "1500/1500 - 0s - loss: 0.6582 - accuracy: 0.7213 - val_loss: 0.7345 - val_accuracy: 0.6980\n",
      "Epoch 00025: early stopping\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x245f66529c8>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Classification neural network\n",
    "import numpy as np\n",
    "import tensorflow.keras\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Split into train/test\n",
    "x_train, x_test, y_train, y_test = train_test_split(    \n",
    "    x, y, test_size=0.25, random_state=42)\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(100, input_dim=x.shape[1], activation='relu',kernel_initializer='random_normal'))\n",
    "model.add(Dense(50,activation='relu',kernel_initializer='random_normal'))\n",
    "model.add(Dense(25,activation='relu',kernel_initializer='random_normal'))\n",
    "model.add(Dense(y.shape[1],activation='softmax',kernel_initializer='random_normal'))\n",
    "model.compile(loss='categorical_crossentropy', \n",
    "              optimizer=tensorflow.keras.optimizers.Adam(),\n",
    "              metrics =['accuracy'])\n",
    "monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=5, \n",
    "                        verbose=1, mode='auto', restore_best_weights=True)\n",
    "model.fit(x_train,y_train,validation_data=(x_test,y_test),callbacks=[monitor],verbose=2,epochs=1000)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculate Classification Accuracy\n",
    " \n",
    "Accuracy is the number of rows where the neural network correctly predicted the target class.  Accuracy is only used for classification, not regression.\n",
    "\n",
    "$ accuracy = \\frac{c}{N} $\n",
    "\n",
    "Where $c$ is the number correct and $N$ is the size of the evaluated set (training or validation). Higher accuracy numbers are desired.\n",
    "\n",
    "As we just saw, by default, Keras will return the percent probability for each class. We can change these prediction probabilities into the actual iris predicted with **argmax**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred = model.predict(x_test)\n",
    "pred = np.argmax(pred,axis=1) # raw probabilities to chosen class (highest probability)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the actual iris flower predicted, we can calculate the percent accuracy (how many were correctly classified)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy score: 0.698\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "\n",
    "y_compare = np.argmax(y_test,axis=1) \n",
    "score = metrics.accuracy_score(y_compare, pred)\n",
    "print(\"Accuracy score: {}\".format(score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculate Classification Log Loss\n",
    "\n",
    "Accuracy is like a final exam with no partial credit.  However, neural networks can predict a probability of each of the target classes.  Neural networks will give high probabilities to predictions that are more likely.  Log loss is an error metric that penalizes confidence in wrong answers. Lower log loss values are desired.\n",
    "\n",
    "The following code shows the output of predict_proba:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Numpy array of predictions\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[0.    , 0.1065, 0.6795, 0.2134, 0.0007, 0.    , 0.    ],\n",
       "       [0.    , 0.755 , 0.2437, 0.    , 0.0013, 0.    , 0.    ],\n",
       "       [0.    , 0.7047, 0.2916, 0.0004, 0.0033, 0.    , 0.    ],\n",
       "       [0.    , 0.3164, 0.6668, 0.0154, 0.0014, 0.    , 0.    ],\n",
       "       [0.    , 0.0462, 0.5769, 0.3767, 0.0002, 0.    , 0.    ]],\n",
       "      dtype=float32)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "As percent probability\n",
      "[ 0.     10.6481 67.9451 21.3382  0.0683  0.0003  0.    ]\n",
      "Log loss score: 0.7310668326690793\n"
     ]
    }
   ],
   "source": [
    "from IPython.display import display\n",
    "\n",
    "# Don't display numpy in scientific notation\n",
    "np.set_printoptions(precision=4)\n",
    "np.set_printoptions(suppress=True)\n",
    "\n",
    "# Generate predictions\n",
    "pred = model.predict(x_test)\n",
    "\n",
    "print(\"Numpy array of predictions\")\n",
    "display(pred[0:5])\n",
    "\n",
    "print(\"As percent probability\")\n",
    "print(pred[0]*100)\n",
    "\n",
    "score = metrics.log_loss(y_test, pred)\n",
    "print(\"Log loss score: {}\".format(score))\n",
    "\n",
    "pred = np.argmax(pred,axis=1) # raw probabilities to chosen class (highest probability)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Log loss](https://www.kaggle.com/wiki/LogarithmicLoss) is calculated as follows:\n",
    "\n",
    "$ \\mbox{log loss} = -\\frac{1}{N}\\sum_{i=1}^N {( {y}_i\\log(\\hat{y}_i) + (1 - {y}_i)\\log(1 - \\hat{y}_i))} $\n",
    "\n",
    "The log function is useful to penalizing wrong answers.  The following code demonstrates the utility of the log function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\jheaton\\AppData\\Local\\Continuum\\miniconda3\\envs\\tensorflow\\lib\\site-packages\\ipykernel_launcher.py:12: RuntimeWarning: divide by zero encountered in log\n",
      "  if sys.path[0] == '':\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtoAAAE0CAYAAAASSJRcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxc9Xnv8e+jfV8syZJ3G2NjjA3EyIQlCTIQAm0CzXppGkq4SWhIk5S2SZuWNGmSJm3olvbV3jbcJr1pm8ShIQs3IQRIYkgLBmw22xjjDRtbsiXL1r6Pnv4xR2awZXRGo9EZaT7v10svzZwzyzOPj0Zf//Sb3zF3FwAAAICplRN1AQAAAMBsRNAGAAAA0oCgDQAAAKQBQRsAAABIA4I2AAAAkAYEbQAAACANCNoAkOHM7CUzu3oKHucaM/tByNs+YWbnpfqcAJDNCNoAkD2+JOkvQt72ryR9Po21AMCsR9AGgCxgZuslVbr75pB3uVfSBjObl8ayAGBWI2gDwAxhZoVm9hUzaw6+vmJmhQn7/8DMWoJ9HzQzN7Ozg93XSXo44baXmdkxM1sUXL/AzDrMbJUkufuApK2Srpm+VwgAswtBGwBmjjskXSLpQkkXSLpY0qclycyulfR7kq6WdLakK06571pJu8auuPujkr4q6RtmVizp3yV92t1fSLjPzuB5AACTQNAGgJnjNyR93t1b3b1N0uck3RTse4+kf3X3He7eF+xLVCWp+5RtfyqpUtITkpol/eMp+7uD+wEAJoGgDQAzx3xJBxKuHwi2je17OWFf4mVJOiGpPHGDuw9L+n+S1kj6a3f3U+5TLqkjtZIBIHsRtAFg5miWtCTh+uJgmyS1SFqYsG/RKfd9TtLKxA1mtkDSZyX9q6S/TpzvHThX0rMp1gwAWYugDQAzx7clfdrM6sysVtJnJP1HsO9uSbeY2blmVhLsS3SfEuZtm5kpPpr9NUkfUDyofyFhf6GkiyQ9mJ6XAgCzH0EbAGaOP5O0RfHR6W2Sngq2yd1/IunvJf1C0h5JjwX3GQz2PyWp08xeH2z/uKR6SX8STBm5RfGg/sZg//WSNrn72Ig5ACBJdvqUPADATGdm50raLqnQ3UeCbddI+oi7/1qI+z8u6QPuvj29lQLA7EXQBoBZwszeLunHkkolfUPSaJhQDQBID6aOAMDs8VuS2iTtlRSTdFu05QBAdmNEGwAAAEgDRrQBAACANCBoAwAAAGmQF3UByaitrfWlS5dG9vy9vb0qLS2N7PlnOvo3efQuNfQvNfRv8uhdauhfauhfarZu3XrM3etSeYwZFbSXLl2qLVu2RPb8mzZtUlNTU2TPP9PRv8mjd6mhf6mhf5NH71JD/1JD/1JjZgdSfQymjgAAAABpQNAGAAAA0oCgDQAAAKQBQRsAAABIA4I2AAAAkAYEbQAAACANCNoAAABAGhC0AQAAgDQgaAMAAABpQNAGAAAA0oCgDQAAAKQBQRsAAABIA4I2AAAAkAYEbQAAACANCNoAAABAGhC0AQAAgDQgaAMAAABpQNAGAAAA0oCgDQAAAKQBQRsAAABIA4I2AAAAkAYEbQAAACANCNoAAABAGhC0AQAAgDQgaAMAAABpQNAGAAAA0oCgDQAAAKQBQRsAAABIA4I2AAAAkAYEbQAAACANCNoAAABAGhC0AQAAgDQgaAMAAABpEGnQNrNrzWyXme0xs09FWQsAAAAwlSIL2maWK+kfJV0nabWkXzez1VHVAwAAAEylKEe0L5a0x933ufuQpI2SboiwHgDAKYZGRjU86lGXAQAzUl6Ez71A0ssJ1w9Jen1EtQBAVhgddXX0D6u9Z1DHeobU3juo9p6h+PXe+Pf2niG1B5e7Bkb0vnML9OaoCweAGcjcoxmpMLN3S3qLu38wuH6TpIvd/WOn3O5WSbdKUn19/UUbN26c9lrH9PT0qKysLLLnn+no3+TRu9TM5v65uwZiUveQq2vQ1T0c/941FP/qDr7Ht8VvN967vkkqK5AqCkwVBaby4KuiwLS8dEhr5s3O/qXbbD72pgP9Sw39S82GDRu2untjKo8R5Yj2IUmLEq4vlNR86o3c/S5Jd0lSY2OjNzU1TUtx49m0aZOifP6Zjv5NHr1LzUzrn7urq39EbT0Dau0eVNvYV88ro8/xEechHesZ1ODI6LiPU1aYp5qyQtWUFmhBfaFqywpUU1qomrIC1ZQVqrY0/r2mrEDVJQXKzbFxH2em9S+T0LvU0L/U0L/oRRm0n5S0wsyWSTos6UZJ742wHgBIq4Hh2MnA3NY9+OoQHWw/Flweip0envNzTbVBMK4pLdTZc8vi1xMC88nLpQUqys+N4FUCAMZEFrTdfcTMPirpp5JyJX3d3XdEVQ8ATMboqOt435BauwbV2j3wqiCdGKDbugfVPTBy2v3NpJrSAtWWFaquvFDL60pVV16ouuB6XXmh5pYXqq6sSBXFeTIbf9QZAJB5ohzRlrvfJ+m+KGsAgPG4u7oHR9TaNaAjnYM62jWgo90DOto5oKNdgycvt3YPamScVTnKCvNOBuZzGyr0phWFpwXouvL4yHNeLucOA4DZKNKgDQBRGJvCcaRrQEe7BnQkCMynXu4bip123/KiPDVUFKm+okiXLK85eXlueaHmVsRHnmvLC1RSwNsrAGQ7fhMAmFX6h2Jq7uxXS8eAmjv7daRzQFt3Dupf9z0RD9JdA+roGz7tfgV5OUFoLtR58yt05aq5qq8oVH0QpBsqijS3opAADQAIjd8YAGaMwZGYjnQOqLljQC2d/WrpDL53DKg5uDxeiK4okJbUDWlhdYkal1arvrxI9ZWvBOj6ikJVFucz/xkAMKUI2gAygrurrWdQh07069CJfjV39Kulo1/NnfHpHC2d/TrWM3Ta/apK8jWvsljzK4u0bnGV5lcVa15lUXxbVTxMb/7vX6qp6Q0RvCoAQDYjaAOYFqOjrtbuQR060afDHf1BoO7ToRP9OnyiX4c6+jV0ynrQ5YV5mlcVD83nza/QvMpizasq0vzg+7zKIqZyAAAyFr+hAEyJ2KjrSNdAPDS/KkDHL7d0DJy2NnRNaYEWVhdr1bxyXb26XguqirWwulgLq0s0r6pIFUX5Eb0aAABSR9AGEFrv4IgOHu+Lf7X36cDxXh083q+D7b06dKL/tGXu6soLtbC6WGsXVOq6NfO0oDoepBdVF2t+VTGj0QCAWY3fcgBOGpsnfbA9HqYPBN/HLh/rGXzV7SuK8rSkplTnLajUdWvnaVF1STAiHQ/SnJkQAJDNCNpAlnF3tXUPat+xXu1P+Ho5CNSJa0ebSfMri7VoTrGuWjVXi2tKtHhOiZbUlGjJnFJVljC1AwCAMyFoA7NU18Cw9rfFQ/QrobpH+9t61ZsQpgvycrQkCM+XLa/VkiBML66Jj04X5jEqDQDAZBC0gRlsODaqA+292tOaEKSDUJ24FF6OSQurS7SstlSNS+borLpSLauNf82rLFZuDutHAwAw1QjawAwwFHM939yl3a3d2tvao93B10vHel/1AcS68kItqy3V1efWnwzSZ9WVatGcEkamAQCYZgRtIIP0DI6cDNJ7Wnu0p7Vbu1t7dLC9T/7gLyXFR6eX1pTq7LllumZ1vVbUl2l5XZmW1ZaqnOXwAADIGARtIALDsVHta+vVC0e69MKRbr3Q0qUXj/bocEf/ydvk55rOqi3TmgWVurBqWG9+/RqtmFuupbWMTgMAMBMQtIE0co+fDXFnSzxQ7zrSrZ0tXdrb1qPhWHzKR36uaXldmdYvrdZ76xfr7LllWjG3TIvnlCgvN0eStGnTJjWdPz/KlwIAAJJE0AamyNDIqHa3dmvH4S4939J1crS6o2/45G3mVRbpnIZyNZ0zV+fOK9c5DeU6q7ZMBXk5EVYOAADSgaANTMLAcEy7jnRre3Onth/u1PbDXdp1pPvkKcZLCnK1sr5c161p0KqGCp3TUK5VDeWqKimIuHIAADBdCNrABHoHR7SzpSseqJvj33e39igWrPZRWZyvNQsqdMvlS3XegkqtmV+hpTWlymHJPAAAshpBG0gQG3Xtbu3WMwc79MzL8a8Xj3ZrbAW9mtICrVlQqavOnau1Cyp13vxKLawulhmhGgAAvBpBG1ntaNeAnj4Zqk9o26HOk2dNrCzO1wWLqnTN6nqtXViltQsqVV9RSKgGAAChELSRNUZio3q+pUtPvnRCWw8c19MHO9TSOSApvvLH6nkVeudFC3XhoipduKhKy2pLCdUAAGDSCNqYtXoHR/TMyx168qXj2vLSCT118IT6gtHqhdXFWr90TjxUL67S6nkVKspnbWoAADB1CNqYNU70Dunx/e168qUT2vLScW1v7lJs1GUmndtQoXdftFDrl81R45I5aqgsirpcAAAwyxG0MWN1Dwzrif3H9ejedj26t107W7okSYV5OXrd4ip9pGm5GpfO0esWV6mCU5MDAIBpRtDGjNE/FNPWAyf06N5jenRvu7Yd7lRs1FWYl6PGpdX6xDUrdenyGq1dUMUJYAAAQOQI2shY7q5dR7v18K42Pfxim7a8dEJDsVHl5ZguWBQfsb50eY3WLa5mfjUAAMg4BG1klM7+Yf33nmMnw/WRrviqIKsaynXzZUt02dm1Wr90jsoKOXQBAEBmI60gUu6uF4/26KGdR7VpV6ueOtih2KirvDBPb1hRq6Zz6vSmlXWaV1kcdakAAABJIWhj2sVGXVsPnNADO47owZ1HdaC9T5J03vwKffiKs3TFyrl63eIq5ecyzxoAAMxcBG1Mi/6hmJ5uHdGP//NZ/eyFVh3vHVJBbo4uXV6jD73xLL15db3qK1hyDwAAzB4EbaTNwHBMm3a16UfPNetnO1vVPxxTedERXblqrt68ul5XrKxTOcvuAQCAWYqgjSk1NDKq/9rTph8926IHnj+qnsERzSkt0NvXLdCCWKs+9GsbWHoPAABkBYI2UubuemL/cX3/6cP6yfYj6uwfVkVRnn5lbYPeev58Xba8Rnm5Odq0aRMhGwAAZA2CNibt5eN9+t5Th3XPU4d08HifSgpy9ZbzGvTW8+fpjSvqCNUAACCrEbSRlL6hEf1k2xF9d+shPbavXZJ02fIa3X71Cl27pkElBRxSAAAAEkEbIe1p7dZ/bD6oe7YeUvfgiBbPKdHvvXml3v66BVo0pyTq8gAAADIOQRtnNDQyqgeeP6L/2HxAm/cdV0Fujq5b26D3XrxYFy+bIzOLukQAAICMRdDGadp7BvVvjx3Qt544qLbuQS2sLtYfXrtK725cqNqywqjLAwAAmBEI2jhpb1uPvvZf+3XP1kMaHBnVlavm6qZLluhNK+uUm8PoNQAAQDII2tCTLx3XXY/s00M7jyo/N0fvXLdQH3jDMp09tyzq0gAAAGYsgnYWe2xvu77y0It6fP9xVZfk62NXrtBvXrqE6SEAAABTgKCdZdxdj+1r11ce2q0n9h/X3PJCffZtq3Xj+sUqLsiNujwAAIBZg6CdRbYeOK4v37/rZMD+07et1o0XL1ZRPgEbAABgqkUStM3sLyW9TdKQpL2SbnH3jihqyQb7j/Xqzvtf0E+2H1FdeaE+d/15+l/rFxGwAQAA0iiqEe0HJf2Ru4+Y2Zcl/ZGkP4yollmrvWdQf/+z3frm4wdVkJej3716pT70pmWcvREAAGAaRJK43P2BhKubJb0rijpmq9io65uPH9Bf/nSX+oZiunH9It1+9UrVlfMhRwAAgOmSCUOb/1vSd6IuYrZ4+uAJ/ckPt2v74S5dfnaNPnf9eTp7bnnUZQEAAGQdc/f0PLDZQ5Iaxtl1h7v/MLjNHZIaJb3Dz1CImd0q6VZJqq+vv2jjxo1pqTeMnp4elZVl5trSfcOuu18c0sMvj6iy0PTrqwp0cUNuRp0mPZP7l+noXWroX2ro3+TRu9TQv9TQv9Rs2LBhq7s3pvIYaQvaEz6x2c2SPizpKnfvC3OfxsZG37JlS3oLew2bNm1SU1NTZM9/Jg+/2KZP3fOcjnYN6JbLl+n2q1eovCg/6rJOk6n9mwnoXWroX2ro3+TRu9TQv9TQv9SYWcpBO6pVR65V/MOPV4QN2Thd98Cwvvjjndr45Ms6e26ZvveRy3XhoqqoywIAAICim6P9D5IKJT0YTG3Y7O4fjqiWGWnrgRP6+LefVktnvz58xXLdfvUKlusDAADIIFGtOnJ2FM87G4yOur76yD791QO7NL+qSN+97TKtW1wddVkAAAA4RSasOoKQ2nsG9bt3P6tHXmzTr54/T3/+jrWqyMC52AAAACBozxg7mjv1oW9s0bHeIX3x7Wv03osXZ9SKIgAAAHg1gvYMcN+2Fv3+3c+qqiRf93z4Mq1dWBl1SQAAAJgAQTuDubv+7me79ZWHdmvd4ir9800XaW55UdRlAQAAIASCdoaKjbo+/YPt+vYTB/XOdQv1pXesUWEeq4oAAADMFATtDDQwHNPtG5/R/TuO6Lc3LNcnrjmH+dgAAAAzDEE7w/QPxfSBbzypR/e260/eulofeMOyqEsCAADAJBC0M8jAcDxkb97Xrr95zwV6x7qFUZcEAACASSJoZ4iB4Zg+9G9b9Ni+dv31uwnZAAAAM11O1AVAGomN6qPfekq/3H1MX37n+YRsAACAWYCgHTF312fu3aGHdrbqCzecp/c0Loq6JAAAAEwBgnbE/unhvfrW4wd1W9Ny3XTp0qjLAQAAwBQhaEfovm0tuvP+Xbrhwvn65DXnRF0OAAAAphBBOyJ7Wrv1if98VusWV+nOd52vnBzWyQYAAJhNCNoR6B4Y1q3/vlUlBXn6p/ddxBkfAQAAZiGW95tm7q5PfW+bDrT36ZsffL3qK4qiLgkAAABpwIj2NPv+04f14+da9PvXrNQlZ9VEXQ4AAADShKA9jQ539OuzP9yh9Uur9VtvWh51OQAAAEgjgvY0cXd94u5nNequv3nPhcrlw48AAACzGkF7mtzz1GE9tq9dd/zqai2aUxJ1OQAAAEgzgvY06Ogb0pfu26l1i6t043rO/AgAAJANCNrT4Mv371Jn/7C++Pa1rJcNAACQJQjaabazpUsbnzyomy9dqnPnVURdDgAAAKYJQTvN7rz/BZUX5ul3rloRdSkAAACYRgTtNNq8r12/2NWmj2w4W5Ul+VGXAwAAgGlE0E4Td9ed97+ghooivf+ypVGXAwAAgGk2YdA2s4+aWfV0FDObbN53XE8d7NBvb1iuovzcqMsBAADANAszot0g6Ukzu9vMrjUzls0I4Z8e3qvasgK9u5Hl/AAAALLRhEHb3T8taYWkr0l6v6TdZvYlM+Mc4mewo7lTj7zYplsuX8ZoNgAAQJYKNUfb3V3SkeBrRFK1pO+a2Z1prG3G+pdf7ldZYZ7ed8mSqEsBAABARPImuoGZfVzSzZKOSfoXSZ9092Ezy5G0W9IfpLfEmeVE75B+vK1FN65fpMpiVhoBAADIVhMGbUm1kt7h7gcSN7r7qJm9NT1lzVz3PHVIQyOjeu/rF0ddCgAAACI0YdB298+8xr6dU1vOzObu+tYTB7VucZVWNXAWSAAAgGzGOtpTaMuBE9rX1qv3vp652QAAANmOoD2F7n2mWUX5ObpuTUPUpQAAACBiBO0pMhIb1X3bWnTVufUqLQwz9R0AAACzGUF7ijy2r13tvUN62/nzoy4FAAAAGYCgPUV+9GyLygrz1HROXdSlAAAAIAMQtKfA6KjrZy+0qumcOs4ECQAAAEkE7Smxo7lLx3oGdeWquVGXAgAAgAxB0J4Cv9jVKjPpipVMGwEAAEAcQXsK/GJXqy5YWKWassKoSwEAAECGIGinqLNvWM+83MFoNgAAAF4l0qBtZp8wMzez2ijrSMWWA8flLl26vCbqUgAAAJBBIgvaZrZI0pslHYyqhqnwxP7jKsjN0YWLqqIuBQAAABkkyhHtv5X0B5I8whpS9sRLx3X+wkqW9QMAAMCrRBK0zex6SYfd/dkonn+q9A2NaNuhTl28bE7UpQAAACDDmHt6BpTN7CFJDePsukPSH0u6xt07zewlSY3ufuwMj3OrpFslqb6+/qKNGzempd4wenp6VFZWdvL6zvaYvvzkgH73okJdUJcXWV0zxan9Q3j0LjX0LzX0b/LoXWroX2roX2o2bNiw1d0bU3mMtKVDd796vO1mtlbSMknPmpkkLZT0lJld7O5HxnmcuyTdJUmNjY3e1NSUrpIntGnTJiU+/+5H9knaqfdd90aW9gvh1P4hPHqXGvqXGvo3efQuNfQvNfQvetM+DOvu2ySdPIXiRCPamWzb4U7NrywiZAMAAOA0rKOdgu3NnVqzoDLqMgAAAJCBIg/a7r50Jo5m9wyOaP+xXoI2AAAAxhV50J6pnm/ukru0lqANAACAcRC0J2nX0W5J0qp55RFXAgAAgExE0J6kva09Ki3IVUNFUdSlAAAAIAMRtCdpb1uPls8tU7BEIQAAAPAqBO1J2tvao+V1LAIPAACA8RG0J6F3cETNnQNaXlcadSkAAADIUATtSdh/rFeSGNEGAADAGRG0J2EsaC+tZUQbAAAA4yNoT8KhE/2SpIXVxRFXAgAAgExF0J6Ewx19qizOV3lRftSlAAAAIEMRtCfh8Il+LahiNBsAAABnRtCehMMd/VrAtBEAAAC8BoJ2ktydEW0AAABMiKCdpM7+YfUOxfggJAAAAF4TQTtJzR0DkqT5jGgDAADgNRC0k3SsZ1CSVFdeGHElAAAAyGQE7SS1dceDdm0ZQRsAAABnRtBO0tiIdm1ZQcSVAAAAIJMRtJN0rGdQRfk5KivMi7oUAAAAZDCCdpKO9QyptqxQZhZ1KQAAAMhgBO0kHesZZH42AAAAJkTQTlJbN0EbAAAAEyNoJyk+dYQPQgIAAOC1EbST4O7q6h9WZUl+1KUAAAAgwxG0kzA8Kg3FRlVZTNAGAADAayNoJ6F32CWJoA0AAIAJEbST0DcS/15RRNAGAADAayNoJ6GPEW0AAACERNBOwtjUkQqCNgAAACZA0E7C2NQRRrQBAAAwEYJ2EsamjlQU5UVcCQAAADIdQTsJTB0BAABAWATtJPSPuIrzc5WfS9sAAADw2kiMSRiMSSUFuVGXAQAAgBmAoJ2EwZhUTNAGAABACATtJAzGnBFtAAAAhELQTkJ8RJsVRwAAADAxgnYShmKuknxGtAEAADAxgnYSmKMNAACAsAjaSRiMOUEbAAAAoRC0kzAUE1NHAAAAEApBOwmsOgIAAICwCNpJGBxh1REAAACEE1nQNrOPmdkuM9thZndGVUdYI7FRjbhUzNQRAAAAhBDJ8KyZbZB0g6Tz3X3QzOZGUUcy+odjkqTiAv4IAAAAgIlFlRpvk/QX7j4oSe7eGlEdoQ2NjEqSCvMY0QYAAMDEograKyW90cweN7OHzWx9RHWENhxzSVJ+LiPaAAAAmJi5e3oe2OwhSQ3j7LpD0hcl/VzS70haL+k7ks7ycYoxs1sl3SpJ9fX1F23cuDEt9U6krW9Un3ykXx9cW6A3LMiPpIaZrqenR2VlZVGXMSPRu9TQv9TQv8mjd6mhf6mhf6nZsGHDVndvTOUx0jZH292vPtM+M7tN0veCYP2EmY1KqpXUNs7j3CXpLklqbGz0pqam9BQ8gT2tPdIjD2vteavVdOGCSGqY6TZt2qSo/v1mOnqXGvqXGvo3efQuNfQvNfQvelHNg/iBpCslycxWSiqQdCyiWkIZjsXnaBcwdQQAAAAhRLUo9Nclfd3MtksaknTzeNNGMsnJoJ1H0AYAAMDEIgna7j4k6X1RPPdkja06wochAQAAEAapMaShGEEbAAAA4ZEaQxpb3o+pIwAAAAiD1BjS8AgfhgQAAEB4pMaQTk4dybOIKwEAAMBMQNAOieX9AAAAkAxSY0isOgIAAIBkkBpDGmIdbQAAACSB1BgSH4YEAABAMkiNIY0t75eXy4chAQAAMDGCdkgjo0HQzqFlAAAAmBipMaRRjwdtcjYAAADCIDaGFAtGtHONqSMAAACYGEE7pJNBO4egDQAAgIkRtEMadZdJMka0AQAAEAJBO6TYqIvBbAAAAIRF0A4p5i4GswEAABAWQTukUUa0AQAAkASCdkixUZoFAACA8MiOIY06I9oAAAAIj6AdEh+GBAAAQDII2iHFGNEGAABAEgjaIcU/DEnSBgAAQDgE7ZCYOgIAAIBkELRDigVnhgQAAADCIGiHxDraAAAASAZBO6SYi6ANAACA0AjaITGiDQAAgGQQtEPiw5AAAABIBkE7pPg62iRtAAAAhEPQDompIwAAAEgGQTukmDvNAgAAQGhkx5Bioy5mjgAAACAsgnZIzvJ+AAAASAJBO6RRzgwJAACAJBC0Q3IXU0cAAAAQGkE7JJdHXQIAAABmEIJ2EhjQBgAAQFgE7ZCcAW0AAAAkgaAdEjkbAAAAySBoJ4EPQwIAACAsgnZYDGkDAAAgCQTtkFysow0AAIDwCNoh8WFIAAAAJCOSoG1mF5rZZjN7xsy2mNnFUdSRLOZoAwAAIKyoRrTvlPQ5d79Q0meC6xmNAW0AAAAkI6qg7ZIqgsuVkpojqiM0d+ZoAwAAILy8iJ73dkk/NbO/UjzsXxZRHQAAAEBamKfpU35m9pCkhnF23SHpKkkPu/s9ZvYeSbe6+9VneJxbJd0aXD1H0q501BtSraRjET7/TEf/Jo/epYb+pYb+TR69Sw39Sw39S8057l6eygOkLWi/5pOadUqqcnc3M5PU6e4VE90vama2xd0bo65jpqJ/k0fvUkP/UkP/Jo/epYb+pYb+pWYq+hfVHO1mSVcEl6+UtDuiOgAAAIC0iGqO9ock/Z2Z5Uka0CtTQwAAAIBZIZKg7e7/JemiKJ47RXdFXcAMR/8mj96lhv6lhv5NHr1LDf1LDf1LTcr9i2SONgAAADDbcQp2AAAAIA0I2pLM7Foz22Vme8zsU+PsLzSz7wT7HzezpQn7/ijYvsvM3jKddWeKEP37PTN73syeM7OfmdmShH0xM3sm+Lp3eivPDCH6934za0vo0wcT9t1sZruDr5unt/Lohejd3yb07UUz60jYx7Fn9nUzazWz7WfYb2b290F/nzOzdQn7sv3Ym6h3vxH07Dkze9TMLsAyYTcAAAZQSURBVEjY95KZbQuOvS3TV3XmCNG/JjPrTPgZ/UzCvtf8uc8GIfr3yYTebQ/e7+YE+7L6+DOzRWb2CzPbaWY7zOx3xrnN1L33uXtWf0nKlbRX0lmSCiQ9K2n1Kbf5iKR/Di7fKOk7weXVwe0LJS0LHic36teUgf3bIKkkuHzbWP+C6z1Rv4YZ0L/3S/qHce47R9K+4Ht1cLk66teUSb075fYfk/T1hOtZfewFPXiTpHWStp9h/69I+okkk3SJpMeD7Vl97IXs3WVjPZF03VjvgusvSaqN+jVkeP+aJP1onO1J/dzP1q+J+nfKbd8m6ecJ17P6+JM0T9K64HK5pBfH+b07Ze99jGhLF0va4+773H1I0kZJN5xymxskfSO4/F1JV5mZBds3uvugu++XtCd4vGwyYf/c/Rfu3hdc3Sxp4TTXmMnCHH9n8hZJD7r7cXc/IelBSdemqc5MlGzvfl3St6elshnC3R+RdPw1bnKDpH/zuM2Sqsxsnjj2Juyduz8a9Ebife80IY69M0nlPXPWSLJ/vPclcPcWd38quNwtaaekBafcbMre+wja8ea+nHD9kE5v+MnbuPuIpE5JNSHvO9sl24MPKP6/xDFFZrbFzDab2a+lo8AMF7Z/7wz+fPVdM1uU5H1nq9CvP5iutEzSzxM2Z/uxF8aZepztx16yTn3fc0kPmNlWi5/9GOO71MyeNbOfmNl5wTaOvSSYWYniQfCehM0cfwGLTwV+naTHT9k1Ze99Ua2jnUlsnG2nLsVyptuEue9sF7oHZvY+SY165WRFkrTY3ZvN7CxJPzezbe6+Nw11Zqow/fv/kr7t7oNm9mHF/7pyZcj7zmbJvP4bJX3X3WMJ27L92AuD974UmdkGxYP2GxI2Xx4ce3MlPWhmLwQjlHjFU5KWuHuPmf2KpB9IWiGOvWS9TdJ/u3vi6DfHnyQzK1P8PyC3u3vXqbvHucuk3vsY0Y7/b2RRwvWFip+5ctzbWPwkO5WK/8kmzH1nu1A9MLOrJd0h6Xp3Hxzb7u7Nwfd9kjYp/j/LbDJh/9y9PaFn/1evrEGf7cdfMq//Rp3yp1OOvVDO1ONsP/ZCMbPzJf2LpBvcvX1se8Kx1yrp+8q+KYcTcvcud+8JLt8nKd/MasWxl6zXeu/L2uPPzPIVD9nfdPfvjXOTKXvvI2hLT0paYWbLzKxA8YPy1BUI7pU09snSdyn+oQIPtt9o8VVJlin+v+0npqnuTDFh/8zsdZK+qnjIbk3YXm1mhcHlWkmXS3p+2irPDGH6Ny/h6vWKzyeTpJ9KuiboY7Wka4Jt2SLMz67M7BzFP7TyWMI2jr1w7pX0m8En8C+R1OnuLeLYm5CZLZb0PUk3ufuLCdtLzax87LLivRt35YhsZmYNwWehZGYXK55X2hXy5x6SmVUq/hfkHyZsy/rjLziuviZpp7v/zRluNmXvfVk/dcTdR8zso4o3KlfxVQl2mNnnJW1x93sV/wf5dzPbo/hI9o3BfXeY2d2K/4IekfTbp/xpetYL2b+/lFQm6T+D982D7n69pHMlfdXMRhV/E/0Ld8+qsBOyfx83s+sVP8aOK74Kidz9uJl9QfFfPJL0+VP+PDirheydFP8g0MbgP8djsv7YkyQz+7biqzvUmtkhSZ+VlC9J7v7Pku5T/NP3eyT1Sbol2JfVx54UqnefUfyzPP8neN8bcfdGSfWSvh9sy5P0LXe/f9pfQMRC9O9dkm4zsxFJ/ZJuDH6Gx/25j+AlRCpE/yTp7ZIecPfehLty/MUHVm6StM3Mngm2/bGkxdLUv/dxZkgAAAAgDZg6AgAAAKQBQRsAAABIA4I2AAAAkAYEbQAAACANCNoAAABAGhC0AQAAgDQgaAMAAABpQNAGgFnIzNab2XNmVhScDW6Hma2Jui4AyCacsAYAZikz+zNJRZKKJR1y9z+PuCQAyCoEbQCYpcysQPFTBQ9IuszdYxGXBABZhakjADB7zZFUJqlc8ZFtAMA0YkQbAGYpM7tX0kZJyyTNc/ePRlwSAGSVvKgLAABMPTP7TUkj7v4tM8uV9KiZXenuP4+6NgDIFoxoAwAAAGnAHG0AAAAgDQjaAAAAQBoQtAEAAIA0IGgDAAAAaUDQBgAAANKAoA0AAACkAUEbAAAASAOCNgAAAJAG/wNvAXy94P6E/wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "from matplotlib.pyplot import figure, show\n",
    "from numpy import arange, sin, pi\n",
    "\n",
    "#t = arange(1e-5, 5.0, 0.00001)\n",
    "#t = arange(1.0, 5.0, 0.00001) # computer scientists\n",
    "t = arange(0.0, 1.0, 0.00001)  # data     scientists\n",
    "\n",
    "fig = figure(1,figsize=(12, 10))\n",
    "\n",
    "ax1 = fig.add_subplot(211)\n",
    "ax1.plot(t, np.log(t))\n",
    "ax1.grid(True)\n",
    "ax1.set_ylim((-8, 1.5))\n",
    "ax1.set_xlim((-0.1, 2))\n",
    "ax1.set_xlabel('x')\n",
    "ax1.set_ylabel('y')\n",
    "ax1.set_title('log(x)')\n",
    "\n",
    "show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix, without normalization\n",
      "[[ 21   1   0   0   0   0   0]\n",
      " [  4 190  54   1   0   0   0]\n",
      " [  0  51 129   7   0   0   0]\n",
      " [  0   0   5   9   0   0   0]\n",
      " [  0   8   0   0   0   0   0]\n",
      " [ 10   7   0   0   0   0   0]\n",
      " [  3   0   0   0   0   0   0]]\n",
      "Normalized confusion matrix\n",
      "[[0.95 0.05 0.   0.   0.   0.   0.  ]\n",
      " [0.02 0.76 0.22 0.   0.   0.   0.  ]\n",
      " [0.   0.27 0.69 0.04 0.   0.   0.  ]\n",
      " [0.   0.   0.36 0.64 0.   0.   0.  ]\n",
      " [0.   1.   0.   0.   0.   0.   0.  ]\n",
      " [0.59 0.41 0.   0.   0.   0.   0.  ]\n",
      " [1.   0.   0.   0.   0.   0.   0.  ]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAEmCAYAAAB/OxvoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZxcVZn/8c+3O2wxiIQEZAsBDCgyEkhEBdkEGXBhUVkiIigaQcEVxyioqMPIT9xF0TAguIWAiCIwIvJTWQaUJIRIhkWCyRCIkAZkNyThmT/uKSgqN923u+t23dv5vnndV1fde+ucp5Pw1KlTZ1FEYGZm9dXV6QDMzGxwnMjNzGrOidzMrOacyM3Mas6J3Mys5pzIzcxqzoncBkTSepJ+LelRSRcPopyjJP22nbF1iqQ9JN3Z6ThszSOPIx/eJL0T+DjwcuBxYC5wekRcP8hyjwZOAnaLiBWDDrTiJAUwISLu7nQsZq3cIh/GJH0c+CbwH8AmwDjge8DBbSh+K+CuNSGJFyFpRKdjsDVYRPgYhgewAfAEcFgv96xDlujvT8c3gXXStb2BxcAngAeBJcB70rUvAM8Ay1MdxwGnAT9pKns8EMCI9PxY4B6yTwV/A45qOn990+t2A24GHk0/d2u69gfgS8ANqZzfAmNW87s14v+3pvgPAd4E3AU8DHym6f5dgRuBf6R7zwLWTteuTb/Lk+n3PaKp/E8Bfwd+3DiXXrNtqmOX9HwzoAfYu9P/NnwMv8Mt8uHrdcC6wKW93HMK8FpgIrATWTI7ten6S8neEDYnS9bflbRhRHyerJU/MyJGRcS5vQUi6UXAt4EDI2J9smQ9N+e+0cAV6d6NgK8DV0jaqOm2dwLvATYG1gZO7qXql5L9GWwOfA44B3gXMAnYA/icpG3SvSuBjwFjyP7s9gU+CBARe6Z7dkq/78ym8keTfTqZ2lxxRCwgS/I/lTQS+CFwfkT8oZd4zQbEiXz42gjoid67Po4CvhgRD0bEUrKW9tFN15en68sj4kqy1uj2A4znWWBHSetFxJKImJ9zz5uBv0bEjyNiRUTMAO4A3tp0zw8j4q6IeBq4iOxNaHWWk30fsBy4kCxJfysiHk/1zwdeBRARsyPiplTvQuAHwF4FfqfPR8SyFM8LRMQ5wF+BPwGbkr1xmrWdE/nw9RAwpo++282ARU3PF6Vzz5XR8kbwFDCqv4FExJNk3RHHA0skXSHp5QXiacS0edPzv/cjnociYmV63Ei0DzRdf7rxeknbSbpc0t8lPUb2iWNML2UDLI2If/ZxzznAjsB3ImJZH/eaDYgT+fB1I/BPsn7h1bmfrFugYVw6NxBPAiObnr+0+WJEXBURbyRrmd5BluD6iqcR030DjKk/ziaLa0JEvBj4DKA+XtPrkC9Jo8i+dzgXOC11HZm1nRP5MBURj5L1C39X0iGSRkpaS9KBkr6SbpsBnCpprKQx6f6fDLDKucCeksZJ2gD4dOOCpE0kHZT6ypeRddGszCnjSmA7Se+UNELSEcAOwOUDjKk/1gceA55InxZOaLn+ALDNKq/q3beA2RHxPrK+/+8POkqzHE7kw1hEfJ1sDPmpwFLgXuBE4Jfpln8HZgHzgL8Ac9K5gdR1NTAzlTWbFybfLrLRL/eTjeTYi/RFYksZDwFvSfc+RDbi5C0R0TOQmPrpZLIvUh8n+7Qws+X6acAFkv4h6fC+CpN0MHAAWXcSZH8Pu0g6qm0RmyWeEGRmVnNukZuZ1ZwTuZlZzTmRm5nVXKmJXNIvJc2WNF/S1L5fYWZm/VXql52SRkfEw5LWI1s3Y680MqH5nqmk6c0jR75o0rYTBjpxsFwjuvsaUtwZ1YzKbOgsWrSQnp6etv6v0P3irSJWrDJZN1c8vfSqiDignfX3V9krtn1Y0qHp8ZbABLJhZc+JiOnAdIBXTZwUl19zQ8khDczYF6/T6RBySU7ltmbb/TWT215mrHiadbbvc5QpAP+c+92+ZgCXrrRELmlvYD/gdRHxlKQ/kC1gZGZWcQLV5yvEMlvkGwCPpCT+crJV9szMqk9AjT7tlpnIfwMcL2kecCdwU4l1mZm1l1vkkFZ6O7Cs8s3MyiPo6u50EIV5eyozszzuWjEzqzHhrhUzs3qTW+RmZrXnFrmZWc25RW5mVmeeEGRmVm+eEGRmNgy4RW5mVmeCbk8IMjOrr5qNI69PpGZmQ0kqdvRZjM6T9KCk25rOzZQ0Nx0LJc1N58dLerrp2veLhOoWuZnZKto6auV84CzgR40TEXHEczVJXwMebbp/QURM7E8FTuRmZnnaNGolIq6VND6/Cgk4HHjDYOqoVCIf0S1Gj1q702HkGr3rSZ0OIdevf3Zap0NYrd1ftlGnQ1gt76xkfSreIh8jaVbT8+lp57Mi9gAeiIi/Np3bWtItwGPAqRFxXV+FVCqRm5lVQsH+76QnIga639wUYEbT8yXAuIh4SNIk4JeSXhkRj/VWiBO5mVmekketSBoBvA2Y1DiX9nFYlh7PlrQA2A6YlVtI4lErZmZ52jRqpRf7AXdExOLnq9RYSd3p8TZkG9bf01dBTuRmZqtIo1aKHH2VJM0AbgS2l7RY0nHp0pG8sFsFYE9gnqRbgZ8Dx0fEw33V4a4VM7NWom1bvUXElNWcPzbn3CXAJf2tw4nczGwVXv3QzKz+ajRE1YnczCyPW+RmZjXnFrmZWY3JfeRmZvXnFrmZWb3VaT0eJ3IzsxbZlp1O5GZm9SWhLidyM7Nac4vczKzm6pTISxtfk/aeu63vO83MqkdSoaMK3CI3M2uldNRE2SPeR0i6QNI8ST+XNLLk+szMBk0Ua41XpUVediLfnmz/uleR7T/3wdYbJE2VNEvSrJ6lS0sOx8ysGCfy590bETekxz8BXt96Q0RMj4jJETF5zNixJYdjZlZMnRJ52X3k0cdzM7NKqkqSLqLsFvk4Sa9Lj6cA15dcn5nZ4KkfRwWUnchvB46RNA8YDZxdcn1mZoMmRFdXV6GjCkrrWomIhcAOZZVvZlamOnWteBy5mVme+uTx0rtWzMzqR+0btSLpPEkPNs90l3SapPskzU3Hm5qufVrS3ZLulPSvRcJ1Ijczy9HG4YfnAwfknP9GRExMx5Wpzh2AI4FXptd8T1J3XxU4kZuZ5WhXIo+Ia4GHC1Z7MHBhRCyLiL8BdwO79vUiJ3Izsxb9nKI/pjE7PR1TC1ZzYlq+5DxJG6ZzmwP3Nt2zOJ3rlRO5mVme4uPIexqz09MxvUDpZwPbAhOBJcDXmmpt1edESo9aMTNrpXKHH0bEA89VJZ0DXJ6eLga2bLp1C+D+vspzi9zMLEeZa61I2rTp6aFAY0TLZcCRktaRtDUwAfhzX+W5RW5mlqNde3ZKmgHsTdaXvhj4PLC3pIlk3SYLgQ8ARMR8SRcB/wOsAD4UESv7qsOJ3MwsR7u6ViJiSs7pc3u5/3Tg9P7U4URuZtaiSkvUFuFEbmaWw4nczKzmnMgHSMCI7moOpPnNhV/sdAi5PnHxrZ0OYbV+94k9Ox3Caq27Vp+znm1NV588Xq1EbmZWFW6Rm5nVWckTgtrNidzMrIWAGuVxJ3Izs1WJrjZNCBoKTuRmZjnctWJmVmdy14qZWa0J3LViZlZ3bpGbmdWc+8jNzOrMfeRmZvWWjSOvTyZ3IjczW4WXsTUzq70a5XEncjOzVcjDD83Mas195GZmw0CN8rgTuZlZnjq1yEvfjkfSuyXNk3SrpB+XXZ+ZWTtIxY4qKLVFLumVwCnA7hHRI2l0mfWZmbVFGzeWkHQe8BbgwYjYMZ07E3gr8AywAHhPRPxD0njgduDO9PKbIuL4vuoou0X+BuDnEdEDEBEPt94gaaqkWZJmLe1ZWnI4ZmZ9a2ws0aYW+fnAAS3nrgZ2jIhXAXcBn266tiAiJqajzyQO5SdyAdHbDRExPSImR8TksWPGlhyOmVkR2YSgIkdfIuJa4OGWc7+NiBXp6U3AFoOJtuxEfg1wuKSNANy1YmZ10Y8W+ZhGr0I6pvazqvcC/9X0fGtJt0j6o6Q9ihRQah95RMyXdDrwR0krgVuAY8us08xs0Po3IagnIiYPqBrpFGAF8NN0agkwLiIekjQJ+KWkV0bEY72VU/rww4i4ALig7HrMzNplKCYESTqG7EvQfSMiACJiGbAsPZ4taQGwHTCrt7I8jtzMLEeZiVzSAcCngL0i4qmm82OBhyNipaRtgAnAPX2V50RuZpajXXlc0gxgb7K+9MXA58lGqawDXJ3eMBrDDPcEvihpBbASOD5vtF8rJ3IzsxztapFHxJSc0+eu5t5LgEv6W4cTuZlZqwrN2izCidzMrIW8sYSZWf3VKI87kZuZ5emqUSZ3Ijczy1GjPO5EbmbWSoJub/VmZlZvw+LLTkkv7u2Ffc39NzOrsxrl8V5b5PPJlqBt/nUazwMYV2JcZmYdI7IhiHWx2kQeEVsOZSBmZlVSoy7yYuuRSzpS0mfS4y3S8opmZsNTwU0lqtKP3ueXnZLOAtYiW8zlP4CngO8Dry43tGp5zbbV3BPj9yfv1ekQVmv5ymc7HYLZgFUkRxdSZNTKbhGxi6RbINt3U9LaJcdlZtYxYvhNCFouqYu092bats1NLTMb1mqUxwv1kX+XbFnFsZK+AFwP/L9SozIz67Bh1UceET+SNBvYL506LCJuKzcsM7POGa4zO7uB5WTdK4VGupiZ1Vl90niBpJx2eZ4BbAZsAfxM0qfLDszMrJOGVdcK8C5gUmODUEmnA7OBL5cZmJlZp2SjVjodRXFFEvmilvtGUGBXZzOz2qpQa7uI3hbN+gZZn/hTwHxJV6Xn+5ONXDEzG7ZqlMd7bZE3RqbMB65oOn9TeeGYmVVDu1rkks4D3gI8GBE7pnOjgZnAeGAhcHhEPKKs0m8BbyJrRB8bEXP6qqO3RbPOHewvYGZWR23uIz8fOAv4UdO5acA1EXGGpGnp+aeAA4EJ6XgNcHb62asio1a2lXShpHmS7moc/f5VzMxqpF2jViLiWuDhltMHAxekxxcAhzSd/1FkbgJeImnTvuooMib8fOCHZG9SBwIXARcWeJ2ZWS1J0C0VOoAxkmY1HVMLVLFJRCwBSD83Tuc3B+5tum9xOterIqNWRkbEVZK+GhELgFMlXVfgdWZmtdWPLvKeiJjcrmpzzkVfLyqSyJelDvgFko4H7uP5dw8zs2Gp5OGHD0jaNCKWpK6TB9P5xUDzpj5bAPf3VViRrpWPAaOADwO7A+8H3tuvkAFJp0k6ub+vMzPrBKnYMUCXAcekx8cAv2o6/25lXgs82uiC6U2RRbP+lB4+Dhzd/3jNzOpFqG3rkUuaAexN1pe+GPg8cAZwkaTjgP8FDku3X0k29PBusuGH7ylSR28Tgi6ll76ZiHhbgV/gFODdZJ33S8mm9puZVdvgWtsvEBFTVnNp35x7A/hQf+vorUV+Vn8La5b29TwS2DnVM4ecRJ6+4Z0KsOW4cYOp0sysbYbFFP2IuGaQZe8BXNq02NZlq6lnOjAdYNKkyX1+O2tmNhTqtF530fXIB8qJ2cxqR9SrRV7mm861wKGS1pO0PvDWEusyM2urLhU7qqBwi1zSOhGxrOj9ETFH0kxgLtlSuJ5EZGa1MOy2epO0K3AusAEwTtJOwPsi4qS+XhsRpwOnDzpKM7MhVqM8Xqhr5dtkSzA+BBARtwL7lBmUmVmnlTwhqK2KdK10RcSilo7/lSXFY2bWcdkythXJ0gUUSeT3pu6VkNQNnAR4GVszG9aG2/DDE8i6V8YBDwC/S+fMzIatGjXIC6218iDZDE0zszWC1L61VoZCkVEr55AzsSciiiyebmZWSzXK44W6Vn7X9Hhd4FBeuIOFmdmwU6fhh0W6VmY2P5f0Y+Dq0iIyM+swMcwmBOXYGtiq3YGYmVVGhabfF1Gkj/wRnu8j7yLbDXpamUGZmXWacrfPrKZeE3naq3Mnsn06AZ5NC5+bmQ1b2YSgTkdRXK9j3lPSvjQiVqbDSdzM1gh1Wv2wyOSlP0vapfRIzMwqRFKhowp627NzRESsAF4PvF/SAuBJsk8dERFrVHL/5zPVXF5m3bW7Ox3Caq01ok6TnM2eV7euld76yP8M7AIcMkSxmJlVQ4VWNiyit0QugIhYMESxmJlVxnCZoj9W0sdXdzEivl5CPGZmHdfOrhVJ2wPNEyu3AT4HvAR4P7A0nf9MRFw5kDp6S+TdwCio0WBKM7O2EN1tapFHxJ3ARIC0FPh9wKXAe4BvRMRXB1tHb4l8SUR8cbAVmJnVjSitj3xfYEHOZj2D0tuwArfEzWzNVHAMeep+GSNpVtPR28qwRwIzmp6fKGmepPMkbTjQcHtL5PsOtFAzs7rrSmuS93UAPRExuemYnleepLWBg4CL06mzgW3Jul2WAF8baKyr7VqJiIcHWqiZWZ2V1LVyIDAnIh4AaPyE5/Z9uHygBQ9k9UMzs2GvhOGHU2jqVpG0aUQsSU8PBW4baMFO5GZmOdqZxyWNBN4IfKDp9FckTSRbXXZhy7V+cSI3M2shii1EVVREPAVs1HLu6HaV70RuZtZKVGZBrCKcyM3MctQnjTuRm5mtQtC2mZ1DwYnczCxHjfK4E7mZ2aqqs2lEEU7kZmYt2j1qpWxO5GZmOerUIi/1TUfSuyT9WdJcST9ISziamVWeCh5VUFoil/QK4Ahg94iYCKwEjsq5b2pj1bClPUtbL5uZDT0Nk82X22BfYBJwc/pl1wMebL0prRQ2HWDSpMlRYjxmZoW4j/x5Ai6IiE+XWIeZWSmq0touosw3nWuAd0jaGEDSaElblVifmVnb9GNjiY4rrUUeEf8j6VTgt5K6gOXAh4BFZdVpZtYOWddKRbJ0AaUOP4yImbxw92gzs1qoUc+Kx5Gbma1KyC1yM7N6c4vczKzG3EduZlZ3covczKz2nMjNzGrOX3aamdWYqM5knyKcyM3McnS1sW9F0kLgcbLFA1dExGRJo8nm2YwHFgKHR8QjAym/TuvCmJkNGRX8rx/2iYiJETE5PZ8GXBMRE8iWNJk20FidyM3MWjS6Vkpea+Vg4IL0+ALgkIEW5ERuZraKou1xAYxp7KmQjqk5BQbZulOzm65vEhFLANLPjQcarfvIzcxa9W8ceU9Td8nq7B4R96fVYK+WdMeg4mvhFrmZWY52bvUWEfennw8ClwK7Ag9I2hQg/Vxl452iKtUiXxnBk8tWdDqMXN11GotkZoOS9ZG35/95SS8CuiLi8fR4f+CLwGXAMcAZ6eevBlpHpRK5mVlVtLHptglwadpxaATws4j4jaSbgYskHQf8L3DYQCtwIjczy9OmTB4R9wA75Zx/iGxv40FzIjczy9HOCUFlcyI3M8tRnzTuRG5mlq9GmdyJ3MysRTa0sD6Z3InczKyVN5YwM6u/GuVxJ3Izs1w1yuRO5GZmq+j3ErUd5URuZpbDfeRmZjXWnwWxqsCJ3Mwsh2rUJHciNzPLUaM87kRuZpanRnl8aDaWkPRhSbdL+ulQ1GdmNihFd5WoSLYfqhb5B4EDI+JvQ1Sfmdmg1Gn4YektcknfB7YBLpP0sbLrMzMbLJH1kRc5qqD0FnlEHC/pAGCfiOgpuz4zs3aoSI4upOObL0uaKmmWpFkP9TjPm1lF1KiPvOOJPCKmR8TkiJi80ZgxnQ7HzAxoTNLv+78q8PBDM7McXdXI0YU4kZuZ5XEif6GIGD8U9ZiZtUPddgjqeB+5mVnlFBx6WGT4oaQtJf0+TYqcL+kj6fxpku6TNDcdbxpouO5aMTPL0cb2+ArgExExR9L6wGxJV6dr34iIrw62AidyM7M8bcrkEbEEWJIePy7pdmDz9pSecdeKmdkqig4+FMCYxlyYdExdbanSeGBn4E/p1ImS5kk6T9KGA43WidzMLEc/+sh7GnNh0jE9vzyNAi4BPhoRjwFnA9sCE8la7F8baKxO5GZmLdq9+KGktciS+E8j4hcAEfFARKyMiGeBc4BdBxqvE7mZWZ42ZXJlWw2dC9weEV9vOr9p022HArcNNFR/2WlmlqOrfUsb7g4cDfxF0tx07jPAFEkTgQAWAh8YaAVO5GZmOdqVxiPi+tUUd2WbqnAiNzNbRYXWGi/CidzMLFd9MrkTuZlZi8YOQXXhRG5mlqNGedyJ3Mwsj1vkA9Qlsc6Iag5tH9FdzbjMrBx1Wsa2UonczKwy6pPHncjNzPLUKI87kZuZtZLaOrOzdE7kZmZ56pPHncjNzPLUKI87kZuZ5alRz4oTuZnZquThh2ZmdVa3Kfqe5WJmVnNukZuZ5ahTi9yJ3Mwsh/vIzcxqLJsQ1OkoinMiNzPL40RuZlZv7loxM6u5On3Z6eGHZmY5VPAoVJZ0gKQ7Jd0taVq7Y3UiNzPL06ZMLqkb+C5wILADMEXSDu0MtdRELumzku6QdLWkGZJOLrM+M7N2UcH/CtgVuDsi7omIZ4ALgYPbGWtpfeSSJgNvB3ZO9cwBZufcNxWYmp4+sf663Xe2KYQxQE+bymq3qsZW1bigurFVNS5Yc2Lbqk3lPOeWObOvGrm2xhS8fV1Js5qeT4+I6U3PNwfubXq+GHjNYGNsVuaXna8HfhURTwNI+nXeTekXnp53bTAkzYqIye0utx2qGltV44LqxlbVuMCxDUZEHNDG4vKa7dHG8kvtWqnRd75mZqVZDGzZ9HwL4P52VlBmIr8eeKukdSWNAt5cYl1mZlV1MzBB0taS1gaOBC5rZwWlda1ExM2SLgNuBRYBs4BHy6ovR9u7a9qoqrFVNS6obmxVjQscWyVExApJJwJXAd3AeRExv511KKKtXTUvLFwaFRFPSBoJXAtMjYg5pVVoZrYGKntm5/Q0XnJd4AIncTOz9iu1RW5mQ0vSyIh4qtNx2NDyzE6rLEmbSHVa8aKzJB0EfCd9oWZrECfyISCpK/10UipI0ubAqWTTmSv551aluCRtBHwY+BqwhaTRHQ7JhtCwS+SS1ut0DK0i4tn0cCtJI6qUABokvaLTMbS4n2wm8M7A26r4ZwZsBiCpCquIPgOsAD4LfB14tvfbO0PSSyv6d1lrwyqRpyE+X5H0ZUkbVCCe3SQdmR5/CLgE+CEwrdFKrwJJJwBnStqk07FA1tKN7MubZ4GXA58CDq5SAkj/1r4v6Qzgg5LW6WQ8EfE4cA3wVmBeRPyjSn9eAJL+BfgS8PaqxVZ3VWhJtIWkDwKHAe8kW9dlc0lfioi/djCsDYEvp9bu+BTfeGBv4AxJ05pa6x2R+lWPBw6KiAc6GUtDRISko4CTgOOA9wL7ACMkXRId/oZe0iHA4cBBwKXA/IhY1smYkovI/u2fJemRiPhGpwNqkPQW4ERgFGltlCr8XQ4XlWkVDoakFwO7kM2YejtwS7r0bUkTOhVXRFxBtiDY24EREXEP2YzXHwObkC1p2WmbATMjYpGktTodTJPtgYsiYh7wSeBussR+WAVacxsA3wQOAZYDHweQtF0ng4qIRRFxDVlj5oT0qaHj0ie9acBHIuL1wC/IGjOV+pRVZ8MikUfEY8CHgI2BQ9OCN8cArwaO7uS3+BFxNXAK8CZJR0TEM+lTwiigCv3Si4A9JG0fEcsBJB2dWp2dNAfYXdIrI2J5RHyHbFbcTmR/dp20EDgTOC4i9o+IZyR9GHhfFd4MI+IW4B3AaZI+0Ol4yPrvu4Cx6fm5wFrAR4B/7VRQw8mw6VqJiGWSniL7+P0vZIvU/Ab4z7QGcCdj+5Wko8k+IewA/BnYmixZddoNwO7AMZL+G1ifbPTDlI5GBX8geyOeIun/A+sBDwPfSf3BnTQb+BXwrKS9gXFkDYdjGm+GnRYR81JsT1cglkck/RzYR9I/IuI2SReTfVo9QtLvK9I1VVvDakJQ+sLpo8B+ZF0Xh0fEHZ2N6nmplXsJcDEwLSIWdjaijKRNyRa6P4hsPZwvpy6NjpK0GfC2dKwAPhERf+lsVJn0Z3ZQOh4CzqxKbFUkaQuy72ImkTVgDgHeDZwGnBoRt3YuuvobVokcIH20fSnwbETc1+l4WknaC1gYEYs6HUurRhdUpz/BtJL0IrJ/q090OpZWja6UqrTEqyx9l7UbWffYlcCLyBbPemNVvmivq2GXyM2s+iTtA3wZ+IBb44PnRG5mQy51Ta1dxU+mdeREbmZWc8Ni+KGZ2ZrMidzMrOacyM3Mas6J3FYhaaWkuZJuk3Rx2qpvoGXtLeny9PggSdN6ufclac2c/tZxmqSTi55vued8Se/oR13jJd3W3xjNyuREbnmejoiJEbEj2fTq45svKtPvfzsRcVlEnNHLLS8B+p3IzdZ0TuTWl+uAl6WW6O2Svkc2M29LSftLulHSnNRyHwUg6QBJd0i6nmxWJun8sZLOSo83kXSppFvTsRtwBrBt+jRwZrrvk5JuljRP0heayjpF0p2Sfke2wFavJL0/lXOrpEtaPmXsJ+k6SXelVfqQ1C3pzKa6q7BmiVkuJ3JbLWUbJhwINKaebw/8KCJ2Bp4k28Fnv4jYBZgFfFzSusA5ZOti70E2yzbPt4E/RsROZCtXzidbIW9B+jTwSUn7AxOAXYGJwCRJe0qaRLbS5c5kbxSvLvDr/CIiXp3qu51sedyG8cBewJvJ1hhfN11/NCJencp/v6StC9RjNuSGzaJZ1lbrSZqbHl9HtlrdZsCiiLgpnX8t2TK8N6SVSNcGbiTbCOJvjXXgJf2EbHGkVm8gW2uDiFgJPCppw5Z79k9HY1niUWSJfX3g0sYmw5IuK/A77Sjp38m6b0YBVzVduyitC/9XSfek32F/4FVN/ecbpLrvKlCX2ZByIrc8T0fExOYTKVk/2XwKuDoiprTcNxFo1ywzkS3g9YOWOj46gDrOBw6JiFslHUu2HnZDa1mR6j4pIpoTPpLG97Nes9K5a8UG6iay9cJfBiBpZNpY4Q5ga0nbpvtWtxzuNcAJ6bXdaUGlx8la2w1XAe9t6nvfXNLGwLXAoZLWk7Q+WTdOX9YHlqRFro5quXaYpK4U8zbAnanuExqLYknaLi3eZVY5bpHbgETE0tSynaHn96s8NSLukjQVuEJSD9mOSDvmFPERYLqk44CVwAkRcaOkG9Lwvv9K/eSvAG5MnwieAN4VEXMkzQTmkm2Mccz6viMAAABbSURBVF2BkD8L/Cnd/xde+IZxJ/BHsqWPj4+If0r6T7K+8znKKl9KtvSqWeV4rRUzs5pz14qZWc05kZuZ1ZwTuZlZzTmRm5nVnBO5mVnNOZGbmdWcE7mZWc39H67X9NGdW+YmAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW8AAAEmCAYAAACtaxGwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3debwcVZ338c/33hASTAhCECRhU5YRGNkCKqCgIA+MEFwRFJCHJYIsKovGB2QQZUAZdVxQjIKAjhAW0YyioCiyCJKwSsAwAYkEFBIIIDuE3/NHnQudTt/uurldt6puvm9e9aKrq/qcX/ft/Pr06XNOKSIwM7N66Sk7ADMzGzgnbzOzGnLyNjOrISdvM7MacvI2M6shJ28zsxpy8l7OSDpZ0o/T7XUkPSWpt8t13C9pl26WmaPOwyU9nJ7PaoMo5ylJb+hmbGWRNFvSTmXHYcVw8u6ylLgelvSahvsOkXR1iWG1FBF/i4gxEbG47FgGQ9IKwNeAXdPzeXRZy0qPv6970XWfpHMlfanTeRGxaURcPQQhWQmcvIsxAvjkYAtRxn+jztYARgGzyw6kCiSNKDsGK54TQzHOAI6TtEqrg5K2kzRT0hPp/9s1HLta0qmSrgeeAd6Q7vuSpD+mr/X/I2k1Sf8t6clUxnoNZXxD0gPp2M2S3t5PHOtJCkkjJL0tld23PSfp/nRej6Spku6V9KikiySt2lDO/pLmpWMntHthJI2W9NV0/hOSrpM0Oh2bnL7qP56e85saHne/pOMk3ZEeN13SKEkbAXPSaY9L+l3j82p6XQ9JtzeQ9IdUzkJJ0xvOC0kbpNvjJJ0vaUGK98S+D1NJB6bY/1PSIkl/lbR7m+d9v6TjU/xPSzpb0hqSfiXpn5J+K+m1DedfLOkfKcZrJG2a7p8CfBT4TN97oaH8z0q6A3g6/U1f6b6SdLmkrzaUP13SOe3+VlZxEeGtixtwP7AL8FPgS+m+Q4Cr0+1VgUXA/mQt9H3T/mrp+NXA34BN0/EV0n1zgTcC44C7gHtSPSOA84EfNsSwH7BaOnYs8A9gVDp2MvDjdHs9IIARTc+hr87T0v6ngBuBicCKwPeAC9KxTYCngHekY18DXgJ26ef1OTOVPQHoBbZLj9sIeBp4d6r/M+k5j2x4XW8C1kqv4d3AYa2eR6vnleo8JN2+ADiBrPEyCtih4bwANki3zwd+DoxNZd4DHJyOHQi8CByansfhwEOA2rwvbiT7ljABeAS4BdgyPf/fAf/ecP5Bqd4Vgf8Cbms4di7pvdVU/m3A2sDoxvdiur1mqvNdZMn/PmBs2f9evA0i15QdwHDbeDV5bwY8AazOksl7f+CmpsfcAByYbl8NnNJ0/GrghIb9rwK/atjfs/Efd4uYFgGbp9sn0zl5fxf4JdCT9u8Gdm44/vqUuEYAJwEXNhx7DfACLZJ3SpbP9sXSdOzzwEVN5z4I7NTwuu7XcPwrwFmtnker58WSyft8YBowsUUcAWxAlpCfBzZpOPbxhr/jgcDchmMrpceu2eZ98dGG/UuB7zbsHwX8rJ/HrpLKHpf2z6V18j6o1XuxYf/9wAPAQho+sLzVc3O3SUEi4k7gF8DUpkNrAfOa7ptH1hrr80CLIh9uuP1si/0xfTuSjpV0d/rK/ThZa318nrglfRzYCfhIRLyc7l4XuCx1ZzxOlswXk7Ui12qMNyKeBvr7wXA8WUv33hbHlnhdUt0PsOTr8o+G28/Q8JwH6DOAgJtSN81B/cQ6kiX/Vs1/p1fiiYhn0s12MeX6G0rqlXR66qZ6kiwJ98XUTqv3TaNfkH0ozYmI6zqcaxXn5F2sfyf7Wt34D/4hsmTYaB2yVmafZV7qMfVvfxbYG3htRKxC9g1AOR/7RWCviHii4dADwO4RsUrDNioiHgT+TvZVva+Mlci6bFpZCDxH1v3TbInXRZJSuQ+2OLeTp9P/V2q4b82+GxHxj4g4NCLWImtNf6evn7sp1hdZ8m/V/HcqykeAvci+wY0j+yYBr/4N+3t/dHrfnEr2wft6SfsOMkYrmZN3gSJiLjAdOLrh7suBjSR9JP2o9GGyfuNfdKnasWR9zguAEZJOAlbu9CBJa6dYD4iIe5oOnwWcKmnddO7qkvZKxy4B9pC0g6SRwCn0875KrelzgK9JWiu1MN8maUXgIuA9knZWNvTvWLJuiz8O6Nln9SwgS7L7pToOouEDQ9KHJE1Mu4vIkt7ipjIWp5hOlTQ2PfdjgB8PNJ5lMJbsuT9K9gH0H03HHwYGNBZd0juA/wsckLZvSZrQ/lFWZU7exTuFrB8YgMjGIO9BlpweJfsKv0dELOxSfVcAvyL7cW0eWUu309dpgJ3JWqeX6NURJ31D774BzACulPRPsh/e3pKez2zgCOAnZK3wRcD8NvUcB/wZmAk8BnyZrG99DtkPrd8ia/XuCewZES/kfN7NDgWOJ3uNN2XJD4FtgD9Jeio9r09GxF9blHEUWSv+PuC69ByHYoTG+WR/uwfJfpy+sen42cAmqRvrZ50Kk7RyKvPIiHgwdZmcDfwwfcOxGlKEL8ZgZlY3bnmbmdWQk7eZWYEknSPpEUl39nNckr4paW6axLVVnnKdvM3MinUusFub47sDG6ZtCtk8i44KTd6SfqZsevbsNK3XzGy5EhHXkP0435+9gPMjcyOwiqTXdyq36AVsDoqIx5StXTFT0qXRtOJbSupZYu9ZYWuNXubVPAu1+UbVHFXV47ECtpybN+9+Fi5c2NV/Cb0rrxvx0rO5zo1nF8wmG9XVZ1pETBtAdRNYckTY/HTf39s9qOjkfbSk96Xba5N9LVgieacnOQ2gZ8zrY8UtDi44pGXzuyu/UHYILY0e2dWluM1qZ/u3TOp6mfHSs6y48d65zn3utjOfi4jBBNHqg6fjMMDCkreyReB3Ad4WEc8oW896VFH1mZl1j2DoVmOeT8MsZbIF4B7q9KAioxsHLEqJ+1+AtxZYl5lZ9wiQ8m2DNwM4II06eSvwRES07TKBYrtNfg0cltYXnsPSs8TMzKqrSy1vSReQLfY2XtJ8sjWPVgCIiLPIlsz4N7IlkJ8hW8ago8KSd0Q8TzYExsysZgQ93fk9KSLaLgIW2TT3IwZari+XZGbWSsWXfXHyNjNrJobyB8tl4uRtZraUrv0YWRgnbzOzVtzyNjOrIbe8zczqZkgn6SwTJ28zs2Z9k3QqzMnbzKwVt7zNzOpG0FvtRd+cvM3Mmnmct5lZTbnP28ysbjzaxMysntzyzm+LjSdwzVWnlB1GS5se/8uyQ2jpp8fsWHYI/dpkwtiyQ+iXKv4P0yrALW8zs5rp3oUWCuPkbWbWilveZmY15Ja3mVndeLSJmVn9iK5dBq0oTt5mZktxy9vMrJ7c521mVkNueZuZ1ZBb3mZmNSP3eZuZ1ZNb3mZm9VP19W+cvM3MmmSXsHTyNjOrFwn1OHmbmdWOW95mZjVU9eRd2FgYSetJurOo8s3MiiQp11YWt7zNzJopbRVW9Cj0EZLOk3SHpEskrVRwfWZmgybytbrLbHkXnbw3BqZFxJuBJ4FPNJ8gaYqkWZJmLVywoOBwzMzyWd6T9wMRcX26/WNgh+YTImJaREyKiEnjV1+94HDMzPKpevIuus87OuybmVXScjvaJFlH0tvS7X2B6wquz8xs8DSArSRFJ++7gY9JugNYFfhuwfWZmQ2aED09Pbm2shTWbRIR9wObFFW+mVmRutltImk34BtAL/CDiDi96fg6wHnAKumcqRFxebsyq71grZlZWbrUbSKpFzgT2J2sQbuvpOaG7YnARRGxJbAP8J1O5Tp5m5k1U1dHm2wLzI2I+yLiBeBCYK+mcwJYOd0eBzzUqVDPsDQza2EA3SbjJc1q2J8WEdMa9icADzTszwfe0lTGycCVko4CXgPs0qlSJ28zsxYGkLwXRsSkdkW1uK952PS+wLkR8dU0Qu9HkjaLiJf7K9TJ28ysSd/0+C6ZD6zdsD+RpbtFDgZ2A4iIGySNAsYDj/RXqPu8zcxa6d4475nAhpLWlzSS7AfJGU3n/A3YGUDSm4BRQNv1QtzyNjNrpu4NFYyIlyQdCVxBNgzwnIiYLekUYFZEzACOBb4v6dNkXSoHRkTbGelO3mZmLXRznHcas315030nNdy+C9h+IGU6eZuZteBrWJqZ1VDVF6Zy8jYza1L2cq95OHmbmbXg5G1mVkNO3gMgYERvNYeen3fEdmWH0NIh583qfFJJfv+ZHcsOoV+jVugtOwSrumrn7molbzOzqnDL28ysbro4SacoTt5mZk0EVDx3O3mbmS1N9HiSjplZ/bjbxMysbuRuEzOz2hG428TMrI7c8jYzqyH3eZuZ1Y37vM3M6icb513t7O3kbWa2FC8Ja2ZWSxXP3U7eZmZLkYcKmpnVjvu8zcxqquK528nbzKyVqre8C79sjaQDJN0h6XZJPyq6PjOzbpDybWUptOUtaVPgBGD7iFgoadUi6zMz64oaXIyh6Jb3u4BLImIhQEQ81nyCpCmSZkmatWDhgoLDMTPrrO9iDFVueRedvAVEuxMiYlpETIqISauPX73gcMzM8sgm6eTZylJ08r4K2FvSagDuNjGzuqh6y7vQPu+ImC3pVOAPkhYDtwIHFlmnmdmgeZIORMR5wHlF12Nm1i2epGNmVlNO3mZmNVTx3O3kbWbWilveZmZ14yvpmJnVj3wxBjOzeqp47nbyNjNrpafi2bvwVQXNzOqomzMsJe0maY6kuZKm9nPO3pLukjRb0k86lemWt5lZEwl6uzTDUlIvcCbwbmA+MFPSjIi4q+GcDYHPka3AukjS6zqV65a3mVkLXVyYaltgbkTcFxEvABcCezWdcyhwZkQsAoiIRzoV2m/LW9LK7R4YEU92DNnMrKYG0OU9XtKshv1pETGtYX8C8EDD/nzgLU1lbJTVqeuBXuDkiPh1u0rbdZvMJlvOtfEp9O0HsE67gs3M6kpkwwVzWhgRkzoU16x5qewRwIbATsBE4FpJm0XE4/0V2m/yjoi12wRjZjasdXFRwflAYz6dCDzU4pwbI+JF4K+S5pAl85n9xpenZkn7SPp/6fZESVsPJHIzs1rJ2d+ds897JrChpPUljQT2AWY0nfMz4J1Z1RpP1o1yX7tCO442kfRtYAXgHcB/AM8AZwHb5Il6uHjrG1crO4SWjt1jo7JD6Nfk795Qdgj9uvLoHcoOwSquW8O8I+IlSUcCV5D1Z5+TrnVwCjArImakY7tKugtYDBwfEY+2KzfPUMHtImIrSbemQB5Lnx5mZsOS6O4knYi4HLi86b6TGm4HcEzacsmTvF+U1EPqYE+XNHs5bwVmZnVU8QmWufq8zwQuBVaX9AXgOuDLhUZlZlayql+AuGPLOyLOl3QzsEu660MRcWexYZmZlaebMyyLknd6fC/wIlnXiWdlmtmwV+3UnSMRSzoBuABYi2x84k8kfa7owMzMylT7bhNgP2DriHgGQNKpwM3AaUUGZmZWlmy0SdlRtJcnec9rOm8EHQaPm5nVWsmt6jzaLUz1dbI+7meA2ZKuSPu7ko04MTMbtiqeu9u2vPtGlMwGftlw/43FhWNmVg21bXlHxNlDGYiZWVUMiz5vSW8ETgU2AUb13R8R1V1Uw8xskKre8s4zZvtc4IdkH0a7AxeRXQnCzGxYkqBXyrWVJU/yXikirgCIiHsj4kTS0oVmZsNVNy9AXIQ8QwWfV/b94V5JhwEPAh0vjmlmVmdV7zbJk7w/DYwBjibr+x4HHDTQiiSdDDwVEf850MeamQ21iufuXAtT/Snd/Cewf7HhmJmVT6ir63kXod0knctY+iKZr4iI93cqPK2LcgDZlZMXkE2rNzOrtpL7s/No1/L+9mAKTte53AfYMtVzCy2St6QpwBSAtdfxBenNrBpq2+cdEVcNsuy3A5c1LGjVfMHNvnqmAdMAtt56Ur8tfTOzoVT1ta/zrue9rJyMzax2RPVb3kV+uFwDvE/SaEljgT0LrMvMrKt6lG8rS+6Wt6QVI+L5vOdHxC2SpgO3kS0re+0yxGdmNuSGxWXQJG0LnE02vnsdSZsDh0TEUZ0eGxGnko0NNzOrlYrn7lzdJt8E9gAeBYiI2/H0eDMb5obD9PieiJjX1Hm/uKB4zMxKly0JW+2md57k/UDqOglJvcBRwD3FhmVmVq7hMFTwcLKuk3WAh4HfpvvMzIatije8c61t8gjZTEkzs+WCVOO1TfpI+j4tJttExJRCIjIzq4CK5+5c3Sa/bbg9Cngf2UJTZmbDVtWHCubpNpneuC/pR8BvCovIzKxkYhhM0mlhfWDdbgdiZlYZJU99zyNPn/ciXu3z7gEeA6YWGZSZWdlEtbN32+Sdrl25Odl1KwFejgivFGhmw1o2SafsKNprOw49JerLImJx2py4zWy5UPVVBfNMIrpJ0laFR2JmViGScm1laXcNyxER8RKwA3CopHuBp8m+UURELFcJ/bXbHFl2CC0tmjmoq9UV6oObTyw7BLNlUoduk3Z93jcBWwHvHaJYzMyqocsrBkraDfgG0Av8ICJO7+e8DwIXA9tExKx2ZbZL3gKIiHuXLVwzs/rq1vT4tKDfmcC7gfnATEkzIuKupvPGAkcDf8pTbrvkvbqkY/o7GBFfy1OBmVnddLnbZFtgbkTcByDpQmAv4K6m874IfAU4Lk+h7X6w7AXGAGP72czMhinRq3wbMF7SrIated2nCSy5pMj8dN+rtUlbAmtHxC/yRtiu5f33iDglb0FmZsNFdvX43KcvjIhJHYpr9sqwa0k9wNeBA3PXSI4+bzOz5U53x3DPB9Zu2J8IPNSwPxbYDLg6DT1cE5ghaXK7Hy3bJe+dlz1WM7N66+J63jOBDSWtTzZbfR/gI30HI+IJYHzfvqSrgeM6jTbpt887Ih4bZMBmZrXU123SjQsQp/kyRwJXAHcDF0XEbEmnSJq8rDEuy6qCZmbDXjevpBMRlwOXN913Uj/n7pSnTCdvM7MWhsOVdMzMlitieFw93sxs+SJKXXQqDydvM7MWqp26nbzNzJYi6Js9WVlO3mZmLVQ8dzt5m5ktrdwLLeTh5G1m1sSjTczMaqrqLe9CP1wk7SfpJkm3SfpeWpTczKzylHMrS2HJW9KbgA8D20fEFsBi4KMtzpvStw7ugoULigrHzCw/1fgCxF2wM7A12SV/AEYDjzSfFBHTgGkAW289KZqPm5kNteW9z1vAeRHxuQLrMDMrxPLc530V8EFJrwOQtKqkdQusz8ysa3qUbytLYS3viLhL0onAlekyPy8CRwDziqrTzKwbsm6Tare8Cx0qGBHTgelF1mFmVoSK95p4nLeZ2dKElueWt5lZXbnlbWZWM8t9n7eZWS3lvLhwmZy8zcxacPI2M6sh/2BpZlYzotwJOHk4eZuZtdBT8X4TJ28zsxbcbWJmVjPuNjEzqyXPsDQzqx+P8zYzq6eK5+5qJe97H32aD/9wZtlhtHTaN44pOwQzGyJZn3e103elkreZWVVUO3U7eZuZtVbx7O3kbWbWgrtNzMxqqNqp28nbzKy1imdvJ28zsybC0+PNzOrHk3TMzOqp4rmbnrIDMDOrJOXc8hQl7SZpjqS5kqa2OH6MpLsk3SHpKknrdirTydvMbCnK/V/HkqRe4Exgd2ATYF9JmzSddiswKSLeDFwCfKVTuU7eZmYtSPm2HLYF5kbEfRHxAnAhsFfjCRHx+4h4Ju3eCEzsVKiTt5lZk7w9Jil3j5c0q2Gb0lTcBOCBhv356b7+HAz8qlOM/sHSzKwF5R9usjAiJrUrqsV90U+d+wGTgB07VerkbWbWQheHCs4H1m7Ynwg8tHR92gU4AdgxIp7vVKi7TczMWujiYJOZwIaS1pc0EtgHmLFEXdKWwPeAyRHxSJ5ChyR5Szpa0t2S/nso6jMzG5QBdnq3ExEvAUcCVwB3AxdFxGxJp0ianE47AxgDXCzpNkkz+inuFUPVbfIJYPeI+OsQ1WdmNijdnB4fEZcDlzfdd1LD7V0GWmbhLW9JZwFvAGZI+nTR9ZmZDZbo6lDBQhTe8o6IwyTtBrwzIhYWXZ+ZWTdUfXp86aNN0pjIKQCjV12z5GjMzJKKZ+/SR5tExLSImBQRk0aOXaXscMzMgPwT5MtSesvbzKyKeire8nbyNjNrxckbImK9oajHzKwbfCUdM7M68pV0zMzqqeK528nbzKylimdvJ28zs6WUOwwwDydvM7MW3OdtZlYzA1jutTRO3mZmrVQ8ezt5m5m10FPxfhMnbzOzFqqdup28zcyW5kk6ZmZ1Ve3s7eRtZtak70o6VebkbWbWQsVzt5O3mVkrbnkPwJOPPMqV3zmv7DBamj7z22WHYGZDyNPjzczqqNq528nbzKyViuduJ28zs2aSZ1iamdVTtXO3k7eZWSsVz91O3mZmrVS818TJ28xsab6SjplZ7dRhenxP2QGYmdnAueVtZtZC1VveTt5mZi24z9vMrGaySTplR9Gek7eZWStO3mZm9eNuEzOzGqr6D5YeKmhm1oJybrnKknaTNEfSXElTWxxfUdL0dPxPktbrVKaTt5lZK13K3pJ6gTOB3YFNgH0lbdJ02sHAoojYAPg68OVO5RaavCV9XtJfJP1G0gWSjiuyPjOzblHO/3LYFpgbEfdFxAvAhcBeTefsBfRdRuwSYGepfcdNYX3ekiYBHwC2TPXcAtzc4rwpwJS0+9Rzt505p0shjAcWdqksRq9wZreKgi7H1kVVjQuqG1tV44LlJ7Z1u1TOK2695eYrVhqp8TlPHyVpVsP+tIiY1rA/AXigYX8+8JamMl45JyJekvQEsBptXqMif7DcAfh5RDwLIOl/Wp2UnuS0VscGQ9KsiJjU7XK7oaqxVTUuqG5sVY0LHNtgRMRuXSyuVQs6luGcJRTZbVLx32rNzIbEfGDthv2JwEP9nSNpBDAOeKxdoUUm7+uAPSWNkjQGeE+BdZmZVdVMYENJ60saCewDzGg6ZwbwsXT7g8DvIqJty7uwbpOImClpBnA7MA+YBTxRVH0tdL0rpouqGltV44LqxlbVuMCxVULqwz4SuALoBc6JiNmSTgFmRcQM4GzgR5LmkrW49+lUrjok90GRNCYinpK0EnANMCUibimsQjOz5UTRMyynpfGMo4DznLjNzLqj0Ja3mQ0tSStFxDNlx2HF8wxLqyxJa3SaqGCvkjQZ+Fb6UcyGOSfvISCpJ/3fiSgnSROAE8mmElfydatSXJJWA44GvgpMlLRqySFZwYZd8pY0uuwYmkXEy+nmupJGVOkffR9Jbyo7hiYPkc3I3RJ4fxVfM2AteGVcbtleAF4CPg98DXi5/enlkLRmRf+WtTOskncajvMVSadJGleBeLaTtE+6fQRwKfBDYGpfa7wKJB0OnCFpjbJjgaxFm8a4vgz8C/BZYK8q/aNP77WzJJ0OfELSimXGExH/BK4C9gTuiIjHq/R6AUj6V+CLwAeqFlsdVaHF0BWSPgF8CPgI2ToqEyR9MSL+t8SwXgucllq166X41gN2Ak6XNLWhVV6K1E96GDA5Ih4uM5Y+ERGSPgocRbba2kHAO4ERki7tNHmhaJLeC+wNTAYuA2ZHxPNlxpRcRPbe/7akRRHx9bID6iNpD+BIYAxpLZIq/C3rrDKtv8GQtDKwFdnA9g8At6ZD35S0YVlxRcQvyRbd+gAwIiLuI5t5+iNgDbLlIcu2FjA9IuZJWqHsYBpsDFwUEXcAxwNzyZL5hyrQahsH/BfwXuBF4BgASRuVGVREzIuIq8gaMIenbwelS9/opgKfjIgdgJ+SNWAq9W2qboZF8o6IJ4EjgNcB70uLynwM2AbYv8xf3yPiN8AJwL9J+nBEvJC+DYwBqtDPPA94u6SNI+JFAEn7p9ZlmW4Btpe0aUS8GBHfIpudtjnZa1em+4EzgIMjYteIeEHS0cAhVfgAjIhbyaZYnyzp42XHQ9Yf3wOsnvbPBlYAPgn8n7KCqrth020SEc9Leobsq/W/ki3y8mvgB2kN3TJj+7mk/cm+CWwC3ASsT5agynY9sD3wMUl/BMaSjVrYt9So4GqyD999Jf0OGE02bfhbqX+3TDcDPwdelrQTsA5ZY+FjfR+AZYuIO1Jsz1YglkWSLgHeKenxiLhT0sVk30o/LOn3Fel2qpVhNUkn/Wj0KWAXsm6JvSPiL+VG9arUmr0UuBiYGhH3lxtRRtLryRaDn0y2/sxpqbuiVJLWAt6ftpeAYyPiz+VGlUmv2eS0PQqcUZXYqkjSRLLfVrYma7S8FzgAOBk4MSJuLy+6ehpWyRsgfW1dE3g5Ih4sO55mknYE7o+IeWXH0qyve6nsbyrNJL2G7L36VNmxNOvrJqlKi7vK0m9T25F1fV0OvIZsgap3V+XH8joZdsnbzKpP0juB04CPu9W9bJy8zWzIpW6nkVX8BloXTt5mZjU0LIYKmpktb5y8zcxqyMnbzKyGnLxtKZIWS7pN0p2SLk6XsVvWsnaS9It0e7KkqW3OXSWtUTPQOk6WdFze+5vOOVfSBwdQ13qS7hxojGbd5uRtrTwbEVtExGZkU5sPazyozIDfOxExIyJOb3PKKsCAk7fZ8sjJ2zq5FtggtTjvlvQdshlya0vaVdINkm5JLfQxAJJ2k/QXSdeRzY4k3X+gpG+n22tIukzS7WnbDjgdeGNq9Z+Rzjte0kxJd0j6QkNZJ0iaI+m3ZItYtSXp0FTO7ZIubfo2sYukayXdk1a/Q1KvpDMa6q7CGiFmr3Dytn4pu8jA7kDftO+NgfMjYkvgabIr3ewSEVsBs4BjJI0Cvk+2rvTbyWa7tvJN4A8RsTnZipCzyVaeuze1+o+XtCuwIbAtsAWwtaR3SNqabAXJLck+HLbJ8XR+GhHbpPruJltqts96wI7Ae8jW6B6Vjj8REduk8g+VtH6OesyGxLBZmMq6arSk29Lta8lWgVsLmBcRN6b730q2pO31aVXPkcANZBdP+GvfOuqSfky2AFGzd5GtbUFELAaekPTapnN2TVvfEr9jyJL5WOCyvgvtSpqR4zltJulLZF0zY4ArGo5dlNZV/19J96XnsCvw5ob+8HGp7nty1GVWOCdva+XZiNii8Y6UoJ9uvAv4TUTs23TeFqCrkzEAAAE0SURBVEC3Zn6JbJGs7zXV8allqONc4L0RcbukA8nWk+7TXFakuo+KiMYkj6T1BlivWSHcbWLL6kay9bY3AJC0UroYwV+A9SW9MZ3X39KyVwGHp8f2pkWL/knWqu5zBXBQQ1/6BEmvA64B3idptKSxZF00nYwF/p4Wkvpo07EPSepJMb8BmJPqPrxv4SlJG6UFsswqwS1vWyYRsSC1YC/Qq9dvPDEi7pE0BfilpIVkVw7arEURnwSmSToYWAwcHhE3SLo+DcX7Ver3fhNwQ2r5PwXsFxG3SJoO3EZ2MYlrc4T8eeBP6fw/s+SHxBzgD2TLCB8WEc9J+gFZX/gtyipfQLaMqVkleG0TM7MacreJmVkNOXmbmdWQk7eZWQ05eZuZ1ZCTt5lZDTl5m5nVkJO3mVkN/X/vsDeI5BDiIAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn import svm, datasets\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "# Compute confusion matrix\n",
    "cm = confusion_matrix(y_compare, pred)\n",
    "np.set_printoptions(precision=2)\n",
    "print('Confusion matrix, without normalization')\n",
    "print(cm)\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cm, products)\n",
    "\n",
    "# Normalize the confusion matrix by row (i.e by the number of samples\n",
    "# in each class)\n",
    "cm_normalized = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "print('Normalized confusion matrix')\n",
    "print(cm_normalized)\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cm_normalized, products, title='Normalized confusion matrix')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.7 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}