MNIST_Handwritten_Digits-STARTER.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "In this project, you will build a neural network of your own design to evaluate the MNIST dataset.\n",
    "\n",
    "Some of the benchmark results on MNIST include can be found [on Yann LeCun's page](https://webcache.googleusercontent.com/search?q=cache:stAVPik6onEJ:yann.lecun.com/exdb/mnist) and include:\n",
    "\n",
    "88% [Lecun et al., 1998](https://hal.science/hal-03926082/document)\n",
    "\n",
    "95.3% [Lecun et al., 1998](https://hal.science/hal-03926082v1/document)\n",
    "\n",
    "99.65% [Ciresan et al., 2011](http://people.idsia.ch/~juergen/ijcai2011.pdf)\n",
    "\n",
    "\n",
    "MNIST is a great dataset for sanity checking your models, since the accuracy levels achieved by large convolutional neural networks and small linear models are both quite high. This makes it important to be familiar with the data.\n",
    "\n",
    "## Installation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Update the PATH to include the user installation directory. \n",
    "import os\n",
    "os.environ['PATH'] = f\"{os.environ['PATH']}:/root/.local/bin\"\n",
    "\n",
    "# Restart the Kernel before you move on to the next step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Important: Restart the Kernel before you move on to the next step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install requirements\n",
    "!python -m pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "## This cell contains the essential imports you will need – DO NOT CHANGE THE CONTENTS! ##\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "import torchvision\n",
    "import torchvision.transforms as transforms\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device(type='cpu')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "device"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the Dataset\n",
    "\n",
    "Specify your transforms as a list if you intend to .\n",
    "The transforms module is already loaded as `transforms`.\n",
    "\n",
    "MNIST is fortunately included in the torchvision module.\n",
    "Then, you can create your dataset using the `MNIST` object from `torchvision.datasets` ([the documentation is available here](https://pytorch.org/vision/stable/datasets.html#mnist)).\n",
    "Make sure to specify `download=True`! \n",
    "\n",
    "Once your dataset is created, you'll also need to define a `DataLoader` from the `torch.utils.data` module for both the train and the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define transforms\n",
    "transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5), (0.5))])\n",
    "\n",
    "# Create training set and define training dataloader\n",
    "## YOUR CODE HERE ##\n",
    "train_data = torchvision.datasets.MNIST(root=\"data\", train=True, download=True, transform=transform)\n",
    "\n",
    "# Create test set and define test dataloader\n",
    "## YOUR CODE HERE ##\n",
    "test_data = torchvision.datasets.MNIST(root=\"data\", train= False, download=True, transform=transform)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#DataLoader\n",
    "train_loader = torch.utils.data.DataLoader(dataset=train_data, batch_size=64, shuffle=True)\n",
    "test_loader = torch.utils.data.DataLoader(dataset=test_data, batch_size=64, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Justify your preprocessing\n",
    "\n",
    "In your own words, why did you choose the transforms you chose? If you didn't use any preprocessing steps, why not?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**By combining these transformations into a Compose pipeline, we efficiently preprocess the input data, preparing it for training while also enhancing the training process by standardizing the input distribution. This preprocessing approach contributes to more stable and effective training of neural network models, ultimately leading to improved performance and generalization ability on unseen data.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore the Dataset\n",
    "Using matplotlib, numpy, and torch, explore the dimensions of your data.\n",
    "\n",
    "You can view images using the `show5` function defined below – it takes a data loader as an argument.\n",
    "Remember that normalized images will look really weird to you! You may want to try changing your transforms to view images.\n",
    "Typically using no transforms other than `toTensor()` works well for viewing – but not as well for training your network.\n",
    "If `show5` doesn't work, go back and check your code for creating your data loaders and your training/test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "## This cell contains a function for showing 5 images from a dataloader – DO NOT CHANGE THE CONTENTS! ##\n",
    "def show5(img_loader):\n",
    "    dataiter = iter(img_loader)\n",
    "    \n",
    "    batch = next(dataiter)\n",
    "    labels = batch[1][0:5]\n",
    "    images = batch[0][0:5]\n",
    "    for i in range(5):\n",
    "        print(int(labels[i].detach()))\n",
    "    \n",
    "        image = images[i].numpy()\n",
    "        plt.imshow(image.T.squeeze().T)\n",
    "        plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of training samples: 60000\n",
      "Image shape: torch.Size([1, 28, 28])\n",
      "Number of classes: 10\n"
     ]
    }
   ],
   "source": [
    "# Explore data\n",
    "## YOUR CODE HERE ##\n",
    "# Get the size of the training dataset\n",
    "num_train_samples = len(train_data)\n",
    "print(\"Number of training samples:\", num_train_samples)\n",
    "\n",
    "# Get the shape of the images in the dataset\n",
    "image_shape = train_data[0][0].size()\n",
    "print(\"Image shape:\", image_shape)\n",
    "\n",
    "# Get the number of classes in the dataset\n",
    "num_classes = len(train_data.classes)\n",
    "print(\"Number of classes:\", num_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build your Neural Network\n",
    "Using the layers in `torch.nn` (which has been imported as `nn`) and the `torch.nn.functional` module (imported as `F`), construct a neural network based on the parameters of the dataset.\n",
    "Use any architecture you like. \n",
    "\n",
    "*Note*: If you did not flatten your tensors in your transforms or as part of your preprocessing and you are using only `Linear` layers, make sure to use the `Flatten` layer in your network!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Net(\n",
       "  (fc1): Linear(in_features=784, out_features=120, bias=True)\n",
       "  (fc2): Linear(in_features=120, out_features=84, bias=True)\n",
       "  (fc3): Linear(in_features=84, out_features=10, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## YOUR CODE HERE ##\n",
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.activation = F.relu\n",
    "        self.fc1 = nn.Linear(28 * 28 * 1, 120)\n",
    "        self.fc2 = nn.Linear(120, 84)\n",
    "        self.fc3 = nn.Linear(84, 10)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x = torch.flatten(x, 1) # flatten all dimensions except batch\n",
    "        x = self.activation(self.fc1(x))\n",
    "        x = self.activation(self.fc2(x))\n",
    "        x = self.fc3(x)\n",
    "        return x\n",
    "#initialize\n",
    "net = Net()\n",
    "net.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "#visualize\n",
    "scripted = torch.jit.script(net)\n",
    "torch.jit.save(scripted, \"mnist_network.pt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Specify a loss function and an optimizer, and instantiate the model.\n",
    "\n",
    "If you use a less common loss function, please note why you chose that loss function in a comment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "## YOUR CODE HERE ##\n",
    "#loss function\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "\n",
    "#specify optimizer\n",
    "optimizer = optim.Adam(net.parameters(), lr=0.001)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running your Neural Network\n",
    "Use whatever method you like to train your neural network, and ensure you record the average loss at each epoch. \n",
    "Don't forget to use `torch.device()` and the `.to()` method for both your model and your data if you are using GPU!\n",
    "\n",
    "If you want to print your loss **during** each epoch, you can use the `enumerate` function and print the loss after a set number of batches. 250 batches works well for most people!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1 training accuracy: 0.94% training loss: 0.18276\n",
      "Epoch 1 validation accuracy: 0.96% validation loss: 0.13452\n",
      "Epoch 2 training accuracy: 0.96% training loss: 0.13200\n",
      "Epoch 2 validation accuracy: 0.96% validation loss: 0.14052\n",
      "Epoch 3 training accuracy: 0.97% training loss: 0.10619\n",
      "Epoch 3 validation accuracy: 0.96% validation loss: 0.11004\n",
      "Epoch 4 training accuracy: 0.97% training loss: 0.09218\n",
      "Epoch 4 validation accuracy: 0.97% validation loss: 0.11047\n",
      "Epoch 5 training accuracy: 0.98% training loss: 0.07756\n",
      "Epoch 5 validation accuracy: 0.97% validation loss: 0.10582\n",
      "Epoch 6 training accuracy: 0.98% training loss: 0.07224\n",
      "Epoch 6 validation accuracy: 0.97% validation loss: 0.10166\n",
      "Epoch 7 training accuracy: 0.98% training loss: 0.06088\n",
      "Epoch 7 validation accuracy: 0.97% validation loss: 0.08723\n",
      "Epoch 8 training accuracy: 0.98% training loss: 0.05775\n",
      "Epoch 8 validation accuracy: 0.97% validation loss: 0.09581\n",
      "Epoch 9 training accuracy: 0.98% training loss: 0.05274\n",
      "Epoch 9 validation accuracy: 0.97% validation loss: 0.09240\n",
      "Epoch 10 training accuracy: 0.98% training loss: 0.04721\n",
      "Epoch 10 validation accuracy: 0.97% validation loss: 0.09276\n"
     ]
    }
   ],
   "source": [
    "## YOUR CODE HERE ##\n",
    "#Number of epochs to train\n",
    "n_epochs = 10\n",
    "\n",
    "# Move model and data to GPU if available\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "net.to(device)\n",
    "\n",
    "# Establish a list for our history\n",
    "train_loss_history = list()\n",
    "val_loss_history = list()\n",
    "\n",
    "for epoch in range(n_epochs):\n",
    "    net.train()\n",
    "    train_loss = 0.0\n",
    "    train_correct = 0\n",
    "    for i, data in enumerate(train_loader):\n",
    "        # data is a list of [inputs, labels]\n",
    "        inputs, labels = data\n",
    "\n",
    "        # Pass to GPU if available.\n",
    "        inputs, labels = inputs.to(device), labels.to(device)\n",
    "\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        outputs = net(inputs)\n",
    "        loss = criterion(outputs, labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "        _, preds = torch.max(outputs.data, 1)\n",
    "        train_correct += (preds == labels).float().mean().item()\n",
    "        train_loss += loss.item()\n",
    "    print(f'Epoch {epoch + 1} training accuracy: {train_correct/len(train_loader):.2f}% training loss: {train_loss/len(train_loader):.5f}')\n",
    "    train_loss_history.append(train_loss/len(train_loader))\n",
    "\n",
    "\n",
    "    val_loss = 0.0\n",
    "    val_correct = 0\n",
    "    net.eval()\n",
    "    for inputs, labels in test_loader:\n",
    "        inputs, labels = inputs.to(device), labels.to(device)\n",
    "\n",
    "        outputs = net(inputs)\n",
    "        loss = criterion(outputs, labels)\n",
    "\n",
    "        _, preds = torch.max(outputs.data, 1)\n",
    "        val_correct += (preds == labels).float().mean().item()\n",
    "        val_loss += loss.item()\n",
    "    print(f'Epoch {epoch + 1} validation accuracy: {val_correct/len(test_loader):.2f}% validation loss: {val_loss/len(test_loader):.5f}')\n",
    "    val_loss_history.append(val_loss/len(test_loader))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the training loss (and validation loss/accuracy, if recorded)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAuEklEQVR4nO3deXxU5d3//9cnOyEBsrBmIQkgiqwSUFAB0VqsC63VqsUqLsWlSkvtT23vu8tt228Xq/a21SpaXOt2a7VUad0FFJWEXRAshgABZQ37EpJ8fn/MgAEHCJDJSTLv5+ORBzNnzpn5zAB5z7muc12XuTsiIiIHigu6ABERaZoUECIiEpECQkREIlJAiIhIRAoIERGJSAEhIiIRKSBEIjCzf5nZlQ29r0hzYhoHIS2FmW2rczcV2A3UhO9f5+5/a/yqjp6ZjQCedPfcgEuRGJUQdAEiDcXd0/beNrNy4Fp3f+PA/cwswd2rG7M2keZITUzS4pnZCDOrMLPbzOxz4BEzyzCzl81snZlVhm/n1jnmHTO7Nnx7rJm9a2Z/CO+7zMzOOcp9C81smpltNbM3zOw+M3vyKN7TCeHX3WRmC83sgjqPfc3MFoVfY5WZ/Si8PTv8PjeZ2UYzm25m+h0gB6V/HBIrOgGZQFdgHKF/+4+E7+cDO4E/H+L4k4ElQDbwe+CvZmZHse9TwEwgC/gF8J0jfSNmlgj8E3gN6ADcDPzNzHqGd/kroSa1dKA38FZ4+y1ABdAe6Aj8BFAbsxyUAkJiRS3wc3ff7e473X2Du7/g7jvcfSvwa2D4IY5f7u4PuXsN8BjQmdAv2Xrva2b5wCDgZ+5e5e7vApOP4r2cAqQBvw0/z1vAy8Bl4cf3AL3MrI27V7r77DrbOwNd3X2Pu093dULKISggJFasc/dde++YWaqZPWhmy81sCzANaGdm8Qc5/vO9N9x9R/hm2hHu2wXYWGcbwMojfB+En2elu9fW2bYcyAnf/ibwNWC5mU01syHh7XcCS4HXzKzMzG4/iteWGKKAkFhx4DflW4CewMnu3gYYFt5+sGajhvAZkGlmqXW25R3F86wG8g7oP8gHVgG4e4m7jybU/PQS8Fx4+1Z3v8Xdi4ALgB+a2ZlH8foSIxQQEqvSCfU7bDKzTODn0X5Bd18OlAK/MLOk8Df78w93nJml1P0h1IexA7jVzBLDl8OeDzwTft4xZtbW3fcAWwg1r2Fm55lZ93B/yGZClwDXRnpNEVBASOz6I9AKWA98APy7kV53DDAE2AD8CniW0HiNg8khFGR1f/IIBcI5hOq/H7jC3ReHj/kOUB5uOrs+/JoAPYA3gG3A+8D97v52g70zaXE0UE4kQGb2LLDY3aN+BiNypHQGIdKIzGyQmXUzszgzGwWMJtRPINLkaCS1SOPqBPyd0DiICuAGd58TbEkikamJSUREIlITk4iIRNRimpiys7O9oKAg6DJERJqVWbNmrXf39pEeazEBUVBQQGlpadBliIg0K2a2/GCPqYlJREQiUkCIiEhECggREYmoxfRBiEjTs2fPHioqKti1a9fhd5aoSklJITc3l8TExHofo4AQkaipqKggPT2dgoICDr6+kkSbu7NhwwYqKiooLCys93FqYhKRqNm1axdZWVkKh4CZGVlZWUd8JqeAEJGoUjg0DUfz9xDzAbF55x7ufm0JS9duDboUEZEmJeYDorqmlgenlfHQtGVBlyIiDWzDhg3079+f/v3706lTJ3Jycvbdr6qqOuSxpaWljB8//rCvMXTo0Aap9Z133uG8885rkOdqKFENCDMbZWZLzGxppPVvzWyYmc02s2ozu+iAx35vZgvN7GMzu9eidJ6alZbMxcW5vDhnFWu36EoLkZYkKyuLuXPnMnfuXK6//nomTJiw735SUhLV1dUHPba4uJh77733sK8xY8aMhiy5SYlaQIQXf7+P0KpXvYDLzKzXAbutAMYCTx1w7FDgVKAv0BsYBAyPVq3XnlbEntpaHp1RHq2XEJEmYuzYsVx//fWcfPLJ3HrrrcycOZMhQ4YwYMAAhg4dypIlS4D9v9H/4he/4Oqrr2bEiBEUFRXtFxxpaWn79h8xYgQXXXQRxx9/PGPGjGHvbNlTpkzh+OOPZ+DAgYwfP/6IzhSefvpp+vTpQ+/evbntttsAqKmpYezYsfTu3Zs+ffpwzz33AHDvvffSq1cv+vbty6WXXnrMn1U0L3MdDCx19zIAM3uG0OIoi/bu4O7l4ccOXBfXgRQgidAi8onAmmgVWpDdmlEnduLJD5Zz4xndSUvW1b8iDe1//rmQRau3NOhz9urShp+ff+IRH1dRUcGMGTOIj49ny5YtTJ8+nYSEBN544w1+8pOf8MILL3zpmMWLF/P222+zdetWevbsyQ033PClMQVz5sxh4cKFdOnShVNPPZX33nuP4uJirrvuOqZNm0ZhYSGXXXZZvetcvXo1t912G7NmzSIjI4Ozzz6bl156iby8PFatWsVHH30EwKZNmwD47W9/y7Jly0hOTt637VhEs4kpB1hZ535FeNthufv7wNvAZ+GfV9394wP3M7NxZlZqZqXr1q07pmLHDStiy65qni1ZefidRaRZu/jii4mPjwdg8+bNXHzxxfTu3ZsJEyawcOHCiMece+65JCcnk52dTYcOHViz5svfWQcPHkxubi5xcXH079+f8vJyFi9eTFFR0b7xB0cSECUlJYwYMYL27duTkJDAmDFjmDZtGkVFRZSVlXHzzTfz73//mzZt2gDQt29fxowZw5NPPklCwrF/0W2SX5XNrDtwApAb3vS6mZ3u7tPr7ufuE4GJAMXFxce08tGA/AwGF2Qy6d1lXDGkK4nxMd9/L9KgjuabfrS0bt163+2f/vSnnHHGGbz44ouUl5czYsSIiMckJyfvux0fHx+x/6I++zSEjIwM5s2bx6uvvsoDDzzAc889x6RJk3jllVeYNm0a//znP/n1r3/NggULjikoovlbcBWQV+d+bnhbfXwD+MDdt7n7NuBfwJAGru9Lxg0rYtWmnUxZ8Fm0X0pEmojNmzeTkxNq3Hj00Ucb/Pl79uxJWVkZ5eXlADz77LP1Pnbw4MFMnTqV9evXU1NTw9NPP83w4cNZv349tbW1fPOb3+RXv/oVs2fPpra2lpUrV3LGGWfwu9/9js2bN7Nt27Zjqj2aAVEC9DCzQjNLAi4FJtfz2BXAcDNLMLNEQh3UX2piamgjj+9At/ateXBqGVqKVSQ23Hrrrfz4xz9mwIABUfnG36pVK+6//35GjRrFwIEDSU9Pp23bthH3ffPNN8nNzd33U15ezm9/+1vOOOMM+vXrx8CBAxk9ejSrVq1ixIgR9O/fn8svv5zf/OY31NTUcPnll9OnTx8GDBjA+PHjadeu3THVHtU1qc3sa8AfgXhgkrv/2szuAErdfbKZDQJeBDKAXcDn7n5i+Aqo+4FhhDqs/+3uPzzUaxUXF3tDLBj0bMkKbnthAU9eczKn9cg+5ucTiWUff/wxJ5xwQtBlBG7btm2kpaXh7nzve9+jR48eTJgwodHriPT3YWaz3L040v5R7YNw9ynAlAO2/azO7RK+6Geou08NcF00azuYrw/I4Q+vfcKD0z5VQIhIg3jooYd47LHHqKqqYsCAAVx3XSC/3o5Yk+ykDlJyQjxjhxZw56tLWLR6C726tAm6JBFp5iZMmBDIGcOx0qU6EVx+cldSk+J5aHpZ0KWINHvqz2sajubvQQERQdvURC4dlM8/561m9aadQZcj0mylpKSwYcMGhUTA9q4HkZKSckTHqYnpIK4+rYDH3i9n0rvL+O/zDpwhRETqIzc3l4qKCo51IKscu70ryh0JBcRB5Gakcl7fzjw9cwU3n9mDtq3qv0yfiIQkJiYe0Qpm0rSoiekQvnt6Eduranh65oqgSxERaXQKiEPondOWU7tn8ch7y6iqPnA+QRGRlk0BcRjjhnVjzZbd/GNufWcJERFpGRQQhzGsRzbHd0rnoemafkNEYosC4jDMjHHDivhkzTbeWaIrMUQkdigg6uH8fl3o3DaFB6d9GnQpIiKNRgFRD4nxcVx9aiEflG1kfsWmoMsREWkUCoh6unRwHunJCTw4TdNviEhsUEDUU3pKIt8+JZ9/LfiMFRt2BF2OiEjUKSCOwNWnFhIfZ/z1XZ1FiEjLp4A4Ah3bpDC6fw7PlVZQub0q6HJERKJKAXGExg0rYueeGp74YHnQpYiIRJUC4ggd1zGdM3q257EZ5ezaUxN0OSIiUaOAOArjhnVjw/YqXphdEXQpIiJRo4A4CqcUZdI3ty0PT19GTa2m3xCRlimqAWFmo8xsiZktNbPbIzw+zMxmm1m1mV10wGP5ZvaamX1sZovMrCCatR6JvdNvLFu/ndcXrQm6HBGRqIhaQJhZPHAfcA7QC7jMzA5cmm0FMBZ4KsJTPA7c6e4nAIOBtdGq9WiMOrETeZmtmKjpN0SkhYrmGcRgYKm7l7l7FfAMMLruDu5e7u7zgf0WWwgHSYK7vx7eb5u7N6nRaQnxcVx7WhGzV2yitHxj0OWIiDS4aAZEDrCyzv2K8Lb6OA7YZGZ/N7M5ZnZn+IxkP2Y2zsxKzaw0iDVvLy7OpV1qoqbfEJEWqal2UicApwM/AgYBRYSaovbj7hPdvdjdi9u3b9+4FQKpSQlccUpX3vh4DZ+u29bory8iEk3RDIhVQF6d+7nhbfVRAcwNN09VAy8BJzVseQ3jiqEFJMXH8fB0nUWISMsSzYAoAXqYWaGZJQGXApOP4Nh2Zrb3tGAksCgKNR6z7LRkvjkwlxdmr2Ld1t1BlyMi0mCiFhDhb/43Aa8CHwPPuftCM7vDzC4AMLNBZlYBXAw8aGYLw8fWEGpeetPMFgAGPBStWo/VtacVsqemlsffLw+6FBGRBmMtZZ3l4uJiLy0tDez1xz1eyszyjcy4fSSpSQmB1SEiciTMbJa7F0d6rKl2Ujc71w0vYtOOPTxXsvLwO4uINAMKiAYysGsmA7tm8PC7y6iuqT38ASIiTZwCogGNG1ZEReVO/vXR50GXIiJyzBQQDegrJ3SkKLs1E6eV0VL6dkQkdikgGlBcnHHt6UUsWLWZ98s2BF2OiMgxUUA0sAtPyiE7LYmJmn5DRJo5BUQDS0mM58ohBbyzZB1LPt8adDkiIkdNAREFl5/SlVaJ8TqLEJFmTQERBRmtk7hkUB6T563i8827gi5HROSoKCCi5JrTCqmpdR55b1nQpYiIHBUFRJTkZabytT6deerDFWzdtSfockREjpgCIoquG9aNrbureXrmiqBLERE5YgqIKOqT25YhRVlMerecqmpNvyEizYsCIsrGDS/i8y27+Oe81UGXIiJyRBQQUTbiuPb07JjOQ9M1/YaINC8KiCgzM747rIjFn29l6ifrgi5HRKTeFBCN4IJ+XejUJkUD50SkWVFANIKkhDiuOrWAGZ9u4KNVm4MuR0SkXhQQjeSyk/NJS07gQZ1FiEgzoYBoJG1SEvn2yflMWfAZKzfuCLocEZHDimpAmNkoM1tiZkvN7PYIjw8zs9lmVm1mF0V4vI2ZVZjZn6NZZ2O56tQCDJik6TdEpBmIWkCYWTxwH3AO0Au4zMx6HbDbCmAs8NRBnuaXwLRo1djYOrdtxQX9uvBsyUo279D0GyLStEXzDGIwsNTdy9y9CngGGF13B3cvd/f5wJeGGZvZQKAj8FoUa2x03x1WxI6qGp78cHnQpYiIHFI0AyIHWFnnfkV422GZWRxwF/CjKNQVqBM6t2HYce155L1ydu2pCbocEZGDaqqd1DcCU9y94lA7mdk4Mys1s9J165rPILTrhhWxfttuXpqzKuhSREQOKpoBsQrIq3M/N7ytPoYAN5lZOfAH4Aoz++2BO7n7RHcvdvfi9u3bH2u9jWZotyxO7NKGidPLqK3V9Bsi0jRFMyBKgB5mVmhmScClwOT6HOjuY9w9390LCDUzPe7uX7oKqrkyM8YNK6Js3XbeXLw26HJERCKKWkC4ezVwE/Aq8DHwnLsvNLM7zOwCADMbZGYVwMXAg2a2MFr1NDXn9ulMTrtWTJz2adCliIhElBDNJ3f3KcCUA7b9rM7tEkJNT4d6jkeBR6NQXqAS4uO45rRC7nh5EbNXVHJSfkbQJYmI7KepdlLHhEsG5dG2VSITp2r6DRFpehQQAWqdnMDlp+Tz6qLPWbZ+e9DliIjsRwERsCuHFpAYF8fD03UWISJNiwIiYB3SU7jwpByen1XB+m27gy5HRGQfBUQTcO3pReyuruXx9zX9hog0HQqIJqB7hzTOOqEjT7xfzvbd1UGXIyICKCCajBvP6MamnXv4/jNzqa750tyFIiKNTgHRRJyUn8EdF5zIGx+v4af/+Ah3TcEhIsGK6kA5OTLfGVLAmi27+fPbS+mQnsKErxwXdEkiEsMUEE3MLWcfx9qtu/jfN/9DhzbJjDm5a9AliUiMUkA0MWbG//tGH9Zvq+KnL31EdloyXz2xU9BliUgMUh9EE5QQH8efvz2AvrntGP/0HErKNwZdkojEIAVEE5WalMCksYPIadeKax4t4ZM1W4MuSURijAKiCctsncRjVw8mOTGeKyfN5LPNO4MuSURiiAKiicvLTOWxqwazbVc1V06ayeYde4IuSURihAKiGejVpQ0PXjGQ8vU7+O7jpezaUxN0SSISAxQQzcTQbtncfUk/SpZv5PvPzKFGa1mLSJQpIJqR8/p24Wfn9eLVhWv4mUZbi0iUaRxEM3PVqYWs2bKbB6Z+Sqc2Kdx8Zo+gSxKRFkoB0QzdNqona7fu4q7XP6F9ejKXDs4PuiQRaYEUEM2QmfG7b/Zl/bYqfvLiArLTkjmrV8egyxKRFiaqfRBmNsrMlpjZUjO7PcLjw8xstplVm9lFdbb3N7P3zWyhmc03s0uiWWdzlBgfx1/GnETvnLbc9PRsZi2vDLokEWlhohYQZhYP3AecA/QCLjOzXgfstgIYCzx1wPYdwBXufiIwCvijmbWLVq3NVevk0GjrTm1SuOaxEpau3RZ0SSLSgtQrIMystZnFhW8fZ2YXmFniYQ4bDCx19zJ3rwKeAUbX3cHdy919PlB7wPZP3P0/4durgbVA+3q9oxiTnZbM41efTEKcceWkmazZsivokkSkhajvGcQ0IMXMcoDXgO8Ajx7mmBxgZZ37FeFtR8TMBgNJwKcRHhtnZqVmVrpu3bojfeoWIz8rlUevGsymHVWh0dY7NdpaRI5dfQPC3H0HcCFwv7tfDJwYvbLCL2rWGXgCuMrdv7QOp7tPdPdidy9u3z62TzB657Tlge8M5NN12xin0dYi0gDqHRBmNgQYA7wS3hZ/mGNWAXl17ueGt9X3BduEX+u/3P2D+h4Xy07v0Z4/XNyPD5dt5IfPzdVoaxE5JvUNiB8APwZedPeFZlYEvH2YY0qAHmZWaGZJwKXA5Pq8WHj/F4HH3f35etYowOj+Ofz3uScwZcHn3PHPhRptLSJHrV7jINx9KjAVINxZvd7dxx/mmGozuwl4ldDZxqRwuNwBlLr7ZDMbRCgIMoDzzex/wlcufQsYBmSZ2djwU45197lH/A5j0LWnF7Fmyy4emr6Mjm1TuHFE96BLEpFmyOrzDdPMngKuB2oInRm0Af7X3e+Mbnn1V1xc7KWlpUGX0WTU1joTnpvLP+au5s6L+nJxcd7hDxKRmGNms9y9ONJj9W1i6uXuW4CvA/8CCgldySRNVFyccedF/Titeza3/30Bby9ZG3RJItLM1DcgEsPjHr4OTHb3PYAat5u4pIQ4HvjOQE7onM6NT85m7spNQZckIs1IfQPiQaAcaA1MM7OuwJZoFSUNJy05gUfGDqZ9ejJXP1pC2TqNthaR+qlXQLj7ve6e4+5f85DlwBlRrk0aSPv0ZB67ejAGXDFpJms12lpE6qG+U220NbO7945aNrO7CJ1NSDNRmN2aSWMHsXF7FVc+UsKWXRptLSKHVt8mpknAVkKXn36LUPPSI9EqSqKjX1477h9zEv9Zs5Xrn5jF7mqNthaRg6tvQHRz95+HJ94rc/f/AYqiWZhEx4ieHfj9RX2Z8ekGbnluHrUabS0iB1HfgNhpZqftvWNmpwI7o1OSRNuFJ+Vy+znH8/L8z/jlK4s02lpEIqrvinLXA4+bWdvw/UrgyuiUJI3humGh0daPvFdOpzYpXDe8W9AliUgTU9+pNuYB/cIT6OHuW8zsB8D8KNYmUWRm/PTcXqzdupvf/Gsx7dOTufCk3KDLEpEm5IhWlHP3LeER1QA/jEI90oji4oy7v9WPIUVZ3Pr8fKZ+ErtraojIlx3LkqPWYFVIYJIT4nnwioH06JjODU/OYn7FpqBLEpEm4lgCQj2bLUSblEQeu2oQGalJXPVICeXrtwddkog0AYcMCDPbamZbIvxsBbo0Uo3SCDq0SeHxawZT686Yhz9kxqfrgy5JRAJ2yIBw93R3bxPhJ93d63sFlDQT3dqn8ehVg4mLg28/9CHjn56jaTlEYtixNDFJC9Qvrx2vTxjO+DN78O+FnzPyrqn89d1lVNd8aUlwEWnhFBDyJSmJ8fzwK8fx2g+GMbBrBr98eRHn/eldSso3Bl2aiDQiBYQcVEF2ax69ahAPXD6QLTv3cPED73PLc/NYv2130KWJSCNQQMghmRmjenfijVuGc+OIbkyet4qRf3iHJ94vp0bzOIm0aAoIqZfUpARuHXU8//r+MPrktuWn/1jI6PveZc6KyqBLE5EoiWpAmNkoM1tiZkvN7PYIjw8zs9lmVm1mFx3w2JVm9p/wj+Z9aiK6d0jjyWtO5k+XDWDd1t1c+JcZ/Pjv86ncXhV0aSLSwKIWEGYWD9wHnAP0Ai4zs14H7LYCGAs8dcCxmcDPgZOBwcDPzSwjWrXKkTEzzu/XhTdvGcG1pxXyXGkFI+96h2dmrtD04SItSDTPIAYDS8PrR1QBzwCj6+7g7uXuPh848BrKrwKvu/tGd68EXgdGRbFWOQppyQn817m9mDL+dHp0TOf2vy/gwr/M4KNVm4MuTUQaQDQDIgdYWed+RXhbgx1rZuP2LoO6bp0mmgtKz07pPDvuFO7+Vj8qKndwwZ/f5Wf/+IjNO7SsqUhz1qw7qd19orsXu3tx+/btgy4nppkZF56Uy5u3jOCKIQU8+cFyRt71Ds/PqtCCRCLNVDQDYhWQV+d+bnhbtI+VALVtlcgvLjiRyTedRn5WKj/6v3l868H3Wfz5lsMfLCJNSjQDogToYWaFZpYEXApMruexrwJnm1lGuHP67PA2aSZ657TlheuH8vtv9uXTdds59953+eXLi9i6S81OIs1F1ALC3auBmwj9Yv8YeM7dF5rZHWZ2AYCZDTKzCuBi4EEzWxg+diPwS0IhUwLcEd4mzUhcnPGtQXm8dctwLhmUx6T3lnHmXVP5x9xVanYSaQaspfxHLS4u9tLS0qDLkEOYt3ITP/3HR8yv2MzQblncMfpEundID7oskZhmZrPcvTjSY826k1qal3557XjxxlP51dd7s3D1Fkb9cTq/+dfHbN9dHXRpIhKBAkIaVXyccfkpXXnrluFceFIOD04t4yt3T+VfCz5Ts5NIE6OAkEBkpSXz+4v68cINQ2ibmsQNf5vNlY+UsEzLnYo0GQoICdTArpn886ZT+fn5vZizvJKv3jONu19bwq49NUGXJhLzFBASuIT4OK46tZA3fzScc/t25t63lnLW3VN58+M1QZcmEtMUENJkdEhP4Z5L+vPsuFNITYrnmsdKueHJWazRutgigVBASJNzclEWr4w/nVtH9eStxWs5666pPPHBcs0UK9LIFBDSJCXGx3HjiO68NmEY/fLa8dOXPuLiB99nyedbgy5NJGYoIKRJ65rVmieuGcw9l/Rj2frtnHvvdO58dbE6sUUagQJCmjwz4xsDcnnjh8P5+oAc7nv7U0b9cRrvLV0fdGkiLZoCQpqNzNZJ/OHifjx17ckAjHn4Q3743Fw2arlTkahQQEizM7R7Nv/+wTBuOqM7k+eu5sy73uEFrTsh0uAUENIspSTG86Ov9mTK90+nqH0at/zfPC7/64eUayS2SINRQEizdlzHdP7vuiH86uu9mb9yM1/94zTue3spVdUHLnMuIkdKASHNXlx4AsA3bhnOmSd04M5Xl3D+n95l1vLKoEsTadYUENJidGyTwv1jBvLwFcVs3bWHix6YwX+/tIAtWsVO5KgoIKTFOatXR17/4XCuGlrIUx+u4Ky7NJ24yNFQQEiL1Do5gZ+d34uXvncq2WnJ3PC32Xz38VJWb9oZdGkizYYCQlq0vrntmHzTqfzX107gvaUbOOvuqUx6dxk1mtdJ5LAUENLiJcTH8d1hRbw2YRiDCzO54+VFfOP+9/ho1eagSxNp0qIaEGY2ysyWmNlSM7s9wuPJZvZs+PEPzawgvD3RzB4zswVm9rGZ/TiadUpsyMtM5ZGxg/jTZQNYvWkno+97j/835WN2VGlNbJFIohYQZhYP3AecA/QCLjOzXgfsdg1Q6e7dgXuA34W3Xwwku3sfYCBw3d7wEDkWZsb5/brw5g9H8K3iXCZOK+Pse6bxzpK1QZcm0uRE8wxiMLDU3cvcvQp4Bhh9wD6jgcfCt58HzjQzAxxobWYJQCugCtgSxVolxrRNTeQ3F/blueuGkJwQx9hHSrj56Tms27o76NJEmoxoBkQOsLLO/Yrwtoj7uHs1sBnIIhQW24HPgBXAH9x944EvYGbjzKzUzErXrVvX8O9AWrzBhZlM+f7p/OCsHrz60eecedc7PDNzhRYnEqHpdlIPBmqALkAhcIuZFR24k7tPdPdidy9u3759Y9coLURyQjw/OOs4pnz/dI7v3Ibb/76ASyd+wNK1WpxIYls0A2IVkFfnfm54W8R9ws1JbYENwLeBf7v7HndfC7wHFEexVhG6d0jjme+ewu++2Ycla7Zyzv9O557XP9HiRBKzEqL43CVADzMrJBQElxL6xV/XZOBK4H3gIuAtd3czWwGMBJ4ws9bAKcAfo1irCBCa1+mSQfmMPL4jv3x5Ef/75n+YOK2MfnltGdg1g+KumZyUn0Hb1MSgSxWJOovm9ANm9jVCv9jjgUnu/mszuwModffJZpYCPAEMADYCl7p7mZmlAY8QuvrJgEfc/c5DvVZxcbGXlpZG7b1IbJrx6XpeW7iG2SsqWbh6y74Bdj06pFFckMHArpkUd82ga1YqoesrRJoXM5vl7hFbaKIaEI1JASHRtqOqmrkrNzGrvJJZKyqZtbySrbtCYyiy05I4KT8jHBoZ9M5pS3JCfMAVixzeoQIimk1MIi1KalICQ7tlM7RbNgC1tc5/1m5j1vJKSpdvZNbySl5btAaApIQ4+uaEmqX2/mSlJQdZvsgR0xmESANau3UXs5dvYtbyjZQur+SjVZvZUxP6P1aU3ZqTumZQ3DV0plGUnUZcnJqlJFhqYhIJyK49NSxYtZnS8kpmhc8yKneE1qdol5rISfkZ4c7vDPrmtqNVkpqlpHGpiUkkICmJ8QwqyGRQQSbQDXenbP12ZpV/0Sz11uLQNB8JccaJOW0p7vpFaHRokxLsG5CYpjMIkYBt3F7F7OXhju/ySuZVbGJ3eE3tvMxWDMzPYGBBJqNO7ET7dPVjSMNSE5NIM1JVXctHqzcze3klpeWVlC6vZP223aQkxnHZ4HyuH96NjjqzkAaigBBpxtxDV0tNnFbGi3NWER9nXFKcx/UjupHTrlXQ5Ukzp4AQaSFWbNjBX6Yu5flZFQBcNDCXG0d0Jy8zNeDKpLlSQIi0MKs27eSBdz7l2ZKV1LjzjQE5fO+M7hRmtw66NGlmFBAiLdTnm3fx4LRPeerDFeypqeX8fl246Yzu9OiYHnRp0kwoIERauHVbd/Pw9DKe+GA5O/fU8LXenblpZHdO6Nwm6NKkiVNAiMSIjdur+Ou7ZTw2Yznbdldzdq+OjD+zB71z2gZdmjRRCgiRGLNpRxWPvFfOI+8tY8uuakYe34GbR3ZnQH5G0KVJE6OAEIlRW3bt4fEZ5Tz87jI27djD6T2yuXlkDwYXZgZdmjQRCgiRGLd9dzVPfrCcidPK2LC9ilOKMhl/Zg+GFGVpHYsYp4AQEQB2VtXw1MwVPDj1U9Zu3U1x1wxuPrMHw3pkKyhilAJCRPaza08Nz5Wu5C/vfMpnm3fRL68d40d2Z+TxHRQUMUYBISIR7a6u4YVZq7j/naVUVO7kxC5tuHlkd87u1UlrVcQIBYSIHNKemlpemrOK+95eSvmGHfTsmM5NI7vztT6diVdQtGgKCBGpl+qaWl6e/xl/eus/fLpuO93at+amkd05v28XEuLjgi5PouBQARHVv3EzG2VmS8xsqZndHuHxZDN7Nvz4h2ZWUOexvmb2vpktNLMFZqb5jUWiLCE+jq8PyOG1CcP587cHkBgfx4Rn53HW3VN5rnQle2pqgy5RGlHUziDMLB74BPgKUAGUAJe5+6I6+9wI9HX3683sUuAb7n6JmSUAs4HvuPs8M8sCNrl7zcFeT2cQIg2vttZ5bdEa/vTWf1i4egs57VoxsGsG+Zmp5Gelkp+ZStesVDqmp6jPopkKasnRwcBSdy8LF/EMMBpYVGef0cAvwrefB/5soUsozgbmu/s8AHffEMU6ReQg4uKMUb078dUTO/LW4rU88cFy5qys5JUFn1FT+8WXy6SEOPIyWoUDozV5mal0DYdHXmYqKYlaa7s5imZA5AAr69yvAE4+2D7uXm1mm4Es4DjAzexVoD3wjLv//sAXMLNxwDiA/Pz8Bn8DIhJiZpx5QkfOPKEjEOrUXr1pJ8s37GDFxvDPhh0s37iDmcs2sr1q/5P9jm2SQ2cdma33nXXkhf/Map2kS2ubqGgGxLFIAE4DBgE7gDfDp0Fv1t3J3ScCEyHUxNToVYrEqMT4OLpmtaZr1pfXn3B3Nm6v+lJwrNi4g/eWrueFLbv22791Uvy+sAg1XYVDJDOVLu1akZSgzvGgRDMgVgF5de7nhrdF2qci3O/QFthA6GxjmruvBzCzKcBJwJuISJNmZmSlJZOVlhxxcsBde2qoqAwFxr4zkA07KFu3nXeWrGN39Rcd4XEGXdq12v+sI7M1BdmpHNcxnURdWRVV0QyIEqCHmRUSCoJLgW8fsM9k4ErgfeAi4C1339u0dKuZpQJVwHDgnijWKiKNJCUxnu4d0une4cuLGtXWOuu27a4THNtDQbJxB68vWsP6bVX79k1OiKNPTlv657Wjf347+ue1I6ddKzVXNaCoBUS4T+Em4FUgHpjk7gvN7A6g1N0nA38FnjCzpcBGQiGCu1ea2d2EQsaBKe7+SrRqFZGmIS7O6NgmhY5tUiLOOLttdzUrN+5g6dptzFu5iTkrN/HEB8t5+N1lALRPTw4FRl47BuS3o29uO9KSm2pLetOngXIi0qztqall8WdbmbOykrkrNjF35SbK1m8HwAyO65C+LzD657ejR4d0jQ6vQyOpRSSmbNpRxdyVobCYEw6NzTv3AKFO8b65XzRLDchrR4c2sTsON6hxECIigWiXmsSInh0Y0bMDELqyqnzDDuasqNwXHA9NK6M6PJYjp12r/Zqmeue01dgNFBAiEgPMjMLs1hRmt+bCk3KB0NVUC1dv3neGMWfFJl5Z8BkACXHG8Z3TGZCXsa8TvDCrdcyNFlcTk4hI2Nqtu5i3cvO+M435FZvZtrsagDYpCfTPz9jXLNU/rx0ZrZMCrvjYqQ9CROQo1NQ6S9duY+7Kyn1nGZ+s2creWUYKs1szqCCDQQWZDC7MJD8ztdldZquAEBFpINt2V7OgYjNzVlYye3klJeWV+zrAO6QnM6gwk8EFmQwqyKRnp6Z/xZQ6qUVEGkhacgJDumUxpFsWEBrc95+125hZvpGSZRspKd/IK/NDfRnpKQkUd83YFxp9ctuSnNB8Or8VECIixyAuzujZKZ2endL5zildcXcqKndSUh4Ki5nLNvL2knVAaPR3v7x2oTOMwkwGds1o0gP51MQkIhJlG7btpqS8cl9oLFy9hZpaJ86gV5c2oT6McGhkpyU3am3qgxARaUK27a5mzopKSpZtZGb5Ruas2LRvksKi7NYMCofF4IJM8jKjO7+UAkJEpAmrqq5lwarNoTOMcD/Gll2hy2s7tkned5XUoIJMenZMb9DxGAoIEZFmpLbWWbJm674+jJLyjazZshsIjccoDl8lNbgwkz45bY9pzQxdxSQi0ozExRkndG7DCZ3bcMWQAtydlRt37nel1FuL1wKQkhjHV3p14k+XDWjwOhQQIiJNnJmRn5VKflYqFw0MTRWybutuSstDfRipSdG5dFYBISLSDLVPT+acPp05p0/nqL2G1usTEZGIFBAiIhKRAkJERCJSQIiISEQKCBERiUgBISIiESkgREQkIgWEiIhE1GLmYjKzdcDyY3iKbGB9A5XT3Omz2J8+j/3p8/hCS/gsurp7+0gPtJiAOFZmVnqwCatijT6L/enz2J8+jy+09M9CTUwiIhKRAkJERCJSQHxhYtAFNCH6LPanz2N/+jy+0KI/C/VBiIhIRDqDEBGRiBQQIiISUcwHhJmNMrMlZrbUzG4Pup4gmVmemb1tZovMbKGZfT/omoJmZvFmNsfMXg66lqCZWTsze97MFpvZx2Y2JOiagmRmE8L/Tz4ys6fNLCXomhpaTAeEmcUD9wHnAL2Ay8ysV7BVBaoauMXdewGnAN+L8c8D4PvAx0EX0UT8L/Bvdz8e6EcMfy5mlgOMB4rdvTcQD1wabFUNL6YDAhgMLHX3MnevAp4BRgdcU2Dc/TN3nx2+vZXQL4CcYKsKjpnlAucCDwddS9DMrC0wDPgrgLtXufumQIsKXgLQyswSgFRgdcD1NLhYD4gcYGWd+xXE8C/EusysABgAfBhwKUH6I3ArUBtwHU1BIbAOeCTc5PawmbUOuqiguPsq4A/ACuAzYLO7vxZsVQ0v1gNCIjCzNOAF4AfuviXoeoJgZucBa919VtC1NBEJwEnAX9x9ALAdiNk+OzPLINTaUAh0AVqb2eXBVtXwYj0gVgF5de7nhrfFLDNLJBQOf3P3vwddT4BOBS4ws3JCTY8jzezJYEsKVAVQ4e57zyifJxQYseosYJm7r3P3PcDfgaEB19TgYj0gSoAeZlZoZkmEOpkmB1xTYMzMCLUxf+zudwddT5Dc/cfunuvuBYT+Xbzl7i3uG2J9ufvnwEoz6xnedCawKMCSgrYCOMXMUsP/b86kBXbaJwRdQJDcvdrMbgJeJXQVwiR3XxhwWUE6FfgOsMDM5oa3/cTdpwRXkjQhNwN/C3+ZKgOuCriewLj7h2b2PDCb0NV/c2iB025oqg0REYko1puYRETkIBQQIiISkQJCREQiUkCIiEhECggREYlIASFyBMysxszm1vlpsNHEZlZgZh811POJHKuYHgchchR2unv/oIsQaQw6gxBpAGZWbma/N7MFZjbTzLqHtxeY2VtmNt/M3jSz/PD2jmb2opnNC//snaYh3sweCq8z8JqZtQrsTUnMU0CIHJlWBzQxXVLnsc3u3gf4M6GZYAH+BDzm7n2BvwH3hrffC0x1936E5jTaO4K/B3Cfu58IbAK+GdV3I3IIGkktcgTMbJu7p0XYXg6MdPey8ISHn7t7lpmtBzq7+57w9s/cPdvM1gG57r67znMUAK+7e4/w/duARHf/VSO8NZEv0RmESMPxg9w+Ervr3K5B/YQSIAWESMO5pM6f74dvz+CLpSjHANPDt98EboB96163bawiRepL305EjkyrOjPdQmiN5r2XumaY2XxCZwGXhbfdTGgVtv+P0Ipse2dA/T4w0cyuIXSmcAOhlclEmgz1QYg0gHAfRLG7rw+6FpGGoiYmERGJSGcQIiISkc4gREQkIgWEiIhEpIAQEZGIFBAiIhKRAkJERCL6/wGvTG8IDvPqKgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAv2klEQVR4nO3dd3yV9fn/8deVzZ5JBILsQFCGEJYKyKobHDioC21F/VZttbXVtj9tbW3tt9Zav7UqdVeUumqddaAyioOh7B1WmAkywghZ1++Pc8AQjpBATk7Oyfv5eOSRc89z5Sj3+9yf+/7cH3N3REREKoqLdAEiIlI7KSBERCQkBYSIiISkgBARkZAUECIiEpICQkREQlJASJ1jZm5mnYOvHzOz/1eZdY/hfa4ws/ePtU6RSFNASNQxs/+Y2b0h5o8xs81mllDZfbn7je7+m2qoqX0wTA6+t7tPcvfvHO++Q7zXGWaWW937FalIASHR6FngSjOzCvOvAia5e0kEahKJOQoIiUavAy2AwQdmmFkz4DzgOTPrb2afmtkOM9tkZn81s6RQOzKzZ8zst+Wm7whus9HMrquw7rlm9qWZ7TKz9Wb2q3KLpwV/7zCz3WY2yMzGm9mMctufamazzGxn8Pep5ZZ9Yma/MbP/mlmBmb1vZi2r+sGYWVZwXzvMbJGZjS637BwzWxzc/wYz+0lwfkszeyu4zddmNt3MdGwQBYREH3ffB7wEXF1u9qXAUnefB5QCtwEtgUHACOB/jrZfMzsL+AkwCugCjKywyp7gezYFzgVuMrMLgsuGBH83dfeG7v5phX03B94GHiYQbg8Cb5tZi3KrfRe4FkgDkoK1VJqZJQJvAu8H93ELMMnMugZXeRK4wd0bAScDHwXn/xjIBVKBdODngJ7BIwoIiVrPAmPNLCU4fXVwHu4+x90/c/cSd18DPA4MrcQ+LwWedveF7r4H+FX5he7+ibsvcPcyd58PvFjJ/UIgUFa4+z+Cdb0ILAXOL7fO0+6+vFwA9q7kvg8YCDQE7nf3Inf/CHgLGBdcXgx0N7PG7r7d3eeWm98KaOfuxe4+3fWQNkEBIVHK3WcA+cAFZtYJ6A+8AGBmmcEmk81mtgv4HYGziaNpDawvN722/EIzG2BmH5tZnpntBG6s5H4P7HtthXlrgTblpjeXe72XwMG+KloD69297Fve42LgHGCtmU01s0HB+X8EVgLvm1mOmd1ZxfeVGKWAkGj2HIEzhyuB99x9S3D+owS+nXdx98YEmkwqXtAOZRPQttz0iRWWvwC8AbR19ybAY+X2e7Rv3BuBdhXmnQhsqERdlbURaFvh+sHB93D3We4+hkDz0+sEzlJw9wJ3/7G7dwRGA7eb2YhqrEuilAJCotlzBK4TXE+weSmoEbAL2G1m3YCbKrm/l4DxZtbdzOoD91RY3gj42t0Lzaw/gWsGB+QBZUDHb9n3O0CmmX3XzBLM7DKgO4EmoGNiZinlf4AvCJx5/NTMEs3sDAJNWJPNLCnYL6OJuxcT+HzKgvs5z8w6B+8K20ngGk5ZqPeUukUBIVEreH1hJtCAwDf7A35C4OBdAPwd+Gcl9/cu8BCBi7cr+eYi7gH/A9xrZgXA3QS/gQe33QvcB/w3eDfQwAr73kbgLqsfA9uAnwLnuXt+ZWoLoQ2wr8JPWwKBcDaB5re/AVe7+9LgNlcBa4LNbjcCVwTndwE+BHYDnwJ/c/ePj7EuiSGma1EiIhKKziBERCQkBYSIiISkgBARkZAUECIiElKln3pZ27Vs2dLbt28f6TJERKLKnDlz8t09NdSymAmI9u3bM3v27EiXISISVcysYg//g9TEJCIiISkgREQkJAWEiIiEFDPXIESk5hQXF5Obm0thYWGkS5FKSklJISMjg8TExEpvo4AQkSrLzc2lUaNGtG/fnsNHfpXaxt3Ztm0bubm5dOjQodLbqYlJRKqssLCQFi1aKByihJnRokWLKp/xKSBE5JgoHKLLsfz3CmtAmNlZZrbMzFaGGqXKzIaY2VwzKzGzsSGWNzazXDP7azjrrA3yd+/nxS/WUVyqx/CLSO0QtoAws3jgEQLPpu8OjDOz7hVWWweMJzhUZAi/AaaFq8bawN3591cbGPXgVO56bQHvLtx89I1E6rhhw4bx3nvvHTLvoYce4qabvn1sqDPOOONgZ9pzzjmHHTt2HLbOr371Kx544IEjvvfrr7/O4sWLD07ffffdfPjhh1WoPrRPPvmE884777j3U53CeQbRH1jp7jnuXgRMBsaUX8Hd1wQHfz/sa7OZ9QXSgffDWGNEbdlVyPXPzeGHk7+iXYsGNKufyJQlW46+oUgdN27cOCZPnnzIvMmTJzNu3LhKbf/OO+/QtGnTY3rvigFx7733MnLkyGPaV20XzoBow6EDwOdy6ADt3yo4pu6fCIwMdqT1JpjZbDObnZeXd8yF1jR35+XZ6xn14FSmr8jjF+dk8epNpzK8WzqfLMujRM1MIkc0duxY3n77bYqKigBYs2YNGzduZPDgwdx0001kZ2dz0kkncc89FUeNDWjfvj35+YHB/O677z4yMzM5/fTTWbZs2cF1/v73v9OvXz969erFxRdfzN69e5k5cyZvvPEGd9xxB71792bVqlWMHz+eV155BYApU6Zwyimn0KNHD6677jr2799/8P3uuece+vTpQ48ePVi6dOnhRX2LF198kR49enDyySfzs5/9DIDS0lLGjx/PySefTI8ePfjzn/8MwMMPP0z37t3p2bMnl19+eRU/1cPV1ttc/wd4x91zj3Rhxd0nAhMBsrOzo2JovA079vHz1xYwdXke/ds35w9je9KhZQMARmal8ercXOas3c6Aji0iXKlI5fz6zUUs3rirWvfZvXVj7jn/pG9d3rx5c/r378+7777LmDFjmDx5Mpdeeilmxn333Ufz5s0pLS1lxIgRzJ8/n549e4bcz5w5c5g8eTJfffUVJSUl9OnTh759+wJw0UUXcf311wPwy1/+kieffJJbbrmF0aNHc9555zF27KGXTQsLCxk/fjxTpkwhMzOTq6++mkcffZQf/ehHALRs2ZK5c+fyt7/9jQceeIAnnnjiqJ/Dxo0b+dnPfsacOXNo1qwZ3/nOd3j99ddp27YtGzZsYOHChQAHm8vuv/9+Vq9eTXJycsgmtKoK5xnEBgJj5B6QEZxXGYOAm81sDfAAcLWZ3V+95dUsd2fS52s588/TmLXma349+iQmTxh4MBwABmemkhhvfLR0awQrFYkO5ZuZyjcvvfTSS/Tp04dTTjmFRYsWHdIcVNH06dO58MILqV+/Po0bN2b06NEHly1cuJDBgwfTo0cPJk2axKJFi45Yz7Jly+jQoQOZmZkAXHPNNUyb9s0l1IsuugiAvn37smbNmkr9jbNmzeKMM84gNTWVhIQErrjiCqZNm0bHjh3Jycnhlltu4T//+Q+NGzcGoGfPnlxxxRU8//zzJCQc//f/cJ5BzAK6mFkHAsFwOYGB5I/K3Q8Mpo6ZjQey3f2wu6Cixbpte7nztfnMXLWNUzu14A8X96Rt8/qHrdcwOYGBHVvw4ZIt3HVOVgQqFam6I33TD6cxY8Zw2223MXfuXPbu3Uvfvn1ZvXo1DzzwALNmzaJZs2aMHz/+mHt7jx8/ntdff51evXrxzDPP8MknnxxXvcnJyQDEx8dTUlJyXPtq1qwZ8+bN47333uOxxx7jpZde4qmnnuLtt99m2rRpvPnmm9x3330sWLDguIIibGcQ7l4C3Ay8BywBXnL3RWZ2r5mNBjCzfmaWC1wCPG5mR47oKFNW5jzz39Wc+dA05ufu5HcX9mDS9weEDIcDRnRLY1XeHtbk76nBSkWiT8OGDRk2bBjXXXfdwbOHXbt20aBBA5o0acKWLVt49913j7iPIUOG8Prrr7Nv3z4KCgp48803Dy4rKCigVatWFBcXM2nSpIPzGzVqREFBwWH76tq1K2vWrGHlypUA/OMf/2Do0KHH9Tf279+fqVOnkp+fT2lpKS+++CJDhw4lPz+fsrIyLr74Yn77298yd+5cysrKWL9+PcOGDeMPf/gDO3fuZPfu3cf1/mG9BuHu7wDvVJh3d7nXswg0PR1pH88Az4ShvLBanb+Hn74yj1lrtjM0M5XfX9SD1k3rHXW7EVnp/OrNxUxZupXvnV75LvEiddG4ceO48MILDzY19erVi1NOOYVu3brRtm1bTjvttCNu36dPHy677DJ69epFWloa/fr1O7jsN7/5DQMGDCA1NZUBAwYcDIXLL7+c66+/nocffvjgxWkIPOvo6aef5pJLLqGkpIR+/fpx4403VunvmTJlChkZ3xwSX375Ze6//36GDRuGu3PuuecyZswY5s2bx7XXXktZWeCGlt///veUlpZy5ZVXsnPnTtydW2+99Zjv1DrA3KPi2u5RZWdne20YMKi0zHlqxmoeeH8ZyQlx3H3+SVzcp02VejF+589TadkwmReuHxjGSkWO3ZIlS8jKUjNotAn1383M5rh7dqj1a+tdTFFpxZYC7nhlPl+t38HIrHTuu/Bk0hunVHk/w7ul88T0HHYVFtM4pfJPXhQRqU56FlM1KC4t45GPV3LuwzNYu20Pf7m8N3+/uu8xhQMEbnctKXOmLY+evh0iEnt0BnGcFm/cxU9fncfCDbs4t0crfj3mJFo2TD6ufZ5yYjOa1U/koyVbOa9n62qqVKR6ubse2BdFjuVyggLiGBWVBM4aHvl4JU3rJ/LoFX04u0eratl3fJwxrGsaHy/bSmmZEx+nf4RSu6SkpLBt2zY98jtKHBgPIiWlaq0aCohjsCB3J3e8Mo+lmwu48JQ23H1ed5o1SKrW9xiRlc5rX27gy3XbyW7fvFr3LXK8MjIyyM3NJZoecVPXHRhRrioUEFVQWFzKX6asYOK0HFo2TOLJa7IZkZUelvcanNmShDjjwyVbFRBS6yQmJlZpZDKJTrpIXUlz123nvP+bwaOfrOLiPm14/7ahYQsHgMYpiQzo2FxPdxWRiFFAHMW+olJ++9ZiLn50JvuKSnnuuv7879heNKkX/ttPh3dLZ8XW3azbtjfs7yUiUpEC4gg+z9nG2X+ZxhMzVnPFgBP5z48GMyQztcbef2RWGgBTluosQkRqngIihD37S7jn3wu5bOJnlDm8cP0AfntBDxrVcKe1di0a0Cm1gZ7uKiIRoYvUFcxYkc+dr81nw459XHtae+44syv1kyL3MY3MSuep/66moLC4xgNKROo2nUEE7Sos5q7X5nPlk5+TFB/HyzcM4p7zT4poOEDgdtfiUmfGivyI1iEidY/OIICPl23l568tYMuuQm4Y2pHbRmaSkhgf6bIA6HNiU5rUS+TDJVurrSOeiEhl1PmAWJW3m+uemUWXtIY8euVp9G7bNNIlHSIhPo5hXVPVq1pEalydb2LqlNqQJ67O5s1bTq914XDA8Kx0vt5TxFfrd0S6FBGpQ+p8QECgnT85oXY0KYUyNDOV+DhTpzkRqVEKiCjQpF4i/do30+2uIlKjFBBRYmRWOks3F5C7Xb2qRaRmKCCixIHnPuksQkRqigIiSnRo2YCOLRvw4RIFhIjUDAVEFBmRlcZnq7axe39JpEsRkTpAARFFhndLp6i0TL2qRaRGKCCiSHb7ZjROSdDtriJSIxQQUSQxPo6hwbGqy8qqPgC5iEhVKCCizMisNPJ3FzEvd0ekSxGRGKeAiDIHelXrdlcRCTcFRJRpWj+Jvu2a6XZXEQm7sAaEmZ1lZsvMbKWZ3Rli+RAzm2tmJWY2ttz8dsH5X5nZIjO7MZx1RpuRWWks2bSLjTv2RboUEYlhYQsIM4sHHgHOBroD48yse4XV1gHjgRcqzN8EDHL33sAA4E4zax2uWqPN8G6BXtVT1MwkImEUzjOI/sBKd89x9yJgMjCm/Aruvsbd5wNlFeYXufv+4GRymOuMOp1SG9C+RX3d7ioiYRXOA28bYH256dzgvEoxs7ZmNj+4jz+4+8YQ60wws9lmNjsvL++4C44WZsbwbunMXLWNvUXqVS0i4VFrv5m7+3p37wl0Bq4xs/QQ60x092x3z05NTa35IiNoZFYaRSXqVS0i4RPOgNgAtC03nRGcVyXBM4eFwOBqqismZLdvTqPkBN3uKiJhE86AmAV0MbMOZpYEXA68UZkNzSzDzOoFXzcDTgeWha3SKJSUEMeQrqlMWape1SISHmELCHcvAW4G3gOWAC+5+yIzu9fMRgOYWT8zywUuAR43s0XBzbOAz81sHjAVeMDdF4Sr1mg1MiuNvIL9LNy4M9KliEgMSgjnzt39HeCdCvPuLvd6FoGmp4rbfQD0DGdtseCMzDTiDD5cspWeGU0jXY6IxJhae5Fajq5Zg0Cvat3uKiLhoICIcsO7pbNo4y427yyMdCkiEmMUEFFuZFYaAFOW6ixCRKqXAiLKdU5rSNvm9fhID+8TkWqmgIhyZsaIbunMWJnPvqLSSJcjIjFEAREDRmals7+kjJmr1KtaRKqPAiIG9O/QnIbJCRojQkSqlQIiBiQlxDEksyUfLd2Cu3pVi0j1UEDEiOHd0tmyaz+LNu6KdCkiEiMUEDFiWNdUzOBDdZoTkWqigIgRLRomc0rbpnq6q4hUGwVEDBmRlc783J1s2aVe1SJy/BQQMWRkVmBMpY91FiEi1UABEUMy0xvSpmk93e4qItVCARFDzIyRWWnMWJlHYbF6VYvI8VFAxJjhWekUFpfx6aptkS5FRKKcAiLGDOzYnPpJ8brdVUSOmwIixiQnxDO4S0s+WrpVvapF5LgoIGLQiKx0Nu0sZPEm9aoWkWOngIhBw7qmYYbGiBCR46KAiEGpjZLpldGUD9UfQkSOgwIiRo3MSmPe+h1sLVCvahE5NgqIGDW8W6BX9SdL8yJciYhEKwVEjMpq1YjWTVJ0u6uIHDMFRIwyM4ZnpTFjZb56VYvIMVFAxLARWensLSrlsxz1qhaRqlNAxLBBHVtQLzFeY0SIyDFRQMSwlMR4Tu/SkilL1KtaRKourAFhZmeZ2TIzW2lmd4ZYPsTM5ppZiZmNLTe/t5l9amaLzGy+mV0Wzjpj2cisNDbs2MfSzQWRLkVEokzYAsLM4oFHgLOB7sA4M+teYbV1wHjghQrz9wJXu/tJwFnAQ2bWNFy1xrJhXdMA1MwkIlUWzjOI/sBKd89x9yJgMjCm/Aruvsbd5wNlFeYvd/cVwdcbga1AahhrjVlpjVPoldFEt7uKSJWFMyDaAOvLTecG51WJmfUHkoBVIZZNMLPZZjY7L08dwr7N8G7pfLV+B/m790e6FBGJIrX6IrWZtQL+AVzr7mUVl7v7RHfPdvfs1FSdYHybEVlpuGusahGpmnAGxAagbbnpjOC8SjGzxsDbwC/c/bNqrq1OOal1Y05onKLrECJSJeEMiFlAFzPrYGZJwOXAG5XZMLj+v4Dn3P2VMNZYJxzoVT1teR77S9SrWkQqJ2wB4e4lwM3Ae8AS4CV3X2Rm95rZaAAz62dmucAlwONmtii4+aXAEGC8mX0V/OkdrlrrghHd0thTVMrnOV9HuhQRiRIJ4dy5u78DvFNh3t3lXs8i0PRUcbvngefDWVtdc1rnlqQkxvHR0q0MydT1GhE5ulp9kVqqT0piPKd3bsmHS7aoV7WIVIoCog4Z3i2d3O37WLF1d6RLEZEooICoQ4Z3C/SqVqc5EakMBUQdckKTFE5u05iPluh2VxE5OgVEHTOiWzpz123n6z1FkS5FRGo5BUQdMyIrjTL1qhaRSlBA1DEnt25CWqNk9aoWkaNSQNQxcXHGiKw0pi7Po6jksMdbiYgcpICog4Z3S2f3/hJmrVGvahH5dpUKCDNrYGZxwdeZZjbazBLDW5qEy+mdW5KcEKfbXUXkiCp7BjENSDGzNsD7wFXAM+EqSsKrXlI8p3ZqobGqReSIKhsQ5u57gYuAv7n7JcBJ4StLwm1EVjrrvt7Lqjz1qhaR0CodEGY2CLiCwBgNAPHhKUlqwje9qnU3k4iEVtmA+BFwF/Cv4CO7OwIfh60qCbvWTevRvZV6VYvIt6tUQLj7VHcf7e5/CF6sznf3W8Ncm4TZyKw0Zq/9mu3qVS0iIVT2LqYXzKyxmTUAFgKLzeyO8JYm4TY8K50yh6nL8yJdiojUQpVtYuru7ruAC4B3gQ4E7mSSKNazTRNaNkzW7a4iElJlAyIx2O/hAuANdy8GdH9klIuLM4Z3S2Xq8jyKS9WrWkQOVdmAeBxYAzQApplZO2BXuIqSmjMiK52CQvWqFpHDVfYi9cPu3sbdz/GAtcCwMNcmNeD0zi1Jio9jiu5mEpEKKnuRuomZPWhms4M/fyJwNiFRrkFyAoM6tdDTXUXkMJVtYnoKKAAuDf7sAp4OV1FSs0ZkpbE6f496VYvIISobEJ3c/R53zwn+/BroGM7CpOYc6FWtTnMiUl5lA2KfmZ1+YMLMTgP2hackqWkZzerT7YRGut1VRA6RUMn1bgSeM7MmwentwDXhKUkiYURWGo9NzWHn3mKa1NeT3EWk8ncxzXP3XkBPoKe7nwIMD2tlUqNGZKVTWuZ8slzNTCISUKUR5dx9V7BHNcDtYahHIqRXRlNaNEjS7a4ictDxDDlq1VaFRFx8nDGsWxqfLNtKiXpViwjHFxBHfdSGmZ1lZsvMbKWZ3Rli+RAzm2tmJWY2tsKy/5jZDjN76zhqlCoY0S2NXYUlzF67PdKlUFRSRmmZnuYiEklHvEhtZgWEDgID6h1l23jgEWAUkAvMMrM33H1xudXWAeOBn4TYxR+B+sANR3ofqT6DM1NJio/jo6VbGdixRaW3Ky4tY8/+EvYUlbI3+HvP/hL27C9hb1Epe4pK2Lu/lN37S9hb9M16u/eXHjJ9cB9FJRSXOi0aJHHj0E5cObAd9ZI0PpVITTtiQLh7o+PYd39gpbvnAJjZZGAMcDAg3H1NcNlhbRruPsXMzjiO95cqapicwICOzXl7/ibqJ8UHDu7Bg/zBg/uBg/r+bw78RVVokqqfFE/9pAQaJAd/J8XTtF4ibZqmHJxukJxA/aR4Psv5mvveWcLE6Tn84IxOXN7/RFISFRQiNaWyt7keizbA+nLTucCA6nwDM5sATAA48cQTq3PXddaY3m34ycvzeOjDFdRLjD94IK+fFE/D5AQa10ukVZPAwbxhcjz1kwMH9QMH/QbJCTQIrt8gOSE4HVivXmI88XGVv3R183D4LGcbD36wnF+9uZjHp+Xwg2GduTS7LUkJx9M6KiKVEc6ACDt3nwhMBMjOzlaDdTUY2zeDc3u0IikhrkoH83AZ2LEF/5wwkJmrtvGn95fxy9cX8ugnq7h1RGcu6pNBYryCQiRcwvmvawPQttx0RnCe1HL1kqr2TT/czIzTOrfk1ZtO5Zlr+9GiYRI/e3UBIx+cyqtzcnUxWyRMwhkQs4AuZtbBzJKAy4E3wvh+EuPMjDO6pvHvH5zGE1dn0yApgR+/PI9Rf57KG/M2UqagEKlWYQsIdy8BbgbeA5YAL7n7IjO718xGA5hZPzPLBS4BHjezRQe2N7PpwMvACDPLNbMzw1WrRBczY2T3dN665XQeu7IPiXFx3Pril5z1l2m8u2CTgkKkmph7bPxjys7O9tmzZ0e6DImAsjLn7QWbeOjD5azK20NWq8bcPiqTkVlpmNWepjKR2sjM5rh7dqhlusInUS8uzji/V2vev20oD17ai71FJVz/3GzGPPJfPl62lVj5EiRS03QGITGnuLSMf83dwF+mrGDDjn30ObEpt4/qymmdW+iMQqSCI51BKCAkZhWVlPHynPX89aOVbNpZSP8OzfnxqEwGVKGXuEisU0BInVZYXMo/Z63nkY9XsrVgP6d1bsHto7rSt12zSJcmEnEKCBECQfH8Z2t5bOoq8ncXMTQzldtHZdKrbdNIlyYSMQoIkXL2FpXw7My1PD5tFTv2FjMyK43bRmVyUusmR99YJMYoIERCKCgs5pn/ruHv03PYVVjCWSedwG2jMul6wvE8o1IkuiggRI5g575inpyxmqdmrGZPUQnn9WzND0d0oXNaw0iXJhJ2CgiRStixt4iJ03J4ZuYaCotLuaB3G24d0YX2LRtEujSRsFFAiFTBtt37eXxaDs99uobiUmdM79ZMGNKRbic0jnRpItVOASFyDLYWFPLYJzm8+MU69hWXckbXVCYM6cigjupwJ7FDASFyHLbvKeL5z9by7KdryN9dRI82TZgwpCNnn3wCCRqPQqKcAkKkGhQWl/La3A08MT2HnPw9ZDSrx/dO78Bl/dpSPymqx96SOkwBIVKNysqcD5ZsYeK0HOas3U6TeolcNbAd15zantRGyZEuT6RKFBAiYTJn7ddMnJbD+4u3kBgfx8V92vD9wR3plKpbZCU6KCBEwiwnbzdPzFjNK3NyKS4tY2RWOjcM6Uh2++aRLk3kiBQQIjUkf/d+npu5huc+W8uOvcX0ObEpE4Z0YlT39Fo1zrfIAQoIkRq2t6iEl2fn8sSMHNZ/vY8OLRvw/cEduLhPBimJ8ZEuT+QgBYRIhJSUlvGfRZuZOC2H+bk7adEgiWtObc9VA9vRrEFSpMsTUUCIRJq781nO10yctoqPl+VRLzGeS7Mz+P7gjrRtXj/S5UkddqSA0M3bIjXAzBjUqQWDOrVg+ZYCJk7L4YUv1vGPz9Zydo9W3DCkIz0zmka6TJFD6AxCJEI27yzkmZlrmPT5WgoKSxjYsTk3DOnEGV1T9SgPqTFqYhKpxQoKi/nnrPU8OWM1m3YWkpnekOsHd2RM7zYkJehRHhJeCgiRKFBcWsZb8zfy+NQclm4uIL1xMtee1oHvDjiRximJkS5PYpQCQiSKuDvTV+QzcVoOM1bm0zA5gcv7teXm4Z1pWl93Pkn10kVqkShiZgzJTGVIZioLN+zk79NzeHrmGlZs3c0z1/bT9QmpMWrgFKnFTm7ThL9cfgq/PDeLqcvzeHP+pkiXJHWIAkIkClw9qD29Mppw75uL2LG3KNLlSB0R1oAws7PMbJmZrTSzO0MsH2Jmc82sxMzGVlh2jZmtCP5cE846RWq7+Djj9xf1ZPveYu5/d2mky5E6ImwBYWbxwCPA2UB3YJyZda+w2jpgPPBChW2bA/cAA4D+wD1m1ixctYpEg+6tG/P9wR2YPGs9n+Vsi3Q5UgeE8wyiP7DS3XPcvQiYDIwpv4K7r3H3+UBZhW3PBD5w96/dfTvwAXBWGGsViQo/GpFJ2+b1+Pm/FlBYXBrpciTGhTMg2gDry03nBueFe1uRmFUvKZ77LuhBTt4e/vbJqkiXIzEuqi9Sm9kEM5ttZrPz8vIiXY5IjRiSmcoFvVvz6CcrWbm1INLlSAwLZ0BsANqWm84Izqu2bd19ortnu3t2amrqMRcqEm1+eV53GiQncNdrCygri43OrlL7hDMgZgFdzKyDmSUBlwNvVHLb94DvmFmz4MXp7wTniQjQsmEyvzgni1lrtjN51vqjbyByDMIWEO5eAtxM4MC+BHjJ3ReZ2b1mNhrAzPqZWS5wCfC4mS0Kbvs18BsCITMLuDc4T0SCxvbNYFDHFvz+3SVs3VUY6XIkBulZTCJRLCdvN2f9ZTqjstJ55Io+kS5HotCRnsUU1RepReq6jqkNuXV4Z95esIkpS7ZEuhyJMQoIkSg3YUgnMtMb8v9eX8ie/SWRLkdiiAJCJMolJcTx+4t6sHFnIX96f3mky5EYooAQiQF92zXnyoEn8szM1czP3RHpciRGKCBEYsRPz+pGy4bJ3PnqAkpKKz69RqTqFBAiMaJxSiL3jjmJxZt28dR/V0e6HIkBCgiRGHLmSScwMiudBz9Yzvqv90a6HIlyCgiRGGJm3DvmJOLN+MXrC4mVfk4SGQoIkRjTumk97jizK9OW5/HGvI2RLkeimAJCJAZdNag9vdo25d43F8fcEKXLNhdw12sL+GDxFop1MT6sFBAiMSg+zrj/oh7s2FfM795ZEulyqs2X67Zz6eOf8uIX67j+udkM/N0UfvPWYhZv3BXp0mJSQqQLEJHwyGrVmOsHd+Sxqau48JQMBnVqEemSjsvMVfl8/9nZpDZK5vUfnEZO3m5emZPLc5+u4ckZq+neqjFj+2YwpndrWjRMjnS5MUEP6xOJYfuKSjnzoWkkxBnv/HAwKYnxkS7pmHy4eAv/88Jc2reoz/PfG0Ba45SDy7bvKeLN+Rt5ZU4u83N3khBnDO+Wxti+GQzrlkZivBpKjuRID+tTQIjEuOkr8rjqyS+4dXhnbv9O10iXU2X//moDt780j5NbN+aZa/vTrEHSt667bHMBr87N5bW5G8jfvZ8WDZIY3bs1Y/tmcFLrJjVYdfRQQIjUcbf98yvemr+Rt28dTGZ6o0iXU2mTPl/LL19fSP/2zXlyfD8aJleuVbyktIzpK/J5ZU4uHyzeQlFpGVnlmqBaqgnqIAWESB23bfd+Rjw4lU6pDXn5hkHExVmkSzqqx6au4v53lzK8Wxp/u6LPMTeP7dhbxJvzAk1Q84JNUMO6pXFxnwyGd0sjKaFuN0EpIESEV+bk8pOX53HfhSdzxYB2kS7nW7k7D7y/jEc+XsX5vVrz4KW9qu06wvItBbw6J5fXvtxAXsF+mtVPZEzvNsEmqMaY1f7grG4KCBHB3bniic9ZkLuTD388lPRyF3pri7Iy59dvLuLZT9cyrn9bfntBD+LDcLZTUlrG9JXBJqhFgSaobic0CjZBtSG1Ud1pglJAiAgAq/P3cOZD0xiZlcbfrugb6XIOUVJaxk9fnc9rczcwYUhH7jq7W418o9+xt4g3528KNEGt30F8nDGsaypj+2YwvFt6zDdBKSBE5KBHPl7JH99bxt+vzmZU9/RIlwPA/pJSbn3xS95btIUfj8rk5uGdI9Lcs3JrAa/M2cBrc3PZGmyCGt2rNWP7tuXkNrHZBKWAEJGDikrKOP//ZrCrsJgPbh9a6TuDwmVvUQk3/GMO01fkc8/53bn2tA4RrQcCZzMzgk1Q7y/eQlFJGV3Tg01Qp7QmrVHNNc+VljmFxaXsKy5lX1EphcWl7C0KTheXUlhUSoPkBIZkph7T/hUQInKIOWu3M/axmYw/tT33nH9SxOrYua+Y656ZxZfrtvOHi3tySXbbiNXybXbuLebN+Rt5dW4uX64LNEGdkZnKxX0zOKNrKiVlTmHR4QftfRUO6vvKHdi/WV52yPJ9RRV+F5dSVHL05031atuUf//gtGP6+44UEHrUhkgd1LddM64c0I5nZq5hTO829G7btMZryN+9n6uf/IIVWwt45Lt9OLtHqxqvoTKa1E/kyoHtuHJgO1Zu3R3siJfLlKVbj2l/SQlx1EuMp15iPPWT4klJjKdeUmC6Wf2k4OvAOilJ36xXL/HQdcsvb1wvsZr/6gCdQYjUUbsKixn14FSaN0jmjZtPq9FHUmzcsY8rn/ycjTv28fhV2Qw9xuaRSCktc2aszGfhhp0kJ8QdctA++DrEQT0lMT4sd2UdD51BiMhhGqck8uvRJ3Pj83N4csZqbhzaqUbed3X+Hq584nN27SvmH98bQL/2zWvkfatTfJwxNDM16oKtqmL7/i0ROaKzTj6BUd3TeejD5azbFv4hSpdu3sUlj33KvuJSXpwwMCrDoS5RQIjUcd8MUbogrEOUfrluO5c9/hkJccZLNwzk5DZ6eF5tp4AQqeNaNanHT8/qxvQV+fz7q/AMUTpzVT5XPPE5Tesn8vKNg+icFj0PDKzLwhoQZnaWmS0zs5VmdmeI5clm9s/g8s/NrH1wfpKZPW1mC8xsnpmdEc46Req6Kwe2o3fbptz71mK276neIUo/XLyF8U/PIqNZPV6+YRBtm9ev1v1L+IQtIMwsHngEOBvoDowzs+4VVvsesN3dOwN/Bv4QnH89gLv3AEYBfzIzne2IhEl8nPH7i3qwq5qHKP33Vxu44fk5ZJ3QiH9OGHTIQD9S+4XzoNsfWOnuOe5eBEwGxlRYZwzwbPD1K8AIC/Rl7w58BODuW4EdQMjbsESkemS1asz1Qzry8pxcZq7MP+79Tfp8LT/651dkt2vGpOsHHnGgH6mdwhkQbYD15aZzg/NCruPuJcBOoAUwDxhtZglm1gHoC9S+LpYiMeaHI7rQrkV9fv6vBRQWlx7zfh6buopf/Gshw7qm8ex1/SP+OA85NrW12eYpAoEyG3gImAkc9n+rmU0ws9lmNjsvL69mKxSJQSmJ8dx3QQ/WbNvLXz9aWeXt3Z0/vreU+99dyvm9WvP4VX2jdhxsCW9AbODQb/0ZwXkh1zGzBKAJsM3dS9z9Nnfv7e5jgKbA8opv4O4T3T3b3bNTU2O7w4pITTm9S0suOqUNj01dxbLNBZXerqzM+dUbi3jk41WM69+Why7rXaO9s6X6hfO/3iygi5l1MLMk4HLgjQrrvAFcE3w9FvjI3d3M6ptZAwAzGwWUuPviMNYqIuX84twsGqUkcNdr8ykrO3rfiJLSMn7yyjye/XQtE4Z05HcXhmegH6lZYQuI4DWFm4H3gCXAS+6+yMzuNbPRwdWeBFqY2UrgduDArbBpwFwzWwL8DLgqXHWKyOFaNEzml+d2Z+66HUz6Yt0R191fUsoPXpjLa3M38ONRmTU20I+Enx7WJyIhuTtXPvk589fv5IPbh3JCk8NvUa2NYzlI1RzpYX1qIBSRkMyM+y7oQVFpGb96Y9Fhy3fuK+aqJ7/gvyvz+ePYngqHGKSAEJFv1b5lA24d0YX/LNrM+4s2H5yfv3s/4yZ+xvzcHTzy3T61cqAfOX4KCBE5oglDOtI1vRF3/3sRBYXFbNyxj0sf/5Sc/N08cU2/WjvQjxw/BYSIHFFifBy/v7gHWwoKueu1BVzy2Kfk7drPP743IObHQ6jrFBAiclR9TmzGVQPb8db8TRrLoQ5R/3cRqZQ7zuxKSmI8l2a3pXNaw0iXIzVAASEildIoJZGfn5MV6TKkBqmJSUREQlJAiIhISAoIEREJSQEhIiIhKSBERCQkBYSIiISkgBARkZAUECIiElLMjAdhZnnA2uPYRUsgv5rKiXb6LA6lz+NQ+jy+EQufRTt3D/lQrZgJiONlZrO/bdCMukafxaH0eRxKn8c3Yv2zUBOTiIiEpIAQEZGQFBDfmBjpAmoRfRaH0udxKH0e34jpz0LXIEREJCSdQYiISEgKCBERCanOB4SZnWVmy8xspZndGel6IsnM2prZx2a22MwWmdkPI11TpJlZvJl9aWZvRbqWSDOzpmb2ipktNbMlZjYo0jVFkpndFvx3stDMXjSzlEjXVN3qdECYWTzwCHA20B0YZ2bdI1tVRJUAP3b37sBA4Ad1/PMA+CGwJNJF1BJ/Af7j7t2AXtThz8XM2gC3AtnufjIQD1we2aqqX50OCKA/sNLdc9y9CJgMjIlwTRHj7pvcfW7wdQGBA0CbyFYVOWaWAZwLPBHpWiLNzJoAQ4AnAdy9yN13RLSoyEsA6plZAlAf2BjheqpdXQ+INsD6ctO51OEDYnlm1h44Bfg8wqVE0kPAT4GyCNdRG3QA8oCng01uT5hZg0gXFSnuvgF4AFgHbAJ2uvv7ka2q+tX1gJAQzKwh8CrwI3ffFel6IsHMzgO2uvucSNdSSyQAfYBH3f0UYA9QZ6/ZmVkzAq0NHYDWQAMzuzKyVVW/uh4QG4C25aYzgvPqLDNLJBAOk9z9tUjXE0GnAaPNbA2BpsfhZvZ8ZEuKqFwg190PnFG+QiAw6qqRwGp3z3P3YuA14NQI11Tt6npAzAK6mFkHM0sicJHpjQjXFDFmZgTamJe4+4ORrieS3P0ud89w9/YE/r/4yN1j7htiZbn7ZmC9mXUNzhoBLI5gSZG2DhhoZvWD/25GEIMX7RMiXUAkuXuJmd0MvEfgLoSn3H1RhMuKpNOAq4AFZvZVcN7P3f2dyJUktcgtwKTgl6kc4NoI1xMx7v65mb0CzCVw99+XxOBjN/SoDRERCamuNzGJiMi3UECIiEhICggREQlJASEiIiEpIEREJCQFhEgVmFmpmX1V7qfaehObWXszW1hd+xM5XnW6H4TIMdjn7r0jXYRITdAZhEg1MLM1Zva/ZrbAzL4ws87B+e3N7CMzm29mU8zsxOD8dDP7l5nNC/4ceExDvJn9PTjOwPtmVi9if5TUeQoIkaqpV6GJ6bJyy3a6ew/grwSeBAvwf8Cz7t4TmAQ8HJz/MDDV3XsReKbRgR78XYBH3P0kYAdwcVj/GpEjUE9qkSows93u3jDE/DXAcHfPCT7wcLO7tzCzfKCVuxcH529y95ZmlgdkuPv+cvtoD3zg7l2C0z8DEt39tzXwp4kcRmcQItXHv+V1Vewv97oUXSeUCFJAiFSfy8r9/jT4eibfDEV5BTA9+HoKcBMcHPe6SU0VKVJZ+nYiUjX1yj3pFgJjNB+41bWZmc0ncBYwLjjvFgKjsN1BYES2A09A/SEw0cy+R+BM4SYCI5OJ1Bq6BiFSDYLXILLdPT/StYhUFzUxiYhISDqDEBGRkHQGISIiISkgREQkJAWEiIiEpIAQEZGQFBAiIhLS/wdopnGY+cyXMgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "## YOUR CODE HERE ##\n",
    "# Plotting training loss\n",
    "plt.plot(train_loss_history, label='Training Loss')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Loss')\n",
    "plt.title('Training Loss')\n",
    "plt.legend()\n",
    "plt.show()\n",
    "\n",
    "# validation loss is recorded\n",
    "plt.plot(val_loss_history, label='Validation Loss')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Loss')\n",
    "plt.title('Validation Loss')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing your model\n",
    "Using the previously created `DataLoader` for the test set, compute the percentage of correct predictions using the highest probability prediction. \n",
    "\n",
    "If your accuracy is over 90%, great work, but see if you can push a bit further! \n",
    "If your accuracy is under 90%, you'll need to make improvements.\n",
    "Go back and check your model architecture, loss function, and optimizer to make sure they're appropriate for an image classification task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on the test set: 97.42%\n"
     ]
    }
   ],
   "source": [
    "## YOUR CODE HERE ##\n",
    "correct = 0\n",
    "total = 0\n",
    "\n",
    "# Set the model to evaluation mode\n",
    "net.eval()\n",
    "\n",
    "# Iterate over the test dataset\n",
    "for images, labels in test_loader:\n",
    "    images, labels = images.to(device), labels.to(device)\n",
    "    \n",
    "    # Forward pass\n",
    "    outputs = net(images)\n",
    "    \n",
    "    # Get predictions\n",
    "    _, predicted = torch.max(outputs.data, 1)\n",
    "    \n",
    "    # Update total number of samples\n",
    "    total += labels.size(0)\n",
    "    \n",
    "    # Update number of correct predictions\n",
    "    correct += (predicted == labels).sum().item()\n",
    "\n",
    "# Compute accuracy\n",
    "accuracy = 100 * correct / total\n",
    "\n",
    "print(f'Accuracy on the test set: {accuracy:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Improving your model\n",
    "\n",
    "Once your model is done training, try tweaking your hyperparameters and training again below to improve your accuracy on the test set!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saving your model\n",
    "Using `torch.save`, save your model for future loading."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model saved to trained_model.pth\n"
     ]
    }
   ],
   "source": [
    "## YOUR CODE HERE ##\n",
    "# Define the file path where you want to save the model\n",
    "model_path = 'trained_model.pth'\n",
    "\n",
    "# Save the model\n",
    "torch.save(net.state_dict(), model_path)\n",
    "\n",
    "print(f'Model saved to {model_path}')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}