Learning_visualisations_v3.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyPZW/BFlp0Z27/j1ZrUatnS",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/malcolmlett/ml-learning/blob/main/Learning_visualisations_v3.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Learning Visualisations v3: Weight visualisation during learning\n",
        "In this notebook I'm going to explore options for visualising the weight changes across the network as it learns. There's a number of different interesting things that we focus on for the duration of training:\n",
        "* how fast/slow the weights are getting updates?\n",
        "* what proportion of weights are getting updated?\n",
        "* is learning moving in a consistent direction or is it oscillating?\n",
        "* what's the learning rate that ADAM is currently using?\n",
        "\n",
        "For the first experiments, I'm going to focus on the issue of whether learning is oscillationg, by examining the sign of individual weight changes."
      ],
      "metadata": {
        "id": "Vi364qffAPOm"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "😧\n",
        "**This notebook is incomplete and needs to be started up again now that I know how to make the changes in a way that is compliant with TensorFlow.**"
      ],
      "metadata": {
        "id": "H-2QEiYLnkxT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import tensorflow as tf\n",
        "import math\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import time"
      ],
      "metadata": {
        "id": "62_JrV43Ex4K"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# First, some basic helper methods\n",
        "def plot_fit(x, y, model, history = None):\n",
        "  y_predict = model.predict(x)\n",
        "\n",
        "  pnum = 1\n",
        "  if history != None:\n",
        "    pnum += 1\n",
        "\n",
        "  plt.figure(layout='constrained')\n",
        "  plt.subplot(1,pnum,1)\n",
        "  plt.plot(x, y, color='blue', label=\"Truth\")\n",
        "  plt.plot(x, y_predict, color='orange',label=\"Prediction\")\n",
        "  plt.legend()\n",
        "  if history != None:\n",
        "    plt.subplot(1,pnum,2)\n",
        "    plt.plot(history.history['loss'])\n",
        "    plt.title('Loss')\n",
        "    plt.xlabel('Epoch')\n",
        "    plt.yscale('log')\n",
        "\n",
        "class LessVerboseEpochLogger(tf.keras.callbacks.Callback):\n",
        "    def __init__(self, display_interval=10):\n",
        "        super(LessVerboseEpochLogger, self).__init__()\n",
        "        self.display_interval = display_interval\n",
        "\n",
        "    def on_epoch_end(self, epoch, logs=None):\n",
        "        if (epoch + 1) % self.display_interval == 0:\n",
        "            print(f'Epoch {epoch + 1:5d}:', end=' ')\n",
        "            for key, value in logs.items():\n",
        "                print(f'{key}: {value:.4f}', end='  ')\n",
        "            print()"
      ],
      "metadata": {
        "id": "RkhbRwIsF1TY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Simple Curve\n",
        "As a first exercise, let's simply fit against a very simple curve. This is useful because it's easier to force it into oscillation so we can compare good and bad training regimes.\n",
        "\n"
      ],
      "metadata": {
        "id": "1kSuYBrbFUN6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Define curve to fit\n",
        "x = np.linspace(0.0, 100.0, 1000)\n",
        "#y = np.sin(x/100.0 * np.pi * 5)\n",
        "y = (x-50)**2 + np.random.normal(size=(1000,))\n",
        "plt.plot(x,y)"
      ],
      "metadata": {
        "id": "lNavAreIFsUW",
        "outputId": "f7323caf-5eb6-4421-86a5-3e4825623759",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 447
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7a35699b6380>]"
            ]
          },
          "metadata": {},
          "execution_count": 3
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjAAAAGdCAYAAAAMm0nCAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABTWUlEQVR4nO3deVyUdeIH8M8czHAPAsKAHOIF4omoiJZpkXimaZeZWrm5btgvsy1rD7djW7u2bWsrt+2w3bTUMvNIjVAxEy8UVBQUBTmHU2a4j5nn98fAFKUJCHxnhs/79ZrXK2Ye4MOTMh+f53vIJEmSQERERGRD5KIDEBEREbUXCwwRERHZHBYYIiIisjksMERERGRzWGCIiIjI5rDAEBERkc1hgSEiIiKbwwJDRERENkcpOkBXMZlMKCgogJubG2Qymeg4RERE1AaSJKGyshL+/v6Qy699ncVuC0xBQQECAwNFxyAiIqIOyM3NRUBAwDVft9sC4+bmBsB8Atzd3QWnISIiorYwGAwIDAy0vI9fi90WmJbbRu7u7iwwRERENuZ6wz84iJeIiIhsDgsMERER2RwWGCIiIrI5LDBERERkc1hgiIiIyOawwBAREZHNYYEhIiIim8MCQ0RERDaHBYaIiIhsTrsKzJo1azBmzBi4ubnBx8cHc+bMQUZGRqtjJk2aBJlM1uqxbNmyVsfk5ORgxowZcHZ2ho+PD5566ik0NTW1Omb//v0YNWoU1Go1BgwYgHXr1nXsJyQiIiK7064Ck5iYiLi4OBw+fBjx8fFobGzElClTUF1d3eq4Rx55BIWFhZbHq6++annNaDRixowZaGhowKFDh/DJJ59g3bp1WL16teWYrKwszJgxA5MnT0ZKSgpWrFiB3/zmN9izZ88N/rhERERkD2SSJEkd/eSSkhL4+PggMTEREydOBGC+AjNy5Ei8+eabV/2cXbt2YebMmSgoKICvry8AYO3atVi1ahVKSkqgUqmwatUq7Ny5E2fOnLF83n333YeKigrs3r27TdkMBgM0Gg30ej33QiIiIrIRbX3/vqExMHq9HgDg6enZ6vn169fD29sbQ4cOxbPPPouamhrLa0lJSRg2bJilvABAbGwsDAYD0tLSLMfExMS0+pqxsbFISkq6Zpb6+noYDIZWj65QWlWPjw5moby6oUu+PhERkbX7/GgOnt+ehsziSmEZOrwbtclkwooVKzBhwgQMHTrU8vz999+P4OBg+Pv749SpU1i1ahUyMjKwZcsWAIBOp2tVXgBYPtbpdL96jMFgQG1tLZycnH6RZ82aNXj++ec7+uO02ZJPjiM1twIKuQyLx/ft8u9HRERkTSRJwsc/ZCOjqBL9e7tigI+bkBwdLjBxcXE4c+YMDh482Or5pUuXWv572LBh8PPzw2233YaLFy+if//+HU96Hc8++yxWrlxp+dhgMCAwMLDTv8/sEf5Iza3AlhN5LDBERNTjnC+qQkZRJVRKOWaN8BeWo0O3kJYvX44dO3Zg3759CAgI+NVjo6KiAACZmZkAAK1Wi6KiolbHtHys1Wp/9Rh3d/erXn0BALVaDXd391aPrnDHSH8o5DKk5umFXjojIiIS4WBmKQBgfH8vaJwchOVoV4GRJAnLly/HV199hb179yIkJOS6n5OSkgIA8PPzAwBER0fj9OnTKC4uthwTHx8Pd3d3hIeHW45JSEho9XXi4+MRHR3dnrhdwttVjUmDegMAtpzIF5yGiIio+0iShO2pBQCACf29hWZpV4GJi4vDp59+ig0bNsDNzQ06nQ46nQ61tbUAgIsXL+LFF19EcnIysrOzsW3bNixatAgTJ07E8OHDAQBTpkxBeHg4Fi5ciNTUVOzZswd/+tOfEBcXB7VaDQBYtmwZLl26hKeffhrp6el49913sWnTJjzxxBOd/ON3zNxR5qtOX53Mh8nU4UlcRERENuVcYSVSciugUsgxJ6KP0CztKjDvvfce9Ho9Jk2aBD8/P8tj48aNAACVSoXvvvsOU6ZMQVhYGJ588knMmzcP27dvt3wNhUKBHTt2QKFQIDo6Gg888AAWLVqEF154wXJMSEgIdu7cifj4eIwYMQJ///vf8cEHHyA2NraTfuwbc9tgH7g7KlGor8PR7HLRcYiIiLrFjlPmqy+TQnujt5taaJZ2DeK93pIxgYGBSExMvO7XCQ4OxjfffPOrx0yaNAknT55sT7xu4+igQEy4L7acyMd3Z4swrp+X6EhERERdSpIk7DxdCACYKXDwbgvuhdRBtw82T/OOP1d03WJHRERk687kG3C5rAaODnLcFuYjOg4LTEfdPKg3VAo5LpfVIK2gaxbNIyIishYtt49uC/OFi7rDq7B0GhaYDnJVK3F7uPkqzBfJeYLTEBERdR1JkrDjlPn20YzhfoLTmLHA3IA7m0dg7z6j42wkIiKyWym5FcivqIWzSoHJoeJvHwEsMDfkpoHecFYpoDPU4VS+XnQcIiKiLtFy9SVmsC+cVArBacxYYG6Ao4MCk5sHMu1oXtiHiIjInphMEnY2F5iZVnL7CGCBuWGzm6eSbU0pQJPRJDgNERFR5zqRcwU6Qx3c1EpMbF6J3hqwwNygSaE+8HRRobSqHt9fKBUdh4iIqFO13D66PdwXjg7WcfsIYIG5YSqlHLNHmq/CfHGCs5GIiMh+GE0/XbzOem4fASwwnWJuhHlvpL3nilHXaBSchoiIqHMcyy5HSWU9NE4OuGmA9dw+AlhgOsXQPu7wdVejttGII1ncG4mIiOxDy+J1sUN8oVJaV2WwrjQ2SiaT4dYw86J2CeeKBKchIiK6cU1GE745rQMAzBwufu+jn2OB6SQxg83TqXed0XE2EhER2bwfLpahvLoBXi4qjO9vfZsWs8B0kpsH9kYvZweUVNbj+0zORiIiItu2LcV8+2jaMC2UCuurC9aXyEaZZyOZtxb4knsjERGRDatrNGJPmvn2Uct7m7VhgelE80aZZyN9e7YI+tpGwWmIiIg6Zm96Marqm9DHwwmRQb1Ex7kqFphONLSPOwb5uqKhyWRZdpmIiMjWfJ2SDwCYNcIfcrlMcJqrY4HpRDKZzHIV5ksuakdERDZIX9uIfeklAGBZqNUascB0sjkR5nuFyZevoLiyTnAaIiKi9tlzRocGowmDfF0RpnUTHeeaWGA6ma+7I4b4uwMAvj/P2UhERGRbvk413z6aPbIPZDLrvH0EsMB0iZjB5kXtPknKFhuEiIioHYoNdTh0sQwAcMcI6719BLDAdIlF0cGQy4BTeXoUVNSKjkNERNQm208VQpKAUUEeCPR0Fh3nV7HAdAEvVzUimqed7T6jE5yGiIiobbal/Hj7yNqxwHSRlktvm47nQpIkwWmIiIh+XVZpNVLz9FDIZZg+zE90nOtigekic0b2gUopR7quEmkFBtFxiIiIflXL1gETBnijt5tacJrrY4HpIhpnB9waat7gMf4sd6gmIiLrJUnSj7OPrHzwbgsWmC4UE26ejfTN6ULeRiIiIquVVmDApZJqqJVyTBniKzpOm7DAdKEpQ3yhUspxobiKt5GIiMhqtWwdEDPYF26ODoLTtA0LTBdyd3TAlOarMNxagIiIrJHRJGF7qnn/vjuseOuAn2OB6WJ3Nm8tsONUIYwm3kYiIiLrknSxDDpDHdwdlZgU2lt0nDZjgeliNw/sDY2TA0oq63HkUpnoOERERK1sOWm+QzBzhD/USoXgNG3HAtPFVEo5pg/TAgC2pRYITkNERPSjmoYmy4KrcyOsf/G6n2KB6Qazmqek7TqjQ32TUXAaIiIis2/TilDTYESQpzMig3uJjtMuLDDdICrEC1p3R+hrG7EvvUR0HCIiIgDA1ubZR3dGWPfO01fDAtMNFHIZZjeP7P7qJGcjERGRePqaRvyQWQrAtmYftWCB6SZ3jjLfW9ybXoyKmgbBaYiIqKf77lwRGo0SQn3d0L+3q+g47cYC003CtO4Y7OeORqPEHaqJiEi4XWfMa79MHaoVnKRjWGC60czh5t09t5/ibCQiIhKnoqYBBy6Ybx/Zws7TV8MC043uGOEPhVyGHzLLkKGrFB2HiIh6qM+O5qKhyYTBfu4Y5Gt7t48AFphuFejpjFsGmVc5TEjnDtVERNT9JEnCF8m5AIAHxwfb3OyjFiww3WxymA8AYHsqd6gmIqLudzpfj4vNO0/b6u0jgAWm280a7ge1Uo5zhQYkX74iOg4REfUwX500r/1ye7jt7Dx9NSww3czDWWVZE2bT8VzBaYiIqCdpMpqwvXlbm7mjbGvrgJ9jgRFg9sgf14ThDtVERNRdvr9QitKqBni5qHDzQNvZefpqWGAEGNPXEx7ODiitasD+jGLRcYiIqIfY3Dx4946R/nBQ2HYFsO30NkqllOOuUQEAgE8PXxachoiIeoIr1Q347qz5H813RwYKTnPjWGAEWTAuGACw/3wJcstrBKchIiJ7ty21AA1GE8L93BHu7y46zg1jgREkxNsFEwZ4QZJ+HBFORETUVVoG794VGSA4SedggRGoZTDvN6cLBSchIiJ7VlZVjxM55qU7bHXvo59jgREoNlwLB4UM6bpKZBZzawEiIuoa21ILYJKAYX008PdwEh2nU7DACKRxdsBNA7wBADtPcYdqIiLqGpuP5wGwn9tHAAuMcDOGmxe125qSz60FiIio06UV6HG20ACVQo47RviLjtNpWGAEmzZUCxeVAlml1TiSVS46DhER2Zkvk80TRWLCfdDLRSU4TedhgRHMRa3EHc1bC/wviWvCEBFR52loMmFrirnA2NPtI4AFxiosiu4LAPjmTCHyK2rFhiEiIruxP6MY5dUN6O2mxkQb3zrg51hgrMBgP3dEhXhCkn6cp09ERHSjNiebB+/OjegDpY1vHfBz7fpp1qxZgzFjxsDNzQ0+Pj6YM2cOMjIyWh1TV1eHuLg4eHl5wdXVFfPmzUNRUVGrY3JycjBjxgw4OzvDx8cHTz31FJqamlods3//fowaNQpqtRoDBgzAunXrOvYT2oiZzQOr9p7j3khERHTjig112Jtufk+xt9tHQDsLTGJiIuLi4nD48GHEx8ejsbERU6ZMQXV1teWYJ554Atu3b8fmzZuRmJiIgoICzJ071/K60WjEjBkz0NDQgEOHDuGTTz7BunXrsHr1assxWVlZmDFjBiZPnoyUlBSsWLECv/nNb7Bnz55O+JGt06RB5kt7xy+Xc2sBIiK6YZuT82A0SRgd3AsDfd1Ex+l0MukG5u6WlJTAx8cHiYmJmDhxIvR6PXr37o0NGzbgrrvuAgCkp6dj8ODBSEpKwrhx47Br1y7MnDkTBQUF8PX1BQCsXbsWq1atQklJCVQqFVatWoWdO3fizJkzlu913333oaKiArt3725TNoPBAI1GA71eD3d329jzYeGHR/D9hVIsjg7G87OHio5DREQ2ymSSMPG1fci7Uou/3z0C82zoCkxb379v6IaYXq8HAHh6egIAkpOT0djYiJiYGMsxYWFhCAoKQlJSEgAgKSkJw4YNs5QXAIiNjYXBYEBaWprlmJ9+jZZjWr6GvVp2S38AwJcn8tHQZBKchoiIbNWx7HLkXamFm6MS04f5iY7TJTpcYEwmE1asWIEJEyZg6FDz1QKdTgeVSgUPD49Wx/r6+kKn01mO+Wl5aXm95bVfO8ZgMKC29uqzdOrr62EwGFo9bE10Py94u6pRVd+Eo1wThoiIOij+rHns6e2DfeGkUghO0zU6XGDi4uJw5swZfP75552Zp8PWrFkDjUZjeQQGBoqO1G5yuQy3h/sA4A7VRETUMfVNRst7SKydbNx4NR0qMMuXL8eOHTuwb98+BAT8eF9Nq9WioaEBFRUVrY4vKiqCVqu1HPPzWUktH1/vGHd3dzg5XX0TqmeffRZ6vd7yyM3N7ciPJtxdkebi9c3pQlTWNQpOQ0REtmbXaR3Kqhvgp3HEbWE+ouN0mXYVGEmSsHz5cnz11VfYu3cvQkJCWr0eGRkJBwcHJCQkWJ7LyMhATk4OoqOjAQDR0dE4ffo0iot/nC4cHx8Pd3d3hIeHW4756ddoOabla1yNWq2Gu7t7q4ctGhXkgf69XVDbaMSOU4Wi4xARkY3ZdNz8D/j5Y4Psbu2Xn2rXTxYXF4dPP/0UGzZsgJubG3Q6HXQ6nWVcikajwZIlS7By5Urs27cPycnJeOihhxAdHY1x48YBAKZMmYLw8HAsXLgQqamp2LNnD/70pz8hLi4OarUaALBs2TJcunQJTz/9NNLT0/Huu+9i06ZNeOKJJzr5x7c+MpkM94w2X4Vp+UNIRETUFvkVtUi6VAYANjXzqCPaVWDee+896PV6TJo0CX5+fpbHxo0bLcf84x//wMyZMzFv3jxMnDgRWq0WW7ZssbyuUCiwY8cOKBQKREdH44EHHsCiRYvwwgsvWI4JCQnBzp07ER8fjxEjRuDvf/87PvjgA8TGxnbCj2z97ozoA4VchpM5FbhYUiU6DhER2YitJ/MhSeZJIX08rj7kwl7c0Dow1swW14H5qYfXHcPe9GI8Oqk/np4aJjoOERFZOUmSEPNGIi6WVOO1u4bj7tG2N5kF6KZ1YKjrtCz7vOVEPowmu+yYRETUiU7l6XGxpBqODnJMs9O1X36KBcZK3TbYBx7ODtAZ6nAws1R0HCIisnJbTpg3bowdooWrWik4TddjgbFSaqUCs5s3eORgXiIi+jV1jT+u/TJ3lH0P3m3BAmPF7hljvn8Zn1aE8uoGwWmIiMhabU8tgKGuCQG9nHDTAG/RcboFC4wVG+KvwdA+7mgwmrgyLxERXdNP135RyGWC03QPFhgrd++YIADAxmM5sNMJY0REdAMKKmpxLPsKZDJgXg+5fQSwwFi9O0b4w9FBjvNFVUjJrRAdh4iIrMw3p82rto8J9oRW4yg4TfdhgbFyGicHTB9qng638RgH8xIR0Y9MJgkbjuYAAGaN9BecpnuxwNiAe5sH825PLUB1fZPgNEREZC2+zyzFpZJquKmVuDOij+g43YoFxgaMDfFEiLcLqhuM2MkNHomIqNnHP2QBAO4eHdgj1n75KRYYG/DTDR4/P5YjOA0REVkDnb4OiedLAACLxwcLTtP9WGBsxLxI8waPJ3IqcL6oUnQcIiISbOfpQkgSMDq4F4K9XETH6XYsMDbCx80Rt4X5AOBgXiIiMo+LBIBZI3rW4N0WLDA25L6x5ttIX53MR32TUXAaIiISJbe8Bim5FZDLgGnDtKLjCMECY0MmDuwNX3c1yqsb8N3ZYtFxiIhIkC+SzRs3juvnBR+3nrP2y0+xwNgQpUKOuyM5mJeIqCdrNJos7wH3jQ0SnEYcFhgb0zIb6WBmKfKu1AhOQ0RE3S3hXDGKDPXwclEhdoiv6DjCsMDYmCAvZ4zv7wVJAjYfzxMdh4iIutn6I5cBAPeMCYRaqRCcRhwWGBvUsjLv5uO5MJq4wSMRUU+RXVqN7y+UQiYD5o/pubePABYYmxQ7RAuNkwMK9HU4mFkqOg4REXWTz5r3PZo4sDeCvJwFpxGLBcYGOTooLHtebORgXiKiHqG+yYhNx83rgD0wruetvPtzLDA2quU2UvzZIpRW1QtOQ0REXW33GR2u1DTCT+OIyaG9RccRjgXGRg32c8eIAA0ajRK+TOZgXiIie/fpYfPg3fvGBEGp4Ns3z4ANuz/KPIDrs6M5MHEwLxGR3crQVeJY9hUo5DLLquw9HQuMDZs1wh9uaiWyy2pw+FKZ6DhERNRFNjRPnb59sC983Xvmyrs/xwJjw5xVSsyOMG/i9Rk3eCQiskvV9U3YciIfALBgXM+eOv1TLDA27r7mdQD2nNGhvLpBcBoiIups21MLUFnfhGAvZ0zo7y06jtVggbFxQ/toMDxAgwajCZuP8yoMEZG9WX/EvFzGgqggyOUywWmsBwuMHVjQPJj30yOXuTIvEZEdSc2twOl8PVQKOe6K5ODdn2KBsQN3jOgDjZMDcstrsT+jWHQcIiLqJC37Hk0fpoWni0pwGuvCAmMHnFQK3B0ZAODHZaaJiMi26WsbsS21AABX3r0aFhg7Mb/5NtLe9GIU6msFpyEiohu15UQe6hpNCPV1Q2RwL9FxrA4LjJ3o39sVY0M8YZKArScLRMchIqIbIEnSj4N3xwVBJuPg3Z9jgbEjLRs8fnr4MhqaTILTEBFRRx3NKkdmcRWcVT9u3kutscDYkTsj+qC3mxr5FbXYm87BvEREtqrl6svskf5wc3QQnMY6scDYEUeHH5v61pP5gtMQEVFHlFbVY9eZQgDAgigO3r0WFhg701Jg9qYXQ1/TKDgNERG11+bjeWg0ShgR6IGhfTSi41gtFhg7M9jPHWFaNzQYTdh2ioN5iYhsickkYcNR89ovLYuU0tWxwNihe0abV2v86GAWV+YlIrIhBy6UILe8Fm6OSswa7i86jlVjgbFD944JhMbJAVml1fg2TSc6DhERtVHL4N15owLgpFIITmPdWGDskItaiUXR5oFfaxMvQpJ4FYaIyNoV6muRcK4IAPDAON4+uh4WGDu1eHxfqJVypObpcTSrXHQcIiK6js+P5sIkAVEhnhjg4yY6jtVjgbFT3q5qzB1lnpH0+bFcwWmIiOjXSJKELSfzAAD3c/Bum7DA2LGWwbzfnC6EvpZTqomIrNXxy1eQW14LV7USU8K1ouPYBBYYOzYy0AOhvm6obzLhqxN5ouMQEdE1bGgevDt1qJaDd9uIBcaOyWQyy0Cw/x2+zMG8RERWqMhQh+2p5nW7WiZg0PWxwNi5ORF94KJS4GJJNZIulYmOQ0REP7P+8GU0mSSM6dsLwwM8RMexGSwwds7N0QF3jvpxl2oiIrIedY1Gy9ovD44PEZzGtrDA9AALx/UFAOxJK4JOXyc2DBERWWw5kY+y6gb08XDClCG+ouPYFBaYHiBU64axIZ4wmiRsOJojOg4REcG879EH318CADx8UwgcFHxLbg+erR6iZWDYhiM5qG8yCk5DREQJ6cW4VFoNd0cl7h0TKDqOzWGB6SFih2ihdXdEaVU9vjldKDoOEVGP1zIu8f6oYLiqlYLT2B4WmB7CQSG3TKled4iDeYmIRDLUNSLponlm6F2RAYLT2CYWmB7kvrFBUCnkSM2twMmcK6LjEBH1WJ8dyUGD0YRQXzf07+0iOo5NYoHpQbxd1Zg1wh8A8MmhbLFhiIh6qIYmEz7+IRsAsOTmEMhkMrGBbBQLTA/z4Pi+AICdpwtRXMkp1URE3W1zci50hjr4uKkxe6S/6Dg2iwWmhxkWoEFkcC80GiWsP8wp1URE3amu0YjX9mQAAH57S3+oldz3qKPaXWAOHDiAWbNmwd/fHzKZDFu3bm31+oMPPgiZTNbqMXXq1FbHlJeXY8GCBXB3d4eHhweWLFmCqqqqVsecOnUKN998MxwdHREYGIhXX321/T8dXVXLVZhPD19GXSOnVBMRdZftqQWoqGlEQC8ny+9i6ph2F5jq6mqMGDEC77zzzjWPmTp1KgoLCy2Pzz77rNXrCxYsQFpaGuLj47Fjxw4cOHAAS5cutbxuMBgwZcoUBAcHIzk5Ga+99hqee+45vP/+++2NS1cxbagWfTycUFbdgG0pBaLjEBH1GJ82bxtwf1QQFHKOfbkR7Z54Pm3aNEybNu1Xj1Gr1dBqtVd97dy5c9i9ezeOHTuG0aNHAwDefvttTJ8+Ha+//jr8/f2xfv16NDQ04KOPPoJKpcKQIUOQkpKCN954o1XRoY5RKuRYGB2Ml3elY+PxXNzDBZSIiLrcmXw9UnMr4KCQ4Z7R/L17o7pkDMz+/fvh4+OD0NBQ/O53v0NZ2Y+7ICclJcHDw8NSXgAgJiYGcrkcR44csRwzceJEqFQqyzGxsbHIyMjAlStXn/5bX18Pg8HQ6kHXNjeiDxRyGZIvX8HFkqrrfwIREd2QloXrpg71g7erWnAa29fpBWbq1Kn473//i4SEBLzyyitITEzEtGnTYDSax1rodDr4+Pi0+hylUglPT0/odDrLMb6+rTe1avm45ZifW7NmDTQajeURGMh2+2t83B1xy6DeAIAPD2YJTkNEZN8MdY34uvmW/QNRQYLT2IdOLzD33Xcf7rjjDgwbNgxz5szBjh07cOzYMezfv7+zv1Urzz77LPR6veWRm5vbpd/PHiyd2A+AeX+k1NwKsWGIiOzYluQ81DYaMcjXFWNDPEXHsQtdPo26X79+8Pb2RmZmJgBAq9WiuLi41TFNTU0oLy+3jJvRarUoKipqdUzLx9caW6NWq+Hu7t7qQb9uXD8vTB1iPp87TnEwLxFRV5AkCf9rvn20ICqYC9d1ki4vMHl5eSgrK4Ofnx8AIDo6GhUVFUhOTrYcs3fvXphMJkRFRVmOOXDgABobGy3HxMfHIzQ0FL169erqyD1Ky8q8nx/LRUllveA0RET259DFMlwsqYaLSoG5o/qIjmM32l1gqqqqkJKSgpSUFABAVlYWUlJSkJOTg6qqKjz11FM4fPgwsrOzkZCQgNmzZ2PAgAGIjY0FAAwePBhTp07FI488gqNHj+KHH37A8uXLcd9998Hf3/xmev/990OlUmHJkiVIS0vDxo0b8c9//hMrV67svJ+cAABThvhisJ87Kuua8OWJPNFxiIjszn+TsgEAc0cFwM3RQWwYO9LuAnP8+HFEREQgIiICALBy5UpERERg9erVUCgUOHXqFO644w4MGjQIS5YsQWRkJL7//nuo1T+OuF6/fj3CwsJw2223Yfr06bjppptarfGi0Wjw7bffIisrC5GRkXjyySexevVqTqHuAg4KOR4cHwwA+CI5D5IkCU5ERGQ/8itqEX/WPARiUXSw4DT2RSbZ6TuWwWCARqOBXq/neJjrqKxrxJiXvkNdowmfPDzWMjuJiIhuzKu70/Hu/osY398LGx4ZJzqOTWjr+zf3QiK4OTrggSjzvwxe35PBqzBERJ3AUNeIDUfNK+8uiu4rNowdYoEhAMDvJvWHo4Mcp/P1OJ2vFx2HiMjmvZ94CRU1jejf2wUxg32u/wnULiwwBADwclXj9nDzlOp1P2SLDUNEZONqG4yWwbtPxYZCqeDbbWfjGSWLR24OAQBsSy2AvqbxOkcTEdG17DhVAENdEwJ6OWFK+NXXL6MbwwJDFsMDPDDQxxVNJgnbubAdEVGH/XTXaTl3ne4SLDDUyr3NO1O/sy8TJhMH8xIRtdex7HLLrtN3R3Jfvq7CAkOtPDAuGG6OShTq67D+yGXRcYiIbM6b350HANwVGYDebtx1uquwwFArjg4KPHKzeZPHt/ZmosloEpyIiMh2nMnX44fMMijkMsRNHiA6jl1jgaFfWHZLf/RydkBJZT2+zywVHYeIyGb85/tLAIAZw/wQ0MtZcBr7xgJDv6BSyjF7pHnDsY8OZnFhOyKiNsi7UoMdpwoBAEsn9hOcxv6xwNBVPTi+L1QKOb6/UIrjl6+IjkNEZPX+m3QZRpOE8f29MLSPRnQcu8cCQ1fV19sF8yLNV2HeP3BJcBoiIutW22DExmO5AICHJ4QITtMzsMDQNS25yXwJ9LtzRbhYUiU4DRGR9dp4LAf62kYEejphchi3DegOLDB0TQN8XBEz2AeSBHx4MEt0HCIiq1TXaMS7+y8CAH47sT8UXLiuW7DA0K9qmVL9ZXIeSqvqBachIrI+nx3NQXFlPfp4OOGe0Vy4rruwwNCvGhviiREBGtQ3mfDfQ9mi4xARWZ0vT+QBAJbd0g8qJd9WuwvPNP0qmUyGpRP7AwA+SbqM6vomwYmIiKxHdmk1zuQbIJMB04b5iY7To7DA0HVNHapFXy9n6Gsb8XnzKHsiIgLeax77csug3vB25bYB3YkFhq5LIZfhkeZFmT78/hIaub0AERHyrtRYbh89dutAwWl6HhYYapN5owLg7apGgb4OX6cUiI5DRCTc2sSLaGpeuC4yuJfoOD0OCwy1iaODAg/f1BcA8PdvM1DfZBQbiIhIoNN5emw4kgOAV19EYYGhNnt4Qgi07o4o1Ndh9xmd6DhERMKsP3IZJgmYOkSL6P5eouP0SCww1GaODgrMHxsEAFh/OEdwGiIiMUoq67E1JR8AsHh8X7FhejAWGGqXe8cEQiGX4Wh2OTJ0laLjEBF1u38nXkRdowkjAz0wrp+n6Dg9FgsMtYtW44jbB/sCAP6686zgNERE3aumoQkbj5uXk3g8ZiBkMm4bIAoLDLXbqmlhkMmA7y+UIqesRnQcIqJu8/6BS6isa0JfL2fcMrC36Dg9GgsMtVuItwsm9PcGALy194LgNERE3UNf04h/J14CAPw+NhRybtooFAsMdciTUwYBAL46mc+rMETUI2xNyUdtoxGhvm6YwW0DhGOBoQ6JCOqFiYN6w2iSsPbARdFxiIi6lL6mEW8lmK84zx8byLEvVoAFhjps+eQBAIAvjuehyFAnOA0RUdf5b1I2yqobMMDHFfdHBYuOQ2CBoRswNsQTY/r2QoPRhP8cuCQ6DhFRl6hvMmJTsnnm0bJb+kOl5FunNeD/BbohjzZfhVl/JAdXqhsEpyEi6nzbUgqQW14Lb1c1pg3Vio5DzVhg6IZMGtQb4X7uqG004qMfskTHISLqVE1GEz49fBkAsDg6GC5qpeBE1IIFhm6ITCbD8lvNV2HWHcqGoa5RcCIios7z5Yk8pObp4aJSYG5kgOg49BMsMHTDpg7RYoCPKyrrmvC/pMui4xARdYomownv7DPPsnw8ZiD6eDgJTkQ/xQJDN0wul1lmJH3w/SVU1zcJTkREdOO2pRYgp7wGni4qPDCOM4+sDQsMdYqZw/0Q7OWMKzWN2HCEO1UTkW0zmiT8a18mAGDJTSFwVnHsi7VhgaFOoVTIETfJfBXm3wcuoa7RKDgREVHH7TxdiEsl1dA4OWBRNK++WCMWGOo0cyL6oI+HE0qr6rHxWK7oOEREHWIySfhX8z5vD08IgZujg+BEdDUsMNRpVEo5lk3qDwBYm3gR9U28CkNEtufbszqcL6qCm1qJByf0FR2HroEFhjrV3ZEB0Lo7olBfh08OZYuOQ0TULpIk4a0E89iXxeP7QuPEqy/WigWGOpWjg8KyU/XbezO5Oi8R2ZSEc8U4W2iAs0qBJTeFiI5Dv4IFhjrd3FEBGOznjsq6JqxN5E7VRGQbJEnC281jXxZGB6OXi0pwIvo1LDDU6RRyGZ6KNV+F+fTwZV6FISKbkHz5ClLz9HB0kOORm/uJjkPXwQJDXWJyqA/C/dxR3WDExxwLQ0RWzmiS8OruDADAjGH+8HZVC05E18MCQ12i1R5JP2RxjyQismpfp+TjaHY5HBQyPDi+r+g41AYsMNRlWvZIMnCPJCKyYkaThLcSzGNfVsQMwrAAjeBE1BYsMNRl5HIZ4iab14X58GAWahq4RxIRWZ+Ec0XILquBh7MDr77YEBYY6lKzhvsjyNMZ5dUN3COJiKzSuuZxeveNCYKLmnse2QoWGOpSSoUcjzavzvvvA5dQUcMZSURkPQ5dLMWhi2VQymV4YFyQ6DjUDiww1OXmjgpAsJczSirr8e5+rgtDRNbjvebfSfPHBiGgl7PgNNQeLDDU5VRKOf4wfTAAYMuJfI6FISKrsPVkPr6/UAqFXIalE7nui61hgaFuMTnUBwG9zDtVv3/gkug4RNTDmUwSXttjXvfl3jGBCPTk1RdbwwJD3UKllOPZaearMGsTL6JQXys4ERH1ZLvTdMivqIWrWonVM8NFx6EOYIGhbjN9mBajg3uhrtFkWXOBiKi71TcZ8fKudADAwxP6wtFBITgRdQQLDHUbmUyGZ6aFAQA2Hc/DpZIqwYmIqCf65FA2cspr4OOmxm9v6S86DnVQuwvMgQMHMGvWLPj7+0Mmk2Hr1q2tXpckCatXr4afnx+cnJwQExODCxda/2u7vLwcCxYsgLu7Ozw8PLBkyRJUVbV+Mzt16hRuvvlmODo6IjAwEK+++mr7fzqyOqP7euLWMB8YTRLeiD8vOg4R9TDV9U34195MAMDvY0O57osNa3eBqa6uxogRI/DOO+9c9fVXX30Vb731FtauXYsjR47AxcUFsbGxqKursxyzYMECpKWlIT4+Hjt27MCBAwewdOlSy+sGgwFTpkxBcHAwkpOT8dprr+G5557D+++/34EfkazN76eEAgB2nCrEmXy94DRE1JN8fiwXhrom9PVyxrxRAaLj0I2QbgAA6auvvrJ8bDKZJK1WK7322muW5yoqKiS1Wi199tlnkiRJ0tmzZyUA0rFjxyzH7Nq1S5LJZFJ+fr4kSZL07rvvSr169ZLq6+stx6xatUoKDQ1tcza9Xi8BkPR6fUd/POpCj204IQWv2iEt/uiI6ChE1EPUNxqlUS98KwWv2iGtP3xZdBy6hra+f3fqGJisrCzodDrExMRYntNoNIiKikJSUhIAICkpCR4eHhg9erTlmJiYGMjlchw5csRyzMSJE6FSqSzHxMbGIiMjA1euXLnq966vr4fBYGj1IOu18vZBUMpl2J9RgqNZ5aLjEFEP8HVKPsqqG+DjpsY9o3n1xdZ1aoHR6XQAAF9f31bP+/r6Wl7T6XTw8fFp9bpSqYSnp2erY672NX76PX5uzZo10Gg0lkdgYOCN/0DUZfp6u+CeMeb/R6/uTockSYITEZE9q6xrxCu7zeu+PDQhBEoF57DYOrv5P/jss89Cr9dbHrm5uaIj0XX8360DoVbKcfzyFezLKBYdh4js2FsJF1BaVY8Qbxc8fFNf0XGoE3RqgdFqtQCAoqKiVs8XFRVZXtNqtSgubv1m1dTUhPLy8lbHXO1r/PR7/JxarYa7u3urB1k3rcbRsnX9q7szYDLxKgwRdb50nQEf/5ANAPjLrHColVz3xR50aoEJCQmBVqtFQkKC5TmDwYAjR44gOjoaABAdHY2KigokJydbjtm7dy9MJhOioqIsxxw4cACNjY2WY+Lj4xEaGopevXp1ZmQSbNkt/eGmViJdV4ktJ/NFxyEiOyNJEp7dchpNJglTwn0xKdTn+p9ENqHdBaaqqgopKSlISUkBYB64m5KSgpycHMhkMqxYsQJ//etfsW3bNpw+fRqLFi2Cv78/5syZAwAYPHgwpk6dikceeQRHjx7FDz/8gOXLl+O+++6Dv78/AOD++++HSqXCkiVLkJaWho0bN+Kf//wnVq5c2Wk/OFmHXi4qxN06AIB5LExVPTd6JKLOc/hSOU7mVMBZpcALs4eKjkOdqN0F5vjx44iIiEBERAQAYOXKlYiIiMDq1asBAE8//TQee+wxLF26FGPGjEFVVRV2794NR0dHy9dYv349wsLCcNttt2H69Om46aabWq3xotFo8O233yIrKwuRkZF48sknsXr16lZrxZD9eGhCXwR7OaO4sh7/TrwoOg4R2QlJkvD6t+aBu3dG9IFW43idzyBbIpPsdPqHwWCARqOBXq/neBgbsOt0IX63/gTcHZVIevY2ro5JRDdsT5oOv/1fMpwcFNj3+0ksMDaire/fdjMLiWxb7BAtQrxdYKhrwoYjOaLjEJGNM5okvLbHfPVlyU0hLC92iAWGrIJcLsNvJ/YDALy19wLKquoFJyIiW/bliTxkFlfBw9kBS2/pJzoOdQEWGLIad48ORLifOyrrmvD6t9zokYg65kp1A17elQ4AeHRSf7g7OghORF2BBYashkIuw3N3DAEAfH4shxs9ElGHfPRDFsqrGzDQxxWLm9eaIvvDAkNWZWyIJ+4Y4Q9JAp7fnsYtBoioXTKLK/HOvkwAwOMxA7lonR1jgSGr8+z0MKiVchzLvoKEc9xigIja7s3vLsAkATGDfTBjmJ/oONSFWGDI6vhpnPDwTSEAgFd2p8PILQaIqA3OFhiw83QhAODJKaGQyWSCE1FXYoEhq7Tslv7QODngQnEVPjmULToOEVk5o0nC45+fhCQBM4b5YbAf1/+ydywwZJU0Tg74v9sGAgD+9s05FOprBSciImv2bZoOF4qr4O6oxPOzh4iOQ92ABYas1kPj+yIiyANNJglvJWSKjkNEVkqSJKxt3oZk8fi+8HZVC05E3YEFhqyWXC7D76eEQiYDPjvKadVEdHUHM0uRmqeHWinntOkehAWGrNqEAd6YNdy8S/kb8Vzcjohaq2s04qWd5wAA90cF8epLD8ICQ1bvidsHQSGXYW96MXacKhAdh4isyF++TkO6rhJujko8dutA0XGoG7HAkNUL8XbB4ui+AICXdp7jtGoiAgCk5lZg4/FcAMBb8yPg6aISnIi6EwsM2YSnp4ZC4+SAQn0dvjqZLzoOEVmBfyZcAADMjeiDyaE+gtNQd2OBIZvg6KDAslv6AwBe3nUO+tpGwYmISKTE8yXYm14MmQx47DbeOuqJWGDIZiy5KQQDfFxRWtWAv3+bIToOEQliNEl4blsaAODB8X0R4u0iOBGJwAJDNkOllOPF2UMBAP87fBmn8zitmqgnennXOWSVVkPj5IAnp4SKjkOCsMCQTYnu74XZI827Va/edgYmDugl6lFyymrwyaHLAIBVU8PgqlYKTkSisMCQzfnD9MFwUSlwMqcCWzigl6jHkCQJq748hQajCVEhnrg/Kkh0JBKIBYZsjq+7o2XQ3su70mGo44Beop7gh8wyJF0qg0ohx+t3jxAdhwRjgSGb9PCEEPTzdkFpVT3e+u6C6DhE1MUkScJrzYP3748KQqCns+BEJBoLDNkklVKO1bPCAQDrDmUjs7hScCIi6kqbk/OQmlsBJwcF4iYPEB2HrAALDNmsSaE+iBnsiyaThOe2nYUkcUAvkT0qrarHX742T5t+ZGI/9HbjfkfEAkM2bvXMcKiUchzMLMWetCLRcYioCzy75TRqG40Y2scdj3PROmrGAkM2LcjLGb+d2A8A8OKOs6iubxKciIg606m8CsSfLYJcBrw8dzgUcpnoSGQlWGDI5j06aQD6eDghv6IWr+xOFx2HiDqJ0SRh1ZenAQCzRvhjaB+N4ERkTVhgyOY5qRR4Zd5wAOYVepMvXxGciIg6w8ZjuThXaIC7oxKrZ4aLjkNWhgWG7MJNA71xV2QAJAn4w5bTaDSaREciohuQU1aD15unTa+IGQQvVw7cpdZYYMhu/HH6YHi6qJBRVIn3D1wSHYeIbsCLO8+ivLoBg/3csTA6WHQcskIsMGQ3ermo8OeZgwEA/0y4gOzSasGJiKgjvjqZh/iz5lmFb947Eg4KvlXRL/FPBdmVOSP74OaB3mhoMuGPW09zbRgiG3OlugF/3XEOAPDYrQMQqnUTnIisFQsM2RWZTIa/zhkKtVKOHzLL8BU3eySyKS99cw5l1Q0Y6OOKx27lmi90bSwwZHeCvVzweIz5F9+LO8z30YnI+iVdLMMXyXmQyYCX5w2DSsm3KLo2/ukgu/TIzf0QpnXDlZpG/GHLadFxiOg6DHWN+ONX5r+rC6KCEBnsKTgRWTsWGLJLDgo5/n7PCMhkwO40HdIK9KIjEdGv+OuOs7hUWo3ebmo8FRsmOg7ZABYYsltD/DWYPswPAPDkplTUNxkFJyKiqzmeXW4Zr/bWfRHQODkITkS2gAWG7Npzs4bAy0WFdF0l/pd0WXQcIvoZQ10jlnxyHI1GCVPCfRHd30t0JLIRLDBk13q7qRE3eQAA4OVd6bhcxrVhiKzJ+sM50Nc2oq+XM/5x70jRcciGsMCQ3Vs8vi/GhniiySRh5aZUNHGbASKrkFlcibcSLgAA4iYPgItaKTgR2RIWGLJ7CrkMq2eGw1WtRPLlK/jgYJboSEQ9XkOTCX/Ycga1jUaMDfHEnIg+oiORjWGBoR5haB8N/jLLvJvtG/HnkaGrFJyIqGd7e+8FHM0uh7NKgb/dOYzbBVC78U8M9Rh3RQbg1jAfNDSZsHJTCm8lEQmSfPkK3tmXCQB47a4RGODjKjgR2SIWGOoxZDIZXp43DBonB6QVGPBv7lhN1O0amkx4clMKTBIwZ6Q/Zgz3Ex2JbBQLDPUoPm6O+ON0847Vr3+bgSOXygQnIupZ/nf4MrLLauDtqsbzs4eKjkM2jAWGepy7RwfgrsgASBLw/PazaOStJKJucTpPj5d3mXeafvy2AVywjm4ICwz1ODKZDH+YPhiuaiXOFhrw+rcZoiMR2T1JkvCXbWfQaJQwdYgWD4wLFh2JbBwLDPVIni4qvH73cADAfw5cwrHscsGJiOzbhwezcCKnAs4qBZ6fPQQymUx0JLJxLDDUY00d6ofZI/1hkoDHNpxETUOT6EhEdin58hX87RvzraNnpoXB191RcCKyByww1KOtmTsMfhpH6Ax1eGJjCiRJEh2JyK4UG+rw0MdHYZKA28N9sZC3jqiTsMBQj+asUuLvd4+Ag0KGPWlF2HVGJzoSkV15ZXcGDHVN6O2mxl9mhfPWEXUaFhjq8cYP8MbvJpk3fFz9dRryK2oFJyKyD5uO5+LLE3kAgP8sGo2AXs6CE5E9YYEhAvDopP4Y6OOK0qp6/GHLad5KIrpB54sq8YctpwEA88cGYmSgh9hAZHdYYIgAODoo8O+FkVDKZUg8X4J1h7JFRyKyWbUNRvx+cyqaTBJuHuiNF7lgHXUBFhiiZv16u2LV1DAAwF93nuPUaqIO+su2MziVp4eHswNenjccSm7USF2Af6qIfuI3N4dg9kh/GE0SntqcitoGo+hIRDblyKUybDqeB5kMeG9BJPp4OImORHaq0wvMc889B5lM1uoRFhZmeb2urg5xcXHw8vKCq6sr5s2bh6KiolZfIycnBzNmzICzszN8fHzw1FNPoamJa3RQ15PJZHhxzlD4aRyRXVaDV/eki45EZDP0tY3449YzAID5Y4MQ3d9LcCKyZ11yBWbIkCEoLCy0PA4ePGh57YknnsD27duxefNmJCYmoqCgAHPnzrW8bjQaMWPGDDQ0NODQoUP45JNPsG7dOqxevborohL9gruj+bI3AKw7lI3vL5QITkRk/YwmCUvWHUNmcRW8XdV4OjZUdCSyc11SYJRKJbRareXh7e0NANDr9fjwww/xxhtv4NZbb0VkZCQ+/vhjHDp0CIcPHwYAfPvttzh79iw+/fRTjBw5EtOmTcOLL76Id955Bw0NDV0Rl+gXbhnUGwuigiBJwIrPU6DT14mORGTVNhzNwfHLV+CmVuJ/S8bCw1klOhLZuS4pMBcuXIC/vz/69euHBQsWICcnBwCQnJyMxsZGxMTEWI4NCwtDUFAQkpKSAABJSUkYNmwYfH19LcfExsbCYDAgLS3tmt+zvr4eBoOh1YPoRvx5ZjjCtG4oq27A3f8+hIoaFmiiqzmRcwVrmrcKWDllEAb7uQtORD1BpxeYqKgorFu3Drt378Z7772HrKws3HzzzaisrIROp4NKpYKHh0erz/H19YVOZ14BVafTtSovLa+3vHYta9asgUajsTwCAwM79wejHsfRQYH3F45GQC8n5JbXYuWmVBhNXB+G6KdKq+qx9L/HUdNgxNgQTyyO7is6EvUQnV5gpk2bhrvvvhvDhw9HbGwsvvnmG1RUVGDTpk2d/a1aefbZZ6HX6y2P3NzcLv1+1DMEeTlj7QORUCnl2JtejP98f0l0JCKrIUkSnvnyNEqrGhDq64YPF4+GXM6tAqh7dPk0ag8PDwwaNAiZmZnQarVoaGhARUVFq2OKioqg1WoBAFqt9hezklo+bjnmatRqNdzd3Vs9iDrD0D4arJ4ZDgB4ZXc6DmWWCk5EZB3e3X8R350rgkohx5v3jYSbo4PoSNSDdHmBqaqqwsWLF+Hn54fIyEg4ODggISHB8npGRgZycnIQHR0NAIiOjsbp06dRXFxsOSY+Ph7u7u4IDw/v6rhEV3X/2CDMHukPSQL+uPUM14ehHi8ltwKv7ckAADw9NZTjXqjbdXqB+f3vf4/ExERkZ2fj0KFDuPPOO6FQKDB//nxoNBosWbIEK1euxL59+5CcnIyHHnoI0dHRGDduHABgypQpCA8Px8KFC5Gamoo9e/bgT3/6E+Li4qBWqzs7LlGbyOXm9WF83NTIKq3Gn78+w/Ew1GM1Gk14frt5UsXskf5YclOI4ETUE3V6gcnLy8P8+fMRGhqKe+65B15eXjh8+DB69+4NAPjHP/6BmTNnYt68eZg4cSK0Wi22bNli+XyFQoEdO3ZAoVAgOjoaDzzwABYtWoQXXnihs6MStYu7owP+fs8IAMAXyXn4R/x5wYmIup/RJOHR9SdwMqcCrmolnpkWBpmM416o+8kkO91212AwQKPRQK/XczwMdapPDmXjL9vSoJDL8N+Hx2LCAG/RkYi6zXPb0rDuUDbUSjnWPhCJyWE+oiORnWnr+zf3QiJqp0XRwZg7qg+MJglxG04gp6xGdCSibvHJoWzLTu3/uHckywsJxQJD1E4ymQx/u3MYRgRoUFHTiMc+O4Emo0l0LKIu9UNmqWXcy6qpYZg+zE9wIurpWGCIOsDRQYG1CyPholIgNU+PFRtT0MgSQ3YqrUCPBz8+CpME3B0ZgGW39BMdiYgFhqij/DRO+Me9I6GUy7DjVCH+sOW06EhEna6u0YiVG1PRaJTgr3HEn2aGc9AuWQUWGKIbMGWIFm/Nj4BcBmxOzsOm41wBmuyHJElYvuEEMooq4eHsgC2PToDGiYvVkXVggSG6QdOH+WFFzCAAwDNfnsLuM4WCExF1jnWHsvHduWI4KGR4e34EtBpH0ZGILFhgiDpB3OQBmDuqD0wS8PvNp5BZXCU6EtEN2ZtehOe3nwUAPHbrQNw8sLfgREStscAQdQKFXIZX5w1HVIgnquqbsOzTZFTXN4mORdQh54sq8fhnKQCAuaP6IG7yALGBiK6CBYaokygVcrx9fwR83NTILK7Cwg+PoK6ReyaRbckurcaST46hsr4JY0M8sWbuMCi4wzRZIRYYok7k4+aI9x6IhJtaiRM5FXhp5znRkYjarNhQh7vWHkJueS36eDhh7QORUCsVomMRXRULDFEniwzuhX8tGAUA+N/hy/joYJbgRETXZzRJeOyzkyitakCQpzP+t2QsPF1UomMRXRMLDFEXuGVQbzwVGwoAeGHHWWw9mS84EdG1SZKEt/dewJGscrioFFj30Bj06+0qOhbRr2KBIeoij07qj4cm9AUAPLk5FXvTi8QGIrqKRqMJv998Cm9+dwEA8MTtg1heyCawwBB1EZlMhj/PCMfciOaNH9efxKGLpaJjEbXyRvx5fHkiDwDw9NRQPDwhRHAiorZhgSHqQnK5DK/cNRwRQR6obTTi8c9TkFvO3avJOqz7IQtrEy8CAJ6ZFoZHJw2AnDOOyEawwBB1MQeFHJ8uicJAH1eUVNZj2afJqG/i9GoSa0+aDi/sOAtJAhZEBWHpzdygkWwLCwxRN3BRK/HfJWPRy9kBaQUGLPzgKMqrG0THoh5q9xkdHttwEiYJmD82CC/dOYxXXsjmsMAQdRM/jRPeuX8UnBwUOJpdjrj1J1DTwNV6qXsdOF+CR9cno8FowrShWrw4e4joSEQdwgJD1I3GD/DGF7+LhrNKgaRLZXh43TE0NJlEx6IeYl96MRZ9dBQmCZg53A//vC8CSgXfBsg28U8uUTcb4q/Bxw+OgataicOXyrF8wwluOUBd7ky+Hss3nAAAhGnd8Mq84VAp+RZAtot/eokEiOrnhXcXjIJKKce3Z4vwOw7spS6UXVqNpf89juoGI0YH98LWuAlwUStFxyK6ISwwRIJMHNQb6x4cA0cHOfZllODRT0/wdhJ1usziKtzz7yQU6OvQv7cL/rNoNBwduL8R2T4WGCKBxg/wxoeLx0CtlCMhvRhxG1hiqPPkV9Ri/n8Oo7iyHqG+bvh8aTR6cX8jshMsMESCTRjgjQ8Wj4ZKKUf82SL832cn0WhkiaEbk5JbgSlvJKKkubx8tnQcerupRcci6jQsMERW4OaBvfH+wkioFHLsTtNhxecpaGKJoQ7al1GMOe/8gOoGI7xcVPjwwdHcWZrsDgsMkZWYFOqDfy+MhINChp2nC/HwJ8ehr20UHYtszPcXSvDQx8cAAL2cHfDdylsQ0MtZcCqizscCQ2RFJof54O35oyCTmRccm/X2QRQZ6kTHIhuReL4Ej3120vLxxt9yzAvZLxYYIiszdagWHz84Bs4qBXLKa7D0v8ehr+GVGPp1CeeK8ODHR1FR0wi1Uo6DqyZjkK+b6FhEXYYFhsgKTQr1wVePTkAvZwek5ukx/z+HUVpVLzoWWSFJkrD5eC5+t/4EJAkYFeSBfb+fxNtGZPdYYIisVKjWPO3V21WNs4UGTH59PxLOFYmORVbEaJLw/PazeOqLU2hoMmFKuC82/jYa/h5OoqMRdTkWGCIrFqp1w6bfjkOgpxMq65qw5JPjeOPbDNGxyAo0Gk14eN0xrDuUDQBYOC4Y/7p/FBy4txH1EPyTTmTl+vV2xXcrb8HCccEAgLf2ZuI/By7BZJIEJyNRcspqMOvtg0g8XwIAeHpqKF6YPYR7G1GPws0wiGyAWqnAi3OGopezA97am4mXvjmHvCs1eH72UNHRqJtdKqnCkk+OI6u0GjIZ8MGi0bhtsK/oWETdjgWGyIasiBkEF7USL+9OxydJl2GSgD/NHAy1knvb2DtJknAwsxSPrj+ByromuDkqse6hsYgM7iU6GpEQLDBENkQul+G3t/SHTAb87Zt0/O/wZWToKrFqWhjfyOyYJEn4R/x5vLU3EwAwItAD/7hnBPr1dhWcjEgc3jAlskFLJ/bH2gcioZTLcDS7HHevPYTPjuaIjkVdoK7RiJlvH7SUl1BfN/xvyViWF+rxWGCIbNTUoVpseXQ8Jof2hkkCnt1yGs9tS0N9k1F0NOokBRW1uPX1/UgrMAAAFkQFYc8TE+Hu6CA4GZF4LDBENmx4gAc+XDwG944OBACsO5SNSa/tR/LlK4KT0Y366mQeYv9xAAX6OqiUcrw4Zyj+OoeDtolasMAQ2Ti5XIZX7hqOjx8ag17ODijU12Hee4ew7ocs0dGoA+qbjPjbN+fwxMZUVNY3wdNFhfcXRmLhuGDIZDLR8YishkySJLtcTMJgMECj0UCv18Pd3V10HKJucbGkCrf9PdHy8W9v6YeVtw/iLCUbcbmsGnEbTuBMvvmW0bJb+mPl7YO4vgv1KG19/+bfCiI70r+3K1JW347Job0BAP9OvISFHxxFhq5ScDK6noRzRZj3XhLO5Bvg6CDHv+6PwDPTwlheiK6BV2CI7NTnR3OwelsaGppMAIDfTeqPp2NDeRvCyhQb6vDUF6csq+qGad3w/sLRCPLiZozUM7X1/ZsFhsiOnSs04OkvTuF0vh6Aeafi39zcD9OH+QlORoD5/8+SdcdQoK8DADw0oS+ejg2Dk4q3/KjnYoFhgSGyeHd/Jl7bk4GWv+0rbx+EZbf05+0JQcqq6vHSN+ew9WQ+TBLQz9sF7z4wCmFa/q4iYoFhgSFq5WyBASs3pSC9eTzMQB9XvHLXcIwK4gq+3UWSJHx5Ih9/3XkWFTWNAICYwT54/e4R8HBWCU5HZB1YYFhgiH5BkiR8djQXz203j42RyYC5EQF46c6hcHTgbYuudCZfj7/uPIvDl8oBAIP93LFm7jCMDPQQG4zIyrDAsMAQXVNagR7PbzuLo9nmN1ONkwNeunMoZg73F5zM/pRW1eNv35zDlhP5AAC1Uo4VMYPwm5tD4KDgLTyin2OBYYEh+lWSJOGD77Pwt13nLGNjwv3c8cq84RgWoBEbzg6UVzfgs6M5eG//RVTVNwEA5oz0x+9jQxHQizOMiK6FBYYFhqhNquqb8Gb8eXxw0Lxyr1IuQ+wQLRaP74uxIZ6C09memoYmbDiSg39+dwGVzcWln7cL1swdhqh+XoLTEVk/FhgWGKJ2ySyuwp+3nkHSpTLLcxMGeOH5O4ZigA93Pr4eSZKw/VQh1nxzDoXN06IBYP7YILw4ewiUvF1E1CYsMCwwRO0mSRL2ny/B1yfz8XVqASQJUMhluGOEP5ZO7IcwrRsXwvsZk0lC4oUSvJ1wASdyKgAAfTycsCg6GPMiA+DtqhYbkMjGsMCwwBDdkHSdAS/vSsf+jJJWz699YBSmDuVCeI1GEz49fBn/TrwEncF8xcXRQY5HJw3A0on9OKuLqINYYFhgiDpFam4FntuehpPNVxcA83L3s0b448HxfeGiVooLJ0BBRS3eiD+PL5LzWj3fv7cL/rckCv4eToKSEdkHFhgWGKJOdSy7HG8lXEDSxTI0mcy/Nnq7qXH/2CDcMyYQfez4jdtkknDgQgm2nsxHwrliy+BcuQxYETMIEwf1RpjWjVddiDoBCwwLDFGXqKhpwNcpBfjohyxcLquxPD8y0AMxg30wItAD0f28bH7Qal2jEd+cLkRagQE/ZJZaVjAGzGNcFowLwvShfujr7SIwJZH9YYFhgSHqUg1NJuw6U4gNR3JwJKu81Wu93dR4cHxfPDShL5xVtnOLqaHJhJKqemw9mY+1iRdRWddkec1VrcTt4b64eaA3Zgz3g1rJqy1EXcEuCsw777yD1157DTqdDiNGjMDbb7+NsWPHtulzWWCIuockScgsrsLR7HLsOq3DwcxSy2vOKgXcHR0Q5ueGZbf0xwAfV6ualZNdWg2ZzLxP1Kl8PT47mmPZo6jF7JH+CNO6Y/7YQO5XRNQNbL7AbNy4EYsWLcLatWsRFRWFN998E5s3b0ZGRgZ8fHyu+/ksMERiVNY1YsuJfLx/4BLyK2p/8bqrWok7I/ogur8XbhroDbfmQcBdPT27sq4ROn0diivrkZJbgQPnS35x5ajFAB9XzBjmh6UT+/W4QcpEotl8gYmKisKYMWPwr3/9CwBgMpkQGBiIxx57DM8888x1P58Fhkis2gYjjmWXI6e8BkkXy3AkqxylVfWtjnF0kEOtVMDTRYXbw31RXd+EYX008HB2gLerGgN93FDV0AStuyMU8l8vOCaTBAnmlYWPZ5dDLpehsKIO+tpGfJGci4sl1df83BBvF0QG98KIAA3uHBUAF5WC690QCWLTBaahoQHOzs744osvMGfOHMvzixcvRkVFBb7++utffE59fT3q63/85WgwGBAYGMgCQ2RFcstrsC21AJfLqvFDZtlVr9BcS4i3Cwb6uKLRaEKhvg7ZZdUY1keD6nojqhuaUFpZjwajCY3G6/9KG9rHHRP6e2NSqA9G9+3FTRWJrEhbC4xVXhstLS2F0WiEr69vq+d9fX2Rnp5+1c9Zs2YNnn/++e6IR0QdFOjpjLjJAwCYx86k6yqRW16DvCu1OF9UiYqaRuRX1EIulyEtX2+Zrg0AWaXVyCptfRXlWPaVq34ftVIOpVyGYQEaeLmoMTxAg0FaN7iqlRjqr4GTigNwiWydVRaYjnj22WexcuVKy8ctV2CIyDrJZDIM9nPHYL+r/wurur4JhfpaFFTUwUWtxKWSKuhrG+HooIBcJkNJZT0CPZ3goJDDzVEJrcYRTg4KVNY1IdzPHfLr3HIiIttmlQXG29sbCoUCRUVFrZ4vKiqCVqu96ueo1Wqo1dYzu4GIboyLWokBPm4Y4OMGAIgM7iU4ERFZE6u88atSqRAZGYmEhATLcyaTCQkJCYiOjhaYjIiIiKyBVV6BAYCVK1di8eLFGD16NMaOHYs333wT1dXVeOihh0RHIyIiIsGstsDce++9KCkpwerVq6HT6TBy5Ejs3r37FwN7iYiIqOexymnUnYHrwBAREdmetr5/W+UYGCIiIqJfwwJDRERENocFhoiIiGwOCwwRERHZHBYYIiIisjksMERERGRzWGCIiIjI5rDAEBERkc1hgSEiIiKbY7VbCdyolgWGDQaD4CRERETUVi3v29fbKMBuC0xlZSUAIDAwUHASIiIiaq/KykpoNJprvm63eyGZTCYUFBTAzc0NMpms076uwWBAYGAgcnNzucdSF+O57h48z92D57l78Dx3j648z5IkobKyEv7+/pDLrz3SxW6vwMjlcgQEBHTZ13d3d+dfjm7Cc909eJ67B89z9+B57h5ddZ5/7cpLCw7iJSIiIpvDAkNEREQ2hwWmndRqNf7yl79ArVaLjmL3eK67B89z9+B57h48z93DGs6z3Q7iJSIiIvvFKzBERERkc1hgiIiIyOawwBAREZHNYYEhIiIim8MC007vvPMO+vbtC0dHR0RFReHo0aOiI9m0NWvWYMyYMXBzc4OPjw/mzJmDjIyMVsfU1dUhLi4OXl5ecHV1xbx581BUVCQosX14+eWXIZPJsGLFCstzPM+dIz8/Hw888AC8vLzg5OSEYcOG4fjx45bXJUnC6tWr4efnBycnJ8TExODChQsCE9seo9GIP//5zwgJCYGTkxP69++PF198sdXeOTzPHXPgwAHMmjUL/v7+kMlk2Lp1a6vX23Jey8vLsWDBAri7u8PDwwNLlixBVVVV54eVqM0+//xzSaVSSR999JGUlpYmPfLII5KHh4dUVFQkOprNio2NlT7++GPpzJkzUkpKijR9+nQpKChIqqqqshyzbNkyKTAwUEpISJCOHz8ujRs3Tho/frzA1Lbt6NGjUt++faXhw4dLjz/+uOV5nucbV15eLgUHB0sPPvigdOTIEenSpUvSnj17pMzMTMsxL7/8sqTRaKStW7dKqamp0h133CGFhIRItbW1ApPblpdeekny8vKSduzYIWVlZUmbN2+WXF1dpX/+85+WY3ieO+abb76R/vjHP0pbtmyRAEhfffVVq9fbcl6nTp0qjRgxQjp8+LD0/fffSwMGDJDmz5/f6VlZYNph7NixUlxcnOVjo9Eo+fv7S2vWrBGYyr4UFxdLAKTExERJkiSpoqJCcnBwkDZv3mw55ty5cxIAKSkpSVRMm1VZWSkNHDhQio+Pl2655RZLgeF57hyrVq2Sbrrppmu+bjKZJK1WK7322muW5yoqKiS1Wi199tln3RHRLsyYMUN6+OGHWz03d+5cacGCBZIk8Tx3lp8XmLac17Nnz0oApGPHjlmO2bVrlySTyaT8/PxOzcdbSG3U0NCA5ORkxMTEWJ6Ty+WIiYlBUlKSwGT2Ra/XAwA8PT0BAMnJyWhsbGx13sPCwhAUFMTz3gFxcXGYMWNGq/MJ8Dx3lm3btmH06NG4++674ePjg4iICPznP/+xvJ6VlQWdTtfqPGs0GkRFRfE8t8P48eORkJCA8+fPAwBSU1Nx8OBBTJs2DQDPc1dpy3lNSkqCh4cHRo8ebTkmJiYGcrkcR44c6dQ8druZY2crLS2F0WiEr69vq+d9fX2Rnp4uKJV9MZlMWLFiBSZMmIChQ4cCAHQ6HVQqFTw8PFod6+vrC51OJyCl7fr8889x4sQJHDt27Bev8Tx3jkuXLuG9997DypUr8Yc//AHHjh3D//3f/0GlUmHx4sWWc3m13yM8z233zDPPwGAwICwsDAqFAkajES+99BIWLFgAADzPXaQt51Wn08HHx6fV60qlEp6enp1+7llgyGrExcXhzJkzOHjwoOgodic3NxePP/444uPj4ejoKDqO3TKZTBg9ejT+9re/AQAiIiJw5swZrF27FosXLxaczn5s2rQJ69evx4YNGzBkyBCkpKRgxYoV8Pf353nuQXgLqY28vb2hUCh+MSujqKgIWq1WUCr7sXz5cuzYsQP79u1DQECA5XmtVouGhgZUVFS0Op7nvX2Sk5NRXFyMUaNGQalUQqlUIjExEW+99RaUSiV8fX15njuBn58fwsPDWz03ePBg5OTkAIDlXPL3yI156qmn8Mwzz+C+++7DsGHDsHDhQjzxxBNYs2YNAJ7nrtKW86rValFcXNzq9aamJpSXl3f6uWeBaSOVSoXIyEgkJCRYnjOZTEhISEB0dLTAZLZNkiQsX74cX331Ffbu3YuQkJBWr0dGRsLBwaHVec/IyEBOTg7PezvcdtttOH36NFJSUiyP0aNHY8GCBZb/5nm+cRMmTPjFMgDnz59HcHAwACAkJARarbbVeTYYDDhy5AjPczvU1NRALm/99qVQKGAymQDwPHeVtpzX6OhoVFRUIDk52XLM3r17YTKZEBUV1bmBOnVIsJ37/PPPJbVaLa1bt046e/astHTpUsnDw0PS6XSio9ms3/3ud5JGo5H2798vFRYWWh41NTWWY5YtWyYFBQVJe/fulY4fPy5FR0dL0dHRAlPbh5/OQpIknufOcPToUUmpVEovvfSSdOHCBWn9+vWSs7Oz9Omnn1qOefnllyUPDw/p66+/lk6dOiXNnj2b03vbafHixVKfPn0s06i3bNkieXt7S08//bTlGJ7njqmsrJROnjwpnTx5UgIgvfHGG9LJkyely5cvS5LUtvM6depUKSIiQjpy5Ih08OBBaeDAgZxGbQ3efvttKSgoSFKpVNLYsWOlw4cPi45k0wBc9fHxxx9bjqmtrZUeffRRqVevXpKzs7N05513SoWFheJC24mfFxie586xfft2aejQoZJarZbCwsKk999/v9XrJpNJ+vOf/yz5+vpKarVauu2226SMjAxBaW2TwWCQHn/8cSkoKEhydHSU+vXrJ/3xj3+U6uvrLcfwPHfMvn37rvo7efHixZIkte28lpWVSfPnz5dcXV0ld3d36aGHHpIqKys7PatMkn6ydCERERGRDeAYGCIiIrI5LDBERERkc1hgiIiIyOawwBAREZHNYYEhIiIim8MCQ0RERDaHBYaIiIhsDgsMERER2RwWGCIiIrI5LDBERERkc1hgiIiIyOawwBAREZHN+X/cjeiQ1/XTDwAAAABJRU5ErkJggg==\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "\n",
        "W = tf.Variable(tf.random.normal((100, 5),dtype=tf.float64),name='W')\n",
        "X = tf.Variable(tf.random.normal((100, 5),dtype=tf.float64),name='X')\n",
        "b = tf.Variable(tf.random.normal((1,   5),dtype=tf.float64),name='b')\n",
        "\n",
        "grads = np.ones((1,100*5 + 100*5 + 1*5));\n",
        "\n",
        "opt = RecordingOptimizer(learning_rate=0.01)\n",
        "opt.apply_gradients(zip(grads, [X,W,b]))"
      ],
      "metadata": {
        "id": "mjzFqGtDRRrJ",
        "outputId": "c7fd98a7-e3b5-4df2-9c4e-988057891df2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 339
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "RecordingOptimizer.apply_gradients\n"
          ]
        },
        {
          "output_type": "error",
          "ename": "InvalidArgumentError",
          "evalue": "{{function_node __wrapped__Sub_device_/job:localhost/replica:0/task:0/device:CPU:0}} Incompatible shapes: [1005] vs. [100,5] [Op:Sub] name: ",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mInvalidArgumentError\u001b[0m                      Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-24-0f5378209bb7>\u001b[0m in \u001b[0;36m<cell line: 17>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     15\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     16\u001b[0m \u001b[0mopt\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mRecordingOptimizer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlearning_rate\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0.01\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 17\u001b[0;31m \u001b[0mopt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mapply_gradients\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgrads\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mW\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mb\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m<ipython-input-24-0f5378209bb7>\u001b[0m in \u001b[0;36mapply_gradients\u001b[0;34m(self, grads_and_vars)\u001b[0m\n\u001b[1;32m      6\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mapply_gradients\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgrads_and_vars\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m       \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"RecordingOptimizer.apply_gradients\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 8\u001b[0;31m       \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mapply_gradients\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgrads_and_vars\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      9\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m \u001b[0mW\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mVariable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrandom\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnormal\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m5\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfloat64\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'W'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/keras/src/optimizers/optimizer.py\u001b[0m in \u001b[0;36mapply_gradients\u001b[0;34m(self, grads_and_vars, name, skip_gradients_aggregation, **kwargs)\u001b[0m\n\u001b[1;32m   1221\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mskip_gradients_aggregation\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mexperimental_aggregate_gradients\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1222\u001b[0m             \u001b[0mgrads_and_vars\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maggregate_gradients\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgrads_and_vars\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1223\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mapply_gradients\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgrads_and_vars\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1224\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1225\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_apply_weight_decay\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvariables\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/keras/src/optimizers/optimizer.py\u001b[0m in \u001b[0;36mapply_gradients\u001b[0;34m(self, grads_and_vars, name)\u001b[0m\n\u001b[1;32m    650\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_apply_weight_decay\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrainable_variables\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    651\u001b[0m             \u001b[0mgrads_and_vars\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgrads\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrainable_variables\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 652\u001b[0;31m             \u001b[0miteration\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_internal_apply_gradients\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgrads_and_vars\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    653\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    654\u001b[0m             \u001b[0;31m# Apply variable constraints after applying gradients.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/keras/src/optimizers/optimizer.py\u001b[0m in \u001b[0;36m_internal_apply_gradients\u001b[0;34m(self, grads_and_vars)\u001b[0m\n\u001b[1;32m   1251\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_internal_apply_gradients\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgrads_and_vars\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1252\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1253\u001b[0;31m         return tf.__internal__.distribute.interim.maybe_merge_call(\n\u001b[0m\u001b[1;32m   1254\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_distributed_apply_gradients_fn\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1255\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_distribution_strategy\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/tensorflow/python/distribute/merge_call_interim.py\u001b[0m in \u001b[0;36mmaybe_merge_call\u001b[0;34m(fn, strategy, *args, **kwargs)\u001b[0m\n\u001b[1;32m     49\u001b[0m   \"\"\"\n\u001b[1;32m     50\u001b[0m   \u001b[0;32mif\u001b[0m \u001b[0mstrategy_supports_no_merge_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 51\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstrategy\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     52\u001b[0m   \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     53\u001b[0m     return distribute_lib.get_replica_context().merge_call(\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/keras/src/optimizers/optimizer.py\u001b[0m in \u001b[0;36m_distributed_apply_gradients_fn\u001b[0;34m(self, distribution, grads_and_vars, **kwargs)\u001b[0m\n\u001b[1;32m   1343\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1344\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mgrad\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvar\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mgrads_and_vars\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1345\u001b[0;31m             distribution.extended.update(\n\u001b[0m\u001b[1;32m   1346\u001b[0m                 \u001b[0mvar\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mapply_grad_to_update_var\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgrad\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgroup\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1347\u001b[0m             )\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/tensorflow/python/distribute/distribute_lib.py\u001b[0m in \u001b[0;36mupdate\u001b[0;34m(self, var, fn, args, kwargs, group)\u001b[0m\n\u001b[1;32m   3011\u001b[0m           fn, autograph_ctx.control_status_ctx(), convert_by_default=False)\n\u001b[1;32m   3012\u001b[0m       \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_container_strategy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscope\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 3013\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_update\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvar\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgroup\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   3014\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3015\u001b[0m       return self._replica_ctx_update(\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/tensorflow/python/distribute/distribute_lib.py\u001b[0m in \u001b[0;36m_update\u001b[0;34m(self, var, fn, args, kwargs, group)\u001b[0m\n\u001b[1;32m   4081\u001b[0m     \u001b[0;31m# The implementations of _update() and _update_non_slot() are identical\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   4082\u001b[0m     \u001b[0;31m# except _update() passes `var` as the first argument to `fn()`.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 4083\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_update_non_slot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvar\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mvar\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mtuple\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgroup\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   4084\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   4085\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_update_non_slot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcolocate_with\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mshould_group\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/tensorflow/python/distribute/distribute_lib.py\u001b[0m in \u001b[0;36m_update_non_slot\u001b[0;34m(self, colocate_with, fn, args, kwargs, should_group)\u001b[0m\n\u001b[1;32m   4087\u001b[0m     \u001b[0;31m# once that value is used for something.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   4088\u001b[0m     \u001b[0;32mwith\u001b[0m \u001b[0mUpdateContext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcolocate_with\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 4089\u001b[0;31m       \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   4090\u001b[0m       \u001b[0;32mif\u001b[0m \u001b[0mshould_group\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   4091\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/tensorflow/python/autograph/impl/api.py\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    594\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0mwrapper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    595\u001b[0m     \u001b[0;32mwith\u001b[0m \u001b[0mag_ctx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mControlStatusCtx\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstatus\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mag_ctx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mStatus\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mUNSPECIFIED\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 596\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    597\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    598\u001b[0m   \u001b[0;32mif\u001b[0m \u001b[0minspect\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0misfunction\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfunc\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0minspect\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mismethod\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfunc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/keras/src/optimizers/optimizer.py\u001b[0m in \u001b[0;36mapply_grad_to_update_var\u001b[0;34m(var, grad)\u001b[0m\n\u001b[1;32m   1340\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_update_step_xla\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgrad\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvar\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mid\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_var_key\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvar\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1341\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1342\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_update_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgrad\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvar\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1343\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1344\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mgrad\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvar\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mgrads_and_vars\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/keras/src/optimizers/optimizer.py\u001b[0m in \u001b[0;36m_update_step\u001b[0;34m(self, gradient, variable)\u001b[0m\n\u001b[1;32m    239\u001b[0m                 \u001b[0;34mf\"`tf.keras.optimizers.legacy.{self.__class__.__name__}.\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    240\u001b[0m             )\n\u001b[0;32m--> 241\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgradient\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvariable\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    242\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    243\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mcompute_gradients\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mloss\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvar_list\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtape\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/keras/src/optimizers/adam.py\u001b[0m in \u001b[0;36mupdate_step\u001b[0;34m(self, gradient, variable)\u001b[0m\n\u001b[1;32m    196\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    197\u001b[0m             \u001b[0;31m# Dense gradients.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 198\u001b[0;31m             \u001b[0mm\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0massign_add\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgradient\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mm\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbeta_1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    199\u001b[0m             \u001b[0mv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0massign_add\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msquare\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgradient\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mv\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbeta_2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    200\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mamsgrad\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/tensorflow/python/util/traceback_utils.py\u001b[0m in \u001b[0;36merror_handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    151\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    152\u001b[0m       \u001b[0mfiltered_tb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_process_traceback_frames\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__traceback__\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 153\u001b[0;31m       \u001b[0;32mraise\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwith_traceback\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfiltered_tb\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    154\u001b[0m     \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    155\u001b[0m       \u001b[0;32mdel\u001b[0m \u001b[0mfiltered_tb\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/tensorflow/python/framework/ops.py\u001b[0m in \u001b[0;36mraise_from_not_ok_status\u001b[0;34m(e, name)\u001b[0m\n\u001b[1;32m   5881\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mraise_from_not_ok_status\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mNoReturn\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   5882\u001b[0m   \u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmessage\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;34m\" name: \"\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0;34m\"\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 5883\u001b[0;31m   \u001b[0;32mraise\u001b[0m \u001b[0mcore\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_status_to_exception\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;32mNone\u001b[0m  \u001b[0;31m# pylint: disable=protected-access\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   5884\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   5885\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mInvalidArgumentError\u001b[0m: {{function_node __wrapped__Sub_device_/job:localhost/replica:0/task:0/device:CPU:0}} Incompatible shapes: [1005] vs. [100,5] [Op:Sub] name: "
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# First, let's try with a simple custom training loop\n",
        "class RecordingOptimizer(tf.keras.optimizers.Adam):\n",
        "    def __init__(self, **kwargs):\n",
        "      super(RecordingOptimizer, self).__init__(**kwargs)\n",
        "      self._prev_grads = None\n",
        "      self._prev_deltas = None\n",
        "      self._sign_change_counts = []\n",
        "\n",
        "    # Per iteration we get a zip list containing entries:\n",
        "    #  - grads: (1, 128), vars: (1, 128)   # layer 1 W\n",
        "    #  - grads: (128,),   vars: (128,)     # layer 1 b\n",
        "    #  - grads: (128, 1), vars: (128, 1)   # layer 2 W\n",
        "    #  - grads: (1,),     vars: (1,)       # layer 2 b\n",
        "    def apply_gradients(self, grads_and_vars):\n",
        "      #print(f\"RecordingOptimizer.apply_gradients\")\n",
        "      grads, vars = zip(*grads_and_vars)\n",
        "      deltas = None\n",
        "      if self._prev_grads != None:\n",
        "        deltas = [cur - prev for cur, prev in zip(grads, self._prev_grads)]\n",
        "        #print(f\"deltas: {deltas[0]}\")\n",
        "        #print(f\"deltas: min={np.min(deltas[0])}, max={np.max(deltas[0])}, mean={np.mean(deltas[0])}\")\n",
        "      if self._prev_deltas != None:\n",
        "        delta_sign_changes = [np.sign(cur) != np.sign(prev) for cur, prev in zip(deltas, self._prev_deltas)]\n",
        "        count = 0\n",
        "        for sign_change in delta_sign_changes:\n",
        "          count += np.sum(sign_change)\n",
        "        self._sign_change_counts.append(count)\n",
        "        #print(f\"signs:  {count}\")\n",
        "      if deltas != None:\n",
        "        self._prev_deltas = deltas\n",
        "      self._prev_grads = grads\n",
        "\n",
        "      super().apply_gradients(zip(grads,vars))\n",
        "\n",
        "    def history(self):\n",
        "      return {\n",
        "          \"grad_sign_changes\": self._sign_change_counts\n",
        "      }\n",
        "\n",
        "loss_fn = tf.keras.losses.MSE\n",
        "opt = RecordingOptimizer(learning_rate=0.01)\n",
        "model = tf.keras.Sequential([\n",
        "    tf.keras.layers.Input(shape=(1,)),\n",
        "    tf.keras.layers.Dense(128, activation='relu'),\n",
        "    tf.keras.layers.Dense(1)\n",
        "])\n",
        "history_loss = []\n",
        "for iter in range(5000):\n",
        "  with tf.GradientTape() as tape:\n",
        "    loss = loss_fn(model(x), y)\n",
        "  if iter % 1000 == 0:\n",
        "    print(f\"Iteration {iter} - loss: {np.mean(loss)}\")\n",
        "  history_loss.append(np.mean(loss));\n",
        "  grads = tape.gradient(loss, model.trainable_variables)\n",
        "  opt.apply_gradients(zip(grads, model.trainable_variables))\n",
        "history2 = opt.history()\n",
        "\n",
        "\n",
        "y_predict = model.predict(x)\n",
        "plt.figure(layout='constrained')\n",
        "plt.subplot(1,3,1)\n",
        "plt.plot(x, y, color='blue', label=\"Truth\")\n",
        "plt.plot(x, y_predict, color='orange',label=\"Prediction\")\n",
        "plt.legend()\n",
        "plt.subplot(1,3,2)\n",
        "plt.plot(history_loss)\n",
        "plt.title('Loss')\n",
        "plt.xlabel('Epoch')\n",
        "plt.yscale('log')\n",
        "plt.subplot(1,3,3)\n",
        "plt.plot(history2['grad_sign_changes'])\n",
        "plt.title('Sign changes')\n",
        "plt.xlabel('Epoch')\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "mP57W0gHUIwL",
        "outputId": "1a3ac492-b316-4ee8-c4af-60fa94a60621",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 612
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Iteration 0 - loss: 1256933.8435615096\n",
            "Iteration 1000 - loss: 557779.4886292251\n",
            "Iteration 2000 - loss: 557776.3264285595\n",
            "Iteration 3000 - loss: 557776.3265501902\n",
            "Iteration 4000 - loss: 557778.5174129655\n",
            "32/32 [==============================] - 0s 3ms/step\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 3 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAosAAAHrCAYAAACn9tfQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAACHEElEQVR4nO3dd3hT9f4H8HfSvUtnWuhiCC2j7FrZUCllCIoDQSmIcNWWK1ZFuVcQEeGKiij2wvXKcICi/hCRi2AtS6CsQtmWYdm0BUon0JXz+yOcU0Jb6EhyTpL363nyJDnn23M+SXuaT75TJQiCACIiIiKiGqjlDoCIiIiIlIvJIhERERHViskiEREREdWKySIRERER1YrJIhERERHViskiEREREdWKySIRERER1YrJIhERERHViskiEREREdWKySIRERFVExoainHjxskdRq1UKhUSExPlDsMqMFkkIjKy5cuXQ6VSYd++fXKHQoTDhw/j8ccfR0hICBwdHdG0aVM8/PDDWLhwodyhkULZyh0AERERmcbOnTvRr18/BAcHY+LEidBoNDh//jx27dqFTz75BJMnT5bKZmZmQq1mnRIxWSQiIrIa7733Hjw8PLB37154enrq7cvNzdV77uDgYMLISMn4lYGISAEOHDiAuLg4uLu7w9XVFQMGDMCuXbv0ypSXl+Odd95Bq1at4OjoCG9vb/Ts2RMpKSlSmezsbIwfPx7NmjWDg4MDAgICMHz4cJw5c8bEr4iU6PTp02jbtm21RBEA/Pz89J7X1Gfx0KFD6NOnD5ycnNCsWTPMnj0by5Ytg0ql0vsbCw0NxdChQ7F9+3Z0794djo6OaN68Ob766qs6xanVavHJJ5+gffv2cHR0hK+vLwYNGlRjV441a9agXbt2cHBwQNu2bbFhwwa9/WfPnsVLL72E1q1bw8nJCd7e3njiiSeqXRNid5EdO3YgKSkJvr6+cHFxwaOPPoorV65Ui2/mzJkIDAyEs7Mz+vXrh2PHjtX4nuXn52PKlCkICgqCg4MDWrZsiffffx9arVav3HfffYcuXbrAzc0N7u7uaN++PT755JM6vV/GxppFIiKZHT16FL169YK7uzumTp0KOzs7/Oc//0Hfvn2xdetWREVFAQBmzpyJuXPn4vnnn0f37t1RWFiIffv2Yf/+/Xj44YcBACNHjsTRo0cxefJkhIaGIjc3FykpKTh37hxCQ0NlfJWkBCEhIUhLS8ORI0fQrl27ev3sxYsX0a9fP6hUKkybNg0uLi744osvaq2BPHXqFB5//HFMmDAB8fHxWLp0KcaNG4cuXbqgbdu29zzXhAkTsHz5csTFxeH5559HRUUF/vjjD+zatQtdu3aVym3fvh2rV6/GSy+9BDc3N3z66acYOXIkzp07B29vbwDA3r17sXPnTowaNQrNmjXDmTNnsGjRIvTt2xfHjh2Ds7Oz3rknT56MJk2a4O2338aZM2ewYMECJCYmYtWqVVKZadOmYd68eRg2bBhiY2Nx8OBBxMbG4tatW3rHunHjBvr06YOLFy/ib3/7G4KDg7Fz505MmzYNly9fxoIFCwAAKSkpePrppzFgwAC8//77AIDjx49jx44dePnll+v2CzImgYiIjGrZsmUCAGHv3r017h8xYoRgb28vnD59Wtp26dIlwc3NTejdu7e0LTIyUhgyZEit57l+/boAQPjggw8MFzxZlN9++02wsbERbGxshOjoaGHq1KnCxo0bhbKysmplQ0JChPj4eOn55MmTBZVKJRw4cEDadu3aNcHLy0sAIGRlZen9LABh27Zt0rbc3FzBwcFBePXVV+8Z46ZNmwQAwt///vdq+7RarfQYgGBvby+cOnVK2nbw4EEBgLBw4UJp240bN6odJy0tTQAgfPXVV9I28TqNiYnRO88rr7wi2NjYCPn5+YIgCEJ2drZga2srjBgxQu+YM2fOFADovWfvvvuu4OLiIpw4cUKv7JtvvinY2NgI586dEwRBEF5++WXB3d1dqKiouOd7Ixc2QxMRyaiyshK//fYbRowYgebNm0vbAwICMHr0aGzfvh2FhYUAAE9PTxw9ehQnT56s8VhOTk6wt7fHli1bcP36dZPET+bl4YcfRlpaGh555BEcPHgQ8+bNQ2xsLJo2bYq1a9fe82c3bNiA6OhodOzYUdrm5eWFMWPG1Fg+IiICvXr1kp77+vqidevW+Ouvv+55nv/7v/+DSqXC22+/XW2fSqXSex4TE4MWLVpIzzt06AB3d3e9czg5OUmPy8vLce3aNbRs2RKenp7Yv39/tXNMmjRJ7zy9evVCZWUlzp49CwBITU1FRUUFXnrpJb2fu3NwkOiHH35Ar1690KRJE1y9elW6xcTEoLKyEtu2bQOgu7ZLSkr0upQoCZNFIiIZXblyBTdu3EDr1q2r7QsPD4dWq8X58+cBALNmzUJ+fj4eeOABtG/fHq+//joOHToklXdwcMD777+PX3/9Ff7+/ujduzfmzZuH7Oxsk70eUr5u3bph9erVuH79Ovbs2YNp06ahqKgIjz/+OI4dO1brz509exYtW7astr2mbQAQHBxcbVuTJk3u+0Xm9OnTCAwMhJeX131eSd3OcfPmTcyYMUPqM+jj4wNfX1/k5+ejoKDgvsds0qQJAEjHFJPGu1+3l5eXVFZ08uRJbNiwAb6+vnq3mJgYAFWDil566SU88MADiIuLQ7NmzfDcc89V63spJyaLRERmonfv3jh9+jSWLl2Kdu3a4YsvvkDnzp3xxRdfSGWmTJmCEydOYO7cuXB0dMT06dMRHh6OAwcOyBg5KZG9vT26deuGOXPmYNGiRSgvL8cPP/xgsOPb2NjUuF0QBJOeY/LkyXjvvffw5JNP4vvvv8dvv/2GlJQUeHt7VxtkUtdj1pVWq8XDDz+MlJSUGm8jR44EoBtclJGRgbVr1+KRRx7B5s2bERcXh/j4+Hqf0xg4wIWISEa+vr5wdnZGZmZmtX1//vkn1Go1goKCpG1eXl4YP348xo8fj+LiYvTu3RszZ87E888/L5Vp0aIFXn31Vbz66qs4efIkOnbsiI8++gjffPONSV4TmR9x0Mjly5drLRMSEoJTp05V217TtsZo0aIFNm7ciLy8vDrVLt7Pjz/+iPj4eHz00UfStlu3biE/P79BxwsJCQGge91hYWHS9mvXrlWrNW3RogWKi4ulmsR7sbe3x7BhwzBs2DBotVq89NJL+M9//oPp06fXWntrKqxZJCKSkY2NDQYOHIiff/5ZbyqPnJwcrFy5Ej179oS7uzsA3YfRnVxdXdGyZUuUlpYC0I28vHs0ZosWLeDm5iaVIeu2efPmGmvI1q9fDwA1docQxcbGIi0tDRkZGdK2vLw8rFixwqAxjhw5EoIg4J133qm2ryG1ezY2NtV+buHChaisrGxQfAMGDICtrS0WLVqkt/2zzz6rVvbJJ59EWloaNm7cWG1ffn4+KioqAFS/ttVqNTp06AAAirh2WbNIRGQiS5curbEf0syZM5GSkoKePXvipZdegq2tLf7zn/+gtLQU8+bNk8pFRESgb9++6NKlC7y8vLBv3z78+OOP0vq4J06cwIABA/Dkk08iIiICtra2+Omnn5CTk4NRo0aZ7HWSck2ePBk3btzAo48+ijZt2qCsrAw7d+7EqlWrEBoaivHjx9f6s1OnTsU333yDhx9+GJMnT5amzgkODkZeXl61wScN1a9fPzz77LP49NNPcfLkSQwaNAharRZ//PEH+vXrV+/1oIcOHYqvv/4aHh4eiIiIQFpaGn7//Xdpap368vf3x8svv4yPPvoIjzzyCAYNGoSDBw/i119/hY+Pj9778Prrr2Pt2rUYOnSoNG1QSUkJDh8+jB9//BFnzpyBj48Pnn/+eeTl5aF///5o1qwZzp49i4ULF6Jjx44IDw9vUJwGJeNIbCIiqyBOyVHb7fz588L+/fuF2NhYwdXVVXB2dhb69esn7Ny5U+84s2fPFrp37y54enoKTk5OQps2bYT33ntPmvbk6tWrQkJCgtCmTRvBxcVF8PDwEKKiooTvv/9ejpdNCvTrr78Kzz33nNCmTRvB1dVVsLe3F1q2bClMnjxZyMnJ0St799Q5giAIBw4cEHr16iU4ODgIzZo1E+bOnSt8+umnAgAhOztb72drmuapT58+Qp8+fe4bZ0VFhfDBBx8Ibdq0Eezt7QVfX18hLi5OSE9Pl8oAEBISEqr97N1xX79+XRg/frzg4+MjuLq6CrGxscKff/5ZrVxtU1xt3rxZACBs3rxZL77p06cLGo1GcHJyEvr37y8cP35c8Pb2Fl544QW9ny8qKhKmTZsmtGzZUrC3txd8fHyEhx56SPjwww+la/fHH38UBg4cKPj5+Qn29vZCcHCw8Le//U24fPnyfd8rU1AJggF7mhIREZFVmTJlCv7zn/+guLi41sEh1iA/Px9NmjTB7Nmz8c9//lPucAyKfRaJiIioTm7evKn3/Nq1a/j666/Rs2dPq0oU734fAEirsfTt29e0wZgA+ywSERFRnURHR6Nv374IDw9HTk4OlixZgsLCQkyfPl3u0Exq1apVWL58OQYPHgxXV1ds374d3377LQYOHIgePXrIHZ7BMVkkIiKiOhk8eDB+/PFHfP7551CpVOjcuTOWLFmC3r17yx2aSXXo0AG2traYN28eCgsLpUEvs2fPljs0o2CfRSIiIiKqFfssEhEREVGtmCwSERERUa0sts+iVqvFpUuX4ObmZrCJQolqIwgCioqKEBgYCLWa38FqwmuSTM3crkteI2Rqdb1GLDZZvHTpkt56qkSmcP78eTRr1kzuMBSJ1yTJxVyuS14jJJf7XSMWmyy6ubkB0L0B4rqqRMZSWFiIoKAg6e+OquM1SaZmbtclrxEytbpeIxabLIpV+O7u7rzoyGTYdFQ7XpMkF3O5LnmNkFzud40ovxMHEREREcmGySIRERER1YrJIhERERHVymL7LBpDZWUlysvL5Q6DDMTGxga2trZm05+JiIhIDkwW66i4uBgXLlwAV0e0LM7OzggICIC9vb3coRARESkSk8U6qKysxIULF+Ds7AxfX1/WRFkAQRBQVlaGK1euICsrC61atTKLSXuJiIhMjcliHZSXl0MQBPj6+sLJyUnucMhAnJycYGdnh7Nnz6KsrAyOjo5yh0RERKQ4rEqpB9YoWh7WJhIREd0bPymJiIiIqFZMFomIiIioVkwWqdG2bNkClUqF/Px8uUMhIiIiA6tXsjh37lx069YNbm5u8PPzw4gRI5CZmalXpm/fvlCpVHq3F154Qa/MuXPnMGTIEDg7O8PPzw+vv/46Kioq9Mps2bIFnTt3hoODA1q2bInly5c37BVaqbt/B3ffZs6c2aDj9u3bF1OmTDForERERKRc9RoNvXXrViQkJKBbt26oqKjAP/7xDwwcOBDHjh2Di4uLVG7ixImYNWuW9NzZ2Vl6XFlZiSFDhkCj0WDnzp24fPkyxo4dCzs7O8yZMwcAkJWVhSFDhuCFF17AihUrkJqaiueffx4BAQGIjY1t7Gu2CpcvX5Yer1q1CjNmzNBL7F1dXaXHgiCgsrIStrYcHE9ERET66lWzuGHDBowbNw5t27ZFZGQkli9fjnPnziE9PV2vnLOzMzQajXRzd3eX9v322284duwYvvnmG3Ts2BFxcXF49913kZycjLKyMgDA4sWLERYWho8++gjh4eFITEzE448/jo8//tgAL7nxBAEoKZHnVtc5we98/z08PKBSqaTnf/75J9zc3PDrr7+iS5cucHBwwPbt2zFu3DiMGDFC7zhTpkxB3759AQDjxo3D1q1b8cknn0g1lGfOnJHKpqeno2vXrnB2dsZDDz1UrdbZ3HE+diIiskaNqkoqKCgAAHh5eeltX7FiBb755htoNBoMGzYM06dPl2oX09LS0L59e/j7+0vlY2Nj8eKLL+Lo0aPo1KkT0tLSEBMTo3fM2NjYezZ/lpaWorS0VHpeWFhYY7ncXGDaNKC4GFi1ql4vV3LjBnBHxZxJFRcDd1TiNsqbb76JDz/8EM2bN0eTJk3uW/6TTz7BiRMn0K5dO6nm2NfXV0oY//nPf+Kjjz6Cr68vXnjhBTz33HPYsWOHYYJVgB49gDNngG+/Bfr0kTsay/L2z0fwy6HLeHlAK8Q/FCp3OERUD4M/+QPHLtf8mUvya+Jsh8e7NMM/h0Q0+BgNTha1Wi2mTJmCHj16oF27dtL20aNHIyQkBIGBgTh06BDeeOMNZGZmYvXq1QCA7OxsvUQRgPQ8Ozv7nmUKCwtx8+bNGifGnjt3Lt555537xq1WA0uX6h5//TVgzau8zZo1Cw8//HCdy3t4eMDe3l6qOb7be++9hz63s6g333wTQ4YMwa1btyxmsusLF4DLlwHOy254xaWVyCspw83ySrlDIaJ6YqKobNdvlKOkrHH/WxucLCYkJODIkSPYvn273vZJkyZJj9u3b4+AgAAMGDAAp0+fRosWLRoe6X1MmzYNSUlJ0vPCwkIEBQVVK+ftDdjZAeXlQHY2EBxc/3M5O+tq+ORwR/fPRuvatavhDgagQ4cO0uOAgAAAQG5uLoIb8iYrjCDoaqUBwM9P3lgskTjfPZv6ie7t18OXcTH/Jp7uHgwXB/Yzp/v76IlI9Gzl06hjNOgvLTExEevWrcO2bdvQrFmze5aNiooCAJw6dQotWrSARqPBnj179Mrk5OQAgFRbpdFopG13lnF3d691uT0HBwc4ODjcN3aVCggMBM6e1dUSNSSPUakM1xQsJ5e7XoRarYZw16d1eXl5nY9nZ2cnPRZXu9FqtY2IUDkKCwGxlwOTRcNTi8kimC0S1eaPk1fw4or9AIC9Z/Lwn2cN+4WfLNMD/m7wd29cC1+9BrgIgoDExET89NNP2LRpE8LCwu77MxkZGQCqapqio6Nx+PBh5IrVNABSUlLg7u6OiIgIqUxqaqrecVJSUhAdHV2fcGt1OxRcumSQw1kMX19fvVHUQNXvT2Rvb4/KSutrKhS/u7i6GrZ2l3RU0GWLrFkkqt3eM9elxxuP5tyjJJFh1StZTEhIwDfffIOVK1fCzc0N2dnZyM7Oxs2bNwEAp0+fxrvvvov09HScOXMGa9euxdixY9G7d2+piXLgwIGIiIjAs88+i4MHD2Ljxo146623kJCQINUMvvDCC/jrr78wdepU/Pnnn/j3v/+N77//Hq+88opBXnRgoO6eyaK+/v37Y9++ffjqq69w8uRJvP322zhy5IhemdDQUOzevRtnzpzB1atXLabm8H7EHFr82yHDqmqGZrZIRKQ09UoWFy1ahIKCAvTt2xcBAQHSbdXtYcX29vb4/fffMXDgQLRp0wavvvoqRo4ciV9++UU6ho2NDdatWwcbGxtER0fjmWeewdixY/XmZQwLC8P//vc/pKSkIDIyEh999BG++OILg82xyGSxZrGxsZg+fTqmTp2Kbt26oaioCGPHjtUr89prr8HGxgYRERHw9fXFuXPnZIrWtMS/FSaLxiF2W2CuSESkPPXqs3i/b/1BQUHYunXrfY8TEhKC9evX37NM3759ceDAgfqEV2diM/RdLa4Wa9y4cRg3bpz0vG/fvrX+Lt955517jip/4IEHkJaWprctNDS02vE6duxoUbVEYrIo/u2QYYk1i1rL+ZMhIrIYVrk2NGsWqb7YDG1ct3NFDnAhIlIgJotEdcBmaOPi1DlERMpllckiR0NTfbEZ2rjUUp9FZotEREpjlcmiWDt07VrV3HlE98JmaOOqaoYmIiKlscpk0ctLt4oLUDV/HtG9sBnauDgamohIuawyWVSpAHFpY2sZEU0NV1RUtbwjm6GNo2o0NLNFIiKlscpkEbC+6XOo4cS/ETc33QouZHjSCi4yx0FERNVZfbKYnS1vHKR8bII2Po6GJiJSLqtNFtkMTXUl/o2wCdp41Fzuj4hIsaw2WWQztOGMGzcOI0aMkJ737dsXU6ZMadQxDXEMQ2HNovFJA1xkjoOs17Zt2zBs2DAEBgZCpVJhzZo1tZZ94YUXoFKpsGDBAr3teXl5GDNmDNzd3eHp6YkJEyagWOzwTGTGmCxacLI4btw4qFQqqFQq2Nvbo2XLlpg1axYqKiqMet7Vq1fj3XffrVPZLVu2QKVSIT8/v8HHMDYmi8YnTZ3DmkWSSUlJCSIjI5GcnHzPcj/99BN27dqFwBr+IYwZMwZHjx5FSkoK1q1bh23btmHSpEnGCpnIZOq1NrQlEZuhLb3P4qBBg7Bs2TKUlpZi/fr1SEhIgJ2dHaZNm6ZXrqysDPb29gY5p5eXlyKOYShshjY+sWaRa0OTXOLi4hAXF3fPMhcvXsTkyZOxceNGDBkyRG/f8ePHsWHDBuzduxddu3YFACxcuBCDBw/Ghx9+WGNySWQuWLPYkJpFQQAqSuS51bPmxcHBARqNBiEhIXjxxRcRExODtWvXSk3H7733HgIDA9G6dWsAwPnz5/Hkk0/C09MTXl5eGD58OM6cOSMdr7KyEklJSfD09IS3tzemTp1arTbo7ibk0tJSvPHGGwgKCoKDgwNatmyJJUuW4MyZM+jXrx8AoEmTJlCpVBg3blyNx7h+/TrGjh2LJk2awNnZGXFxcTh58qS0f/ny5fD09MTGjRsRHh4OV1dXDBo0CJcNUHXMmkXj4wAXUjqtVotnn30Wr7/+Otq2bVttf1paGjw9PaVEEQBiYmKgVquxe/fuGo9ZWlqKwsJCvRuRElltzaKYLObkAFotoK5P2lx5A/hepjlUniwGbF0a/ONOTk64du0aACA1NRXu7u5ISUkBAJSXlyM2NhbR0dH4448/YGtri9mzZ2PQoEE4dOgQ7O3t8dFHH2H58uVYunQpwsPD8dFHH+Gnn35C//79az3n2LFjkZaWhk8//RSRkZHIysrC1atXERQUhP/7v//DyJEjkZmZCXd3dzg5OdV4jHHjxuHkyZNYu3Yt3N3d8cYbb2Dw4ME4duwY7G7PsH7jxg18+OGH+Prrr6FWq/HMM8/gtddew4oVKxr8fgFMFk2hagUXZoukTO+//z5sbW3x97//vcb92dnZ8PPz09tma2sLLy8vZNfShDV37ly88847Bo+VyNCsNln099fdV1Tolv3z9ZU3HmMTBAGpqanYuHEjJk+ejCtXrsDFxQVffPGF1Pz8zTffQKvV4osvvpCaBZctWwZPT09s2bIFAwcOxIIFCzBt2jQ89thjAIDFixdj48aNtZ73xIkT+P7775GSkoKYmBgAQPPmzaX9YnOzn58fPD09azyGmCTu2LEDDz30EABgxYoVCAoKwpo1a/DEE08A0CW7ixcvRosWLQAAiYmJmDVrVkPfMgmboY1PzRVcSMHS09PxySefYP/+/dL/RkOYNm0akpKSpOeFhYUICgoy2PGJDMVqk0U7O12CeOWKruaoXsmijbOuhk8ONs71Kr5u3Tq4urqivLwcWq0Wo0ePxsyZM5GQkID27dvr9VM8ePAgTp06BTc3N71j3Lp1C6dPn0ZBQQEuX76MqKgoaZ+trS26du1a68CEjIwM2NjYoE+fPvWK+07Hjx+Hra2t3nm9vb3RunVrHD9+XNrm7OwsJYoAEBAQgNzc3AafF+DqLaai4tQ5pGB//PEHcnNzERwcLG2rrKzEq6++igULFuDMmTPQaDTV/t9UVFQgLy8PGrGT/F0cHBzg4OBg1NiJDMFqk0VA16x45Qpw8SIQGVmPH1SpGtUUbEr9+vXDokWLYG9vj8DAQNjaVv3KXVz0X0NxcTG6dOlSY7OtbwOrXmtrVjYGsTlapFKpGp18iE3Q7u5cvcWYqpqhiZTn2WeflVpGRLGxsXj22Wcxfvx4AEB0dDTy8/ORnp6OLl26AAA2bdoErVar90WXyBxZdbLYtClw8GBVQmCJXFxc0LJlyzqV7dy5M1atWgU/Pz+4u7vXWCYgIAC7d+9G7969Aei+Oaenp6Nz5841lm/fvj20Wi22bt1a7Z8tAKlms7Kysta4wsPDUVFRgd27d0vN0NeuXUNmZiYiIiLq9Noaik3QJsJmaJJZcXExTp06JT3PyspCRkYGvLy8EBwcDG9vb73ydnZ20Gg00uDA8PBwDBo0CBMnTsTixYtRXl6OxMREjBo1iiOhyexZ7WhoQJcsArqaRdLNEebj44Phw4fjjz/+QFZWFrZs2YK///3vuHDhAgDg5Zdfxr/+9S+sWbMGf/75J1566aVqcyTeKTQ0FPHx8XjuueewZs0a6Zjff/89ACAkJAQqlQrr1q3DlStXapzAtlWrVhg+fDgmTpyI7du34+DBg3jmmWfQtGlTDB8+3CjvhYiDW0xDXMFFy2yRZLJv3z506tQJnTp1AgAkJSWhU6dOmDFjRp2PsWLFCrRp0wYDBgzA4MGD0bNnT3z++efGCpmoTgzRzdaqaxbFBMCSaxbrw9nZGdu2bcMbb7yBxx57DEVFRWjatCkGDBgg1TS++uqruHz5MuLj46FWq/Hcc8/h0UcfRUFBQa3HXbRoEf7xj3/gpZdewrVr1xAcHIx//OMfAICmTZvinXfewZtvvonx48dj7NixWL58ebVjLFu2DC+//DKGDh2KsrIy9O7dG+vXr6/W9GxoTBZNQwWu4ELy6tu3b726rdw5pZjIy8sLK1euNGBURMpg1cmipdcs1pR03W+fRqPBl19+WevP2draYsGCBdWWubrTli1b9J47Ojpi/vz5mD9/fo3lp0+fjunTp9/zGE2aNMFXX31V6znHjRsnzdEoGjFiRKP7LLIZ2jQ4zyJR/W0/eRU9W/nIHQZZAatuhhZriyw1WaTGY82iaag5Gpqo3o5eqr1Fh8iQrDpZFGsW2QxNtRFrFpksGpeKA1yIiBSLySKA3FygrEzeWEiZxC8SbIY2Da7gQkSkPFadLHp76ybnBoBaVmMiK8dmaNMQV3DRMlckIlIcq04W1er69VtkfyrLc6/faVERUFKie8yaRePiABciIuWy6mQRqNv0OTY2NgCAMrZVW5wbN24AqL76C6C/eouLeSzYY7aqVnBhtkhEpDRWPXUOULfpc2xtbeHs7IwrV67Azs4OarXV59hmTxAE3LhxA7m5ufD09JS+ENyJTdCmo+YAFyIixbL6ZLEuzdAqlQoBAQHIysrC2bNnTRMYmYSnpyc0Gk2N+zgS2nRUnDqHiEixrD5ZrOv0Ofb29mjVqhWboi2InZ1djTWKIo6ENj2mikREysNksR6ruKjVajg6Oho3IFIMNkObDkdDExEpl9V3vuP60FQbLvVnOmyGJiJSLqtPFi19fWhqONYsmk7VaGgiIlIaq08WxUSguBgoLJQ3FlIWDnAxnarl/pguEhEpjdUni66uunn0ADZFkz4OcDEdNSflJiJSLKtPFgE2RVN1XL3FxDjPIhGRYjFZBAe5UHVcvcW0xD6LWmaLRESKw2QRrFmk6jgS2rSkFVxkjoOIiKpjsgjWLFJ1HAltWir2WSQiUiwmi2DNIlXHmkXTkqbOYbZIRKQ4TBZRt/WhybqwZtG02AxNRKRcTBZR9/WhyXqwZtHEuIILEZFiMVlEVbJ4+TKg1cobCynDhQu6e/Fvg4yLK7gQ3Z/q/kWIjILJIgB/f10H+4oK4MoVuaMhJTh/XncfFCRvHNZCbIbWMlskIlIcJosA7Ox0CSPAfoukq10W/w6YLJqGis3QRGZp/7nrcodAJsBk8TYOciFRbi5QXg6o1eyzaCoqtq8RmaXH/r1T7hDIBJgs3sZBLiQSm6ADAgBbW3ljsRZVzdCsWSQiUhomi7dxrkUSiV8YOLjF9JgrEtWOlwfJhcnibVzFhUScY9H0VOI8i/w0JCJSHCaLt7FmkURMFk1P7LLIZmgiIuVhsngbB7iQiMmi6XEFFyLzcyn/ptwhkIkwWbyNA1xIJK7ewmTRdFSclZvI7FwrLpM7BDIRJou3iYnBtWvArVvyxkLyEr8wcNoc02EzNBGRcjFZvM3LC3Bw0D0Wa5bIOrEZ2vRUbIYmIlIsJou3qVQc5EJAaWnVko9MFk2HK7gQESkXk8U7cPocEn/3jo6At7e8sViTqmZoWcMgIqIaMFm8A2sW6cIF3X2zZlyCzpQ4GpqISLmYLN6B0+eQmCxy9RbTqhoNzXSRiEhpmCzegdPnkPhFoVkzeeOwNmKyyGZoIiLlYbJ4BzZD053N0GQ6VaOhmS2SPLZt24Zhw4YhMDAQKpUKa9askfaVl5fjjTfeQPv27eHi4oLAwECMHTsWl+6qWcjLy8OYMWPg7u4OT09PTJgwAcXFxSZ+JUSGx2TxDhzgQkwW5cFWaJJbSUkJIiMjkZycXG3fjRs3sH//fkyfPh379+/H6tWrkZmZiUceeUSv3JgxY3D06FGkpKRg3bp12LZtGyZNmmSql2By/HJnPWzlDkBJ7qxZFAQOcLBGTBblIdYsshma5BIXF4e4uLga93l4eCAlJUVv22effYbu3bvj3LlzCA4OxvHjx7Fhwwbs3bsXXbt2BQAsXLgQgwcPxocffohAzsVFZow1i3cQr+WbN4H8fFlDIZkwWZSHmvMskpkpKCiASqWCp6cnACAtLQ2enp5SoggAMTExUKvV2L17d43HKC0tRWFhod6NSInqlSzOnTsX3bp1g5ubG/z8/DBixAhkZmbqlbl16xYSEhLg7e0NV1dXjBw5Ejk5OXplzp07hyFDhsDZ2Rl+fn54/fXXUVFRoVdmy5Yt6Ny5MxwcHNCyZUssX768Ya+wHpycgCZNdI/ZFG19KiqqVu9hsmhaKrAan8zHrVu38MYbb+Dpp5+Gu7s7ACA7Oxt+fn565WxtbeHl5YXs7OwajzN37lx4eHhIt6CgIKPHTtQQ9UoWt27dioSEBOzatQspKSkoLy/HwIEDUVJSIpV55ZVX8Msvv+CHH37A1q1bcenSJTz22GPS/srKSgwZMgRlZWXYuXMnvvzySyxfvhwzZsyQymRlZWHIkCHo168fMjIyMGXKFDz//PPYuHGjAV7yvXH6HOuVnQ1otYCtLXDX/3wysqrR0KxZJGUrLy/Hk08+CUEQsGjRokYda9q0aSgoKJBu58+fN1CURIZVrz6LGzZs0Hu+fPly+Pn5IT09Hb1790ZBQQGWLFmClStXon///gCAZcuWITw8HLt27cKDDz6I3377DceOHcPvv/8Of39/dOzYEe+++y7eeOMNzJw5E/b29li8eDHCwsLw0UcfAQDCw8Oxfft2fPzxx4iNjTXQS69Z06bA0aNMFq3RnXMsqtlBw6Q4wIXMgZgonj17Fps2bZJqFQFAo9EgNzdXr3xFRQXy8vKg0WhqPJ6DgwMcHByMGjORITTqI7GgoAAA4OXlBQBIT09HeXk5YmJipDJt2rRBcHAw0tLSAOj6dbRv3x7+/v5SmdjYWBQWFuLo0aNSmTuPIZYRj1ETQ/X94PQ51ov9FeWj4goupHBionjy5En8/vvv8L5rPdDo6Gjk5+cjPT1d2rZp0yZotVpERUWZOlwig2rwaGitVospU6agR48eaNeuHQBdnw17e3upw6/I399f6rORnZ2tlyiK+8V99ypTWFiImzdvwsnJqVo8c+fOxTvvvNPQlyPhxNzWi6u3yIfN0CS34uJinDp1SnqelZWFjIwMeHl5ISAgAI8//jj279+PdevWobKyUvq88vLygr29PcLDwzFo0CBMnDgRixcvRnl5ORITEzFq1CijjYTmjB1kKg2uWUxISMCRI0fw3XffGTKeBjNU3w+xVklMHMh6sGZRPtJnHnNFksm+ffvQqVMndOrUCQCQlJSETp06YcaMGbh48SLWrl2LCxcuoGPHjggICJBuO3fulI6xYsUKtGnTBgMGDMDgwYPRs2dPfP7553K9JCKDaVDNYmJiojThaLM7Plk1Gg3KysqQn5+vV7uYk5Mj9dnQaDTYs2eP3vHE0dJ3lrl7BHVOTg7c3d1rrFUEDNf3Q3w5bIa2PkwW5aNWsxma5NW3b997Tt1Ul2mdvLy8sHLlSkOGdU+siCdTqVfNoiAISExMxE8//YRNmzYhLCxMb3+XLl1gZ2eH1NRUaVtmZibOnTuH6OhoALp+HYcPH9brCJySkgJ3d3dERERIZe48hlhGPIYxiU2QrFm0PlwXWj5VA1z46UdkLni5Wo961SwmJCRg5cqV+Pnnn+Hm5ib12fDw8ICTkxM8PDwwYcIEJCUlwcvLC+7u7pg8eTKio6Px4IMPAgAGDhyIiIgIPPvss5g3bx6ys7Px1ltvISEhQaoZfOGFF/DZZ59h6tSpeO6557Bp0yZ8//33+N///mfgl1+dmCjk5gKlpQAHqlkP1izKp6rPorxxEBFRdfWqWVy0aBEKCgrQt29fvT4bq1atksp8/PHHGDp0KEaOHInevXtDo9Fg9erV0n4bGxusW7cONjY2iI6OxjPPPIOxY8di1qxZUpmwsDD873//Q0pKCiIjI/HRRx/hiy++MPq0OQDg7V2VIHKQi/XQalmzKKeq0dDMFonMBQfYWI961SzWpYnI0dERycnJNS7GLgoJCcH69evveZy+ffviwIED9QnPIFQqXbJw+rQuebirpZ0s1JUrQHm5bn7FWqZEIyPiPItERMrFqYdrwBHR1kf8XWs0gJ2dvLFYI6lmkckiEZHiMFmsAZNF68P+ivK6PRiaA1yIiBSIyWINmCxaHyaL8lKBU+cQESkVk8UaMFm0PkwW5cUVXIiIlIvJYg2YLFofLvWnDMwViYiUh8liDTgxt/VhzaK81Co2QxOZG365sx5MFmsgJgyXLwMVFfLGQqbBmkV5qTjAhYhIsZgs1sDPD7C11U3UfHuRGrJggsAJueVWlSzKGwcREVXHZLEGNjZAYKDuMZuiLd/168DNm7rHrFmUB5uhiYiUi8liLcQaJrHGiSyX+IXA2xtwdJQ3FmslruDC0dBERMrDZLEWYrJ4/ry8cZDxsQlafmyGJiJSLiaLtQgK0t0zWbR8HNwiv6rl/pgtEhEpDZPFWojJ4rlz8sZBxifWLDJZlI/YDM1ckYhIeZgs1oI1i9ZDrFkUf+dkeioOcCEiUiwmi7Vgsmg9OCG3/NScZ5HovlT3L0JkFEwWayEmi5cvA+Xl8sZCxsVkUX6q2x+DWuaKRGaDl6v1YLJYCz8/wM5O14fq0iW5oyFj4gAX+UmjofnxQ0SkOEwWa6FWsynaGhQVAQUFusfss3h/WVlZ6NevHyIiItC+fXuUlJQY5LicOoeISLls5Q5AyYKCgL/+YrJoycTfracn4OYmayhmYdy4cZg9ezZ69eqFvLw8ODg4GOS4VVPnGORwRER0m8oAnV2ZLN4Dp8+xfOLvNjhY3jjMwdGjR2FnZ4devXoBALy8vAx2bGnqHDZDE5kNDrixHmyGvgc2Q1s+a0oWt23bhmHDhiEwMBAqlQpr1qypViY5ORmhoaFwdHREVFQU9uzZI+07efIkXF1dMWzYMHTu3Blz5swxWGxshiYiUi4mi/fAZNHyib9ba+ivWFJSgsjISCQnJ9e4f9WqVUhKSsLbb7+N/fv3IzIyErGxscjNzQUAVFRU4I8//sC///1vpKWlISUlBSkpKQaJTRwNzVyRiEh5mCzeA5NFy2dNNYtxcXGYPXs2Hn300Rr3z58/HxMnTsT48eMRERGBxYsXw9nZGUuXLgUANG3aFF27dkVQUBAcHBwwePBgZGRk1Hq+0tJSFBYW6t1qo+I8i0REisVk8R6YLFo+8XdrDcnivZSVlSE9PR0xMTHSNrVajZiYGKSlpQEAunXrhtzcXFy/fh1arRbbtm1DeHh4rcecO3cuPDw8pFvQPapvq/osEhGR0jBZvAcxgbh6Fbh5U95YyDjEmkVraIa+l6tXr6KyshL+/v562/39/ZGdnQ0AsLW1xZw5c9C7d2906NABrVq1wtChQ2s95rRp01BQUCDdzt/jWxdHQxMRGYch/q9yNPQ9eHoCLi5ASYmuBuqBB+SOiAxJq2XNYn3FxcUhLi6uTmUdHBzqPLXOnVM7CIIgJY9EVEVp36WUFg8ZD2sW70GlYlO0JbtyBSgr0/2eAwPljkZePj4+sLGxQU5Ojt72nJwcaDQao5//ztSQtYtERMrCZPE+mCxaLrEJOjBQt7SjNbO3t0eXLl2QmpoqbdNqtUhNTUV0dLTRz39nTSJzRSLDKK/UouhWudxhkAVgM/R9MFm0XNbWBF1cXIxTp05Jz7OyspCRkQEvLy8EBwcjKSkJ8fHx6Nq1K7p3744FCxagpKQE48ePN3ps+jWLAjjdL1Hj9f9oC87n3cT+6Q/Dy8Ve7nDIjDFZvA8mi5bL2ga37Nu3D/369ZOeJyUlAQDi4+OxfPlyPPXUU7hy5QpmzJiB7OxsdOzYERs2bKg26MUY9PosGv1sRNbhfJ5uZObuv64hrn2AzNGQOWOyeB9MFi2XNc2xCAB9+/a97zyGiYmJSExMNFFEVVR31CSyzyKRYX2ZdobJIjUK+yzeh5hIMFm0PNbWDK1kqjv+E3F9aCLD2vVXHgpusu8iNRyTxfsQaxbFWiiyHNbWDK1kHA1NZFxlFVq5QyAzxmTxPsREoqgIKCiQNxYyrLNndfchIfLGQXeNhmaySESkKEwW78PFBWjSRPeYTdGW49YtQJxSkM3Q8tOrWWQzNBGRojBZrAMOcrE84u/SxQXw8pI3Frp7BRf54iDrtW3bNgwbNgyBgYFQqVRYs2aN3n5BEDBjxgwEBATAyckJMTExOHnypF6ZvLw8jBkzBu7u7vD09MSECRNQXFxswldhWvcbMEeWg8liHTBZtDxiE3RwsH6iQvLQGw0tYxxkvUpKShAZGYnk5OQa98+bNw+ffvopFi9ejN27d8PFxQWxsbG4deuWVGbMmDE4evQoUlJSsG7dOmzbtg2TJk0y1UsgMhpOnVMHTBYtjzi4hf0VleHutaGJTO1e654LgoAFCxbgrbfewvDhwwEAX331Ffz9/bFmzRqMGjUKx48fx4YNG7B371507doVALBw4UIMHjwYH374IQKtfU1RMmusWawDsU8bR0RbDg5uURZOyk1KlpWVhezsbMTExEjbPDw8EBUVhbS0NABAWloaPD09pUQRAGJiYqBWq7F79+4aj1taWorCwkK9W32wVYRMhcliHbBm0fJY24TcSsdJuUnJsrOzAaDaakb+/v7SvuzsbPj5+entt7W1hZeXl1TmbnPnzoWHh4d0CzKzebxUzFatBpPFOuBci5aHzdDKwmZoskbTpk1DQUGBdDvPGglSKCaLdRAaqrs/dw7Qcl5Ti8AJuZWFk3KTkmk0GgBAjjjf1m05OTnSPo1Gg9zcXL39FRUVyMvLk8rczcHBAe7u7no3IiVislgHTZsCNjZAeTlw+bLc0VBjCQKX+lMavUm5ZYyDqCZhYWHQaDRITU2VthUWFmL37t2Ijo4GAERHRyM/Px/p6elSmU2bNkGr1SIqKsrkMRMZEkdD14GtLdCsmW5QxNmzuuSRzNeVK0Bpqa7pk79LZdCvWWS6SKZXXFyMU6dOSc+zsrKQkZEBLy8vBAcHY8qUKZg9ezZatWqFsLAwTJ8+HYGBgRgxYgQAIDw8HIMGDcLEiROxePFilJeXIzExEaNGjeJIaJKVIbqWMlmso5AQXaJ45gzw0ENyR0ONITZBBwQAdnbyxkI6HA1Nctu3bx/69esnPU9KSgIAxMfHY/ny5Zg6dSpKSkowadIk5Ofno2fPntiwYQMcHR2ln1mxYgUSExMxYMAAqNVqjBw5Ep9++qnRYub3KjIVJot1JA6E4CAX8yc2QbO/onJwbWiSW9++fe9Zq61SqTBr1izMmjWr1jJeXl5YuXKlMcJTJLYCWA/2WawjsW8bB6uZP06bo0xivsi1oYmIlIXJYh1xYm7LwZpFZZLqFpkrEhkcv4RRYzBZrCMmi5aDNYvKJDZFa/mZRkSkKEwW60hMLMRl4sh8sWZRmcSaRdaAEBEpC5PFOhKTxYIC3Y3MF2sWlUnqs8hckYhIUZgs1pGrK+DtrXvM2kXzVVZWNbE6axaVRVwfmrkiEZGyMFmsB3H6HCaL5uv8eV3NlaMj4OsrdzSkR6pZZLpIRKQkTBbrgcmi+RMXaGjZElDzr19R1GyGJiJSJH5c1kNoqO6eyaL5ysrS3YeFyRsHVaeCAdakIiKT4fc668FksR7EmsUzZ2QNgxrh4kXdPfsrKo84wEXLqkUiIkVhslgPbIY2fxcu6O6bNpU3DqpOmjqHuSIRkaIwWawHJovmT0wWmzWTNw6qTpyUm7kiEZGyMFmsBzFZzM0Fbt6UNxZqGDHRZzO08lTVLDJdJCJSEiaL9dCkCeDmpnvMZf/MT2VlVbLYvLm8sVANxNHQ8kZBRER3YbJYDyoVB7mYs8uXdZNy29qyGVqJ1GIzNLNFohopbb4ApcVDxsNksZ7Yb9F8iQl+SAhgYyNrKFQDlfTJw2yRiEhJmCzWE5NF8yUmi+J8maQsHA1NRKRM9U4Wt23bhmHDhiEwMBAqlQpr1qzR2z9u3DioVCq926BBg/TK5OXlYcyYMXB3d4enpycmTJiA4uJivTKHDh1Cr1694OjoiKCgIMybN6/+r84ImCyaL3FCbiaLyiSOhtYyWSQiUpR6J4slJSWIjIxEcnJyrWUGDRqEy5cvS7dvv/1Wb/+YMWNw9OhRpKSkYN26ddi2bRsmTZok7S8sLMTAgQMREhKC9PR0fPDBB5g5cyY+//zz+oZrcGKiwT6L5oc1i8om1SyyGZrI8IxwWfFKtR629f2BuLg4xMXF3bOMg4MDNBpNjfuOHz+ODRs2YO/evejatSsAYOHChRg8eDA+/PBDBAYGYsWKFSgrK8PSpUthb2+Ptm3bIiMjA/Pnz9dLKuXAmkXzxWTRsEJDQ+Hu7g61Wo0mTZpg8+bNjTqeimtDExEpklH6LG7ZsgV+fn5o3bo1XnzxRVy7dk3al5aWBk9PTylRBICYmBio1Wrs3r1bKtO7d2/Y29tLZWJjY5GZmYnr16/XeM7S0lIUFhbq3YxBTBYvXQLKy41yCjISJouGt3PnTmRkZDQ6UdThaGgio+HQZWoEgyeLgwYNwldffYXU1FS8//772Lp1K+Li4lBZWQkAyM7Ohp+fn97P2NrawsvLC9nZ2VIZf39/vTLic7HM3ebOnQsPDw/pFmSkWZf9/AAHB0CrrVoNhJSvsrJqbkwmi8qkluZZZLZIRGQoKgN8UzB4sjhq1Cg88sgjaN++PUaMGIF169Zh79692LJli6FPpWfatGkoKCiQbufPnzfKedRqICxM9zgz0yinICO4dAmoqADs7ICAALmjqV1RURGmTJmCkJAQODk54aGHHsLevXsNeo77DVITJScnIzQ0FI6OjoiKisKePXv09qtUKvTp0wfdunXDihUrGh0Xm6GJiJTJ6FPnNG/eHD4+Pjh16hQAQKPRIDc3V69MRUUF8vLypH6OGo0GOTk5emXE57X1hXRwcIC7u7vezVjatdPdnzhhtFOQgYkjoYOClD3H4vPPP4+UlBR8/fXXOHz4MAYOHIiYmBhcvHixxvI7duxAeQ39IY4dO1btGhLVZZDaqlWrkJSUhLfffhv79+9HZGQkYmNj9a7d7du3Iz09HWvXrsWcOXNw6NCher5afYb49ktERIZn9GTxwoULuHbtGgJuV+dER0cjPz8f6enpUplNmzZBq9UiKipKKrNt2za9D8GUlBS0bt0aTZo0MXbI9yW2cLMZ2nyIyaJYK6xEN2/exP/93/9h3rx56N27N1q2bImZM2eiZcuWWLRoUbXyWq0WCQkJGD16tNTNAwAyMzPRv39/fPnllzWeJy4uDrNnz8ajjz5aayzz58/HxIkTMX78eERERGDx4sVwdnbG0qVLpTJNmzYFAAQEBGDw4MHYv39/Q186gKqaRS2rFokMbun2M3KHQGas3slicXExMjIykJGRAQDIyspCRkYGzp07h+LiYrz++uvYtWsXzpw5g9TUVAwfPhwtW7ZEbGwsACA8PByDBg3CxIkTsWfPHuzYsQOJiYkYNWoUAgMDAQCjR4+Gvb09JkyYgKNHj2LVqlX45JNPkJSUZLhX3gjiUnFGaukmIzCHZLGiogKVlZVwdHTU2+7k5ITt27dXK69Wq7F+/XocOHAAY8eOhVarxenTp9G/f3+MGDECU6dObVAcZWVlSE9PR0xMjN65YmJikJaWBkBXO1lUVARA9z9h06ZNaNu2bY3HS05ORkREBLp163bP83JSbiLjWbz1tNwhkBmrd7K4b98+dOrUCZ06dQIAJCUloVOnTpgxYwZsbGxw6NAhPPLII3jggQcwYcIEdOnSBX/88QccHBykY6xYsQJt2rTBgAEDMHjwYPTs2VNvDkUPDw/89ttvyMrKQpcuXfDqq69ixowZsk+bIxJrFpksmg9zSBbd3NwQHR2Nd999F5cuXUJlZSW++eYbpKWl4fLlyzX+TGBgIDZt2oTt27dj9OjR6N+/P2JiYmqsiayrq1evorKyssZBZuIAs5ycHPTs2RORkZF48MEHMXbs2FqTwYSEBBw7duy+fS/FSbmZKxIRKUu951ns27cvhHt89d+4ceN9j+Hl5YWVK1fes0yHDh3wxx9/1Dc8kwgO1t2Lo2tJ+cRpc5ScLALA119/jeeeew5NmzaFjY0NOnfujKefflqv28bdgoOD8fXXX6NPnz5o3rw5lixZIiVextK8eXMcPHjQKMe+1/8XIiIyPa4N3QDi1CsXLgBlZbKGQnVkDjWLANCiRQts3boVxcXFOH/+PPbs2YPy8nI0b9681p/JycnBpEmTMGzYMNy4cQOvvPJKo2Lw8fGBjY1NjYPMahtgZgjq2/+NmCoSmQd+r7MeTBYbwM8PcHLSXShsila+srKqwUjmMseii4sLAgICcP36dWzcuBHDhw+vsdzVq1cxYMAAhIeHY/Xq1UhNTcWqVavw2muvNfjc9vb26NKlC1JTU6VtWq0WqampiI6ObvBx70fFSbmJiBSp3s3QpBu1GRoKHD+uq7Fq0ULuiOhezp3TJSBOTsBd3fAUZ+PGjRAEAa1bt8apU6fw+uuvo02bNhg/fny1slqtFnFxcQgJCcGqVatga2uLiIgIpKSkoH///mjatGmNtYzFxcXSVFZA1SA1Ly8vBN/uY5GUlIT4+Hh07doV3bt3x4IFC1BSUlJjHIZS1XLObJGISEmYLDZQWJguWRT7wpFyiU3QoaF3JiTKVFBQgGnTpuHChQvw8vLCyJEj8d5778HOzq5aWbVajTlz5qBXr156S2NGRkbi999/h6+vb43n2LdvH/r16yc9F2cZiI+Px/LlywEATz31FK5cuYIZM2YgOzsbHTt2xIYNG6oNejEkjoYmujdeGtQQhlgVi8liA4nNmWIiQsplLv0VAeDJJ5/Ek08+WefyDz/8cI3bxdkKanK/QWqixMREJCYm1jmWxhIH5Wj5iUhEpCjss9hAYuLBmkXlM6dk0ZpV1SwyWyQiUhImiw0k1iwyWVQ+JotmQlwbWt4oiIjoLkwWG4jN0ObDXOZYtHbss0hkXpTeB5wMh8liA4mJx+XLwK1b8sZC98aaRfOgllZwYbZIZAzs4kENxWSxgby8AFdX3eOzZ+WNhWpXUgLk5uoem8sci9ZKqqXg5xlRo53KLa62jbkiNRSTxQZSqTjIxRyIvxsPD6BJE1lDofuQJuWWOQ4ic3GvZuC0v65V26Y1cLbI5NN6MFlsBPZbVD42QZsP8YPP0B9oRIZQWVmJ6dOnIywsDE5OTmjRogXeffddvaZdQRAwY8YMBAQEwMnJCTExMTh58qSMUevjlUUNxWSxEVizqHxMFs0Pc0VSovfffx+LFi3CZ599huPHj+P999/HvHnzsHDhQqnMvHnz8Omnn2Lx4sXYvXs3XFxcEBsbi1sK6djOa4saipNyNwJrFpWPyaL5UKnYDE3KtXPnTgwfPhxDhgwBAISGhuLbb7/Fnj17AOhqFRcsWIC33npLWsv9q6++gr+/P9asWYNRo0bJFruIg8eooViz2AhiAsJkUbk4bY754KTcpGQPPfQQUlNTceLECQDAwYMHsX37dsTFxQHQrbGenZ2NmJgY6Wc8PDwQFRWFtLS0Go9ZWlqKwsJCvZsx8dKihmLNYiO0aKG7P31a3jiodqxZNB/q219d+XlGSvTmm2+isLAQbdq0gY2NDSorK/Hee+9hzJgxAIDs7GwAqLZ+ur+/v7TvbnPnzsU777xj3MCJDIA1i40gJot5ebobKY+YLHLaHOVTcQkXUrDvv/8eK1aswMqVK7F//358+eWX+PDDD/Hll182+JjTpk1DQUGBdDt//rzB4q1poDRrFqmhWLPYCM7OQNOmwMWLwKlTQPfuckdEd7p+HSgo0D1msqh8KilX5CcaKc/rr7+ON998U+p72L59e5w9exZz585FfHw8NBoNACAnJwcBAQHSz+Xk5KBjx441HtPBwQEODg5Gj13Ea4saijWLjdSype5eQbMj0G1iraKfH+DiIm8sdH9c7o+U7MaNG1Cr9T8ybWxsoNVqAQBhYWHQaDRITU2V9hcWFmL37t2Ijo42aay3yiux/9x1k56TlEtVYz1z/bBmsZFatAC2buUgFyVif0Uzc7tqUctkkRRo2LBheO+99xAcHIy2bdviwIEDmD9/Pp577jkAutH8U6ZMwezZs9GqVSuEhYVh+vTpCAwMxIgRI4wSU21frBJW7Efqn7lGOeedMrOLjH4OUgYmi43UvLnu/q+/5I2DqmOyaF44GpqUbOHChZg+fTpeeukl5ObmIjAwEH/7298wY8YMqczUqVNRUlKCSZMmIT8/Hz179sSGDRvg6Oho0lhNkSgCNa8SQ5aJyWIjickiR0QrD5NF86Li+BZSMDc3NyxYsAALFiyotYxKpcKsWbMwa9Ys0wVGZALss9hITBaVS6ztZbJoHtTipNzMFonMQuN7wpG5YLLYSOIAl4sXgZs35Y2F9IkJvPg7ImWr+uBhtkhkDIb+IqZitmg1mCw2kpcX4Ompe8x+i8pRWVm1eos4HyYpm9QMzVyRyCh2Zxm2jyFzRevBZLGRVKqqZOTUKXljoSoXLgDl5YC9vW4uTFI+cXoH5opExrH5zytyh0BmismiAYjNnOy3qBxi4t68OWBjI28sVEesWSQyKyq2Q1sNJosGICaLrFlUDvF3wSZo8yF+7GiZLRKZBeaK1oPJogGwGVp5OLjF/HDqHCLzYoiVQcg8MFk0ANYsKg9rFs1P1dQ5TBeJjIFrQ1NDMVk0ADFZPHsWKCuTNxbSEZNF1iyaDzZpEREpE5NFA9BoAGdnQKvVJYwkL0FgM7Q5kkZDs/KDqEZK+z7FL3jWg8miAXD6HGXJzgZu3ADUaiAkRO5oqK6q+iwyWyQiUhImiwbC6XOUQ/wdhITo5lkk88KaRSIiZWGyaCCsWVQODm4xT+KcbVomi0Rmga3Q1oPJooFwRLRycHCLeRI/eDgamsg8sM+i9WCyaCBshlYO8XfAmkXzouY8i0RGZejvYZxn0XowWTQQMTH56y+gslLeWKwdaxbNk4qzchMRKRKTRQMJCgLs7HTzLF64IHc01o3JonmSmqGZLRKZBTZDWw8miwZiYwM0b657zH6L8snLA/LzdY/F3weZB6likbkikVlo4sLpJqwFk0UDEpui2W9RPmKiHhiomyidzMntSblljoKI6qZrSBO5QyATYbJoQBwRLT8ObjFfrFkkMi9shrYeTBYNiMmi/Nhf0XyJnztaZotERsErixqKyaIBsRlafkwWzRcHQxMRGZ4haoCZLBrQnTWLrByRB5uhzZea7dBERIpkK3cAliQ0FFCrgRs3gOxsICBA7oisD2sWTSc0NBTu7u5Qq9Vo0qQJNm/e3KjjsWaRyLzwe531YM2iAdnbA8HBusdsija94mIgJ0f3mDWLprFz505kZGQ0OlEEqlaD4AcQUd0s23FG7hDISjBZNDAOcpHPmTO6+yZNAE9POSOhBpFaoZktEtVFduGtepXnpUUNxWTRwJgsyicrS3cfFiZvHI1RWVmJ6dOnIywsDE5OTmjRogXeffddgyZQ27Ztw7BhwxAYGAiVSoU1a9bUWC45ORmhoaFwdHREVFQU9uzZo7dfpVKhT58+6NatG1asWNHouKpWcCEi4zDs1cWpc6wHk0UD44ho+YjveWiorGE0yvvvv49Fixbhs88+w/Hjx/H+++9j3rx5WLhwYY3ld+zYgfLy8mrbjx07hhyxTf4uJSUliIyMRHJycq1xrFq1CklJSXj77bexf/9+REZGIjY2Frm5uVKZ7du3Iz09HWvXrsWcOXNw6NCher5afeLa0Fpmi0REisJk0cBYsygfseKrY0dZw2iUnTt3Yvjw4RgyZAhCQ0Px+OOPY+DAgdVq9QBAq9UiISEBo0ePRmVlpbQ9MzMT/fv3x5dfflnjOeLi4jB79mw8+uijtcYxf/58TJw4EePHj0dERAQWL14MZ2dnLF26VCrTtGlTAEBAQAAGDx6M/fv313is5ORkREREoFu3bvd87VLNItvKiIgUhcmigYnJ4smT7B9iamKC3q6dvHE0xkMPPYTU1FScOHECAHDw4EFs374dcXFx1cqq1WqsX78eBw4cwNixY6HVanH69Gn0798fI0aMwNSpUxsUQ1lZGdLT0xETE6N3rpiYGKSlpQHQ1U4WFRUBAIqLi7Fp0ya0bdu2xuMlJCTg2LFj2Lt37z3Pq2aTFpGRGe4i23biCp5bvs9gxyNl49Q5Bta8ue6+oADIywO8veWNx5pYwhyLb775JgoLC9GmTRvY2NigsrIS7733HsaMGVNj+cDAQGzatAm9evXC6NGjkZaWhpiYGCxatKjBMVy9ehWVlZXw9/fX2+7v748///wTAJCTkyPVTFZWVmLixIn3rTm8H7EZml+yiJRv7NLqrR1kuZgsGpizMxAYCFy6pKtdZLJoGvn5uuQcqErYzdH333+PFStWYOXKlWjbti0yMjIwZcoUBAYGIj4+vsafCQ4Oxtdff40+ffqgefPmWLJkiZR4GUvz5s1x8OBBgx6zaoALs0UiIiVhM7QRtGmju79dCUMmINYq+vsDrq7yxtIYr7/+Ot58802MGjUK7du3x7PPPotXXnkFc+fOrfVncnJyMGnSJAwbNgw3btzAK6+80qgYfHx8YGNjU22ATE5ODjQaTaOOfU9cwIXIyHhxUcMwWTSC8HDd/fHj8sZhTSyhCRoAbty4AbVa/7K0sbGBVqutsfzVq1cxYMAAhIeHY/Xq1UhNTcWqVavw2muvNTgGe3t7dOnSBampqdI2rVaL1NRUREdHN/i49yNNym20MxCZN14bJBc2QxtB69a6+5Mn5Y3DmojJojk3QQPAsGHD8N577yE4OBht27bFgQMHMH/+fDz33HPVymq1WsTFxSEkJASrVq2Cra0tIiIikJKSgv79+6Np06Y11jIWFxfj1B3D9bOyspCRkQEvLy8E316CKCkpCfHx8ejatSu6d++OBQsWoKSkBOPHjzfaa+fS0EREhmeI/6lMFo1AnBRanCSajM9SahYXLlyI6dOn46WXXkJubi4CAwPxt7/9DTNmzKhWVq1WY86cOejVqxfs7e2l7ZGRkfj999/h6+tb4zn27duHfv36Sc+TkpIAAPHx8Vi+fDkA4KmnnsKVK1cwY8YMZGdno2PHjtiwYUO1QS+GJPZZ1DJbJCJSFCaLRiDWbp0+rcvoOcu98YlN/mIXAHPl5uaGBQsWYMGCBXUq//DDD9e4vVOnTrX+TN++fes0l2FiYiISExPrFIchqHmhEBEpEvssGkHLloCNDVBUpBsVTcYlCJaTLFozFdeGJiJSJCaLRmBvX9UcykEuxnflCnD9ui7ZaNVK7mioodhnkYhImZgsGok4yCUzU944rIH4HoeGAk5OsoZCjcLR0KRsFy9exDPPPANvb284OTmhffv22LevahUTQRAwY8YMBAQEwMnJCTExMTipoJGO/CJGDcVk0UiYLJqOOJ+l+J6TeWLNIinZ9evX0aNHD9jZ2eHXX3/FsWPH8NFHH6FJkyZSmXnz5uHTTz/F4sWLsXv3bri4uCA2Nha3bt2SMXKixqt3srht2zYMGzYMgYGBUKlUWLNmjd7+unyzysvLw5gxY+Du7g5PT09MmDABxcXFemUOHTqEXr16wdHREUFBQZg3b179X52MxIm5mSwan/gei+85mSeu4EJK9v777yMoKAjLli1D9+7dERYWhoEDB6LF7T5HgiBgwYIFeOuttzB8+HB06NABX331FS5dulTtc5LI3NQ7WSwpKUFkZCSSk5Nr3F+Xb1ZjxozB0aNHkZKSgnXr1mHbtm2YNGmStL+wsBADBw5ESEgI0tPT8cEHH2DmzJn4/PPPG/AS5cGaRdNhzaJlYM0iKdnatWvRtWtXPPHEE/Dz80OnTp3w3//+V9qflZWF7OxsxMTESNs8PDwQFRWFtLS0Go9ZWlqKwsJCvRuREtV76py4uDjExcXVuO/ub1YA8NVXX8Hf3x9r1qzBqFGjcPz4cWzYsAF79+5F165dAejmlhs8eDA+/PBDBAYGYsWKFSgrK8PSpUthb28vrZE7f/58vaTyTqWlpSgtLZWey33RiYnL2bPAjRu6NaPJOFizaBnEqXM4GpqU6K+//sKiRYuQlJSEf/zjH9i7dy/+/ve/w97eHvHx8cjOzgaAanOR+vv7S/vuNnfuXLzzzjtGj52osQzaZ7Eu36zS0tLg6ekpJYoAEBMTA7Vajd27d0tlevfurTfRcGxsLDIzM3H9+vUazz137lx4eHhIt6CgIEO+tHrz8QG8vHSPFdS/2eKUlgJ//aV7zJpF81bVDE2kPFqtFp07d8acOXPQqVMnTJo0CRMnTsTixYsbfMxp06ahoKBAup0/f96AERMZjkGTxbp8s8rOzoafn5/efltbW3h5eemVqekYd57jbkq76FQqNkWbwunTgFYLuLsDGo3c0VBjqKSaRZkDIapBQEAAIiIi9LaFh4fj3LlzAADN7X9AOTk5emVycnKkfXdzcHCAu7u73s2YeG1ZJ0Osd2Axo6FNfdHVBZNF47uzvyIXALEMHOBCStSjRw9k3vXP/MSJEwgJCQEAhIWFQaPRIDU1VdpfWFiI3bt3Izo62qSxEhmaQZPFunyz0mg0yM3N1dtfUVGBvLw8vTI1HePOc5gDMVkUExoyPPG9ZX9F88cBLqRkr7zyCnbt2oU5c+bg1KlTWLlyJT7//HMkJCQA0NWMT5kyBbNnz8batWtx+PBhjB07FoGBgRgxYoS8wRM1kkGTxbp8s4qOjkZ+fj7S09OlMps2bYJWq0VUVJRUZtu2bSgvL5fKpKSkoHXr1npzWikdp88xPvG9ZX9F86fipNykYN26dcNPP/2Eb7/9Fu3atcO7776LBQsWYMyYMVKZqVOnYvLkyZg0aRK6deuG4uJibNiwAY6OjjJGXoW19tRQ9R4NXVxcjFOnTknPs7KykJGRAS8vLwQHB0vfrFq1aoWwsDBMnz5d75tVeHg4Bg0aJHUMLi8vR2JiIkaNGoXAwEAAwOjRo/HOO+9gwoQJeOONN3DkyBF88skn+Pjjjw3zqk3kzmZoQWAzqTGIyymyZtH8sWaRlG7o0KEYOnRorftVKhVmzZqFWbNmmTAqIuOrd7K4b98+9OvXT3qelJQEAIiPj8fy5csxdepUlJSUYNKkScjPz0fPnj2rfbNasWIFEhMTMWDAAKjVaowcORKffvqptN/DwwO//fYbEhIS0KVLF/j4+GDGjBm1TpujVC1aADY2QHExcPkycDsXJgMRhKpm6PBweWOhxpNGQzNbJCJSlHoni3379r3nP/O6fLPy8vLCypUr73meDh064I8//qhveIpibw+EhQGnTulqF5ksGtbly0BRkS4hv72IApkxtZrN0ETGxO9h1FAWMxpaqcTmUQ5yMTyxCbp5c8DBQd5YqPFYs0hEpExMFo2M0+cYD5ugLQz7LBIRKRKTRSNjsmg8nDbHsnA0NJFxcZAlNRSTRSNjsmg8HAltWTgamujemOuRXJgsGpmYyJw5A9y8KWsoFofN0Jalam1oZotEdXWjrELuEMgKMFk0Ml9fwNtbf5oXarzCQuDiRd1j1ixaBtYsEtXfjJ+Pyh0CWQEmi0amUgHt2ukeHz4sbyyWRGzW12gAT09ZQyEDUd/OFjkamqjufjl4qc5leWlRQzFZNIH27XX3R47IG4clEWtpucyf5ahqhiYiIiVhsmgCYs0ik0XDEQe3sL+iBZFqFmWOg4iI9DBZNAGxZpHN0IbDwS2WhwNciOpPvFq0WgE/Z1zE2WslssZDlqney/1R/YkDMC5cAEpKABcXeeOxBMeO6e6ZLFoODnAharifDlzEqz8clDsMslCsWTQBLy/Ax0f3+ORJeWOxBKWluvW2AaBtW3ljIcPhpNxEDbf3TJ7cIZAFY7JoIg88oLvn5NyNl5kJVFYCHh5AQIDc0ZChsGaRyLh4aVFDMVk0ETFZPHFC3jgswdHb04q1bcvlqyyJ+vbv8kROkbyBEJkTZoBkAkwWTUTsW7d/v7xxWII7k0WyHFeKSgEA6Wevc65FIiPIOJ8vdwhkpqxvgEtlGZC39462rrvuG7tdEGosE9se2NpRgG8ZgIuGONe9zm+o12WM9+aucg04Zqty4PWhAh7pCuBYPV6Hod6bFhMB11CQYZ25dkN6XFqhhaOdjYzREFmeU7nFcodAZsr6ksXyfCClp8lPGwngf6/ffrLV5Ke3KPGR0L2hAJAhQwABg5gsGlnRrQomi0R1wKmmyBSsL1lU2QCuLW8/Fju8qWp+XqftdSsrQIUDB4DKShXatQecHOtxjPrE1aDX0MDXJkNcWkGFb1aooNUCjz+ugqtrI4/ZkLicAkGGd7O8UnpcdKscvm4OMkZDZJkEQYDq9v+301eKMeAjXe1FE2c7/O/vvRDo6YSW/1iPCi2TUKpifcmigzfwiOnnr1EBeOkdYPdu4LvvgKdGmDwEi/DXKSB+EeDoCIz9N9jr1oLcuiNZvH6jXMZIiMxPXQf7Xbh+E0FezgCA6WuqlhW7fqMcc3/9Ewuf7sRE0cIYYiAoP2pNqEMH3T1Xcmm4g7fnnG3VClDzr9eiPPNgiPT4JEdEE9VLfcaEVWoFvLQiHTtPX9Pb/svBS1i+I8vAkZEl4MetCUVE6O7FdY2p/nbt0t336CFvHGR4T3RphiAvJwDA6v0XZY6GyLxkXa37Mn8f/paJ9Yeza9w385djhgqJLAiTRRMSp89hsthw4rQ5kZH3LkfmR6VS4ZNRnQAAe8/m1evDj8jaXbh+s07lBAFYm3HJyNGQpWGyaEJisnjyJFDOLlkNIq4JLdbSkmXpHNwE/dv4QRCAhZu4NiaRofX+YDMqtFq5wyAzw2TRhIKCABcXoKKiam1jqrviYuDsWd1jTshtuf4+oBUA4OeMS7hw/cZ9ShNRfeUUlsodApkZJosmpFKxKboxxPfM3x/w9pY3FjKejkGe6NHSG5VaAct2nJE7HCJFEwe2cOlTMiYmiybGQS4Nx2X+rMfEXs0BAN/tOYeCm+yzQXQ/TBbJmJgsmpiYLB45cu9yVB2TRevR5wFftPZ3Q0lZJf4v/YLc4RARWTUmiyYmzrUozhdIdcfBLdZDpVJhzIPBAIAfmSwS1UqcQPt8Xt1GQxM1BJNFExOnfMnMBG7y2q4X1ixal2EdAmFvo8axy4U4eqlA7nCIiKwWk0UTCwgAfHwArbYq+aH740ho69PExR4xEX4AgJ84STcRkWyYLJqYSgV07Kh7zKbouhMHBGk0gJeXvLGQ6QzrEAgASDmeA6E+65kREZHBMFmUgdgUnZEhaxhmRayFZX9F69LrAV/Y26hx9toNnL5SLHc4RERWicmiDMRkkTWLdSeOHm/XTt44qEpoaCg6dOiAjh07ol+/fkY5h6uDLaJb6CbVTDmWa5RzEBHRvTFZlIGYLB46VDWhKt2bWAvL/orKsnPnTmRkZGDz5s1GO0e/1r66c52+arRzEJkDflyQXJgsyqBNG8DODigoqBq0QbUTBGDPHt3jBx+UNxYyvajmuprF9LPXUVHJNW2JiOrDEJVSTBZlYG9fVUPGpuj7O3sWKCrSJdjicomWKjQ0FCqVqtotISHBYOfYtm0bhg0bhsDAQKhUKqxZs6bGcsnJyQgNDYWjoyOioqKwR8zYb1OpVOjTpw+6deuGFStWGCy+u7X2d4O7oy1ulFXi6KVCo52HiIhqxmRRJhzkUneHDunuw8N1CaMl27t3Ly5fvizdUlJSAABPPPFEjeV37NiB8vLqy+EdO3YMOTk5Nf5MSUkJIiMjkZycXGscq1atQlJSEt5++23s378fkZGRiI2NRW5uVb/B7du3Iz09HWvXrsWcOXNwSPxFGZharUL3MN0Q+D1ZeUY5BxER1Y7Jokw4yKXuDh/W3bdvL28cpuDr6wuNRiPd1q1bhxYtWqBPnz7Vymq1WiQkJGD06NGorKyUtmdmZqJ///748ssvazxHXFwcZs+ejUcffbTWOObPn4+JEydi/PjxiIiIwOLFi+Hs7IylS5dKZZo2bQoACAgIwODBg7F///4aj5WcnIyIiAh069atTu9BTTqHNAEAHLyQ3+BjEBnSv/71L6hUKkyZMkXaduvWLSQkJMDb2xuurq4YOXJkrV/aiMwJk0WZMFmsu337dPfi/JTWoqysDN988w2ee+45qFSqavvVajXWr1+PAwcOYOzYsdBqtTh9+jT69++PESNGYOrUqQ0+b3p6OmJiYvTOFRMTg7S0NAC62smioiIAQHFxMTZt2oS2tYw+SkhIwLFjx7B3794GxQMA7QI9AADH2AxNCrB371785z//QQdx/dbbXnnlFfzyyy/44YcfsHXrVly6dAmPPfaYTFESGY6t3AFYKzFZ/OsvoLAQcHeXNx4lE7vKRUfLG4eprVmzBvn5+Rg3blytZQIDA7Fp0yb06tULo0ePRlpaGmJiYrBo0aIGn/fq1auorKyEv7+/3nZ/f3/8+eefAICcnBypZrKyshITJ05sVM3h/bQN1F0gWddKUFxaAVcH/usieRQXF2PMmDH473//i9mzZ0vbCwoKsGTJEqxcuRL9+/cHACxbtgzh4eHYtWsXHqxhdF5paSlKS0ul54WF/DJEysSaRZl4ewPNmukeG6mrl0W4fh24dEn3+K4v8RZvyZIliIuLQ2Bg4D3LBQcH4+uvv8aqVatga2uLJUuW1FgTaUjNmzfHwYMHcfDgQRw5cgQvv/yyUc/n7eqAAA9HCAJw/DI/UEk+CQkJGDJkiF7NOwCkp6ejvLxcb3ubNm0QHBws1cjfbe7cufDw8JBuQUFBRo2dqKGYLMqITdH3d+yY7j4oCHBzkzcWUzp79ix+//13PP/88/ctm5OTg0mTJmHYsGG4ceMGXnnllUad28fHBzY2NtX6WuXk5ECj0TTq2I0h1i4evVggWwxk3b777jvs378fc+fOrbYvOzsb9vb28PT01Nvu7++P7OzsGo83bdo0FBQUSLfz588bI2yiRmOyKCMmi/cnJovWtszfsmXL4OfnhyFDhtyz3NWrVzFgwACEh4dj9erVSE1NxapVq/Daa681+Nz29vbo0qULUlNTpW1arRapqamIlrEvwAP+um8Lp6+UyBYDWa/z58/j5ZdfxooVK+Do6GiQYzo4OMDd3V3vRmRohmhoYscfGTFZvD9rXBNaq9Vi2bJliI+Ph61t7ZeoVqtFXFwcQkJCpCboiIgIpKSkoH///mjatGmNtYzFxcU4deqU9DwrKwsZGRnw8vJCcHAwACApKQnx8fHo2rUrunfvjgULFqCkpATjx483/Auuo+a+rgDANaJJFunp6cjNzUXnzp2lbZWVldi2bRs+++wzbNy4EWVlZcjPz9erXZS7Rp7IEJgsykhMFg8fBiorARsbeeNRovR03b01jYT+/fffce7cOTz33HP3LKdWqzFnzhz06tUL9vb20vbIyEj8/vvv8PX1rfHn9u3bp7eWc1JSEgAgPj4ey5cvBwA89dRTuHLlCmbMmIHs7Gx07NgRGzZsqDboxZRa+LoAAP5izSLJYMCAATgszuN12/jx49GmTRu88cYbCAoKgp2dHVJTUzFy5EgAummszp07J2uNPJEhMFmUUcuWgJMTcPMmcPKkbhlAqlJZCYhT93XtKm8spjRw4EAIdVyf6eGHH65xe6dOnWr9mb59+9bp+ImJiUhMTKxTHKYg1ixmF97iiGgyOTc3N7Rr105vm4uLC7y9vaXtEyZMQFJSEry8vODu7o7JkycjOjq6xpHQROaEfRZlZGNTNcKXTdHV/fkncOMG4OICtG4tdzQkNw8nO/i4OgAA/mJTNCnQxx9/jKFDh2LkyJHo3bs3NBoNVq9eLXdYRI3Gr+Yyi4wEdu/WJYtPPSV3NMoiNkF37swmetJp7uuCq8Wl+OtKCTo085Q7HLJyW7Zs0Xvu6OiI5OTkey6lSWSOWLMoMw5yqZ24cos1NUHTvYV6OwMAzufdkDkSIiLrwWRRZmKymJEhaxiKxGSR7tasiS5ZvHD9psyREJmecafaJ6odk0WZiX0WL10Crl6VNxYlqagADhzQPWaySKJmTZwAABfyWbNIRGQqTBZl5uYGtGqleyzWpJFuMu5bt3RrZrdsKXc0pBRNPW8ni6xZJCIyGSaLChAVpbvftUveOJRETJy7dAHU/Cul25p56ZqhL+XfRKW2btMLERFR4/BjWAHEKbiYLFa5M1kkEvm7OcBWrUJ5pYDcoltyh0NEZBWYLCqA2CfvwAGgjnMxWzwObqGa2NqoEeCpW5eXTdFERKbBZFEBOnTQzSOYm6sb6GLtysqqphJiskh3a+YpjojmIBciIlNgsqgATk5A27a6x3v3yhuLEhw5oksYPT2B5s3ljoaURqxZzC4olTkSIiLrwGRRIdhvscqdTdAqTixGd/F31yWLOYXss0hEZApMFhWCyWIVcZk/Dm6hmvi76daHZrJIRGQaTBYVQkwW9+7VTUhtzcTJuDt3ljcOUiaNB2sWiYhMicmiQrRuDXh4ADduAIcPyx2NfCoqql5/p07yxkLK5Cc1Q7PPIhGRKTBZVAi1mpNzA0Bmpm7lFldXoEULuaMhJdLcThZzi25By4m5iYiMzuDJ4syZM6FSqfRubdq0kfbfunULCQkJ8Pb2hqurK0aOHImcnBy9Y5w7dw5DhgyBs7Mz/Pz88Prrr6PCCtpm2W+xqgk6MpIrt1DNfG/3WSyvFJB3o0zmaIiILJ9RPo7btm2Ly5cvS7ft27dL+1555RX88ssv+OGHH7B161ZcunQJjz32mLS/srISQ4YMQVlZGXbu3Ikvv/wSy5cvx4wZM4wRqqIwWQQ2bdLdi+8F0d3sbNTwcbUHwH6LRESmYJRk0dbWFhqNRrr5+PgAAAoKCrBkyRLMnz8f/fv3R5cuXbBs2TLs3LkTu25nSL/99huOHTuGb775Bh07dkRcXBzeffddJCcno6zMsmsRunfX3Z84AVy7Jm8sctm5U3ffv7+8cZCyidPn5LLfIhGR0RklWTx58iQCAwPRvHlzjBkzBufOnQMApKeno7y8HDExMVLZNm3aIDg4GGlpaQCAtLQ0tG/fHv7+/lKZ2NhYFBYW4ujRo7Wes7S0FIWFhXo3c+PtDTzwgO7xnj3yxiKHggJdn0UA6NZN3lhI2cRkMZs1i0RERmfwZDEqKgrLly/Hhg0bsGjRImRlZaFXr14oKipCdnY27O3t4enpqfcz/v7+yM7OBgBkZ2frJYrifnFfbebOnQsPDw/pFhQUZNgXZiLW3BS9f7/uPiQE8PWVNxZSNilZLGCySERkbLaGPmBcXJz0uEOHDoiKikJISAi+//57ODk5Gfp0kmnTpiEpKUl6XlhYaJYJ44MPAl99ZZ3JorjUIWsV6X7EQS5Xi9kMTUR0Lyo0fik0o4839fT0xAMPPIBTp05Bo9GgrKwM+fn5emVycnKg0WgAABqNptroaPG5WKYmDg4OcHd317uZI7FmcfduQKuVNxZTu3OZP6J7EQe4XCu27H7MRERKYPRksbi4GKdPn0ZAQAC6dOkCOzs7pKamSvszMzNx7tw5REdHAwCio6Nx+PBh5ObmSmVSUlLg7u6OiIgIY4cru/btAScn/f571oLJItWVjytrFomITMXgyeJrr72GrVu34syZM9i5cyceffRR2NjY4Omnn4aHhwcmTJiApKQkbN68Genp6Rg/fjyio6Px4O0qtYEDByIiIgLPPvssDh48iI0bN+Ktt95CQkICHBwcDB2u4tjaVjXDWlNTdF4ekJWle8w1oel+mCwSEZmOwZPFCxcu4Omnn0br1q3x5JNPwtvbG7t27YLv7RELH3/8MYYOHYqRI0eid+/e0Gg0WL16tfTzNjY2WLduHWxsbBAdHY1nnnkGY8eOxaxZswwdqmJZ4yAXsVaxeXPgrvFPRNV4sxmaiMhkDD7A5bvvvrvnfkdHRyQnJyM5ObnWMiEhIVi/fr2hQzMb1pgsbt2qu+/RQ944yDyINYtFpRW4VV4JRzsbmSMiIrJcXFBNgcQ1oo8cAYqK5I3FVDZv1t336ydvHGQe3B1tYW+j+/fFpmgiIuNisqhAgYFAcLBuNLTYPGvJiourps3p21fWUMhMqFQqNkUTEZkIk0WFsqam6J07gYoK3WTcYWFyR0PmgoNciIhMg8miQllTsii+xp495Y2DzIs41yKTRSIi42KyqFB3JouCIG8sxiYmi+JrJqoLb6lmkc3QRETGxGRRoTp1AuzsgNxc4MwZuaMxnvJyJovUMGyGJiIyDSaLCuXoqEsYASAtTd5YjOnwYeD6dd3cipGRckdD5qSqGZo1i0RExsRkUcHEPnybNskbhzGJtYpiTSpRXYk1i9dYs0hEZFRMFhVswADd/R9/yBuHMYnLhPfvL28cZH7YDE1EZBpMFhVM7MN34gRw9aq8sRiDIOimzQE4vyLVn48b51kkIjIFJosK5uUFhIfrHos1cJbk1CkgOxuwtQW6dJE7GjI33i66msW8G2WoqNTKHA0RkeVisqhwcXG6+y1bZA3DKMTlv7t1A5yc5I2FzI+Xiz3UKl0Ndd4N1i6S5Vu+84zcIZCVYrKocD166O7F5lpLsn+/7n7gQHnjIPNko1bBy4VN0WQdzlwtQcHNcrnDICvFZFHhHnpId3/4MJCXJ28shiauB921q7xxkPkSm6I5yIUsHRNFkhOTRYXTaIC2bXVNbZbUb/HcOeD4cUClArp3lzsaMlfiIBcmi2Rsc+fORbdu3eDm5gY/Pz+MGDECmZmZemVu3bqFhIQEeHt7w9XVFSNHjkROTo5Bzm/hC3mREQkG+OthsmgGHn5Yd29JyeLKlbr7nj0BPz95YyHzVTXXIpuhybi2bt2KhIQE7Nq1CykpKSgvL8fAgQNRUlIilXnllVfwyy+/4IcffsDWrVtx6dIlPPbYYzJGTWQYtnIHQPfXpw+wYIFlzbeYnq67HzxY3jjIvInN0FdYs0hGtmHDBr3ny5cvh5+fH9LT09G7d28UFBRgyZIlWLlyJfrfnjh22bJlCA8Px65du/BgDeuZlpaWorS06m+3sLDQuC+CqIFYs2gGxJVcjh2zjPkWtdqq0d3R0bKGQmZOaoYuYs0imVZBQQEAwMvLCwCQnp6O8vJyxMTESGXatGmD4OBgpNWyZuvcuXPh4eEh3YKCgowfOFEDMFk0Az4+QESE7rElLP23f78u6XV3rxrAQ9QQPrdrFq+VsGaRTEer1WLKlCno0aMH2rVrBwDIzs6Gvb09PD099cr6+/sjOzu7xuNMmzYNBQUF0u38+fO1nlMQ2GuR5MNk0UwMGaK7X7tW3jgMQWzNiYnhetDUOBzgQnJISEjAkSNH8N133zXqOA4ODnB3d9e7ERmaCqpGH4PJopkYOlR3/9tvumZcc/bbb7r7O1priBqEA1zI1BITE7Fu3Tps3rwZzZo1k7ZrNBqUlZUhPz9fr3xOTg40Go2JoyQyLCaLZuLBBwEXF+DKFeDQIbmjabjcXGDHDt3j233AiRrM+45kkc10ZEyCICAxMRE//fQTNm3ahLCwML39Xbp0gZ2dHVLvmLYiMzMT586dQ7QBOmfzr5vkxNHQZsLeHujbF/jf/3Q1cx07yh1Rw+zcqasZDQ4GWreWOxoyd963V3Apq9Si8GYFPJzZr4GMIyEhAStXrsTPP/8MNzc3qR+ih4cHnJyc4OHhgQkTJiApKQleXl5wd3fH5MmTER0dXeNI6PridyGSE2sWzYi4LN7GjfLG0RjietDDh8sbB1kGRzsbuDnqvvNe5SAXMqJFixahoKAAffv2RUBAgHRbtWqVVObjjz/G0KFDMXLkSPTu3RsajQarV6+WMWoiw2DNohmJiwNeflk332JREeDmJndE9SMIVcki51ckQ/FxdUDRrQpcLSpFC19XucMhC1WXbg6Ojo5ITk5GcnKyCSIiMh3WLJqRVq2AFi2A8nLzXM3l0CHg4kXAyUnXpE5kCD6u4ohoDnIhS8Z2aJIPk0UzExenu//1V3njaAixVnHAAMDRUd5YyHJII6LZDE1EZBRMFs2M2Hz766/m1+H5f//T3YtzRhIZgrdYs1jEZJGIyBiYLJqZvn11tXLnzwNHj8odTd3l5QHiildi7SiRIYg1i1fYDE0WzNwqB8iyMFk0M3f29xNr6szBxo26KXPatgVCQuSOhgwhNDQUHTp0QMeOHdGvXz/Z4qiamJs1i0RExsBk0QyJzbjffCNvHPUh9ldkE7Rl2blzJzIyMrB582bZYqga4MJkkYjIGJgsmqEnnwRUKuDIEeDSJbmjub/KyqoBOZwyhwytaoALm6GJiIyByaIZ8vMDunbVPV63Tt5Y6mLHDuDaNcDDA3joIbmjUb6LFy/imWeegbe3N5ycnNC+fXvs27fPYMfftm0bhg0bhsDAQKhUKqxZs6bGcsnJyQgNDYWjoyOioqKwZ88evf0qlQp9+vRBt27dsGLFCoPFV19issgBLmTJ2GWR5MRk0Uw9/rju/rvv5I2jLj74QHc/aBBgx9XY7un69evo0aMH7Ozs8Ouvv+LYsWP46KOP0KRJkxrL79ixA+Xl5dW2Hzt2DDk5OTX+TElJCSIjI+85cfCqVauQlJSEt99+G/v370dkZCRiY2ORm5srldm+fTvS09Oxdu1azJkzB4dkWrRcHA1dUlaJm2WVssRARGTJmCyaqaee0t1v2aLspujycmDDBt3jJ56QNxZz8P777yMoKAjLli1D9+7dERYWhoEDB6JFixbVymq1WiQkJGD06NGorKxKkjIzM9G/f398+eWXNZ4jLi4Os2fPxqOPPlprHPPnz8fEiRMxfvx4REREYPHixXB2dsbSpUulMk2bNgUABAQEYPDgwdi/f3+Nx0pOTkZERAS6detWp/egvlwdbOFgq/tXxn6LRESGx2TRTIWE6Jp0BQFQ8tKj+/cDFRW6x8OGyRuLOVi7di26du2KJ554An5+fujUqRP++9//1lhWrVZj/fr1OHDgAMaOHQutVovTp0+jf//+GDFiBKZOndqgGMrKypCeno6YmBi9c8XExCDt9vxHJSUlKCoqAgAUFxdj06ZNaNu2bY3HS0hIwLFjx7B3794GxXM/KpWqqimaySIRkcExWTRjI0bo7n/6SdYw7umXX3T3jz0G2NvLG4s5+Ouvv7Bo0SK0atUKGzduxIsvvoi///3vtdYSBgYGYtOmTdi+fTtGjx6N/v37IyYmBosWLWpwDFevXkVlZSX8/f31tvv7+yM7OxsAkJOTg549eyIyMhIPPvggxo4da7Saw7rgkn9k6TjPIsnJVu4AqOEeewyYOhXYtg0oLATc3eWOSF9lJSC2WrIJum60Wi26du2KOXPmAAA6deqEI0eOYPHixYiPj6/xZ4KDg/H111+jT58+aN68OZYsWQKVSmXUOJs3b46DBw8a9Rz1wbkWiYiMhzWLZqxFC+CBB3TNvD//LHc01X3xBXD5sq5G8R7d4+gOAQEBiIiI0NsWHh6Oc+fO1fozOTk5mDRpEoYNG4YbN27glVdeaVQMPj4+sLGxqTZAJicnBxqNplHHNhZvzrVIRGQ0TBbN3DPP6O6XLJE3jpokJOjuNRrAwUHeWMxFjx49kJmZqbftxIkTCKll2ZurV69iwIABCA8Px+rVq5GamopVq1bhtddea3AM9vb26NKlC1JTU6VtWq0WqampiI6ObvBxjamqzyKbockyCWyHJhkxWTRz48YBajWwdStw8qTc0VQpLdU1QwPArFnyxmJOXnnlFezatQtz5szBqVOnsHLlSnz++edIEDPvO2i1WsTFxSEkJASrVq2Cra0tIiIikJKSgmXLluHjjz+u8RzFxcXIyMhARkYGACArKwsZGRl6tZdJSUn473//iy+//BLHjx/Hiy++iJKSEowfP94or7ux/N0dAQDZBbdkjoSIyPKwz6KZCwoCYmN1K6QsXQrMnSt3RDriwBagqvaT7q9bt2746aefMG3aNMyaNQthYWFYsGABxowZU62sWq3GnDlz0KtXL9jfMXooMjISv//+O3x9fWs8x759+/TWck5KSgIAxMfHY/ny5QCAp556CleuXMGMGTOQnZ2Njh07YsOGDdUGvShFgIcuWbxccFPmSIiILA+TRQvw/PO6ZPFf/wLee09X0yi3L77Q3f/zn4CNjbyxmJuhQ4di6NChdSr78MMP17i9U6dOtf5M375969SklZiYiMTExDrFIbdATycAwCXWLBIRGZwC0gpqrDvXWxaTNDlduAD89pvu8XPPyRsLWQexZvFqcSnKKrQyR0NkeOyxSA1liMkxmCxaAEfHqsebNskXh2j5ct2cYD17As2byx0NWQMvF3vY26ohCEBOIWsXyfIkbz4ldwhkxZgsWoh163T369cDJSXyxSEIwNdf6x7XMi0gkcGpVKo7+i0yWSTL88fJq3KHQFaMyaKFiIvT1eIVFQGffSZfHKmpwIkTgIsL8OST8sVB1oeDXIiIjIPJooVQq3WruQDAm29WTVtjauJsLc89p7wVZciyBXrcHuSSz5pFIiJDYrJoQZ5+umr9ZTkm6T5+XNcMrlIBL79s+vOTddOwZpGIyCiYLFoQd3cgKkr3+KuvTH/+Tz7R3T/yiG4pQiJTEqfPuXidySJZly93npE7BLJwTBYtzLffAra2wI4dwO7dpjvv1atVCWojlyYmapBQbxcAQNY1GUd4Ecngw42Z9y9E1AhMFi1M06bAsGG6x48+arrzLloE3LwJdO4M9O5tuvMSicJ8dcniuWs3UFHJuRaJiAyFyaIFev553f3ly8Ady/0aTWEhMG+e7vGrrxpmAlCi+gpwd4SDrRoVWgEX89kUTURkKEwWLdDgwUBwsO6xKdaKfvhhoLgYCAsDRo0y/vmIaqJWq6Sm6L+usimaiMhQmCxaKHFi7MWLgT17jHeevLyq4z/7rDLWpSbrFeZzu9/iFSaLRESGwo92C9W7N9Cype7xlCm6lVWMQZzb8e7HRHIQ+y2evlIscyRERJaDyaIF27hRd5+WVlXTaEjHj1fN57h9u27VFiI5tdG4AQCOXiqUORIi0ykqrZA7BLJwTBYtWPPmVbV98fG6AS+GIghARITucdOmQI8ehjs2UUO1b+oBADh+uZAjoomIDITJooW7c87DOXMMd9z27ase/+tfhjsuUWOEervA1cEWpRVanMxlUzQRkSEwWbRwGk1VMvfvfwP79jX+mBcuAEeP6h4/8ADwzDONPyaRIajVKrQN1C1KnnE+X95giIgsBJNFK/DGG7opbbRaIDoauHat4ccqKwPatKl6bojkk8iQopp7AwC2n7oqcyRERJaByaKV+OwzwNcXqKgAfHyA/PyGHcfNDSi5PSvJxo2650RK0ruVDwBg+8mrqNQaaRoAIiIrwmTRSnh7A6tWVT0PCgJu3arfMQYP1tUsAsD06cDAgYaLj8hQOgZ5wsPJDgU3y7GDtYtERI2m6GQxOTkZoaGhcHR0RFRUFPYYc3ZpK9CvH/D997rHxcWAk5NuUu37KS7WLeH366+65w89BMyaZbw4iRrD1kaN4R0DAQArdp+VORqyRkr77Fr90kOY3L8lnu8ZJlsMTnY2+N/fe8p2fmocxSaLq1atQlJSEt5++23s378fkZGRiI2NRW5urtyhmbXHHwc6dKh67u0N/P577eXnzq3e1Lx5s3FiIzKUMVEhUKmAjUdzsPuvRnTSJaonJX52dQ5uglcHtsZbQyNki6F7mBciAtxlOz81jmKTxfnz52PixIkYP348IiIisHjxYjg7O2Pp0qVyh2bWVCrg4EH91VYefli3XaUCzp3T1SR+/bXu+T/+UVXuiSd08yva25s+bqL6aK1xw5NdggAAL3yTjl8PX0ZZBeddJONT2mfXG4Pa6D2f82j7WkoaT/82fvjXSNOflwzHVu4AalJWVob09HRMmzZN2qZWqxETE4O0tLQaf6a0tBSlpaXS88JCruBwL++/r5tIe/hw/e0hITWXT0jQDZIhMhfTh0XgeHYhDl0owIsr9sNWrYLGwxEu9rZwsFNDdWdhVdUzVc2bScG6hjTBP4fIV2smqu9nlyk+tx7t1FTv+eioYAR4OGL88r01lrezUaG80rADw5aO6wYAEIy17iwZnSKTxatXr6KyshL+/v562/39/fHnn3/W+DNz587FO++8Y4rwLMYjj+gGrPzjH8CHH9ZcZvRo4IsvdP0bicyJq4MtVk2KxsJNJ7Fq73lcKynDhes35Q6LjMDLWRnNHfX97DLF55ans121beH3aA5+oU8LLNx0ymDnf7C5l8GORQ0T4OHY6GMoMllsiGnTpiEpKUl6XlhYiKCgIBkjMg92dsAHH+hqGi9f1q0j7eAAxMQAjo6sWSHz5mRvg6mD2uD12Na4VHALOYW3cLOsEqUVlQB03SruvAcA8aEgCFDxAjALPq7KSBbrqz6fW3++Owjf7jmHgpvlyCkshbO9DS7l34QgAEM6BGDDkWyk/pkDO7Ua7k52SOzfEr1a+cDRzqbasTQejliT0AO7/rqGqDAvtPRzxcncYqhVKnRo6oFnHgxB4sr92HvmOnzdHBAfHYIPfztR6+t4qmsQLhXcxMMR/thwJBs7T1/D4me6oFkTJ7T0c5XKqVQq7P7HABTdKseZqzcQ5OWM6WuOoFcrH7Twc8Xvx3OQV1IGJzsb2NmoEeDhiMsFt+Bsb4PswlvYknkFzX1d8EhkIBb8frLWeDTujii6VY6Sskq97a4OtvjPs10w5ovdAIBerXzw2sDW8HKxx8vfHcDQDoHoEtIEu/66hhBvFwR5OSHIyxkz1x7FmKgQXCsuRdEt3Trc3cO8cCKnCEW3KrD24CUk9GuJS/k3EejpCA8ne2Scz0eItzNCvJwx85ej6POAL/zdHeHmaAetIOB83g1oBcDRTo3zebovsXHtNDh+uRAh3i5Yd+gSrhaX4p+DI7Biz1n0aumLSwU30drfDelnr+Ovq8V4rkcYlmzPwuioYFzKv4Wsq8Vo6acbZODv7oC5v/6Jyf1bws2x+heG+lIJCqwXLisrg7OzM3788UeMGDFC2h4fH4/8/Hz8/PPP9z1GYWEhPDw8UFBQAHd3dqol4+Lf2/3xPSJTM/XfXGM/u3iNkKnV9W9OkQNc7O3t0aVLF6SmpkrbtFotUlNTER0dLWNkRERENeNnF1kqxTZDJyUlIT4+Hl27dkX37t2xYMEClJSUYPz48XKHRkREVCN+dpElUmyy+NRTT+HKlSuYMWMGsrOz0bFjR2zYsKFax2EiIiKl4GcXWSJF9lk0BPb9IFPi39v98T0iUzO3vzlzi5fMn1n3WSQiIiIiZWCySERERES1YrJIRERERLViskhEREREtWKySERERES1YrJIRERERLViskhEREREtWKySERERES1YrJIRERERLViskhEREREtWKySERERES1spU7AGMRl7wuLCyUORKyBuLfmYUutW4QvCbJ1MztuuQ1QqZW12vEYpPFoqIiAEBQUJDMkZA1KSoqgoeHh9xhKBKvSZKLuVyXvEZILve7RlSCuXzlqietVotLly7Bzc0NKpVKb19hYSGCgoJw/vx5uLu7yxShMvG9qd293htBEFBUVITAwECo1ezdURNrvyb5Gk3P3K5LXiN8jaZW12vEYmsW1Wo1mjVrds8y7u7uivhlKRHfm9rV9t6YQ82FnHhN6vA1mpY5XZe8RnT4Gk2rLteI8r9qEREREZFsmCwSERERUa2sMll0cHDA22+/DQcHB7lDURy+N7Xje2M81vDe8jVSY1jDe8vXqFwWO8CFiIiIiBrPKmsWiYiIiKhumCwSERERUa2YLBIRERFRrZgsEhEREVGtrC5ZTE5ORmhoKBwdHREVFYU9e/bIHZLJzZw5EyqVSu/Wpk0baf+tW7eQkJAAb29vuLq6YuTIkcjJyZExYuPatm0bhg0bhsDAQKhUKqxZs0ZvvyAImDFjBgICAuDk5ISYmBicPHlSr0xeXh7GjBkDd3d3eHp6YsKECSguLjbhqzBv5nJdzp07F926dYObmxv8/PwwYsQIZGZm6pWpy/Vz7tw5DBkyBM7OzvDz88Prr7+OiooKvTJbtmxB586d4eDggJYtW2L58uXGfnnV/Otf/4JKpcKUKVOkbZb0+swJrxFl/g1ZzTUiWJHvvvtOsLe3F5YuXSocPXpUmDhxouDp6Snk5OTIHZpJvf3220Lbtm2Fy5cvS7crV65I+1944QUhKChISE1NFfbt2yc8+OCDwkMPPSRjxMa1fv164Z///KewevVqAYDw008/6e3/17/+JXh4eAhr1qwRDh48KDzyyCNCWFiYcPPmTanMoEGDhMjISGHXrl3CH3/8IbRs2VJ4+umnTfxKzJM5XZexsbHCsmXLhCNHjggZGRnC4MGDheDgYKG4uFgqc7/rp6KiQmjXrp0QExMjHDhwQFi/fr3g4+MjTJs2TSrz119/Cc7OzkJSUpJw7NgxYeHChYKNjY2wYcMGk73WPXv2CKGhoUKHDh2El19+2eJenznhNaLMvyFrukasKlns3r27kJCQID2vrKwUAgMDhblz58oYlem9/fbbQmRkZI378vPzBTs7O+GHH36Qth0/flwAIKSlpZkoQvncnSxqtVpBo9EIH3zwgbQtPz9fcHBwEL799ltBEATh2LFjAgBh7969Uplff/1VUKlUwsWLF00Wu7ky5+syNzdXACBs3bpVEIS6XT/r168X1Gq1kJ2dLZVZtGiR4O7uLpSWlgqCIAhTp04V2rZtq3eup556SoiNjTX2SxIEQRCKioqEVq1aCSkpKUKfPn2kD0JLeX3mhteI8v6GrO0asZpm6LKyMqSnpyMmJkbaplarERMTg7S0NBkjk8fJkycRGBiI5s2bY8yYMTh37hwAID09HeXl5XrvU5s2bRAcHGyV71NWVhays7P13g8PDw9ERUVJ70daWho8PT3RtWtXqUxMTAzUajV2795t8pjNiblflwUFBQAALy8vAHW7ftLS0tC+fXv4+/tLZWJjY1FYWIijR49KZe48hljGVO9JQkIChgwZUi0GS3l95oTXiI7S/oas7RqxleWsMrh69SoqKyv1fjkA4O/vjz///FOmqOQRFRWF5cuXo3Xr1rh8+TLeeecd9OrVC0eOHEF2djbs7e3h6emp9zP+/v7Izs6WJ2AZia+5pr8bcV92djb8/Pz09tva2sLLy8sq37P6MOfrUqvVYsqUKejRowfatWsHAHW6frKzs2t8veK+e5UpLCzEzZs34eTkZIyXBAD47rvvsH//fuzdu7faPkt4feaG10jVfnHfvcrwGjEOq0kWqUpcXJz0uEOHDoiKikJISAi+//57/pMmqqOEhAQcOXIE27dvlzsUgzl//jxefvllpKSkwNHRUe5wyMzxGrEcVtMM7ePjAxsbm2ojknJycqDRaGSKShk8PT3xwAMP4NSpU9BoNCgrK0N+fr5eGWt9n8TXfK+/G41Gg9zcXL39FRUVyMvLs8r3rD7M9bpMTEzEunXrsHnzZjRr1kzaXpfrR6PR1Ph6xX33KuPu7m7UL3Tp6enIzc1F586dYWtrC1tbW2zduhWffvopbG1t4e/vb9avzxzxGqnaL+67VxleI8ZhNcmivb09unTpgtTUVGmbVqtFamoqoqOjZYxMfsXFxTh9+jQCAgLQpUsX2NnZ6b1PmZmZOHfunFW+T2FhYdBoNHrvR2FhIXbv3i29H9HR0cjPz0d6erpUZtOmTdBqtYiKijJ5zObE3K5LQRCQmJiIn376CZs2bUJYWJje/rpcP9HR0Th8+LDeF4yUlBS4u7sjIiJCKnPnMcQyxn5PBgwYgMOHDyMjI0O6de3aFWPGjJEem/PrM0e8RnSU8jdktdeILMNqZPLdd98JDg4OwvLly4Vjx44JkyZNEjw9PfVGJFmDV199VdiyZYuQlZUl7NixQ4iJiRF8fHyE3NxcQRB0w/6Dg4OFTZs2Cfv27ROio6OF6OhomaM2nqKiIuHAgQPCgQMHBADC/PnzhQMHDghnz54VBEE3dY6np6fw888/C4cOHRKGDx9e49Q5nTp1Enbv3i1s375daNWqFafOqSNzui5ffPFFwcPDQ9iyZYve1FM3btyQytzv+hGnzRg4cKCQkZEhbNiwQfD19a1x2ozXX39dOH78uJCcnCzbtBl3jvQUBMt7feaA14iy/4as4RqxqmRREARh4cKFQnBwsGBvby90795d2LVrl9whmdxTTz0lBAQECPb29kLTpk2Fp556Sjh16pS0/+bNm8JLL70kNGnSRHB2dhYeffRR4fLlyzJGbFybN28WAFS7xcfHC4Kgmz5n+vTpgr+/v+Dg4CAMGDBAyMzM1DvGtWvXhKefflpwdXUV3N3dhfHjxwtFRUUyvBrzZC7XZU1/JwCEZcuWSWXqcv2cOXNGiIuLE5ycnAQfHx/h1VdfFcrLy/XKbN68WejYsaNgb28vNG/eXO8cpnT3B6GlvT5zwWtEuX9D1nCNqARBEExdm0lERERE5sFq+iwSERERUf0xWSQiIiKiWjFZJCIiIqJaMVkkIiIioloxWSQiIiKiWjFZJCIiIqJaMVkkIiIioloxWSQiIiKiWjFZJCKyUCqVCmvWrJE7DCLF4jVSN0wWiYiMYNy4cVCpVNVugwYNkjs0IkXgNWI+bOUOgIjIUg0aNAjLli3T2+bg4CBTNETKw2vEPLBmkYjISBwcHKDRaPRuTZo0AaBr/lq0aBHi4uLg5OSE5s2b48cff9T7+cOHD6N///5wcnKCt7c3Jk2ahOLiYr0yS5cuRdu2beHg4ICAgAAkJibq7b969SoeffRRODs7o1WrVli7dq1xXzRRPfAaMQ9MFomIZDJ9+nSMHDkSBw8exJgxYzBq1CgcP34cAFBSUoLY2Fg0adIEe/fuxQ8//IDff/9d74Nu0aJFSEhIwKRJk3D48GGsXbsWLVu21DvHO++8gyeffBKHDh3C4MGDMWbMGOTl5Zn0dRI1FK8RhRCIiMjg4uPjBRsbG8HFxUXv9t577wmCIAgAhBdeeEHvZ6KiooQXX3xREARB+Pzzz4UmTZoIxcXF0v7//e9/glqtFrKzswVBEITAwEDhn//8Z60xABDeeust6XlxcbEAQPj1118N9jqJGorXiPlgn0UiIiPp168fFi1apLfNy8tLehwdHa23Lzo6GhkZGQCA48ePIzIyEi4uLtL+Hj16QKvVIjMzEyqVCpcuXcKAAQPuGUOHDh2kxy4uLnB3d0dubm5DXxKRQfEaMQ9MFomIjMTFxaVak5ehODk51amcnZ2d3nOVSgWtVmuMkIjqjdeIeWCfRSIimezatava8/DwcABAeHg4Dh48iJKSEmn/jh07oFar0bp1a7i5uSE0NBSpqakmjZnIlHiNKANrFomIjKS0tBTZ2dl622xtbeHj4wMA+OGHH9C1a1f07NkTK1aswJ49e7BkyRIAwJgxY/D2228jPj4eM2fOxJUrVzB58mQ8++yz8Pf3BwDMnDkTL7zwAvz8/BAXF4eioiLs2LEDkydPNu0LJWogXiPmgckiEZGRbNiwAQEBAXrbWrdujT///BOAbhTmd999h5deegkBAQH49ttvERERAQBwdnbGxo0b8fLLL6Nbt25wdnbGyJEjMX/+fOlY8fHxuHXrFj7++GO89tpr8PHxweOPP266F0jUSLxGzINKEARB7iCIiKyNSqXCTz/9hBEjRsgdCpEi8RpRDvZZJCIiIqJaMVkkIiIiolqxGZqIiIiIasWaRSIiIiKqFZNFIiIiIqoVk0UiIiIiqhWTRSIiIiKqFZNFIiIiIqoVk0UiIiIiqhWTRSIiIiKqFZNFIiIiIqrV/wPn7tgYPRGQTwAAAABJRU5ErkJggg==\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Now let's attempt to use this in a normal model\n",
        "model = tf.keras.Sequential([\n",
        "    tf.keras.layers.Input(shape=(1,)),\n",
        "    tf.keras.layers.Dense(128, activation='relu'),\n",
        "    tf.keras.layers.Dense(1)\n",
        "])\n",
        "#opt = tf.keras.optimizers.Adam(learning_rate=0.01)\n",
        "opt = RecordingOptimizer(learning_rate=0.01)\n",
        "model.compile(optimizer=opt, loss='mse', metrics=['mse'])\n",
        "#history = model.fit(x, y, epochs=50, batch_size=len(x), verbose=0, callbacks = [LessVerboseEpochLogger(10)])\n",
        "history = model.fit(x, y, epochs=50, batch_size=len(x), verbose=0)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "ZKUyekptkOX0",
        "outputId": "09aaf7f6-7892-4515-9eb4-d1e49cb530e2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "error",
          "ename": "TypeError",
          "evalue": "<tf.Tensor 'gradient_tape/sequential_57/dense_114/MatMul/MatMul:0' shape=(1, 128) dtype=float32> is out of scope and cannot be used here. Use return values, explicit Python locals or TensorFlow collections to access it.\nPlease see https://www.tensorflow.org/guide/function#all_outputs_of_a_tffunction_must_be_return_values for more information.\n\n<tf.Tensor 'gradient_tape/sequential_57/dense_114/MatMul/MatMul:0' shape=(1, 128) dtype=float32> was defined here:\n    File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n    File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n    File \"/usr/local/lib/python3.10/dist-packages/colab_kernel_launcher.py\", line 37, in <module>\n    File \"/usr/local/lib/python3.10/dist-packages/traitlets/config/application.py\", line 992, in launch_instance\n    File \"/usr/local/lib/python3.10/dist-packages/ipykernel/kernelapp.py\", line 619, in start\n    File \"/usr/local/lib/python3.10/dist-packages/tornado/platform/asyncio.py\", line 195, in start\n    File \"/usr/lib/python3.10/asyncio/base_events.py\", line 603, in run_forever\n    File \"/usr/lib/python3.10/asyncio/base_events.py\", line 1909, in _run_once\n    File \"/usr/lib/python3.10/asyncio/events.py\", line 80, in _run\n    File \"/usr/local/lib/python3.10/dist-packages/tornado/ioloop.py\", line 685, in <lambda>\n    File \"/usr/local/lib/python3.10/dist-packages/tornado/ioloop.py\", line 738, in _run_callback\n    File \"/usr/local/lib/python3.10/dist-packages/tornado/gen.py\", line 825, in inner\n    File \"/usr/local/lib/python3.10/dist-packages/tornado/gen.py\", line 786, in run\n    File \"/usr/local/lib/python3.10/dist-packages/ipykernel/kernelbase.py\", line 361, in process_one\n    File \"/usr/local/lib/python3.10/dist-packages/tornado/gen.py\", line 234, in wrapper\n    File \"/usr/local/lib/python3.10/dist-packages/ipykernel/kernelbase.py\", line 261, in dispatch_shell\n    File \"/usr/local/lib/python3.10/dist-packages/tornado/gen.py\", line 234, in wrapper\n    File \"/usr/local/lib/python3.10/dist-packages/ipykernel/kernelbase.py\", line 539, in execute_request\n    File \"/usr/local/lib/python3.10/dist-packages/tornado/gen.py\", line 234, in wrapper\n    File \"/usr/local/lib/python3.10/dist-packages/ipykernel/ipkernel.py\", line 302, in do_execute\n    File \"/usr/local/lib/python3.10/dist-packages/ipykernel/zmqshell.py\", line 539, in run_cell\n    File \"/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py\", line 2975, in run_cell\n    File \"/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py\", line 3030, in _run_cell\n    File \"/usr/local/lib/python3.10/dist-packages/IPython/core/async_helpers.py\", line 78, in _pseudo_sync_runner\n    File \"/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py\", line 3257, in run_cell_async\n    File \"/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py\", line 3473, in run_ast_nodes\n    File \"/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py\", line 3553, in run_code\n    File \"<ipython-input-90-a7094fa33ad1>\", line 11, in <cell line: 11>\n    File \"/usr/local/lib/python3.10/dist-packages/keras/src/utils/traceback_utils.py\", line 65, in error_handler\n    File \"/usr/local/lib/python3.10/dist-packages/keras/src/engine/training.py\", line 1807, in fit\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/util/traceback_utils.py\", line 150, in error_handler\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py\", line 832, in __call__\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py\", line 888, in _call\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py\", line 695, in _initialize\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/polymorphic_function/tracing_compilation.py\", line 178, in trace_function\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/polymorphic_function/tracing_compilation.py\", line 283, in _maybe_define_function\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/polymorphic_function/tracing_compilation.py\", line 310, in _create_concrete_function\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/framework/func_graph.py\", line 1059, in func_graph_from_py_func\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py\", line 598, in wrapped_fn\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/polymorphic_function/autograph_util.py\", line 41, in autograph_handler\n    File \"/usr/local/lib/python3.10/dist-packages/keras/src/engine/training.py\", line 1401, in train_function\n    File \"/usr/local/lib/python3.10/dist-packages/keras/src/engine/training.py\", line 1384, in step_function\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/distribute/distribute_lib.py\", line 1681, in run\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/distribute/distribute_lib.py\", line 3271, in call_for_each_replica\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/distribute/distribute_lib.py\", line 4069, in _call_for_each_replica\n    File \"/usr/local/lib/python3.10/dist-packages/keras/src/engine/training.py\", line 1373, in run_step\n    File \"/usr/local/lib/python3.10/dist-packages/keras/src/engine/training.py\", line 1154, in train_step\n    File \"/usr/local/lib/python3.10/dist-packages/keras/src/optimizers/optimizer.py\", line 543, in minimize\n    File \"/usr/local/lib/python3.10/dist-packages/keras/src/optimizers/optimizer.py\", line 276, in compute_gradients\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/backprop.py\", line 1066, in gradient\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/imperative_grad.py\", line 67, in imperative_grad\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/backprop.py\", line 148, in _gradient_function\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/ops/math_grad.py\", line 1702, in _MatMulGrad\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/ops/math_grad.py\", line 1683, in _MatMulGradAgainstSecondOnly\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/ops/gen_math_ops.py\", line 6192, in mat_mul\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/framework/op_def_library.py\", line 796, in _apply_op_helper\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/framework/func_graph.py\", line 670, in _create_op_internal\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/framework/ops.py\", line 2652, in _create_op_internal\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/framework/ops.py\", line 1160, in from_node_def\n\nThe tensor <tf.Tensor 'gradient_tape/sequential_57/dense_114/MatMul/MatMul:0' shape=(1, 128) dtype=float32> cannot be accessed from here, because it was defined in FuncGraph(name=train_function, id=134369476716448), which is out of scope.",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-90-a7094fa33ad1>\u001b[0m in \u001b[0;36m<cell line: 11>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      9\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcompile\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moptimizer\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mopt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mloss\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'mse'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmetrics\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'mse'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m \u001b[0;31m#history = model.fit(x, y, epochs=50, batch_size=len(x), verbose=0, callbacks = [LessVerboseEpochLogger(10)])\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m \u001b[0mhistory\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mepochs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m50\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mverbose\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/keras/src/utils/traceback_utils.py\u001b[0m in \u001b[0;36merror_handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     68\u001b[0m             \u001b[0;31m# To get the full stack trace, call:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     69\u001b[0m             \u001b[0;31m# `tf.debugging.disable_traceback_filtering()`\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 70\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwith_traceback\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfiltered_tb\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     71\u001b[0m         \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     72\u001b[0m             \u001b[0;32mdel\u001b[0m \u001b[0mfiltered_tb\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/execute.py\u001b[0m in \u001b[0;36mquick_execute\u001b[0;34m(op_name, num_outputs, inputs, attrs, ctx, name)\u001b[0m\n\u001b[1;32m     51\u001b[0m   \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     52\u001b[0m     \u001b[0mctx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mensure_initialized\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 53\u001b[0;31m     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,\n\u001b[0m\u001b[1;32m     54\u001b[0m                                         inputs, attrs, num_outputs)\n\u001b[1;32m     55\u001b[0m   \u001b[0;32mexcept\u001b[0m \u001b[0mcore\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_NotOkStatusException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mTypeError\u001b[0m: <tf.Tensor 'gradient_tape/sequential_57/dense_114/MatMul/MatMul:0' shape=(1, 128) dtype=float32> is out of scope and cannot be used here. Use return values, explicit Python locals or TensorFlow collections to access it.\nPlease see https://www.tensorflow.org/guide/function#all_outputs_of_a_tffunction_must_be_return_values for more information.\n\n<tf.Tensor 'gradient_tape/sequential_57/dense_114/MatMul/MatMul:0' shape=(1, 128) dtype=float32> was defined here:\n    File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n    File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n    File \"/usr/local/lib/python3.10/dist-packages/colab_kernel_launcher.py\", line 37, in <module>\n    File \"/usr/local/lib/python3.10/dist-packages/traitlets/config/application.py\", line 992, in launch_instance\n    File \"/usr/local/lib/python3.10/dist-packages/ipykernel/kernelapp.py\", line 619, in start\n    File \"/usr/local/lib/python3.10/dist-packages/tornado/platform/asyncio.py\", line 195, in start\n    File \"/usr/lib/python3.10/asyncio/base_events.py\", line 603, in run_forever\n    File \"/usr/lib/python3.10/asyncio/base_events.py\", line 1909, in _run_once\n    File \"/usr/lib/python3.10/asyncio/events.py\", line 80, in _run\n    File \"/usr/local/lib/python3.10/dist-packages/tornado/ioloop.py\", line 685, in <lambda>\n    File \"/usr/local/lib/python3.10/dist-packages/tornado/ioloop.py\", line 738, in _run_callback\n    File \"/usr/local/lib/python3.10/dist-packages/tornado/gen.py\", line 825, in inner\n    File \"/usr/local/lib/python3.10/dist-packages/tornado/gen.py\", line 786, in run\n    File \"/usr/local/lib/python3.10/dist-packages/ipykernel/kernelbase.py\", line 361, in process_one\n    File \"/usr/local/lib/python3.10/dist-packages/tornado/gen.py\", line 234, in wrapper\n    File \"/usr/local/lib/python3.10/dist-packages/ipykernel/kernelbase.py\", line 261, in dispatch_shell\n    File \"/usr/local/lib/python3.10/dist-packages/tornado/gen.py\", line 234, in wrapper\n    File \"/usr/local/lib/python3.10/dist-packages/ipykernel/kernelbase.py\", line 539, in execute_request\n    File \"/usr/local/lib/python3.10/dist-packages/tornado/gen.py\", line 234, in wrapper\n    File \"/usr/local/lib/python3.10/dist-packages/ipykernel/ipkernel.py\", line 302, in do_execute\n    File \"/usr/local/lib/python3.10/dist-packages/ipykernel/zmqshell.py\", line 539, in run_cell\n    File \"/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py\", line 2975, in run_cell\n    File \"/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py\", line 3030, in _run_cell\n    File \"/usr/local/lib/python3.10/dist-packages/IPython/core/async_helpers.py\", line 78, in _pseudo_sync_runner\n    File \"/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py\", line 3257, in run_cell_async\n    File \"/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py\", line 3473, in run_ast_nodes\n    File \"/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py\", line 3553, in run_code\n    File \"<ipython-input-90-a7094fa33ad1>\", line 11, in <cell line: 11>\n    File \"/usr/local/lib/python3.10/dist-packages/keras/src/utils/traceback_utils.py\", line 65, in error_handler\n    File \"/usr/local/lib/python3.10/dist-packages/keras/src/engine/training.py\", line 1807, in fit\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/util/traceback_utils.py\", line 150, in error_handler\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py\", line 832, in __call__\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py\", line 888, in _call\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py\", line 695, in _initialize\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/polymorphic_function/tracing_compilation.py\", line 178, in trace_function\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/polymorphic_function/tracing_compilation.py\", line 283, in _maybe_define_function\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/polymorphic_function/tracing_compilation.py\", line 310, in _create_concrete_function\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/framework/func_graph.py\", line 1059, in func_graph_from_py_func\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py\", line 598, in wrapped_fn\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/polymorphic_function/autograph_util.py\", line 41, in autograph_handler\n    File \"/usr/local/lib/python3.10/dist-packages/keras/src/engine/training.py\", line 1401, in train_function\n    File \"/usr/local/lib/python3.10/dist-packages/keras/src/engine/training.py\", line 1384, in step_function\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/distribute/distribute_lib.py\", line 1681, in run\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/distribute/distribute_lib.py\", line 3271, in call_for_each_replica\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/distribute/distribute_lib.py\", line 4069, in _call_for_each_replica\n    File \"/usr/local/lib/python3.10/dist-packages/keras/src/engine/training.py\", line 1373, in run_step\n    File \"/usr/local/lib/python3.10/dist-packages/keras/src/engine/training.py\", line 1154, in train_step\n    File \"/usr/local/lib/python3.10/dist-packages/keras/src/optimizers/optimizer.py\", line 543, in minimize\n    File \"/usr/local/lib/python3.10/dist-packages/keras/src/optimizers/optimizer.py\", line 276, in compute_gradients\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/backprop.py\", line 1066, in gradient\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/imperative_grad.py\", line 67, in imperative_grad\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/backprop.py\", line 148, in _gradient_function\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/ops/math_grad.py\", line 1702, in _MatMulGrad\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/ops/math_grad.py\", line 1683, in _MatMulGradAgainstSecondOnly\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/ops/gen_math_ops.py\", line 6192, in mat_mul\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/framework/op_def_library.py\", line 796, in _apply_op_helper\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/framework/func_graph.py\", line 670, in _create_op_internal\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/framework/ops.py\", line 2652, in _create_op_internal\n    File \"/usr/local/lib/python3.10/dist-packages/tensorflow/python/framework/ops.py\", line 1160, in from_node_def\n\nThe tensor <tf.Tensor 'gradient_tape/sequential_57/dense_114/MatMul/MatMul:0' shape=(1, 128) dtype=float32> cannot be accessed from here, because it was defined in FuncGraph(name=train_function, id=134369476716448), which is out of scope."
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "It looks like trying to wrap the Optimizer to collect stats isn't going to work well. On first glance, it appears that the highly optimised TF classes use their computation graph not just for the loss function calculation, but also in how the gradients are applied. This seems to be evidenced also by the way that my custom loop is so much slower than using the standard model.fit() function (about 10x slower).\n",
        "\n",
        "Additionally, when running my custom training loop for 5000 iterations, it learns a flat line instead of the curve. So something is just wrong.!\n",
        "\n",
        "Ironically, the grad sign change plot shows quite clearly that there's oscillation in the weight changes.\n",
        "\n",
        "It appears that I will need to override Model.fit() instead, as per: https://www.tensorflow.org/guide/keras/customizing_what_happens_in_fit"
      ],
      "metadata": {
        "id": "NCdhp8xPnYwi"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## MNIST Image Classification\n",
        "As a first exercise, I'll look at the weight updates"
      ],
      "metadata": {
        "id": "Q2mrdo35FDRN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "fashion_mnist = tf.keras.datasets.fashion_mnist\n",
        "(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()\n",
        "class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']\n",
        "print(f\"train_images shape: {train_images.shape}\")\n",
        "print(f\"train_labels shape: {train_labels.shape}\")"
      ],
      "metadata": {
        "id": "6Wf7oBuqFkNi"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}