tutorial/30.lin.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Linear models, loss functions, gradients, SGD\n",
    "(c) Deniz Yuret, 2019-2021\n",
    "* Objectives: Define, train and visualize a simple model; understand gradients and SGD; learn to use the GPU.\n",
    "* Prerequisites: [Callable objects](https://docs.julialang.org/en/v1/manual/methods/#Function-like-objects-1), [Generator expressions](https://docs.julialang.org/en/v1/manual/arrays/#Generator-Expressions-1), [MNIST](20.mnist.ipynb), [Iterators](25.iterators.ipynb)\n",
    "* New functions: \n",
    "[accuracy](http://denizyuret.github.io/Knet.jl/latest/reference/#Knet.accuracy), \n",
    "[nll](http://denizyuret.github.io/Knet.jl/latest/reference/#Knet.nll), \n",
    "[Param, @diff, value, params, grad](http://denizyuret.github.io/Knet.jl/latest/reference/#AutoGrad),\n",
    "[sgd](http://denizyuret.github.io/Knet.jl/latest/reference/#Knet.sgd),\n",
    "[progress, progress!](http://denizyuret.github.io/Knet.jl/latest/reference/#Knet.progress), \n",
    "[gpu](http://denizyuret.github.io/Knet.jl/latest/reference/#Knet.gpu), \n",
    "[CuArray](https://cuda.juliagpu.org/stable/usage/overview), \n",
    "[load, save](https://juliaio.github.io/FileIO.jl/stable), \n",
    "\n",
    "\n",
    "<img src=\"https://www.oreilly.com/library/view/tensorflow-for-deep/9781491980446/assets/tfdl_0401.png\" alt=\"A linear model\" width=300/> ([image source](https://www.oreilly.com/library/view/tensorflow-for-deep/9781491980446/ch04.html))\n",
    "\n",
    "In Knet, a machine learning model is defined using plain Julia code. A typical model consists of a **prediction** and a **loss** function. The prediction function takes some input, returns the prediction of the model for that input. The loss function measures how bad the prediction is with respect to some desired output. We train a model by adjusting its parameters to reduce the loss.\n",
    "\n",
    "In this section we will implement a simple linear model to classify MNIST digits. The prediction function will return 10 scores for each of the possible labels 0..9 as a linear combination of the pixel values. The loss function will convert these scores to normalized probabilities and return the average -log probability of the correct answers. Minimizing this loss should maximize the scores assigned to correct answers by the model. We will make use of the loss gradient with respect to each parameter, which tells us the direction of the greatest loss increase. We will improve the model by moving the parameters in the opposite direction (using a GPU if available). We will visualize the model weights and performance over time. The final accuracy of about 92% is close to the limit of what we can achieve with this type of model. To improve further we must look beyond linear models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Set display width, load packages, import symbols\n",
    "ENV[\"COLUMNS\"]=72\n",
    "using Statistics: mean\n",
    "using Base.Iterators: flatten\n",
    "using IterTools: ncycle, takenth\n",
    "import Random # seed!\n",
    "using MLDatasets: MNIST\n",
    "using FileIO: load, save\n",
    "import JLD2 # data file format\n",
    "import CUDA # functional, CuArray\n",
    "using Knet: Knet, AutoGrad, dir, Data, minibatch, Param, @diff, value, params, grad, progress, progress!, CuArray\n",
    "# The following are defined for instruction even though they are provided in Knet\n",
    "# using Knet: accuracy, nll, sgd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "600-element Data{Tuple{Array{Float32,2},Array{Int64,1}}}\n",
      "100-element Data{Tuple{Array{Float32,2},Array{Int64,1}}}\n"
     ]
    }
   ],
   "source": [
    "# Load MNIST data\n",
    "xtrn,ytrn = MNIST.traindata(Float32); ytrn[ytrn.==0] .= 10\n",
    "xtst,ytst = MNIST.testdata(Float32);  ytst[ytst.==0] .= 10\n",
    "dtrn = minibatch(xtrn, ytrn, 100; xsize = (784,:), xtype = Array)\n",
    "dtst = minibatch(xtst, ytst, 100; xsize = (784,:), xtype = Array)\n",
    "println.(summary.((dtrn,dtst)));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Model definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Linear([0.010250208813939895 0.009156469384186117 … -0.008744043935401897 0.0027893835288007015; 0.012809294862061355 -0.0027828944916268284 … -0.01291731632961187 0.028698271336794062; … ; -0.007299245195363846 0.006931397646413979 … 0.00029883093930691213 -0.00042994722598638653; -0.014150554416036985 0.011703104539148282 … 0.013981758139057205 0.004469080535429366], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# In Julia we define a new datatype using `struct`:\n",
    "struct Linear; w; b; end\n",
    "\n",
    "# The new struct comes with a default constructor:\n",
    "model = Linear(0.01 * randn(10,784), zeros(10))\n",
    "\n",
    "# We can define other constructors with different inputs:\n",
    "Linear(i::Int,o::Int,scale=0.01) = Linear(scale * randn(o,i), zeros(o))\n",
    "\n",
    "# This one allows instances to be defined using input and output sizes:\n",
    "model = Linear(784,10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# We turn Linear instances into callable objects for prediction:\n",
    "(m::Linear)(x) = m.w * x .+ m.b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(\"784×100 Array{Float32,2}\", \"100-element Array{Int64,1}\")"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x,y = first(dtst) # The first minibatch from the test set\n",
    "summary.((x,y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1×100 LinearAlgebra.Adjoint{Int64,Array{Int64,1}}:\n",
       " 7  2  1  10  4  1  4  9  5  9  …  1  3  6  9  3  1  4  1  7  6  9"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y' # correct answers are given as an array of integers (remember we use 10 for 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10×100 Array{Float64,2}:\n",
       " -0.0250313   -0.0816154   -0.00423346  …  -0.135169   -0.0115012\n",
       "  0.0122202    0.191724    -0.0426294      -0.0232773   0.0648005\n",
       " -0.0499727    0.205248     0.0562747       0.0852388   0.030561\n",
       "  0.0344634   -0.0917553    0.0138946      -0.0437487  -0.0509927\n",
       " -0.0362311   -0.0544444   -0.0221727      -0.0451954  -0.172924\n",
       "  0.00775462  -0.107705    -0.00175397  …  -0.0442647  -0.0156054\n",
       " -0.00403154  -0.0260729    0.0224417      -0.137707   -0.00150977\n",
       "  0.0255925    0.00819857   0.0125202       0.0554805  -0.0272517\n",
       " -0.0155763   -0.0461483    0.0882222      -0.0856574  -0.080045\n",
       "  0.0395347    0.113438    -0.0488667       0.0662365   0.0560977"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ypred = model(x)  # Predictions on the first minibatch: a 10x100 score matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can calculate the accuracy of our model for the first minibatch\n",
    "accuracy(model,x,y) = mean(y' .== map(i->i[1], findmax(Array(model(x)),dims=1)[2]))\n",
    "accuracy(model,x,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.12749999999999995"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can calculate the accuracy of our model for the whole test set\n",
    "accuracy(model,data) = mean(accuracy(model,x,y) for (x,y) in data)\n",
    "accuracy(model,dtst)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Loss function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "nll (generic function with 1 method)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# For classification we use negative log likelihood loss (aka cross entropy, softmax loss, NLL)\n",
    "# This is the average -log probability assigned to correct answers by the model\n",
    "function nll(scores, y)\n",
    "    expscores = exp.(scores)\n",
    "    probabilities = expscores ./ sum(expscores, dims=1)\n",
    "    answerprobs = (probabilities[y[i],i] for i in 1:length(y))\n",
    "    mean(-log.(answerprobs))\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.2918972870616603"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# model(x) gives predictions, let model(x,y) give the loss\n",
    "(m::Linear)(x, y) = nll(m(x), y)\n",
    "model(x,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.2918972870616603"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can also use the Knet nll implementation for efficiency\n",
    "(m::Linear)(x, y) = Knet.nll(m(x), y)\n",
    "model(x,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# If the input is a dataset compute average loss:\n",
    "(m::Linear)(data::Data) = mean(m(x,y) for (x,y) in data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.296847605139261"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Here is per-instance average negative log likelihood for the whole test set\n",
    "model(dtst)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Bonus question:** What is special about the loss value 2.3?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Calculating the gradient using AutoGrad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/latex": [
       "Usage:\n",
       "\n",
       "\\begin{verbatim}\n",
       "x = Param([1,2,3])          # user declares parameters with `Param`\n",
       "x => P([1,2,3])             # `Param` is just a struct wrapping a value\n",
       "value(x) => [1,2,3]         # `value` returns the thing wrapped\n",
       "sum(x .* x) => 14           # Params act like regular values\n",
       "y = @diff sum(x .* x)       # Except when we differentiate using `@diff`\n",
       "y => T(14)                  # you get another struct\n",
       "value(y) => 14              # which carries the same result\n",
       "params(y) => [x]            # and the Params that it depends on \n",
       "grad(y,x) => [2,4,6]        # and the gradients for all Params\n",
       "\\end{verbatim}\n",
       "\\texttt{Param(x)} returns a struct that acts like \\texttt{x} but marks it as a parameter you want to compute gradients with respect to.\n",
       "\n",
       "\\texttt{@diff expr} evaluates an expression and returns a struct that contains the result (which should be a scalar) and gradient information.\n",
       "\n",
       "\\texttt{grad(y, x)} returns the gradient of \\texttt{y} (output by @diff) with respect to any parameter \\texttt{x::Param}, or  \\texttt{nothing} if the gradient is 0.\n",
       "\n",
       "\\texttt{value(x)} returns the value associated with \\texttt{x} if \\texttt{x} is a \\texttt{Param} or the output of \\texttt{@diff}, otherwise returns \\texttt{x}.\n",
       "\n",
       "\\texttt{params(x)} returns an iterator of Params found by a recursive search of object \\texttt{x}.\n",
       "\n",
       "Alternative usage:\n",
       "\n",
       "\\begin{verbatim}\n",
       "x = [1 2 3]\n",
       "f(x) = sum(x .* x)\n",
       "f(x) => 14\n",
       "grad(f)(x) => [2 4 6]\n",
       "gradloss(f)(x) => ([2 4 6], 14)\n",
       "\\end{verbatim}\n",
       "Given a scalar valued function \\texttt{f}, \\texttt{grad(f,argnum=1)} returns another function \\texttt{g} which takes the same inputs as \\texttt{f} and returns the gradient of the output with respect to the argnum'th argument. \\texttt{gradloss} is similar except the resulting function also returns f's output.\n",
       "\n"
      ],
      "text/markdown": [
       "Usage:\n",
       "\n",
       "```\n",
       "x = Param([1,2,3])          # user declares parameters with `Param`\n",
       "x => P([1,2,3])             # `Param` is just a struct wrapping a value\n",
       "value(x) => [1,2,3]         # `value` returns the thing wrapped\n",
       "sum(x .* x) => 14           # Params act like regular values\n",
       "y = @diff sum(x .* x)       # Except when we differentiate using `@diff`\n",
       "y => T(14)                  # you get another struct\n",
       "value(y) => 14              # which carries the same result\n",
       "params(y) => [x]            # and the Params that it depends on \n",
       "grad(y,x) => [2,4,6]        # and the gradients for all Params\n",
       "```\n",
       "\n",
       "`Param(x)` returns a struct that acts like `x` but marks it as a parameter you want to compute gradients with respect to.\n",
       "\n",
       "`@diff expr` evaluates an expression and returns a struct that contains the result (which should be a scalar) and gradient information.\n",
       "\n",
       "`grad(y, x)` returns the gradient of `y` (output by @diff) with respect to any parameter `x::Param`, or  `nothing` if the gradient is 0.\n",
       "\n",
       "`value(x)` returns the value associated with `x` if `x` is a `Param` or the output of `@diff`, otherwise returns `x`.\n",
       "\n",
       "`params(x)` returns an iterator of Params found by a recursive search of object `x`.\n",
       "\n",
       "Alternative usage:\n",
       "\n",
       "```\n",
       "x = [1 2 3]\n",
       "f(x) = sum(x .* x)\n",
       "f(x) => 14\n",
       "grad(f)(x) => [2 4 6]\n",
       "gradloss(f)(x) => ([2 4 6], 14)\n",
       "```\n",
       "\n",
       "Given a scalar valued function `f`, `grad(f,argnum=1)` returns another function `g` which takes the same inputs as `f` and returns the gradient of the output with respect to the argnum'th argument. `gradloss` is similar except the resulting function also returns f's output.\n"
      ],
      "text/plain": [
       "  Usage:\n",
       "\n",
       "\u001b[36m  x = Param([1,2,3])          # user declares parameters with `Param`\u001b[39m\n",
       "\u001b[36m  x => P([1,2,3])             # `Param` is just a struct wrapping a value\u001b[39m\n",
       "\u001b[36m  value(x) => [1,2,3]         # `value` returns the thing wrapped\u001b[39m\n",
       "\u001b[36m  sum(x .* x) => 14           # Params act like regular values\u001b[39m\n",
       "\u001b[36m  y = @diff sum(x .* x)       # Except when we differentiate using `@diff`\u001b[39m\n",
       "\u001b[36m  y => T(14)                  # you get another struct\u001b[39m\n",
       "\u001b[36m  value(y) => 14              # which carries the same result\u001b[39m\n",
       "\u001b[36m  params(y) => [x]            # and the Params that it depends on \u001b[39m\n",
       "\u001b[36m  grad(y,x) => [2,4,6]        # and the gradients for all Params\u001b[39m\n",
       "\n",
       "  \u001b[36mParam(x)\u001b[39m returns a struct that acts like \u001b[36mx\u001b[39m but marks it as a\n",
       "  parameter you want to compute gradients with respect to.\n",
       "\n",
       "  \u001b[36m@diff expr\u001b[39m evaluates an expression and returns a struct that\n",
       "  contains the result (which should be a scalar) and gradient\n",
       "  information.\n",
       "\n",
       "  \u001b[36mgrad(y, x)\u001b[39m returns the gradient of \u001b[36my\u001b[39m (output by @diff) with respect\n",
       "  to any parameter \u001b[36mx::Param\u001b[39m, or \u001b[36mnothing\u001b[39m if the gradient is 0.\n",
       "\n",
       "  \u001b[36mvalue(x)\u001b[39m returns the value associated with \u001b[36mx\u001b[39m if \u001b[36mx\u001b[39m is a \u001b[36mParam\u001b[39m or the\n",
       "  output of \u001b[36m@diff\u001b[39m, otherwise returns \u001b[36mx\u001b[39m.\n",
       "\n",
       "  \u001b[36mparams(x)\u001b[39m returns an iterator of Params found by a recursive search\n",
       "  of object \u001b[36mx\u001b[39m.\n",
       "\n",
       "  Alternative usage:\n",
       "\n",
       "\u001b[36m  x = [1 2 3]\u001b[39m\n",
       "\u001b[36m  f(x) = sum(x .* x)\u001b[39m\n",
       "\u001b[36m  f(x) => 14\u001b[39m\n",
       "\u001b[36m  grad(f)(x) => [2 4 6]\u001b[39m\n",
       "\u001b[36m  gradloss(f)(x) => ([2 4 6], 14)\u001b[39m\n",
       "\n",
       "  Given a scalar valued function \u001b[36mf\u001b[39m, \u001b[36mgrad(f,argnum=1)\u001b[39m returns another\n",
       "  function \u001b[36mg\u001b[39m which takes the same inputs as \u001b[36mf\u001b[39m and returns the gradient\n",
       "  of the output with respect to the argnum'th argument. \u001b[36mgradloss\u001b[39m is\n",
       "  similar except the resulting function also returns f's output."
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@doc AutoGrad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Linear"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Redefine the constructor to use Param's so we can compute gradients\n",
    "Linear(i::Int,o::Int,scale=0.01) = \n",
    "\n",
    "Linear(Param(scale * randn(o,i)), Param(zeros(o)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Set random seed for replicability\n",
    "Random.seed!(226);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Linear(P(Array{Float64,2}(10,784)), P(Array{Float64,1}(10)))"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use a larger scale to get a large initial loss\n",
    "model = Linear(784,10,1.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "17.3364869902235"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can still do predictions and calculate loss:\n",
    "model(x,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "T(17.3364869902235)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# And we can do the same loss calculation also computing gradients:\n",
    "J = @diff model(x,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "17.3364869902235"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# To get the actual loss value from J:\n",
    "value(J)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2-element Array{Param,1}:\n",
       " P(Array{Float64,2}(10,784))\n",
       " P(Array{Float64,1}(10))"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# params(J) returns an iterator of Params J depends on (i.e. model.b, model.w):\n",
    "params(J) |> collect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10×784 Array{Float64,2}:\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0  …  0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0  …  0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# To get the gradient of a parameter from J:\n",
    "∇w = grad(J,model.w)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "∇b = grad(J, model.b) = [0.5906946123008457, -0.027041397089362467, -0.08694323408456664, -0.10750809160810809, -0.05254391972254851, -0.09056807340274403, -0.14070630316059043, 0.015127334068075577, -0.03230399769325696, -0.06820692960774405]\n"
     ]
    }
   ],
   "source": [
    "# Note that each gradient has the same size and shape as the corresponding parameter:\n",
    "@show ∇b = grad(J,model.b);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Checking the gradient using numerical approximation\n",
    "\n",
    "What does ∇b represent?\n",
    "\n",
    "∇b[1] = 0.59 means if I increase b[1] by ϵ, loss will increase by 0.59ϵ"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "value(model.b) = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "17.3364869902235"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Loss for the first minibatch with the original parameters\n",
    "@show value(model.b)\n",
    "model(x,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.1"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# To numerically check the gradient let's increase the last entry of b by +0.1.\n",
    "model.b[1] = 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "value(model.b) = [0.1, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "17.39572020341206"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We see that the loss moves by ≈ +0.79*0.1 as expected.\n",
    "@show value(model.b)\n",
    "model(x,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Reset the change.\n",
    "model.b[1] = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Checking the gradient using manual implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Without AutoGrad we would have to define the gradients manually:\n",
    "function nllgrad(model,x,y)\n",
    "    scores = model(x)\n",
    "    expscores = exp.(scores)\n",
    "    probabilities = expscores ./ sum(expscores, dims=1)\n",
    "    for i in 1:length(y); probabilities[y[i],i] -= 1; end\n",
    "    dJds = probabilities / length(y)\n",
    "    dJdw = dJds * x'\n",
    "    dJdb = vec(sum(dJds,dims=2))\n",
    "    dJdw,dJdb\n",
    "end;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0; … ; 0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0], [0.5906946123008457, -0.027041397089362464, -0.08694323408456665, -0.10750809160810809, -0.0525439197225485, -0.09056807340274403, -0.14070630316059046, 0.015127334068075577, -0.03230399769325697, -0.06820692960774406])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "∇w2,∇b2 = nllgrad(model,x,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "true"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "∇w2 ≈ ∇w"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "true"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "∇b2 ≈ ∇b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Training with Stochastic Gradient Descent (SGD)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "sgdupdate! (generic function with 1 method)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Here is a single SGD update:\n",
    "function sgdupdate!(func, args; lr=0.1)\n",
    "    fval = @diff func(args...)\n",
    "    for param in params(fval)\n",
    "        ∇param = grad(fval, param)\n",
    "        param .-= lr * ∇param\n",
    "    end\n",
    "    return value(fval)\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "sgd (generic function with 1 method)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We define SGD for a dataset as an iterator so that:\n",
    "# 1. We can monitor and report the training loss\n",
    "# 2. We can take snapshots of the model during training\n",
    "# 3. We can pause/terminate training when necessary\n",
    "sgd(func, data; lr=0.1) = \n",
    "    (sgdupdate!(func, args; lr=lr) for args in data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model(dtst) = 2.3142570870549655\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "┣████████████████████┫ [100.00%, 6000/6000, 00:12/00:12, 491.39i/s] \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model(dtst) = 0.28047171743026245\n"
     ]
    }
   ],
   "source": [
    "# Let's train a model for 10 epochs to compare training speed on cpu vs gpu.\n",
    "# progress!(itr) displays a progress bar when wrapped around an iterator like this:\n",
    "# 2.94e-01  100.00%┣████████████████████┫ 6000/6000 [00:10/00:10, 592.96/s] 2.31->0.28\n",
    "model = Linear(784,10)\n",
    "@show model(dtst)\n",
    "progress!(sgd(model, ncycle(dtrn,10)))\n",
    "@show model(dtst);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Using the GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model(dtst) = 2.3094196f0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "┣████████████████████┫ [100.00%, 6000/6000, 00:07/00:07, 853.96i/s] \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model(dtst) = 0.28053972f0\n"
     ]
    }
   ],
   "source": [
    "# The training would go a lot faster on a GPU:\n",
    "# 2.94e-01  100.00%┣███████████████████┫ 6000/6000 [00:02/00:02, 2653.45/s]  2.31->0.28\n",
    "# To work on a GPU, all we have to do is convert Arrays to CuArrays:\n",
    "if CUDA.functional()  # returns true if there is a GPU\n",
    "    atype = CuArray{Float32}  # CuArrays are stored and operated in the GPU\n",
    "    dtrn = minibatch(xtrn, ytrn, 100; xsize = (784,:), xtype=atype)\n",
    "    dtst = minibatch(xtst, ytst, 100; xsize = (784,:), xtype=atype)\n",
    "    Linear(i::Int,o::Int,scale=0.01) = \n",
    "        Linear(Param(atype(scale * randn(o,i))), \n",
    "               Param(atype(zeros(o))))\n",
    "\n",
    "    model = Linear(784,10)\n",
    "    @show model(dtst)\n",
    "    progress!(sgd(model,ncycle(dtrn,10)))\n",
    "    @show model(dtst)\n",
    "end;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "## Recording progress"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "trainresults (generic function with 1 method)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function trainresults(file, model)\n",
    "    if (print(\"Train from scratch? (~77s) \"); readline()[1]=='y')\n",
    "        # We will train 100 epochs (the following returns an iterator, does not start training)\n",
    "        training = sgd(model, ncycle(dtrn,100))\n",
    "        # We will snapshot model and train/test loss and errors\n",
    "        snapshot() = (deepcopy(model),model(dtrn),model(dtst),1-accuracy(model,dtrn),1-accuracy(model,dtst))\n",
    "        # Snapshot results once every epoch (still an iterator)\n",
    "        snapshots = (snapshot() for x in takenth(progress(training),length(dtrn)))\n",
    "        # Run the snapshot/training iterator, reshape and save results as a 5x100 array\n",
    "        lin = reshape(collect(flatten(snapshots)),(5,:))\n",
    "        # FileIO.save and FileIO.load can be used to store models in files\n",
    "        save(file,\"results\",lin)\n",
    "    else\n",
    "        isfile(file) || download(\"https://github.com/denizyuret/Knet.jl/releases/download/v1.4.9/$file\", file)\n",
    "        lin = load(file,\"results\")    \n",
    "    end\n",
    "    return lin\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train from scratch? (~77s) stdin> n\n"
     ]
    }
   ],
   "source": [
    "# 2.43e-01  100.00%┣████████████████▉┫ 60000/60000 [00:44/00:44, 1349.13/s]\n",
    "lin = trainresults(\"lin149.jld2\",Linear(784,10));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Linear model shows underfitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "using Plots; default(fmt = :png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlgAAAGQCAIAAAD9V4nPAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nO3deXgT5fo38PuZSSZL0zTd9wItUpClLLIo+47IphXkcFgURUQRjytHQQE3xPOK/gA9qCgKKIoHZJODCwdQ9qUFAREEqaWlC23TPckkmXn/mBJq2Qo2Dc18PxcX18wzk8mdNO03z8wzM0yWZQIAAFArztcFAAAA+BKCEAAAVA1BCAAAqoYgBAAAVUMQAgCAqiEIAQBA1bwehA6HY/78+ePGjXvttdfKy8svXcHlcs2ZM+ezzz7zdiUAAACX8noQPvLII+vXrx82bNi+fftGjhx56Qrz589ftGjRunXrvF0JAADApZhXT6g/d+5ckyZNMjIyoqOjbTZbRETEzp0727Rp41nhxIkT99133+DBg0+dOrVq1SrvVQIAAHBZ3u0RHjhwICkpKTo6mogMBkOnTp12797tWSpJ0sMPP7xo0SKdTufVMgAAAK5E49Wt5+bmhoaGembDw8NzcnI8swsWLGjZsmW3bt22bNly2YcfO3bsmWee0Wq11RufffbZLl26EJHSl2WMXfax5wusdqcUHx162aXw17ndbp7nfV2FSl39ww/ehg+/D13vh1+j0VxzZe8GoV6vdzqdnlmHw2EwGJTp33///d///ve+ffuu8vC33367cePGAwYMqN6YmJiofATtdjvP8xrN5V/CLz9utxYUNZ4y6a++BriCysrKwMBAX1ehUqIoEpEgCL4uRKXw4fchp9MpSVLt9yPWJjK9G4RxcXGZmZmyLCulZGZm3nvvvcqiLVu25Obmtm/fnoisVqsoirfffnv1HadExPN827Zt77777stunLvg8s/NOEZ0xaXwl13tzQcvU955vP++gg+/D3njw+/dn2W3bt3cbrey5/Po0aMnTpwYPHhwcXHxl19+OXbs2IyMjAMHDhw4cGDy5Mn9+vXbvHlzHT41I8JtNQAA4Jq82yMUBGH+/PmjR4++/fbb9+3b99prrwUHB6elpY0ePdputwcHByur6fV6QRCCgoLq8rk5RrjDFAAAXIt3g5CI/v73v/fp0+fYsWNNmzZt3LgxEbVq1er06dPV9/BOmzZNOeZRp9AnBACAa/N6EBJRdHS0cgaFQhCExMTE6it4uoZ1Cj1CAAC4Nr893ssYYWg5AABck98GIQAAQG34bxAy7BoFAIBrQxACAICq1cdgGR/BqFEAaGB+/vnniooKX1dxkzIYDG3btvXGlv03CDFUBgAamrFjxzLGjEajrwu56djt9srKyhMnTnhj434bhBxOnwCABmj58uXV71UHihMnTgwbNsxLG/ffY4TYMQoAALXgt0EYIDC7S/J1FQAAcLPz2yCM0FOp89qrAQCAyvltEIboObtLdrh9XQcAANzc/DYIOY4FauQzZThQCAAAV+O3QUjEzFr5dJmvqwAAgJubHwchmbXsVAl6hAAAcDX+G4SMzFo6jV2jAABwVf4bhMTMAp0qRRACANSTzMzMQ4cO+bqK6+a/QciYWSOfLvV1GQAADdm0adOsVmstV/7mm2/eeOMNr9bjDX4bhIwoQEuZ5TLOqgcAuGEff/xxeXm5r6vwLr8NQmKMJznCwM5WYO8oAMCNeO655+x2+5gxY/r377979+633nrrvffeS01NjY2NPXny5LBhw1avXt2xY8fExMRZs2ZVf+DatWu7dOmSlJQ0duzY/Px8Itq8eXPnzp1jY2NTUlJWr17tcrmeeuqpW265JSEhoV+/fsXFxT56iUR+fNFtIiKSm5rpVCk1CfR1IQAAN8TmIqtYT9/mQ3RMz/+p5cUXX3z33Xf//e9/x8bGmkymJUuWbNy48csvv/zwww8NBsPWrVsZYxs2bCgvL+/evXu/fv2UR6WlpT3wwAPr169v167d9OnTR40a9b///e9vf/vbf//73y5dupw7d66oqGjVqlUHDx48ePCgwWBIT08XBKF+XuNl+XEQMiJqamanSuX+sbgnEwA0SBsypX/sqadLZC3toRkY96e/loGBgYyxoKCg4OBgpeW+++7r1auXZ4UXX3wxKiqKiAYMGJCenq7Vaolo+fLlf//737t3705E8+bNCwkJ+eOPPxhjBw4caNasWUxMTExMzNGjR4uKitLT07t169apU6f6eYFX4r9ByBjJcpKZncbAUQBosEYlcqMSb6JjWPHx8dVnIyIilImAgICKigqLxUJEOTk57du3V9pNJlNYWFhubu66devmzp37/PPPt2vXbsGCBSNHjjxz5szUqVOzs7PHjBkzf/58H3YKb6L3t44xIpmSAgkDRwEAbhjHcZJ0ccwhY9fewRYbG3vmzBlluqysrKCgIC4urnv37ps2bcrJyenatevUqVN5nn/++eePHDmyZ8+ebdu2rVq1ylsvoBb8NwiJEclNgxhOJQQAuGFNmjRZt27dwYMHaz+e5f7771+5cuV3332Xn5//5JNP9urVy2w2L1269Ny5c4wxjUZjsVi2bNmyY8eO8vJyQRBkWVa6kr7iv0HIGElSUiA7XYYb1QMA3KAlS5YcPHjw+eefP378+K233tqoUSPPot69e+v1emW6efPmTZo0SUhIaNu2bevWrVetWjVv3rzevXvzPP/ll1/yPL9t27ZBgwZ17NgxIyNj8eLFbrf7lVde6dChw8iRIx966KEhQ4b46PUR+fExQj4o1FV8PkRLQVo6VyHHBmC8DADAdevYseOnn36qTN9+++3VF61fv94zPW3aNGXirrvuIqIBAwYMGDCg+sqejSji4uJqrOBDftsj1EbGu/LOElHTIHYKhwkBAOAK/DYIOZOFSJYqSpW9o74uBwAAblJ+G4REpAmPc+WfxRkUAABwFX4dhJHxzvws5eIyAAAAl+XPQaiNiHPlZykXl/F1LQAAfu7cuXOFhYW+ruJG+HMQaiLiXHlnm5pxn3oAgBs0f/78srKyKy2dN2+e3W5XpmfOnFljaGhD4edB6MzPCtaRhqPzdl9XAwDQAL300ktXOZX+n//8Z2VlZX3W4w1+ex4hEWnCY93WfNntUsbLhOtxKiEAwHV4+eWXHQ7HlClTjEbjc889JwjC3Llzz5w5Ex4e/sgjj+zatYuIJk6cKAjCiy++6HnUqlWrVqxY4XA4hg8fPmXKFMbY119/vWTJEqvVmpCQ8Morr8TExMyZM2f37t0ajaZbt26vvPKK714ikX8HIeM1fFCouzC3qTn6VKncJQJBCABwHUaPHv3GG288+uijkZGRSUlJt91224wZM/r27Zubm1tZWTlu3LjXX3/9iSeeMJvNnivO/Oc//3n66aeXLVsWEBAwadKkkpKS0aNHP/LII2vWrImPjz9+/Lgsy6+//npmZuYnn3zidDrT09N9+xrJv4OQLuwdVYLQ17UAAFy3yv0/FK2cXz/PFTrxJUOrLtVbmjVrxnFc69atlZtOWK1Wo9EYExOTkJDgWSclJSUkJMQz+957782YMaN3795E9P/+3/+bOHHinXfeyXGcwWBISEhQHlhUVGQwGCwWS2hoaPPmzevn1V2F3wdhvCs/Kym+0w/ZCEIAaHiMt/U1duhdT0/GrjFq5JNPPpk5c+aUKVMGDBjwyiuv3HLLLZeuk5mZmZycrEy3aNEiOzu7ZcuWU6dOHTZsmEajSU1NnT179owZMx577LHGjRu3bdv2iSeeuPfee+v+tVwPfx4sQ1VnUJxNCsQ59QDQMDFGHF9P/y53iyXGmHzhzgVDhw49fPjwoUOHQkJCJkyYUGOpIiIiIjc3V5k+d+5cSEiIVqudMWNGVlbW6tWr9+/fP2/evLi4uHXr1uXl5U2ZMmX8+PEZGRnefQ+vxc+DUOkR4mZMAAA3JiYmZt++fVartaKi4ttvv5UkKSEhoWPHji6XS1m6d+9eq9XqdDqV9UeOHDl//vyioiKbzfbqq6+OGjXq999/P3LkCBG1a9cuMTHR5XJt3bq1pKTEaDT27t1bEITq9zv0Cb8Pwjhn3tkoA9ndVCL6uhoAgIbmnXfeWbRoUefOnX/88cc333wzOjo6Pj7+o48+ev/994lo0aJF8+bN69y58759+2JiYkJDQ6dOndq3b9927dolJSVFRES88cYbxcXFEyZMiIyMbNy4cUVFxfTp03fs2NG6devY2NhevXrNnTs3MTHRt6+xZq/2pjJ58uT27dtPnjz5skvtdjvP81qt9uobOff8vVEzP27/rfHjHnz7MAwcrTNlZWWBgYG+rkKlRFEkIkEQfF2ISnnvw9+mTZsVK1a0adPGGxtv0E6cODFs2LATJ044nU5JknQ6XR1u3M97hESkCY91nc/G3lEAALgsFQRhZJwz72xSIJ2+4kWCAABAvVQQhBFxrvws3IwJAAAuy/+DULkHRXMLO2pFEAIAQE3+H4SaiHhnftbtEey3EjnX5utqAADgJqOCIAyPdRflask9MI7bmOnjs1UAAOBm4/9ByDRa3hziKswd3oitzUAQAgDAn/h/ENKF68vcFc/tyJPLnL6uBgAAbiYqCcI4Z36WSUt3RLLNWegUAgDARX5+9wmFNiJOzDpFRMMbcev+kEc28XVBAACXk5KSctddd+GaQZdyOp0Gg8FLG1dFEGoi4ioPbiWiEY245/c7RYkXVNETBoAGZvHixZ5bN0AN3vt+oJYgdOZnEVGkgZoHse05cv9YXHQUAG46AQEBSUlJvq5CdVTRM+KDQsklSpVlVLV3FIcJAQCgiiqCkIg04XGu/GwiuqcxW/vHTXzHDQAAqF+qCcKIOFf+WSK6JYiZtXTgPKIQAACIVBWEymFCIhrRiGHvKAAAKNQShNrIeKVHSETDG3Fr/0CPEAAAiNQThELjFo7TR2WXk4g6RbASkU6WIAsBAEA1QchbwrSxifZje4mIEQ1NYOvQKQQAAPUEIREF3Na38sAWZXpEY241LsANAACqCkJD2+6O00fcZVYi6hPNrA7acg6dQgAAtVNREDJBr2/ZxZb+IxFpOJrdnnt+vxtJCACgcioKQiIK6Ni3cn/V3tH7EjmnRLhVLwCAyqkrCHW3tHWXWZ05GUTEMZrTnptxQJLQKwQAUDF1BSExZrytj3InCiIa1ogL0NCq39EpBABQL5UFIZGxY9/K/T+QVBV+r3TgX0qTXIhCAAC1Ul0QaiMTeHOo49RhZbZfLIsPoGWnkIQAACqluiAkImPHvhUXhswQ0dyO/MtpksPtw4oAAMBnVBmEHXrbj+2RHTZltlM4ax1CH/yKTiEAgBqpMQi5ALMusZXt552elldv4+cedp+3+7AoAADwDTUGIREZO/ar2PutZzYlhE1sxo3+nwsn2AMAqI1Kg1DfqotUUVaZvt3T8nIHnmP0+iHsIAUAUBeVBiHjNcF/e7JkzWKpvFhp4Rit6KV5/1fp+2z0CgEAVESlQUhEQkKysUPv4rUfeFoiDbSiF3//dve5SmQhAIBa1EcQrl27dsKECU888cTJkyert6enp7/wwgvjxo2bOnXq999/Xw+V1GAePEHM+NV2dI+npVc0m9yCG7PVjYOFAAAq4fUg/OKLL6ZMmdK/f3+LxdK1a9fz5897Fh0/fjwwMPCuu+5q0qTJqFGjvvjiC28XUwMTdMGj/1H81ULJVu5pnNmW0/M0Ow3nFQIAqILG20/wr3/964033hg7diwRpaWlffzxx9OnT1cWjRkzxrNaYWHh+vXrR48e7e16atA1baO/tVPpxk8sI6cqLRyjZT01nda54gOkh5urd9cxAIBKePcPvcPhSE9P79mzpzLbs2fPPXv2XLpaTk7Otm3bunbt6tViriRo+CTbsb2Ok4c8LREG2j6En3dYWngMg0gBAPycd3uEeXl5siyHhYUps+Hh4Tk5OdVX2Lhx49ChQ4loxIgRjzzySI2HOxyORYsWrV+/vnrj008/3alTJyKy2+08z2u12r9ep37Ig4VfvBM4+XUWYFZaQok29qK7tgo2u/vRZOwmvYyKigrGmK+rUClRFIlIEARfF6JS+PD7kNPplCTJ6XTWcn2j0chx1+jyeTcIDQYDXfilJSK73W40GquvMGTIEFmWMzIyJk6c+PTTT7/zzjt/Kk6j6dat28CBA6s3JicnKxvhOK6ugpDa9+DOn6385OXQKXM5U1DVExlp611yv/9yLl74Zxt86Gtyu901fppQbzQaDSEIfQcffh9SglCn09Vy/WumIHk7CENDQ/V6fWZmZkhICBFlZmbGxsZeulrjxo2nTZv27LPP1ghCnufbtm07YsSIy26cu6BOSg26cxxjrODd58Ife5M3ByuNjQJp611yn01umdiMtjhe+Cd1+ObD9VLeebz/voIPvw9548Pv3Z8lx3F333338uXLichms61evTo1NZWI1q5dm52dnZ+fr6wmy/IPP/zQrFkzrxZzTeZBY43tep5/9zl3qdXTGBfAtg7mPzslTd3lFnHEEADA73h91Ojs2bP79Olz+PDh7Ozs5ORk5Yjg5MmTP/jgg7fffru4uDg6Ovr06dMajWbt2rXeLuaazIPGkiyf/3O/MDaA7R2uefBH9x3rXV/15ZsEYjcpAID/8HoQNmvW7Lffftu/f7/ZbE5JSVGOMO/fvz8sLGzIkCG//PJLXl5edHR0ixYtbpJdDeY7xxHJBf9+PvShWZrQaKUxUEur+vIf/Cp13eBa2kMzMA5ZCADgJ7wehERkMBh69OhRvSUhIUGZaN26devWreuhhutivnM8Z7Lkv/0Py7CHjJ36e9ofbs61sLAxW92TmnMz23Ic0hAAoOG7KTphNyFT92HhU/9V9uPawqWvShWlnvbuUWz/CM32HKnbBteRIlyHDQCgwUMQXpE2KiHiH+9owmLy3pxiP37A0x5loB8Gax5pwQ34r+uJ3e4Klw9rBACAvwpBeDVMow0aOjF4zNPWL/+vaPk8V0HV1QAY0fhbuEP3aK0OarPa9W0WuoYAAA0VgvDa9Mnto174UBvTJP+df1hXLXCXFCrtkQZa1ot/tyv/6E73PT+4fylGHAIANDwIwlphgj6w76ioF5ZwemPem1NKNnzsLqs613BQHDs+UjMojvXb5Bq1xX26FHEIANCQIAivA2cMDBr2UORz/5YdlXlzHy5aNtdx+ggRCRw93Jz7daT21mDqvM41bTdu7QsA0GAgCK8bHxRquXdq1EufCE1aFn+1MO+NyeU7NkiVZWYtzW7P/zpSq+Wo9WrXAz+6MawUAODmVx/nEfolTh9g6j7M1H2Y49SRil0bSzcuFRq3MLTtHtL6jrc6m2e155eekAZ/625koukp3JAEnHMIAHCTQhD+VbqmrXVNW8ui3f7L/spDP5as/VCb0MzQqstjye2n3Bf/+SlpxgHpn/ukh5pz45pyYXpflwsAAH+GIKwbTNAb2nY3tO0uiw778f324/vLtq4mWR6e3H5UcrsToW0Wnw1K/sp5RyQbfwt3T2OORw8RAODmgCCsY0zQGVK6GVK6EZErP8t+Is2Wti3y90VzjIEvN7r1YFnzpTubT9/TKDWR/1sS1z4MeQgA4GMIQi/SRMSZIuJM3YeRLDvzzooZx28/80v7sxuc1vP5vzfavTVxualpYrOkPu0TW4bVxe2FAQDg+iEI6wVj2qgEbVRCQJeBRCTZKyOyTydnnT53+njlrvX6zed26SPE0EZh8QmNmzbWRcZrwmOZUNv7LwMAwF+BIPQBTm/UJbXWJbVu1pOISHa7Kn/LSj+eeeCPjNDjP7ZxZ4VX5vABZiEiVhMWowmP1YRG8iGRmpAoLsDs69oBAPwNgtD3GK9p3bxx6+aNiXqcrZA3ZsrfZ0knMvLv0J7rI+W0yc2NOHNcsua6i/Jkt1sTEsmHRGos4bwljLeE88ERvDmYt4QxAQNSAQBuBILw5hIfwKa0YFNacE4pZnd+9LdZ0tvZ8q9OuXNL1iua6xNS2Zbls9J8l7XAXXzeeTLNXZTvLi1ylxQQ43hLOG8O5s2hnCmID7Rw5hDeZKmaDgjCvlYAgMtCEN6ktBz1iGI9ovjXbqMSkX7KlbfmSFPT9KdK4juEJXSNYnc04m6PZCFC1fqSvdJdUiCVWt2lhVJ5ibvU6szPksqLpbJid1mxVFFCjHEBQbwpiAswV/0zBnLGQC4gkDMEckYTZwzkjCZmMDEenwoAUBH8yWsAggQaksCGJPBEVCLSrjx5V770r5/dBwrkRibWJYJ1jmCdwg0tIxK0kQlX2ogs2t3lJVJ5sVRRJlWWShVlUkWpKz9LspVLleVSZZlUWSbZymVbBfE8pw9gBhNnMHL6AKY3csZATm9kOiOnNzCdkdMbmd7olshpCWE6PWcwMUHPNBj4CgANEoKwgQkS6M54dmc8T0QuiQ4XyXvz5R258vwjUlaF3DaUdQhj7cNY+1DW3MKqn7bPBL0mRE8hkdd8Clm0S7YKyVYh25X/KyVbuWSvlO0VrqJS2W6THJWyrcJZWeZwOWXRIdnKZdEuS25Ob2SCgWkFTm9kOgPTCkxn4AQ9abScwcQ0WibomM7IeJ4zmIjXcDo9aQSmFZigYxotpzMQp+H0BuJ4772BAAA1IAgbMA1HHcJYhzD2KBERlTpp/3n5YIH8Tab8Srp0rkJuHcLahrK2oaxNCGsdzEy167MxQc8Lej4o9OqrlZWVBQYGXpyX3JLdJjsqZacoOWyyvVJ2OWWHTRLt5HJKtnLZKUqV5bI1X3a5JHsFuV2Swy47RXKJkmgnl0tyVJLkluyVJEmc3kgcx3RGxvFMp2e8hmkF0giM1zCdnhjjDCYi4gQ98RrG80xnICKmMzKOI47n9EYiYlqBaQUiYnojMY6IOIOJGBERZ6iqnDOarus9BwD/gyD0H2Yt9Y1hfWOquoGlTjpUKB8ulA+cl5f8Kh0vlqOMLCWE3RpMrYLZrRaWbGFCHd59hOM5o4nqKFckewVJsmyvlGW37LDLbpfsdJDLKbtcsmgnWZZs5UQkiXZyu2SXS6osJyLZmi9LUlWaEsmiQ3Y5iUi2V5AsE5GyGhFJtrKqiQstSvQSEdPqmEYgImKMMwQoS5lWRxf2/V7Mzgt5XDWtD7j4ZhgCiFX9IJhGy7QXRyoxvZFV6/IyvZFxF38MTDAQX22pRltjlBPTCEyrk0SRiCSXQETEcUrwA8CNQRD6LbNWGW5T9efYLdNvJfJRq3zMSqvPyHOsUka53NjEWlhYcwvdamHNLSw5qLa9Rm+rCpV67K4p0UtEstMhu0QiIlmWbBXKUlm0k9tFRLIsyxcaPXlcNW2vuLg1W1X0EpHsEmWn6Fkk2ytlyf3nWenirGgjd7WlLqcsOqrXKbtE2fmnFpIkJfirU/Y213yRnIbTGS7z4q8cpdW/AVzmcdXy/vIr6AxUi7FXF79P1ALTaG74ZCEm6OtkLJhot1foL1MD0wrVv/SAQpali781f5nb7dbd1reutqZAEKoFz6i5hTW3sHubVLWIEp0skX8tln8tpm/Oym8dkU6UyBaBNQuiZkGsWRBrFkRNzSwxkOlUcMyuWn+uAewsFUWRiARBuNIKnt7wnxrdTlm0X2bty0Vp1UOcDrpkOxcfZyunq95zU3LYlC8QV3fx+0QteHYA3AC5uECuRT3X5HY6Re1lvh/Izku+pgARYxwzBFx7vdqRJEno0KeutqZAEKqXwFGrYNYq+OI3epnobLl8soR+K5VPlMj/OyefKqXMcjnSwJqaKcnMkswsMZASA1mimakgHBswJuhw5qj31DxADvXI6XRK1Xai1AkEIVzEiBJMLMFE/WIvpqNLoswK+VQpnS6Vfy+V9+bT72XS76Uyz3RNzK7GJtY4kBqbWJNA1shECSYWdMVeCgDAzQhBCNeg4SgxkCUGEsX+6WhQRkF5AQVklMkZ5XSiRP42S8oop8xymWeUYGKNTCzBRHEBLD6AGplYvIlijExbh2NzAADqCIIQblCoTm4cyG675JaKVgdlVsiZ5XJGGWVVyD8XUWa5dLaCcivlUD3FGllsAEswUbSRxRopNoDFGCkugAXeHIN0AECFEIRQx4J1FKxjKSE1A1KSKdcmZ1XQuUr5bDmdq5SPWym7UsqppLMVsixTXACLNFBsAIsyUEwAizRQtIFFGynCwMJxRXEA8BoEIdQTjlGMkcUYiegyA+4rXJRVIefZqv7PrpAPF1KuTcqppDybXCpSuIFFGSjKQBEGFm2kcD0L11OUkUUaqqa5qw3jBwC4IgQh3BQCNJQcxJKD6LIxKUp03ibn2CjPRvk2+VwlnSmT952nnErpvJ3O2+UCO4XpKVzPwvQUZWDhegrTszA9RRgoQs9C9RSmZ6E6wkFKALgUghAaAIGj2AAWe+EyL5eu4JbpvJ0K7PJ5O+VWygV2KrDLvxTT1hw6b5MK7FTokAvsFKilcAML1VGYnkJ1LFRPoToWpqcwPYXoWKiOQnQUqmd6nBoCoCYIQvAHPKMoA0UZlIy84k7SIgedt8uFdip0KP9TgV0+U0aFDipySEp7kYN4RiE6FqKjEF3VRLCOQnQsWEchOgoWWLCOgi9MAEBDhyAEFVGCja6wA9ajwkVFDrnIQUUOKrLLRQ6yilTkkH8vI6uDrA7JKioTcrFIFoGCdSxYVzVhEcgikEWoagkSmEWgIIEsOgoSmBmDYwFuPghCgJoCNBSgYfFX3hPrIRMVO8gqylYHFYtV0VgsUrFD/qWYih1UIkrFIpWIVCxSiSiXO8miI4vAzFoKEihIYGaBggQya8kiMIuOzFoyC8ysJbNAgVoKFligljQ4tAngTQhCgBvHLpwuQoGehquRZCoRySrKJSKVilQiyqVOKhGp1ElWUf69jEqdVCpKpU4qFanMScWiXCqSwJNZSyYtswhk1lKgwAJ4ZtJQiMFt1jKTlkxaCtSSRWAmLZk0ZLowjcFBALWBIASoPxy7EJxVanXOR6WLypxU5pRLRCoRqcwpW21yuZMqJFYiyrk2KndSuZOsoqRMlLuoRJTLnMQzCtBUhWKAhgK0ZBFYgIaMGqXHyYwaZQUyaphBQyuEt2cAAB3tSURBVBaBAjRk0JCSr8hRUAkEIcDNzqgho4YiDRfjUxSJiIRr3U/S4b4YipUuqnCSVZSViTInlTrlAjtVusjqoEqXZHNTiUgVLiV35TInEVGglgK1TM8rE6TnKVDLArWk48ksUICG6XiyCGTQkJ4ni8D0PBk1ZNYqvVhm1JAabl0CDR2CEMBv6XjS8RSqq97vvI7rDjglKndSmVO2uancSaVOsruo3CWXOcnuojInVbjkMiedLiWbi+xusoqS3U02F5U6yeGmMqdc4SRRIotAAk8mDQvQksBRsI4EjgKqzWo5MmmqUtOsJS1HQQITeFI6rzqezFqm4ciC67mDdyAIAeDytFyNHbmK676Ej9VBokQVLrncSaJExQ5ySFTpksud5JTI6iCnRFZRzq4kh5tKRHLJVCJKDjdVuqjCRaKbSkTZJVOJSBqOArWk48h4IUeVwUQWgXGMLELVCkrQ6nkyXIjSAA0TOFL29yoPMWsZz8gsEI9rEqkeghAAvOvC2ZZ/NVDpQifV7iabu6q7Weokt0TFouyWqUSsWsEhkVWUlX5qpYscbqpwSaJESvSWOcklUYkoS0SlIrnlqmgM1DIN88Qk8RdalPhUAtXAMz1PzKWxBEgCRwFa0jAK1DLGqjqsQQJxjEwapuVISWK4+eGnBAANhtJJJaI6iVUPJU1LnbJbropGpaXMKbvkqvhU+qaVLtkqUqmN3DZZlKjCSS6ZypySMh6YiIpFkmUqd8lOiZQdxUonlYiCBUZUFbTKUVVlnzBd+K4QoCGBr+rOepJVWV9ppAtBq0QsI7IIjIiUzjHcMAQhAKidcqGDS3YC05XytazMFRhoqOXGXRIpI4+sokxU1R9VeqvKHmNZpmKRiKoSV5TIKsqyTKdLybO+KFGFSyKiYgfJVPVwiahElD0P9CSu0pHVcmSqmiUNRzwjs/ZiajIii46Iqvq4RGTRESNSurlEVZ1auhDSnq0ZNUzHEZFf7VVGEAIAeJHmQi/2ckFLf7EvW50ncZWOrLKXmC5EqUumMqdMRMouZU8X1uaWrSIR0e9lRERKN5eIyl2SUyIisjqI6OLWKl2yQyIiKhFJkoljFCQQUdWBWyLyDBX2XAvCIjBGF9f0BLanQ6zjyaghoqqOMlWLW8/WPBNamcLr+tKGCEIAAH9wrcSlOgxdD0+gKgOgiKoOytKFHc50oSvsWdMT2Erfl4iUgVF0oadLRJUuSYlbz9aqTxwfQfo6zUIEIQAA3CDlGhEX1EcAO51OSarD7RER4QArAACoGoIQAABUDUEIAACqhiAEAABVQxACAICqIQgBAEDVEIQAAKBqCEIAAFA1BCEAAKgaghAAAFQNQQgAAKqGIAQAAFVDEAIAgKohCAEAQNUQhAAAoGoIQgAAUDUEIQAAqBqCEAAAVA1BCAAAqlbbIFy6dOkXX3yhTOfl5Q0cODA4OLh///5ZWVleqw0AAMDrahWEsixPnz5dFEVl9rnnntu1a9fYsWP/+OOP+++/34vVAQAAeJmmNiuVlpaeP3++Xbt2RORwONasWfPCCy88//zz6enp7du3P3/+fHh4uJfrBAAA8Ipa9QhtNhsRmUwmItq5c2d5efmQIUOIqGXLlkR09uxZb1YIAADgRbUKwvDwcJ1Od+DAASL64osvoqKilAgsKCggIqPR6NUSAQAAvKdWQcjz/NixYx966KHBgwcvXbp0woQJHMcR0b59+3Q6XaNGjbxcJAAAgLfU6hghES1atCgqKiotLe25556bOXOm0rhnz57Ro0cbDAavlQcAAOBdtQ1CvV7/6quvembPnz//66+/PvPMM2FhYd4pDAAAoD7U9jzCSZMmzZs3T5netWtXUlJSjx49GjVq9N///tdrtQEAAHhdrYLQ7XavWLEiJSVFmZ0+fXpcXNx33303fPjwqVOnut1ub1YIAADgRbXaNVpUVGS325s2bUpEBQUFu3fvXrp0af/+/W+99da4uLizZ882btzYu2UCAAB4R62CkOd5InK5XET07bffut3uvn37EpFygLCgoODqQZiWlvbBBx+Iovj3v/9deaAiJyfnP//5T3p6uk6nGzx48NChQ//CCwEAALgRtdo1GhISEhkZuXz5cofD8dFHH6WkpMTExBBRZmYmXYjDK/ntt9969+7drFmzbt263XvvvVu3bvUsev/99w8ePNitW7eWLVs++OCDCxcu/GuvBQAA4LrVdtTo7NmzH3300blz5xLRZ599pjRu3rw5NDQ0ISHhKg9ctGjRqFGjnnrqKSIqKCh46623evfurSx66aWXlPMRiYjn+U8++eTxxx+/sZcBAABwY2o7avSRRx7ZvXv3okWLdu/e/be//U1pNBgMc+fO9YTZZe3Zs6dnz57KdM+ePffs2XPxuas9MC8vDxcsBQCA+lfbHiERdezYsaKiYseOHV988UV0dHSrVq3uv/9+jeYaW8jNzQ0NDVWmw8LCioqKHA6HTqervs7x48fffvvt7777rsZjHQ7HokWL1q1bV73xmWee6dSpExHZ7Xae57Vabe1fAtShiooKxpivq1Ap5VYwgiD4uhCVwoffh5xOpyRJTqezlusbjcar99ao9kFotVqHDx/+008/ERHP88opE23atPnmm2/i4uKu8kCDweC5f5PD4dBoNDWiKyMjY9CgQfPnz+/cuXPN4jSa7t27Dxw4sHpjcnKycnVTjuMQhD7kdrtxmVlfUb6AIgh9BR9+H1KCsEZv6iqumYJU+yB87LHHDh48uGDBgvvuuy8iIqKoqGjDhg3PPvvsuHHjqo9/uVRsbKwypoaIMjMzY2JiqpeVlZXVr1+/Z5999sEHH7z0sTzPp6SkDB8+/LJb5i6o5UuAuoU334eUdx7vv6/gw+9D3vjw12pbdrt9zZo177zzzuOPPx4REUFEISEhEyZMWLp06bZt27Kzs6/y2NTU1JUrVyqnXixfvjw1NZWIduzYcejQoby8vAEDBkyaNGnq1Kl18VoAAACuW616hIWFhQ6H4/bbb6/R3rVrVyI6d+5cbGzslR57//33r1y58rbbbjOZTAUFBdu2bSOit99+u0mTJg6H47fffvvggw8++OADIoqIiNi9e/cNvxIAAIAbUKsgDA0N1el0u3btatWqVfX2Xbt2EZFyTuGVGI3G7du3Hzx40Ol0duzYUTmk9/7772s0GpfL9eSTT3rWVE7bBwAAqE+1CkK9Xp+amvrkk086HI5Ro0ZFRkYWFRVt3Ljx2Wef7d2791W6gwqO4zp27Fi9xXMOPm5eAQAAvlXbwTLvvvtuVlbWtGnTpk2b5hk1mpKSsmzZMm+WBwAA4F21DUKLxbJt27Zt27b99NNPBQUFwcHBnTt3HjhwIPZnAgBAg3YdJ9Qzxnr37u25QBoRbd68ecaMGQcPHvRCYQAAAPXhL52KUVRUlJaWVlelAAAA1D+cEwoAAKqGIAQAAFVDEAIAgKpdY7CM2+2WZflKSyVJqut6AAAA6tU1gjAxMdFzyWwAAAD/c40gHDduXGFhYf2UAgAAUP+uEYSvvvpq/dQBAADgExgsAwAAqoYgBAAAVUMQAgCAqiEIAQBA1RCEAACgaghCAABQNQQhAACoGoIQAABUDUEIAACqhiAEAABVQxACAICqIQgBAEDVEIQAAKBqCEIAAFA1BCEAAKgaghAAAFQNQQgAAKqGIAQAAFVDEAIAgKohCAEAQNUQhAAAoGoIQgAAUDUEIQAAqBqCEAAAVA1BCAAAqoYgBAAAVUMQAgCAqiEIAQBA1RCEAACgaghCAABQNQQhAACoGoIQAABUDUEIAACqhiAEAABVQxACAICqIQgBAEDVEIQAAKBqCEIAAFA1BCEAAKgaghAAAFQNQQgAAKqGIAQAAFVDEAIAgKohCAEAQNUQhAAAoGoIQgAAUDUEIQAAqBqCEAAAVA1BCAAAqoYgBAAAVUMQAgCAqiEIAQBA1RCEAACgaghCAABQNQQhAACoGoIQAABUDUEIAACqhiAEAABVQxACAICqIQgBAEDVEIQAAKBqCEIAAFA1BCEAAKgaghAAAFRNUw/P8eOPP3766accxz344INdunTxtNvt9t27dx88eLCwsHDu3Ln1UAkAAEANXu8R7tmzZ+jQoZ07d27btu3AgQMPHz7sWZSWlvbUU0/t2LHjzTff9HYZAAAAl+X1HuHbb789bdq0hx9+mIhOnz69cOHCJUuWKIvuuOOO9PT0Y8eObdiwwdtlAAAAXJbXe4R79+7t2bOnMt2zZ889e/Z4+xkBAABqz+s9wtzc3NDQUGU6PDw8Jyen9o91OBwLFy5cu3Zt9cZnnnmmc+fORGS323me12q1dVgt1F5FRQVjzNdVqJQoikQkCIKvC1EpfPh9yOl0SpLkdDprub7RaOS4a3T5vB6Eer1e+aUlIrvdbjQaa/9YjUbTo0ePQYMGVW9s0aKFshGO4xCEPuR2u6/rpwl1SKPREILQd/Dh9yElCHU6XS3Xv2YKUj0EYVxcXGZmptKHy8zMjIuLq/1jeZ5PSUkZNmzYZZdyF9RNoXCd8Ob7kPLO4/33FXz4fcgbH36v/yzvvffeZcuWybIsSdKKFStSU1OJ6Icffvj111+9/dQAAADX5PUe4bRp03r16tWlSxeXy8XzvDJ8dPbs2SNGjIiMjLzttttEUZQkKSkpKSIiYvfu3d6uBwAAoDqvB2FISEhaWtqBAwc4juvQoQPP80S0Zs0ag8FgNBq///57z5rKIgAAgPpUH1eW0Wg01S8oQ0QRERHKRGJiYj0UAAAAcCU43gsAAKqGIAQAAFVDEAIAgKohCAEAQNUQhAAAoGoIQgAAUDUEIQAAqBqCEAAAVA1BCAAAqoYgBAAAVUMQAgCAqiEIAQBA1RCEAACgaghCAABQNQQhAACoGoIQAABUDUEIAACqhiAEAABVQxACAICqIQgBAEDVEIQAAKBqCEIAAFA1BCEAAKgaghAAAFQNQQgAAKqGIAQAAFVDEAIAgKohCAEAQNUQhAAAoGoIQgAAUDUEIQAAqBqCEAAAVA1BCAAAqoYgBAAAVUMQAgCAqiEIAQBA1RCEAACgaghCAABQNQQhAACoGoIQAABUDUEIAACqhiAEAABVQxACAICqIQgBAEDVEIQAAKBqCEIAAFA1BCEAAKgaghAAAFQNQQgAAKqGIAQAAFVDEAIAgKohCAEAQNUQhAAAoGoIQgAAUDUEIQAAqBqCEAAAVA1BCAAAqoYgBAAAVUMQAgCAqiEIAQBA1RCEAACgaghCAABQNQQhAACoGoIQAABUDUEIAACqhiAEAABVQxACAICqIQgBAEDVEIQAAKBqCEIAAFA1BCEAAKgaghAAAFQNQQgAAKqGIAQAAFVDEAIAgKp5PQjdbvfixYvHjBkzffr0vLy8Gks3b958//33T548OT09/Xq3fOTIkd9++62OyoTrI0nS+vXrfV2Feh0/fvz48eO+rkK91q1bJ8uyr6tQqZMnTx49erRut+n1IJw9e/bixYtTU1OtVmvv3r1dLpdn0bfffjtmzJjevXsnJyf36tXr9OnT17XlFStWbNy4sa7rhVpxOp0TJ070dRXq9dVXX61evdrXVajX/fff73a7fV2FSq1fv/7zzz+v221q6nZzNdhstnfffffbb7/t2LHjPffc06xZs2+++Wb48OHK0rfeemvmzJkTJkwgoqNHj7733ntvvfWWV+sBAACowbs9wpMnT4qieNtttxERY6xHjx579uzxLN27d2/Pnj2V6Z49e+7du9erxQAAAFzKuz3C3NzckJAQxpgyGxYWlpOTo0zbbLbS0tLQ0NBLF3kEBQVNnz79xRdfrN6YlJQUHBxMRGfOnBEEYdu2bV59CXBZsiwbjcbBgwf7uhCVyszMJKIDBw74uhCVMplMQ4cO9fxlg/qUlZXlcrlqf4x80aJFiYmJV1/Hu0FoMBgcDodn1m63BwQEKNOCIPA871lqt9uNRmONh8+cOdNgMNRobNq0qRKE4FuPPfaYr0sA8I1HH33U1yVAbYWFhV1zHe8GYWxsbFFRUXl5uclkIqLMzMyOHTsqi3iej4qKyszMTE5OVhbFxcXVeLjZbJ4zZ45XKwQAAJXz7jHCpKSkVq1aKSN8zp07t2XLltTUVFEUP/vss7KystTU1GXLlhGR0+lcuXJlamqqV4sBAAC4FPP22TDbtm0bOXJkhw4djhw5Mnbs2Hnz5hUWFoaFhZ04ccJkMvXq1Ss8PLy0tDQ8PHzTpk16vd6rxQAAANTg9SAkouLi4kOHDiUkJChHLCVJys7Ojo6O1mg0Tqdz//79er2+Xbt213Xkeffu3StXruR5/sEHH2zVqpXXaocqxcXF69evP3jwIMdxAwYMuPPOO4no888/P3v2rLJCWFjYgw8+6NMa/dnSpUvz8/OV6ZiYmHHjxhFRRkbG4sWLS0tLhw0bNmjQIJ8W6M9++umnXbt2VW956qmnNm7cePLkSWU2ICBg6tSpvijNb4mimJ6enpaWZrfbn3zySaVRkqRPPvlk7969CQkJjz/+uNlsJqLS0tKFCxdmZmZ26dJlwoQJHHcjuznr4xJrFoulV69ennE7HMfFx8drNBoi0mq1d9xxR/v27a83BQcNGtS0adPw8PBu3bqdOnXKK3VDNR9++OF//vOfxMTE+Pj4Bx544M033ySi999/f9++fVar1Wq1lpaW+rpGf7ZgwYJDhw4pb3VZWRkRFRUVdenSxeVytWvXbsKECV999ZWva/RbNpvNesHmzZs/+ugjjUazYsWK7du3K43FxcW+rtHfbN26dcKECWvWrJk1a5an8YUXXliwYEHXrl3T09M93/wGDRqUnp7etWvXBQsWzJgx4wafT26AUlNTZ82apUxPmjTpiSee8Gk5quB0Oj3Ty5cvT05OlmW5R48eX3/9te+KUpG2bdtu2bKlesv8+fP79++vTH/yyScdOnTwRV2q06tXr7lz58qyfM8993z44Ye+LsfP7du3LzAwUJkuLS0NDAz8+eefZVl2Op2RkZHbt2/fvn17RESEKIqyLB85csRsNpeVld3AEzXIi27v2rWrT58+ynSfPn127tzp23rUQOnBK6xWq+cMlq+//vqll1768ssvq188D7zhiy++mDVr1urVqyVJIqKdO3dW/y04ePCg3W73aYH+78yZMzt37hw/frwy+91337344ovLli2rfpIYeMmRI0f0en3r1q2JSKPRdO/efceOHTt37uzRo4dWqyWiVq1a6XS6n3/++QY23vCC0O125+fne04NCQ8PP3funG9LUpXs7OxXXnll5syZRNShQ4dGjRppNJrXXnutX79+yELvuf3222NiYjiOmzFjxtChQ2VZzs3Nrf5bQESXXpIC6tZHH300cODAmJgYImrZsuUtt9wiCMK7777buXNnm83m6+r8XPUPPBGFh4fn5OTUaIyIiLix3wLvnkfoDTzPC4LgdDqVWVEULz3pHrykoKBg0KBBjz/++F133UVE8+fPV9qfeuqpW265ZdOmTcOGDfNpgX7rvffeUyYef/zxpKSk7du363Q6URSVRmUCg669SpKkZcuWLViwQJl9+eWXlYnp06e3bt165cqVuAy9V+l0Os+ffSISRTEkJISIKioqPI0Oh+PGfgsaXo+QiKKjo7OyspTps2fPxsbG+rYelbBarQMGDBg6dGiNi94RkclkSk5OVq77BV4VEhLSpEmTzMzM2NjY6r8FWq02IiLCt7X5t82bNzscDuUrYHWCILRp0wYffm+LiYnJzc31ZGFWVlZMTExsbKxn4LrL5crNzb2xOGiQQThixAjlJH1Jkr788ssRI0b4uiL/V1JSMmjQoJ49e77++utKiyiKnt1Bp0+fTktLa9u2re8K9Gd2u91zFOro0aPHjx9PSUkZMWLEmjVrlOOCK1euHDp0KM/zPi3Tz3300Ufjx49XDkdJkqSM3SWi3NzcH3/8ER9+b0tJSYmIiNiwYQMRZWdn79ixY/jw4cOGDduxY4fyjXDDhg0RERFt2rS5ka3X3QCf+pOdnZ2YmNi/f3/l1IvS0lJfV+T/XnrpJY7jOlzQtWvXjIwMi8WinFNoNpufeeYZX9fot44cORIaGjpo0KBBgwYFBgbOnj1blmWn0zlw4MA2bdoMHz48MjLyyJEjvi7TnxUUFOh0umPHjimzymUj+/btO2TIEIvFMnHiREmSfFuhn8nJyenQoUPz5s15nu/QocOIESNkWV6zZk1oaOjIkSPj4+OnT5+urPncc8/Fx8ePGjUqLCzshgex18cJ9d5gs9l++uknQRC6du2qfEcDryoqKqp+shRjrEmTJllZWcqtolu2bBkfH++76vzfH3/88csvv/A836pVK2WwBhFJkrR3716r1dq1a9egoCDfVujfbDZbXl5e48aNPS15eXk///yzKIq33nprkyZNfFeaf3K5XNX3Nmu1WuUvTHZ2dlpaWpMmTapfR+Xo0aNnzpxp3779DR8ma6hBCAAAUCca5DFCAACAuoIgBAAAVUMQAgCAqiEIAQBA1RCEAACgaghCAL+Vl5f3wQcf4BqkAFfX8K41CtDQ9e/f//Tp0zUaBw8evGjRorp9otOnT0+ePDk5OTk6OrputwzgTxCEAPXt7Nmzbrd70qRJ1RtvvfVWX9UDoHIIQgAfiI+PV25ldSVut7ugoMBiseh0uhqLysvLy8vLw8PDL724qNPpLCwstFgsl16Dv6Kiwu12m83m6o2SJJ0/f95kMgUEBNzoSwFo8HCMEOAmMnr06MGDBy9ZsiQqKioqKio0NHTWrFmeyz8dPXq0R48eZrM5Ojo6IiLi1Vdf9SwqKip64IEHgoODo6OjDQZDSkqK5w6lhYWFd999t9lsDgoKSklJUa6KV15ePn78eIPBEBUVZTKZ4uLivv76a5+8ZACfQ48Q4CZSXl6+d+/eEydOfPrpp/Hx8Z9++unLL78cFBT01FNP5ebm9unTJzAwcP369bGxscuXL3/xxRedTuecOXPsdnu/fv1OnTo1d+7cHj16VFZWbtmypbKyUtnm448/Pn78+H/+8585OTmPPfbYpEmTdu/ePWPGjE2bNn3++ecpKSllZWUHDhzQaPDXANSqLi4UDgDXITk5+dLfxIULF8qyrNzubu/evZ6VBw8eHB4e7nK5Zs2axRg7fPiwZ9Hdd98dEBBQWlr64YcfEtGaNWtqPNHOnTuJ6OGHH/a0vPvuu0R07ty5O+6447777vPyCwVoGPAdEMAHmjVrNmvWrOotHTt2VCbi4uI6derkaU9NTd20aVNWVtahQ4datGhR/XZr991339dff33w4MHvv/8+JCTk7rvvvuxz3XnnnZ5pZUhOZmZmu3btlixZMm3atFGjRnXp0gXdQVAzfPoBfCA8PHzMmDGXXRQVFVV9VrnpUlZWVmZmZo2zIJSbzhQWFhYUFFzlBjTBwcGeaUEQiEgUxblz5/I8/9lnny1cuNBisYwaNWru3LkhISF/4TUBNFQYLANwcykoKKg+m5+fT0TR0dEBAQGXXRQUFGSxWPLy8q7rWQIDA//v//4vLy/vwIED06ZNW7Zs2SOPPPKXawdokBCEADeXP/744/jx457ZzZs3m83m+Pj4Ll26HDt2LCMjw7No48aNgiC0b9++Z8+e+fn527dvv97nUm7/PWfOnJEjR+7Zs6dO6gdocBCEAD5QUFDw5Z999913yiKDwTB+/Phjx46VlZUtWrRo1apVjz32mFarnTJlik6nGz169NGjR61W64IFC5YvX/7QQw+FhIQ88MADiYmJY8eO3bBhQ2lpaW5u7meffXb48OGrFDBt2rRNmzbl5+e7XK4DBw5s3769Q4cO9fLSAW46OEYI4AMnTpwYPXp09ZaUlJQBAwYQUfv27Xv37t22bVuXy8UYGz9+/Jw5c4goMTFx48aNDzzwQOvWrYmI5/mJEyfOnz+fiAIDA7ds2fLAAw8MGzZM2VpYWNimTZuuUkBGRsbw4cNdLpcyO2DAgMWLF3vhhQI0AEy+cEIuANQPSZIu/b1jjHEcN2TIkJKSkp9++qmoqOjUqVOxsbE1RsFIkvTrr7+Wl5c3bdr00rEt2dnZ2dnZZrO5adOm1xwIWl5e/scff9hstoSEhIiIiL/+ugAaKAQhwE3EE4S+LgRARXCMEAAAVA3HCAFuIlOnThVF0ddVAKgLdo0CAICqYdcoAACoGoIQAABUDUEIAACqhiAEAABVQxACAICqIQgBAEDV/j/v9xc/Kg+QewAAAABJRU5ErkJggg=="
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Demonstrates underfitting: training loss not close to 0\n",
    "# Also slight overfitting: test loss higher than train\n",
    "trnloss,tstloss = Array{Float32}(lin[2,:]), Array{Float32}(lin[3,:]) \n",
    "plot([trnloss,tstloss],ylim=(.0,.4),labels=[\"trnloss\" \"tstloss\"],xlabel=\"Epochs\",ylabel=\"Loss\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlgAAAGQCAIAAAD9V4nPAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nO3dd3wUZf4H8O8zs70km2x6IwUIJYgSQSAgSD8UxV8EFeVsoaon4o+D3ykcZwV7O0VBQSwIcggcIiCIYAGB0JEOSUggfVM2m60zvz8mLEtoAbNZkvm8/7jXzDyTmWefW/fDM88zM0wURQIAAJArLtAVAAAACCQEIQAAyBqCEAAAZA1BCAAAsoYgBAAAWUMQAgCArCma4Bzl5eVvv/32yZMne/ToMW7cOJ7nvUWHDx/etm3boUOHhg8f3rVrVyI6cODA8uXLjxw5Eh4e/sgjj3Ts2LEJaggAALLl9x6hKIqDBw8+duzYnXfeOX/+/KlTp/qWjh8/fsmSJfPnz9+zZ4+0Zdq0aeXl5QMHDlQqlV27ds3OzvZ3DQEAQM783iP86aef8vPzt27dyvN8Wlpat27dZsyYERQUJJVu3LiRiHr37u3df8WKFRxXF88nTpxYvHhxenq6vysJAACy5fce4ZYtW3r37i1dDm3fvr1er9+3b9/lKsSdq1JJSUlYWJi/awgAAHLm9x5hYWGh2Wz2roaFhZ05c6Yhf/jll18eOnRo2bJl9bZ//vnnixcv9s1LxtjSpUulZY/H4zsGeR5RPHDoRPt2KRy7qk8ADXK5lgd/QssHClo+UARB8I2Ay1MoFIxd4Uff70Go0WhsNpt31eFw6HS6K/7VunXrJk+e/N1335lMpnpFM2fOfPrpp2NjY71bGGPer6PNZjMajRc9puh2Bc2f4py9Qo+vrh9cpuXBr9DygYKWD5Ta2lqDwdDAna+YgtQEQRgXF7d7925p2eVynTlzxjfDLmrjxo2jR49etmzZzTfffGGpQqEYOHBgamrqRf+W47hL/UtB5HlG4qXL4U+5TMuDX6HlAwUtHyiN3vJ+/3/xrrvu+vXXX3Nzc4no22+/jYmJueGGG3Jzc1evXn3R/X/77bf77rvv66+/zsjIaPTKMCK8bAMAAHz5PQhbtWr1zDPP9OjR44477nj88cffeustxtimTZsmTZpERI8//nhKSsqOHTv+8Y9/pKSk/PjjjxMmTLDZbFlZWSkpKSkpKdJujQc5CAAA52FN8z7Co0eP5ubmdu7cOTw8nIisVmtFRUVcXFxRUVFNTY13t6ioqPLycqfT6d1iMBgiIiJ8D5Wamrpy5cpLXRqtrq6+5FV7wZPzzJ2hr34XpPzzHwjqu1zLgz+h5QMFLR8oVqu14WOEDdEUT5YhojZt2rRp08a7ajAYpI8RGRlZb8+GTKW5VowROoQAAHAeOY30MsbEpukAAwBAsyGnICQiQpcQAADOI68gZAyzZQAA4DxyCkLGGGIQAADO10STZa4fSEIAaBY8Hs+2bdsCXYvrS9euXRWKxo8teQWhyKTbRfCwUQC43lVVVfXu3Vt6USsQ0Y4dO4qKikJDQxv9yPIKQgCAZsRoNG7ZsiXQtbhe+CMCJXIaIyQSRUyWAQCA88gsCFkTPUkHAACaC3kFIeFhowAAcD55BaFIDPNGAQDAl7yCkIgJ6BICAIAPuQUhAADAeWQXhOgQAgCAL3kFoYiHjQIAwPlkFoTERFEIdC0AAOA6Iq8gJFwaBQC4Vi+99NLBgwcDXYvGJ68gFHHzBADAtVq2bFlBQUGga9H4ZBaEGCMEALgm77777tGjR6dOnTpw4MAlS5YsXrz4lVdeGTduXFxc3Pfffz9x4sSvvvrqtttuS0xMHDt2rMPhIKJNmzb169cvMTHxzjvvPH78OBG9+uqrH3744d133x0bG3v8+PHbb7998eLFN998c79+/QL40eT10G1RZCL6hADQbJU7yO5pih8xg5IFKc/bkpWV9cknn0yfPr1Pnz5arfadd955/vnnP/vss1mzZvE8P3369O3bt3/55Zcmk2nIkCGff/55z54977333mXLlt18881ffvnlsGHD9u7de+DAgXXr1i1evLhjx456vf7HH38URXHZsmUajaYJPtSlyCsICZdGAaA5e+Z3z9r8ppjxN7E9/9xN510y1Ol0CoXCYDCEhIRIW2699daRI0d6d5g0aVLbtm2J6J577tm5c+f+/fvvu+++9u3bV1RU9OnT5/nnn//jjz+I6P7777/11lu9fzVjxoyEhIQm+ESXIa8gxEO3AaBZm38rT8QHuhZ14uPjfVcjIiKkBZ1OZ7PZioqKDhw4cODAAWlj69atpeul9f6q3mpAyCsICe/kBQC4VhzH+fYlOO5ys0wSEhJiY2PffffdetsZY5dZDQh5TZYh3D4BAHCtkpKS1qxZk52dXVhYeMWdx48f//nnn3/zzTcWiyUvL2/OnDlOp7MJKnkN5BaEeOg2AMA1mjVrltVqffbZZzdv3pyYmNi+fXtvUbdu3cxms7SckJDQoUOH1NTUDRs2LFq0qEePHpmZmUeOHOE4rmPHjr4jgv369VOr1U39MS7Q/MbMUlNTV65cmZqaetHS6upqo9F4qb89/Eym8v8+Sw4z+K128nX5lgf/QcsHir9b3mKxJCcnWywW/52ieQkNDT127FhoaKjVajUYGvNnXF49QtxHCAAA9cgrCAljhAAAcD55BaGIIAQAgPPJKwiJGMMt9QAA4ENuQUg2d6BrAAAA1xN5BSHHsSOVeB8hAMC1WLBgQU5OzqVK582b10zfTSGvIOQZHa7ApVEAgGvx3nvvHTt27FKlb7zxRm5ublPWp7HILAg5hiAEALgG8+bNO3ny5AsvvDBy5MiVK1fm5+dnZWX17Nlz8ODBn3zyyQcffFBQUDB9+vSRI0euXbuWiNavXz9y5Mh+/fq9+OKL0jNlnn766XXr1j344IP9+/fPzc2dMGHCf//738GDB7/88suB/WjyetaogqNDCEIAgKs3aNCgN95447777uvWrVtsbGxWVlaXLl2mT59eUVGRl5fXqVOn11577cEHH7zhhhsSEhLWrFnz+OOPf/zxxzExMTNnznzmmWfee++9NWvWbNiw4dVXXw0LC6usrFywYEFlZeXLL7/cuHfHXwN5BSHP2BmbUOsmrbw+NwC0EGWfPF97YGsTnMh42z3Bwx713ZKQkKDT6dq0aZOenk5E5eXlWq02IiKiVatWnTt3JiKNRpOamiqVvvbaay+++GL//v2J6P33309JSZGevj158uQhQ4YQ0d69e91u95w5c4KCgprg41yevAKBMZZkoEOV4k3mwD/vHADgapkfnU5ik8z4Y1cYOHvnnXeefvrpV155pXfv3tOnT+/evbtv6bFjx6ZMmfLcc89Jq+Hh4VVVVUSUkpLi3Sc8PPx6SEGSWxASUaqJHbAgCAGgeWKMWMDeR8h8XunatWvXX375pbS0dO7cucOGDSsuLvYtjYyMnDp1amZmZr0j8Dx/0eXAktdkGWLULpgdsGCYEADgqsXExOzYscNisdjt9u+//97hcISFhfXt29ftdouiGBMTs337dovF4nA4Hn300eeff16aRFpTUyNNn7luySwIibULFvcjCAEArt7MmTM3b97cs2fPhQsXfvXVV0lJSbGxsRMmTFi0aBHHcS+99NLq1at79OjxzTffjB079rHHHhs0aFBUVFSHDh1++uknIoqPj9doNNKhVCqV7/uYAkter2E6888HbGPeGfCr6eS9srsm7G94GVCgoOUDBa9hamJ4DVOjSTCwUrtY7Qp0PQAA4PoguyDkGKUGs4O4mxAAAIhIdkHIGIliWgjDMCEAAEhkFoRERNQxBBNHAQCgjtyCkBGJHUPY/nIEIQAAEDVNENrt9rVr165fv97hcFxYeurUqezsbN8tp0+fXr58+e7duxu/KnWXRulAReMfGwAAmiO/30VQXFzcq1ev+Ph4t9tdXl6+efPmkJAQqWjXrl1DhgypqqpyOp0ej0fa+P33348ePbpfv347duwYOnTo+++/35i1YUQixRuY1SWWOyhU3ZjHBgCA5sjvPcJ33nmnU6dOGzZs+Omnn+Lj4+fMmeMtSkxM3LRp07Zt23z3nzJlyltvvbVkyZKtW7d+8cUXf/zxR6NXiRF1MLE/MEwIAABNEITffvvtqFGjiIgxdt999y1fvtxbFBIS0q5dO447V4cjR44cPXr0nnvuIaKIiIj+/fv77v/nMWIiiUSUFoqJowAAQNQEl0bz8/Pj4+Ol5YSEhPz8/MvsXFBQEBYWptVqpdX4+PiCgoJ6+7hcruXLl0dHR3u3MMakrCUij8fjvcp6EYx53G7m8bQPEveXix4PsrDRXKHlwW/Q8oHi75ZXKBQxMTG+r2uQucrKSo+PBv4Vx3GMXeEtC34PQqfTqVDUnUWpVF50voyXw+Hw7iztX11dXW8fj8ezefNm3ycbMca8zzh3Op2XOYUoik6n0+NwtDVwy3OYw4Gfj0Zz+ZYH/0HLB4q/W57n+bVr11ZUYGrfOVqt1uFwOBwOpVLZwD/RaDSBD8Lo6OjS0lJpuaSkxLcnd9Gdy8rKRFGU6l1SUuLtTXppNJo333zzUs8a9Xg8Op3uUsevYpxWq1HodOnR9MdvLp0Os2UazeVbHvwHLR8oTdDyOp0uLi7Or6dojgRBaNyW9/sYYUZGxo8//igt//jjjxkZGUR0qSd9p6amajSa33//nYgEQdi0aZO0f6NhRCIRUZSWiKi4tjGPDQAAzZHfe4RPP/30bbfdFhER4XK5Pvvssy1bthBR7969hw8fPmHChBdeeKG0tFQUxWnTphmNxmefffapp5569NFH//73v69bty4kJGTQoEGNWp2zSUjUMYTtt4j9tHhDLwCArPk9CNPT0zdu3PjFF19wHPfzzz936NCBiCZOnNimTRuO40JCQkJCQl555RUikl6r8dxzz6WkpPz8888dO3b84IMPfOeUNi7pQWv9YhCEAACy1hSv5UtPT09PT/fd4p3kOXXq1Ho7S1NAvTs0LqZQiC6ntNzRxPbiQWsAALInr2eNKmNTnHlHpGXcSggAACS3IFQnpzlP7JeWpZcxIQkBAGROXkGoSunkOF4XhKFq0ikovwZRCAAga/IKQmVkvGC3eSrLpNWMSG5DAYIQAEDW5BWExJg6uYPj7NXR4a3Yt7kIQgAAWZNZEErDhGevjt6RwP10Wqh2BbZGAAAQSLILQlVKmuP4Pmk5WEXdI9i6fCGwVQIAgACSXxDGtvZYSoSaKmn17kQOV0cBAORMdkFIHKdKbOc4Wfe+37sTudWnBCf6hAAAciW/ICRSpZy7mzBSS+2C2U9n0CkEAJApOQah2uduQpKujuagSwgAIFNyDEJVQqqrMFd01L2E6e5EtjxHENAnBACQJTkGIVMoVXGtnbmHpNXWQSxMw34vQRICAMiRHIOQiNTJaedfHWW4OgoAIE8yDULfuwmpbpgQPUIAADmSaRCqkzo4Tx0V3XUPlbnJzDwi4a1MAAAyJNMgZGqtMiLedeqId8tdrRg6hQAAMiTTICQidUoabqIAAAD5BqHq/CDMiGQFNvF4FTqFAADyIt8gVKd0cuYcJKGuF8gzGt+Om7kTnUIAAHmRbxByOiMfEu44ecC7ZVpnfnOh+GsROoUAADIi3yAkIn3G7daN//GuahX0fDr3v797kIQAAPIh7yDsNsiZd8RVmOvdMro15xFpyQlcIAUAkAtZByFTqvS9hlk3LvNu4Ri93Z2f8rtgcwewXgAA0HRkHYREZOg1rHbfb56KEu+WnpGsWwR75wA6hQAAsiD3IOR0Bl23gdbNK3w3vtaNe3Of54wtUJUCAICmI/cgJCJj3/+p+X2dUGv1bkkyskfacjOyPQGsFQAANA0EIfGmME2HbjW/rfbd+OyN/Ko8YTvezQQA0NIhCImIjANGWjctF11O75ZgFb3Xkx+10VPlCmC9AADA7xCERETKyARlXGtb9o++G+9J4gbFsazNuEAKANCSIQjrGPuPqN7wDYnnXQt98xb+WJX40SHMIAUAaLEQhHXUKZ14Y0jVuq/O28jTkv789B2e3WUYLAQAaJkQhOeYH51uy95o3fSt78bWQey9nvzIHz3VGCwEAGiJEITncIbg8ImvVG/61rbtB9/t9yZzvSPZ2F8wWAgA0AIhCM/Dm8LDxr5QuepT+x/bfbe/15PfXy6+vR+DhQAALQ2CsD5lVCtz1szyr95wnDj32l6dgr4fwr97QMDEGQCAFgZBeBGqhNTQB6eUz3/JmXPQuzFOz374C//iLuHLY8hCAICWA0F4cZp26SGjJpfO+5fvE2dSgtiGofzU7QLe0wQA0GIgCC9J075rxKQ3rT+vLP/yNe9DZ9oGs9WD+ae2eFbl4YYKAICWAEF4OYqwmIhJb4kuZ/E7kz2WYmnjDaFs+UDFYz+75x8RBKQhAEAzhyC8AqbWmh/6h+7G3sVvP127f6u08ZYItmaIYu4hoftK95ZihCEAQDOGIGwAxowD7g198O9V3y0o+WCa6/RJIrrJzH69UzHlBm7URs/IDZ5cK+IQAKBZQhA2lLpN58i/f6jvOqB0zrPlX77mqbYwohFJ3P5MRdtgunm5+6Xdgh333AMANDcIwqvBmK7rgMj/+5g3mIpmjav+canodukV9OLN/La7FLvKxA5L3f85iQmlAADNCYLwqnFaQ/BdYyImveU8+UfRrLG1e38loiQjW9qf//RW/oVdwm3fufeU40opAEDzgCC8RorwWPNjM0LunVS19qvityc5cw4RUd9oln234t5kbvD37tE/eXbhnRUAANc9BOGfom7TOfKZ9/Tdh5R9+nzFfz4QnQ6e0fj23OERyhtC2V3rPP2+c/83D3dZAABcv/wehIWFhUOGDDEYDCkpKStWrPAtcrvdjz/+eEhISHh4+PPPPy9t/O2337p27Wo2m+Pi4l566SV/V68RcJy++5DIf8wVPe6i2eMdx/cRUbCKptzAHb9XMaYd9/xOocNS95v7hPwa5CEAwHXH70H4t7/9LTY2try8/KOPPnrwwQeLi4u9RXPmzPntt9+OHz+enZ09b9687777johGjBgxevTosrKyn3/++e233167dq2/a9goOI0+ZOTfTJkTyxfOqlj2oeh2EZGSo/tTuO3DFXN7839UiDd96+69yv3+H0JhbaCrCwAAZ/k3CC0Wy/Lly6dPn65SqQYMGHDLLbcsWrTIW/rpp59Onjw5NDQ0ISEhKytr/vz5DoejsLBw6NChRJSUlNShQ4fc3Fy/1rBxaTp0jZzyb09FSfGbf/N9eUXvKDavN396lHJaZ35bsdhhqWvEBs/RSnQQAQACT+HXo588eVKj0SQmJkqrnTp1OnLkiLf06NGjaWlp3qKlS5eq1eonn3xyypQp48ePP3ToUFFR0d13333hYZ1Op8Ph8K4yxlQqlR8/xtXgDCbzozNs2Rsti97idEZjv0ztDb2IMSJScnR7PLs9nq918+8eEDL+6743mZvRhQ/XBLrSAAAy5t8gLC8vNxgM3tWgoKDDhw9Ly06n02q1Go1GadVoNJaVlRHRoEGDJk+e/Prrr+fm5g4bNiw4OLjeMaurq9PT0xlj3i2MsZKSEmnZarX67+NchbY369ukuw5tr9jwH8uKeaqew9Q39SHVucSbmEz3xrLZB/j23/BPpHomtnFr/ft/hd9dLy0vP2j5QEHLB0pNTY0oNvSKmk6n43n+8vv499c3LCysurrau1pRURERESEtq1Qqo9FYVVUlrVZWVkZERJw5cyYzM3P79u1paWlut7tPnz7vv//+5MmTfY9pNBo3btyYmpp6qZN6wzXwuvWnbv2dOQerf1xavX6Run267qY+mvZdmVJFREYjfdiHnqkSn93Bd16tmNCBn9ieC2vOvcPrqOVlBi0fKGj5gGCM+Xax/jz/jhEmJiY6nc7jx49Lq3v37m3btq23NDU1de/evb5FJ0+eVCqV0vVShULRtWvXgwcPXnjY5kWV2N786PSoGQs0qV2sv6w6889R5V+8Wrv3V8FuI6LWQWxxP/6XYYoyu9j2G9dff/IcqsDYIQBA0/FvEJpMpnvuueef//yn1WpduXLlzp0777//frvdnpmZeerUqaysrNdff/306dOHDh2aO3duVlZWhw4dOI776KOP3G73oUOHvv322549e/q1hk2G0xn13YeET3wl6h/zVIkdan5bfeafD5S8//fqDUtcBSdSgtg7PfiD9yhbGenWVe4713mW5QgOPLkUAMD//D4w9e67744bNy4pKSk6Ovqbb74xm802my0vL8/lco0ZM+bEiRNdu3ZVKpVTpkwZOHAgEa1cufLZZ5+dMWOGyWQaM2bMww8/7O8aNjHOYDL0usPQ6w7R6XAc32s/uKNs/oueylI+JII3hU02hU8KiThQQvl7ildWl7Wh0lB7iUKl1rRP17TvqkntwumDAv0JAABaFNbwIcfrRGpq6sqVKy81RlhdXd0cr9qLTofbUuSpKPVUlHgsJSSKvCmsXG1eWRU+rzCMr62+V9iZUbkjrnAfHxWvT2rPaXRMo+O0ek6jZ1p93apGz2l0TK0NyEdopi3fAqDlAwUtHyhWq7Vxxwib+VTFloKp1MrIBGVkgu9GPdHjRI8Tnaw2ZJcO/W/pX/aUuChnf/ucnNZKWytlaRSXa6Zancsq2m2CwybabYLdRh43bwrjTeF8SLjCFME05+UiU2s5tZSaOqbRcbrzvkyM47mgUMZf5FshOh2e6nJOrWVqnTTZBwCgZUAQNgNJRpZkZPckERFPdHOuNX13mbitTNxdRrvLRIdHHHETNzKJy4hijEh0OT0VJVLn0l1eLNjOm+EtWooFe11kCnabWFtzXqngEarKOX0QbwrnTWGczuiptngsxZ6KUtHt5I2hUtySKDKNjtMZ+WAzHxKhMIXzIeFutd4V04o3hXHaxvyXGgCAvyEIm59WBtbKwO5qVbd6rEpcfEKc+JunwkEjktk9Scpu4bHq8NhrPLooeqrKPZYST2WJYLPyQSG8KYI3hfmOTYpul2i3CbZqT2Wpp6LUXV7szD/mKD3jrCr3VJSIoqgICedN4VJYchodU+s4Tf1+JB8aqYxK4E3h9c/vcQvVFsFaJdhrvJkt2muE2hrBXlO36nL6/gmnNZy9OKxlap3CHKWMSVKYo4nDM+UB4MowRthy/FEhLjkhfJsjnraJA2O5v8SzwbFcRFONGHpbXrDbpC6pWFsj1FrrYsxuk56/WkcU3WWF7sJc0eVURLVSRsQKjlppfFSwVnJGE6cP5rQ6ptZzPldxpSu6nEbHlGrfUws2n7PYbe7S064zuZ7qckVEvDK6lcIcLU1E4k1hipBIplJTyyLn73xgoeUDBWOEcEkdTGxmF35mFyqoEb/PF1fkik9tcSUaWEYk6xnJMiJZgoFd+Sh/GqfRcVGtlFGtrrwrkVBT5TqT6y49zam1vCmMD43kjSGN0pMTnXZXUZ7rdI6nvMh5fJ+7otRTWeopL6rXm6yHKVWcRl83hqo1nIthjY5p9LwpTBnVShEZf9FhVABoptAjbMncAm0vFbcUib8UiVuKBJ5jvSLZbTHstmjWNriRQ7FltLzocgq1NaLDJthrxFrpeuzZy7MOm9tS7D6T4y4r5EOjlNGtOJ1RdNTW7eCwkcfDpIm7Gq10tZbYeYnO1Frv/F6mUNZd6a21CnYbCQJTac6FrkLptpR4Kko8laWe8mLBVl1XqtXXzQ0+m82cRmfnVEGJbfmgkEA1mmy1jO98c4QeIVwFBUc9IliPCDa5ExHxJ6rFnwvFjafFl3YJAlG/aJYRxVoHsWQjxeuZAgNqREyp4pUqosuFiuhxu4vzXWdyhFprXZdRrec0OuJ56fKsNAtJqK0hOu9fmaK91m2tkK7iii4Xp9Uxjb4uMjlOdNrdZWfqQtfjVpjCeVOYsm0XPiSC0xtFh/3sEGmNdHy3pUS05wp2m6uitKi0gIiU0YnKqARFZEK9DitTa7yVZFrdeUUczweHNaPBVN9/eYh2m3RVnEhsyH1Eossp2GvI45H+GRGQ+sP1CUEoI8lGlmxkD7UhIjpWJW48Lf5eIn59XDhRTUW1YpyeJRupbTBrZ2Jtg1lqMCUYWFNcS21uGK9QRicqoxMDXZE6Ur/EU21xn8l1Fea6i06JwnkPJToXHnabaLedV+RxC7YqRUS8MjpRGdVKGRVfb/yVqbRn71j18y2qgkew24Raq1hrE+w1QnWFu7LUYyn2WEo8FaVCTZVQWy3U1jCl+rwOcV3PmwS7TbCddx+R6LRzWj2nNRAx6d8QxHGcRk8cJ9htoqOW0xnOXQb3OeB5H5/juWCzNPmr3pSxhpBaXjq76DjvNaSix+2pLJfGxT2VpZ7qCsYrpLFwqSb1/zWjUNaNkWsNTKNjf+LfLoKj1vtlEM7/PhCRUGs9O9xeKzhsjOPPNZH6gu8AY3V3M1/QjJxWT83n9wOXRoGIyClQbrV4vJoOVYiHK8UjleKRSiq1i4lG1spQN081JYi6mFmbS1xTRcsHyp9sedFR6yo65Tpz0l2Y5yrKE93ueqXnJiJJ0eLzq8dU5z0knnEc0+g5rb7uR1Oh9C2VLi+fu3untkaUZgJLG90uTqPjtEam0XFaHWcw8aZwxdlZTpwhmNMapa5zgz+YKNTWCLXWuv6iRndetEil0gVwx7l/JQi1599u5PEIlWXuihJPeZGnsrTezUjnh5Oe8bzodNQdrdZ6Lra1eunm3fP+luf5oFA+JEKKWN5gEj2uumsJ9hrphuDzauJynr3rqUasvYp3L1yIU2vr/h9Uazmtvn6p1uCTfFpREETf+dtO+0Ub+VwDOs5+hNoauppKMpX67AxzPdPq6+VovX/M6UdPC+5w8zW3wIXQIwQiIhVHbYJZm2AaEnfuy1frppNWMbeacq1irlVcepKmbhOqXGLXMNY1nKWHsTbBLMnI9PgSNWdMrVUltFUltL3yrr6/eg6baLeJTsd55YLHO+rprrKI5/+US7+/itBI3/A4141QNfaLVxjjdIZ6j4xoaGnDVJWXGVS8N9RFwcOUqrNPfbrK2JY90fJ/tW0AACAASURBVGk//3Ypq+/AQt3lfU3dPylsQiOfHb9hcElaBXUwsQ4mIjqXjsW1tL1U3F4iLDgqHqsUTlSLwSpKNrJ4rTLF5InTswQDizdQnJ6ZW9p9CrLXGOHRkjClijMYOYMp0BVpCZhKw6s0FBTaoL0b+02QCEK4OhFauj2e3R5/7kWXp23iiSo6WOoscbP9FvH7fCHXSgU1os1NsXoWo6M4PYvWUSsDSzRQKyNLNLBgPKMNAK4bCEL4s2J0LEZHnfUeo/G8C0G1biqwiWdsdKpGPG2jY1XiDwVirpVyqkWeUayexekpWsfi9RStY4kGlhxEiQamvsKrpAEAGhmCEPxFq6DWQax1EPleWZVYHFRgE/Nr6IxNPFVDe8rE5TnCiWo6VSNGaFhKEHUMYTeZ2Y1mlhaCaAQA/0IQQgCEqClEzdJCqF5GekTKrxGPV9HecnFzofjuAeFoldgmiHUJY+lhLD2MdQ5lOnxnAaBR4UcFriM8k27VoH4xdQHp8NB+i5hdKmaXiguOCH9UiClG1jqIJRgo3sDidBRvYLE6itIxDTqOAHBNEIRwXVPzJPUFpVWXQPvKxRyreMpKeTXi9hLKrxEKaqioVtTwFK1jkVpKDWY3hbEbQ1kndB8BoAHwOwHNiZKjLmGsS9hFbuqvcNIZm3jGRn9UiNtLxLmHhIMVYisDi9OTScWCVRSkoiAli9RSvIEl6ClOz0JwgwcAIAihxTCpyKRi7U3nLqu6BDpYIRbVUoVTrHBQlYuqnOKuMvpvnpBnpfwa0SlQjI6ZNWRWk1nNzBoK0zCzmsI1ZNYws5oitCyyqd5jBQCBgiCEFkvJ0Q2hUihe4rFwLiq0iaUOKrNTmUOU/jfPSqV2KnMIZXYqtosVDorWsTg9xejqepB6Bal4IiKDQsrOuvgMVVOIGs/ZAWh+8F8tyJdRScZg1qZu7eJh6fDQGZtYYKOCGrHCSURkdZFLICKqdol7y6nETmV2ocxB5Q6yOES3QCFqMqmYQUkmFSk4ClIyNU8GJYWoKETNQtQUoqIwDQvTULiGhWsvcWIAaCoIQoDLUfOUaGSJRrpUUtbj8JDFSRUOscZNFge5Rap2iXYPVTvJ4qSiWvFQBVmcVGYXSuxUYhfLHRSuIbOaaRVkUJKSI5OKaXiS8lIKzmAVGRQsWEU6BekUZFIxNU+YBwTQWPAfE0BjUvMUpaUorW9qXi5B3QKV2MniFGvddX1Ni1O0e8jiIIuDcqzirjKqdFKNW6hwkM1NNg9VOkW7h2rddcEZpFDrlW6dgkxq0iuYiiPpEq5UyhEFq85VQMFRpJbi9SxKR/F6zKoFIEIQAgSWgqNoHUXrGhqcvqTgLLBYOY3e5qYKB9W4RYeHfC/hCkQW57nH+LsE2lVGp6xCYS3l14g8I5OK6RRkUJLU4wxSsiAVBasoRMWCVMQzOjsyylQcaRWk4cmoJAVHISqm4MiovETlAJoPBCFAc2VQEhEp9KLR6M3OqxtwrHaRxSHa3FTjpgon2dxilZOqXFTpJItTzLGSIJLFQURkdQsugWxucnioykUegSxO0S1QtYs0PGkVZFCwUDXF6ilCy2J0FKllYRoyq1m4hsI0ZNZgGhFcv/DdBJAvo5KMymvpjPqSrtNWu8RyB522UXGtWGCjY1XiluK6oVBpOq4g1t2UYlZTmIZFailSy2L0FKll0VoyqSlIyaRBUIAmhi8dAPwpGp40PIWoWYKBbjTTpdK01k1lDrHMQWV2KrWLRbVUbBd/KaTiWuG0jSqdVOkUbR6qdVOImjQ8aXkmDXOGqEnByKhkdV1PJSk50imYmiPp2izHKFhFKo70CqZTkPSU9iAl8WffhhKsYr4vRpEu+QJ4IQgBoCloFRSnYHF6ae2SQSQSVTio1iPaPVTtIrdAFge5BLK6xVo3eTfWuEWbm9wCHXGRR6QqJzkEsrmFGhc5BSKiSicJZ8dGK52i7yvNpSJpvFPNkU7BQtR1M3JD1XW3tURoKVJLZjXTKShETToF0ykoCAOiLRSCEACuI0x6OclFkrKRO3FSrNo9ZHOLFU6qcZPNTeUOsaSWSh3ijhIqtpPFIdjcZHGQzS3a3FTlIqOSdArSK5hJTSpSGdRug4Ipz87U9XZDpXwlqptYpOJIf0GI+nZhg1XE+Xw+nlHQ2UvWUg8Y/ApBCABypFWQtu737ypGSatdUmSKFgeVVTk4taraJbrPTinydkMdQt1k3VwreURyeMjmrn8o3y6sb/+ViDwiVblE7xndAhGdN32XzkYsESk5adIv0/Ek3UUjinUzh2vc5PQQUV3QSp1g7x010iVi6cJyXZvwTMMTY2RSnTuFR6Aq18VbQ7qfNUR17op0M4UgBABoKKNSumOEkZGq1YLRyBq9q3opZ6fvimcfbFQXkC6BrK66EVbpLhrGKCWIyOdxgBUOEs92gj3ieSEtiFTprDtFrUewe85tkU7Bc5e8JmyXzugUa1zkESlIdfHdLsqgYCFq6cERdU+NUPOk5MigYERkUhPvM/Sr4sk76zhYRSaRDFfVdleCIAQAaAakHmGI+lK5G+D5P9K9NA1X5RItDrI4yeIQLQ6qcpHTU/dACSI6UV039OsUqMYtOD1Uc7ZLXeGk+d1Z7+DGrDyCEAAA/izF2YHSBgpRs1Z13bqrjnCr1X61f3J5DRqEdblc69evLygoaNxzAwAABFyDgrCkpGTgwIE5OTl+rgwAAEBTa1AQhoWF6fX6yspKf9cGAACgiTUoCFUq1RNPPDF79mybzebvCgEAADSlhk6W0ev1hw8fTk5OHjBgQExMDMedS9BZs2b5p24AAAB+19Ag/OKLL5xOJxGtXr26XhGCEAAAmq+GBuHhw4f9Wg8AAICAuIr7CEVR/Omnn3bt2pWfnx8dHZ2WljZo0CCeb87P1QEAANlraBBWVFTceeedP//8MxFxHCcIAhF17tx51apVcXFxfqwgAACAPzX0qeaPP/54dnb2O++8U1hY6PF4SktL58+fX1BQMHr0aL/WDwAAwK8a1CO02+3/+c9/3n333bFjx0pbzGbzww8/HB4efscddxQUFMTGxvqzkgAAAP7SoB5hWVmZw+Ho2bNnve0ZGRlEdPr06cavFwAAQJNoUBCazWa1Wr1ly5Z623/77TciiomJafx6AQAANIkGXRrVaDSZmZlPP/20w+EYMWJEZGRkeXn5qlWrpkyZ0rdvX1wXBQCA5quhs0bff//9U6dOPfnkk08++STP8x6Ph4huuOGGzz///PJ/6Ha7P/roo19//TUlJWXSpElms9m3dNWqVUuXLjUYDBMmTOjYsaO0cffu3fPnzy8rK0tNTZ0yZYpGo7n6zwUAANAgDZ01GhISsmnTpg0bNsycOXP8+PHTp0//7rvvdu7cecV7J6ZNm7ZgwYLMzMzc3NxBgwaJougtWrFixSOPPNK/f/+YmJjevXtLr3lau3Ztv379wsPDb7/9dpvNhqebAgCAf4kNUFxcnJyc/MMPPzRkZ19VVVUGg2H//v2iKLrd7tjY2PXr13tLMzIyPvjgA2l55MiRM2bM8Hg8iYmJX3311WWO2bZt20OHDl3mjFdbSWgUaPlAQcsHClo+UKqrqxv3gA3qESqVyhMnTuh0uqtN2X379mk0GumaJ8/zvXr18s64EQRh27Ztffv2lVb79OmzdevWEydOFBQUdOzY8Z///OfLL7+cl5d3tWcEAAC4Kg0aIzSZTD179ly3bt2Fd1BcXmFhYVhYmHc1PDy8sLBQWi4vL3e5XN4hw/Dw8DNnzuTk5KhUqqysrDFjxhw6dOjGG2/cvXt3QkKC7zFtNtvYsWP1er13C2Ns8eLF0nJNTQ1j7KoqCY0CLR8oaPlAQcsHSk1NTcN31ul0vq9LuqiGTpb517/+9de//tXhcAwbNiwqKsq3KDk5+VJ/pdVqpXdWSOx2e0hIiLeIiLyldrtdp9OpVKqampq33npLukPx+PHjc+fOfeGFF3yPqVKp7r33Xt+pqowxb2/V4/FcQ88V/jy0fKCg5QMFLR8ogiA0vOWvmILU8CB88MEHi4qKZs2adeFLl0Sf+S/1xMXFFRYWOp1OlUpFRHl5eWlpaVKRXq83mUx5eXnSdJu8vLzY2Nj4+HgiSkpKkvZJTk729iDP1Vih6N+/f2pq6kXPyHFcQz42NDq0fKCg5QMFLR8ojd7yDQ3CuXPn2u32qz16WlpafHz80qVLR40alZub++uvv3788cc2m23FihXDhw/PzMxcuHBhz549HQ7HkiVLpk6dmpSU1LVr1w0bNowePdrtdv/0009ZWVlXe1IAAICGa1AQ1tbW/vHHH7fffru3P9dAjLG33npr9OjRX3zxxe7du5955plWrVqdOnVq1KhR+fn5zz33XN++ffv27VtSUhIfH3/PPfcQ0VtvvZWZmblkyZITJ04kJCQgCAEAwK8aFIQVFRXTpk3zzvC8Kn/5y18OHz68Z8+exMREaTQxJiYmJycnKiqK5/lDhw7t2LFDr9ffeOON0rBzRkbG0aNHd+7cGRER0b59+2s4IwAAQMM1KAgjIiLCwsLy8vJuueWWaziH2Wzu16+fd5Xn+VatWknLGo2mV69e9fY3Go19+vS5hhMBAABcrQaNN/I8/8ILL8yYMePEiRP+rhAAAEBTauhkmR9//LG4uLhdu3YdO3b0vTWQiH744Qc/VAwAAKApNDQIiahLly7+qwcAAEBANDQIlyxZ4td6AAAABMQVxginTp26evVq7+rChQtzcnK8q+vXr7/11lv9VDMAAIAmcIUg/Prrr/fs2SMtC4Lw0EMPbd261VtaXFz8888/+7F2AAAAfobnAwEAgKwhCAEAQNYQhAAAIGsIQgAAkLUr3z7x2Wef+c6ImT179sKFC6Xl06dP+6teAAAATeIKQRgZGZmfn797925pNTo6uqioqKioyLtDdHS0H2sHAADgZ1cIwm3btjVNPQAAAAICY4QAACBrCEIAAJA1BCEAAMgaghAAAGQNQQgAALKGIAQAAFlDEAIAgKwhCAEAQNYQhAAAIGsIQgAAkDUEIQAAyBqCEAAAZA1BCAAAsoYgBAAAWUMQAgCArCEIAQBA1hCEAAAgawhCAACQNQQhAADIGoIQAABkDUEIAACyhiAEAABZQxACAICsIQgBAEDWEIQAACBrCEIAAJA1BCEAAMgaghAAAGQNQQgAALKGIAQAAFlDEAIAgKwhCAEAQNYQhAAAIGtNEYTHjx9ftmzZH3/8cWFRWVnZihUrfvnlF0EQfLe7XK7s7Ozi4uImqB4AAMiZ34Pwk08+6dGjx9KlSwcOHPjKK6/4FmVnZ7dr1+7zzz+fOHHiXXfd5ZuFL7/8crdu3ZYsWeLv6gEAgMz5Nwjtdvu0adOWLl361Vdfbdy48YUXXigtLfWWzpgx46mnnlq6dOnvv/9+8ODBNWvWSNv37du3atWqjIwMv9YNAACA/B2Ev/zyi1qtvvXWW4mobdu2HTt2/P7776Uiu92+Zs2a+++/n4i0Wu1dd921fPlyInK73WPHjv3www+VSqVf6wYAAEBECr8ePT8/Pz4+3rsaHx+fn58vLZ8+fVoQBG9pfHz8Dz/8QESvv/56RkbGzTfffKljulyuZcuWRUdHe7cwxh588EFp2ePxeDyeRv8gcEVo+UBBywcKWj5QrqrlOY5jjF1+H/8GocvlUijOnUKpVDocDm8RY4zned+iw4cPf/bZZ9u3b7/MMT0ez2+//WY0Gr1bGGMjRoyQlp1Op/cU0JTQ8oGClg8UtHygOByOhl8y1Gg0AQ7CqKgo30HBkpKS/v37e4tEUSwrK4uIiJCKoqOjX3vttZiYmBdffJGIjh8/vnLlSrPZLF0+9dJoNK+//npqaupFz+jxeHQ6nb8+D1waWj5Q0PKBgpYPFEEQGrfl/TtG2K1bt5MnT0qXQ61W67Zt23r16kVEbrc7ODi4U6dOGzZskPbcsGFDr169HnjggXvvvTc5OTk5OVmj0ZjNZikmAQAA/MS/PcLIyMiHHnpoxIgR48aNW7RoUf/+/Tt06LBz58709HS73T516tTJkydXV1fv3bs3Nzd31KhRRqPxtttuk/528eLFGRkZ3h4kAACAP/g3CIno3//+9/z587du3Xr77bePHTuWiOLi4l577TWFQvHAAw+YzeZVq1aZzeatW7f6DvsR0ZgxY9q2bevv6gEAgMwxURQDXYerk5qaunLlykuNEVZXV9cLVGgaaPlAQcsHClo+UKxWq8FgaMQD4lmjAAAgawhCAACQNQQhAADIGoIQAABkDUEIAACyhiAEAABZQxACAICsIQgBAEDWEIQAACBrCEIAAJA1BCEAAMgaghAAAGQNQQgAALKGIAQAAFlDEAIAgKwhCAEAQNYQhAAAIGsIQgAAkDUEIQAAyBqCEAAAZA1BCAAAsoYgBAAAWUMQAgCArCEIAQBA1hCEAAAgawhCAACQNQQhAADIGoIQAABkDUEIAACyhiAEAABZQxACAICsIQgBAEDWEIQAACBrCEIAAJA1BCEAAMgaghAAAGQNQQgAALKGIAQAAFlDEAIAgKwhCAEAQNYQhAAAIGsIQgAAkDUEIQAAyBqCEAAAZA1BCAAAsoYgBAAAWUMQAgCArDVFEP7rX/8KCwszmUzjx493u92+Rd9++21iYqLBYBg0aNCZM2eI6J133klLS1OpVHFxcc8//7woik1QQwAAkC2/B+F33303b968nTt35uTk7Nix44MPPvAWFRcX//Wvf/3kk08sFktiYuKTTz5JRKIozp8/32q1rl69es6cOV988YW/awgAAHLm9yCcP39+VlZWQkKCyWSaPHnyp59+6i1atGhRt27d+vfvr1Qqp0+fvnLlyrKyskmTJnXt2lWlUt1www2DBw/euXOnv2sIAABy5vcgPHLkSFpamrSclpZ29OhRb9HRo0c7deokLcfHx+t0upMnT3pLbTbbpk2bevbseeEx7XZ7rQ+73e7PTwAAAC2Zwt8nsFgsBoNBWjYajTabrba2VqvVSkWJiYnePYOCgsrLy6VlURTHjx/frl27e+65p94Bq6uru3XrxnHnIpwxVlRUJC1brVa/fRS4HLR8oKDlAwUtHyg1NTUNnz6i0+l4nr/8Pn4PQrPZXFVVJS1XVlYaDAYpBesVEVFFRUV4eDgRiaL45JNPHj9+fO3atYyxegc0Go0bN25MTU291BmNRmMjfwZoGLR8oKDlAwUtHxCMMW//qlH4/dJou3bt9u7dKy3v2bOnXbt2Fy06efKk0+lMSkoiov/7v//77bffVq1a1bgfFQAA4EJ+D8KsrKx58+YdOHCgoKDg9ddfz8rKIqJXX311wYIF99133+7du5ctW1ZVVTV9+vTMzEyTyfTcc88tWLDgtddeO3HiRHZ2tu+oIQAAQKPz+6XRAQMGTJs2bdiwYU6nc/To0WPGjCEii8Wi1WpDQ0P/85//TJkyZfz48X379v3www+J6MiRI3FxcVOnTpX+fPDgwS+99JK/KwkAALLFmt0d66mpqStXrrzUGGF1dTWu2gcEWj5Q0PKBgpYPFKvV2szGCAEAAK5nCEIAAJA1BCEAAMgaghAAAGQNQQgAALKGIAQAAFlDEAIAgKwhCAEAQNYQhAAAIGsIQgAAkDUEIQAAyBqCEAAAZA1BCAAAsoYgBAAAWUMQAgCArCEIAQBA1hCEAAAgawhCAACQNQQhAADIGoIQAABkDUEIAACyhiAEAABZQxACAICsIQgBAEDWEIQAACBrCEIAAJA1BCEAAMgaghAAAGQNQQgAALKGIAQAAFlDEAIAgKwhCAEAQNYQhAAAIGsIQgAAkDUEIQAAyBqCEAAAZA1BCAAAsoYgBAAAWUMQAgCArCEIAQBA1hCEAAAgawhCAACQNQQhAADIGoIQAABkDUEIAACyhiAEAABZQxACAICs+T0IPR7PnDlzRo0aNXXq1KKionqla9asefjhh8eNG7dr1y7v/h9++OGl9r+i1atXO53ORqg3XKXly5cHugpyZLfb16xZE+hayFFlZeXGjRsDXQs5Ki4u/vXXXxv3mH4PwpkzZ86ZMyczM9Nisdx2221ut9tbtHbt2lGjRt12222pqal9+/Y9fvw4Ec2YMePjjz+W9u/Xr5/H47mq002YMKG6urqRPwNcicfjefjhhwNdCzmyWCxPPvlkoGshRzk5Oc8++2ygayFHe/bsmT17duMe079BWFtb++9//3vu3LmZmZkfffSRy+X67rvvvKVvvPHGc88999BDD02ePDkzM/ODDz6w2WwffPDBvHnzpP0dDsfq1av9WkMAAJA5/wbhkSNHnE7nzTffTESMsVtvvXXr1q3e0t9//71Pnz7Scp8+fX7//fcjR4643e4uXbpcdH8AAIBGp/Dr0QsLC0NDQxlj0mpYWNiZM2ek5dra2qqqKrPZ7FtUWFhoNpsvur+XVqvNyMjguHMRzhhLT0/3Lt9///0KhX8/F9QjiqLRaBw6dGigKyI7LpfL7Xaj5ZteTU1NaWkpWr7pVVZWnjp1quEt//777ycnJ19+H/8GhlardTgc3lW73a7X66VllUrF87y31G6363S6y+zvNX/+/AvnZXTt2lVamDhxYuN+BGigxx9/PNBVAACoLyws7Ir7+DcIY2Njy8vLrVarwWAgory8PG9i8TwfFRWVl5eXmpoqFcXFxcXGxpaVldXU1Ej5l5eX171793rHvOmmm2666Sa/VhsAAOTDv2OEKSkpaWlpX331FRGdPn16w4YNmZmZTqfzyy+/rK6uzszMXLhwIRG5XK5FixZlZma2bt26Y8eOixYtIqKCgoIff/wxMzPTrzUEAACZY6Io+vUEP/3004gRI9LT0/ft2/fggw/Onj27rKwsLCzs8OHDBoOhb9++4eHhVVVV4eHhq1ev1mg0GzduHDlyZHp6+t69ex966KFXXnnFr9UDAACZ83sQElFFRcXu3bsTEhKkEUtBEAoKCqKjoxUKhcvl2r59u0ajuemmm7xzZOrt30But3vevHm7du1KTk5+4oknLhxchEZUWVm5cuXKHTt2MMYGDhx4++23E9HXX3+dm5sr7RAaGjpmzJiA1rFl2r1799q1a72rjzzySEREhMfj+eSTT7Kzs5OSkp544glpJAIa1zfffHPixAnvqtlszsrK+vDDD6uqqqQtycnJI0aMCFDtWhpBEPbv35+dnV1cXDx27NiQkBBp++rVq1etWhUUFDRhwoRWrVqRzy9/SkrKE088odPpruF0TfGINZPJ1LdvX2+qcRwXHx8vTexUKpU9e/bs0qWLNwUv3L+BJk2aNH/+/IyMjF9++eV//ud/GrH+cKFPP/10yZIlSUlJrVq1GjNmzEsvvUREc+fO3bp1q8VisVgs3l8HaFy///77ggULLGdJT5yYPHny3LlzMzIytmzZMnz48EDXsWWyWq3eZl+wYMH69euJaPbs2YcOHZI2Wq3WQNex5SguLr7jjjtWrFgxbdq0srIyaePixYsfffTRLl26uFyuHj16VFRUENHf/va3BQsWZGRkbN68+dqH0sQWoaSkRKPRnDhxQhTF2tra4ODg7OzsQFeqJXO5XN7lRYsWpaSkiKLYr1+/b775JnCVkoU5c+aMGDHCd0tZWZlWqz169Kgoina73WQybdu2LUC1kwWHwxEeHr5u3TpRFFu1arVz585A16jFkp6XKX23RVG86aabPvvsM2m5f//+b7/9dnFxsUajOXnypCiKNpvNaDTu2rXrGk7UQh66nZ2dHR0dnZSUREQajaZHjx6N/jA68OV7p2ZFRYX3wsWKFStmzJixaNEi32fpQeM6duzYs88++/7770t32e7atSs8PLx169ZEpFarMzIy8OX3qxUrVqjV6n79+kmr8+fPnzlz5qpVq0T/DzPJWW1t7a5du7zN3q9fv19//TU7Ozs2NjYxMZGItFrtNf/yt5AgLCws9L1ZJDw8/MI78cEfCgsLZ86c+dxzzxFRly5dEhMTFQrF7Nmz+/bt63K5Al27FigiIqJnz55Go3Hz5s3t27fft2/fmTNn8OVvSp9++uljjz3G8zwR9e3bNywsTBCEJ5988oEHHgh01Voy6Vvt/aqHh4efPn263i9/RETEtX35W8gTWNRqte/PrtPp1Gg0AayPTJSXlw8ZMmTcuHF33XUXEb322mvS9meeeSY1NfW///0vBmsb3d1333333XdLy+PGjXvppZcyMzPrffm1Wm2Aatfy5efnr1+//t///re0umDBAmlh/PjxKSkpe/bs6dy5c8Aq16JJP+kul0takL7n9X75HQ7Htf3yt5AeYWxsbEFBgffSRH5+fmxsbGCr1OJVVFQMGjRo4MCB//rXv+oV6fX69u3b5+XlBaRi8nHzzTefOnUqNjb29OnTgiBIG/Pz82NiYgJbsRZs/vz5t91224VT+WJiYmJiYvCd95+IiAiFQpGfny+tnjp1KiYmprF++VtIEHbv3p3neWke14kTJ3bu3ClN6Ac/qaqqGjJkSEZGhrcX6HK5bDabtHzy5Mnt27ffeOONgatgi1VZWSktuFyuZcuWde7cuVu3bhqNRrqnIicnZ/v27XfccUdA69hiiaL42WefPfroo9KqzWbzdke2bt2an5+flpYWuNq1cAqF4o477pAez2K325cvX3733XdLjx7bsGEDER07dmz37t3X+MvfCDN7rg9ffPFFWFjYyJEjY2JiXnjhhUBXp4V7/vnnOY7r0qVLenp6enr6LbfcUlBQEBwcPHDgwKFDhwYFBU2aNCnQdWyZevfufcsttwwfPjwpKalLly5FRUWiKH711VfSlz82NnbmzJmBrmOLtX79epPJZLPZpNXNmzdHREQMHTp04MCBBoPhzTffDGz1Wph+/fpJbyJKS0tLT0+vqanZu3dvZGTkXXfd1alTp6FDh7rdblEUFy5c6P3lf/HFF6/tXE1xQ32TTv1LFgAABdFJREFUyc3N3bt3b5s2bdq1axfourRw0o1T3lXGWFJSUkFBwf79+wVB6NixY0JCQgCr14LV1NTs2rWrvLw8Pj6+c+fO3tew5OXl7dmzp3Xr1u3btw9sDVuwioqK2tra6Oho75bjx48fPnxYpVJ16tQpMjIygHVreXJzc31fzJ6YmMhxXGVl5S+//BIaGnrLLbd4v/x//pe/RQUhAADA1WohY4QAAADXBkEIAACyhiAEAABZQxACAICsIQgBAEDWEIQAsrBkyZJNmzYFuhYA1yPcPgEQMB999NHs2bMv3L5///5re7/oZbRr165nz56ffvpp4x4WoAVoIQ/dBmiOKioqTp48+fTTT4eGhvpuVyqVgaoSgAwhCAECbOLEidLbBC9FehO3yWSqt93lcpWVlQUFBV20+1heXi6Kotlsrrfd7XZbLJbw8PB62ysrK10ul9lsZoxd9WcAaM4wRghwPVqxYkVoaOjatWu7d+8eEhISGho6YMCAgoICqdThcEyaNCk0NDQ6Ojo4OHjYsGHep/IT0ccff5yUlGQ2m8PCwsxm8xtvvOEtevvtt8PDwyMiIkJDQ9955x1p4/z58+Pi4kwmU3h4uF6vf/jhh5vwgwIEHoIQ4HrkdDotFstf//rX4cOH79+//+uvv969e/cdd9zhdruJaOzYse+///4//vGP3bt3L1y4cMuWLQMGDKitrSWiV199ddy4cb169dq0adO+ffveffdd7zG///77pUuXfv3111u2bOnTp8/kyZMPHDiwc+fOrKys+++/f9++fQcPHly6dOmF7xgCaOEa7VHhAHCVZs2adeF/kl26dBFFccmSJUQ0YcIE787SlpUrV544cYIx9vTTT3uLVqxYQUSffPJJRUWFXq+/8847LzxXampqeHh4ZWWltGqxWFQq1csvvzx37lzGmPeNCgAyhDFCgAB78803fV9c4DtxJjMz07s8fPhwnud37twpCIIoivfee6+3aNiwYTqdbtOmTTExMTU1NY899thFT9SzZ8+goCBp2WQyRUdHnzp1asCAAUR0++23Z2VlDR48+MIxRYAWD0EIEGDDhg271GSZqKgo77JSqQwLC8vPzw8JCSEi39fQM8aio6PLyspKS0uJKC4u7qJHqzc3Va1WO53Orl27fvnll7NmzXrggQd4nu/Ro8esWbMyMjL+/OcCaC4wRghw/ZKCTeLxeMrLy6Ojo/V6PRGVlJT47llSUhIcHCzNLC0qKrqqs9x///179uwpKCj45JNPysvLhwwZUlxc3BjVB2geEIQA1681a9Z4lzds2OByudLS0rp3705E3333nbdo06ZNVVVVPXr06N69u0qlkkYTr1ZMTMxDDz303nvvWa3W/fv3//nKAzQXuDQKEGCrV6+u93LzwYMHSwvz5s3r3LnzsGHD9u/fP3HixKSkpDvvvFOj0QwdOvTVV19t06bNX/7ylwMHDjz22GPR0dEPPvigyWR66qmn3njjjbi4uKysrLCwsL179x47dmz06NGXOvvChQsrKyuHDh2akJBQVlb22WefaTSajh07+vczA1xPEIQAAfbUU0/V27Jr1y5p4a233vrf//3f+++/n4jatm27YsUKjUZDRF9++eUjjzwyatQoURSJ6IYbbvj222+l66KvvPKKQqF44403XnzxRSJSKBRTpky5zNkFQZg5c+bf/vY3aTUhIWHx4sX1ghmgZcOzRgECRhRFQRAu3M7z/DfffDNy5MijR48mJiYeOHCAMZaWlsZx541lFBYW5uXlhYSEtGnTpt4RbDbbwYMHeZ5PTEy88JE09QiCkJOTU1paajabW7VqpVDg38cgL/jGAwQMY4zn+cvvo1AoOnfufNGiqKgo32mlvnQ6XXp6egOrwXFccnIy7qMH2cJkGQAAkDUEIcD1KD09/aOPPoqIiAh0RQBaPowRAgCArKFHCAAAsoYgBAAAWUMQAgCArCEIAQBA1hCEAAAgawhCAACQtf8H81tlKu8o1MwAAAAASUVORK5CYII="
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# this is the error plot, we get to about 7.5% test error, i.e. 92.5% accuracy\n",
    "trnerr,tsterr = Array{Float32}(lin[4,:]), Array{Float32}(lin[5,:]) \n",
    "plot([trnerr,tsterr],ylim=(.0,.12),labels=[\"trnerr\" \"tsterr\"],xlabel=\"Epochs\",ylabel=\"Error\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Visualizing the learned weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tbody><tr><td style='text-align:center;vertical-align:middle; margin: 0.5em;border:1px #90999f solid;border-collapse:collapse'><img style='max-width: 100px; max-height:100px;display:inline' src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAHAAAABwCAAAAADji6uXAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAAmJLR0QA/4ePzL8AAAVcSURBVGje7dpJjFxXFQbgr9rVg9N2TDxAEjuJHY/EBDIolmCBIRLGShAskijZsIjCwgixZRfJCsmCNUgIiQ0rlA2KBJESMVhgQIwiSkISD7QZYtIekIf20GM1i/9eXpl9VUvlOlL1e/Xefa9en//95/zn3Nt6HmNYwAjaYktd21bZX4X5MmYJ45gr13XK8REslrHKvq4xI/psg/+D7dGy08GoYLJcnqR+Ohr8WmV/rIxrCU4jZX/p/7a6xtX7DbZL+4/hkmAwquFitQ4uCXbj5fwmPIzL+Bj+heP4Txkzo+Fyxawb18F3af8xXCU8qTaB68K1K4LfGfwD0ziIaziNc9iAN/AePopZDVZrynaDvA+V34Pt0v5juCgcqTGyg3XlSeaxRXB5EKfK+YM4JlhexK24G5/H3wX7U+XcMcF2q8TfwXdp/zFsa+LcknDnHombH+BlwfI24doHWC/4fQk/xLv4NN7Eh/EYzuOf2CyxdhGrbwqX9v0HW4fFvx2Njy8Krn8QXi3gAezF0a6nnBHOHsOz+D0e12jVdfgb3sd2ybmD79KViaUj4vfq43eEO3cJhvuwFr8Vjv1CYusPsKNcdwQH8BHBdAZ/xeu4XzC8OVy6Mpqmu/67Wp5iXPTlFvwJG0XDzEg+nMdXJdYexo8kfh4RTk/iZ+W+y+Vew/qwJ9bulJ1x0aNn8DZuwR3l+AbJc/dLDbEOt+Mr2Ik/lyf/TTm/VWIo7JIYvaTpAQy2S/uPYdWki1LzXS9PUXXILtGT20SftETL7C03eERi6R8l9k6JLpoT7JXr6rsy+C5dGV1a+2SrJWbeJ7qG1IFzoj1PC8/WdN3gDF7DF/FzPCrvwW7BviVcHvZpembt2gcj2KyTXPdMOfZviaM/FQyfxwmpPxZF8zwsefR8GXuvpve6IBgOa4ueWXtBtMyy+LjmvxHpyzwkPZqpckHtqxJOTko98QXpu90muqfmv+Wyv95Q0/TI2t3xrfp8RDToNPZIrV5tU/fFwuFvSS1JcuqoYDYpvK7ad9gv7Ym16xxRS9OvPq3psSzgQhnznNQaj5Tv38fX8VTXDU9KLXlNsBuVGF3j9eC7dGXmLTpSSyxKb3RWeiuXBb/PCR9n8V3RpNvwYzfyktT4o2XsXPl+RZMLB9+lK6NL5zXzDQuS+06IRt0qdcYn8Sq+rMHts3ih7G8RLVq17UbROWPyPuw0xLBH1m7JfFOnfK5KHUE4uEd49LronSnpzWzHK6J3nsR+0bA7pS9+WvLhrdKrmzHEsEfW7kgMHRM+nhXM7sAhwWi/4Hwcv8Mn8B18DZ/Ct0XL7hP+1vngO6UGuSo14/xN4dL+Y1j/LJbtxnLiqfI0T0rNcFD63ZWTu2Wu6YDomo9LzDxZzp2TPsH6su0Y6tIe2f/qwzrfu090y2W8JfXFWXxG+DYrOfMJ4dv38CHJkX+RPs00XpT3YYdw8pKmVzPYLu0/hmNdX85Lj4XksSnJiWvxDfxSOLVW8Ngu/DpQjt0tXP61praYduPaqMF36crNW0xI/PuJ9EsXxP/npZbYKP23Q6JpfiVrL+r840ua+YrjUqtcEJ06Ie/EtZvCpf2fx/+m4NUSH1+SfllH1sS8J3P405IXxyTO3iK4HsXTMie8WMYtl8+kcHGi/NhwPU1PrN0R31ZtOiZzETPCozuFV7WXdo/omvc1Ncgboj0nhIuT5T+pWLY0a1cH36UrUx8uC+8mNGsKN4t+mRA9WddSLGjW0awp50c1a0jrmvG6Rq7mwnnDfNgja9eagoYn42U7IfxsyfrCavOCW10zWucm6jHlHsvl+rp+ddVK/IeD/4P/BTDkYd8NLUYhAAAAAElFTkSuQmCC\"></td><td style='text-align:center;vertical-align:middle; margin: 0.5em;border:1px #90999f solid;border-collapse:collapse'><img style='max-width: 100px; max-height:100px;display:inline' src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAHAAAABwCAAAAADji6uXAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAAmJLR0QA/4ePzL8AAAWASURBVGje7drbj9xjHAbwz0xnt7R0t1VnLUXROgt1iIrzOakLpcEFFxKJC9y4EIREJCL+ABFESBwqcT7HWYlzqQiqaS3WWUvZtnt08by/TElc7kwyu28ymZ133ndm9vv8nuf7fb7vr3YDapiCYTTQhU1lfqzMjZVHvawdwLSyZ7jMVY8RbCn76hjC1DJf1+LR+V/YqKNb4lxhNVTmhiXuA9Xi8jyI7bADZmI1/izvVzhui1FN3Eflmuj8kLYewynCuVFsIzGvCY6Ea7thI3YSjv5S1u1cfnFXef095gjmNeHerPLZ601iOE6jMSixHhN+1TW1cUfh2UXYLBj3C6Zf4DPsWfb2l+dPMFuuiynl87YT/CYxHJfR6BLMRsvEBuHXcZp8GsHXZd32WI6TyvynZd/+eB+9eBPfCdabcEjZO3VChLTlX1i7UbOmqTjZW+YIzz4uc8eiT3AZEAxPwOeileswX3T3K+HkcNnXJTm280Paeh5Wf2zGdCwQLlY15SosxK64C7uU+Rr2FpzeFZ39VbDrFkzfxDxNHa5PiJC2HsOKgz2C25FYibl4AEvwJH7GWXgZV+MJPC2auxyX4i/Ji2cK957DWqmPRqRW7fyQtl5LK3/YLTp6JR6V/DdbOPQdluFLvCB1znzR0zU4GPfgfrwj18Du4jfuwLk4yGRNM06jUReOrJcaZAXOxuVST9ZxonDsJdHVLtHdw7CP1KdX42G5Fvqwl2CozG3Trv+w87+wMSaYDJTnp3Co6OMtEvsXpLZcXNY8hHNwp2B4jmjsN8K//cR7bN0LmCa1UOeHtPVaerPwcFh48xtOEa+xXLj3f2NbyZfP4zLh7q04Qri4BjMkP34vutz5IW2fljbEG/yIB/+zaDr+3ur1XOmH/iS5cg+8glPFW/4pNdAHuLvMrxI+d35IW4/hiPRXSE3zwX8WHCz4HojT8F55XiN1zGzR3rW4T3h8pniUBVLLEm6uacd/2Plf2Bguf1R1Rw/+EG9wjfiDv8ULrhM9bJTHIsl3K3CV9AKOxVvSpztJeqs7ynWydkKEtPUYdkvtsVG83lLBYUhqlnmCQb/kNppnSQvxe3n/8bL/XhyNZ0SXLyh7aqLJnR/S1mM4JPVMFy4pv2C1eMAZeBbXCt+6Bd998XZ573ipX28XXOeIJ1wmfmTrMenxx2U0qrOjWZLLejV5ubPktiHxFRukXu2XvLlUdPMM8RcbJTfeL17iHZws+P9e9nd+SFvvLa4TDIdFGy+U3tsb0s++XWJ/vnDvCrwmeL8o/n9hWT9HvMiR4vEXCzeJxv4xIULann7piPDrVcGjT+rPfcrrPUVHZ4lnXCY+/zLxGFPFxz+O0+Ws+FSpdXqlpp0rvOz8kLaehzdJvdEjOJ0ruNSlFl0hGP0ltcobgvFK4eA6vC5aeoxcC1OFcwPC5a/FP66fECFtzzn+NDnbnS7+oPolL8uZxfXSO/tN8DqjrH9MOLlE+miH4jbh3SHCvX7BeUDyaueHtD3n+NXZUJecD/6Io8r8WZIrKw+yv/S4fxKP+LlgXfn/RWXdZnwk/O2Ts8WeCRHS9uTDMemZrZa+2VrBZoHUmj2Cx3nC1QdwuGjuYNk/T2rYDZI364Jzn1wb1bXS+SFtDw8HhVM/CI6flTdn4gDRxPVyrrhYNPNiPCJ9mCWC0V3C10XiOXqFz7OlfzqJ4biM2nXivYcEy02igzW5r+J94eJM0dSDRG+PkPPGbyXXfSgcnCE9u+6ybtfy+Vuk3un8kLYew1sEO+ILxwSruiaWW4RnVU4bkNpzs2A+KBo7U7MPV92TyL/vD+78kLan10bz/m6iq9W9hFOEU6OSKxuCV3UzVbfg2NCsi6oaqbrfuPIbk2fA4zL+AZN0aHTEFN8DAAAAAElFTkSuQmCC\"></td><td style='text-align:center;vertical-align:middle; margin: 0.5em;border:1px #90999f solid;border-collapse:collapse'><img style='max-width: 100px; max-height:100px;display:inline' src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAHAAAABwCAAAAADji6uXAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAAmJLR0QA/4ePzL8AAAWTSURBVGje7dpdjF1VGQbg53TOzLTFUirTAlVoilhpCylUg0HFNDX8hEjUBExMiCYSvIHwk2BCGn5sYuKNMeGKEGKMtVEbMBp/alLRQqyARVtqWqwtyCA0TaHt8CPT6enMHC/etd3TG+7OmeT0rGRyztp7rbXPfO9+v/f7vrUaD2IAbTTL5yTmYBrzMDFjzEkMlj40yriJGfMHMVXWmCzXlf4cXW69/8BmQ+zdFHwGyq9o41TpVzi1S3+w3GuXRaYxv8xvolU+JzE0Y/6ZYdLZwbB66rDYv+rPE1za5ftcwW9M8Hm7jB3GUXyojKkwG1BzuiG4975Ju48hse08tf/7uGB3dvl7uVy7tkzaiatwDG+URebiTXwE3y/Xz8GIYDtH35d2qDVb4u9aYuv5WFt+yUmsxCW4uIx5R/DbiQW4EFtxDV7Bdrwv2P8VX1Rr7EzO965Ju4/hkDqeOQ+rBLcTGMVPBddd4h9b4jv/jWVYhz/iiHD1eFl4CItwGB9V+9zeN+ns6CHRrgXiH38rPnG74PoJLMSz+CfuxquijQewBF/BHbhB3oEDWCHcPlHmz9Ta3jVp1x/YeACL8br4y5uEdzskdpkv/NuH/+I6/BIbsB9/Fu5uK/fewg/wmfJ9St6B8wXz3jdp93nYFFsPi14twNPYLb5wW7n/DRyU+PMqwa4l+BzBt/EXiY1uwzdFK5Uxk+Kze9+k3cewihXPEj1bKFp2GZYKN9/GHvwMVwje9+NSvCbxze/wacF9r8SkXxCtHBa9rXLP3jZp9zGs+NES/fqb2H2Z4LYHvxF+NUQrf4i/C1/XiuYtlnhmGi9hucSqL5e5bXXu0tsm7T6GM2tsA+X7HskLdpT+oOBwo8SYQ+In2/gJNmKLcHi15IYviYYudnru2fsm7X5Ms1FqnQ3B5Q38SZ07HMbn8TX8uHw/iofxI8HnefGje0VPN5TPi4Xba/Tzww62ZvXUJt4TXzqEc4U7nxR+blLXU7+OF/EEbpFazIi8C5vKwu+V/qKyzrS67tbbJp2d/HBQXb8+F0+pMV0p/DwmcelS/FzyxuskvxgR3u6SmOdbZcwa8b3j5Rn93KIjrbFRXaduib3HhHP7JW+8RPi3SvK8adG9U6J5x0X/viy1tf3CvyX4D34hutjP8TvSmm3h4GKJL9fjGfy6XH8dt4ouLpCay52ijTsl7rmp9MfxAv6Fz0lsuhyPS7zT18OOtGa1fzuFz4ovfVd85nLh0/fK4LniW3fjD+JD75J49RF8V2o0a8t6h8u1UfHHfV/akdYcFL9JcDuIT+ECyRtW4B/l/ojg15KY8xHBa6vUYraKj71eOPuk1FhXCQdn5T/s/Qc2K43aWy6clBrN0xLXNEXnhiVmuRUXlf4LwrH1UuOeEP17TfTxHIlxhkRP+3WajrT/7z1VT75M9ibukJrMKeHWo4LJZtwj3HpOdG631Omek7zxV1IfP1/wO1vehb4edqQ1q/rJCuHPBB7DV4VrbdyMq0Ub7y0Td+By0c/nBaf1uE/0dI3kj0eFi1Xc2/sm7T6G1ZmXQdl/OCZadkjOVowKnkfxJcn7TpZfel4Zd6PUdo6IFj4reeEh9b7xXOFi75t0duLS6tzhEtlXGlH70M24Uo3zAXxH8oVdsjd1C24X3o0Jl7cIb5c6/SxU75u0+zn+Q+q9p6slv9gjvrCN30sOv670d4kPnSN55OXC182ST1wo78KEaOKUOqY5M0w6O/v4Vcx4XLizWvbhL/iAifvkPMZKiYEG8DHJG18RnRwXTa32t/q+tCOtWdVKW/iw8Ga/1D+bgsMJdYw5XsZNquvYLYljRkt/RK2ZU2pfPX1GmHR2at7V+fqWxI9TomEtwWRAzmNMSmwypj73ParGaKrcrzS2WnugXOvnFh1p/wMXzGpYDtMgJgAAAABJRU5ErkJg\"></td><td style='text-align:center;vertical-align:middle; margin: 0.5em;border:1px #90999f solid;border-collapse:collapse'><img style='max-width: 100px; max-height:100px;display:inline' src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAHAAAABwCAAAAADji6uXAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAAmJLR0QA/4ePzL8AAAWHSURBVGje7dpJbFxVGgXgr5zyECcmJGkgQMIUM4QxgoZmbAgISItBiAW0GDYICVZsQIINQsAGsUJhQbMCCVoIIboRLKARAoluEmZIhzATSOJ0MChjJzhlPLA496m8YOsqqewrPfmN95X/8875/3Pvrd2PXvwqbQ5Gy7lxjKEL3Zgs9/yKnnJ9HPVyX9XmlnsmS38T5Xyt9NXS1vkvrNcl1t1olF9Qxw78LHFfJJjtwQ84CofgW8HwKCzAL+XZRjnfW44nSj89MyKkrcewhgMSYxLvin+7BI99+FLw2IQfsRzby7UG5qFPOFgr2wTma/JxckaEtPUYjkvcD2hqZU1wqeEbbBEs9pXrG8pWtTEM4CwsFDznlOf3C/8mzWI4Ta1e09Q95e+IYHokfhJMhnFo2U7HV+XcWeXcTlyD8zTz4zC24T3sNqul09TqE8KZqiaZj4OEcyM4G0uFj/04Dk/gdk3sx3CO1DHPCf7n4gKsFe2dJ9h2fkjbw8Oq7cIglgku+7AXBws+H+Hrsv+Y8GwxzsDHuEn4NoJjSp+jotUHa9ZMLW2d/8I6zfp/QHBbhu/Lr2ng/9iMlXgSJwpOK0UrTxM+zhW9XFD6+RFHS17sm9JnZ4e0PVq6X7RuXPLasHCtH5/hWuHbIC4T/HZK/bJRatU/Ck+XCeeendLn6XjJrJZOU6t3a3rwDaKhY1LL9IsmfihecbfgcireEe19tOwvwQniOao8ugn/LOfmi6fs/JC2HsMx0dJtwpka/i1aeK74gfOFQ324C3eXh2/G07hOeHkhXhSN7ca9UgP1Cz+7Z0RIW4/hpOS8/0ke24QrcbzgOojVuAFrcCk+LQ/fKjh+i1dFVzdLHhwo92wSj7Ko/HedH9LWY1j5im4MSd15ueTDOwXfl3Ej7igPrRE+/gcr8JrUPScKrpfiTdHekyTfNkqfnR/Slr+wdr/E92fB8BhcLTXKEP4gWrhkykODuB4PCP8WSr7bKPj3SF30N3widdBou/7Dzn9hvVdy4GIZb3mkXFgneP63HH8gnBoo9x4h/PuT5LoXcJvkw+Hy7KjUS8PC9SNnREjbM17aJVwawHp8JzjtKOeXYqtw8nXNuaeFgl/lBdfhcRyLi+XbeF90+grBs/ND2notvU90sIHPRfvOFG9wSzm/VuqYUfHzX+Ov5b6tosNbZIxtULz8G+I9Ptfk+CkzIqTt0dLKX3whGtgo++tk7LSa590rnv/Pgt/68twByZcPyxjpoHC4T7zJLhmvGZkRIW3PWNuEcGxQYt0lurpRsOnD8+I5bhTf8aGMg6/APThZ6tKF4lNeETy7ZQxuiYwPdH5I2+ctunCVzPWeIPz5Dv8Q/9cl9en54hXvFe9XaeWojHu/i4vK8UeC9+GSP2fz4bS0+qRoZeUH4JlyvFawWCo+Y5v4hDelxjxU5iyWC14HRHsbMlazr9zTXbbZ9TTT0upkzmFCcPpF6sq/lxsekbHTH4Rvk6KbW8VfPIVVUtOs1pznvUq+jaHST8X1zg9p+8ZLKy7uF59wpvDqdcFvuXBps8z3foW3JNc1BPdLxDPegrelfl0wpe+ZEdLWe4uHxO9Va6H2Co6LhZ89UotsL8dbRBOPk9rlL5IXjxXM58vY6Brh7DzxkL1meThNrT5edqp81S/cqdZj7JFas0/mJx4TjEdEY4eFj0NSo+4u+ys113P0yTdS6Wlnh7T1PHxQ+FJpak2wGRIN/ULq1AuEb2+Ix5grWL8sGB0mfnGVJNnjhY975Ruo1u10fkjbs56mpokfqSV7NNf1fioeolq7+K/f6Wi7YDcqelqtsVqk+X3MeotpafWpa0mrdbxES5dLLlst9eocyZ01yYtHCG69pY9qnnhEsK/mfKtcOzNC2vIX/gZv1W5h2JoR7AAAAABJRU5ErkJg\"></td><td style='text-align:center;vertical-align:middle; margin: 0.5em;border:1px #90999f solid;border-collapse:collapse'><img style='max-width: 100px; max-height:100px;display:inline' src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAHAAAABwCAAAAADji6uXAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAAmJLR0QA/4ePzL8AAAWCSURBVGje7drdj1TlHQfwz7KzAisWy4vVxbWA1VKgGoXUWLVJxaaWqKRpTHy90N5ZE4w3pr0hEi/lqolXJsYLrYlRGzVqg5WqFDUltCJarSC2IAV5ky0ssAuzvfg+xzP+ATObDPMkJ2fPOc/MM/v7nu/39/b0rcUp9KMpYwJ9GMdguTeuHhNl/kS57it/T8FoeTa1XJ/C6TJHudfR0f0LNibUGFa/oNly3SzHAE6UewMYK2eC0Tga5XNTWu4R/Kp3pPtN2nkM4Wyc9E1unVTj1oevcLA8awo+52AY3ym/fC/2lzlnlXszy7nCtftN2nkMm8LDhtrGBwSfo4LpaHl+CJcLrqP4vhrXS8oxXzDehn+IFg9iRN6F7jdp5zFsiC72C5YnsFV4uBB/xQW4Hg9gHx7DzfgVXsUcLMPbuA+PCN53Ca6b8JbaZ3a3STuP4ahoZhPHsRFHyrEQNwjfrhRufYKV2C4Y3YJL8We8gcfxEpaL/hIc39fT0jaNxqDYtiH4HMdVguni8mwXdoq+XoyPcSdexw7R3iGsFqw+wvfUsehczBNt7n6Tdh7D8bLqScHrA8zCKkzDT/EyrsU/y5wdgscG3C/6ehpPYTZ+Ur7zuy0LjZQ1ut+kncewX2LMptiY6OhubMYM4dhH5dftEF19TrR2h/jT/0m8s0y4uEV0uBpDwuXuN+nk8XCg5WY/vo0fCbabBLcVwtH1kke8ixdwq/i8jfhMNPVTeTeq752hF5e2aTSmSLw4JnniKvF7VcxyIX6BPfg7lpb5S/BDwfFpTBecFpf/4iCubllomWhy95t0crR0TPzZYfxA7dN2C16vS94xF3/D7VgnOciQYHxRmVcdyrxryt+fqms43W3SyePhgPBqjsSol4lmrpd8/jeiqeeKv/tYcozHRFeXik9cgg/Ll19bzr/HIr1aW5tG47TgNwXnCZ+G8RB+LTHLsMSgy8VHbijz/yD+bwUW4DbB+xWJi94vizxd5vTi0raMr3lIsDyJP5br9ZJfzJKc8Qmp2xwWn7mofHavxCuv4I4yf7XUBPbgNck7B88Ik05OzbvqTYxJHrdF4tB3hD+DeE/6SbPwb8GyX2KY7aKpz+Jz4eo6PCh+9b84VhbrfpNOjpY2RQO/JbHHfKmTvSn9h7VSB90v+cJM4esKPIOfS93mLvxM4p51Zf5mwW5UON39Ju34gn1VD3hEfNopyfVWSkwyIbYfkdzhbdHc2bhbajFXSP/pT6KZ4+IPr5IazotSD5h9Rpi08zys+r9nSc/iHqmP9uN3wsVPJF69VLBaIvHpmPi7IeHdSnwhuC8QnzlHcv/penFpm0ZjQmy+UHL8udInmoF/ST30yfJsuujiRWXOdZL3fym9qV34peB6SPAew/l6WtrG0Wio+4bTpXZ2tfi0bYLTjRJzzpOYZabwdqfEoXMlX5wnddRpWCO9qK3S69h9xpi08xhWe132idZdIdyaJnHMF+LbBoVP84V3NwnX3pA62ufl86sEy2o/1cXSm5oqnOx+k07OXoyqzzdWzk9JDni96OKF0pO6QPzb2aK/B/BjwfymMndzuV6sxvuEvBM9DNsyGqRWOiB2Piz8e1Nq1L+V/O9RPK+uz6xU19iqWGi56O5norUj5dlRiVd7+WFbRt8a9X7E2cKtwXI9IXW2W2SvxRHpzy8q80aEd9cJ/n+Rd2GfOq84KrHRTPG73W/Syekf9qv3tZ0vXKz2924XDIakDl71qTbgXtHXhtQGFuA/EhMNC567JMataqXdb9LO8/DhllWb6j3ZR0UTj0suuFAwPCU4jYh27izXu8T/Dav3dhxr+b5q31X3m3Ty6jQVD6tR5Rx95TzQcm9C4s7WvlW1R7+616/u21f3ejxsy/g/fzJ+pi2PEDcAAAAASUVORK5C\"></td><td style='text-align:center;vertical-align:middle; margin: 0.5em;border:1px #90999f solid;border-collapse:collapse'><img style='max-width: 100px; max-height:100px;display:inline' src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAHAAAABwCAAAAADji6uXAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAAmJLR0QA/4ePzL8AAAWGSURBVGje7drfj1xlHQbwz+yedmZLaQsFtkDRUlswRAgUkIQAGqrUECFcAEZikHDhhZF4SfiRAPdwxYWRBBIjUQmgxpgYwi/5kaBFqSVAQCnUggULBRTKLtvZWS6e980Mf8DOJLP7Jidn9sx7ztn5Pud5vt/vc97WbTIazJfPrbLBBA6Xfa/Mm8RcmbNQ5nXRKfvewPnzWDFwjQlDHuN/w+YwVkqMFzAl+ByFdViF/wpuLWzFf8q2Bu+WOZ0yp1uu1WAGnwqOHcF2/EM6fAwr51oS455gugKn4GjB6ULBdiX2l+/36nPzCKzGgXLhj/Fv/KPMHeTzeId0+BhOCv9agsUc2jges4LFerwhXNqDY4WbE3gC/8PF+BL+gLNxEs4VjP8qz8Kyli7KaFbo6x/h0qRgNid6+i+cjL/hAxwnfNyDS7AZvy379fgQG0RHf12OfYS1SyKkw8ewK/mqwSEcKbw5Ee9gu/BvSvA4Hy/hWuzCQ/gOdpRzdguvH8D1uByvCY7L+XBRRtP4Yg06K3juxOl4Be/hbtHGdeXYkzgHz+J53Cc8nBbdXIFXcYzk0/p8jH9Ih4/hgvBmQTSwLTq4U3LcvjJxGy7A68LbUyVXvis4dcp3b+Es4d5BXIq3cYJo6/iHdDQYVv61JR9+KDHfKdhslLql4nIZfif8XC+4vVYuuE7q1n14UWqeDZJvv7wkQjoaDCdF+1ZKj/BK+ftsqWd2S5+wXfLiwzhDeHWwbHVsFKxekvr1sOB7LL6yJEI6GgyJLs6LHjbCvw3CqydxleDykfDwn5LjVkldWz2ANdITfoKry/w/yTPx8ZII6fAxnJfYr5XcNoUzRTtPENxuxdPS811X5t5TLnC19BWbJA9+gG8LxnfgfsFwD762JEI6fAzrXVvCpUNSW54jvtpqyZVwi9SgPfy/zFuLK/Emfo9vCX9/Kn7dj/Ab8d2aJRHS4WO4UjS0h89E73bgCtwl3NxQjt0p/eEh4etN4q09IRz8Af4sPH5PMJyWZ2CqnDf+IR1NPuwJB1eJZu7A41KzbBcPbR1+jl/iL1K3dKTu+ZXku9elJnpbatyDgvEPJaeuXhIhHT6GvYE7z0geew7XCJaNxP84nCa9xHbxwx+UmvX7UtfMC0/b+BnOE5y/Kfxc9mkWZTStgbt+Kpp3htQzh6VXvEw09nT8QnCt7wuPEe7eIDx7TDy3oyVHbpH3VDWvjn9IR/PeYlZ8mrZw8QCeETy3iT5uKidskV5vteTKnninB0SPtwi+p9UbSN9Z/bfxD+lotLSt//53M54SH/si8Vq2iA++W2rT98U/3Sge6qxo7EPlF3TEu6k+wJz0JfNLIqSj6Q8r/+bxVeHZWvFoXsbXhY/fLSc9hRsF3wPC053l3HfEj5kWz636c2ukTh3/kI4mH86Kzk2V/2Cf+Cw/xqOCyya8IFh2pYbdLHq6S7zt++V94ptSB50sPf+RZVvm4aKMZlLyWF271JW47xI+/UQ4+rjgBt8r+71Sr14ivs4mwfRp0dGXpedcr+/Jjn9IR6OldQ1NW2rKlsT/7+XYWcK5P0od2sK9uBmPSI//mWhqIx7PI/IsbNXPt4M18PiGdOg3bN0u8e3qr1uaFf18QeqTI+R97rT4ahOijbPig3akF2wEt/1Sy9Rad408J8tauiijdZvgV73MBeFU7Q33S92yV2qTqrukX9wmeXSmnDddzu3or1es61CXtXRRRrMgOlrXAE8Kx2b0ObQV3xAsOsLROf11apNlXltwagSzwWt29bk+3iEdnV9ax7zEutY3FZe6BrUrfWFXfz1j1ci61rTni/mvfl7W0kUZzeAa/aqT1UfriR8zU76r66cmBNee/rrhhYGL1j6/eqh1DWt7FL9w/G/4OcVBaJMt0Q2TAAAAAElFTkSuQmCC\"></td><td style='text-align:center;vertical-align:middle; margin: 0.5em;border:1px #90999f solid;border-collapse:collapse'><img style='max-width: 100px; max-height:100px;display:inline' src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAHAAAABwCAAAAADji6uXAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAAmJLR0QA/4ePzL8AAAV5SURBVGje7dpZjJ1jGAfw35n5ZrrMdDNaYk9pJaqWJgjiQkQIFyIhjQvrlRCRCHFDLIlbES5EiC1c2UNCqmKNktJaEntpUS2lLarTzkw7Lv7v55z2fs4kn3lvzjnveZd8z//7P8//ed63dae0UfRjHL1oYU/5HENVxrTK+B7sLb9b5XdvmTOGEcwoY3pKf2/53tXW/A2r2tbE3sTeNUY92hgQHHsF0xH0YbfgT96BnrJm57yqY92utuZvWFXaGOwVW4/ZF8dxwW6sfE7rGDOOQezAQFl0rMyr34nRslbN12abtPsYtrRx6xFu1TjtEJx24y9sFbzq/2u+7dT2wbuEm3325V7LlC+doFbV3Bjv6NyL7dgs+G3AutJ/oODxd/mvD7OxAIcIpkeU/2eUsTPwjykeTlCr6l07/eVxgtcyHCN4vIgfsVSw+6b092E1tuHDsuhiLBS8DxNu1jqp+Sbt+oatOwW/2WL3LTgBR4qPnC/YLMASfI6ny/iDMCw4rRMcN+BUfIs18j7M046VzTdp93lYx7EBwWgd1gpBD8UdOF/i2rMYwgV4FCdLfBzCKXgPVwt+0yVmDmvr11qrNtuk3cewRzg1GwcIJwfE7gfjZvyAN3GR8PFuXCc+8wP8LLjuxG94DteUdVaIX56prZuabdLJwXBAeDeOWRL3KjwvWB6AX8Q3Xip8PR0bBecL8YXEz6dwlfB2k/jbPcLXuf8Lk3Yfw1HRMoeL/yPxa7PgthrLhUuPCwcfwfs4TfztS8LPxbhX3oFfBfOR0t+S3KT5Jp0cHvaL3hyQGLZGcKht38INpe91/IRLBJ+NooWW4lUcX/pmiR7dIjyeZkqXTlCr6ty7r3T0SezajTNxOW7CEzgX95Rxd+FW0aSD+B1vCf4H4STJCSss0q6lNt+k3cewV3KCrySn+FFwXI7X8IlomrPxcsfEPXiwzFkrfnet6NgRvCF6dUkZv0s0avNNOjk1737JKf4UPGfgYfF9s0VvfoZXOiauEw00V/znjcK5nyVenig69nHRNvO163PNNmn3MdwrddEvcRQ+lvzgQHwq9RiSq9dnEGdgJe7HLZLXE+4twxXiXz8VXv8h+M2ajCds/obVNMFmGO+IjvkCR0sc/FD87JNlwsWC9fuiY6/CeomFx8g7sF20zybhcH22VZ9jdbU1f8P/covpokEW4BysEq7NEoxHJCc8q0x8QGLcRtwnGuZBnCcxdKHomNMlTxkVPJtv0u5j2F927Rc9+q341pclTxwSjXqUcPMZ4eLXEg83S231azm3WCNxdJXUaLbgWHkXpnKLCWnVmGBHuPKb+M75ojWXCp9WSIxchrclz1gpmG8oc67E9xIHt5Y158g7UD9Z8006OfnhP2XnHXhXcr3tZcCo6JojJWb2CY57RKv8UMbAY9pn9+QccVDqpP2m8sMJahXh2a8Sr4a178YQf1lJvNtSxr4ttZpFuB4vlPm79lv8z/I5UNYZmYwnbP6G1Vj5skP4crDg9YfUab4TDA8RvbpEzqAuE77OkxxyQGpq73UsXvOuZ7/vXW3N3/C/c/xB0R0nSI5wnGiRxZLDr5cYd1sZV+O5SnzmevviRzj7t+SG/eKHm2/SyYmH9d20OeIfl2vfo3lIYuSQ1F7qs8J3JPZNkzxirZzfj4pG2ia+c1j4PVP73mKzTdr9uxh3CF/GBMdF4j//Eg6eKf71I8kNN0v+NyS1nell/hzBfLXU3yrBcag81aipc/wJalV9F6qSmPiK5Bf12fvnolHnSi7xFa4t40+RWsz0ssZhwtOZ2rWB+on6TJ3jT1Br3V5/ETvXd/M77/TW9zV2CnbLRHP+Xn7PK2O3Cf6DZc5W0ar73xtvtkm7vuG/sc9qwRy5zjwAAAAASUVORK5C\"></td><td style='text-align:center;vertical-align:middle; margin: 0.5em;border:1px #90999f solid;border-collapse:collapse'><img style='max-width: 100px; max-height:100px;display:inline' src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAHAAAABwCAAAAADji6uXAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAAmJLR0QA/4ePzL8AAAWWSURBVGje7drZiyXVHQfwz52+3W0744zGBWcco2JciE7EUVFcSEI0ouOSuKFBoz74IIL64IOCuyKiPkiIBiQPIQmZJIoKeTAhEDFRDJlxwWXQiYmjRmfRid3tbD29XB++p6zrH9D3QnUfaOpW3VN1+v6+9f3+ttO6F7swgham0cFMObYwhKny3YJyrdV13pHRKeczZX4bA9hdvh8s3/d0NH/BNgyLnQfK30Q5dtR4ttVYKfO7r1X4tcrnPcp91b1TXXN6Opq/YLviUmXnbt4R7uzAZPk8KRhPlfsmhMP7YVSwG8D2Mn9Q+Fodm2/S3mPYEpvPqHWxOp8Svg0JxruwDUvxHywqc7+BZViHvbCiHFv4uDyzegeab9LeY9gRnKYFpxmMCY9GcDiOF44NCD7P4ib8X3R4reB1Jo4RLEfwIb5T7t1Yntl8k/bHH3Zz8Qv8F4txMDZgT8HtNnwg+rkG/8b+OAdX4cfC3Z14T7i6N47C/8qva75J+4NhxcEJ8X2LhTubRSc3YDl+jxtFJ9fgCvwBr4vGnlfOP5F34Rbh4j5lscqHNtuk/fGHBMdp0cYlwskBnIGncRyeFAxX4gn8pNxzP67Fo4LZTwXjQ3EI/i7vxa45YdLeYzijjkOXiPbthYvxOV7E98Vnno5TcRBewyO4VDRzMU4S3R0X7n2EE8WfbsKbc8Kk/cFwpzp2bGOhxB/rxf4D4vuOxG8lThnCY4L9D/ArnFUeWuWU38RWwfNTvDsnTNrzBVu3C247xf9dLXHmfmL7zThM4tP1guMvJJe4U/RyGW7GtyQmHSz3j+A3gt8p+GxOmLT3PBySWGZYNPIznCB53Xmih9tFLz8R/3eh6Oq48PNT/AxHC5f3Ff1dgVvxSxxR5jXfpP3JD/cQHJeL3zoJL0gs8wwuwh8l1/gC1+A5wWqb5BZrhcNbhHNjwsORct9awbf5Ju2PP6xW3iZ53bDo4TtSkxkXTM4Vri3Bn/Cg5I1PiC98WeLYK/Cjcm2r8Hi0PLf5Ju09hgvUdcwxiTsWSnz5kPi453EJThZct+AyPCW4Xyi+siO5/vGSgywRXT6365c136T94WHVe2iJrj4pccrZon9n4m08XK5tw3eFd+tFX0+Teto03hdfuk5qPlslN5yvl87KaFf5RHe/ryV5wYpy/hHOlzrbOfgLXhVtXCVxzYv4m+Qcm0U/78GfBcul6rp6s03aewzbXStPiI4uFT4eJ7p5hmBxluA4KPXP53CD8O4lXF6eMygx7DLR3CoXmfeHszK+qnm3BIt3RC8PwT8lVt1HYtO2YDOONwTT3wmuV0ot5yipA1wgtZuX1X2oeR7OymhXvfUdkhsOSkz5lsQ3w1LznsBfJcdfIz60qqWtEp84JfW0w6WHMYTV6jrsfL10Vka7LRxZIDgMCv924ecS3/wa10nef4Tw73Hh5qOiucPC1/NFN0el9rO9nFe9keabtD9aOiHYHSS+cIvElaMSV96COyS3eFb6v2dLz+lofA//Eh6PCU9XSq45Krwckvio+Sbtfb30PtHAyXLcUzBdKTZfLXnewYLhx2XucvGf66THuEji043Sk9oh/nENDpBYaT6mmZXxVUxT7ZfZJHW21VLrfgD/KNffFa7+UPrBh0kP4wV8W3KRCfGVG8vcas/jjjlj0v5gOKmuma4TLZ0W+4/LfouOxDo7JQ88VHT3WOkjXiB+8Xmps45KDDtYFlrYr1/Y/AVbd1cfyvE96cXvL/q4QXKE66Xv8Gr5L1dJf2m3aO0r5f4PRW+r3nK1V3FK3onmm7T3GN7l6/u4Z4SXVd9hkfBpEw4s/+F24epYubZUXTNfpN6zuFC9b7x7/2OzTdq/HvCUeh9vtX+7yss76lrchLom11HnftVxt5qDk+p6erV3tfkm7fmCXwK1j4MqwXjs+wAAAABJRU5ErkJg\"></td><td style='text-align:center;vertical-align:middle; margin: 0.5em;border:1px #90999f solid;border-collapse:collapse'><img style='max-width: 100px; max-height:100px;display:inline' src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAHAAAABwCAAAAADji6uXAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAAmJLR0QA/4ePzL8AAAV+SURBVGje7dpLbFzVHQbwnyfXDk4cEiABJ4BQAPMI4ZFCqSoQLQ8BgiAelVjAAtQF226QqNSqSCxYVIIFqlqgElKLKti1pUIsyhshAkTiGRAhiEcTAklMQ+LGsWPPsPjO7UzZdTFjaTxHupp775x77vj/+fv+3/+cM3SftBYaOILhjntDmC/nzXK/KveGS/9GR98hLCn3Dpdr3+vb09b/L6zmOy6agk2jHE3MlfNKcKqvlwgmhzHScT2MmXI9V74bwmzHs/0d0t5j2NLmUKXNqRrbUcGmxqDGrFX6z5ZnZ7BUm3fzpd+Rcq+BZYsipL3HsI55Q5uDzY4OhwSvGcHvG/y7XP8HYzgBJ2KP4LVC8BzpGJfg3f8h7T2Gc9r5qynYNLTz4F5MCgf3Cn4NHCzff4m1guUenC4YnoI1wuMR+V8YXRQh7T2GI4JPpzc5KBybLJ/7sA6/xw4cj1/hTvwCZ5fnzi7j7MWzOA0TOBarDDxNl1o1L7rXxLfleLXcGxZe3Yxd2C84TOBJbMMnwrEGnsJqvI4XBctRHKXti/o/pL3HcIlwbQQH8BY+xE+wWzRxZ+n8Dm4Q7ZzE1eX+c/gMywX347FBdPlI6TuEYxZFSHuP4ax4lf14v5xfJFg9hpvw59J5CtfgIZyHvwq33hYd3Yat+LXo8a7yF02X5+u829PW/y+s6jqi9jLTuA5Xlg6viOe8Eh/gDMHmEK7AA8KvacHzZ7hb/g8my5gj/rf27O+Q9h7DYcl9X4nejeMs0dVJnCP1xNvl143iftyCF3AcPhJfOl7Guahcr8Vr+LFwcIBhV1pFctYuwe0SwWKf5LTbxZ/UdcIXuLd83ip+9EzJh6/il+JzZvG4cHCqPL90UYS05y8c+o3E+w3h4QPiSdbgGcFgAv8SwO/CHcLVH0itMS7ec0z09jKpIyvh5IT427FFEdLe87Ce+1wjc2HnCx9/J77zK/wJ6/Fx6bsWK0VjLxAcv5T8OCY8fL+MsV3qw1O1a4z+DunCzbWtkjp+N/6Cq/Am3hPPckB8z/nlV76BC8uzo6K9jwim2wXXZ8v36wXz2jf1tPX/C6shqemGBafnpE5/WTT2HMFrk9SKn2CL4LRRcBmXfLga75b+Ix0vmZeaZTDX1pVWNSXeSyXOhwWXLaKvP5Q68UTR0KNFJ88TX3q91IUr8Wh55nXxrXVriQYPMOxKq5rCt1mpJf4oeBwS/HYKB1eIP71EuPkxfo6/Ca4HJZ8uEz/7z46XHKO9TtX/IV0YT9MSvzEp/nMH7sHXeFry5GXiQ28X7bxfdHNf+dUfyBrUGeVevd50srZW13sF+jukC1MfzpaL5dq1+nap+ybEm+yXudTNws3fClc34O+l35myBrJdas4LcK7oaL1G3P8hXbh8WO+P2Sh8ew8Pik7uKAc8jBtlnrzec3GWeJ01Ml+6U+rCDWWs/aK1Ay3tSqvq9d5KeDgt/nSdxP+A5MV9Mm92gvidwzhJ1qqekTxIculmfC71ymrxrEe01/X7O6QLlw/nJeYt/EPmVaYEj60yr/ZT8ZZP41qZ05nCxcK9LbhN8L1F1oAPSK5lkA+71KpWOam1dLdoYs2jFVIrVsK5Y8WnLMePJMeNCd8uF55uErw/LMeU9r6o/g/pwuTDIdG5WZl7OVk0db3ksU2yjvg8PpV9M4/J/HZL8JsRPd1cnp2UXLlF8B8zWMfvUvvv/tKGaN5Rsv47LbicJOtL5won10nNcSmeEL2ck/p/VekPfyhjLNPey9hcFCHt/brFfYJBS7R0XGqEjaKTy/+PwbaV/jtl38a3Zey58v0Aw6606vv7unfJ2uBLwruW1IG09wd/JPVgvf6/R/R0VmqRlnifmTJmVT4HvrQr7TtHS2uMTTk2PQAAAABJRU5ErkJg\"></td><td style='text-align:center;vertical-align:middle; margin: 0.5em;border:1px #90999f solid;border-collapse:collapse'><img style='max-width: 100px; max-height:100px;display:inline' src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAHAAAABwCAAAAADji6uXAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAAmJLR0QA/4ePzL8AAAV8SURBVGje7dpLbNzVFQbw39hjOwlxXjakEB6RqdKWhygQhYYWUd4RqtpSgVhU6oIVbIAFSyQqVWKN1E0XBYGQCqVVqwJqYUF5CISABQJEmpIA5eEGApg6b5Kxh8V3b/8GsfWMNPaV/pr/6947c775zjnfubd1F1roYqicd6S1y/V8eT5f3qnnwwv6dst1bXWMofLufBlvSI/b4E/Yrrav+BxfcD6PEcwJVuPluoODgslJOIAZzf9g4Zhz5bq1ZEzaewyHBbcOxjCKL3BUw7Whchwu9w4JZkdxQulTP4fLMar5L1R+zi0Jk/Yew+rjqt07+K/gegzfxhHh4DrM4pLS5z9YU957SrBdj73YWJ6NL/hVyxguSmsTvhAbHxMsRnEKVuM8TON8PC2+8WlcgGdwHc7Bx4L9WnxL/O0ewXOdhtODbdLeY1i5MSz4HcbngtsanIj3sR07BcuzsKPcP19i5M/xYjnOFux3yv+hjr96SZi09xjW3LPmkbOC33rxg+/iL4Lr//ABThO+vS5cnBYc9wsffyrxcRoPlPOLsHlJmLT3GNY8slWOYazCBryHF3BreedswfZj7BLcXsNNgu8GiZG/Lc+uk5z1MeH3Mg8XpbXbkl92JS89IpzbL3bfIdg9gbdwmcTC6wXz7XhZ/Oq/8KMyxiFcXMb6M1aW+4Nv0t5jWPFrC4YTWCF8OoJ95fml+KvgdRv+KH7yTslfTseFeAcPlT6/wlXiS+scg2/S3mM4InrwhPK5Et/V8HCN4PM7/BJv4BWJidsk99wueuKecu8afKdM0NHUCarOHGyT9h7DjsTAA4LjLrwtPrErGDwoOvED3Isbyucs7seneBw/Fh1xEf4tsXQKm0RbLPNwUdr/c5qO8G6D+MbL8Yj40inBYC9+I/FzdsEg+4SLW6Q+8Ad8H2eWsS6R+FhrcYNt0v7ow6q9WxLTpsWnXinaoiOa4VnJP2//2iDTOFdymkfLGJtFoxBsd1mu0yxSa9daZltyy4OSi+zET4Rv6/Ck4LgZN0t8rG2s9HtXdMiU+NKtkh/Nl8/lOs2itPaw8KMjeegVOBV342rBr4OfSez7xFfxI3nOJsHyh/hHuT5Y+u8rY4/14xcO/oTtltRchsTeI8KZX+Al4dSn2C01mXsXdN4u/vdayYcmZT1jsjx/CX8XjDeWcQffpP2JhzX3HxU89+IMfITf40apuTwi2uGZ0nlG9N9WiYMXCPd2S0zcKxplssyxcA1jcE3an7yU1LuPiK74SOptXcFuFT4T/fhUeX9SdP8E/in4PC9+80zh3X7RI7UGdHBJmLT3GB7TrNG2RANsktxlXHKax/EDyTPvwx2CzzviJ2+UmLlb4umE+OU9wu2jZfzlvHRRWruuCw0JV6ZEo78ueuB7ou+nRG9slPran2Sd8EPh727NetOFUqvbU95ZqyxULgmT9qdO0xU/uVpwWy+8msDDmjXcTRIDXxZMRkX7PSScnRGu7serUi84XN5dIbWgwTdpf9YtiI0P4W9SIz0Db0rMawsun5RveIfU31YJ326RmLhVYunbguUXwlGaNYvBN2l/9mLUGtiI8O15sf8WwW6b+NIOnivHpKwrzeJkWZd6UWLeNomVcxptX+Pu4Ju0P9qiVWZeodnv9KasT2wWbo1Jne0m8YkHhJefiw5cJzqyLTXT1YLzhnKv7rsafJP2fMLWrzXr+HU9eF5qoDOi8d4TvL6pTQmeI7Jm1RU8T5ZcZoUm3q7qxy8c/AnbdW32mGbvUlf2/q4XHm4RLFYKH2dE640LTp8J7yaEl12p24xp9oiPLphjsE3an3g4p9nLXfdjj2v22K/R+MG6P61qvbb4y+Ol70mSvxz3VT1Y9/MPvkl7PuGX0Chpw6QyVCEAAAAASUVORK5C\"></td></tr></tbody></table><div><small>(a vector displayed as a row to save space)</small></div>"
      ],
      "text/plain": [
       "10-element Array{Base.ReinterpretArray{Gray{Float64},2,Float64,Array{Float64,2}},1}:\n",
       " [Gray{Float64}(0.4908239636570215) Gray{Float64}(0.5143737541511655) … Gray{Float64}(0.5024939212016761) Gray{Float64}(0.495916529558599); Gray{Float64}(0.5094658164307475) Gray{Float64}(0.513299941085279) … Gray{Float64}(0.5058013712987304) Gray{Float64}(0.5018126231152564); … ; Gray{Float64}(0.5092319445684552) Gray{Float64}(0.511765138246119) … Gray{Float64}(0.5012233739253134) Gray{Float64}(0.4948901003226638); Gray{Float64}(0.49531134590506554) Gray{Float64}(0.5102947317063808) … Gray{Float64}(0.4997979305917397) Gray{Float64}(0.5044707474298775)]\n",
       " [Gray{Float64}(0.48987679835408926) Gray{Float64}(0.49530668929219246) … Gray{Float64}(0.5130084594711661) Gray{Float64}(0.5001193022544612); Gray{Float64}(0.4970226907171309) Gray{Float64}(0.5205637812614441) … Gray{Float64}(0.5065035792067647) Gray{Float64}(0.5052895932458341); … ; Gray{Float64}(0.5168877840042114) Gray{Float64}(0.49479521717876196) … Gray{Float64}(0.5057567786425352) Gray{Float64}(0.5140665620565414); Gray{Float64}(0.49830957339145243) Gray{Float64}(0.4946436043828726) … Gray{Float64}(0.5196775104850531) Gray{Float64}(0.48503570817410946)]\n",
       " [Gray{Float64}(0.4891350856050849) Gray{Float64}(0.5030100019648671) … Gray{Float64}(0.5027714800089598) Gray{Float64}(0.5089084636420012); Gray{Float64}(0.4925731043331325) Gray{Float64}(0.49041686952114105) … Gray{Float64}(0.5068537299521267) Gray{Float64}(0.5084607899188995); … ; Gray{Float64}(0.5056128348223865) Gray{Float64}(0.5002068866742775) … Gray{Float64}(0.4984070745995268) Gray{Float64}(0.49468112317845225); Gray{Float64}(0.5087750880047679) Gray{Float64}(0.500645040825475) … Gray{Float64}(0.5053641679696739) Gray{Float64}(0.5114392694085836)]\n",
       " [Gray{Float64}(0.4918334651738405) Gray{Float64}(0.5163223557174206) … Gray{Float64}(0.50553307402879) Gray{Float64}(0.5091525819152594); Gray{Float64}(0.5072587495669723) Gray{Float64}(0.5132663827389479) … Gray{Float64}(0.48877352476119995) Gray{Float64}(0.5140587333589792); … ; Gray{Float64}(0.5092744007706642) Gray{Float64}(0.5078554907813668) … Gray{Float64}(0.500650086265523) Gray{Float64}(0.49698861106298864); Gray{Float64}(0.5129081290215254) Gray{Float64}(0.4867618391290307) … Gray{Float64}(0.4893770758062601) Gray{Float64}(0.48876699432730675)]\n",
       " [Gray{Float64}(0.5098865265026689) Gray{Float64}(0.49293799977749586) … Gray{Float64}(0.49498313339427114) Gray{Float64}(0.4943969347514212); Gray{Float64}(0.5064350008033216) Gray{Float64}(0.4777735751122236) … Gray{Float64}(0.5059327948838472) Gray{Float64}(0.4981385412393138); … ; Gray{Float64}(0.5041584065183997) Gray{Float64}(0.5121502699330449) … Gray{Float64}(0.5022772999946028) Gray{Float64}(0.49385563423857093); Gray{Float64}(0.5116583108901978) Gray{Float64}(0.4953797049820423) … Gray{Float64}(0.5103746270760894) Gray{Float64}(0.5167519059032202)]\n",
       " [Gray{Float64}(0.4963116191793233) Gray{Float64}(0.4985369035275653) … Gray{Float64}(0.48472269531339407) Gray{Float64}(0.49451918667182326); Gray{Float64}(0.47798573784530163) Gray{Float64}(0.501429944881238) … Gray{Float64}(0.5177054069936275) Gray{Float64}(0.5100058363750577); … ; Gray{Float64}(0.49064289685338736) Gray{Float64}(0.4991192268207669) … Gray{Float64}(0.49084132071584463) Gray{Float64}(0.4836706444621086); Gray{Float64}(0.5067608170211315) Gray{Float64}(0.495110041461885) … Gray{Float64}(0.49275354528799653) Gray{Float64}(0.508804983459413)]\n",
       " [Gray{Float64}(0.5032706707715988) Gray{Float64}(0.5004886631504633) … Gray{Float64}(0.49449901888146996) Gray{Float64}(0.5041607609018683); Gray{Float64}(0.47683392465114594) Gray{Float64}(0.5136641198769212) … Gray{Float64}(0.5083635468035936) Gray{Float64}(0.49915948684792966); … ; Gray{Float64}(0.4937529778108001) Gray{Float64}(0.48974801879376173) … Gray{Float64}(0.49833616032265127) Gray{Float64}(0.4979303772561252); Gray{Float64}(0.4905413128435612) Gray{Float64}(0.48986768908798695) … Gray{Float64}(0.5038345188368112) Gray{Float64}(0.4931643372401595)]\n",
       " [Gray{Float64}(0.5095663638785481) Gray{Float64}(0.48027458414435387) … Gray{Float64}(0.4809955582022667) Gray{Float64}(0.5001712987723295); Gray{Float64}(0.5101840989664197) Gray{Float64}(0.507750787306577) … Gray{Float64}(0.4842640347778797) Gray{Float64}(0.49829891650006175); … ; Gray{Float64}(0.4977661466691643) Gray{Float64}(0.5211146771907806) … Gray{Float64}(0.5045258770696819) Gray{Float64}(0.4986607431201264); Gray{Float64}(0.4845365984365344) Gray{Float64}(0.503006627317518) … Gray{Float64}(0.5025216990616173) Gray{Float64}(0.49614120298065245)]\n",
       " [Gray{Float64}(0.5098621947690845) Gray{Float64}(0.5081747900694609) … Gray{Float64}(0.519333703443408) Gray{Float64}(0.49926065909676254); Gray{Float64}(0.4983776818262413) Gray{Float64}(0.4976025358773768) … Gray{Float64}(0.49493960477411747) Gray{Float64}(0.4786314833909273); … ; Gray{Float64}(0.511407676152885) Gray{Float64}(0.49439472798258066) … Gray{Float64}(0.491908460855484) Gray{Float64}(0.4813338480889797); Gray{Float64}(0.49946358660236) Gray{Float64}(0.5032675282564014) … Gray{Float64}(0.490285805426538) Gray{Float64}(0.5066517656669021)]\n",
       " [Gray{Float64}(0.4988562040962279) Gray{Float64}(0.5033672582358122) … Gray{Float64}(0.48556274827569723) Gray{Float64}(0.5004890039563179); Gray{Float64}(0.49528156174346805) Gray{Float64}(0.5084399841725826) … Gray{Float64}(0.4910677056759596) Gray{Float64}(0.5000185667031474); … ; Gray{Float64}(0.4983965229475871) Gray{Float64}(0.5122394608333707) … Gray{Float64}(0.49630676419474185) Gray{Float64}(0.4902312057092786); Gray{Float64}(0.49863077502232045) Gray{Float64}(0.5013216945808381) … Gray{Float64}(0.4944363934919238) Gray{Float64}(0.4870054926723242)]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "\"Epoch 99\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Let us visualize the evolution of the weight matrix as images below\n",
    "# Each row is turned into a 28x28 image with positive weights light and negative weights dark gray\n",
    "using ImageShow\n",
    "for t in 10 .^ range(0,stop=log10(size(lin,2)),length=20) #logspace(0,2,20)\n",
    "    i = ceil(Int,t)\n",
    "    f = lin[1,i]\n",
    "    w1 = reshape(Array(value(f.w))', (28,28,10))\n",
    "    w2 = clamp.(w1.+0.5,0,1)\n",
    "    IJulia.clear_output(true)\n",
    "    display([MNIST.convert2image(w2[:,:,i]) for i=1:10])\n",
    "    display(\"Epoch $(i-1)\")\n",
    "    sleep(1) # (0.96^i)\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
{"cell_type":"markdown","metadata":{"id":"S4XMqPTS1d8a"},"source":["## Colab setup"]},
{"cell_type":"code","metadata":{"id":"BrHjOFFsxf7W"},"source":["# Julia/IJulia installation cell: run once (takes 30 seconds) and reload page before running the Julia cells in Google Colab\n","%%shell\n","wget 'https://julialang-s3.julialang.org/bin/linux/x64/1.6/julia-1.6.3-linux-x86_64.tar.gz' -O /tmp/julia.tar.gz\n","tar -x -f /tmp/julia.tar.gz -C /usr/local --strip-components 1\n","rm /tmp/julia.tar.gz\n","julia -e 'using Pkg; Pkg.add(\"IJulia\")'"],"execution_count":null,"outputs":[]},
{"cell_type":"code","metadata":{"id":"bkBZp_6YP4gf"},"source":["# Package installation cell: run once (takes ~3 mins) to install packages used by this notebook\n","ENV[\"JULIA_CUDA_USE_BINARYBUILDER\"]=\"false\"  # Use this for faster installation on Colab, otherwise CUDA libraries will be downloaded\n","using Pkg; pkg\"activate .\"; pkg\"add CUDA Knet AutoGrad IterTools MLDatasets Plots ImageShow FileIO JLD2\""],"execution_count":null,"outputs":[]}
 ],
 "metadata": {
  "accelerator": "GPU",
  "kernelspec": {
   "display_name": "Julia 1.7",
   "language": "julia",
   "name": "julia-1.7"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}