Neural translation - seq2seq.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural translation - seq2seq\n",
    "## French questions to english questions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Difficulties:**\n",
    "\n",
    "1. Output of arbitrary length\n",
    "1. Order of tokens in the input and the output is not the same"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%reload_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.text import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset\n",
    "http://www.statmt.org/wmt15/translation-task.html\n",
    "\n",
    "Obtained by web crawling millions of sites and using simple heuristics such as replacing *en* with *fr* etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "PATH = Path('data/translate/')\n",
    "TMP_PATH = PATH/'tmp'\n",
    "TMP_PATH.mkdir(exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = 'giga-fren.release2.fixed'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "en_fn = PATH/f'{filename}.en'\n",
    "fr_fn = PATH/f'{filename}.fr'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training a full translation model takes a long time. In this example we therefore focus only on questions that start with *What*, *Where*, *Wh...* etc. and end with a *?*.\n",
    "\n",
    "Compiling makes the regular expressions faster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "re_enquest = re.compile('^(Wh[^?.!]+\\?)')\n",
    "re_frquest = re.compile('^([^?.!]+\\?)')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "lines = ((re_enquest.search(enquest), re_frquest.search(frquest)) for enquest, frquest in zip(open(en_fn, encoding='utf-8'), open(fr_fn, encoding='utf-8')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(None, None)\n",
      "(None, None)\n",
      "(None, None)\n",
      "(None, None)\n",
      "(None, None)\n",
      "(<_sre.SRE_Match object; span=(0, 15), match='What is light ?'>, <_sre.SRE_Match object; span=(0, 25), match='Qu’est-ce que la lumière?'>)\n",
      "(None, None)\n",
      "(None, None)\n",
      "(None, None)\n",
      "(None, None)\n",
      "(None, None)\n",
      "(None, None)\n",
      "(None, <_sre.SRE_Match object; span=(0, 72), match=\"Astronomes Introduction Vidéo d'introduction Qu'e>)\n",
      "(None, None)\n",
      "(None, None)\n",
      "(None, None)\n",
      "(None, None)\n",
      "(None, None)\n",
      "(None, None)\n",
      "(<_sre.SRE_Match object; span=(0, 11), match='Who are we?'>, <_sre.SRE_Match object; span=(0, 15), match='Où sommes-nous?'>)\n",
      "(<_sre.SRE_Match object; span=(0, 23), match='Where did we come from?'>, <_sre.SRE_Match object; span=(0, 17), match=\"D'où venons-nous?\">)\n"
     ]
    }
   ],
   "source": [
    "for i, l in enumerate(lines):\n",
    "    print(l)\n",
    "    if i == 20:\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "questions = [(e.group(), f.group()) for e, f in lines if e and f]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle.dump(questions, (PATH/'fr-en-questions.pkl').open('wb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "questions = pickle.load((PATH/'fr-en-questions.pkl').open('rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('What would we do without it?', 'Que ferions-nous sans elle ?'),\n",
       " ('What is the absolute location (latitude and longitude) of Badger, Newfoundland and Labrador?',\n",
       "  'Quelle sont les coordonnées (latitude et longitude) de Badger, à Terre-Neuve-etLabrador?'),\n",
       " ('What is the major aboriginal group on Vancouver Island?',\n",
       "  'Quel est le groupe autochtone principal sur l’île de Vancouver?')]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "questions[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "52328"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(questions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "en_questions, fr_questions = zip(*questions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Tokenizer` is a fastai wrapper around *spacy* that uses multiple processors for speedup."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "en_tok = Tokenizer.proc_all_mp(partition_by_cores(en_questions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "fr_tok = Tokenizer.proc_all_mp(partition_by_cores(fr_questions), 'fr')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['what', 'would', 'we', 'do', 'without', 'it', '?']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "en_tok[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['que', 'ferions', '-', 'nous', 'sans', 'elle', '?']"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fr_tok[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Average length of the questions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13.345895123069868"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean([len(q) for q in en_tok])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16.26809738572084"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean([len(q) for q in fr_tok])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Discard questions that are too long"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "keep = np.array([len(q) < 30 for q in en_tok])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "en_tok = np.array(en_tok)[keep]\n",
    "fr_tok = np.array(fr_tok)[keep]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle.dump(en_tok, (PATH/'en_tok.pkl').open('wb'))\n",
    "pickle.dump(fr_tok, (PATH/'fr_tok.pkl').open('wb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "en_tok = pickle.load((PATH/'en_tok.pkl').open('rb'))\n",
    "fr_tok = pickle.load((PATH/'fr_tok.pkl').open('rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Numericalization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "def toks2idxs(tok, pre):\n",
    "    freq = Counter(t for q in tok for t in q)\n",
    "    itos = [s for s, c in freq.most_common(40000)]\n",
    "    itos.insert(0, '_bos_')  # beginning of sequence token\n",
    "    itos.insert(1, '_pad_')  # padding token\n",
    "    itos.insert(2, '_eos_')  # end of sequence token\n",
    "    itos.insert(3, '_unk_')  # unknown token\n",
    "    stoi = collections.defaultdict(lambda: 3, {t:i for i,t in enumerate(itos)})  # if string not found, set to '_unk_'\n",
    "    indcs = np.array([([stoi[t] for t in q] + [2]) for q in tok])\n",
    "    np.save(TMP_PATH/f'{pre}_indcs.npy', indcs)\n",
    "    pickle.dump(itos, open(TMP_PATH/f'{pre}_itos_pkl', 'wb'))\n",
    "    return indcs, itos, stoi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "en_indcs, en_itos, fr_stoi = toks2idxs(en_tok, 'en')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "fr_indcs, fr_itos, fr_stoi = toks2idxs(fr_tok, 'fr')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_indcs(pre):\n",
    "    indcs = np.load(TMP_PATH/f'{pre}_indcs.npy')\n",
    "    itos = pickle.load(open(TMP_PATH/f'{pre}_itos_pkl', 'rb'))\n",
    "    stoi = collections.defaultdict(lambda: 3, {t:i for i,t in enumerate(itos)})\n",
    "    return indcs, itos, stoi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "en_indcs, en_itos, en_stoi = load_indcs('en')\n",
    "fr_indcs, fr_itos, fr_stoi = load_indcs('fr')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'what would we do without it ? _eos_'"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "' '.join([en_itos[i] for i in en_indcs[0]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'que ferions - nous sans elle ? _eos_'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "' '.join([fr_itos[i] for i in fr_indcs[0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ! pip install git+https://github.com/facebookresearch/fastText.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "import fastText as ft"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "en_vecs = ft.load_model(str((PATH/'word_vectors'/'wiki.en.bin')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "fr_vecs = ft.load_model(str(PATH/'word_vectors'/'wiki.fr.bin'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_vecs(lang, ft_vecs):\n",
    "    vec_dict = {w: ft_vecs.get_word_vector(w) for w in ft_vecs.get_words()}\n",
    "    pickle.dump(vec_dict, open(PATH/f'wiki.{lang}.pkl', 'wb'))\n",
    "    return vec_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "en_vec_dict = get_vecs('en', en_vecs)\n",
    "fr_vec_dict = get_vecs('fr', fr_vecs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "en_vec_dict = pickle.load(open(PATH/'wiki.en.pkl','rb'))\n",
    "en_vec_dict = pickle.load(open(PATH/'wiki.en.pkl','rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "ft_words = en_vecs.get_words(include_freq=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "ft_word_dict = {k:v for k,v in zip(*ft_words)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "ft_words = sorted(ft_word_dict.keys(), key=lambda x: ft_word_dict[x])  # sorted by frequency"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['by',\n",
       " 'as',\n",
       " 'for',\n",
       " 's',\n",
       " 'on',\n",
       " 'was',\n",
       " 'is',\n",
       " 'a',\n",
       " 'to',\n",
       " '(',\n",
       " ')',\n",
       " \"'\",\n",
       " 'and',\n",
       " 'in',\n",
       " '-',\n",
       " 'of',\n",
       " '</s>',\n",
       " 'the',\n",
       " '.',\n",
       " ',']"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ft_words[-20:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2519370"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(ft_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dimensionality of the word vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "dim_en_vec = len(en_vec_dict['and'])\n",
    "dim_fr_vec = len(fr_vec_dict['and'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(300, 300)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dim_en_vec, dim_fr_vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Mean and stdv of word vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "en_vecs = np.stack(list(en_vec_dict.values()))\n",
    "# keys are words, values are the vectors of size 300"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2519370, 300)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "en_vecs.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.0075652334, 0.29283327)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "en_vecs.mean(), en_vecs.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model data object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`en_indcs` are the numericalized questions.\n",
    "\n",
    "**Percentile:**\n",
    "*Ist beispielsweise eine Stichprobe von Schuhgrößen gegeben, so ist das empirische 0,35-Quantil diejenige Schuhgröße s , so dass 35 % der Schuhgrößen in der Stichprobe kleiner als s  sind und 65 % größer als s sind.* [source](https://de.wikipedia.org/wiki/Empirisches_Quantil)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "en_len_99 = int(np.percentile([len(o) for o in en_indcs], 99))\n",
    "fr_len_97 = int(np.percentile([len(o) for o in fr_indcs], 97))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(29, 33)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "en_len_99, fr_len_97"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Truncate the questions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "en_indcs_trunc = np.array([q[:en_len_99] for q in en_indcs])\n",
    "fr_indcs_trunc = np.array([q[:fr_len_97] for q in fr_indcs])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A `Dataset` object needs a `__getitem__` and a `__len__` method. This example is actually very general and can be used for any arrays..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Seq2SeqDataset(Dataset):\n",
    "    def __init__(self, x, y):\n",
    "        self.x, self.y = x, y\n",
    "    def __getitem__(self, idx):\n",
    "        return A(self.x[idx], self.y[idx])\n",
    "    def __len__(self):\n",
    "        return len(self.x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "??A()  # returns a np.array if len == 1 else returns a list of np.arrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Split into train and validation set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "trn_keep = np.random.rand(len(en_indcs_trunc)) > 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "en_trn, fr_trn = en_indcs_trunc[trn_keep], fr_indcs_trunc[trn_keep]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "en_val, fr_val = en_indcs_trunc[~trn_keep], fr_indcs_trunc[~trn_keep]  # tilde negates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(45218, 5039)"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(en_trn), len(en_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create datasets for french to english translation. Swap arguments to create a english to french model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "trn_ds = Seq2SeqDataset(fr_trn, en_trn)\n",
    "val_ds = Seq2SeqDataset(fr_val, en_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'que ferions - nous sans elle ? _eos_'"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "' '.join([fr_itos[o] for o in fr_trn[0]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'what would we do without it ? _eos_'"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "' '.join([en_itos[o] for o in en_trn[0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Everything still looking as expected :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs = 125"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we want to fully utilize the GPUs capabilities, we train in batches. The length of a minibatch tensor is set by the sequence length of the longest question in that batch. The other questions are padded. To save time and memory, we want to avoid very long and very short questions in one batch because that would mean lot's of padding. For the validation set we simply sort the questions. For training we use the `SortishSampler` which groups *longer* questions together and *shorter* questions together while preserving some randomness.\n",
    "\n",
    "For language models it's better to pad before the start of the sequence because we need the final hidden state to predict the next token or for classification...\n",
    "\n",
    "For sequence to sequence models it is better to pad after the end of the sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "trn_sampler = SortishSampler(en_trn, key=lambda x: len(en_trn[x]), bs=bs)\n",
    "val_sampler = SortSampler(en_val, key=lambda x:len(en_val[x]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both samplers simply return indices:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "i = next(iter(trn_sampler))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'why do birds live near the ocean ? _eos_'"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "' '.join([en_itos[o] for o in en_trn[i]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'pourquoi certains oiseaux vivent - ils à proximité de la mer ? _eos_'"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "' '.join([fr_itos[o] for o in fr_trn[i]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "trn_dl = DataLoader(trn_ds, bs, transpose=True, transpose_y=True, num_workers=1, pad_idx=1, pre_pad=False, sampler=trn_sampler)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "val_dl = DataLoader(val_ds, int(1.5*bs), transpose=True, transpose_y=True, num_workers=1, pad_idx=1, pre_pad=False, sampler=val_sampler)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reminder: the ModelData object combines the training and validation dataloaders and a path to story temp stuff. When you have a ModelData object you can create a learner and then call `fit`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "modeldata = ModelData(PATH, trn_dl, val_dl)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Let's look at an example batch:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "it = iter(trn_dl)\n",
    "its = [next(it) for i in range(3)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(32, 18), (25, 8), (21, 9)]"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[(len(x), len(y)) for x,y in its]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "en ce qui concerne l’ adhésion des 10 nouveaux états membres , quelle est la proposition correcte ? _eos_ _pad_ _pad_ _pad_ _pad_ _pad_ _pad_ _pad_ _pad_ _pad_ _pad_ _pad_ _pad_ _pad_\n",
      "which of the following is correct as regards the accession of the 10 new member states ? _eos_\n",
      "\n",
      "quels stimulateurs de croissance sont autorisés au canada ? _eos_ _pad_ _pad_ _pad_ _pad_ _pad_ _pad_ _pad_ _pad_ _pad_ _pad_ _pad_ _pad_ _pad_ _pad_ _pad_\n",
      "which products are approved in canada ? _eos_\n",
      "\n",
      "qui a -t -on atteint avec le programme ? _eos_ _pad_ _pad_ _pad_ _pad_ _pad_ _pad_ _pad_ _pad_ _pad_ _pad_ _pad_\n",
      "who has been reached by the program ? _eos_\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for x, y in its:\n",
    "    print(' '.join([fr_itos[o] for o in x[:,0]]))\n",
    "    print(' '.join([en_itos[o] for o in y[:,0]]))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## First model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_hid, n_layers = 256, 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_emb(wordvecs, itos, em_sz):\n",
    "    emb = nn.Embedding(len(itos), em_sz, padding_idx=1)\n",
    "    weights = emb.weight.data\n",
    "    not_found = []\n",
    "    for idx, word in enumerate(itos):\n",
    "        try:\n",
    "            weights[idx] = torch.from_numpy(wordvecs[word] * 3)  # wordvecs is a dict\n",
    "            # wordvecs have stdv of ~0.3, embedding of 1, the found and not found ones should have same stdv\n",
    "        except:\n",
    "            not_found.append(word)\n",
    "    print(len(not_found), not_found[5:10])\n",
    "    return emb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Seq2SeqRNN(nn.Module):\n",
    "    def __init__(self, vecs_enc, itos_enc, emb_sz_enc, vecs_dec, itos_dec, emb_sz_dec, nh, out_seqlen, nl=2):\n",
    "        super().__init__()\n",
    "        self.nh, self.nl, self.out_seqlen = nh, nl, out_seqlen\n",
    "        \n",
    "        # Encoder\n",
    "        self.emb_enc = create_emb(vecs_enc, itos_enc, emb_sz_enc)\n",
    "        self.emb_enc_drop = nn.Dropout(0.15)\n",
    "        self.rnn_enc = nn.GRU(emb_sz_enc, nh, num_layers=nl, dropout=0.25)  # sz 300 in, sz 256 out\n",
    "        self.outp_enc = nn.Linear(nh, emb_sz_dec, bias=False)               # sz 256 in, sz 300 out\n",
    "        \n",
    "        # Decoder\n",
    "        self.emb_dec = create_emb(vecs_dec, itos_dec, emb_sz_dec)\n",
    "        self.rnn_dec = nn.GRU(emb_sz_dec, emb_sz_dec, num_layers=nl, dropout=0.1)  # square to enable weight tying\n",
    "        self.out_drop = nn.Dropout(0.35)                                           # sz 300 in, 300 out\n",
    "        self.out = nn.Linear(emb_sz_dec, len(itos_dec))\n",
    "        self.out.weight.data = self.emb_dec.weight.data                            # weight tying! not transposed?\n",
    "        \n",
    "    def forward(self, inp):\n",
    "        seqlen, bs = inp.size()\n",
    "        \n",
    "        # Encoder\n",
    "        h = self.initHidden(bs)\n",
    "        emb = self.emb_enc_drop(self.emb_enc(inp))\n",
    "        enc_outp, h = self.rnn_enc(emb, h)  # h[1] is same as enc_outp[-1]!\n",
    "        # pdb.set_trace()\n",
    "        h = self.outp_enc(h)\n",
    "\n",
    "        # Decoder\n",
    "        dec_inp = V(torch.zeros(bs).long())  # 0 is 'beginning of seq' token\n",
    "        res = []\n",
    "        for i in range(self.out_seqlen):\n",
    "            emb = self.emb_dec(dec_inp).unsqueeze(0)  # adds unit axis at beginning so that rnn 'loops' once\n",
    "            outp, h = self.rnn_dec(emb, h)  # 'loops' once\n",
    "            outp = self.out(self.out_drop(outp[0]))  # 0 because 'loops' once, predictions for words\n",
    "            res.append(outp)\n",
    "            dec_inp = V(outp.data.max(1)[1])  # [1] to get argmax\n",
    "            if (dec_inp == 1).all(): break    # if all sentences in the minibatch have _pad_ as input, break\n",
    "        return torch.stack(res)\n",
    "        \n",
    "    def initHidden(self, bs):\n",
    "        return V(torch.zeros(self.nl, bs, self.nh))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Decoder:** `outp`is the same as `h[1]` (has two layers)\n",
    "```\n",
    "p outp.shape\n",
    "torch.Size([1, 187, 300])\n",
    "\n",
    "p h.shape\n",
    "torch.Size([2, 187, 300])\n",
    "```\n",
    "\n",
    "**Encoder:**\n",
    "```\n",
    "p enc_outp.shape\n",
    "torch.Size([33, 187, 256])\n",
    "\n",
    "p h.shape\n",
    "torch.Size([2, 187, 256])\n",
    "```\n",
    "\n",
    "`enc_outp[-1]` and `h[1]` are the same\n",
    "\n",
    "(187 is bs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loss function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Seq2SeqLoss(pred, target):\n",
    "    seqlen, bs = target.size()\n",
    "    seqlen_pred, bs_pred, n_probs = pred.size()\n",
    "    \n",
    "    # we need to pad if target seqlen is larger than prediction seqlen\n",
    "    if seqlen > seqlen_pred:\n",
    "        pred = F.pad(pred, (0,0,0,0,0,seqlen-seqlen_pred))\n",
    "    \n",
    "    # but we only compare until the seqlen of the target\n",
    "    pred = pred[:seqlen]\n",
    "    \n",
    "    # cross_entropy can't handle rank 3 tensors currently, we need to flatten\n",
    "    return F.cross_entropy(pred.view(-1, n_probs), target.view(-1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "opt_fn = partial(optim.Adam, betas=(0.8, 0.99))  # like bind in c++"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(300, 300)"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dim_fr_vec, dim_en_vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3097 ['l’', \"d'\", 't_up', 'd’', \"qu'\"]\n",
      "1285 [\"'s\", '’s', \"n't\", 'n’t', ':']\n"
     ]
    }
   ],
   "source": [
    "rnn = Seq2SeqRNN(fr_vec_dict, fr_itos, dim_fr_vec, en_vec_dict, en_itos, dim_en_vec, n_hid, en_len_99)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### It's kind of understandable that these tokens where not found in the word vectors we downloaded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = Learner(modeldata, SingleModel(to_gpu(rnn)), opt_fn=opt_fn)  # SingleModel makes the model one layer group"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.crit = Seq2SeqLoss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e7ee66c85b9c498d89b575fa9b128563",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 67%|██████▋   | 241/362 [00:23<00:12, 10.06it/s, loss=37.1]"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY8AAAEOCAYAAABxdpuaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VPW9//HXZ7JBQhISEgKEhLDvCIgoggquSG2t1lZta13aahe73dve1vb+bnftrbW93axatdZWra0rKhWQKotsArIa9jVhSSAsgbAl+fz+mEEjnUCCmZxM8n4+HvPIOd9zzsxnvgzzme/5nvP9mrsjIiLSGKGgAxARkfij5CEiIo2m5CEiIo2m5CEiIo2m5CEiIo2m5CEiIo2m5CEiIo2m5CEiIo2m5CEiIo2m5CEiIo2WGHQATSknJ8eLioqCDkNEJG4sXrx4t7vnNva4VpU8ioqKWLRoUdBhiIjEDTPbcibH6bSViIg0mpKHiIg0mpKHiIg0mpKHiIg0WsySh5kVmNnrZlZsZqvM7GuR8h+YWamZLY08JtVz/EQzW2Nm683sO7GKU0REGi+WV1tVA//p7kvMLB1YbGbTI9t+5e6/qO9AM0sAfg9cBpQAb5nZZHd/J4bxiohIA8Ws5eHuO9x9SWS5EigG8ht4+GhgvbtvdPdjwN+Aq2MTqYhI/FpZup/5G/dQW9u8U4o3S5+HmRUBI4AFkaI7zWy5mT1qZllRDskHttVZL6HhiUdEpM14dM4m7nxyCWbN+7oxTx5m1gF4Fvi6ux8A/gD0BoYDO4D7oh0WpSxqWjWz281skZktKi8vb6KoRUTiw4JNFYzumY01c/aIafIwsyTCieMJd38OwN13uXuNu9cCfyR8iupkJUBBnfXuwPZor+HuD7n7KHcflZvb6DvsRUTiVsneKkr3HWZ0UXazv3Ysr7Yy4BGg2N1/Wae8a53drgFWRjn8LaCvmfU0s2TgBmByrGIVEYlHb22uAGB0z07N/tqxvNpqLHATsMLMlkbKvgvcaGbDCZ+G2gzcAWBm3YCH3X2Su1eb2Z3AVCABeNTdV8UwVhGRuLNgYwUZ7RLp3yW92V87ZsnD3ecQve9iSj37bwcm1VmfUt++IiICCzdVcE5RNgmhZu4tR3eYi4jEpX1Vx9i4+xBnF0W7YDX2lDxEROLQ8pL9AJzVvWMgr6/kISISh1aUhpPHkPzMQF5fyUNEJA4t27aPnjlpZLZPCuT1lTxEROLQitL9DA2o1QFKHiIicaes8gg79h9hWHclDxERaaAlW/YBMCygznJQ8hARiTvT3tlJZvskRhQqeYiISAMcq65l+ju7uGxQHkkJwX2FK3mIiMSRNzfspvJINZOGdgk0DiUPEZE48s8VO0hPSWRsn5xA41DyEBGJE8drapn2zi4uHZRHSmJCoLEoeYiIxIl5G/awr+o4Vw4J9pQVKHmIiMSNf67cQVpyAhf2C37iOyUPEZE4UF1Ty9RVu7h4YB7tkoI9ZQVKHiIicWHhpgoqDh1jUgs4ZQVKHiIicWHKyh20T0pgfP/OQYcCKHmIiLR4NbXOqyt3MWFALu2Tgz9lBUoeIiIt3qLNFew+eJQrh3QNOpR3xSx5mFmBmb1uZsVmtsrMvhYpv9fMVpvZcjN73syiDs5iZpvNbIWZLTWzRbGKU0SkpZu9bjcJIWPCgJZxygpi2/KoBv7T3QcC5wFfNrNBwHRgiLsPA9YCd53iOSa4+3B3HxXDOEVEWrSFmysY0i2DDimJQYfyrpglD3ff4e5LIsuVQDGQ7+7T3L06stt8oHusYhARiXdHq2tYum0f5xRlBx3K+zRLn4eZFQEjgAUnbboN+Gc9hzkwzcwWm9ntsYtORKTlWl6yn2PVtYzu2bKSR8zbQGbWAXgW+Lq7H6hT/j3Cp7aeqOfQse6+3cw6A9PNbLW7z4ry/LcDtwMUFhY2efwiIkFauKkCoG21PMwsiXDieMLdn6tTfjNwFfApd/dox7r79sjfMuB5YHQ9+z3k7qPcfVRubvC37IuINKX5G/fQL68DWWnJQYfyPrG82sqAR4Bid/9lnfKJwLeBj7h7VT3HpplZ+oll4HJgZaxiFRFpifYfPs78jXuY0EJuDKwrli2PscBNwMWRy22Xmtkk4HdAOuFTUUvN7AEAM+tmZlMix+YBc8xsGbAQeMXdX41hrCIiLc7rq8s4XuNc0UKGJKkrZn0e7j4HsCibpkQpO3GaalJkeSNwVqxiExGJB6+u3EleRgrDuwc3V3l9dIe5iEgLVHWsmjfWlnHF4C6EQtF+hwdLyUNEpAWatbacI8drmTi45Z2yAiUPEZEW6dWVO8lKTWpx93ecoOQhItLCHKuuZUZxGZcNyiMxoWV+TbfMqERE2rC5G3ZTebSaiS3wKqsTlDxERFqYpxZupWNqEuf3zgk6lHopeYiItCCbdh9i2ju7uOm8Hi1irvL6KHmIiLQgj8zZSFIoxGfGFAUdyikpeYiItBDVNbVMXrqdq4Z1JTc9JehwTknJQ0SkhViydR8HjlRz2aC8oEM5LSUPEZEW4vU1ZSSGjLF9W25H+QlKHiIiLcTrq8sYVZRFRrukoEM5LSUPEZEWoHTfYVbvrGyRw69Ho+QhItICPLOoBIArh3QNOJKGUfIQEQlYTa3z90XbGNcnh8JOqUGH0yBKHiIiAZu1rpzSfYe5cXRh0KE0mJKHiEjAHp+7mZwOyXFxie4JSh4iIgFat6uS19eU85kxRSQnxs9XcvxEKiLSCj08exPtkkJ8+rweQYfSKDFLHmZWYGavm1mxma0ys69FyrPNbLqZrYv8zarn+Jsj+6wzs5tjFaeISFC27qniubdL+MSoArLTkoMOp1Fi2fKoBv7T3QcC5wFfNrNBwHeAGe7eF5gRWX8fM8sGvg+cC4wGvl9fkhERiVe/mLaGxFCIOyf0CTqURotZ8nD3He6+JLJcCRQD+cDVwJ8ju/0Z+GiUw68Aprt7hbvvBaYDE2MVq4hIc1uzs5LJy7bz2XE96ZzRLuhwGq1Z+jzMrAgYASwA8tx9B4QTDBDtdsp8YFud9ZJImYhIq/DC0lISQsZt43oGHcoZiXnyMLMOwLPA1939QEMPi1Lm9Tz/7Wa2yMwWlZeXn2mYIiLNxt2ZsmIH5/fuFHd9HSfENHmYWRLhxPGEuz8XKd5lZl0j27sCZVEOLQEK6qx3B7ZHew13f8jdR7n7qNzc3KYLXkQkRlZtP8CWPVVMGhofQ5FEE8urrQx4BCh291/W2TQZOHH11M3Ai1EOnwpcbmZZkY7yyyNlIiJxb/Ky7SSEjCsGdwk6lDMWy5bHWOAm4GIzWxp5TAJ+BlxmZuuAyyLrmNkoM3sYwN0rgB8Db0UeP4qUiYjEtW0VVfx57mY+NLRr3J6yAkiM1RO7+xyi910AXBJl/0XA5+qsPwo8GpvoRESan7vzk1feIWTGXZMGBB3OB6I7zEVEmsmf3tzM1FW7+Oolfema2T7ocD4QJQ8RkWawZOtefvLKO1w+KI87LuwVdDgfmJKHiEgz+P2/1tMxNZlfXj+cUKi+M/rxQ8lDRCTG1u2qZMbqMj4zpgcdUmLW1dyslDxERGLs/jc20C4pxGfGFAUdSpNR8hARiaG5G3bz/Nul3Dq2Z1xfmnsyJQ8RkRg5Wl3D955fSY9OqXztkr5Bh9OkWsfJNxGRFujJBVvZtPsQf7r1HNolJQQdTpNSy0NEJAYqjxznt/9az/m9OzG+X+sbd0/JQ0QkBh57czMVh47xnSsHEB7qr3VR8hARaWJVx6p59M1NXDKgM8O6dww6nJhQ8hARaWJ/W7iNvVXH+dKE3kGHEjNKHiIiTajswBF++691nNszm7N7ZAcdTswoeYiINBF351vPLOfw8RruvnZo0OHElJKHiEgT+cv8LcxcW853Jw2kd26HoMOJKSUPEZEmsKH8IHdPKeaifrncdF6PoMOJOSUPEZEmcM+U1SQlhLj3umGt8tLckyl5iIh8QCtL9/Na8S4+f0EvOme0CzqcZqHkISLyAbg7905dQ2b7JG4ZWxR0OM0mZmNbmdmjwFVAmbsPiZQ9DfSP7NIR2Ofuw6McuxmoBGqAancfFas4RUQ+iAdnbWTm2nL+56pBZLRLCjqcZhPLgREfA34HPH6iwN2vP7FsZvcB+09x/AR33x2z6EREPqCFmyr4+aur+dCwrtzahlodEMPk4e6zzKwo2jYL9yZ9Arg4Vq8vIhJLB44c5xtPL6UgO5Wff6xtdJLXFVSfxwXALndfV892B6aZ2WIzu70Z4xIRaZBfTlvLzgNH+NX1w0lrJVPLNkZQ7/hG4KlTbB/r7tvNrDMw3cxWu/usaDtGksvtAIWFhU0fqYjIScorj/LUwq18bGQ+Iwuzgg4nEM3e8jCzROBa4On69nH37ZG/ZcDzwOhT7PuQu49y91G5ua1vzHwRaXkem7uJYzW13HFR6x348HSCOG11KbDa3UuibTSzNDNLP7EMXA6sbMb4RETqVVZ5hD/P3cKVQ7q0+iFITiVmycPMngLmAf3NrMTMPhvZdAMnnbIys25mNiWymgfMMbNlwELgFXd/NVZxiog0xt2vFHOsupZvXTEg6FACFcurrW6sp/yWKGXbgUmR5Y3AWbGKS0TkTC3aXMELS7fzlYv70DMnLehwAqU7zEVEGui+aWvJ6ZDCl8b3CTqUwCl5iIg0wNwNu5m3cQ9fntCb9skJQYcTuAYlDzP7mpllWNgjZrbEzC6PdXAiIi3Fb2esJy8jhRtH65YAaHjL4zZ3P0D4yqdc4FbgZzGLSkSkBVleso95G/fwuXG9aJekVgc0PHmcuO9+EvAnd19Wp0xEpFV7cNZG0tslcsPogqBDaTEamjwWm9k0wsljauQ+jNrYhSUi0jJs2XOIf67YwafP60F6Gxo193QaeqnuZ4HhwEZ3rzKzbMKnrkREWrWHZ28iMRTi1vOLgg6lRWloy2MMsMbd95nZp4H/5tTDqYuIxL09B4/y90XbuGZEfpuZIbChGpo8/gBUmdlZwH8BW6gzT4eISGv053lbOFpdy+cv7BV0KC1OQ5NHtbs7cDXwa3f/NZAeu7BERIJVdayax+dt5rJBefTp3HbHsKpPQ/s8Ks3sLuAm4AIzSwDUcyQirdbf39rGvqrjfOEitTqiaWjL43rgKOH7PXYC+cC9MYtKRCRA1TW1PDxnE2f3yOLsHtlBh9MiNSh5RBLGE0CmmV0FHHF39XmISKs0ZeVOSvYe5g71ddSrocOTfILw8OgfJzz3+AIzuy6WgYmIBMHdeXDmBnrlpnHpwLygw2mxGtrn8T3gnMjMfphZLvAa8EysAhMRCcLSbftYtf0AP71mCKGQBtKoT0P7PEInEkfEnkYcKyISN154u5SUxBAfPqtb0KG0aA1tebxqZlN5bwbA64Epp9hfRCTuHK+p5aXlO7h0UB4ZGorklBqUPNz9W2b2MWAs4QERH3L352MamYhIM5u1tpyKQ8e4Znh+0KG0eA2ehtbdnwWejWEsIiKBeq14F+kpiVzYLzfoUFq8U/ZbmFmlmR2I8qg0swOnOfZRMyszs5V1yn5gZqVmtjTymFTPsRPNbI2ZrTez75zZWxMRaTh3Z+aacsb2ySE5UV26p3PKGnL3dHfPiPJId/eM0zz3Y8DEKOW/cvfhkce/9ZtE7l7/PXAlMAi40cwGNeztiIicmfVlB9m+/wgX9VeroyFill7dfRZQcQaHjgbWu/tGdz8G/I3wmFoiIjEzc205gE5ZNVAQbbM7zWx55LRWVpTt+cC2OuslkTIRkZiZtW43fTt3IL9j+6BDiQvNnTz+APQmPLHUDuC+KPtEuyvH63tCM7vdzBaZ2aLy8vKmiVJE2hR3Z3nJPkYVRfs9K9E0a/Jw913uXuPutcAfCZ+iOlkJUHei4O7A9lM850PuPsrdR+XmqrkpIo2388AR9lUdZ2DX03XlygnNmjzMrGud1WuAlVF2ewvoa2Y9zSwZuAGY3BzxiUjbVLwjfPGokkfDNfg+j8Yys6eA8UCOmZUA3wfGm9lwwqehNgN3RPbtBjzs7pPcvdrM7gSmAgnAo+6+KlZxioi8sz2cPAZ00Rx3DRWz5OHuN0YpfqSefbcDk+qsT0HDn4hIMyneUUlhdirpGpKkwXQnjIi0ecU7DjCwq1odjaHkISJtWtWxajbtOaT+jkZS8hCRNu3trftwh2HdM4MOJa4oeYhIm/Za8S5SEkOM6ZUTdChxRclDRNosd2dGcRnn9+5E++SEoMOJK0oeItJmbSg/xNaKKi7WXOWNpuQhIm3WjOJdAFw8oHPAkcQfJQ8RabNeWbGDofmZGgzxDCh5AAeOHA86BBFpZlv2HGJ5yX6uGtb19DvLv2nzyaPyyHE+8ts5/PcLKzhyvCbocESkmby8fAcAH1LyOCNtPnm0S0rgisFd+Ov8rUz69Wz+OGujWiIirZy789Ky7Yws7Ej3rNSgw4lLbT55JCWEuGvSQP50yzlkpibx0ynFXPC/r3P/G+s5dLQ66PBEJAbmb6xg9c5KPj6q4PQ7S1RtPnmcMGFAZ57/0lheunMcIws78vNX13DRvW/w3JISjtfUBh2eiDShR9/cRHZaMteM0CSlZ0rJ4yRDu2fyp1tH8+wXzyc/qz3/8fdljPjRdD7357d4csFWjlarX0Qknm3dU8Vrxbv41LmFtEvSjYFnKmZDsse7s3tk8fwXz2d68S5mrS1nzvrdvFZcxv1vrOdbV/Tnw8O6EQpFmzFXRFqypxdtxYBPnlsYdChxTcnjFEIh44rBXbhicBcA3ly/m7unFPO1vy3lvmlruXhAZy7qn8uYXp30C0YkDtTUOs8sLuGifrl0zdS9HR+EkkcjjO2Tw0t3juOl5dt5cel2/vbWVh6bu5mUxBAjC7MY0DWdAV3SGVmYRZ/OHTBTy0SkJZm1tpxdB47yw4+oo/yDUvJopFDIuHp4PlcPz+fI8RoWbKrgjTVlLNmyl78t3MbhyL0ieRkpjOuTy1XDunJB3xwSE9S9JBK0l5ZtJys1iYsHaCyrD0rJ4wNol5TARf1yuahfLgC1tc6WiioWbNwT6SPZxbNLSsjpkMzEIV0YUZDFhAGdyU5LDjhykbbH3Zmzfjdj++SQnKgfcx9UzJKHmT0KXAWUufuQSNm9wIeBY8AG4FZ33xfl2M1AJVADVLv7qFjF2ZRCIaNnTho9c9K4YXQhx6preWNNGc8tKeXZxaX8df5WkhKMiwd05rqzCxjfP5cktUhEmsWG8oOUVR5lbB/N29EUYtnyeAz4HfB4nbLpwF3uXm1m/wvcBXy7nuMnuPvuGMYXc8mJIS4f3IXLB3ehptYp3nGAF5eW8vzbpUxdtYtOacnccn4Rt43rSVqKGoEisfTm+j0AjFPyaBIx+8Zy91lmVnRS2bQ6q/OB62L1+i1NQsgYkp/JkPxM/mviAGatLeeJBVu5b/paHpq9kauHd+P83jn0y0une1Z7Xb0l0sTmrN9NQXZ7CrI1HElTCPLn7m3A0/Vsc2CamTnwoLs/1HxhxV5SQohLBuZxycA8lm7bx5/e3MQ/FpXw1/lb392nS0Y7CjulckGfHG48t5CcDikBRiwS32prnQUb93DlEA2C2FQCSR5m9j2gGniinl3Guvt2M+sMTDez1e4+q57nuh24HaCwMP5u+hle0JFf3zCCY9W1FO84wMbdB9m65zBbK6rYUH6Q+6av5f9mrOP83p34xKgCrhjcRZ19Io20paKKA0eqGVHYMehQWo1mTx5mdjPhjvRL3N2j7ePu2yN/y8zseWA0EDV5RFolDwGMGjUq6vPFg+TEEGcVdOSsgvd/uNeXHeS5JSVMXradrzz1Njkdkvno8HxG9shiSLdMCrLb634SkdNYUbofgCH5mQFH0no0a/Iws4mEO8gvcveqevZJA0LuXhlZvhz4UTOG2aL06dyB/5o4gG9e3p+Z68p5Yv4WHpu7mYfnbAKgZ04an7+gF9eOzFc/iUg9VpbuJzkhRL+89KBDaTVieanuU8B4IMfMSoDvE766KoXwqSiA+e7+BTPrBjzs7pOAPOD5yPZE4El3fzVWccaLUMiY0L8zE/p35sjxGtbuqmR5yX6efmsb331+Bb96bS0fOasb4/rmMDQ/U30kInWsKNnPgK7pOuXbhKyeM0dxadSoUb5o0aKgw2hW7s7cDXt4dM4mZq/bzbHI8PFdM9sxuFsmI3t05MohXemZkxZwpCLBqK11zvrhND48vBt3XzM06HBaHDNbfCb30unmgjhnZoztk8PYPjkcOlrN8pL9rNq+nxWl4cdrxbv4+atrGNg1g0+OLuD6cwr160valC0VVVQerWaY+jualJJHK5KWksiY3p0Y07vTu2Wl+w7z6sqdTF5ayv97cRUPzNzI1y7py8UDO+vUlrQJS7ftBcJz9UjTUfJo5fI7tuez43py29giZq3bzS+mruG/nl0OQGF2KpcM7MzHzy5gULeMgCMViY256/eQ2T6JAV30GW9KSh5thJlxUb9cLuybw8JNFSwv2c+8jXt4YsFW/vTmZi4blMe3Jw6gT+cOQYcq0mRO9AmO6dWJBE3e1qSUPNoYM+PcXp04t1cnPn9hL/ZVHeOv87fw+9c3MP2dmQzoks4Vg7vw6fN6kJuu01oS37ZVHKZ032HuuKhX0KG0Ouo5beM6piZz58V9mfmt8fzPVYPIaJfEb/61jiv+bxaTl22nOnL1lkg8mrshPLbq+XX6AaVpKHkIAJ0z2nHbuJ78/QtjmP6NC+ma2Y6vPvU2F937Bve/sZ49B48GHaJIo81et5vO6Sn0ztXp2Kam01byb/p0TufFL49lxuoyHp+3mZ+/uob/e20dVwzuwqUDO3NB31xNaCUtXtWxav61uoxrR+ZrCJ8YUPKQqBITQlwxuAtXDO7Cul2VPD5vC6+s2MFLy7ZjBmd178j4/rlM6N+ZYd0z9Z9TWpx/rS7j8PEarhrWLehQWiXdYS4NVlPrrCjdzxtrynh9TTnLS/bhDucUZXHXpIGMLMwKOkSRd33hL4tZvHUv8++6RFdanYLuMJeYSwgZwws6MrygI1+/tB97Dh7llRU7+M2M9Vx7/1wuHZjHxCFdmDS0C6nJ+mhJcI4cr+H1NWXccE6BEkeMqMNczlinDil8ZkwRM781nq9f2pel2/byzX8s4/JfzWL2uvKgw5M2bPXOSo5W13JeL11lFStKHvKBpaUk8vVL+7Hwu5fy5OfOJSUxxGceXcjv/rWOmtrWc1pU4seJ+Ts0JEnsKHlIkwmFjPP75PDyVy7gw8O68Ytpa7nqt3NYvGVv0KFJG7OiZB9ZqUnkd2wfdCitlpKHNLn2yQn8+obh/P6TI9lfdYyPPzCXe6YUc+R4TdChSRuxovQAQ7t31FWAMaTkITFhZnxoWFemfuNCrj+nkAdnbeRDv5nN21vVCpHYOjFZ2tB8DYQYS0oeElPp7ZK459qhPH7baA4fq+Fjf5jLnU8u4fU1ZUGHJq3UOzsOUFPrDM3vGHQorZqShzSLC/vl8uo3LuTm84uYt2EPt/7pLX72z9XqUJcmV7zjAACDNc1ATCl5SLPJaJfE9z88mPnfvYRPn1fIAzM3cMND83hrc4X6Q6TJbK2oIinB6KbO8piKafIws0fNrMzMVtYpyzaz6Wa2LvI36m3JZnZzZJ91ZnZzLOOU5pWUEOInHx3Kr64/i9U7K/n4A/MY8aPpfOPppSxRn4h8QCUVh+melaqbA2Ms1rcBPwb8Dni8Ttl3gBnu/jMz+05k/dt1DzKzbOD7wCjAgcVmNtnd9c3SilwzojsT+ndmwaYK3lhTxsvLd/D826UMzc/kYyPzuWlMkb4ApNG2VlTRPUutjliLacvD3WcBFScVXw38ObL8Z+CjUQ69Apju7hWRhDEdmBizQCUwHVOTuWJwF+65dhjz7rqE7394EI7zg5fe4ZY/LaTi0LGgQ5Q4s7WiisLs1KDDaPWC6PPIc/cdAJG/naPskw9sq7NeEimTVqxDSiK3ju3Jy1+5gJ9dO5T5G/dw6S9n8vi8zZpPRBpk/+Hj7D98XMmjGbTUDvNo5yqiXpZjZreb2SIzW1RervGUWosbRhfy0lfG0aNTKv/z4irOvXsG3/rHMkr2VgUdmrRg2yrCn48CJY+YCyJ57DKzrgCRv9Eu+C8BCuqsdwe2R3syd3/I3Ue5+6jc3NwmD1aCM6BLBs998Xxe+eo4PnVuIZOXbeeyX87iwZkbdHWWRHXix4VaHrEXRPKYDJy4eupm4MUo+0wFLjezrMjVWJdHyqSNMTMGd8vkh1cP4V/fHM/5vTtxzz9Xc9G9r/OXeZs5Vq051uU9W0+0PLKUPGIt1pfqPgXMA/qbWYmZfRb4GXCZma0DLousY2ajzOxhAHevAH4MvBV5/ChSJm1Yfsf2PHLLOTz1+fMozE7l/724ist+NZOpq3bSmiY1kzO3reIwGe0SyUxNCjqUVk8zCUpccndmri3n7inFrN11kPN6ZfONS/sxume2BsNrwz7+wFwOH6/h5a9cEHQoceNMZxJsqR3mIqdkZozv35kpX72AH189mLW7DnL9Q/P56O/f5JXlOzTsSRu0YOMe3tq8lw8N1ZzlzUFzhUpcS0wIcdOYIq47u4BnlpTwyOyNfPnJJRRmp/Lp8wr52MjudOqQEnSYEmPuzs+nriEvI4Vbzi8KOpw2QS0PaRXaJydw03k9mPGf43ng0yPpnJ7C3VNWc949M7jzySXMXb9b/SKt2Nvb9rF4y17uvLgv7ZMTgg6nTVDLQ1qVhJAxcUhXJg7pytpdlTy1cCvPLSnl5eU7GJqfyaiiLDaUH+LaEflcObQLKYn6omkNnl9SSkpiiI8O1ymr5qIOc2n1jhyvYfLS7fzmX+soO3CU3PQUSvcdJjFkDOqWwWUD8xjZI4vhBR1JS9HvqXhzrLqW0Xe/xgV9c/ntjSOCDifunGmHuf6nSKvXLimBT5xTwHVnd+d4bS1JoRAz15bz1uYK5m3cw33T1wKQmpzApQPzGN0zm7N7ZNEvL10DM8aBmWvL2Vd1nGtHaASj5qTkIW1GKGSkhMKnqSYM6MyEAeFh1SoOHWNl6X6mrNjBa8VlTF4WHsygQ0oi5xRl8fHeswWTAAAPSElEQVRRBVw2KI+kBHURtkTTVu0ks30S4/rmBB1Km6LkIW1edloyF/bL5cJ+ubg7JXsPs2hLBYu37OX11eV86Ykl5KancNN5PfjsuJ46tdWC1NY6b6wt58J+uUruzUz/C0TqMDMKslMpyE7lmhHdqal1Zq4t4y/ztvDL6Wt5cOYGOnVIoXduGiMKsxhR2JGzCjqS0U53NAdh1fYDlFceZUJ/jWvX3JQ8RE4hIWRcPCCPiwfk8fbWvTz/din7qo6zeucB3lhbjjuYwbDuHblqaFeuO7s7WWnJQYfdZry+pgwzuLCfkkdzU/IQaaBwS+O9WZMPHDnO8m37WbSlghnFZfx0SjH3/LOYpIQQPXPSmDikCx8fVUC+5tKOmZlryxmWn0mObgRtdkoeImcoo124k3Zc3xy+fmk/Vu88wJTlOzhSXcvSbfv49Yx1/HrGOnrlpHHxgM7cfH4R7rCh/CAHjlSTlZpEr9wOdMtsp/G4zsCx6lpWlO7n5jE9gg6lTVLyEGkiA7pkMKBLxrvr2yqqeHFpKUu27uOROZv44+xNUY9LTU6gT+cO9MntQJ+8Dozv15lB3TKi7ivvWburkmPVtQzr3jHoUNokJQ+RGCnITuXOi/sCsLH8IK+vKSctOYGinDSy05KpOHSMDeUHWV8WfszdsIfn3i7l56+uYVDXDLLSkthQdoiUpBAjC7M4q3smyYkJJCeGyO/YnhGFHWmX1HbvkF9Ruh+AYd0zA46kbVLyEGkGvXI70Cu3w7+Vn9er0/vW9x46xt8XbWPO+t1UHqlmTO9OVB2rZs763Tz/dun79m2XFKJXTge6dWxP96z25HdsT7eO7RnQNZ3eUV6rtVleso/M9kmaNTAgSh4iLUhWWjJ3XNSbOy7q/b5yd6e88ihOeLiV9WUHeXP9HjbtPsi2iirmb9zDwaPV7+4/uFsG14zIp1duGimJCQzrnkl6K7uceHnJfoZ1z1R/UUCUPETigJnROaPdu+s9OqVxycC8d9fdnQOHqynZV8XCTRW88HYpP3mluM7x0D8vnbO6d6Rfl3T656UzrCAzbu9POXK8hjU7K7n9wl5Bh9JmKXmItAJmRmZqEpmpmQzulsmtY3uyZc8h9lYdp/LIcRZv2cviLXuZXryLpxdtAyBkMKhbBuf27MS5PbM5t1cnMtvHRzKZumon1bXOqKKs0+8sMaHkIdJK9eiURo9Il8oFfd+7iW73waOs3lHJws0VLNi4h7/M38IjczYRMuielUpiyMBgz8FjHKuupV9eBwZ0yWB0z2wuHtA58Jsgj1bXcO/U8EUF4/t1DjSWtkzJQ6SNyemQwri+Ke8OJHjkeA1Lt+1j7vrdbKmootbDY0ZlpSWRGAqxdlfl+1oshdmpTBzShbN7ZJGWnMjQ/EwyU2PfYqmuqWXh5goemb2Jkr2H+etnhxHSqMeBafbkYWb9gafrFPUC/sfd/6/OPuOBF4ETF8Y/5+4/arYgRdqQdkkJnNer079d+VWXu7O8ZD9z1u9myZa9PDpnEw/N2vjudjPontWeSUO60jE1maQEIzc9hbO6dySjfRLJiSHaJyWc8RD3xTsO8JlHF1JeeZT0don8x2X9NIpuwJo9ebj7GmA4gJklAKXA81F2ne3uVzVnbCISnZlxVkF4EEgID2O/fd9h9h8+zrKSfRw+Fm69/HH2RmrrmV8uNTmBc3tmM7BrBr1zO9CjUyoZ7ZMoyEo95dSx1TW1fOuZZbg7939qJOP755KarJMmQQv6X+ASYIO7bwk4DhFphOy0ZLIjfR9j+7zXAqipdY7X1HKsppZtFVWs2n6Aw8dqOFpdw7aKw8zfuIfZ63ZTXSfDmEFRpzQGdk1nVI9szu2VzYAuGSSEjENHq/nplGJWlh7g958cyaShXZv9vUp0QSePG4Cn6tk2xsyWAduBb7r7qmg7mdntwO0AhYWFMQlSRBomIWQkhBJol5TA4G7hK79Odrymlq0VVZTsDbdcNpQdZM3OSpZt28+UFTsByEpNYnC3TFZu38++quPccn4Rk4Z2ae63I6cQ2BzmZpZMODEMdvddJ23LAGrd/aCZTQJ+7e59T/ecmsNcJL6V7jvMwk3h1sk72w8wuFsmnzy3gLN7ZAcdWqsVj3OYXwksOTlxALj7gTrLU8zsfjPLcffdzRqhiDSr/I7tuWZEd64Z0T3oUOQ0gpy38UbqOWVlZl0sMuaAmY0mHOeeZoxNREROIZCWh5mlApcBd9Qp+wKAuz8AXAd80cyqgcPADR7U+TUREfk3gSQPd68COp1U9kCd5d8Bv2vuuEREpGGCPG0lIiJxSslDREQaTclDREQaTclDREQaTclDREQaLbA7zGPBzMqBQ0CsbybMBPbH+LjT7Vvf9mjlDSk7eT2H1l+Pjd2memyabW21Hs/0/3R925qqHnu4ey6N5e6t6gEsaobXeCjWx51u3/q2RytvSFmU9VZfj43dpnpUPX6Q4870/3RLrUedtjozLzXDcafbt77t0cobUnam7+mDCLoeG7tN9dg029pqPZ7p/+n6tgVaj63qtBWAmS3yMxjkS95P9dg0VI9NQ/XYNJqyHltjy+OhoANoJVSPTUP12DRUj02jyeqx1bU8REQk9lpjy0NERGJMyUNERBpNyUNERBqtTSUPMxtvZrPN7AEzGx90PPHMzNLMbLGZXRV0LPHKzAZGPovPmNkXg44nXpnZR83sj2b2opldHnQ88crMepnZI2b2TEP2j5vkYWaPmlmZma08qXyima0xs/Vm9p3TPI0DB4F2QEmsYm3JmqgeAb4N/D02UbZ8TVGP7l7s7l8APgG0yctQm6geX3D3zwO3ANfHMNwWq4nqcaO7f7bBrxkvV1uZ2YWEv/gfd/chkbIEYC3hWQlLgLcIT2+bANxz0lPcBux291ozywN+6e6faq74W4omqsdhhIc5aEe4Tl9unuhbjqaoR3cvM7OPAN8BfufuTzZX/C1FU9Vj5Lj7gCfcfUkzhd9iNHE9PuPu153uNQOZSfBMuPssMys6qXg0sN7dNwKY2d+Aq939HuBUp1P2AimxiLOla4p6NLMJQBowCDhsZlPcvTamgbcwTfV5dPfJwGQzewVoc8mjiT6PBvwM+GdbTBzQ5N+PDRI3yaMe+cC2OuslwLn17Wxm1wJXAB3RNLd1Naoe3f17AGZ2C5HWXEyjix+N/TyOB64l/ENmSkwjiy+NqkfgK8ClQKaZ9fE6U1q3cY39PHYCfgqMMLO7IkmmXvGePCxKWb3n4dz9OeC52IUTtxpVj+/u4P5Y04cS1xr7eXwDeCNWwcSxxtbjb4DfxC6cuNXYetwDfKGhTx43Heb1KAEK6qx3B7YHFEs8Uz02DdVj01A9No2Y1mO8J4+3gL5m1tPMkoEbgMkBxxSPVI9NQ/XYNFSPTSOm9Rg3ycPMngLmAf3NrMTMPuvu1cCdwFSgGPi7u68KMs6WTvXYNFSPTUP12DSCqMe4uVRXRERajrhpeYiISMuh5CEiIo2m5CEiIo2m5CEiIo2m5CEiIo2m5CEiIo2m5CGBMbODzfAaH2ngEPNN+Zrjzez8MzhuhJk9HFm+xcxaxPhrZlZ08lDfUfbJNbNXmysmCZ6Sh8S9yNDTUbn7ZHf/WQxe81Tjwo0HGp08gO8Cvz2jgALm7uXADjMbG3Qs0jyUPKRFMLNvmdlbZrbczH5Yp/wFC89YuMrMbq9TftDMfmRmC4AxZrbZzH5oZkvMbIWZDYjs9+4veDN7zMx+Y2ZzzWyjmV0XKQ+Z2f2R13jZzKac2HZSjG+Y2d1mNhP4mpl92MwWmNnbZvaameVFhsX+AvANM1tqZhdEfpU/G3l/b0X7gjWzdGCYuy+Lsq2Hmc2I1M0MMyuMlPc2s/mR5/xRtJachWd8fMXMlpnZSjO7PlJ+TqQelpnZQjNLj7QwZkfqcEm01pOZJZjZvXX+re6os/kFoM3NkdNmubseegTyAA5G/l4OPER4FNAQ8DJwYWRbduRve2Al0Cmy7sAn6jzXZuArkeUvAQ9Hlm8hPNESwGPAPyKvMYjwXAcA1xEeEj0EdCE838t1UeJ9A7i/znoW743S8DngvsjyD4Bv1tnvSWBcZLkQKI7y3BOAZ+us1437JeDmyPJtwAuR5ZeBGyPLXzhRnyc978eAP9ZZzwSSgY3AOZGyDMIjbKcC7SJlfYFFkeUiYGVk+XbgvyPLKcAioGdkPR9YEfTnSo/mecT7kOzSOlweebwdWe9A+MtrFvBVM7smUl4QKd8D1ADPnvQ8J4bbX0x4noxoXvDw/CPvWHhGSYBxwD8i5TvN7PVTxPp0neXuwNNm1pXwF/Kmeo65FBhk9u4I2Rlmlu7ulXX26QqU13P8mDrv5y/Az+uUfzSy/CTwiyjHrgB+YWb/C7zs7rPNbCiww93fAnD3AxBupQC/M7PhhOu3X5TnuxwYVqdllkn432QTUAZ0q+c9SCuj5CEtgQH3uPuD7ysMT5Z0KTDG3avM7A3CU98CHHH3mpOe52jkbw31f7aP1lm2k/42xKE6y78lPJ3x5EisP6jnmBDh93D4FM97mPfe2+k0eEA6d19rZmcDk4B7zGwa4dNL0Z7jG8Au4KxIzEei7GOEW3hTo2xrR/h9SBugPg9pCaYCt5lZBwAzyzezzoR/1e6NJI4BwHkxev05wMcifR95hDu8GyITKI0s31ynvBJIr7M+jfDopgBEftmfrBjoU8/rzCU8nDaE+xTmRJbnEz4tRZ3t72Nm3YAqd/8r4ZbJSGA10M3Mzonskx65ACCTcIukFriJ8FzXJ5sKfNHMkiLH9ou0WCDcUjnlVVnSeih5SODcfRrh0y7zzGwF8AzhL99XgUQzWw78mPCXZSw8S3jinJXAg8ACYH8DjvsB8A8zmw3srlP+EnDNiQ5z4KvAqEgH8ztEma3N3VcTnkY1/eRtkeNvjdTDTcDXIuVfB/7DzBYSPu0VLeahwEIzWwp8D/iJux8Drgd+a2bLgOmEWw33Azeb2XzCieBQlOd7GHgHWBK5fPdB3mvlTQBeiXKMtEIakl0EMLMO7n7QwvM4LwTGuvvOZo7hG0Cluz/cwP1TgcPu7mZ2A+HO86tjGuSp45kFXO3ue4OKQZqP+jxEwl42s46EO75/3NyJI+IPwMcbsf/ZhDu4DdhH+EqsQJhZLuH+HyWONkItDxERaTT1eYiISKMpeYiISKMpeYiISKMpeYiISKMpeYiISKMpeYiISKP9f73ur9AKzmZwAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.sched.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 3e-3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f2d96c9b3d854c0083895c6b5f4827b8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=12), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 14%|█▍        | 52/362 [00:05<00:30, 10.00it/s, loss=8.32]\n",
      "epoch      trn_loss   val_loss                              \n",
      "    0      5.44016    5.701974  \n",
      "    1      4.310874   4.549533                              \n",
      "    2      3.890306   4.157954                              \n",
      "    3      3.719702   4.019643                              \n",
      "    4      3.322341   3.865383                              \n",
      "    5      3.221692   3.795207                              \n",
      "    6      3.220406   3.697457                              \n",
      "    7      3.244708   3.7222                                \n",
      "    8      2.893455   3.646579                              \n",
      "    9      2.846069   3.616925                              \n",
      "    10     2.784473   3.573879                              \n",
      "    11     2.667841   3.572502                              \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[3.572502047460784]"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.fit(lr, n_cycle=1, cycle_len=12, use_clr=(20, 10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('first_model')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load('first_model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "import itertools"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {},
   "outputs": [],
   "source": [
    "x, y = next(itertools.islice(iter(val_dl), 25, None))\n",
    "probs = learn.model(V(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([18, 187, 17573])"
      ]
     },
     "execution_count": 164,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "probs.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds = to_np(probs.max(2)[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "qu' est ce que l ' t_up ipv ? _eos_\n",
      "what is t_up api ? _eos_\n",
      "what is the t_up ? _eos_\n",
      "\n",
      "qu' entend - on par autres méthodes de déclaration ? _eos_\n",
      "what is alternate reporting ? _eos_\n",
      "what are other other ? ? _eos_\n",
      "\n",
      "que montrent - elles ? _eos_\n",
      "what does it show ? _eos_\n",
      "what do they do ? _eos_\n",
      "\n",
      "qui est en conflit ? _eos_\n",
      "who is in conflict ? _eos_\n",
      "who is conflict ? _eos_\n",
      "\n",
      "pourquoi encourager davantage la recherche ? _eos_\n",
      "why encourage more research ? _eos_\n",
      "why encourage more promote research ? _eos_\n",
      "\n",
      "qui sont les vulnérables ? _eos_\n",
      "who are the vulnerable ? _eos_\n",
      "who are vulnerable ? ?\n",
      "\n",
      "pourquoi encourager davantage la recherché ? _eos_\n",
      "why encourage more research ? _eos_\n",
      "why encourage we more ? _eos_ _eos_\n",
      "\n",
      "qu’ est -ce que des données identifiables ? _eos_\n",
      "what is identifiable data ? _eos_\n",
      "what is the ? ? _eos_\n",
      "\n",
      "pourquoi était - il l' adversaire de sir john ? _eos_\n",
      "why were they opponents ? _eos_\n",
      "why was he shoemaker ’s ? ? _eos_\n",
      "\n",
      "qu' est -ce que l' admissibilité partagée ? _eos_\n",
      "what is shared eligibility ? _eos_\n",
      "what is the - eligibility ?\n",
      "\n"
     ]
    }
   ],
   "source": [
    "start = 110\n",
    "for i in range(start, start + 10):\n",
    "    print(' '.join([fr_itos[o] for o in x[:, i] if o != 1]))  # Input\n",
    "    print(' '.join([en_itos[o] for o in y[:, i] if o != 1]))  # Target\n",
    "    print(' '.join([en_itos[o] for o in preds[:, i] if o != 1]))  # Prediction\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bidirectional encoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Seq2SeqRNNBidirectional(nn.Module):\n",
    "    def __init__(self, vecs_enc, itos_enc, emb_sz_enc, vecs_dec, itos_dec, emb_sz_dec, nh, out_seqlen, nl=2):\n",
    "        super().__init__()\n",
    "        self.nh, self.nl, self.out_seqlen = nh, nl, out_seqlen\n",
    "        \n",
    "        # Encoder\n",
    "        self.emb_enc = create_emb(vecs_enc, itos_enc, emb_sz_enc)\n",
    "        self.emb_enc_drop = nn.Dropout(0.15)\n",
    "        self.rnn_enc = nn.GRU(emb_sz_enc, nh, num_layers=nl, dropout=0.25, bidirectional=True)\n",
    "        # sz 300 in, sz 256 out\n",
    "        self.outp_enc = nn.Linear(2 * nh, emb_sz_dec, bias=False)  # *2 because of bidirectional\n",
    "        # sz 256 in, sz 300 out\n",
    "        self.drop_enc = nn.Dropout(0.05)\n",
    "        \n",
    "        # Decoder\n",
    "        self.emb_dec = create_emb(vecs_dec, itos_dec, emb_sz_dec)\n",
    "        self.rnn_dec = nn.GRU(emb_sz_dec, emb_sz_dec, num_layers=nl, dropout=0.1)  # square to enable weight tying\n",
    "        self.out_drop = nn.Dropout(0.35)                                           # sz 300 in, 300 out\n",
    "        self.out = nn.Linear(emb_sz_dec, len(itos_dec))\n",
    "        self.out.weight.data = self.emb_dec.weight.data                            # weight tying! not transposed?\n",
    "        \n",
    "    def forward(self, inp):\n",
    "        seqlen, bs = inp.size()\n",
    "        \n",
    "        # Encoder\n",
    "        h = self.initHidden(bs)\n",
    "        emb = self.emb_enc_drop(self.emb_enc(inp))\n",
    "        enc_outp, h = self.rnn_enc(emb, h)\n",
    "        # pdb.set_trace()\n",
    "        h = h.view(2,2,bs,-1).permute(0,2,1,3).contiguous().view(2,bs,-1)\n",
    "        h = self.outp_enc(self.drop_enc(h))  # h[1] is same as enc_outp[-1]!\n",
    "\n",
    "        # Decoder\n",
    "        dec_inp = V(torch.zeros(bs).long())  # 0 is 'beginning of seq' token\n",
    "        res = []\n",
    "        for i in range(self.out_seqlen):\n",
    "            emb = self.emb_dec(dec_inp).unsqueeze(0)  # adds unit axis at beginning so that rnn 'loops' once\n",
    "            outp, h = self.rnn_dec(emb, h)  # 'loops' once\n",
    "            outp = self.out(self.out_drop(outp[0]))  # 0 because 'loops' once, predictions for words\n",
    "            res.append(outp)\n",
    "            dec_inp = V(outp.data.max(1)[1])  # [1] to get argmax\n",
    "            if (dec_inp == 1).all(): break    # if all sentences in the minibatch have _pad_ as input, break\n",
    "        return torch.stack(res)\n",
    "        \n",
    "    def initHidden(self, bs):\n",
    "        return V(torch.zeros(self.nl * 2, bs, self.nh))  # because of bidirectional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3097 ['l’', \"d'\", 't_up', 'd’', \"qu'\"]\n",
      "1285 [\"'s\", '’s', \"n't\", 'n’t', ':']\n"
     ]
    }
   ],
   "source": [
    "rnn = Seq2SeqRNN(fr_vec_dict, fr_itos, dim_fr_vec, en_vec_dict, en_itos, dim_en_vec, n_hid, en_len_99)\n",
    "learn = Learner(modeldata, SingleModel(to_gpu(rnn)), opt_fn=opt_fn)\n",
    "learn.crit = Seq2SeqLoss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1b58c126f38e41778a02de0927b4ed3c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=12), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss                              \n",
      "    0      5.267118   5.180488  \n",
      "    1      4.31144    4.448733                              \n",
      "    2      3.889166   4.067809                              \n",
      "    3      3.632571   3.874861                              \n",
      "    4      3.259331   3.805607                              \n",
      "    5      3.262788   3.709997                              \n",
      "    6      3.083301   3.682233                              \n",
      "    7      3.000043   3.629029                              \n",
      "    8      3.079843   3.577867                              \n",
      "    9      2.810955   3.538162                              \n",
      "    10     2.795676   3.544407                              \n",
      "    11     2.611297   3.525392                              \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[3.525392242647567]"
      ]
     },
     "execution_count": 172,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.fit(lr, n_cycle=1, cycle_len=12, use_clr=(20, 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Without bidirectional the final loss was 3.57"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('bidirectional_model')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load('bidirectional_model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Teacher forcing (without bidirectional for now)\n",
    "**Idea:** Especially in the beginning the model predicts only garbage. Let's kickstart training by whispering some answers to it just as a teacher might do during an oral exam.\n",
    "\n",
    "We reduce the teacher forcing probability over time. In order to do that we create a new `Seq2SeqStepper` and override the `step` method of `Stepper`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Seq2SeqStepper(Stepper):\n",
    "    def step(self, xs, y, epoch):\n",
    "        # This line implements the decay of the teacher forcing prob\n",
    "        self.m.pr_force = (10-epoch)*0.1 if epoch<10 else 0\n",
    "        \n",
    "        # The rest ist just the stepper method from the library\n",
    "        xtra = []\n",
    "        output = self.m(*xs, y)\n",
    "        if isinstance(output,tuple): output,*xtra = output\n",
    "        self.opt.zero_grad()\n",
    "        loss = raw_loss = self.crit(output, y)\n",
    "        if self.reg_fn: loss = self.reg_fn(output, xtra, raw_loss)\n",
    "        loss.backward()\n",
    "        if self.clip:   # Gradient clipping\n",
    "            nn.utils.clip_grad_norm(trainable_params_(self.m), self.clip)\n",
    "        self.opt.step()\n",
    "        return raw_loss.data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Seq2SeqRNNTeacherForcing(nn.Module):\n",
    "    def __init__(self, vecs_enc, itos_enc, emb_sz_enc, vecs_dec, itos_dec, emb_sz_dec, nh, out_seqlen, nl=2):\n",
    "        super().__init__()\n",
    "        self.nh, self.nl, self.out_seqlen = nh, nl, out_seqlen\n",
    "        \n",
    "        # Encoder\n",
    "        self.emb_enc = create_emb(vecs_enc, itos_enc, emb_sz_enc)\n",
    "        self.emb_enc_drop = nn.Dropout(0.15)\n",
    "        self.rnn_enc = nn.GRU(emb_sz_enc, nh, num_layers=nl, dropout=0.25)  # sz 300 in, sz 256 out\n",
    "        self.outp_enc = nn.Linear(nh, emb_sz_dec, bias=False)               # sz 256 in, sz 300 out\n",
    "        \n",
    "        # Decoder\n",
    "        self.emb_dec = create_emb(vecs_dec, itos_dec, emb_sz_dec)\n",
    "        self.rnn_dec = nn.GRU(emb_sz_dec, emb_sz_dec, num_layers=nl, dropout=0.1)  # square to enable weight tying\n",
    "        self.out_drop = nn.Dropout(0.35)                                           # sz 300 in, 300 out\n",
    "        self.out = nn.Linear(emb_sz_dec, len(itos_dec))\n",
    "        self.out.weight.data = self.emb_dec.weight.data                            # weight tying! not transposed?\n",
    "        \n",
    "        self.pr_force = 1. ### Teacher forcing ###\n",
    "        \n",
    "    def forward(self, inp, y=None):\n",
    "        seqlen, bs = inp.size()\n",
    "        \n",
    "        # Encoder\n",
    "        h = self.initHidden(bs)\n",
    "        emb = self.emb_enc_drop(self.emb_enc(inp))\n",
    "        enc_outp, h = self.rnn_enc(emb, h)\n",
    "        # pdb.set_trace()\n",
    "        h = self.outp_enc(h)  # h[1] is same as enc_outp[-1]!\n",
    "\n",
    "        # Decoder\n",
    "        dec_inp = V(torch.zeros(bs).long())  # 0 is 'beginning of seq' token\n",
    "        res = []\n",
    "        for i in range(self.out_seqlen):\n",
    "            emb = self.emb_dec(dec_inp).unsqueeze(0)  # adds unit axis at beginning so that rnn 'loops' once\n",
    "            outp, h = self.rnn_dec(emb, h)  # 'loops' once\n",
    "            outp = self.out(self.out_drop(outp[0]))  # 0 because 'loops' once, predictions for words\n",
    "            res.append(outp)\n",
    "            dec_inp = V(outp.data.max(1)[1])  # [1] to get argmax\n",
    "            if (dec_inp == 1).all(): break    # if all sentences in the minibatch have _pad_ as input, break\n",
    "                \n",
    "            if (y is not None) and (random.random() < self.pr_force):\n",
    "                if i >= len(y): break  # predicted seq is already longer than y => nothing to whisper\n",
    "                dec_inp = y[i]         ### Teacher forcing ###\n",
    "            \n",
    "        return torch.stack(res)\n",
    "        \n",
    "    def initHidden(self, bs):\n",
    "        return V(torch.zeros(self.nl, bs, self.nh))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3097 ['l’', \"d'\", 't_up', 'd’', \"qu'\"]\n",
      "1285 [\"'s\", '’s', \"n't\", 'n’t', ':']\n"
     ]
    }
   ],
   "source": [
    "rnn = Seq2SeqRNNTeacherForcing(fr_vec_dict, fr_itos, dim_fr_vec, en_vec_dict, en_itos, dim_en_vec, n_hid, en_len_99)\n",
    "learn = Learner(modeldata, SingleModel(to_gpu(rnn)), opt_fn=opt_fn)\n",
    "learn.crit = Seq2SeqLoss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2520c8150c4f4e30b40da1fae61ecc36",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=12), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  1%|          | 3/362 [00:00<00:35,  9.99it/s, loss=tensor(10.9706, device='cuda:0')]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/fabiograetz/anaconda3/envs/deeplearning/lib/python3.6/site-packages/ipykernel/__main__.py:17: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss                                                         \n",
      "    0      3.798883   11.010717 \n",
      "    1      3.230599   7.161752                                                         \n",
      "    2      3.058165   5.452962                                                         \n",
      "    3      3.285037   4.634273                                                         \n",
      "    4      2.992541   4.289666                                                         \n",
      "    5      2.89953    4.058496                                                         \n",
      "    6      2.951971   3.809111                                                         \n",
      "    7      2.826471   3.737335                                                         \n",
      "    8      2.820316   3.596977                                                         \n",
      "    9      3.075896   3.557067                                                         \n",
      "    10     2.818115   3.518843                                                         \n",
      "    11     2.707423   3.508829                                                         \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[3.508829193894968]"
      ]
     },
     "execution_count": 191,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.fit(lr, n_cycle=1, cycle_len=12, use_clr=(20, 10), stepper=Seq2SeqStepper)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Without bidirectional and teacher forcing the final loss was 3.57\n",
    "1. Without teacher forcing but with bidirectional the final loss was 3.525\n",
    "\n",
    "Interesting question: why is the validation loss so high initially compared to the previous tries?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('teacher_forcing_model')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load('teacher_forcing_model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Attention"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "def random_tensor(*sz):\n",
    "    return torch.randn(sz)/math.sqrt(sz[0])  # division by sqrt is initialization\n",
    "def random_param(*sz):\n",
    "    return nn.Parameter(random_tensor(*sz))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 341,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Seq2SeqRNNAttentionModel(nn.Module):\n",
    "    def __init__(self, vecs_enc, itos_enc, emb_sz_enc, vecs_dec, itos_dec, emb_sz_dec, nh, out_seqlen, nl=2):\n",
    "        super().__init__()\n",
    "        self.nh, self.nl, self.out_seqlen = nh, nl, out_seqlen\n",
    "        \n",
    "        # Encoder\n",
    "        self.emb_enc = create_emb(vecs_enc, itos_enc, emb_sz_enc)\n",
    "        self.emb_enc_drop = nn.Dropout(0.15)\n",
    "        self.rnn_enc = nn.GRU(emb_sz_enc, nh, num_layers=nl, dropout=0.25)\n",
    "        self.outp_enc = nn.Linear(nh, emb_sz_dec, bias=False)\n",
    "        \n",
    "        # Decoder\n",
    "        self.emb_dec = create_emb(vecs_dec, itos_dec, emb_sz_dec)\n",
    "        self.rnn_dec = nn.GRU(emb_sz_dec, emb_sz_dec, num_layers=nl, dropout=0.1)\n",
    "        self.out_drop = nn.Dropout(0.35)\n",
    "        self.out = nn.Linear(emb_sz_dec, len(itos_dec))\n",
    "        self.out.weight.data = self.emb_dec.weight.data  # weight tying!\n",
    "        \n",
    "        # Attention\n",
    "        self.W1 = random_param(nh, emb_sz_dec)\n",
    "        self.l2 = nn.Linear(emb_sz_dec, emb_sz_dec)\n",
    "        self.l3 = nn.Linear(emb_sz_dec + nh, emb_sz_dec)\n",
    "        self.V = random_param(emb_sz_dec)\n",
    "        \n",
    "        self.pr_force = 1. ### Teacher forcing ###\n",
    "        \n",
    "    def forward(self, inp, y=None, ret_attn=False):\n",
    "        seqlen, bs = inp.size()\n",
    "        \n",
    "        # Encoder\n",
    "        h = self.initHidden(bs)\n",
    "        emb = self.emb_enc_drop(self.emb_enc(inp))\n",
    "        enc_outp, h = self.rnn_enc(emb, h)  # h[1] is same as enc_outp[-1]!\n",
    "        h = self.outp_enc(h)  # from hidden size to emb_sz_dec\n",
    "        \n",
    "        dec_inp = V(torch.zeros(bs).long())  # '_bos_' token for every seq in minibatch\n",
    "        res, attns = [], []\n",
    "        # See Luong et al 2015 section 3.1 equation 'score(h_t, \\bar h_s)'\n",
    "        # Multiply all hidden states (\\bar h_s) of the encoder with Matrix W_alpha\n",
    "        w1 = enc_outp @ self.W1\n",
    "        # I think that this implementation is slightly different from concat in Luong et al 2015\n",
    "        # because they concat the hidden states and then apply a single matrix multiplication.\n",
    "        # Here we have two different matrices with output size emb_sz_dec, apply them first\n",
    "        # and then concatenate\n",
    "        for i in range(self.out_seqlen):\n",
    "            \n",
    "            # Attention\n",
    "            w2 = self.l2(h[-1])  # $W_alpha$ times $h_t$ in Luong et al\n",
    "            u = F.tanh(w1 + w2)\n",
    "            a = F.softmax(u @ self.V, 0)  # See equation 7 in Luong et al, V is $v_a^T$\n",
    "            \n",
    "            attns.append(a)\n",
    "            c_t = (a.unsqueeze(2) * enc_outp).sum(0)  # context vector = weighted sum of encoder hidden states\n",
    "            \n",
    "            emb = self.emb_dec(dec_inp)\n",
    "            \n",
    "            weighted_enc = self.l3(torch.cat([emb, c_t], 1))  \n",
    "            \n",
    "            # ATTENTION\n",
    "            # In the Luong et al Paper things are slightly different\n",
    "            # the context vector c_t and the current decoder hidden state\n",
    "            # are concat together and then fed into a linear layer with tanh activation.\n",
    "            # The result called attentional hidden state is fed into a second linear layer with softmax\n",
    "            # activation which gives the probabilities for every token.\n",
    "            \n",
    "            # The fastai implementation takes the context vector c_t, concats it with the INPUT\n",
    "            # of the decoder (dec_inp) and feeds this into a linear layer.\n",
    "            # This input (dec_inp) that is 'mixed' with the context vector \n",
    "            # is then fed into the decoder rnn. The hidden state that is fed into\n",
    "            # the decoder rnn is NOT 'mixed' with the context vector as opposed to Long et al 2015\n",
    "            \n",
    "            outp, h = self.rnn_dec(weighted_enc.unsqueeze(0), h)  # 'loops' once because of unsqueeze(0)\n",
    "            outp = self.out(self.out_drop(outp[0]))  # 0 because 'loops' once, predictions for words\n",
    "            res.append(outp)\n",
    "            \n",
    "            dec_inp = V(outp.data.max(1)[1])  # [1] to get argmax\n",
    "            \n",
    "            if (dec_inp == 1).all(): break\n",
    "            if (y is not None) and (random.random() < self.pr_force):\n",
    "                if i >= len(y): break  # predicted seq is already longer than y => nothing to whisper\n",
    "                dec_inp = y[i]         ### Teacher forcing ###    \n",
    "            \n",
    "        res = torch.stack(res)\n",
    "        if ret_attn: res = res, torch.stack(attns)\n",
    "        return res\n",
    "        \n",
    "    def initHidden(self, bs):\n",
    "        return V(torch.zeros(self.nl, bs, self.nh))  # if all sentences in the minibatch have _pad_ as input, break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 342,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3097 ['l’', \"d'\", 't_up', 'd’', \"qu'\"]\n",
      "1285 [\"'s\", '’s', \"n't\", 'n’t', ':']\n"
     ]
    }
   ],
   "source": [
    "rnn = Seq2SeqRNNAttentionModel(fr_vec_dict, fr_itos, dim_fr_vec, en_vec_dict, en_itos, dim_en_vec, n_hid, en_len_99)\n",
    "learn = Learner(modeldata, SingleModel(to_gpu(rnn)), opt_fn=opt_fn)\n",
    "learn.crit = Seq2SeqLoss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 2e-3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 344,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1054e232a509413ba1bab9c8a0f68a3a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=12), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  0%|          | 1/362 [00:00<01:10,  5.10it/s, loss=tensor(9.8161, device='cuda:0')]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/fabiograetz/anaconda3/envs/deeplearning/lib/python3.6/site-packages/torch/nn/functional.py:995: UserWarning: nn.functional.tanh is deprecated. Use torch.tanh instead.\n",
      "  warnings.warn(\"nn.functional.tanh is deprecated. Use torch.tanh instead.\")\n",
      "/home/fabiograetz/anaconda3/envs/deeplearning/lib/python3.6/site-packages/ipykernel/__main__.py:17: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 14%|█▎        | 49/362 [00:03<00:22, 13.88it/s, loss=tensor(6.9882, device='cuda:0')]\n",
      "epoch      trn_loss   val_loss                                                         \n",
      "    0      4.191049   14.424263 \n",
      "    1      3.060529   5.687311                                                         \n",
      "    2      3.068104   5.495779                                                         \n",
      "    3      2.772898   4.794846                                                         \n",
      "    4      2.817782   4.081442                                                         \n",
      "    5      2.894404   4.314808                                                         \n",
      "    6      2.759051   3.80134                                                          \n",
      "    7      2.73077    3.705336                                                         \n",
      "    8      2.769924   3.509056                                                         \n",
      "    9      2.944073   3.422912                                                         \n",
      "    10     2.801802   3.399807                                                         \n",
      "    11     2.691524   3.3661                                                           \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[3.3660999226120283]"
      ]
     },
     "execution_count": 344,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.fit(lr, n_cycle=1, cycle_len=12, use_clr=(20, 10), stepper=Seq2SeqStepper)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Without bidirectional and teacher forcing the final loss was 3.57\n",
    "1. Without teacher forcing but with bidirectional the final loss was 3.525\n",
    "1. With teacher forcing but without bidirectional the final loss was 3.508829\n",
    "\n",
    "Now we are down to 3.366.\n",
    "\n",
    "Now, let's combine all methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 346,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('attention_model')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 347,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load('attention_model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bidirectional, teacher forcing, and Attention"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Seq2SeqRNNAll(nn.Module):\n",
    "    def __init__(self, vecs_enc, itos_enc, emb_sz_enc, vecs_dec, itos_dec, emb_sz_dec, nh, out_seqlen, nl=2):\n",
    "        super().__init__()\n",
    "        self.nh, self.nl, self.out_seqlen = nh, nl, out_seqlen\n",
    "        \n",
    "        # Encoder\n",
    "        self.emb_enc = create_emb(vecs_enc, itos_enc, emb_sz_enc)\n",
    "        self.emb_enc_drop = nn.Dropout(0.15)\n",
    "        self.rnn_enc = nn.GRU(emb_sz_enc, nh, num_layers=nl, dropout=0.25, bidirectional=True)\n",
    "        self.drop_enc = nn.Dropout(0.25)\n",
    "        self.outp_enc = nn.Linear(2 * nh, emb_sz_dec, bias=False)\n",
    "     \n",
    "        # Decoder\n",
    "        self.emb_dec = create_emb(vecs_dec, itos_dec, emb_sz_dec)\n",
    "        self.rnn_dec = nn.GRU(emb_sz_dec, emb_sz_dec, num_layers=nl, dropout=0.1)\n",
    "        self.out_drop = nn.Dropout(0.35)\n",
    "        self.out = nn.Linear(emb_sz_dec, len(itos_dec))\n",
    "        self.out.weight.data = self.emb_dec.weight.data  # weight tying!\n",
    "        \n",
    "        # Attention\n",
    "        self.W1 = random_param(2 * nh, emb_sz_dec)\n",
    "        self.l2 = nn.Linear(emb_sz_dec, emb_sz_dec)\n",
    "        self.l3 = nn.Linear(emb_sz_dec + nh * 2, emb_sz_dec)\n",
    "        self.V = random_param(emb_sz_dec)\n",
    "        \n",
    "        self.pr_force = 1. ### Teacher forcing ###\n",
    "        \n",
    "    def forward(self, inp, y=None, ret_attn=False):\n",
    "        seqlen, bs = inp.size()\n",
    "        \n",
    "        # Encoder\n",
    "        h = self.initHidden(bs)\n",
    "        emb = self.emb_enc_drop(self.emb_enc(inp))\n",
    "        enc_outp, h = self.rnn_enc(emb, h)\n",
    "        h = h.view(2,2,bs,-1).permute(0,2,1,3).contiguous().view(2,bs,-1)\n",
    "        h = self.outp_enc(self.drop_enc(h))\n",
    "        \n",
    "        dec_inp = V(torch.zeros(bs).long())  # '_bos_' token for every seq in minibatch\n",
    "        res, attns = [], []\n",
    "        # See Luong et al 2015 section 3.1 equation 'score(h_t, \\bar h_s)'\n",
    "        # Multiply all hidden states (\\bar h_s) of the encoder with Matrix W_alpha\n",
    "        w1 = enc_outp @ self.W1\n",
    "        # I think that this implementation is slightly different from concat in Luong et al 2015\n",
    "        # because they concat the hidden states and then apply a single matrix multiplication.\n",
    "        # Here we have two different matrices with output size emb_sz_dec, apply them first\n",
    "        # and then concatenate\n",
    "        for i in range(self.out_seqlen):\n",
    "            \n",
    "            # Attention\n",
    "            w2 = self.l2(h[-1])  # $W_alpha$ times $h_t$ in Luong et al\n",
    "            u = F.tanh(w1 + w2)\n",
    "            a = F.softmax(u @ self.V, 0)  # See equation 7 in Luong et al, V is $v_a^T$\n",
    "            \n",
    "            attns.append(a)\n",
    "            c_t = (a.unsqueeze(2) * enc_outp).sum(0)  # context vector = weighted sum of encoder hidden states\n",
    "            \n",
    "            emb = self.emb_dec(dec_inp)\n",
    "            \n",
    "            weighted_enc = self.l3(torch.cat([emb, c_t], 1))  \n",
    "            \n",
    "            # ATTENTION\n",
    "            # In the Luong et al Paper things are slightly different\n",
    "            # the context vector c_t and the current decoder hidden state\n",
    "            # are concat together and then fed into a linear layer with tanh activation.\n",
    "            # The result called attentional hidden state is fed into a second linear layer with softmax\n",
    "            # activation which gives the probabilities for every token.\n",
    "            \n",
    "            # The fastai implementation takes the context vector c_t, concats it with the INPUT\n",
    "            # of the decoder (dec_inp) and feeds this into a linear layer.\n",
    "            # This input (dec_inp) that is 'mixed' with the context vector \n",
    "            # is then fed into the decoder rnn. The hidden state that is fed into\n",
    "            # the decoder rnn is NOT 'mixed' with the context vector as opposed to Long et al 2015\n",
    "            \n",
    "            outp, h = self.rnn_dec(weighted_enc.unsqueeze(0), h)  # 'loops' once because of unsqueeze(0)\n",
    "            outp = self.out(self.out_drop(outp[0]))  # 0 because 'loops' once, predictions for words\n",
    "            res.append(outp)\n",
    "            \n",
    "            dec_inp = V(outp.data.max(1)[1])  # [1] to get argmax\n",
    "            \n",
    "            if (dec_inp == 1).all(): break\n",
    "            if (y is not None) and (random.random() < self.pr_force):\n",
    "                if i >= len(y): break  # predicted seq is already longer than y => nothing to whisper\n",
    "                dec_inp = y[i]         ### Teacher forcing ###    \n",
    "            \n",
    "        res = torch.stack(res)\n",
    "        if ret_attn: res = res, torch.stack(attns)\n",
    "        return res\n",
    "        \n",
    "    def initHidden(self, bs):\n",
    "        return V(torch.zeros(2 * self.nl, bs, self.nh))  # if all sentences in the minibatch have _pad_ as input, break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3097 ['l’', \"d'\", 't_up', 'd’', \"qu'\"]\n",
      "1285 [\"'s\", '’s', \"n't\", 'n’t', ':']\n"
     ]
    }
   ],
   "source": [
    "rnn = Seq2SeqRNNAll(fr_vec_dict, fr_itos, dim_fr_vec, en_vec_dict, en_itos, dim_en_vec, n_hid, en_len_99)\n",
    "learn = Learner(modeldata, SingleModel(to_gpu(rnn)), opt_fn=opt_fn)\n",
    "learn.crit = Seq2SeqLoss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2a73d76be29b47a0895ae0a80d5b0393",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=12), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  0%|          | 1/362 [00:00<01:07,  5.39it/s, loss=tensor(10.5020, device='cuda:0')]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/fabiograetz/anaconda3/envs/deeplearning/lib/python3.6/site-packages/torch/nn/functional.py:995: UserWarning: nn.functional.tanh is deprecated. Use torch.tanh instead.\n",
      "  warnings.warn(\"nn.functional.tanh is deprecated. Use torch.tanh instead.\")\n",
      "/home/fabiograetz/anaconda3/envs/deeplearning/lib/python3.6/site-packages/ipykernel/__main__.py:17: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss                                                         \n",
      "    0      3.432203   9.22238   \n",
      "    1      3.837512   8.494652                                                         \n",
      "    2      3.28513    6.461766                                                         \n",
      "    3      3.276439   5.529211                                                         \n",
      "    4      3.003907   6.119911                                                         \n",
      "    5      3.133035   4.014019                                                         \n",
      "    6      2.863579   3.807283                                                         \n",
      "    7      2.899184   3.745806                                                         \n",
      "    8      3.003482   3.547384                                                         \n",
      "    9      3.096476   3.428664                                                         \n",
      "    10     2.82979    3.357279                                                         \n",
      "    11     2.626434   3.325287                                                         \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[3.3252869728976076]"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.fit(lr, n_cycle=1, cycle_len=12, use_clr=(20, 10), stepper=Seq2SeqStepper)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd8VFX+//HXSW+UEEIvARQQpIoIooCICGLZVb9fdb+6ug13dV1119+KHXvZxuJa1rZrWXvZVVGkKKigYGhKh9BBIKEngdTz+2PuTGZSZ5JMcmf2/Xw88uDOnTszn1zNe+4999xzjLUWERGJfDHNXYCIiDQOBbqISJRQoIuIRAkFuohIlFCgi4hECQW6iEiUUKCLiEQJBbqISJRQoIuIRIm4cLxp27ZtbVZWVjjeWkQkKi1dujTPWpvZkPcIS6BnZWWRnZ0djrcWEYlKxphtDX0PNbmIiEQJBbqISJRQoIuIRAkFuohIlFCgi4hECQW6iEiUUKCLiEQJVwX6jHkbWbAht7nLEBGJSK4K9L8vyOELBbqISL0EFejGmBuNMauMMauNMTeFq5jE+FiOl5aF6+1FRKJanYFujDkZ+AUwHBgEnG+MOTEcxSTFxXC8pDwcby0iEvWCOUI/CfjaWltorS0FFgA/DEcxSfGxFJUq0EVE6iOYQF8FjDbGZBhjUoDzgK7hKCYhLobjJWpyERGpjzpHW7TWrjXGPArMAfKBlUBp5e2MMVOAKQDdunWrVzFJ8bEKdBGRegrqoqi19nlr7VBr7WjgALCxmm2esdYOs9YOy8ys35C+SfExfH/4eL1eKyLy3y6o8dCNMe2stfuMMd2Ai4GR4SjGWjhWrCN0EZH6CHaCi3eMMRlACXC9tfZgOIo5sX0aG/flh+OtRUSiXlCBbq09M9yFAKQkxFFYXKV5XkREguCqO0VTEmI5XlJOWblt7lJERCKOqwI9NcFzwqCjdBGR0Lkq0JMTYgEo1IVREZGQuSrQUxMV6CIi9eWqQE9xmlwKitTkIiISKpcFuucI/ZjuFhURCZnLAl1H6CIi9eWyQFcbuohIfbkq0Cu6LSrQRURC5apAT/H1clGTi4hIqNwV6GpyERGpN1cFelJcLMZAoS6KioiEzFWBHhNjSI6PpUBH6CIiIXNVoIN3xEUFuohIqFwX6KmJsbooKiJSD64L9OT4WB2hi4jUg+sCPSVBR+giIvXhukBPTVQbuohIfbgu0JPjYzVRtIhIPbgu0FMT4yhQk4uISMhcF+jJCTpCFxGpj6AC3RhzszFmtTFmlTHmNWNMUrgKSlEvFxGReqkz0I0xnYHfAMOstScDscDl4SooxbkoWl5uw/URIiJRKdgmlzgg2RgTB6QAu8NVkHeAruOlOkoXEQlFnYFurd0F/BHYDnwPHLbWzg5XQakacVFEpF6CaXJJBy4CegCdgFRjzJXVbDfFGJNtjMnOzc2td0HJ3kkuihToIiKhCKbJZTywxVqba60tAd4FTq+8kbX2GWvtMGvtsMzMzHoX5BsTvURdF0VEQhFMoG8HRhhjUowxBjgbWBuugryBXqAjdBGRkATThr4YeBtYBnznvOaZcBWUmuidV1RH6CIioYgLZiNr7T3APWGuBaiYKFpH6CIioXHdnaKpmihaRKReXBfoKd4jdHVbFBEJiesCPS3R2+SiI3QRkVC4LtCT4mMwBgoV6CIiIXFdoBtjSE2IU5OLiEiIXBfo4OmLriYXEZHQuDLQ0xJ1hC4iEipXBnpKYqza0EVEQuTOQE+II1+BLiISElcGepozyYWIiATPlYGekhCriaJFRELkykBPTYhTLxcRkRC5M9AT4zTBhYhIiFwa6J4mF2s1UbSISLBcGegpCXGUWygqLW/uUkREIoYrA907hK66LoqIBM+dga6JokVEQubOQHeO0NV1UUQkeK4MdN8kF2pyEREJmisDvUWSJ9CPHlegi4gEy6WBHg/AkeMlzVyJiEjkqDPQjTF9jDEr/H6OGGNuCmdRLXWELiISsri6NrDWrgcGAxhjYoFdwHvhLKplso7QRURCFWqTy9lAjrV2WziK8UqMiyE+1ugIXUQkBKEG+uXAa+EoxJ8xhhZJ8RzVEbqISNCCDnRjTAJwIfBWDc9PMcZkG2Oyc3NzG1xYWmIc+TpCFxEJWihH6JOAZdbavdU9aa19xlo7zFo7LDMzs8GFtUiKU5OLiEgIQgn0K2iC5havFklxHNWNRSIiQQsq0I0xKcA5wLvhLadCWmI8Ow8UNtXHiYhEvDq7LQJYawuBjDDXEqCsvJzdh4835UeKiEQ0V94pCtCtTQoAZeWa5EJEJBiuDfSuTqCrp4uISHBcG+i+AbqK1BddRCQYLg50z+3/6rooIhIc1wZ6WqLnCF3T0ImIBMe1gV4xJrqaXEREghEBga4jdBGRYLg40D1t6GpyEREJjmsD3duGriN0EZHguDbQUxJiiTHqhy4iEizXBroxhrTEOF0UFREJkmsDHTzt6BpxUUQkOC4PdI2JLiISLFcHumYtEhEJnqsDPT42hoOFxc1dhohIRHB1oJeUlbNuz9HmLkNEJCK4OtC7pCcDYK3GRBcRqYurA71/p1YA6ukiIhIEVwd6emoCAAcL1I4uIlIXVwd6hhPoBxToIiJ1cnWgpyvQRUSCFlSgG2NaG2PeNsasM8asNcaMDHdhoCN0EZFQxAW53V+BWdbaS40xCUBKGGvy8bWhqy+6iEid6gx0Y0xLYDRwDYC1thhokoRNTYglITaG/TpCFxGpUzBNLj2BXOAfxpjlxpjnjDGpYa4L8Iy42CY1Qb1cRESCEEygxwFDgaestUOAAmBq5Y2MMVOMMdnGmOzc3NxGKzA9NYEDBRpCV0SkLsEE+k5gp7V2sfP4bTwBH8Ba+4y1dpi1dlhmZmajFZiRmsCBgqJGez8RkWhVZ6Bba/cAO4wxfZxVZwNrwlqVn/TUBA4W6ghdRKQuwfZyuQH4l9PDZTPwk/CVFCgjNYG8o0VYazHGNNXHiohEnKAC3Vq7AhgW5lqq1SU9maNFpRwqLPF1YxQRkapcfacoQLuWSQDsVzu6iEitXB/obVK8d4uqHV1EpDauD/T01HhAt/+LiNTF9YHeRrf/i4gExfWBnu40uezPVxu6iEhtXB/oSfGxJMfHsjmvoLlLERFxNdcHOsCxkjLeXbarucsQEXG1iAj0kzq2bO4SRERcLyIC/Zx+7TEGSsrKm7sUERHXiohAb98yEWshTxdGRURqFBmB3sJzt+jeIwp0EZGaREagO7f/7zl8vJkrERFxr4gI9O5tPVOYbs7Lb+ZKRETcKyICvWVSPC2T4tirI3QRkRpFRKADtG2RSF6+bv8XEalJ5AR6aiK56uUiIlKjiAn0zJaJ5B1VoIuI1CRiAr19iyT2HlEbuohITSIn0FsmUlBcRn5RaXOXIiLiShEU6N6bi3SULiJSnYgJ9HYtEwHYefBYM1ciIuJOQQW6MWarMeY7Y8wKY0x2uIuqTp/2LQBYvv1gc3y8iIjrhXKEfpa1drC1dljYqqlFRprnCH3Wqj3N8fEiIq4XMU0uXuv2HG3uEkREXCnYQLfAbGPMUmPMlHAWVJtBXVs310eLiLhesIE+ylo7FJgEXG+MGV15A2PMFGNMtjEmOzc3t1GL9IqPMQDMW7s3LO8vIhLJggp0a+1u5999wHvA8Gq2ecZaO8xaOywzM7Nxq3TcMfkkADbt06iLIiKV1RnoxphUY0wL7zIwAVgV7sKqM7hraxJiYzhQqEG6REQqiwtim/bAe8YY7/avWmtnhbWqGhhjyEhL0DC6IiLVqDPQrbWbgUFNUEtQTmiXxsKc/c1dhoiI60Rct8WubVLIPVrE4WMlzV2KiIirRFygn94rA4AteQXNXImIiLtEXKD37eAZAuCXLy9t5kpERNwl4gK9Q6tkAPZo1EURkQARF+hpiXHEOjcYlZfbZq5GRMQ9Ii7QAS4a1AnQUbqIiL+IDPRzT+4AwIEC3WAkIuIVkYHeJjUBgIO6Y1RExCciAz09xRPo2Vs12YWIiFdEBnqn1p75Rb/bdbiZKxERcY+IDPSUBM+IBZ+u24e16ukiIgIRGugA3TNSgOB7uhwvKQtnOSIizS5iA/36sScA8N3Ouptd/rNiF33vmsW1L2dz8xsrKFP/dRGJQhEb6GP6eCbRuH/mmjq3vevfnuHbP1m9l/eW72Lrfo0DIyLRJ2IDvX1Lz4XRHQeO1dmc0rdDy4DHZ/9pgdreRSTqRGyg+1uUk1fjc6t3H2bJ1gNV1j+1ICfgcVm51VACIhLRIjrQ373udAD++MmGGrd56KO1ADxz1Sl0bp3suynpsVnrfdscPV7CSXfP4vzHvwxjtSIi4RXMFHSuNbRbOgBrvj9S7fPWWgqKyoiLMUzo34EJ/TtgraXHbR8BnsG9bn3nW95autP3PqVl5cTFRvT3nIj8l4rq5Drlgbms2HGIUr+mFGMMt03qC8CKnYd8Ye51wh0fU1pW3qR1iog0hogP9DsnnwRUHajrtSXbfetuGn9iwHOn9mgDwMVPLvKt+3/n9vEtr9x5KCy1ioiEU8QHeq/MNADWVmp2ue3d7wDo0TaVm8b3DnhuUJfWAY+nXdCP6886wTe93c9ezA5XuSIiYRN0oBtjYo0xy40xH4azoFCdkpVObIxh4aaKni7r9lSE+0M/HFDlNd4JMgA6tUrimlE9AHj1FyMAOFRYUuULQkTE7UI5Qr8RWBuuQuqrZVI8PdqmkpOb71v3zRZPN8WnrxzKSOeouybPX3Nqtev3HNbkGSISWYIKdGNMF2Ay8Fx4y6mfLunJ5OQW+EI9J7eAlIRYJp7cscbX3Ow0w3gnnfb67JaxAOzX5BkiEmGCPUKfDvwecGX3j86tk9m0L5+z/7SAw8dKWL/nqK+/eU1uHH8iWx+ZjDEmYH3n1skkxMawYe/RcJYsItLo6uyHbow5H9hnrV1qjBlby3ZTgCkA3bp1a7QCg+E/c9Gge2c36L0S4mLo27EFizfvb2hZIiJNKpgj9FHAhcaYrcDrwDhjzCuVN7LWPmOtHWatHZaZmdnIZdbuyhHdq6w7zemaWB/FpeWs3HmY5ds1I5KIRI46A91ae5u1tou1Ngu4HPjUWntl2CsLwem92rL1kclcfmpX37qGBPqvx3mG5l26TYEuIpEj4vuh+2uRFOe3HF/v95k8oCMtkuLYtr+wMcoSEWkSIQW6tXa+tfb8cBXTUGf1bedbzmyRWO/3McbQJT2F95bvaoyyfL7deYgteRqLXUTCI6qO0Lump/iWzxtQc5fFYKz9/gj5RaVs2pdf98bVOHK8JGDM9QMFxVz4t4Wc9cf5DapLRKQmURXoHVol+ZYT4hr2q3nHiKlPoGdNncnAabOZMW8TADm5+fz61WUB21hreW3Jdg4fK9E47CLSKCJ6+NzK4mNjWHbXOcTHmro3rsMlQ7vwwMy1IU9Xd+R4iW/5L3M3UFpezuOfbgrYJmvqTC4a3In/rNjtG3PGa87No8lqm8rR46V19qUXEfFnwjEV27Bhw2x2duQPcJU1dSYAmx6cFNQY6cWl5fS+8+MGf+6Fgzrx/srdrLr3XNISo+o7V0RqYIxZaq0d1pD3iKoml3DZdiC43i4/e/Eb3/IJ7dICnuvZNpXXp4wI6n3eX7kbgBcXbQ2uQBERFOi1ev/XowDPpNLT3l9d5/beoXxf+8UI/vXz03zrtzx8Hp/eMpYRPTPY+sjkgNdcN7YX8343hjevHUm/joGTWf/hk/X8c+GWhv4aIvJfQoFeC29AA/xz0VZ21HGkXlpeTpvUBEb2yqB9yyQ2P3QeGx6YVGW8mKHdPOOxL7/rHH4/sS+9MtMY3qONb45Uf9M+WMPeI7WP/FhaVs7E6Z9z939WMW/t3oB2fBH576FAr0VqYhy/ObtitqPZa/bWuv0rX28PmDkpJsZU29vm3etGseGBSaRXuuiZFB/LoqnjWP/ARC4a3Mm3/rSH5lFQVEpN1ztW7T7Cuj1HeemrbfzsxWwGTmvYeDYiEpkU6HX47Tm9+W7aBADu/3BNjduF2vWwpm6VnVonkxgXy60T+3LF8IqhDPrf8wn97v6kyvbr9xzlB08srLI+a+pMJv31i4DJPkQkuinQg+A/jMCzn29mc27VvunbneaY353Tu8pz9dGpdTIPXzyQl3463LfuWEkZx0vKArY7//EvfMtL7xzP8KyKMWzWfn+Ee/5Td9t/c7DWclRNQyKNSoEepFd+5rnI+eBHa5k848sqz290bkCqa4akUI3unRnQtr5+T+A47SVlnjODuBhDRloib/5yJDc4g4sBLN5ygE9W72nUmhpi9uo9ZE2dSY/bPmLAtNka0VKkESnQg3TGiW2Z0K894DlSPlYceKS8+9AxALq1Sany2oYa2i2dL35/FgA3vLbct95/3tP1D0zyLf9uQh/uv6i/7/G1Ly9t9Jrqa0qlWn745CIKi0ubqRqR6KJAD8HTV57iW9647yhvL93pe3yP060xXHd3dklPBjxNOyt2HALgi425AJzcuWXAxNcAV43M4kW/5prqmokaS3m55UBBMTsOFFbbjOJtJqppGIV+d3/CIx+va9Dnj/vjfB6btY7iUldOqiXSJBToIYiJMbznNH9c+LeF3PLWSt/dpF7B3FFaH8YYrhzhmQnqB08sxFrLQx95QvCdX1Xt7ggwpncm0y8bDMDNb6xo9Jr2HTlOUWkZPW//iKH3z+HMxz5jwLTZ9L7jYz5YuZtt+wv485wN9L1rFl/l7Oee91cFvP5/h3XxLT+9IKfKvgyGtZYZn25kc14BT87PaZQ7dSPZok15PPSR6+Zylyai+8pDdFKlm3+gYoiArIzGb27xd9+FJ/PK19uBwMk3EuNia3zNRYM7cdMbK1i58zCb9uVXuYO1vv40e32VMWq8isvKA5qGAK549mvfct8OLbhh3ImcN6ADb2bvDNhu1qrva53c+1BhMW8v3clPR/VgwcZcfvKPb6psM3/9Psb2aVfNq6Pfj55bDMAp3dM5q0874mNNlfsgJHrpCD1ESfE1h+e9F50c1s+OiTGMP8kTVJc+/RVQMSpkTfz/mMf/eQFZU2ey72jtNyrVpbzc1hjmwXh9yggmD+yIMYbHrxjC/Rf15wdOv/t7qrkj9+jxihEph9w/hwdmrmVRzn7++Mn6at//168ur3Z9pCgrt9zy1koW5eSF9LqCooprEde+vJTed37MCwu3NnJ14mYK9Hr48IYzmHXTmVw7umfA+jG9wz+X6gntWgQ87tuh6hlDZSvuPifg8fAH5zWohsmPB/byOX9gR9677nT+ctkgtj4yma2PTOZbp+8+QJ/2FTXfNP5EWqdUXGe4YFAnrhqZxfTLhwCw90gR89fv8z1fUlbOgGmz6Xn7R5zz5wV4761atfswWW1Tfdstuf1snrnKc40jv6g0ou+W/XJTHm8v3cmPnl0c1PbDHphL1tSZPDm/6pdsbfdOSPRRk0s9nNy5FQAntvf0MrllQm/+Z1jX2l7SaMb1bcfTC3J8j4OZmal1SgIr75nAoHsr7iA9fKyEVsn1m6bP27vm6StPYeLJHXzrh3RL9y239Ou7/+ilA/nX19uYt24fN42vu5/+Nf/4hmHd03nyyqG+bplQ0TUUYO6aveQXlTL+pPY8d7VngLoJ/TvQNi2BvPxiBk6bXWXcnEiwbX8BV7+wxPd46baDnNK9Yr8+NT+HR2dVXEBukRTH0eOeI/MnPvP8f3FqVjrfbK1oktt96BidWieHu3RxAQV6A1w8pDOJcTGcN6BjlV4m4XJK93R+MiqLif07sHV/Ab3bB9cm3io5nud+PIyfv+QZ1njumr1cckqXOl5V1VvZOwBITYgNCPPq+Afq4K6t63zvv191iq+LZfa2g/zt0020rvSl06d9C3p3aMEHzoiUfTsEnrH88yfDOf/xqvcJNNTSbQc5ITONVin1n6s2GGP+MD/g8SVPLaJtWgLZd56DtTYgzAFfmPt765ens+NAIVc+v5ht+wvJ3naQC51AX737MGmJcXTPSOVQYTFb8goY0LlV2C7mS9PSf8UGiIkxXDCoU5OFOUBsjOGeC/pzWs8MLju1W0gXvMb3a+8bxuDhenYT9E7I0TOzcS6u+ju3fwd+43dT1H9W7GZGpbb6D244gwy/rqHdM1IDnj+5cysuHORpj5/yUuOMyV9QVMolTy1i0H2zOXwsfE05L3+11bf867Mq9kNefjF5+UX0uO2jGl97mXOGeMlQz5d01zYpvHedZ7TQ37y2nKypM5k4/XMmz/jS96Ux+L45/PDJRQE9gz5bv48Z8zayfX8hxyvdmVxaVtEldN2eI75uszXZe+R4tdcBNuw9yqHC4mpeIQ2lI/T/Mt5hDPLyi/hs3b6AibWDMahra3YfOsZ/rh8VjvL47YQ+jOiVwY+eXRwQnv+45lS2HygkIS6Gey7oxz+dseL9B0/zunZMT95fuZvZa/ayYsehoM4OavPGNzt8y4PuDV9Tzl3OMA2/GtuLW87tw2WnduXMxz4DPO3kXq/+/DRfb5ZXf3EaQ7qmk5wQy6OXDgwYwK3yPRHr/O4y9p+svNx6Ht/7wWrmr/eE9J/nbPA973/mBJ4vm7995vmi/cc1p1b7/1BRaRmnPeS5VvPNHeOZ9v5qHr5kAKt2HfZdG1h3/8RaOxlI6OqcscgYkwR8DiTi+QJ421p7T22viZYZi6LVsAfmkpdfBHjGag/2KN9ay9D75zDx5A48fPHAcJbIi4u2+nq8fHnrWXRJD+wSerCgmO0HChlUQ1gPvX+Ob+TLlfdMqPf1AqBK//gh3Vr7jn4by/GSMvreNQuAtfdNJDnBE3SLN+/nsmcquny+ee1IhvdoU+17VOfqF5awYEPVI+mz+7Zj3rp91bwidP9zSheytx3ks1vG+tYFe0/BmN6ZDOramitP68bwh+bx6s9P4/QT2jZKXZGmqWYsKgLGWWsHAYOBicaY4KbeEVeaffNo33Jtp/GV7Tp0jIOFJdX2xW9sPx7Z3bdcOcwB0lMTagxzgEVTx/mW/XvN+LPW1jwk8a7DZE2dyROfbfK103u7iC7ffoiB0z6p8bX1sWrXYQBG9szwhTnAaT0rxgZ6/IohIYU5wIs/He7refTxjWf6/tt7w/z5qxuUHwC8tXQnW/IKfHcJL9lyIOjXLtiQy4x5G3nemcjFe+YRDl9v3s+Ih+ZF9d3EdQa69fB2L4h3fjRNfQSrfCq+ZnfFmDBFpWXVDgX8yMfrOONRz+n/UL/eLOFijOGL35/F0jvH1+v1/qfyn2+ovj/3L17KpsdtH7EoJ4+i0sCxeWZ+9z3gmTVq3Z6j/GBwJ35+Zk96Znra7I8cL6XHbR9hrWXumr0Ul5az+9AxDhUWkzV1JtPnbqjyebXxDucw44ohVZ5bftc5fHbLWC4Y1KnKc6E4qWNLercPvIg8LCvwC2L2zaNZc9+5Aev6dmjBlofP47NbxvKGM43iFcO78c6vRgZs93/PLSZr6kz+9++eeyQSarjQ+odLB3LLhMDeTn9fsBnwXCMKxzzHAJc/8zV7jhzn1ne+5VhxGaVl5Vhr+f7wMd+gd08vyGFbiBPDu0lQbejGmFhgKXAC8IS1Nnxfo9Ikltx+NsOdNs7pczdUmbzjV2N7sTk3n7vO78fGffkBXSUr9ywJl64NHOhs/QMT6XPnLN5ZtpOEuBgevngA4PkCe/aLzcxd6zlK9bbp+reNx1W60N3ZGUvn4xvPpM+ds3zrn5yfwx+qucFp+tyNXHN6VkCf+9o8MNNzu3513VDTUxOqTIbSWFolx3PN6VmM7t2Ws/q08zW/Lb79bO77YA0P/XCAr2dPj7ap9GibWuM1hG93Hg54vOFBz4BxhwqLiYuNIWdfPv9esYuLBnfmu12HgapfemXllu92HWZgl4Zd96jM/6ar95bv4r3lu6psc93YXjw5P4dHPl7Ho5cM4LJTuzVqDU2hzjb0gI2NaQ28B9xgrV1V6bkpwBSAbt26nbJt27bGrFPCpD7jp0RS/+7ed37sO8VeOHUcGakJvrbqyl79xWmc3qst6/YcYeL0LwKeq3wBL5j99vDFA7hieNVQKCgq5YOVuxnfrz1t0xL5+4IcX6+jpti3j81ax5Pzc/jZGT246/x+jfKe1e2P2n6XkrJy/ufprxh1Qoav//yPR3bnpa88uTHvd2MCpoBsqEWb8hrUnDP75tFVzm4aW2O0oYcU6M6H3gMUWGv/WNM2uigaOUIJ9Jm/OYPe7VsQH2F9lv1/x6yMFLbur5gb9tPfjeH+D9fw2fpcfjqqB7ec2ztgZqi1900ECGjXBnh32U5+++bKOj/bP9Ty8os4VFjCc19s5nWn58wfLh3I/3v7WwD+evlgLhrcuR6/YfNbuu0g32w9wFUjuvPJ6j38YHBnYoLszvvusp2M6Z1JRlqi779VbIwh56HzGlRTflEp6/ccoX+nVr4v8a5tktlx4Fi93i+UDgT10SSBbozJBEqstYeMMcnAbOBRa+2HNb1GgR45vt15iAv/tpBRJ2Tw/NWn8lXOfmJiTMDdil6RdGTub+m2A1zy1FcB6/p1bMnEkzv4uj32vvNjTuvRhv6dWvmal16fMoIRPaufsMRay8JN+7nyec9RX0JsDJ/cPJoWSXEkxMWEPK9rv44t+ejGM0P91aLO/PX7uMYZcG39AxNrHXiuLtf/a5nvWojX0jvH8+WmPEb2yuDVxduZPndjwPMLp45jzuo95OYX+c4c/FXX46qxNFWgDwReBGLxXER901p7X22vUaBHvrJyS4yBwuIykuNjgz7acqsdBwo5//EvfX3bK385VT5T+W7ahICpB2uycFMehcVlnONMfuJ16oNzyT1aFHR9/75+VIP7y0eL0x6ay94jRfRp34JP/Hpkef159npmfLqJJbefTbuWSYCn2+fAabOZccVg32idE6d/HtD3/pdjejF1Ut+A9yoqLSP/eCnLtx+iqLScyQOrjvT57+W7uMlv+OlwHdg0S5NLMBTo4laXPLWIK0d044dDAoc9OP3heew+XDEKZUP/aP37lXtNHtiRmd96jhjvu6g/4/q2Iy+/WEFeif++u3ZMT26bVDGiaOUv3pyHziO/ylC5AAAH3ElEQVQ2xjDkvtkcLKy4EW32zaP50bNfM6RbOnOcC/7+/ftD5d+fX4Eu4nKHj5X4Bi9ryB++v8rhs/j2s8lMS/QNjBbpZz3h9OMXlvC53w1Rd04+iVEntGXSXwMvVndrk0K7Folkb6t+btp7LujHvR94RpxsaBBPn7uB6XM3hq3ZRYEu0ohmrdpDz8zURuvNkL31AK98vY0//e/gJh3vJ1pc/+oy3xkNwFUjuvPy16H1npt105m0So5n75GiBp8Jrd59mMkzvuS2SX25dkyvBr1XdRToIhK1rLX86pVlzFq9J2D9t9MmcLy4zHcfhdfsm0eTlZHKm9k7uPPfq7j7/H789IwejVqT96zr09+N8Q1Qt2b3EfLyixjdwPkQmurWfxGRJmeM4emrTuGJHw0NWN8yKZ52LZPY7HRrvGxYV7Y+Mpne7VuQEBfD/53WjY0PTmr0MAdom+a58esvfr1jzpvxBT9+YUnY7nANhUZbFBFXG927YrAu/3CPiTHVtosbY4iPDU8T11e3jePEOz5m2baDfPTd9wF3oM5Zs5cJ/WufIyDcFOgi4motkuKZdoHnjtbquhU2Je9NdbsOHeO6fy0LeG7Ky0ub/V4NNbmIiOtdM6oH14xq/CaU+kiKrzk2r68U8k1NR+giIiFYeOs4nvtyC2mJcRSXljOgcysS4mL48QtLmPnd98wot83Wq0mBLiISgoy0RG6dGHjHaUlZOZ1bJ/PwxQOatYuqAl1EpIHiY2NY6DepSnNRG7qISJRQoIuIRAkFuohIlFCgi4hECQW6iEiUUKCLiEQJBbqISJRQoIuIRImwjIdujMkFQhuJvkJbIK8Ry2lMbq4NVF9DuLk2UH0N4ebaoKK+7tbaBg2qHpZAbwhjTHZDB3kPFzfXBqqvIdxcG6i+hnBzbdC49anJRUQkSijQRUSihBsD/ZnmLqAWbq4NVF9DuLk2UH0N4ebaoBHrc10buoiI1I8bj9BFRKQemjzQjTFbjTHfGWNWGGOynXVtjDFzjDEbnX/TnfXGGDPDGLPJGPOtMWZo7e9er3peMMbsM8as8lsXcj3GmKud7TcaY64Oc33TjDG7nH24whhznt9ztzn1rTfGnOu3fqKzbpMxZmoj1dbVGPOZMWatMWa1MeZGZ32z779aanPLvksyxiwxxqx06rvXWd/DGLPY2Q9vGGMSnPWJzuNNzvNZddUdpvr+aYzZ4rf/Bjvrm+NvI9YYs9wY86Hz2BX7rpb6wr/vrLVN+gNsBdpWWvcYMNVZngo86iyfB3wMGGAEsDgM9YwGhgKr6lsP0AbY7Pyb7iynh7G+acAt1WzbD1gJJAI9gBwg1vnJAXoCCc42/Rqhto7AUGe5BbDBqaHZ918ttbll3xkgzVmOBxY7++RN4HJn/dPAr5zl64CnneXLgTdqqzuM9f0TuLSa7Zvjb+O3wKvAh85jV+y7WuoL+75zS5PLRcCLzvKLwA/81r9kPb4GWhtjGnXab2vt58CBBtZzLjDHWnvAWnsQmANMDGN9NbkIeN1aW2St3QJsAoY7P5ustZuttcXA6862Da3te2vtMmf5KLAW6IwL9l8ttdWkqfedtdbmOw/jnR8LjAPedtZX3nfeffo2cLYxxtRSd7jqq0mT/m0YY7oAk4HnnMcGl+y76uqrQ6Ptu+YIdAvMNsYsNcZMcda1t9Z+D54/RKCds74zsMPvtTup/Y+ysYRaT3PU+Wvn9OwFb5NGc9bnnMYOwXMk56r9V6k2cMm+c07JVwD78Pyx5gCHrLWl1XyWrw7n+cNARlPWZ6317r8Hnf33F2NMYuX6KtURrvqmA78Hyp3HGbho31VTn1dY911zBPooa+1QYBJwvTFmdC3bVjfbanN2y6mpnqau8ymgFzAY+B74k7O+WeozxqQB7wA3WWuP1LZpDXWErb5qanPNvrPWlllrBwNd8BwZnlTLZzV7fcaYk4HbgL7AqXiaAm5t6vqMMecD+6y1S/1X1/I5TbrvaqgPmmDfNXmgW2t3O//uA97D8z/yXm9TivPvPmfznUBXv5d3AXY3QZmh1tOkdVpr9zp/bOXAs1ScJjZ5fcaYeDyB+S9r7bvOalfsv+pqc9O+87LWHgLm42k/bW2M8U7e7v9Zvjqc51vhaYpryvomOk1Z1lpbBPyD5tl/o4ALjTFb8TSBjcNzROyWfVelPmPMK02y7+pq2G/MHyAVaOG3vAhPm9AfCLyI9pizPJnAiwVLwlRXFoEXHUOqB8+37RY8Fy7SneU2Yayvo9/yzXjaAQH6E3iRZzOei3pxznIPKi7s9W+EugzwEjC90vpm33+11OaWfZcJtHaWk4EvgPOBtwi8sHeds3w9gRf23qyt7jDW19Fv/04HHmnmv42xVFx0dMW+q6W+sO+7Ris8yF+up7MDVwKrgTuc9RnAPGCj828bv1/8CTxti98Bw8JQ02t4Tr1L8Hwj/qw+9QA/xXNRZRPwkzDX97Lz+d8C7xMYUnc49a0HJvmtPw9PT48c735vhNrOwHMK+C2wwvk5zw37r5ba3LLvBgLLnTpWAXf7/Y0scfbDW0Cisz7JebzJeb5nXXWHqb5Pnf23CniFip4wTf634bz3WCoC0xX7rpb6wr7vdKeoiEiUcEu3RRERaSAFuohIlFCgi4hECQW6iEiUUKCLiEQJBbqISJRQoIuIRAkFuohIlPj/w4pPNcbYX4gAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.sched.plot_loss(n_skip=500)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/fabiograetz/anaconda3/envs/deeplearning/lib/python3.6/site-packages/torch/nn/functional.py:995: UserWarning: nn.functional.tanh is deprecated. Use torch.tanh instead.\n",
      "  warnings.warn(\"nn.functional.tanh is deprecated. Use torch.tanh instead.\")\n"
     ]
    }
   ],
   "source": [
    "x, y = next(itertools.islice(iter(val_dl), 25, None))\n",
    "probs, attns = learn.model(V(x), ret_attn=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds = to_np(probs.max(2)[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pourquoi ? _eos_\n",
      "why the best source ? _eos_\n",
      "why ? _eos_\n",
      "\n",
      "à qui pouvez - vous faire confiance ? _eos_\n",
      "who can you trust ? _eos_\n",
      "who can you trust ? ? _eos_\n",
      "\n",
      "pourquoi ? _eos_\n",
      "why or why not ? _eos_\n",
      "why ? _eos_\n",
      "\n",
      "quels types de projets ? _eos_\n",
      "what types of projects ? _eos_\n",
      "what types of projects are _eos_\n",
      "\n",
      "qui et pourquoi ? _eos_\n",
      "which ones and why ? _eos_\n",
      "who and why why why ?\n",
      "\n",
      "quel est le ratio des étudiants étrangers et canadiens dans votre établissement ? _eos_\n",
      "why is this important ? _eos_\n",
      "what is the ratio of of foreign and and in in your institution ? _eos_\n",
      "\n",
      "qu’ est -ce que l’ intégrité commémorative ? _eos_\n",
      "what is commemorative integrity ? _eos_\n",
      "what is integrity integrity ? _eos_\n",
      "\n",
      "qu’ est -ce que le financement de projet ? _eos_\n",
      "what is project funding ? _eos_\n",
      "what is funding project ? _eos_\n",
      "\n",
      "quels sont les avantages ? _eos_\n",
      "what are the benefits ? _eos_\n",
      "what are the benefits ? _eos_\n",
      "\n",
      "pourquoi l’ inspection est-elle nécessaire ? _eos_\n",
      "why is inspection necessary ? _eos_\n",
      "why is the inspection necessary ? _eos_\n",
      "\n"
     ]
    }
   ],
   "source": [
    "start = 100\n",
    "for i in range(start, start + 10):\n",
    "    print(' '.join([fr_itos[o] for o in x[:, i] if o != 1]))  # Input\n",
    "    print(' '.join([en_itos[o] for o in y[:, i] if o != 1]))  # Target\n",
    "    print(' '.join([en_itos[o] for o in preds[:, i] if o != 1]))  # Prediction\n",
    "    print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/fabiograetz/anaconda3/envs/deeplearning/lib/python3.6/site-packages/torch/nn/functional.py:995: UserWarning: nn.functional.tanh is deprecated. Use torch.tanh instead.\n",
      "  warnings.warn(\"nn.functional.tanh is deprecated. Use torch.tanh instead.\")\n"
     ]
    }
   ],
   "source": [
    "x, y = next(itertools.islice(iter(val_dl), 21, None))\n",
    "probs, attns = learn.model(V(x), ret_attn=True)\n",
    "preds = to_np(probs.max(2)[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {},
   "outputs": [],
   "source": [
    "i = 103\n",
    "attn = to_np(attns[...,i])\n",
    "input_sequence = ' '.join([fr_itos[o] for o in x[:, i] if o != 1])\n",
    "output_words = [en_itos[o] for o in preds[:, i] if o != 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"de quel type d' autorisation ai - je besoin ? _eos_\""
      ]
     },
     "execution_count": 212,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_sequence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['what', 'type', 'of', 'authorization', 'do', 'i', 'need', '?', '_eos_']"
      ]
     },
     "execution_count": 213,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output_words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3QAAAJCCAYAAACIxIW/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xt83Hd95/vXZ66SRpJ1sXyTL3KCSeJASIIS0nLpAgEC3Sa0QDew7IGWbk5bcuAsLY+Gspu29LSnhV5PN21JC1u2LaQB2q3bmg0hTQq0JLFDQhLbceL4Kl9lyfaMZI00o/meP2ZGHkujuc/8ZjTv5+PhRzQzP818mOBvfp/v9/P9fM05h4iIiIiIiLQen9cBiIiIiIiISGWU0ImIiIiIiLQoJXQiIiIiIiItSgmdiIiIiIhIi1JCJyIiIiIi0qKU0ImIiIiIiLQoJXQiIiIiIiItSgmdiIiIiIhIi1JCJyIiIiIi0qICXgew2OrVq93IyIjXYYhIjT311FNnnXNDXsdRDY1PIiuPxiYRaUbljE1Nl9CNjIywe/dur8MQkRozsyNex1AtjU8iK4/GJhFpRuWMTSq5FBERERERaVFK6ERERERERFqUEjoREREREZEWpYRORERERESkRSmhW+TAmRiplPM6DBGRy5ybnmM8Nut1GCIiJXPOceBMzOswRFY8JXQ5TkfjvP33v83Xvz/mdSgiIpf55b97jo995WmvwxARKdl3D5zl1t/7NgfHp7wORWRFU0KX40x0lpSD77084XUoIiKXOXkhzpGJaa/DEBEp2ckLcQDGzs14HInIyqaELkcsngBg15FJjyMREblcNJ5gfGoW51QSLiKtIRZPAjA5PedxJCIrmxK6HNFMQndscoZTmVklEZFmEIsnScw7LswkvA5FRKQkU5mEbkIJnUhdKaHLEZ1JLvy867BW6USkeWQrCM6oMYqItIjsuDU5rXFLpJ6U0OXIrtAFfMZuJXQi0iTmkiniiRSAOl2KSMtQyaVIYyihyxHNDDw3jQyw6/A5j6MREUnLznIDnImpHFxEWkNsNj12TUwpoROpJyV0OaIzCXrCAV53xQAvnIourNiJiHgpO8sNWqETkdahFTqRxlBClyMWT9LbGeSmkQFSDp4+et7rkERELptcOhNVQicirSGqhE6kIZTQ5YjGE/R0BLh+Ux9+n7HrkPbRiYj3Lluhm1JCJyKtYSozGaUulyL1pYQuR3QmQW9HkEg4wLUbetXpUqSFmdltZrbfzA6Y2T15Xv+wmY2b2TOZPz/jRZylyO6hi4T8KrkUkZaRnYy6MJMgMZ/yOBqRlUsJXY50yWUAgNEtAzxz7DxzSQ1AIq3GzPzAfcA7ge3A+81se55L/8Y5d33mz583NMgyZI9UuXJNt44tEJGWEYsn6QimbzXPXdQqnUi9KKHLkS65DAJw89Z+ZpMpnj9xweOoRKQCNwMHnHMHnXNzwAPAHR7HVLHsHrorh7q1QiciLSExn2ImMc/IYATQPjqRelJClyNdcpleoXvtlgEAnUcn0pqGgWM5j8cyzy32HjN71sy+ZmabGhNa+bJlSyODES7MJIgn5j2OSESksKnMuLVlsAuASR1dIFI3SugyUinH1Gy6yyXAUE+YrasjPHlI59GJtCDL85xb9PgfgBHn3HXAt4Av5X0js7vMbLeZ7R4fH69xmKWJxtNHqqxbFQbgrBqjiLSslbS/t5Cp2UsTUaDGKCL1pIQuY3ouScpBT2aFDmB0Sz9PHZkklVp8HygiTW4MyF1x2wicyL3AOTfhnMtmRn8GvDbfGznn7nfOjTrnRoeGhuoSbDGxeJKejgBDPemETmWXIq1ppe3vLSRbKr45u0KnhE6kbpTQZWTPSunN7KEDuGlkgHMXExw8O+VVWCJSmV3ANjPbamYh4E5gR+4FZrY+5+HtwL4GxleWWGZ/75qeDgA1RhFpXStqf28h2VLxTf1dmGmFTqSelNBlZNuCZ0suAUZH+gHYdVhllyKtxDmXBO4GHiKdqD3onNtjZp8xs9szl33MzPaY2Q+AjwEf9iba4qIz6Q68WqETaXk129/bDOXghWQTur6uIH2dQSanNW6J1IsSuoxsW/DcksutqyOs7g7pPDqRFuSc2+mce6Vz7krn3G9knrvXObcj8/OnnHPXOude45x7s3PuBW8jXl5sNr1CNxgJYaYVOpEWVrP9vc1QDl5IdqK8pyPIQCSkkkuROlJClxGdyazQ5ZRcmhmjWwbYrRU6EfFQdg9dwO9jMBLSCp1I66rZ/t5ml12h6+kIMBgJM6EulyJ1o4QuIza7tOQS0mWXRycvcjoa9yIsEZHMkSrpsWl1d1gJnUjrWlH7ewvJdrns6QhohU6kzpTQZeQruYR0YxRAZZci4gnn3MIKHcCa3g7GY5pgEmlFK21/byHReIKQ30c44GegWwmdSD0Fil/SHi7Vel/+lWzf0Etn0M/uw+f499dt8CI0EWlj8USKZMrRk1mhG+oOc+B0zOOoRKRSzrmdwM5Fz92b8/OngE81Oq5ay52IGoyEOHdxjlTK4fPl20YoItXQCl1GNJ6kI5ieScoV9Pu4cUufVuhExBPRRZNNa3rDjE/N4pzOxxSR5pWb0A1EQqQcnM/0KxCR2qoqoTOz28xsv5kdMLN78rz+s2b2nJk9Y2bfXebwzKYQnUkszIAvNrplgH0nowureCIijbL4SJWh7jCJecf5ixqPRKR5Zc/PhHRCB+joApE6qTihMzM/cB/wTmA78P48CduXnXOvds5dD3wW+L2KI62zWDxJb0f+CtSbRgZIOfj+0fMNjkpE2l00fvn+3oWz6KZ0YyQizWvqspLL9LilTpci9VHNCt3NwAHn3EHn3BzwAHBH7gXOuWjOwwhLz1ppGtF4YkmHy6zrN/fh9xm7VXYpIg126UiVTMllJqE7E1VCJyLNa3HJJaDGKCJ1Uk1TlGHgWM7jMeB1iy8ys48CnwBCwFuq+Ly6is4kWNUVyvtadzjA9vW92kcnIg2XPcspe2zBpRU6dboUkeYViyfoDqfHrcHu9P3VhBI6kbqoZoUuX5uiJStwzrn7nHNXAr8E/Ne8b2R2l5ntNrPd4+PjVYRUuUIll5A+j+6ZY+eZS6YaGJWItLtLh/Omb4zW9HYAWqETkeaWu0LX36UVOpF6qiahGwM25TzeCJwocP0DwLvzveCcu985N+qcGx0aGqoipMoVKrkEuHlkgHgixZ4TFxoYlYi0u8VdLiMhP51Bvw4XF5GmlUo5puYuTZSHAj56OgJK6ETqpJqEbhewzcy2mlkIuBPYkXuBmW3LefijwEtVfF5dRWeSCyVN+bxmUx8Az5+ILnuNiEitxeIJ/D6jK5Q+UsXMGOoJqymKiDStqbkkznFZ9/DBiA4XF6mXivfQOeeSZnY38BDgB77onNtjZp8BdjvndgB3m9mtQAI4B3yoFkHXWjwxz9x8asmh4rnW9XYQ8vsYm7zYwMhEpN1ly5bMLlW5r+kJq+RSRJrW1KLuvJBujKKETqQ+qmmKgnNuJ7Bz0XP35vz88Wrev1Gii855ysfnM4b7Ozl2TgmdiDRO+ozMy4fqoZ4wL52Z8igiEZHCsnt/uy9L6MIcPz/jVUgiK1pVB4uvFNGZbBe5wvntxv5Ojk1qMBKRxkk3bLp8sim9QqculyLSnGILe38Xl1yqskCkHpTQcWngKbSHDmDTQJdW6ESkoXI7xWUN9YSJxpPEE/MeRSUisrxYvpLL7nTJpXNNeySxSMtSQgdEs+c8dRZeods80MX5i4mFBFBEpN6i8cRls9xw6Sy6s2qMIiJNaGErS05CNxgJkZh3xGaTXoUlsmIpoSO9RwVKWKHr7wJQ2aWINEz+ksvMWXQ6ukBEmtDU7OXnZ0K6KQrA5JQao4jUmhI6lh7cu5xNA50AKrsUaQFmdpuZ7TezA2Z2T4Hr3mtmzsxGGxlfqdIrdEtLLgGdRSciTSlvyWUmoZtQp0uRmlNCR26Xy8Ill5dW6JTQiTQzM/MD9wHvBLYD7zez7Xmu6wE+BjzR2AhLk0o5pmaTSxo2rckkdFqhE5FmlD0/szPoX3huMJIet3R0gUjtKaEjXXIZWDTw5NPXFaQ7HGDsnEouRZrczcAB59xB59wc8ABwR57rfh34LNCULSOzh/MuPlJlIBLCTCt0Iq1opVQPFBKLJ+kOX35+5kB3puRSnS5Fak4JHfkP7s3HzDJHF2iFTqTJDQPHch6PZZ5bYGY3AJucc/9Y6I3M7C4z221mu8fHx2sfaQH5ypYAAn4fg5GQEjqRFrNSqgeKydedd1AllyJ1o4SOdMlloUPFc+noApGWkG92ZqFXtpn5gN8HfqHYGznn7nfOjTrnRoeGhmoYYnHZhk359vcO9XQwHmvKhUURWd6KqB4oJp3QXT5udQT9dIX8aooiUgdK6EjfNBXrcJm1qb+LY5MzOkdFpLmNAZtyHm8ETuQ87gFeBTxmZoeBW4AdzVbalF2hyzc+DfWEtUIn0npqVj3QzGJ5mjlBulxce+hEak8JHflLA5azaaCTmcS8SgZEmtsuYJuZbTWzEHAnsCP7onPugnNutXNuxDk3AjwO3O6c2+1NuPllz7zMNz6t6QmrKYpI66lZ9YCX5eDFxOJJesJLx63BSEj3TyJ1oISOTMllGSt0oE6XIs3MOZcE7gYeAvYBDzrn9pjZZ8zsdm+jK120QEI31BPm7NQsqZSqBURaSM2qB7wsBy8mNqsVOpFGKm1ZaoWLziSLHlmQtWkgk9Cdm+GGzf31DEtEquCc2wnsXPTcvctc++8aEVO5Fkou8+zxHeoOk5h3XJhJ0J9pNiAiTW+hegA4Trp64APZF51zF4DV2cdm9hjwi81WPVBMvj10AAORMC+envIgIpGVTSt0ZGu9S1uh29ifOVxcK3QiUmfLdbkEWNOrs+hEWs1KqR4oxDm37FaWwe4QEzq2QKTm2n6FLjmfYnpuvuSSy0g4wGAkpIROROouOpMgFPARDiw9I3OoO53QjcdmuWpdT6NDE5EKrYTqgULiiRTzKbfMCl2IeCLFxbkkXaG2vwUVqZm2X6G7VNJU+sCyUUcXiEgDROPJZSeb1vR2AHBGRxeISBMp1MxpIHsWnY4uEKkpJXQLJU2lrdABbOrv5NjkTL1CEhEB0jdGvct04B3qubRCJyLSLKIFSsWzh4urMYpIbbV9QpftIrfcTVM+mwa6OHF+hnl1lxOROooWOFIlEvLTGfRrD52INJVSVuiU0InUlhK6mUxCl6eL3HI2D3SRTDlOXtAqnYjUT6GGTWbGml4dLi4izaVQ5dNgJF1ZoLPoRGpLCV2B0oDlXDqLTgmdiNRPLF74SJWhbiV0ItJcpmaXv68a6M6u0GncEqklJXQLJZdl7KEbyBxdoMYoIlJH0ZkEPeHlx6Y1vWE1RRGRpnKp5HLp2BUJ+QkFfFqhE6kxJXQVlFxu6OvEZzCmowtEpI6WO8spSyt0ItJssiWX3eGlY5eZMRgJMakulyI11fYJXaGBZzlBv4/1qzo5dk4llyJSH4n5FDOJ+YKTTUM9YaLxJPHEfAMjExFZXrTIfVV/V0hNUURqrO0Tumg8QU84gN9nZf3exv5OHS4uInUTK2F/75qe9Fl0WqUTkWYRiyfoLnBfNdgdUsmlSI0poZtJllVumbVJh4uLSB0V2oeStXAW3ZQSOhFpDsVKxQciWqETqbW2T+jSbcFLL7fM2tTfxenorEqdRKQusit0hc7IzCZ0Z6JK6ESkOUwpoRNpuLZP6KLxRFkdLrOynS6Pn9c+OhGpvWzDpkIrdGu0QiciTSY2u/z5mQCDkRBTs0lmk5oQF6kVJXQzhc95Ws6mgexZdCq7FJHaK+WMzIFICDMYj+roAhFpDrF4smCjuYHM4eJapROpnbZP6IrNJC1n4XBxdboUkTrI7qFbVWCPb8DvYzAS1gqdiDSNUvbQAUzo6AKRmmn7hC46kyy4R2U5a3rChAI+nUUnInVRygodpPfRqculiDSLdG+CAiWX3emETit0IrXT1gmdc45YPFFRl0ufz9jY16lOlyJNysxuM7P9ZnbAzO7J8/rPmtlzZvaMmX3XzLZ7Eedysit0xc7IXNMT5owSOhFpErF44Yny7AqdEjqR2qkqoSvhhukTZrbXzJ41s0fMbEs1n1dr03PzpFzxGfDlbBzo4tikSi5Fmo2Z+YH7gHcC24H350nYvuyce7Vz7nrgs8DvNTjMgmLxJJGQn4C/8DCtFToRaRZzyRSzyVTB+6rBbMmlEjqRmqk4oSvxhulpYNQ5dx3wNdI3TU0j20Wuki6XAJv6tUIn0qRuBg445w465+aAB4A7ci9wzkVzHkYA18D4iorOlLa/N5vQpVJNFb6ILKPVqwcKKeX8zN6OIH6fMTmtiSiRWqlmha6UG6ZHnXPZjOdxYGMVn1dz0czAU0nJJcDmgS7OX0wsDGAi0jSGgWM5j8cyz13GzD5qZi+Tnmz6WINiK0mxxgJZa3rCJFOO8zMah0Sa3UqoHigke35moVJxn8/o79JZdCK1VE1CV9INU46PAN/I94KZ3WVmu81s9/j4eBUhlSdWYtOB5Vw6ukBllyJNxvI8t2QJyzl3n3PuSuCXgP+a9428Gp9mS9vfmz1cXGWXIi2h5asHCin1vmowElKXS5EaqiahK+mGCcDMPgiMAp/L97pz7n7n3KhzbnRoaKiKkMpTfcllOqE7qk6XIs1mDNiU83gjcKLA9Q8A7873gnfjU6krdB0AnInpLDqRFtDy1QOFlFJyCenGKFqhE6mdahK6km6YzOxW4NPA7c65pppCrrbkctNAJwBj2kcn0mx2AdvMbKuZhYA7gR25F5jZtpyHPwq81MD4iirW+jtLK3QiLaXlqwcKic2WtkI30K2ETqSWqknoSrlhugH4POlk7kwVn1UX1ZZcruoM0hMOcEwrdCJNxTmXBO4GHgL2AQ865/aY2WfM7PbMZXeb2R4zewb4BPAhj8LNq1jr76xsQqejC0RaQstXDxSSva8qVvk0GAmpy6VIDVWWyZC+YTKz7A2TH/hi9oYJ2O2c20G6xLIb+KqZARx1zt2+7Js2WLbkstKEzszSRxec0x46kWbjnNsJ7Fz03L05P3+84UGVyDlHtMQVuu5wgK6QXyt0Iq1hYTIcOE56MvwDuReY2TbnXLZioOmqBwpZOD+z2ApdJMSFmQSJ+RTBIkeziEhxFSd0UNIN063VvH+9ReNJOoI+wgF/xe+xqb+TQ2enaxiViLS72WSKxLwrebJJZ9GJtIYSJ8PvzmxXSQDnaLLqgULKaYoCcO7i3MI+YBGpXFUJXasrdY9KIZsGuvjOS2dxzpFZhRQRqUq5+3vX9ITVFEWkRbRy9UAxsXiCjqCv6KrbQCRdKj45rYROpBbaep07OlPaHpVCNvV3MpOY56za74pIjURnsvtQShuf1vZ2cOqCEjoR8dbUbLKkifKBzArdpO6dRGqivRO6eGnnPBWycBadOl2KSI1cav1dWkI33N/JifNxUqmWOa5KRFagaLy041YGu9MJnRqjiNRGmyd0pc0kFXLpcHEldCJSG6V2issa7utkbj7F2WntoxMR78RKvK9aWKFTQidSE22d0MVmElWXXG7sz55Fp06XIlIb0RIP583asCo9Dp04r7JLEfFOLJ6gJ1z8vqq/K4SZVuhEaqWtE7palFx2hQKs7g5phU5EaqbcMzKHMxNLxzWxJCIeipVYcun3GX2dQSZVVSBSE22e0JU28BSzsb9Le+hEpGbK3UO3oS+7QqeETkS8k+4eXtq4NRAJqeRSpEbaNqGLJ+aZS6ZK3qNSyKaBLo5N6kZKRGojOpPEZxAJlXZj1NsRoDsc4LgSOhHx0FQZvQkGI2Em1OVSpCbaNqEr95ynQjb1d3Li/Azz6jAnIjUQiyfoDgfw+Uo729LM2NDXoYRORDwzn3JMz81rhU7EA22b0F3qIld9yeXmgS6SKcfJC7qZEpHqldopLtdwX6dKLkXEM1OZ+6ruEpqiAAx0K6ETqZW2TeiiM5kVuhqVXAIquxSRmqikYdMGJXQi4qGFyqeSSy5DnLs4p/MzRWqgfRO67ApdZ21W6AAOT0xX/V4iIpU0bNrQ18m5iwkuziXrFJWIyPLK7c47EAmRcnA+M8EuIpVr24QuVuY5T4UM93XSHQ7wwslo1e8lIhKLJ8suB8+eialVOhHxwtRsNqEr7b4qe7j4xJSOLhCpVtsmdNGZ7B666hM6n8+4al0P+07Gqn4vEZHoTKLssSl7dMFxHS4uIh4o97iV4cyYNaZJKJGqtW9Ct9DlsvqSS4Br1vew71QU51QLLiLVKecsp6yFhE6Hi4uIB8otudw8mN6ucnRC5/iKVKttE7pYPIHfZ3QG/TV5v2vW9xKLJ9U2XESqkko5pmbL73K5tieM32cquRQRT2RX6LpLTOiGusN0hfwcUUInUrW2TeiiM+k9KmalnfNUzNXregFUdinSJMzsNjPbb2YHzOyePK9/wsz2mtmzZvaImW3xIs7FpueSpFz51QMBv491vR1K6ETEEwvN5kqcjDIzNg90cXRSDeVEqtW+CV0FbcELuXpdD4Aao4g0ATPzA/cB7wS2A+83s+2LLnsaGHXOXQd8DfhsY6PM71LZUvnjkw4XFxGvxOJJgn4jHCj91nLzQJdW6ERqoG0TulgFbcELiYQDbBnsYt8pJXQiTeBm4IBz7qBzbg54ALgj9wLn3KPOueydxOPAxgbHmFe5+1ByDfd1KqETaXKtWj1QzNRsgp6OYFmVTyOrIxydvKiz6ESq1LYJXSVd5Iq5Zl0vL6jkUqQZDAPHch6PZZ5bzkeAb9Q1ohKVezhvrg19nZy6EGdeN0ciTamVqweKqWSifPNAF7PJFGdiOrpApBrtm9DF65DQre/l0MS0DvYV8V6+KeK8WY6ZfRAYBT63zOt3mdluM9s9Pj5ewxDzK7f1d64NfZ0kU45x3RyJNKuWrR4oJhZP0h0ub9zakul0eWRC++hEqtG2CV2tSy4Brl7fg3Ow/5RW6UQ8NgZsynm8ETix+CIzuxX4NHC7cy5vFuScu985N+qcGx0aGqpLsLmq2UM33J89i057UkSaVM2qBxo92VRMJcetbBmIAHBkUmOWSDXaNqGLztS2KQrA9vXpTpcvKKET8douYJuZbTWzEHAnsCP3AjO7Afg86WTujAcx5hWdqfyMzGEdLi7S7GpWPdDoyaZi0hPl5d1XbejrIOAznUUnUqXaLlG1iOR8ium5+ZqXXA73ddIdDrBPnS5FPOWcS5rZ3cBDgB/4onNuj5l9BtjtnNtB+iapG/hqZhP/Uefc7Z4FnVFu6+9c2cPFdXSBSNMqt3rgR5arHmg2lVQ+Bfw+hvs7OaySS5GqtGVCNzVbeRe5Qnw+4+p1PUroRJqAc24nsHPRc/fm/Hxrw4MqQSWtv7O6wwFWdQY5fk4JnUiTWqgeAI6Trh74QO4FOdUDtzVT9UAxsQp7E6TPotMKnUg12rLkMjqTmQGvccklpBujvHAyhnPqMici5cs2bCqn9XeuDX2dWqETaVLOuSSQrR7YBzyYrR4ws2yFQG71wDNmtmOZt2sazjmmZivrTbBlUGfRiVSrLVfoLrUFr/3//KvX9xB7PMnYuRk2DXTV/P1FZGWrtmHTcF8HY1qhE2larVo9UMj03DwpR9ldLiHdGOXCTIILFxOs6qr9RLtIO2jPFbqFtuD1WaEDNUYRkcqkO8VVPjZt0OHiItJgsSruqzZnjy6Y1D46kUq1Z0K3UHJZ+xW6q9b2YIb20YlIRdIdeKtZoeskFk8uTFyJiNTbVLzy3gSXzqJT2aVIpdozoVsouaz9Cl0kHGDLQJcSOhGpSCyepCdc3QodwEkdXSAiDRKtIqHbnNmeosYoIpWrKqEzs9vMbL+ZHTCze/K8/iYz+76ZJc3svdV8Vi3FqmgLXopr1veq5FJEKlLtHroNfTpcXEQaq5qSy65QgKGeMEd0dIFIxSpO6MzMD9wHvBPYDrzfzLYvuuwo8GHgy5V+Tj1kD+7trkNTFICr1/VyeGKai3PJury/iKxc1e6h29ivw8VFpLFiVazQAYyo06VIVapZobsZOOCcO+icmwMeAO7IvcA5d9g59yyQquJzai4aT9ATDuD3VdYWvJhr1vfgnBqjiEh5kvMppufmq9pDN9QdJug3HV0gIg1TbUK3eSCikkuRKlST0A0Dx3Iej2Wea3rVljQVs9Dp8qQSOhEp3dRs9qao8hU6n89Yt6pDh4uLSMNUU3IJ6cYop6Jx4on5WoYl0jaqSejyLW9VdJq2md1lZrvNbPf4+HgVIZUm3UWufmedbOzvpCccUGMUESlLtbPcWcM6XFxEGmhqNonPIBLyV/T7Wwa7cA7GzmmVTqQS1SR0Y8CmnMcbgROVvJFz7n7n3KhzbnRoaKiKkEoTjSfq1hAFwMy4en0PL5xSQicipbswU5sOvBuU0IlIA8XiSbrDAcwq28qS7XR5+KwSOpFKVJPQ7QK2mdlWMwsBdwI7ahNWfdW75BLSjVFeOBnDuYoWLUWkDV3qwFv9Ct2paJzkfFNtXxaRFSpaZTOnLYMRAI5oH51IRSpO6JxzSeBu4CFgH/Cgc26PmX3GzG4HMLObzGwMeB/weTPbU4ugqxWN17fkEtL76GKzSca0j0VESlTtPpSs4b5OUg5ORdXpUkTqr9qJ8v6uID3hAEd1dIFIRaqaBnbO7QR2Lnru3pyfd5EuxWwq0Zlk1TPgxVyzvgeAfSejbMqUEoiIFJI9nLeaLpdw6Sy6E+fjbOzX+CMi9ZU+bqXyccvM2DzYpRU6kQpVdbB4K3LOVX3OUymuWteDGexTp0sRKVGtVuguJXSqEBCR+puaTVY9bm0Z7OKozqITqUjbJXTTc/OkXPUz4MV0hQKMDEbU6VJESlbLLpcAx5XQiUgD1KI3weaBCMfOXWQ+pd4DIuVqu4Tu3PQcUH0XuVJco06XIlKG6EyCzqCfoL+6obkz5GcgElJCJyINUYuEbstgF4l5x8kLGrdEytV2Cd1zxy8A6ZLIert6XS9HJi8ynTksWESkkFp24N3Q16GSSxGpu+xWlu5wlSWXmX4DKrsUKV/bJXRPHpp+yFWzAAAgAElEQVSkM+jnVcOr6v5Z16zvxTl44ZT20Yk0mpndZmb7zeyAmd2T5/U3mdn3zSxpZu/1IsbFYrPVNRbINdzXyXF12RWROptNpkjMu+pX6Fbr6AKRSrVdQvfEoUleu6W/6pKmUlydWQVU2aVIY5mZH7gPeCewHXi/mW1fdNlR4MPAlxsb3fKOTc4w1BOuyXtlDxfXWZgizaUVJ5sKiWaaOVXbPXxdbwchv48jWqETKVtbJXQXZhK8cCrKTSMDDfm8jf2d9HQE1BhFpPFuBg445w465+aAB4A7ci9wzh12zj0LNMXp29F4gj0nLnBzjcan4b5Opufmic6o5FukWbTqZFMhUwvNnKorufT7jI0DnRyd1Fl0IuVqq4TuqSOTOAc3b21MQmdmXLOulxd0dIFIow0Dx3Iej2WeK5uZ3WVmu81s9/j4eE2Cy2fXoUlSDm65crAm75c9umDsvGa7RZpIy002FVOr7ryQ3kenFTqR8rVVQvfEoUmCfuOGzX0N+8yr1/fwwqkYKbXhFWkky/NcRX8JnXP3O+dGnXOjQ0NDVYa1vMcPThDy+7hxc39N3m8453BxEWkaLTfZVEysRit0AFsGIxyduKhScZEytVVC9+ShSV6zsY+OoL9hn3nN+l6mZpOMqTmBSCONAZtyHm8ETngUS0m+d3CCGzbXbnzS4eIiTanlJpuKOTSRLpGsxf7fzQNdxGaTTGaOmBKR0rRNQjczN89zYxe4qUHlllnZxihPHZ1s6OeKtLldwDYz22pmIeBOYIfHMS3rwkyCPSei3HJFbcotAQYjIUIBn86iE2kuLTfZVMzDe08zMtjFyGBX1e+1JfMe6nQpUp62SeiePnqOZMo1bP9c1quHV3HlUIQ/euQAifmWKIcXaXnOuSRwN/AQsA940Dm3x8w+Y2a3A5jZTWY2BrwP+LyZ7fEq3icPpff3/lCN9s8B+HyWPrpACZ1IM2mpyaZiYvEE33v5LG/bvhazfIuP5ckmdDqLTqQ8bZPQPXFoEp/Ba7fUZn9KqQJ+H5965zUcPDvNA08ebehni7Qz59xO59wrnXNXOud+I/Pcvc65HZmfdznnNjrnIs65QefctV7F+vjBCUIBH9dvqu3+Xh0uLtJcWm2yqZjH9o+TmHe8/dp1NXm/jf1dmKHGKCJlqs0Jti3gyUOTXLO+l94abNot11uvWcMtVwzw+996iTtuGPYkBhFpXt97eYLXbu6v+f7eDas6+ZcXvWuWICJLOed2AjsXPXdvzs+7SJdiNr2H955mMBKqWTOnjqCfdb0dHNHRBSJlaYsVurlkiqePnWt4uWWWmfHpd21ncnqOP33sZU9iEJHmdP7iHPtO1Xb/XNZwfydnYrPMJudr/t4i0t7mkike3X+Gt16zBr+v+nLLrM0DXSq5FClTWyR0zx2/QDyR4nUeJXQAr964indfv4EvfPeQSqBEZMETddg/l5XtdHn6wmzN31tE2tsThyaIxZO8bXttyi2ztgx2qSmKSJnaIqF78lC6w+RNI94ldAC/+I6rcMDvfHO/p3GISPN4/OAE4YCP12xaVfP3zp5Fp8YoIlJrD+89TUfQxxtesbqm77tlMMJ4bJaLc8mavq/IStYWCd2uw5NcORRhsLv6M1KqsbG/i59+/Vb+7unjPH/8gqexiEhz+N7LE4yO9BMO1P58TCV0IlIPzjke3nuaN20bojNU27FrodOlVulESrbiE7r5lGPX4Ulu3lr7cqZK/Pybr6SvM8hv/NM+nKvoLFERWSHOTc/xwqkYt9RpfFq3qgPQ4eIiUlvPH49y8kKct21fW/P33jIQAdTpUqQcKz6he+FUlFg86en+uVy9HUE+/tZtfO/gBI/uP+N1OCLioScOTQD12T8H6Y5xq7vDSuhEpKa+ufcUPoO3XlP7hG6zzqITKduKT+gW9s81SUIH8IHXbWHr6gj/784XSOqwcZG29fjBSTqDfq7bWNvz53IN93dy4MxU3d5fRNrPw3tPMzoywEAkVPP3XtUZpK8rqKMLRMqw4hO6XYcnGe7rXNhL0gxCAR+/dNvVvHRmigd3j3kdjoh4JLt/LhSo31B827Xr2H3kHP/47Im6fYaItI+jExd54VSMt9eh3DJry0CXSi5FyrCiEzrnHE8emmyacstc77h2LTeN9PN7D7/I1Kw6OYm0m4mpWfafjtXl/Llc//mNW3nNxlX8t//1PGdi8bp+loisfN/cewqAt9f4uIJcmwcjSuhEyrCiE7qDZ6c5OzXn2YHihZgZv/yuazg7Ncu9f/+8GqSItJknMuXg9U7oAn4fv/uT13Nxbp5f/luNNSJSnW/uPc3V63oW9rrVw5aBLo6fn2Fyeo69J6I8vPc0f/Gvh/iNf9rLR//6+/zn/7mbl8dVSi6SFfA6gHpqxv1zuW7Y3M/H37qNP3zkJW7c3M8Hb9nidUgi0iCPH5ygK+Tnuo21P39usVes6eaT77iK/+ef9vH17x/nva/dWPfPFJGVZ3J6jt2HJ/nom19R18/ZMtjFfMpx468/fNnz4YCP4b5OJqbn+Ik//jfu/0+v5XV1nhQTaQUrOqHbdWiS1d0hrlgd8TqUZX38rdv4wdh5fu0f9rB9Qy83bu73OiQRaYD0/rkBgv7GFEr89Ou38s09p/m1HXv44SsH2dBE+4pFpDU8su80KUddjivI9fbt6zj85mlWdQYZ7utiuD/dC2F1dwgz4+jERX7qL57kg194gs++9zp+/AZNUkl7W9Ell08cmuTmrQOYmdehLMvnM/7gP1zPulUd/PxffZ+zU7NehyQidXZ2apaXzkzxQw2cWfb5jM+97zrmneOXvv6sSi9FpGwP7z3Nut4OXj1c38qCVV1BPvmOq7nrTVfyo9et5/pNfQz1hBfu5zYPdvG3P/d6RrcM8F/+5gf8wbde1JgmbW3FJnRj5y5y/PwMN400Z7llrr6uEH/6wddy7uIcd3/5+zrKQGSFe/xg+vy5W65o7Pi0ZTDCL7/rGr7z0ln++omjDf1sEWltM3PzfPulcd62fW1TTJSv6grypZ++mffcuJE/+NZL/MKDP2A2Oe91WCKeWLEJ3a7D6f1zzdgQJZ9rN6ziN3/81Tx+cJLPPbTf63BEpI6+9/IEkZC/7rPc+fzH123mjdtW85s793FkQuc8iUhpvnvgLPFEirdfW99yy3KEAj5+533X8QtveyV/+/Rx/o8vPMn5i3NehyXScCs2oXvy0Dl6OgJcva7X61BK9p7XbuQ/3bKFz3/7IDufO+l1OCJSJ48fnOCmrQMEGrR/LpeZ8dvvuQ6/z/jkV59lPqUyJREp7uG9p+gJB3jd1uZqQmJm/F9v3cYf3nk9Tx89z7vv+1d2/OAECVU7SRup6m7CzG4zs/1mdsDM7snzetjM/ibz+hNmNlLN55XjyUMT3DQygN/nfVlAOf7bv9/ODZv7+ORXf8CBMzGvwxFpWc06Pp2Jxnl5fLqh++cW29DXya/+2LU8eXiST//dczyy7zQnL8xoD4pIAzTr2FTIfMrxyL4zvPnqNYQCzbkWcMf1w/zVz7wOM+NjX3maN/72o/zxYwc4N60VO1n5Ku5yaWZ+4D7gbcAYsMvMdjjn9uZc9hHgnHPuFWZ2J/DbwH+oJuDlOOd46cwU335xnH95cZyXx6d572s31eOj6ioU8PHH//FGfuyPvsv/+ZdPce+PXctVa3tY2xtuipp1kVbQbONTrscbdP5cMT9x4zDfOzjBA7uO8cCuYwD0dQW5Zl0v2zf0cs36Xq5e18Mr1nTTEfR7GqvIStHMY9NizjmOTFzk2y+N888vnGFieq7u3S2rdfPWAR75xI/w2Itn+OJ3D/PZ/72f/++Rl/jxGzby068fYdvaHq9DFKmLao4tuBk44Jw7CGBmDwB3ALmD0h3Ar2Z+/hrw383MXI2mgc9Nz/HdA2f5zkvjfPvFs5yKxoH0mUs/84atfOiHW/Nct/WrOvmj99/Ih//Hk3zoi08C0NsR4JVre3jluh6uWpu+yertCNIR9BEO+AkHfXRk/hkO+FZ88uecI/v/Ipd9DDnPucx11X1O9ms07LLH6eey19iS5xb/fv73Lv/fUaG/OrkvLb4q9/eWvlbg85b5HrOPwwEfvuZcBfd8fJpLpjgdjXMqGufUhTino3FOXojzrwfO0hMOcO0Gb8vBzYzfed9r+JUf284Lp2LsOxll38koe0/G+OsnjhBPpDLXweaBLrat6eGVa7vZtrabbWt6WNvbQdBv+H1GwOcj4DcCPlvxY49IlTwfm5bjnCMaT/K9l8/y7ZfS91bHJmcA2NjfyUfesJV3XLuuniHUhM9nvOXqtbzl6rXsPxXjf/zrIb7+/TG+8uRRbt46wNbBCAPdIQYjIQZy/vR3hQgFfAR8RjDgI+jzLYxxGtek2VWT0A0Dx3IejwGvW+4a51zSzC4Ag8DZKj4XgANnpnjb7/8LzqWTnTdsW82btg3xxlcOMbwCzlf6oSsHefKXb2XfqSgvno6x/1SMl05P8U/PnuTLM8W705mBzwyfpW/cfJZOSrKPYVHyUWysWnxDn/tzoSRj4Zrc5y5PElzOxbmvXfpdl/d9xHs7P/ZGtnucmCzD0/Hp5/7qKb7x/Kklz3cG/axb1cHPvPEKT/bP5dPTEeSmkYHLOgLPpxyHzk7z4ukYL55Ojz0vno7x2P4zJIvsufMZ6RsgDIyFsSc7Ji0ZavJMkixW8BMLTkhUppQxLf/vFYql8sErz7d26bUK7zNrfXva6KG52H8LfnJ0I792x6saE0x5PB2bvvPSOD//19/HufTf83nncM4xn3Lk/tXuDgf4oSsHueuNV/DGbUNsGexqyaTmqnU9/NZ7ruOT77iKrzx5lJ3PneLR/WeYnJ4rOpblCvovjWlG+u9ddlxLP15+Yreiv2wVjmvZsevye7Ts77klzy15zwL3arVWcMK70JdW8++ztInyMt5yyX9DFl/78m+8q+aT4dUkdPkiWRxzKddgZncBdwFs3ry5pA+/YnWEX3z7VdxyxSCv2biqaW6OamlVV5Bbrhi8rDTLOceZ2Cwvn5liem6e2eQ8s4kU8UX/dC49OKcy/0w/vjRgL5dglfMXLN9qVd7fy65y5Rvw8qyAWc7v5XuezE1hvtfNWPIfndzriik44Lmlr+UbNPO915LPKTJIlPJ95n0t5zcXX5f7cMlrJfyHOt9q5ZrecNHf84in49Obr17D1et6WbcqzNreDtav6mRdbwe9nYGWuCny+4xXrOnmFWu6eder1y88n5hPcfjsNC+enmLy4hzz8ymSKZf+k/15PnuTmPk7kBmHnIPF90+F/r4tVul//OuR8FS68l7j+5CK9zx6cYNWn89b/gOv39TXwEjK4unYtK63g/fcuBG/Lz3B6/MZfrP0BLDP6Aj6uGlkgOs39RFcQfdVg91h7n7LNu5+yzbg0mrkuek5JqbnmJye4/zFORLzjsR8KvMnPa4l5lMkUpePaQuVQa7y+4BiKh3XLM81pUzkF65EKhhq2SpPlCr/zFK+s/J/r/R/R/UeHqtJ6MaA3E1qG4ETy1wzZmYBYBUwufiNnHP3A/cDjI6OlvSvy+czPvrmV1QQdmszM9b2drC2t8PrUESamafj00+Ott7+3VIE/T62re3RPhSRynk6Nm1b28Ov3n5tBWGvLGbGqs4gqzqDjKyOeB2OSNWqmX7ZBWwzs61mFgLuBHYsumYH8KHMz+8F/rneNeAiImh8EpHmpLFJRGqu4hW6TF333cBDgB/4onNuj5l9BtjtnNsBfAH4SzM7QHp26c5aBC0iUojGJxFpRhqbRKQeqim5xDm3E9i56Ll7c36OA++r5jNERCqh8UlEmpHGJhGptZWz41VERERERKTNKKETERERERFpUdZs+2zNbBw4UsavrKYGZ7PUiGLJT7Es1SxxQONi2eKcG2rA59RNmeNTO/47LoViyU+x5NeIWDQ2eUex5KdY8mu3WEoem5ouoSuXme12zo16HQcoluUoluaNA5orlpWkmb5XxZKfYslPsaxszfSdKpb8FEt+imV5KrkUERERERFpUUroREREREREWtRKSOju9zqAHIolP8WyVLPEAc0Vy0rSTN+rYslPseSnWFa2ZvpOFUt+iiU/xbKMlt9DJyIiIiIi0q5WwgqdiIiIiIhIW1JCJyIiIiIi0qJaNqEzs9vMbL+ZHTCzezyO5bCZPWdmz5jZ7gZ/9hfN7IyZPZ/z3ICZPWxmL2X+2e9hLL9qZscz380zZvauBsWyycweNbN9ZrbHzD6eeb7h302BWBr+3ZhZh5k9aWY/yMTya5nnt5rZE5nv5W/MLFTvWFYyjU8Ln63xaWkcGpvyx6KxqQE0Ni18tsam/LFofMofS/OPT865lvsD+IGXgSuAEPADYLuH8RwGVnv02W8CbgSez3nus8A9mZ/vAX7bw1h+FfhFD76X9cCNmZ97gBeB7V58NwViafh3AxjQnfk5CDwB3AI8CNyZef5PgZ9r9L+zlfJH49Nln63xaWkcGpvyx6Kxqf7fscamS5+tsSl/LBqf8sfS9ONTq67Q3QwccM4ddM7NAQ8Ad3gckyecc98GJhc9fQfwpczPXwLe7WEsnnDOnXTOfT/zcwzYBwzjwXdTIJaGc2lTmYfBzB8HvAX4Wub5hv1/ZoXS+JSh8SlvHBqb8seisan+NDZlaGzKT+PTsrE0/fjUqgndMHAs5/EYHv1LznDAN83sKTO7y8M4stY6505C+i8EsMbjeO42s2czZQUNKWHIZWYjwA2kZ1Q8/W4WxQIefDdm5jezZ4AzwMOkZ2zPO+eSmUu8/vvU6jQ+FabxKUNj05IYNDbVl8amwjQ25dD4tCSGph6fWjWhszzPeXn+wuudczcC7wQ+amZv8jCWZvMnwJXA9cBJ4Hcb+eFm1g18Hfi/nXPRRn52CbF48t045+adc9cDG0nP2F6T77JGxLJCaXxqHZ6NTxqbltLYVHcam1qH7p2Wj0XjUx6tmtCNAZtyHm8ETngUC865E5l/ngH+jvS/aC+dNrP1AJl/nvEqEOfc6cxfghTwZzTwuzGzIOlB4K+dc3+bedqT7yZfLF5+N5nPPw88RroOvM/MApmXPP37tAJofCqs7ccnjU2FaWyqG41NhbX92AQan4pp1vGpVRO6XcC2THeZEHAnsMOLQMwsYmY92Z+BtwPPF/6tutsBfCjz84eAv/cqkOwAkPHjNOi7MTMDvgDsc879Xs5LDf9ulovFi+/GzIbMrC/zcydwK+m69EeB92Yu8/T/MyuAxqfC2np80ti0bCwam+pPY1NhbT02ZT5X41P+WJp/fKq0m4rXf4B3ke548zLwaQ/juIJ0p6gfAHsaHQvwFdJLzgnSs28fAQaBR4CXMv8c8DCWvwSeA54lPSCsb1AsbyC99P0s8Ezmz7u8+G4KxNLw7wa4Dng685nPA/fm/P/4SeAA8FUg3Mj/H6+0PxqfFj5f49PSODQ25Y9FY1MD/mhsWvh8jU35Y9H4lD+Wph+fLBOQiIiIiIiItJhWLbkUERERERFpe0roREREREREWpQSOhERERERkRalhE5ERERERKRFKaETERERERFpUUroREREREREWpQSOhERERERkRalhE5ERERERKRFKaETERERERFpUUroREREREREWpQSOhERERERkRalhE5ERERERKRFKaETERERERFpUUroREREREREWpQSOhERERERkRYVqOaXzew24A8BP/DnzrnfWvT6h4HPAcczT/1359yfF3rP1atXu5GRkWrCEpEm9NRTT511zg15HUc1ND6JrDwam0SkGZUzNlWc0JmZH7gPeBswBuwysx3Oub2LLv0b59zdpb7vyMgIu3fvrjQsEWlSZnbE6xiqpfFJZOXR2CQizaicsamaksubgQPOuYPOuTngAeCOKt5PREREREREylBNQjcMHMt5PJZ5brH3mNmzZvY1M9uU743M7C4z221mu8fHx6sISUREREREpH1Uk9BZnufcosf/AIw4564DvgV8Kd8bOefud86NOudGh4ZauoxdREREZFlmdpuZ7TezA2Z2T57XP2xm42b2TObPz3gRp4i0jmqaoowBuStuG4ETuRc45yZyHv4Z8NtVfJ6IiIhIy6pX/wERaW/VrNDtAraZ2VYzCwF3AjtyLzCz9TkPbwf2VfF5nnv0hTPMJue9DkNEMkqY6f6Eme3NlH0/YmZbcl6bz5kB37H4d2V5+0/FePF0zOswRFqR+g/U0VwyxT+/cNrrMEQaruKEzjmXBO4GHiKdqD3onNtjZp8xs9szl33MzPaY2Q+AjwEfrjZgrxybvMhP/cUu/v7pE8UvFpG6y5npfiewHXi/mW1fdNnTwGim7PtrwGdzXptxzl2f+XM7UpKZuXk++IUnuPfvn/c6FJFWVLP+A7LUt/ad5qf/YjcHzkx5HYpIQ1V1Dp1zbiewc9Fz9+b8/CngU9V8RrM4fzEBwMvjGiREmsTCTDeAmWVnuhdKl5xzj+Zc/zjwwYZGuAJ96XuHGY/N0t8V9DoUkVZUav+BrzjnZs3sZ0n3H3jLkjcyuwu4C2Dz5s21jrMlTUzPATAem+UVa7o9jkakcaopuWwrsdl0Qnd4YtrjSEQko9SZ7qyPAN/IedyR6a77uJm9ux4BrjTReII/eexlAC7MJDyORqQlldR/wDk3m3n4Z8Br872RGsotFYunx6XzF+c8jkSksZTQlWh6Nr137vDZix5HIiIZpcx0py80+yAwCnwu5+nNzrlR4APAH5jZlcv8ro5Vyfjz7xziwkyCH75yUAmdSGXarv9AI8XiSQAmldBJm1FCV6KpzArdkclpUqm894wi0lhFZ7oBzOxW4NPA7Tmz3jjnTmT+eRB4DLgh34doFjxtYmqWL3znID/66vX88JWDxBMpNYkSKVO79R9otEsrdJpwkvZS1R66djKVWaGLJ1KcjsVZv6rT44hE2t7CTDdwnPRM9wdyLzCzG4DPA7c5587kPN8PXMzsUVkNvJ7LG6bIIn/y2MvMJOb5L297Jf/28lkAojNJhnr8Hkcm0lraqf9Ao2VX6M5Na4VO2otW6Eo0lRkkQGWXIs2gxJnuzwHdwFcXHU9wDbA7MwP+KPBbec6BkoyTF2b4n48f4Sdu3Mgr1nSzqjPdEEVllyLSTBYSOq3QSZvRCl2JpmdzErqJaX7oykEPoxERKGmm+9Zlfu/fgFfXN7qV44/++QDOOT7+1m0A9CqhE5EmpKYo0q60QleiqdkkPeEAIb9PnS5FpG0cPjvNg7uO8YGbN7NpoAuA3o50QheNK6ETkeZxaYVOCZ20F63QlWhqNklvZ5COoI/DZ5XQiUh7+INvvUjAb3z0La9YeC5bchnVCp2INJFsQqemKNJutEJXoql4kkjYz9bVEY5MaA+diKx8+0/F+PsfnODDP7yVNT0dC89rD52INKNsyaVW6KTdKKEr0fRcku5wgJHBCIcndHSBiKx8v/vN/XSHAvzsj1xx2fO9nenijguaBReRJuGcY2o2iVl6skn3adJOlNCVKBZPEgkH2LI6QjyR4kxstvgviYi0qGeOneebe09z15uuoK8rdNlr4YCfjqBPe+hEpGlMz82TcrCut4OU0x5faS9K6Eo0PZukpyPA1sEIAIe0j05EVrA/eewAA5EQP/WGrXlfX9UZVMmliDSNbLlltnmTji6QdqKErkRTs+mSyy2D6YHiiDpdisgK5Zxj9+FzvOXqNXSH8/fOUkInIs0k2xBlU382odM+OmkfSuhKNDWbLrnc0NdJyO/jkBI6EVmhTkdnmZie41Ubepe9prcjSHQmuezrIiKNlF2h25xZodNZdNJOlNCVwDmXLrkMB/D7jE0DnRw5q06XIrIy7TlxAYBrh1cte41W6ESkmUQzK3TZSqpz0xqfpH0ooSvBTCK90TaSKT3aujqiw8VFZMXacyKKGVyzfvkVOiV0ItJMFkouB1RyKe1HCV0JpjKDRHdHOqHbkjm6wDm1xBWRlef54xfYOhhZdv8cQG9nUAeLi0jTyJZcbujrwO8zJXTSVpTQlWBqNpPQZW5uRjJHF5yO6ugCEVl59pyIsr3A/jlIJ3Sx2STzOutJRJpAdoWutyNIX2dQXS6lrSihK8GShC5Tn62ySxFZac5fnOP4+Rmu3bD8/jlIl1zCpVlxEREvxeIJ/D6jK+SnryuopijSVpTQlSBbchlZSOjSZ9Ed1ll0IrLC7D0RBeDaIit02YRO++hEpBlMxdPHS5kZ/V0hNUWRtqKErgSLV+h0dIGIrFTPZztclpjQ6egCEWkGsXiSnkyvg76ukPbQSVupKqEzs9vMbL+ZHTCzewpc914zc2Y2Ws3neWVxQqejC0RkpdpzIsr6VR0MdocLXtebuXHSCp2ININoPElPR3qiqb8ryHntoZM2UnFCZ2Z+4D7gncB24P1mtj3PdT3Ax4AnKv0sr03PXt7lEtJll9pDJyIrzZ4T0aKrcwCrulRyKSLNIxZPLKzQ9Ue0QiftpZoVupuBA865g865OeAB4I481/068FkgXsVneSq2aIUO0p0udXSBiKwkF+eSHByfYnuRhiigPXQi0lxi8eRC5UBfV5DZZIqZuXmPoxJpjGoSumHgWM7jscxzC8zsBmCTc+4fC72Rmd1lZrvNbPf4+HgVIdXH9GySgM8IBy59XTq6QERWmn0nY6QcvKqEFbreTGlTVF0uRcrSLttVGi02m8gpuQwBOlxc2kc1CZ3leW5hucrMfMDvA79Q7I2cc/c750adc6NDQ0NVhFQfU/EkkUznpCwdXSAiK83ebEOU4eIrdF0hPwGfaYVOpAzttF2l0XKbovRnSsKV0Em7qCahGwM25TzeCJzIedwDvAp4zMwOA7cAO1pxpmlqdv6yckvQ0QUizaDYTLeZfcLM9prZs2b2iJltyXntQ2b2UubPhxobeXPacyJKX1eQDas6il5rZqzqDIWtxd0AACAASURBVCqhEylP22xXaSTnHLHMsQWQ7nIJqDGKtI1qErpdwDYz22pmIeBOYEf2RefcBefcaufciHNuBHgcuN05t7uqiD0wNZtYktBljy44PKFOlyJeKHGm+2lg1Dl3HfA10jdImNkA8CvA60jfYP2KmfU3KvZmlW2IkluNUIgSOpGy1Wy7ilwyk5hnPuUWSi4HIiq5lPZScULnnEsCdwMPAfuAB51ze8zsM2Z2e60CbAbTs/OXdbiES0cXaIVOxDNFZ7qdc48657KzLo+TriQAeAfwsHNu0jl3DngYuK1BcTelxHyK/adivKqEhihZPZ1BokroRMpRs+0qzd5/oJFi8XTzup6cpigA57RCJ20iUPyS5TnndgI7Fz137zLX/rtqPstLsdnkQke3XDq6QMRT+Wa6X1fg+o8A3yjwu8NLfoP0TRNwF8DmzZsrjbXpvXR6irn5FNtLaIiStaozyAXNgIuUo5ztKgDrSG9XWVLh5Jy7H7gfYHR0tK1bbscyzZkWErrOTMnltMYnaQ9VHSzeLqZnk/SEl+a+I6sjHJm4qKMLRLxRcKb7sgvNPgiMAp8r93ebvWlTrezJNkQpY4VOJZciZWub7SqNFM2s0GW774YCPrrDAa3QSdtQQleCdJdL/5LnRwa7mEnMcyamowtEPFBsphsAM7sV+DTpm6LZcn63new5EaUz6Gfr6kjJv9PbEVi4kRKR4tppu0ojTS0quYR02aX20Em7qKrksl1MzybpDucpuczc+Bw6O83a3uJd4USkphZmuoHjpGe6P5B7Qaa5wOeB25xzZ3Jeegj4zZxGKG8HPlX/kJvXnhMX2L6hF7+vtIYocGmFzjlXciMVkXbXLttVGunSHrpL92r9XSEldNI2tEJXhHOOqbkk3XlX6NIJ3RHtoxNpuBJnuj8HdANfNbNnzGxH5ncnSbcF35X585nMc20plXLszXS4LMeqziDzKcf03HydIhMRKW7xHjrIrtCp5FLag1boirg4N49zLOlyCemjC4J+49BZHV0g4oViM93OuVsL/O4XgS/WL7rWcWTyItNz8xUldAAXZpYe7SIi0iiLu1xCeoXu6KTuz6Q9aIWuiKnZ9CARyXOzkj66oEsrdCLS0ippiALQm0nodHSBiHgpFk9gBpFQbkIX5Jy6XEqbUEJXRDahW272eetghEM6i05EWtjzx6ME/cYr1/aU9Xu5K3QiIl6JxpN0hwP4cvYA93WFiMaTJOdTHkYm0hhK6IrIdk5aLqHbMqijC0Skte05cYFta3oIBcr7T4ISOhFpBrF4cuHIgqz+Lo1P0j6U0BUxXWyFbrWOLhCR1uVcZQ1R4FJCp5JLEfFSLL50H29/JH24uBqjSDtQQldErMAeOrh0dMFhlV2KSAs6HZ1lYnqOVw2Xt38OLh3iqxlwEfFSLJ68rCEKpEsuAc7r6AJpA0roisiu0C0eKLKyRxccVmMUEWlBzx/PNkQpf4WupyOAmVboRMRbsdnEkvu0gS6t0En7UEJXRKEul3Dp6ILDE2qNKyKtZ8+JKGZwzfryEzqfz+gJB7RCJyKeSq/QXb6Hri+zh06Hi0s7UEJXRLEul9mjC1RyKSKtaM+JC2wdjCw7aVVMb2eQaKZ5lIiIF/KVXGb30KnkUtqBEroipuJJAj4jXKD7m44uEJFWtedElGsr2D+XtaozqBU6EfHUVJ4VukjIT9BvKrmUtqCErojp2STdHQHMbNlrdHSBiLSic9NzHD8/U9H+uSwldCLipXhinrn51JIVOjOjryukFTppC0roiojNJomECpci6egCEWlFe09GgcoaomSt6gyqKYqIeCaWKfnuzdO8rr8ryLlpjU+y8imhK2IqT132YlsGdXSBiLSePSeyHS4rL7ns7dAKnYh4JxZPjz+LSy4hfXTBpFbopA0ooStiei5ZtFnA5oEuAI6dm2lESCIiNfHs2AU2rOpgINM8oBKrupTQiYh3sit0+Sbf+7uCKrmUtqCEroipeHLZDpdZQz1hAM5OqeRSRFrD3hNRvvH8Kd589Zqq3mdVZ5DZZIp4Yr5GkYmIlO5SQrd0ha6/K6SmKNIWlNAVMTVbPKGLhAN0Bv1MKKETkRaQnE/xS19/lv6uEJ98x1VVvVdvZ/omKhrXTZOINN6lksul92rZpihqWicrnRK6IkpJ6ABW94Q4O6VlfRFpfl/810M8d/wCv3b7tfR1VV5uCZcaEagxioh4IbtCl+9erb8r+P+zd+/xcdXnve8/z9x0lyxLsjGWje8EUwgXcWkAw0lIQi6FtCEpSbtDsrNDsxt22s05e5fs9JCUNOfk0iZtT9ltSEOTXlIgl7Zu4oZCSAIkIWAI2NgG37hYvkqybGkkzUgz8zt/zBp5LI+kkTSz1mj0fb/QS6M1a2b9GFs/r2c9v/U8jKUdQ6NaQSDVbU4BnZndYGYvmdleM7uzwPMfNbPtZvacmT1hZhvncrwgDCXTRTXcbWuo0ZJLEal4r/YN8aWHd3P9eUt5+wVnzfn9WrwMne6jE5Eg5FYHNE+y5BKyLVpEqtmsAzozCwP3AG8DNgLvKxCwfdM5d4Fz7iLgC8CXZj3SAGQyLpuhm6bKJUB7Y40ydCJS0ZxzfOK724mGQvzxu35lyv6axVJAJyJBGs/QFVxymZ2fTug+Oqlyc8nQXQ7sdc7td86NAvcDN+Xv4JwbyPuxAZhXi5iHvZv8m4pZctkYU4ZORCrat57p5mf7+rjz7a/jrJbakrzn+D10I6mSvJ9ItVsIq5v8NJhI0RALEw6deYEqV8G3X5UupcrNJaBbDhzI+7nb23YaM/uYme0jm6H7eKE3MrPbzGyrmW3t6emZw5BKK+5d9SlmyWV7Yw3Hh0bJZOZVzCoyrxVxYrTJzJ41s5SZ3TzhubR3wvScmW32b9TBODaY4I+/t5PLVy/mfZetLNn7KkMnUryFsLrJb4OJsYIVLoHxe4QV0Em1m0tAV2itzhnRjHPuHufcWuAPgD8s9EbOuXudc13Oua6Ojo45DKm04snJ0/gTtTfGSGccJ3RSI+KLIk+MXgM+CHyzwFuMOOcu8r5uLOtgK8CnN+8gkcrwud+4gFCBK9mzpYBOZEaqfnWT3+LJVMEKl5AtigJacinVby4BXTewIu/nTuDQFPvfD7xrDsfz3XhAVxOedt+2RvWiE/FZMSdGrzjntgGZIAZYKR7acYQt24/we29az5qOxpK+dzQcoj4WVpVLkeKUbHWTZA0mJg/ocheclKGTajeXgO5pYL2ZrTazGHALcNqyJTNbn/fjO4A9czie74bGA7rCqfx87QroRPxW1InRFGq9pd5Pmtm8utg0EwOJMe761xc4b1kzt21aU5ZjNNdGlaETKU7JVjdV6u0qfptqyWUkHKK5NqIMnVS96dcSTsI5lzKz24GHgDBwn3Nuh5ndDWx1zm0Gbjez64ExoB+4tRSD9svg+D1002fo2huz67RV6VLEN0WdGE1hpXPukJmtAR41s+3OuX1nHMTsNuA2gJUrS3fvmV/+7OE99Awm+eoHuoiGy9N6tKVOAZ1IkWazuumvCj3hnLsXuBegq6trwS7LHEykWLG4ftLnWxtiytBJ1Zt1QAfgnNsCbJmw7a68x783l/cPWi5D1zSDDF2fMnQifpnpidFpnHOHvO/7zezHwMXAGQHdfD9penJ/H9es7+DCzkVlO4YCOpGija9uAg6SXd30/vwdzGy9cy63omnerW7y20AiNWmGDrKFUfqVoZMqV57LtVUidw9dMRm6lrookZBpyaWIf6Zd9j0ZM2s1sxrvcTtwFbCzbCMN0METI5zTNvnV61JoroswkFDbApHpOOdSQG510y7gwdzqJjPLFWe63cx2mNlzwB3Ms9VNfhtMjNE8RfG61vooJ5Shkyo3pwxdtZtJlctQyFjcEKN3UJOGiB+KWfZtZpcB/wy0Ar9mZn/knDsfOA/4ipllyF7Y+pxzruoCuoHEGCdHxuhsrSvrcZrrouw6PFjWY4hUi2pf3eSn0VSGZCozaVEUgNb6GHuPxX0clYj/FNBNIZ5MEQ0bNZHpM3SQXXbZN6QMnYhfijgxeprsUsyJr/sZcEHZBxiwg/0jAHS2ljdDpyWXIhKEwUR23mmcol/wovqoiqJI1dOSyykMJVNTThITtTXG6FFRFBGpEN3jAV15M3QtdVHiyRSp9ILuDiEiPssVr5vqHrrW+hjxZIrRlOYnqV4K6KYQT6RomEFA19FYo6IoIlIxuvuHAVi+qMxLLr2TqUHdRyciPjoV0E19Dx3AiRFdcJfqpYBuCvFZZOh640mcm3eF8ESkCnX3j1AXDbO4IVbW4+Sa92rZpYj4Kbfkcroql4CWXUpVU0A3hZkGdO2NNSTGMgyNpss4KhGR4nT3D9PZWodZoZZ9paOATkSCMFBUhi4b0PUPKUMn1UsB3RSGkqmiKlzmqBediFSS7v6Rst8/B9DiLWkaSCigExH/5KqRN091D11D9jn1opNqpoBuCoPJmd1D19aYvQqkXnQiUgkOnhgpe4VLOHUypQydiPjp1JLL6TN06kUn1UwB3RSGkimaZrjkEqBXlS5FJGCDiTFODJe/Bx1oyaWIBCNXFGWq1VTjSy6VoZMqpoBuCjOtcnkqoFOGTkSCdfCEPz3oQAGdiARjMDFGXTRMNDz56WxdLExNJKQMnVQ1BXSTyGQcQ6PpGVe5BOgd1KQhIsHqPp4N6Jb7kKGrjYaIho2BEbUtEBH/DCZSUy63zGmtj9GvgE6qmAK6SQyNemn8GQR00XCIRfVR+oaUoRORYOV60Pmx5NLMaKmLKkMnIr4qNqBbVB/VkkupagroJjGUzLYemEmVS4C2hpiWXIpI4Lr7R6iNhmgrcw+6nOa6KAMK6ETERwOJsSl70OW01se05FKqmgK6ScST2ROTmdxDB9n76FQURUSClm1ZUF/2HnQ5LXVRtS0QEV8VveSyQRk6qW4K6CaRq5w0kyqXkAvolKETkWBlWxaUf7llTnOtllyKiL8GE2NFLrlUhk6qmwK6SeSWXM48Qxejd1ABnYgEq7t/2NeATvfQiYjfBhMpmmqKWXKZzdA553wYlYj/FNBNIrfkciZFUSCboRtIpBhNZcoxLBGRacWTKfqHx3xpWZCjgE5E/DaTKpfpjGMgoUq8Up0U0E0iniuKMsOArs3rRadKlyISlIP9XsuCRf5m6AZGdAVcRPwxls4wMpYuqijKIq+5uJZdSrVSQDeJuHdz/0yrXLZ7vej6VBhFRALiZ8uCnOa6CBmXzQ6KiJTbkDfXFJehywZ9Kowi1WpOAZ2Z3WBmL5nZXjO7s8Dzd5jZTjPbZmY/NLNz5nI8Pw2N5u6hC8/odbkMXY8Ko4hIQLq9DJ3fSy4BLbsUEV+MF68rqspl9mK7motLtZp1QGdmYeAe4G3ARuB9ZrZxwm6/BLqccxcC3wa+MNvj+W0wkSIWDlETmVlA15FbcqkMnYgEpLt/mJpIaHzFgB9yAd3AiDJ0IlJ+uTYpxfahAy25lOo1lwzd5cBe59x+59wocD9wU/4OzrkfOeeGvR+fBDrncDxfDSVTM87OAbQ3ZScNtS4QkaDkWhb41YMOsm0LQBk6kelU8+omP+UydM0zWXI5pPlJqtNcArrlwIG8n7u9bZP5MPDvczier+LJ1IzvnwOoj0Woi4bVukDEB0WcGG0ys2fNLGVmN0947lYz2+N93erfqMsv11TcT81acikyrWpf3eSnU0sup8/QNddGCZkydFK95hLQFbr0W7C8mZn9NtAFfHGS528zs61mtrWnp2cOQyqdeDJFQ2zmAR1ks3R9Q5o0RMqpyBOj14APAt+c8NrFwKeAK8iuNviUmbWWe8x+yQZ0/hVEgfwllwroRKZQ1aub/DQ4vuRy+nO1UMhoqYuqKIpUrbkEdN3AiryfO4FDE3cys+uBTwI3OucKpq2cc/c657qcc10dHR1zGFLpxIvsbVJIW0ONllyKlF8xJ0avOOe2ARMbQ74VeNg5d9w51w88DNzgx6DLbSiZ4vjQKMv9Dui8JU25+1pEpKCqXt3kp5kURYHsfXQqiiLVai4B3dPAejNbbWYx4BZgc/4OZnYx8BWywdyxORzLd0OjKRpm2IMup72xhl4VRREpt5meGJXqtRXt4An/K1wCNMYimGnJpcg0qnp1k58GZ9healF9lBPK0EmVmnVA55xLAbcDDwG7gAedczvM7G4zu9Hb7YtAI/AtM3vOzDZP8nYVJ55IzbipeE5HU0wZOpHyK/rEaC6vnW8nTUH0oIPskqbm2qgCOpGpVfXqJj8NJlLEIsVXI1eGTqrZ7CIWj3NuC7Blwra78h5fP5f3D1I8OfuArq2hhuNDo2QyjlDIvypzIgtMUSdGU7z2ugmv/XGhHZ1z9wL3AnR1dRUbMAbmVA86fwM6yN5Hp4BOZErjq5uAg2RXN70/f4e81U03zLfVTX4aSKSKqnCZs6g+xotHBss4IpHgzKmxeDWbS0DX3hgjnXGc0ImNSDlNu+x7Cg8BbzGzVq8Yylu8bfNed/8INZHQeE9MPzXXRVQURWQK1b66yU+DibGiKlzmtNZHlaGTqjWnDF21Smccw6PpWd9D1+adSPXGkyxu8K+xr8hC4pxLmVnuxCgM3Jc7MQK2Ouc2m9llwD8DrcCvmdkfOefOd84dN7PPkA0KAe52zh0P5H+kxA72j7Dc5x50OcrQiUyvmlc3+SmenFnxutaGGMOjaRJjaWqjM+8zLFLJFNAVMDQ6s8pJE7XnBXQbljaVbFwicroiToyeZpKS3865+4D7yjrAAHT3D/teECWnpS7KkZOJQI4tIgvL4AyrkS/yKvGeGB7jrBYFdFJdtOSygKFkNqCbS1EUQJUuRcR33f0jLF/k//1zkA3oBrxS4iIi5TSYGKOppvgll+csbgDguQP95RqSSGAU0BUQ905IZr3kssHL0A2q0qWI+Gd4NEXf0GggBVEAVbkUEd/MNEN35ZrFLG2u4cGt3WUclUgwFNAVEM9l6Ga55LKlLkokZPQNKaATEf8cDLDCJUBzXZTRVIbEWDqQ44vIwpEN6IrP0EXCIW6+tJMfv3RMS8Ol6iigKyA+xyWXoZCxuCFG76CWXIqIf061LAjuHjpQc3ERKa90xs24KArAey5dQcbBd55Vlk6qiwK6AuZ6Dx1kC6MoQycifso1FV8RUIYuF9CpdYGIlFPuwvtMA7pV7Q1csXox39p6AOcqvq2oSNEU0BUwmChBQNdUQ4+KooiIj7pPjBCLhMYr7fqtWRk6EfHBYCI7xzTPYMllznu7VvBK3zBPvVwVnWpEAAV0BZUkQ9cQU1EUEfFVd/8InYvqCIX870EHWnIpIv4Yv/A+i1oHb79gGY01ER7YeqDUwxIJjAK6AnKp/NlWuYRshq5vKKmUvoj4pttrKh6U8SWXCQV0IlI+uYBuNv2C62Jhfu31Z7Nl++HxTJ/IfKeAroB4Mk0sEiIWmf3H09YQIzGWYWhU1d5ExB8H+4cDq3AJ0OydXJ0c1kmSiJRPLhCbSZXLfL952QoSYxn+7fnDpRyWSGAU0BUQT47NabklMH4PS19cyy5FpPxGRtP0xkcDq3AJ+ffQqbm4iJTPXDJ0AK/vbGHD0kYe1LJLqRIK6AqIJ1JzDujaGmMA9CqgExEfHDyRrXAZZIYuGg7REAvrHjoRKavBWVa5zDEz3tu1gucOnGD30cFSDk0kEAroCogn03O6fw5OZeh61ItORHxwIOCm4jktdVHdQyciZTWXKpc5v37xcqJh44GnlaWT+U8BXQHx5BhNcwzoOpq8JZfqRSciPjgYcFPxnOa6qDJ0IlJWg4kU0bBRM5daB401XH/eUv75lwcZTWVKODoR/ymgK2AomaahJjyn91jc4C25VIZORHzQ3T9CLByiI6AedDkdTTV0e8GliEg5DCbGaKqNYja3Fi3v7VrB8aFRfrjraIlGJhIMBXQFxJMpGueQxofsvSSL6qPK0ImIL7r7h1neGlwPupwr17Sx6/AAxwYTgY5DRKrXYCI16/vn8m3a0MFZzbUqjiLzngK6AuLJFI1zzNBBtnWBiqKIiB+6+0dYvijY++cArju3A4CfvNQT8EhEpFqVKqALh4x3X7qcn+zu4chJXYSS+UsBXQGlqHIJ2cIoWnIpIn7o7h8JvCAKwMZlzSxpquHHuxXQiUh5DCbGaKqZ20qqnPdcuoKMg+88212S9xMJggK6CdIZx8jY3KtcArQ31dCrJZciUmaJsTS98WRFBHRmxrUbOnh8dw+ptAoNiEjplSpDB7CqvYErVi/mwa0HyGRcSd5TxG9zCujM7AYze8nM9prZnQWe32Rmz5pZysxunsux/BL3epuUJEPXEKN3UAGdiJRXd4VUuMy57twlDCRSPHfgRNBDEZEqlA3oSpOhA3hP1wpe7Rtm+8GTJXtPET/NOqAzszBwD/A2YCPwPjPbOGG314APAt+c7XH8NlTKgK6xhoFESuVwRaSsuvuDbyqe7+r17YRDxo91H53IGarxYrjfBhJjJcvQAVy7IXvv7xN7e0v2niJ+mkuG7nJgr3Nuv3NuFLgfuCl/B+fcK865bcC8iWjGM3QlmCjaGtWLTqScijgxqjGzB7znf2Fmq7ztq8xsxMye877+2u+xl9LBE5WVoWupi3LJykX8ePexoIciUlGq9WK4nzIZRzxZuiWXkG238rqzmvipAjqZp+YS0C0H8uu8dnvb5rVcQFeSe+ga1YtOpFyKPDH6MNDvnFsHfBn4fN5z+5xzF3lfH/Vl0GXS3T9CNGwsaQq2B12+685dwgsH1b5AZIKqvBjup6HRFM5R0oAO4Op17Wx9pZ+R0XRJ31fED3MJ6Ao1O5rV3aRmdpuZbTWzrT09wS7RiSeyAV1TiYqiACqMIlIe054YeT9/w3v8beBNNtdOtBWou3+EsxcF34MuX24Jk9oXiJymKi+G+yl34b2U99ABXLW+ndF0hq2vHi/p+4r4YS4BXTewIu/nTuDQbN7IOXevc67LOdfV0dExhyHN3VApM3QNXkCnwigi5VDMidH4Ps65FHASaPOeW21mvzSzn5jZNZMdpJIuOE1me/cJ1nU0Bj2M05x/djMdal8gMlFVXgz302DuwnuJM3SXr1pMNGy6j07mpbkEdE8D681stZnFgFuAzaUZVnAGS1kUpSm75LJvSEsuRcqgmBOjyfY5DKx0zl0M3AF808yaCx2kki44FXLg+DCv9A1z1br2oIdyGrUvECmoKi+G+2kwMQaUPkPXUBPh4pWtuo9O5qVZB3Te1e7bgYeAXcCDzrkdZna3md0IYGaXmVk38B7gK2a2oxSDLqdSVrmsj0Woi4aVoRMpj2JOjMb3MbMI0AIcd84lnXN9AM65Z4B9wIayj7gMcleTN22orIAO4LpzO9S+QOR0VXkx3E8DZcrQAVyzrp0dhwY4rgvxMs/MqQ+dc26Lc26Dc26tc+6z3ra7nHObvcdPO+c6nXMNzrk259z5pRh0OeXuoSvFkkvIZumUoRMpi2JOjDYDt3qPbwYedc45M+vwiqpgZmuA9cB+n8ZdUk/s6eWs5lrWVtiSS4Br1nWofYFInmq9GO6nF7qzveKWNteW/L2vWt+Oc/CzfcrSyfwyp4CuGsVHU8QiIWKR0nw07Y019MaVoRMptWJOjICvAW1mtpfs0spca4NNwDYze55ssZSPOufm3Z3w6Yzjib29XL2+nUqs9dJSr/YFIhNV48VwvyTG0nzj569y3bkdLF9U+r6bFy5voakmomWXMu+UPl89z8UTqZJUuMxpa6gZb/orIqXlnNsCbJmw7a68xwmyV7knvu47wHfKPsAye+HgSU6OjHHN+spbbplz3blL+OJDL3FsMMGSptJfUReRhWPzc4fojSf5yDVryvL+kXCIK9e2qTCKzDvK0E0wlEyVbLklQIeWXIpImeROOiqtIEq+XPuCx3brBElEZi+Tcdz7+H42LmvmDWvbpn/BLF29rp0Dx0d4rU8X42X+UEA3QTyZKklBlJy2hhqOD42SycyqKrGIyKQe39PDxmXNtDdWTkPxicbbF7ykZZciMns/2d3D3mNxbtu0pqxLzK/2VjwoSyfziQK6CUod0LU3xkhnHCdGxkr2niIiQ8kUz7zaX9HLLSGvfcGeXrUvEJFZ++rj+1nWUss7LlxW1uOsaW9gWUut7qOTeUUB3QTxZIrGEpbCbW/ymourMIqIlNBTLx9nLO3GryZXsuvO7eDkyBjPd6t9gYjM3AsHT/KzfX186KpVRMPlPXU1M65a185P9/WS1uoqmScU0E0wlEyX9B66tgYvoFMvOhEpocf39FITCXHZqsVBD2Va16zrIGSofYGIzMpXH99PY02EWy5f6cvxrl7XzonhMXYeGvDleCJzpYBugsFEaZdcdrZmy+r+3//6Ag88/RqJsXTJ3ltEFq4n9vZw+erF1EbDQQ9lWtn2Ba0K6ERkxg6eGOF72w5zy2UraK6N+nLMN6zLFl3RfXQyXyigm2AomaKxpnQnSCsW1/O/f+sSaiJh/uA727nqc4/yZ4/s1hJMEZm1IycT7D4a5+oKrm450XXndrD94El6tFpBRGbg6z99GYAPXb3at2Muaarl3KVNuo9O5g0FdHlS6QwjY2kaa0p7BejtFyzj+x+/mm9+5AouWrGIP3tkD2/43KN84rvb2HN0sKTHEpHql7tqPB/un8u57twlADy2W1k6ESnOQGKMf3rqAO+8cFlZGolP5er17Tz1ynGtrJJ5QQFdnqHR7C9tQwkzdDlmxhvWtvO1D17GI3dcy82XdvLdZw/y5i8/xp889JLaGohI0Z7Y00N7Y4zzzmoOeihFy7VX+Ldth3BO852ITO/+p14jnkyVrZH4VK5e185oKsMzr/b7fmyRmVJAl+fIyQQATSWsclnIuiWN/D+/fgE//8SbeG9XJ3/5o7189B+eYSiZKutxRWT+y2QcT+ztBItUZgAAIABJREFU46p17YRC5evFVGqhkPHhq1fz45d6+Kuf7At6OCJS4cbSGf72p6/wq2va+JXlLb4f//LVi4mEjMf3aNmlVD4FdHm+/PBu6qJhrt2wxJfjLW6I8fl3X8infm0jj+w6yrv/6md09w/7cmwRmZ9ePDJIbzw5r+6fy/notWv4tdefzRcfeon/2HEk6OGISAX7/rbDHD6Z4LZN/mfnABpqIlyyslX30cm8oIDO88SeXn6w4wi3v3EdZ7XU+nZcM+NDV63m6x+6nIMnRrjpL3/K068c9+34IjK/PLE3ew/aNes7Ah7JzJkZX7z5Qi5c3sLvP/CcSoKLSEHOOe59bD/rljRy7Ybg5rqr1rXzwqGT9A+NBjYGkWIooCOb1v/0v+3gnLZ6/ss1/lVRyrdpQwf/8rGraK6L8v6vPskDT78WyDhEpLI9vqeX9Usafb3wVEq10TBf/UAXzbVRPvJ3W1X1UkSAbBC35+ggf/P4fj5w31PsPDzAR65ZHejS8qvXt+Ec/Hx/X2BjECmGAjrgGz97hb3H4tz1zo3URILr6bS2o5F/+d2ruHJNG3/wne3c/W87SatYioh4EmNpnnr5+LyqblnIkuZa/ubWLvqGknz0H54hmVIVOZGF6MTwKN/bdoj/+e3necPnHuXNX36MP/7+Lg6dGOG/XreWX7+4M9Dxvb5zEY01EfWjk4pX3uof80DPYJI/f2QP153bwRtf58+9c1NpqY/ytx+8jD/+/i7u++nLHB1I8OXfvIhYRLG3yEK39ZV+kqkM18zzgA7gV5a38KX3XsTv/uOzfOK72/nT97wes/lT5EVEZm80leHPHtnNVx7bTzrjaKqNcPW6dj7+pg6uWd9OZ2t90EMEIBIOceWaNt1HJxVvwQd0X/jBiyRSae5658aKOZmIhEN8+sbzWb6ojs9u2UViLM09v3UJtdHgsociErzH9/YQDRtXrG4Leigl8fYLlvHfr9/Alx/ZzYalTXz02rVBD0lEymzvsUF+/4HneOHgAO++pJP3X7GS13e2EAlX5oXra9a388iuo/zrcwe56aLlQQ9HpKAFHdD98rV+vvVMN79z7RrWdDQGPZwzfGTTGupiYf7wX17gP3/9ab76gS4aahb0H5nIgvbEnl4uWdlaVfPAx9+0jj3HBvn8D15kRWs977hwWdBDEpEycM7x90++yme/v4v6WJiv/KdLeev5ZwU9rGm9p6uTLdsP8/sPPMdQMs37r1gZ9JBEzlCZl0N8kMk4Pr15B0uaavhvb1wf9HAm9dtXnsOfvuf1PLm/jw/c9xQDibGghyQiAeiLJ9lxaKAqllvmMzP+5D2v5/Wdi/jYN5/lf3zreU4Oa54TqSbHBhN86OtPc9e/7uDKNW089Pub5kUwB1Afi/CN/3w5123o4H/983a++tj+oIckcoYFG9B9+5lunu8+ySfe/joaK/xq97sv7eQv338Jzx84wfu/+iTHVT5XZMHJ3ZR/9TxsVzCd2miY+2+7kt+9bi3f/eVB3vSln7Bl+2GcU1EokfnuoR1HuOHPHufn+/q4+6bz+fqHLmNJ8/yq0lsbDfOV/9TFOy5Yxme37OJLD+/W/CQVZU4BnZndYGYvmdleM7uzwPM1ZvaA9/wvzGzVXI5XKidHxvj8D17k0nNaedc8WQ/99guWce8HLmX30Ti33Ptzjg0mgh6SiPjgyMkED249wL2P7aelLsoFy1uCHlJZ1EbD/M8bXsfm26/irJYafvcfn+V3/v4Zjg5orpPqMl/PnWZiKJni/qde46Z7fsrv/P0znL2olu9//Go+8KurKqZewUzFIiH+4n0X896uTv7ih3v4zPd2KaiTijHr1JSZhYF7gDcD3cDTZrbZObczb7cPA/3OuXVmdgvweeA35zLguXLO8eeP7OH48CjfuPHyeTWxvPF1S/n6By/jv/zdVt771z/nt644h7VLGljb0Uhnaz3hAHu1iATBzG4A/hwIA3/jnPvchOdrgL8DLgX6gN90zr3iPfcJsnNUGvi4c+4hH4c+qVxrgsd29/DYnh52H40DsKSphv/x1nOr/vf8/LNb+JffvYqvPfEyX3p4N9d/6Sd88u3n8ZuXrZhX87UU5pwjlXGMpjKMpTOMpjIkUxmSqTSJsTO/j6a8L2/f3GtG09nXJcbSjIymSXiPc19vOm9pRRbZma/nTsVwzrH94En+6akDbH7uIEOjadYvaeRTv7aR37rinKqo1h0OGZ/7jQtpqIlw309fJp4c4//9jQurfl6WyjeXtYaXA3udc/sBzOx+4CYgf1K6Cfi09/jbwF+ambkSXNJIZxwnhkdxQMY5yP5Hxjmcg1TacejkCK/2DfFq3zCvHh/mtb5hXukbYjCR4v1XrORX5uGV7jesa+fvP3wF/+2bz/LZLbvGt8fCIVa117O2o5HV7Q0010Wpi4api4Wz373HtdEw0bARDhmRUIhwKPc4+90MjNx3YOLPk5jsRGuqP+qJz+Tv6iY+6858WGj/3LZT+zgmDmHK48yC5X0yMznfnM256Wx/c059Lq7AtqleN/UBz15UF1j11bmcGJnZRuAW4HzgbOARM9vgnCtJQ7RfvtZPz2ASR+5zzv49zH2aI6Np+odHOTkyRv/wKP3DY5wczj7eeyxOMpUhFglx+arF3HxpJ5s2dHDu0qYFE9BEwiF+59q1vPX8s7jzu9u487vb+dOHd3N2Sy1ntdRyVnMtS73vZzXX0lwXJRLOzmO5eS0azn4PGXl/Dt7vgDu1LeNc9iuT99g5Mi7770w6c/rPGedty2QDk7RzpNPZxxnnGEtnGEs7UunM+OOxdIZUxo0HLrkgJTn+PT0epIylHMkJwUsqnckexxtj7vjpvN9P49Q8nJu7Q5b9/w+ZYXmPcyefoVBuH/Nen9s3957e+xX4a5f9+3zq73Vurk1748v/7FIZRyrtsv9/6cys57GJaqMhar1/32qjYWoip36OVmjFRAI+dxpNZYgnU+M/T3zL3O/F+J+t9zjjsrUHRtMZkmNeQD2W9v5OZzjYP8yDW7vZeXiA2miId154Nu+7fAWXrGytunkrFDLueudGmmoi/MWje+mNj/KGtW001UZoqInQmPuqjdAQi4z/XuV+t8Z/z6Y4Rv55zvi/22ec3xQ+95l43lPwd3X8Nae2586fc9/z/+ydNwfm9s9tm2xuLYZNeFDoXMrGfy7uPGuyp077zLwBn/E55v274CZ8LuQ+i8zpnxV5n//EzyL/8z7tOMC7Llpe8t+LuQR0y4EDeT93A1dMto9zLmVmJ4E2YM4NPY4OJHjD5x4tat9IyOhsrWNlWwMXr1zE2o5G3tu1Yq5DCMyl57Tys0+8if6hUfb3xtl3bIh93veXjgzyHzuPqiG5+OL7H7+a888O7MLIrE+MvO33O+eSwMtmttd7v5+XYmD/36N7efTFY9PuFzJYVB9jUV2URfVRljbXcsXqNq7Z0M6Vq9uoiy3sViWr2hv4p49cyXefPciT+/s4MpBgf88QP9vXx2AiNf0bVKBYJERNOERNNEQsHCIWyX7VRMLEIiGiYaMlFvWeM2LhEJFwiLAZoZARDnHqsXeCmH+xIP9kBDgtQHXjwdapE5jccxnv5CyTmXiCePoJy6kTrLyLfXmPcxcJw17geOqioff/Gjbv/9P7ioTGA7HaSIiavO813nPZ1536rKLhEJGQzddAIdBzp5/u7eVDX396rm9T0MZlzXzmXb/CTRedTXNttCzHqBRmxh1vOZem2ihfeOjFouZ7kZybXr98Vhf1pzKXgK7QUCZGEcXsg5ndBtwGsHJlceVgW+qi/NGN53v/mGTj+tzVj5C3bVlLLecsbuDsRbUV299kLlobYlzasJhLz1l82vaMdyV4eDTFiLf8ZHg0uyxlZCxNOndlefx7hnQGUulMwasTuKnzWNNdM5zR1ZT8qzBTvE+hK8c2YT87ddnnjKs8+e89l1+q07N9E58rPjs5E7PJlOa/7rTPrMCVsTPfc/LjdS4KtPnrXE6MlgNPTnhtwRtqZzM/3fXOjdzx5g3e68888a2NhlhUH6OpJkJIS3WmZGa8+9JO3n1p52nbh0dTHDmZ4MjJBPFkKpsF8uazsbQ3v3lzGuSyVqd+/09dNcfLXmWDpVw2K5zLaHnBSS6bNVmwEgpxKnAJh4iEs1nC3ONIOBuczdMgREon0HOndUsa+aMbz5/wPmce3OzUip1Qbu4yGw+yayJewB3NPm6qjbBycf2C+/v9kU1r+NBVqxgaTRNPphhKphhMpMYfDyVTp2c5x7Nf2Z+LOj/Ky+YVyp4XfM7OzNpPvPgCp86f8+dDyJ8XT/3ZW97+49lG780njme6vweFsmTjz02SkZy43xnvOc2ZVaHPLD87mD/2U48LxRj575P/WU74mdyfyYT3Ht9eWnMJ6LqB/DRXJ3Bokn26zSwCtADHJ76Rc+5e4F6Arq6uos51G2oi3PqGVTMf9QIQCll2qeUCv7ovVW8uJ0ZFnTDB7OanVe0Nxewmc1Afi7Cmo7Eie4iKTCHQc6cVi+t17lRikXCIlroQLXXVnZWUyjaXtNXTwHozW21mMbL3o2yesM9m4Fbv8c3Ao6VYAy4iwsxOjJhwYlTMa0VESk3nTiJScrMO6JxzKeB24CFgF/Cgc26Hmd1tZjd6u30NaPPuT7kDOKM8r4jILM3lxGgzcItXHnw1sB54yqdxi8gCpXMnESmHOXXUds5tAbZM2HZX3uME8J65HENEpBDvnrjciVEYuC93YgRsdc5tJnti9PfeidFxskEf3n4Pki2gkgI+VqoKlyIiU9G5k4iU2pwCOhGRIM3lxMg591ngs2UdoIiIiEiZVV/pRxERERERkQXCKu0+WzPrAV6dwUvaKUFvlhLRWArTWM5UKeMA/8ZyjnOuw4fjlM0M56eF+GdcDI2lMI2lMD/GorkpOBpLYRpLYQttLEXPTRUX0M2UmW11znUFPQ7QWCajsVTuOKCyxlJNKulz1VgK01gK01iqWyV9phpLYRpLYRrL5LTkUkREREREZJ5SQCciIiIiIjJPVUNAd2/QA8ijsRSmsZypUsYBlTWWalJJn6vGUpjGUpjGUt0q6TPVWArTWArTWCYx7++hExERERERWaiqIUMnIiIiIiKyIM3bgM7MbjCzl8xsr5ndGfBYXjGz7Wb2nJlt9fnY95nZMTN7IW/bYjN72Mz2eN9bAxzLp83soPfZPGdmb/dpLCvM7EdmtsvMdpjZ73nbff9sphiL75+NmdWa2VNm9rw3lj/ytq82s194n8sDZhYr91iqmean8WNrfjpzHJqbCo9Fc5MPNDeNH1tzU+GxaH4qPJbKn5+cc/PuCwgD+4A1QAx4HtgY4HheAdoDOvYm4BLghbxtXwDu9B7fCXw+wLF8Gvi/AvhclgGXeI+bgN3AxiA+mynG4vtnAxjQ6D2OAr8ArgQeBG7xtv818F/9/jOrli/NT6cdW/PTmePQ3FR4LJqbyv8Za246dWzNTYXHovmp8Fgqfn6arxm6y4G9zrn9zrlR4H7gpoDHFAjn3GPA8QmbbwK+4T3+BvCuAMcSCOfcYefcs97jQWAXsJwAPpspxuI7lxX3fox6Xw54I/Btb7tvf2eqlOYnj+anguPQ3FR4LJqbyk9zk0dzU2GanyYdS8XPT/M1oFsOHMj7uZuA/pA9DvgPM3vGzG4LcBw5S51zhyH7CwEsCXg8t5vZNm9ZgS9LGPKZ2SrgYrJXVAL9bCaMBQL4bMwsbGbPAceAh8lesT3hnEt5uwT9+zTfaX6amuYnj+amM8aguam8NDdNTXNTHs1PZ4yhouen+RrQWYFtQZbrvMo5dwnwNuBjZrYpwLFUmr8C1gIXAYeBP/Xz4GbWCHwH+H3n3ICfxy5iLIF8Ns65tHPuIqCT7BXb8wrt5sdYqpTmp/kjsPlJc9OZNDeVneam+UPnTpOPRfNTAfM1oOsGVuT93AkcCmgsOOcOed+PAf9M9g86SEfNbBmA9/1YUANxzh31fgkywFfx8bMxsyjZSeAfnXPf9TYH8tkUGkuQn413/BPAj8muA19kZhHvqUB/n6qA5qepLfj5SXPT1DQ3lY3mpqkt+LkJND9Np1Lnp/ka0D0NrPeqy8SAW4DNQQzEzBrMrCn3GHgL8MLUryq7zcCt3uNbgX8NaiC5CcDz6/j02ZiZAV8DdjnnvpT3lO+fzWRjCeKzMbMOM1vkPa4Drie7Lv1HwM3eboH+nakCmp+mtqDnJ81Nk45Fc1P5aW6a2oKem7zjan4qPJbKn59mW00l6C/g7WQr3uwDPhngONaQrRT1PLDD77EA/0Q25TxG9urbh4E24IfAHu/74gDH8vfAdmAb2QlhmU9juZps6nsb8Jz39fYgPpspxuL7ZwNcCPzSO+YLwF15f4+fAvYC3wJq/Px7XG1fmp/Gj6/56cxxaG4qPBbNTT58aW4aP77mpsJj0fxUeCwVPz+ZNyARERERERGZZ+brkksREREREZEFTwGdiIiIiIjIPKWATkREREREZJ5SQCciIiIiIjJPKaATERERERGZpxTQiYiIiIiIzFMK6EREREREROYpBXQiIiIiIiLzlAI6ERERERGReUoBnYiIiIiIyDylgE5ERERERGSeUkAnIiIiIiIyTymgExERERERmacU0ImIiIiIiMxTCuhERERERETmKQV0IiIiIiIi81Qk6AFM1N7e7latWhX0MESkxJ555ple51xH0OOYC81PItVHc5OIVKKZzE0VF9CtWrWKrVu3Bj0MESkxM3s16DHMleYnkeqjuUlEKtFM5iYtuRQREREREZmnFNCJiIiIiIjMUwroRERERERE5ikFdCIiIiIiIvOUAroKlck47nviZQYTY0EPRaRimdkNZvaSme01szsLPP9RM9tuZs+Z2RNmtjHvuU94r3vJzN7q78hlosf39PCL/X1BD0NE5DQnh8f42hMvk864oIciMikFdBVq97FB7v7eTv7t+cNBD0WkIplZGLgHeBuwEXhffsDm+aZz7gLn3EXAF4Avea/dCNwCnA/cAPxv7/0kAOmM478/8Bxfenh30EMRETnND3Yc5jPf28nP9vUGPRSRSSmgq1CDiRQA+3viAY9EpGJdDux1zu13zo0C9wM35e/gnBvI+7EByF1ivQm43zmXdM69DOz13k8C8IuX++iNjzLgzXsiIpWiZzAJwMM7jwY8EpHJKaCrUPFk9sRmnwI6kcksBw7k/dztbTuNmX3MzPaRzdB9fCavFX98f1t2JYKWmItIpckFdI/sPIpzWnYplUkBXYWKJ3IB3VDAIxGpWFZg2xn/2jrn7nHOrQX+APjDmbwWwMxuM7OtZra1p6dn1oOVwlLpDD944QhwamWCiEil6IlnA7pDJxPsPDwwzd4iwVBAV6GGvAxdd/8wibF0wKMRqUjdwIq8nzuBQ1Psfz/wrpm+1jl3r3OuyznX1dHRMYfhSiG/ePk4fUOjrFvSyGBiTFfARaSi9A6Osn5JI2bwyM5jQQ9HpCAFdBUqt+Qy4+DVvuGARyNSkZ4G1pvZajOLkS1ysjl/BzNbn/fjO4A93uPNwC1mVmNmq4H1wFM+jFkm+N62w9THwtz4+rPJOBga1QUsEakcPfEkr1vWzCUrW3lkl+6jk8qkgK5C5QI60H10IoU451LA7cBDwC7gQefcDjO728xu9Ha73cx2mNlzwB3Ard5rdwAPAjuBHwAfc84pkvBZdrnlYd503lI6mmoAGBjRfXQiUjl6BpO0N8a4/rylbD94ksMnR4IeksgZFNBVqHgiRTScvc1HlS5FCnPObXHObXDOrXXOfdbbdpdzbrP3+Pecc+c75y5yzv0fXiCXe+1nvded65z796D+Hxayn+/vo394jHdcsIzm2iig++ikehTRJ3OTmT1rZikzu3nCc7ea2R7v61b/Ri35RkbTxJMpOppqePPGJQD8cJeWXUrlUUBXoYZGUyyqj3F2S60Ko4hIVfr+tsM0xMJcd24HTbURQJUupToU2SfzNeCDwDcnvHYx8CngCrLtVD5lZq3lHrOcqdcriNLRWMPajkZWtdVr2aVUJAV0FWowkaKpJsKajkYtuRSRqjOWzvCDHUe4fuNSaqNhmuuyGboBBXRSHYrpk/mKc24bkJnw2rcCDzvnjjvn+oGHgRv8GLSc7pjXsqC9qQYz4/rzlvKzvX2n3RYjUgkU0FWooWSKhpoIazsa2N8zpMpvIlJVfr6vjxPecksgL0OnEyWpCnPpdak+mRUiP0MHcP3GpYymMzy+Wy1spLIUFdAVsQ78DjPbaWbbzOyHZnZO3nNpM3vO+9o88bVSWDyZorEmwtoljcSTqfGrRCIi1eD72w7TWBNh04ZsK4hcQKeiKFIliu51OdvXqkdm+eWaii/xijZ1ndPKovooD2vZpVSYaQO6IteB/xLocs5dCHwb+ELecyNeQYKLnHM3IkWJJ9M01ERY094IwL5jWnYpItUht9zyzd5yS2C8KMqAMnRSHWbaJ3PGr1WPzPLrGUxiBosbYgBEwiHeeO4SfvTiMVLpiStlRYJTTIaumHXgP3LO5ZqlPUl28pE5iCfHaKqNsHZJAwD7elUYRUSqw0/39nJy5NRyS4DaaJhYOKQll1Itpu2TOYWHgLeYWatXDOUt3jbxWW88yeL6GJHwqdPl6zcupX94jGdfOxHgyEROV0xAN9O13B8G8kuA13pLAp40s3cVeoGWDZxpKJmmoSbMWc211MfCytCJSNX4/rbDNNVEuGZD+2nbm2ojKooiVaGYPplmdpmZdQPvAb5iZju81x4HPkM2KHwauNvbJj7rGUyO98jM2bShg1g4pGqXUlEiRexT9DpwM/ttoAu4Nm/zSufcITNbAzxqZtudc/tOezPn7gXuBejq6lL1D7L30DXURDAz1nQ0qNKliFSF0VSGh3Yc4c3nL6UmEj7tuea6qDJ0UjWcc1uALRO23ZX3+GkmWdHknLsPuK+sA5Rp9cSTtDeeHtA11kS4cm0bj+w8yv96+3kBjUzkdMVk6Ipay21m1wOfBG50zo1X8HDOHfK+7wd+DFw8h/EuCKOpDKOpDE012Xh7bUcj+9WLTkSqwE/39jKQSPHOC5ed8VxTbUR96ESkYhTK0AG8+bwl7O8dYq9WT0mFKCagm3YduJldDHyFbDB3LG97q5nVeI/bgauAnaUafLUa8vqbNHgB3Zr2Rg6eGGFkNB3ksERE5ux72w7TVBvh6nVnFnForo2qyqWIVATnHL3xwgHdm85bCqBll1Ixpg3oilkHDnwRaAS+NaE9wXnAVjN7HvgR8DnnnAK6aeQaVjbmMnReYZT9vboSJCLzVzKV5j92HuGt559FLHLmPz/ZDJ2WXIpI8OLJFImxDO2NsTOeO3tRHeef3cwjOxXQSWUo5h66YtaBXz/J634GXDCXAS5EZwR0HdnWBft7hjj/7JYpXzuWzvC1J17m/VesHC8DLiJSCZ7Y08tgIsU7Ciy3BBVFEZHKketBVyhDB/DmjUv58x/uoS+epK2x8D4ifimqsbj4azyg8xrtrm5vwIyiCqM8truHz/37i/zHDl01EpHK8v3th2mpi3LV2vaCzzfXqiiKiFSG3vgoAB2NtQWfv/68pTgHj754rODzIn5SQFeB4hPuoauNhlm+qI59RRRGeWx3tu1Dd//wNHuKiPjrxcODXHpOa8HllgBNtVGGR9Nq2Csigctl6NqbzlxyCXD+2c0sa6nVfXRSERTQVaC4d4U6V+UScpUui8jQ7ekF4MDxkfIMTkRklvqGknRMsTSpuS475ylLJyJB6xlMAEw6Z5kZ15+3lMd295IYU9E6CZYCugo0scolwJqOBvb3DJHJTN6m77W+YV7uzWbxlKGThcDMbjCzl8xsr5ndWeD5O8xsp5ltM7Mfmtk5ec+lvSJO+YWcpEwyGUdffHTSq92QzdCBAjoRCV5vfJRwyGitn3zOumpdOyNjaV48MujjyETOpICuAk28hw6yGbqRsTSHBxKTvu4ne7LLLV+/YhHd/crQSXUzszBwD/A2YCPwPjPbOGG3XwJdzrkLgW8DX8h7bsQ5d5H3dSNSVgOJMVIZR1vD5Bm6Jm/OU2EUEQlaz2CStoYYoZBNus+5ZzUBsOeoAjoJlgK6CjR+D13s9IAOmHLZ5WO7e+hsreOade0cGUjoPhSpdpcDe51z+51zo8D9wE35OzjnfuScy6WrnwQ6fR6jeHrj2ftR2gqUAM/JVeZVQCciQeuZpAddvhWtdcTCITUYl8ApoKtA8USK+liYcN5VobUd2V50+yaZNMbSGX6+r49NGzpYsbiOdMZx+OTk2TyRKrAcOJD3c7e3bTIfBv497+daM9tqZk+a2bvKMUA55VTFuOkzdFpyKSJB640naZ+mHUEkHGJNR4MCOglcUX3oxF9Do6nT7p+DbB+UpprIpJUun321n3gyxab1HeMnRd39I6xYXF/28YoEpNA6mII3mZrZbwNdwLV5m1c65w6Z2RrgUTPb7pzbV+C1twG3AaxcuXLuo16gTmXopiiKksvQjShDJyLB6hlMsmFp07T7rVvSyLbukz6MSGRyytBVoHgyfVqFS8hWU1qzpJH9vYWvAj22p4dIyHjDujY6W+sAOKDCKFLduoEVeT93Aocm7mRm1wOfBG50ziVz251zh7zv+4EfAxcXOohz7l7nXJdzrqujo6N0o19g+rwMXftUSy5V5VJEKoBzjt4illwCrF/SxIH+YUZGVelSgqOArgLFE2NnZOgA1rY3sO9Y4QzdT3b3cMnKVpproyxrqcMMFUaRavc0sN7MVptZDLgFOK1apZldDHyFbDB3LG97q5nVeI/bgauAnb6NfAHqiycJGSyaomJcY42KoohI8E6OjDGWdtMuuQRYv7QR52BfEa2lRMpFAV0FGkqmaagJn7F97ZJGjgwkxoum5PTGk7xwcIBNG9oBiEVCnNVcq9YFUtWj1pMxAAAgAElEQVSccyngduAhYBfwoHNuh5ndbWa5qpVfBBqBb01oT3AesNXMngd+BHzOOaeArox64qMsboiddm/wRJFwiIZYWBk6EQlUrql4MRm6dUuyRet0H50ESffQVaDBZIrli+rO2J4rjPJyzxAXdLaMb3/Caya+acOp5WArWuuVoZOq55zbAmyZsO2uvMfXT/K6nwEXlHd0kq+viAIDkO1FN6gMnYgEaDygK2LOWtXWQDhkCugkUMrQVaChZIrGAhm6NV7rgolp/cd297C4IcavnH0qyOtsreOgAjoRqRC98eSULQtymmojDIwoQyciwemJ5zJ0089ZsUiIVW317DmmXnQSHAV0FSieTJ3WVDznnLZ6QnZ6QJfJOB7b08s169tPa37Z2VrH4ZMjjKkXnYhUgL6h0aIydM11UQaTytCJSHBOZehqi9p//ZIm9ihDJwFSQFeB4skz2xYA1ETCrFxcz/681gW7jgzQG0+yaf3p1fc6W+vJODiiXnQiUgH64qO0NRSz5DKie+ikKpjZDWb2kpntNbM7CzxfY2YPeM//wsxWedujZvYNM9tuZrvM7BN+j32h64kniYVD45V3p7NuSSOv9g0zmtJFdAmGAroKM5rKMJrKnNG2IGdNR+NpGbqf7O4B4BqvIErOeOuC4yqMIiLBSoyliSdTRS65jKoPncx7ZhYG7gHeBmwE3mdmGyfs9mGg3zm3Dvgy8Hlv+3uAGufcBcClwO/kgj3xR+/gKO2NMcwmL+KUb/3SRtIZxyt9hSuRi5SbAroKM+RVsCyUoYNsYZT9vUOkM9n+yY/t7uG8Zc0saTp9WUBna7ahuAqjiEjQck3Fiykw0KwMnVSHy4G9zrn9zrlR4H7gpgn73AR8w3v8beBNlo0gHNBgZhGgDhgFBvwZtkA2Q1dMhcucXKXLPUe17FKCUVRAV8SygTvMbKeZbTOzH5rZOXnP3Wpme7yvW0s5+GqUa0nQOGlA18hoKsOhEyMMJVM882r/eLuCfMsW1RIy1LpARALX6zUVLzpDlxjDOVfuYYmU03LgQN7P3d62gvt4bVhOAm1kg7sh4DDwGvAnzrnj5R6wnNIzOLOAbm1HI2aoMIoEZtqArshlA78EupxzF5KdiL7gvXYx8CngCrJXqz5lZq2lG371mS6gy1W63NsT5+f7+hhLO66dcP8cQDQcYllLnTJ0IhK4Pi9DV1xRlAhjaUdS96LI/FZord7EqxST7XM5kAbOBlYD/6eZrTnjAGa3mdlWM9va09Mz1/FKnt4i26zk1EazNQ5UGEWCUkyGbtplA865HznncqmgJ4FO7/FbgYedc8edc/3Aw8ANpRl6dRoP6ApUuYRTvej2HYvz2J4e6mNhLl1VOEZe3qqATkSC1zfDDB3AgHrRyfzWDazI+7kTODTZPt7yyhbgOPB+4AfOuTHn3DHgp0DXxAM45+51znU557o6Os68sCuzk844+ma45BJgXUcje7XkUgJSTEBXzLKBfB8G/n2Wr13w4tPcQ7e4Icai+ij7e4f4ye4efnVNGzWRM3vWQbYwipZcikjQemaSofMuZqkXncxzTwPrzWy1mcWAW4DNE/bZDORuRbkZeNRl1xq/BrzRshqAK4EXfRr3gnd8aJSMY+YB3dJGXu4dIqV2URKAYgK6YpYNZHc0+22yV5G+OJPXatnAKbmiKJNVuTQz1rQ38JOXeni1b5hNGya/KtfZWs/hgYTK6IpIoPriozTWRKiNFr74lK/Zy9ANKkMn85h3T9ztwEPALuBB59wOM7vbzG70dvsa0GZme4E7gFyNgnuARuAFsoHh3zrntvn6P7CA9c7gAlS+9UuaGE1neE3VxSUAxTTYKGbZAGZ2PfBJ4FrnXDLvtddNeO2PJ77WOXcvcC9AV1fXgr4TPp6YOkMH2Ztvn33tBMA0AV0dzsHhkyOc09ZQ2oGKiBSpN54sarklZPvQAQyo0qXMc865LcCWCdvuynucINuiYOLr4oW2iz/Gm4rPMEO3Plfp8lh8vN6BiF+KydBNu2zAzC4GvgLc6K33znkIeIuZtXrFUN7ibZNJTHcPHcBab9JYsbiOVW31k+63Qq0LRKQC9A0VX2CguU4ZOhEJznhAN8MMXe7cbK8Ko0gAps3QOedSZpZbNhAG7sstGwC2Ouc2k11i2Qh8y2vC+Jpz7kbn3HEz+wzZoBDgbpXendr4PXSxyf9o1rRns23XbuiYsullrrm47qMTkSD1Do5yzhQXn/LlMnTqRSciQRhfcjnDDF1jTYSzW2oV0EkgillyWcyygeuneO19wH2zHeBCM5RMURcNEw5NHqhd2LmIhliYd1xw9pTvtayllnDIlKETkUD1DSW55JziOtaMV7kcUYZORPzXM5ikLhqmITb9Pb8TrVvapF50EoiiAjrxTzyZmnK5JcBZLbW88EdvnTI7BxAJhziruVYBnYgEJp1xHB8apaPIe+gaYmFCpgydiASjx2tZMN05ViHrlzTyj7/oI5NxhKa4MC9SasXcQyc+iifTkzYVz1fsRKPWBVLNzOwGM3vJzPaa2Z0Fnr/DzHaa2TYz+6GZnZP33K1mtsf7unXia6U0+oezJcDbirwfxcxoqo3qHjoRCUS2qXhxF6AmWr+kkcRYhoMndCFd/KWArsLEE2NFBXTF6myt58BxTSxSfcwsTLa899uAjcD7zGzjhN1+CXQ55y4Evg18wXvtYuBTwBXA5cCnvMJNUmK5puIzKQHeVBtRlUsRCUTP4MybiuesG690qWWX4i8FdBVmKJmmoWbm67Yns2JxHUcHEyRT6ZK9p0iFuBzY65zb75wbBe4HbsrfwTn3I+dcLkX9JNnWKQBvBR52zh13zvUDDwM3+DTuBSVXYKDYtgWQ7UWnDJ2IBKEUAZ0Ko4jfFNBVmMFkisaaaMner7O1PtuL7kSiZO8pUiGWAwfyfu72tk3mw8C/z/S1ZnabmW01s609PT1zGO7CNJsmvU21EQZGlKETEX+NpTP0D4/NuKl4zqL6GB1NNew5qoBO/KWArsIMJVM0ljBDd6p1gZZdStUpdCOpK7ij2W8DXWRbrMzotc65e51zXc65ro6OjlkNdCE7teRyBhm6uigDytCJiM9y89VsM3SQvY9ujzJ04jMFdBWmmCqXM6FedFLFuoEVeT93Aocm7mRm1wOfBG50ziVn8lqZu954kkjIaK4tfuVBU21EVS5FxHezbSqeb92SRvYei+NcwWuEImWhgK7CxJMpGkpYFOWsZvWik6r1NLDezFabWQy4Bdicv4OZXQx8hWwwdyzvqYeAt5hZq1cM5S3eNimxvvgobY2xGZXwbq5Vhk5E/DfbpuL51i9pJJ5McXQgOf3OIiWigK6CjKYyjKYyNJUwoIuEQyxrqeWAMnRSZZxzKeB2soHYLuBB59wOM7vbzG70dvsi0Ah8y8yeM7PN3muPA58hGxQ+DdztbZMS640naWuY2clRc22EeDJFJqMr3CLin9Jk6JoAVboUf6mxeAUZSmaXGJUyQweworVeGTqpSs65LcCWCdvuynt8/RSvvQ+4r3yjE4DeodEZX+1uqo3iHMRHUzNaqikiMhc9XoZuTvfQLfVaFxyNc8163Xct/lCGroLEvYCulH3oQM3FRSQ4ffEk7Q0za9LbXJedA3UfnYj4qWcwSVNNhNro7IvTtTXEaK2PqjCK+EoBXQUpX0BXz9GBpHrRiYivnHPZJZczqHAJ2QwdoF50IuKrnvjse9DlmJlXGEVLLsU/CugqSG7JZSmrXMKpSpeH1ItORHw0PJomMZaZcU+nJm8OVC86EfFTz2ByTgVRctYtaWKPKl2KjxTQVZDBMt1Dp9YFIhKEXMW4thkGdM3K0IlIAHoHk3MqiJKzfkkjJ4bH6BsaLcGoRKangK6CDJVryeXiekDNxUXEX72zaCoOpzJ0uodORPxUiiWXcHphFBE/KKCrIPFEeQK6s5priYSMA8eVoRMR/4z3dJrxkstshk696GQ+M7MbzOwlM9trZncWeL7GzB7wnv+Fma3Ke+5CM/u5me0ws+1mVuvn2BeixFiawUSqJAHduiXZgE730YlfFNBVkHiZllyGQ8bZi+qUoRMRX/V5GbqZF0VRhk7mNzMLA/cAbwM2Au8zs40Tdvsw0O+cWwd8Gfi899oI8A/AR51z5wPXAbq6UWa5HnQzXVFQyFnNtTTWRNirSpfik6ICuiKuMm0ys2fNLGVmN094Lu019B1v6iuFlavKJah1gYj4ry93D90MG4vXRsP/f3t3Hh93dR76//PMptFoX7xJsryzeAMvmJ2UUAhLCFlIINDUaZKSttCb3Ka3TZqWJrRJE3qb5TZLA4QEsgG/JNy6Nw6EJKQQsMFmsbHMZsubZMvWLs1Is5/fH9/vyEKW5JE0M9+Z0fN+veY131m/j8bS8TznPOccfB4XA8P6HVYVrE3APmNMqzEmCjwE3DDmOTcAD9jHPwWuEBEBrgJ2G2N2ARhjuo0xukx1lnVlYA+6lNRKl7p1gcqV0yZ0afYyHQY+DPx4nLcYNsaca1/eNcN4i1ooEqfU68btkoy/t5XQ6QidUip3uoIRKv0efJ6pF4NU+r0M6AidKlyNwJFRt9vs+8Z9jjEmDvQDdcAZgBGRx+3O8r/JQbyzXmqEbk55ZqpbV2hCp3Ionf9lT9vLZIw5aIzZDSSzEOOsEYzEM75lQUpTTYATgxHCMe3kU0rlRlcoOu0lwCv9Hl3lUhWy8Xpmx65hP9FzPMAlwK329XtE5IpTTiBym4jsFJGdnZ2dM4131utMzfmtmHnJJVgLo3QORujRlS5VDqST0KXTyzQZv93gbBeRd08pulkmGElkpdwSRu9Fp6N0Sqnc6BqMUD/FcsuUCr9HR+hUIWsDFo663QQcneg59ry5KqDHvv+/jTFdxpghYCuwfuwJjDH3GGM2GmM2zpkzJws/wuzSNWjP+Z1mmzXWRcvqAfj5i20ZeT+lJpNOQpdOL9Nkmo0xG4FbgK+JyLJTTqC9TIBVcpm9hM7auuCIll0qpXKkOxSd8oIoKZWlXh2hU4VsB7BCRJaIiA+4GRi7jsAWYLN9fCPwW2PtRP04sFZEAnai9zZgb47inrU6g2FqAt5plYiPZ3VjFZuW1PK9Zw4ST2gBm8qudH5r0+llmpAx5qh93Qr8Dlg3znO0lwlr24KyEndW3nthrW4urpTKre5gZMpbFqRU+D26KIoqWPacuDuwkrNXgUeMMS0icpeIpNYT+C5QJyL7gL8CPm2/thf4ClZS+DLwojHmF7n+GWabzsHpt1cT+dglS2jvG+axlo6Mvq9SY6UzHDTSywS0Y/Uy3ZLOm4tIDTBkjImISD1wMXD3dIMtdsFInIbq0qy899wKP1636MIoSqmciCWS9A7Fpj9C5/fqtgWqoBljtmKVS46+785Rx2Hg/RO89odYWxeoHOkKRjOywuVoV5w9j8V1Ae59+gDXrVmAtYipUpl32hG6dHqZROQ8EWnDapi+IyIt9svPBnaKyC7gSeBLxhgtG5hAMBKnPEsjdLoXnVIql3rthQBmMkKnCZ1SKlc6ByMZT+jcLuEjlyxh15E+Xjzcm9H3Vmq0tAqFjTFbjTFnGGOWGWO+YN93pzFmi328wxjTZIwpM8bU2RthYox51hizxhhzjn393ez9KIUvlMVVLkH3olPFR/fIzF8jK8ZNc4Suwu9lOJYgpnNPlFI5kI2SS4AbNzRRVerl3qcOZPy9lUrJzMxPlRGDkThlWVoUBaCpOqAjdKpo6B6Z+a07aK8YN80vSJV255aO0imlsi0UiTMcS2R8hA4g4PNw6/nNPL63g0PdoYy/v1KgCV3eiMaTRONJyn3ZHaHr1L3oVPHQPTLzWHcoNUI33ZJLL4CudKmUyrrUpuLZGKED2HzRYjwu4XvPHMzK+yulCV2eCEWsXuhsllwurLW2LtBROlUkcrJHpm6rMj0jezpNu+TSagsHhnWETimVXakS8WyM0AHMq/Rz/doGHtl5hH5dvVdlgSZ0eSJoJ3RZLbm0Nxc/ovPoVHHI+h6ZoNuqTFdXKILP46Jimm1aZamO0CmlcuPkCN30OqDS8ZFLljAUTfDQ84ezdg41e2lClydSCd10v/yko9FO6Np1hE4Vh6zvkammr2swSn2Zb9rLdI+M0GlCp5TKsjeODyICS+rLsnaO1Y1VXLi0ju8/e1AXe1IZpwldngjlYIRuboUfj0to79OEThWFkT0yRcSHtUdmWqtVikiNiJTYx6k9MnVLlQzqDkWmvSAKWPvQAQzooihKqSzb0z7AsjnlBLK4jgHAxy5dwrH+MFtfOZbV86jZRxO6PDGYgzl0bpewoNqvI3SqKOgemfmtKxiZUflS5ciiKJrQKaWyq+VoP6sbKrN+nsvPnMvS+jK++/sDGDOVGQJKTS67XREqbSOLomRxhA6gsbpUR+hU0TDGbAW2jrnvzlHHO7BKMce+7llgTdYDnMW6g1HOmj/9L0jlI4uiaMmlUip7uoIRjvWHWd1YlfVzueyNxv/+/+7h+QM9nL+0LuvnVLODjtDlidwldAEdoVNKZZUxhu5gdEZLgLtdQnmJR0folFJZ1XJ0AIBVDdlP6ADet76JmoCX+36vG42rzNGELk+kvrRkcw4dWAujHB8ME43rhFylVHYMhONEE8kZrxhX4ffooihKqaza094PwMoclFwClPrc/NEFi/j1q8d59KW2nJxTFT9N6PJEKGJt9p3tEbqmmlKMgY7+cFbPo5SavbrtPZ2muwddSoXfo9sWKKWyquVoP4vqAlTZW6Xkwp+9bRkXLq3jfz68i3ufas3ZeVXx0oQuTwQjMUq9btyu6S3xna6mamvrgrY+3YtOKZUdXUFrU/GZlFyCtTCKllwqpbJpT/sAq3NUbplSVuLhe39yHtetWcAXtr7KF7e+SjKpi6So6dNFUfJEMJLI6gqXKboXnVIq20ZG6MpmltBV+D102u+llFKZ1j8U43DPEDdvWnj6J2dYicfN//ngOurKfdzzVCudgxHuvnEtXreOtaip04QuTwQj8ayXWwIsqCpFBF3pUimVNV0he4SuYmYll5WlXlq7QpkISSmlTtFyzJo/l+sRuhS3S/j8u1Yxt6KE//2rN+gJRfnWreuzvp6CKj7aDZAnQjlK6HweF3MrSmjTETqlVJZ0DUYQgdpAJubQacmlUio7WtpTK1zmZkGU8YgId7x9BV967xqefrOTW+57jh67U0ypdGlClyeC4ThlJe6cnKuxulRLLpVSWdMdilAT8OGZYelQhd/LwHBMN+BVSmXFnqP9LKjyUzfD+b6ZcPOmZv7jjzbw2rEBbvz2sxzu1rUOVPo0ocsTuSq5BGisCWjJpVIqa7oGo9SVzWx0DqxFUeJJQzim26yowiMiV4vI6yKyT0Q+Pc7jJSLysP34cyKyeMzjzSISFJG/zlXMs82e9v6c7T+XjqtWzeeHHzufnqEo7/nWM7x4uNfpkFSB0IQuT+Q0oasu5Vj/sK6opJTKiu5QZMZbFoBVcgnoXnSq4IiIG/gmcA2wEvigiKwc87SPAr3GmOXAV4Evj3n8q8Avsx3rbBWKxGntCrG60blyy/Gct7iWn//5RZT7PXzwnu38Yvcxp0NSBSCthC6NXqbLRORFEYmLyI1jHtssIm/al82ZCrzYhCLxnE2CbawpJZYwnBjU1eOUUpnXHYzOeMsCsBZFAXQvOlWINgH7jDGtxpgo8BBww5jn3AA8YB//FLhCRARARN4NtAItOYp31nn12ADGOLcgymSWzinn0b+4mNWNVdz+4xf5j//er6XnalKnTejS7GU6DHwY+PGY19YC/wicj9W4/aOI1Mw87OIzGInnZNsCOLkXXbvuRaeUyoLOYCQjCd3JETpdGEUVnEbgyKjbbfZ94z7HGBMH+oE6ESkD/hb4fA7inLX2tNsrXDbmX0IHUFvm40cfO593rl3Al375Gn/36CvEElp+rsaXzgjdaXuZjDEHjTG7gbG/ae8AnjDG9BhjeoEngKszEHdRicaTRONJyn25G6EDdKVLpVTGReIJBsNx6jNQclmZSuiGdYROFRwZ576xQywTPefzwFeNMcFJTyBym4jsFJGdnZ2d0wxz9tpzdID6ch/zKp1fEGUifq+b/3PzOm6/fBk/ef4IH/n+Di1BV+NKJ6FLp5dpRq+d7Y1SKGL1PudqhK5xZIROEzqlVGZ1B63ltjOxalylP1VyqSN0quC0AaN3q24Cjk70HBHxAFVAD1ZV090ichD4JPB3InLH2BMYY+4xxmw0xmycM2dO5n+CIpdaEMWucs1bLpfwv95xFl9+3xq27e/m/d/eRldQp8wUCmMM4ViCnlCUtt4h9p0YzMp50skg0ullmtFrjTH3APcAbNy4cdYVCQfthC5Xc+jKSjzUBLy6dYEqeCJyNfB1wA3cZ4z50pjHLwO+BqwFbjbG/HTUY5uBv7dv/rMx5gHUjI0kdBlY5bLCTui0R1oVoB3AChFZArQDNwO3jHnOFmAzsA24EfitsSZKXZp6goh8DggaY76Ri6Bni3AswZsnglxx9lynQ0nbTec101Bdyp8+uJPN9z/PT267YKTTS+XWicEwe48O0B2M0h2K0B2KWsdB67h3KMpwNMFQNMFwLMHY6Y/7v3gtbldmOxLSySDS6WWa7LV/MOa1v0vztbNGKGoldBU5SujAKrvUETpVyEbN770Sq63ZISJbjDF7Rz0tNb/3r8e8NjW/dyNWJ9ML9mt1jegZSvUc11dkbg6djtCpQmOMidujao9jdTjdb4xpEZG7gJ3GmC3Ad4EfiMg+rJG5m52LeHZ5vWOQRNLk5YIok7l0xRy+fesG/vTBnXzs+zt58KOb8Htzs4fxbNY5GOG5A91s29/N9tZu9neG3vK4z+2irtxnXcpKWFJfRsDnIeBzE/C5KfW5CXjdBHweSn3Z+fdKJ4NIp5dpIo8DXxy1EMpVwGemHGWRC4ZzO0IHVtnl2F9IpQrMyPxeABFJze8dSeiMMQftxyac32s/nprf+5Psh13cRhK6spkndAGfG7dLdJVLVZCMMVuBrWPuu3PUcRh4/2ne43NZCW6W23M0vxdEmczlZ83lKzedyyceeom/+NGLfOdDG/C6dReyTDHG0N43zAuHetlxsIftrT3sO2FNZy0v8XDe4ho+sHEh65prmFtRQl25j/ISj+Olu6fNINLpZRKR84BHgRrgehH5vDFmlTGmR0T+CSspBLgr9QVKnRTM8Rw6gMbqAE+90YUxxvFfQqWmabw5uufP4LXpzg1Wk+gOWSWX9RUzL7kUESr8HgaGdYROKZU5LUcHqPR7aLIXiSs07zqngcFwjM8+uodPPbKLr910Lq4Ml/DNFtF4kr3HBnjhUC8vHOrhhUO9HB+wOiZTCdz7NzRxwdI6VjVU4snT5DmtDCKNXqYdWOWU4732fuD+GcRY9EYSuhyXXA7HEvQOxajNwFwXpRyQ9fm9YC3aBNwG0NzcnObbz15dgxFK7dKSTKj0e3WETimVUS3t/axuzP8FUSZz6/mL6B+Ocfdjr1NV6uWuG1YV9M+TTeFYgrbeIY70DHO4Z2jkcqRniANdISJxq4insbqUC5bWsWFRDRsW1XDmvIq8TeDGyl0GoSYUciKhS6102TusCZ0qVDmZ3zvbF22aqu5QlLoMbFmQUuH36Bw6pVTGxBJJXu0Y5MMXLXY6lBn787cto38oxneeaqWq1Mtfv+NMp0NyRCJpONIzxJHeIdp6h0eStzb79onBt64K6ve6aK4N0Fwb4JLl9axrthK4+VV+h36CmdOELg8MOjCHLlVm0N43xJqmwqshVwqd35uXjvUPZ2TLgpQKv0dXuVRKZcy+E0Gi8SSrGiqdDmXGRIRPX3MW/cMxvvHkPqpKvfzpZUudDivrwrEEu9v62XGwhx0HrTLJ0R1/HpewoNrPwpoAf3DmHJpqrORtoZ3E1Zf7im40UxO6PBCKJABnRuh0c3FVqHR+b/5pOdrP9tYebr98Wcbes9Lv5XDPUMbeTyk1u+1pL9wFUcYjInzhPWsYDMf5wtZXcbuEj1yyxOmwMsoYw+62fh5r6WDHgR52t/UTTVhlkivmlvPOtQ2sW1jNoroATbUB5lWUFEypZKZoQpcHgpEYpV53xvekmEx1wEvA59atC1RB0/m9+eVfH7fmctx2WeYSugq/l4FhHaFTSmVGy9EBynxultSVOR1KxrhdwldvOpd4Msld/28v8WQyo+2wU3pDUR59qZ1Hdh7htY5BPC5hTVMVf3LxYjYurmXDohqdNmTThC4PBCOJnJZbgtWj01RTqpuLK6UyYntrN797vZPPXHMWVaWZ2+y2slTn0CmlMmdPez8rGyqLblVIn8fFN25Zzycffpkvbn2NWMJw++XLnQ5ryhJJwzP7unh45xGeaDlONJHknKYqvvCe1Vx/ToNupj4BTejyQDASH9lAN5caq3VzcaXUzBljuPux15hXWcLmDC80UOH3EozGSSZN0X0BU0rlViJp2HtsgA9sXHj6Jxcgr9vF1286F69L+NfHXyeWSPKJK1YUxHyxUCTOj587zPefPUh73zDVAS+3XtDMTect5Kz5hT/fMds0ocsDoUicspLs7Bw/mcaaUl460pfz8yqlissTe4/z4uE+/uW9a/B7M9uWVfo9GAODkXhGR/6UUrPPga4QQ9FE0cyfG4/H7eLfPnAuHreLr/36TeIJw6euOiNvk7qBcIwfbDvEfU+30jsU48KldXzm2rO4cuU8Sjy5/25cqDShywPBcDynC6KkNFYH6BuKEYw4c36lVOFLJA3/+vjrLK0v4/0bxp2uOCOp8prBcEwTOqXUjLQcTS2IUtwjPm6XcPf71uJ1C994ch+xRJJPX3NWXiV1fUNR7n/mIN9/5gAD4ThvP2sut1++nA2Lak7/YnUK/RafB4KROA3Vud/7orHm5F50Z86vyPn5lVKF7+cvtn9j53IAAB3XSURBVPHmiSDfunV9VlYVS5WjDwzHrbVKlVJqmva091PicbF8TrnToWSdyyV84d1r8LhcfOepViLxJH9/3dmOr/7YFYxw39MH+MG2g4SiCd6xah5/+fYVRT1qmgua0OWBUNSpEbqTe9FpQqeUmqpwLMHXfv0ma5uquGb1/Kyco2LUCJ1SSs3EnvYBzlpQ6XhSkysul3DXDavwul3c/8wBnj/Qw7+8dw3nLKzOeSxtvUPc+1QrD+04QiyR5J1rG7j98uX6/TNDNKHLA8FwPOerXMKozcV1pUul1DT8cPsh2vuGufvGtVkr5akstdpGXelSKTUTxhj2HO3nXec0OB1KTokI//DOs9m0pIY7/7OF93zrGTZftJhPXXVmTgYT9p0Y5Nu/a+U/X25HBN67romPv20pS2fBKGkuaUKXB4KROOUOrHI5p7wEn9tFm650qZSaosFwjG8+uY9Lltdz8fL6rJ0nNUI3oCN0SqkZONIzzGA4PitL+0SEq1cv4KLl9fzrY6/z/WcP8vieDu66YTV/uHJeVs65u62Pbz25n8f3dlDicfGhCxfxp5cupcGuDlOZpQmdw2KJJJF4knJf7v8pXC5hQbVfR+iUUlN279MH6B2K8TdXn5nV81T6dYROKTVz21q7AFjdMPsSupRKv5d/evdq3r2ukb/7+St87MGdXLN6Pp971yrmVc58LYfWziC/3NPB1leO0XJ0gEq/hzsuX86HL1pMXXlJBn4CNRFN6BwWilhfUpwYoQPdi04pNXWdgxHue7qV69YsYG1Tdudi6Bw6pdRMReIJ/v23+1jVUMmqhuJe4TIdGxbV8F9/eQn3Pt3K13/zJr997QQbF9dw4dI6LlxWx9qmarxpzjN88/ggW1/p4Jd7jvFaxyAA65ur+Yd3ruQDG5tG2nCVXZrQOSzV6+zEHDqw5tH97vVOR86tlCpM3/rdPiLxJJ+66oysn8vncVHicTGgI3RKqWn60fbDtPUO88X3rMHlyp+l+53k87i4/fLlXLdmAQ9uO8S21m7+96/eACDgc7NxcS0XLK1ldUMVQ9EEfUNReodi9A1F6RuK0TsUZX9nkP2dIUTgvEW1/OP1K7l69XwWVGlZZa5pQuewUNT6klLhUELXWB3gxGCESDyhGzgqpU7LGMMvX+ngqpXzcjapvbLUqyN0SqlpGQzH+MaT+7hoWR2XrsjefN9Ctbi+jDuvXwlAbyjKcwe62ba/m22t3dz92OunPN/ndlEd8FIT8NFUE+DDFy3mHavmMzcDJZtq+jShc1jQ4RG61F50R/vCLKkvcyQGpVThONQ9RMdAOKsLoYxV4fdY+9ApVUBE5Grg64AbuM8Y86Uxj5cADwIbgG7gJmPMQRG5EvgS4AOiwP8yxvw2p8EXkXufPkBPKMrfXp1fG2vno5oyH1evXsDVqxcA1p5x+04EqfB7qA74qAl4KfW69XPMQ2kVyIrI1SLyuojsE5FPj/N4iYg8bD/+nIgstu9fLCLDIvKyffmPzIZf+IJ5MIcOdOsCpVR6trd2A3DB0rqcnbMm4KNjIJyz8yk1UyLiBr4JXAOsBD4oIivHPO2jQK8xZjnwVeDL9v1dwPXGmDXAZuAHuYm6+Iye7+vE3muFrr68hAuW1rGqoYrG6lICPo8mc3nqtAndDBslgP3GmHPty59lKO6iMZLQOTiHDqzNxZVS6nS2t3ZTX17Csjm5G9G/eFkdLx3upSsYydk5lZqhTcA+Y0yrMSYKPATcMOY5NwAP2Mc/Ba4QETHGvGSMOWrf3wL47dE8NUX//ts3czbfVyknpTNCN+1GKXNhFq/UKpdOlVzOr/LjEh2hU4VJqwdyyxjD9tYeLlham9Ne2mvWLCBp4LE9HTk7p1Iz1AgcGXW7zb5v3OcYY+JAPzB26Pt9wEvGGO3NmKJD3SF+/NxhbjpvoW5irYpeOgndTBulJSLykoj8t4hcOt4JROQ2EdkpIjs7O2fXioupVS6dGqHzul3Mq/Tr5uKq4Gj1QO6l5s/lstwS4Kz5FSytL2PrK8dyel6lZmC8Hg8zleeIyCqsNuvj455gFn93Sse//eoNPG7hE1escDoUpbIunYRuJo3SMaDZGLMO+CvgxyJyygYgxph7jDEbjTEb58yZk0ZIxSMUSQBQ5nNuhcnG6lIdoVOFSKsHcsyJ+XMAIsK1axawvbWbbi27VIWhDVg46nYTcHSi54iIB6gCeuzbTcCjwB8bY/aPd4LZ/N3pdPa097Nl11E+cvGSjGyYrVS+Syehm3ajZIyJGGO6AYwxLwD7AS1kHiUUjVPqdeNJcwPHbGis0c3FVUHKevWAeisn5s+lXGuXXT7ecjzn51ZqGnYAK0RkiYj4gJuBLWOeswVr0ROAG4HfGmOMiFQDvwA+Y4x5JmcRF5G7H3+d6oCXj79tmdOhKJUT6WQRM2mU5thlUYjIUmAF0JqZ0IvDYDju2Py5lMbqUjr6wySSYwdelcprWa8eAC1rSnFq/lzK2QsqWFwX0LJLVRDsDqQ7gMeBV4FHjDEtInKXiLzLftp3gToR2YfVDqXmAd8BLAf+YdQ837k5/hEK1rP7unjqjU5u/4PlVJV6nQ5HqZw4bSZhjImLSKpRcgP3pxolYKcxZgtWo/QDu1HqwUr6AC4D7hKROJAA/swY05ONH6RQhSJxKhzasiClqSZAPGk4PhCmwd7GQKkCMJXqgbYx1QMGiIBVPSAiqeqBnWNPYoy5B7gHYOPGjbO218Op+XMpqbLL7zzVSk8oSm2Zz5E4lEqXMWYrsHXMfXeOOg4D7x/ndf8M/HPWAyxCxhi+/NhrNFT5+dCFi5wOR6mcSavOzxiz1RhzhjFmmTHmC/Z9d9rJHMaYsDHm/caY5caYTcaYVvv+nxljVhljzjHGrDfG/Ff2fpTCFIzEKStxbv4cnNxcXMsuVYHR6oEccmr+3GjXrllAImn4VYuudqmUOtWvXz3BrrZ+PnnlGfi9zn63UiqXnJu4pQAroXNqhcuU1Obibb26F50qHDMsaboM2C0iu7AWS9HqgdNwcv5cyqqGSpprA/xCyy6VmpZoPMmvWjpIFukUi+8/e4CGKj/vXTd2OrVSxc3ZTEIRDMdpqHZ2BaZUQqcrXapCM4OSpp8BP8t6gEXC6flzKamyy3ufbqU3FKVGyy6VmpJf7e3gjh+/xIMf2cRlZxTXyphvHh/kmX3d/M3VZzq60JxSTtDfeIeFos6P0JX63NSV+bTkUik1Lqfnz412nV12+cReXe1Sqam6cuU8agJefvL8YadDybgHtx3C53Fx08aFp3+yUkVGEzqHBfNglUuw5tG16QidUmoczx1wfv5cyurGShbWlmrZpVLTUOJxc+OGJp7Ye5wTg2Gnw8mYgXCMn73YxvVrG6grL3E6HKVyThM6hwUjccodXuUS7M3FdYROKTWO7a09js+fSxERrl29gGf2ddE3FHU6HKUKzs2bmoknDT99oc3pUDLm5y+0MRRNsPkiXdlSzU6a0DkolkgSiScp9+VHQne0bxhrNXellLJY8+e6Od/h+XOjXbtmAXEtu1RqWpbNKef8JbU89PyRolgcJZk0PLjtEOuaq1nbVO10OEo5QhM6B4UicYD8GKGrKSUcS9Id0h5vpdRJh3uGONafH/PnUtY2VdFYXaqbjCs1Tbec38zhniGe2d/ldCgz9vt9XbR2hdh84WKnQ1HKMZrQOahvKAaQH3PodKVLpdQ4UvvPXbi01uFITrJWu5zP7/d10T8cczocpQrOO1bNL5rFUR7cdpD6ch/XrJnvdChKOUYTOof0D8f45MMv43EJKxdUOh0Oy+aWA/DoS+0OR6KUyifW/Dkfy+aUOx3KW1y7ZgGxhOHXWnap1JT5vW7et76JX7Ucp3Mw4nQ403akZ4jfvHaCD25qpsSjG4mr2UsTOgf0DUX5o/ueo+VoP9+6dT2rG6ucDollc8r5k4sX8/1nD/JLLWNSSjF6/lxd3syfSzl3YbWWXSo1A8WwOMoPth/CJcKt5+tiKGp204Qux3pCUT5473O8fnyQez60katW5U+JwGeuOZtzFlbzNz/dzcGukNPhKKUclo/z51JEhGtWz+fpN7sYCGvZpVJTtXyuvTjKjsMFuTjKcDTBwzuOcPWq+cyv8jsdjlKO0oQuhzoHI3zwnu20dga57483cvlZc50O6S18HhffvGUdLpfwFz96kXAs4XRISikH5eP8udGuWbOAaCKpZZdKTdMt5zdzqHuIZ/d3Ox3KlG3Z1U7/cIw/vlBH55TShC5Hjg+EufmebRzuGeJ7Hz6Py86Y43RI42qqCfDVm85h77EBPv9fe50ORynloHydP5eybmE1DVV+vvzYa9z3dKuO1Ck1Re9YNZ/qAlwcxRjDA88e4qz5FWxakp8dTkrlkiZ0OXC0b5ibvrONjv4wD3xkExctr3c6pEm9/ax5/PkfLOMnzx/m0ZcKt7ZeKTV9+Tx/LsXlEv79lnUsqivjn3/xKhd+8Td8bksLh7q1ZFypdKQWR3m8paOgFkfZeaiXvccG2HzR4rxtn5TKJefXyy9CiaThQFeQ3W39vNLez2N7OgiG4zz40U1sWFQYPUmfuvIMXjjYy9/9fA+rG6pYMa/C6ZCUUjmUz/PnRtuwqJZHPn4hr7T1871nDvCj5w7xwLaD/OHZ8/joJUs4f0n+bIiezxJJQyyRJJ40JJIGDBgMxjrEmMnnWBkgaaznJ+z3MAYSxjqOJ5NEYkmiiSTRuHWJxK3bkViCcNy+jiUIx5LWdTxBLG7FFU0kiSWSxBL27ZHXJonEE0Ts94vErOMPbFzIP717dU4+u0L3wU3NfPf3B/jZi2382duWOR1OWh549iCVfg83nNvgdChK5QVN6DIgFInz5OsnePlwH7vb+2lp7ycUteaflXrdrGmq4rPXWguOFAqP28W/37KOa7/+NH/xoxf5zzsuJuDTXxelZot8nz831pqmKr5y07n87TVn8cPth/jRc4d5Yu9xKvwe6sp81Jb5qC0robbMS21ZCXVlPir8Hkp9bgI+D6VeN6U+N6VeNwGfm7ISD5WlnrxbCt0YQyiaoH84Rv9QjIFwjMFwnMFwjIFh6zh1XzASJxxLMBS1LqOPo/EE8aQhnjDEkklOk6/llEuskaMSj4sSjxuvR/C6XfjcLrxuFx634HW5KC/xUFfmpsTrGnluicdFidfFuoU1Tv8YBWP53HI2LanloecPc9ulS3G58rcDpHMwwo6DPTy2p4MPX7RYv5coZdO/hGlKJK1ypJ+92MZjezoYiibweVysXFDJ+zY0saaxinMWVrNsTjnuPG4cJzOv0s/Xb17Hh+5/jr9/dA//9oFztKe7CCSThqFYgqFonGH7y92w3TMeGdUznuolH44lCI963nAswbB9/E83rGZhbcDpH0llUHcwwjP7u/nh9sN5PX9uIvMq/XzqqjO5/fLlbNl1lL1HB+gORekJRWjvG+aV9j56QlFiifQyGJ/HRaXfQ6XfS4XfQ2Wpl/ISD2UlHvvaffLY58HvdeN2gdvlwuMS3C4ZuU4aiCeT9giTGTmOJQzD0QTBSJyQfRkcOU4wGI5ZCdxwjIFw3BpBm0Sp102F34oplaRW+D3MqywZSVxLPG48LsHjduF1Cx6XnSi5BZcIIoIAItjXwumafxHBLYJLrHJYtwguF7jkZELm84y6uE8mYn6fC7/Xjd/jxusW/b8mx27Z1MwnH36Zba3dXJwn00LiiSSvdQzywqFeXjxsXY70DANQE/Cy+aLFzgaoVB5JK6ETkauBrwNu4D5jzJfGPF4CPAhsALqBm4wxB+3HPgN8FEgA/8MY83jGonfAm8cH+flL7fzfl9o51h+mosTDu85p4D3rGlm/qAavu7imJV6yop5PXLGCr/36TX7z2gnWNFaxpqmKtfZ1Y3VpQf3Ha4whmkgSiiSsL0vRk1+aUknNUPRkwjJs92qnSpFSvdkJ+zieTE54LsH6MiNifUFyCfYXJcbcPnnsEk75PMd+vMmkscuYIJFMkkhapU7xpCFqlx6F7bKjVDlSOJZk2E7iwrGJY57wZxHrS+LYUYxIfOrvlUnaNs1cOJZg58Fent7Xye/f7KLl6AAAlX4Pf3XlGQX19z2a3+vmAxsXjvuYMYbBSJxgOP6WkSvrb966LxSJMzBqtGv06FdHf5hQxBoBC0UTp02wpsLjEsr9VnKYShirAz4W1ZVRVeqlstRDVanXOvZb1xV+6/4KO+kstv+HClUhtU9Xr55P9X95+fHzhx1L6I71D/Py4T5ePtLHS4f7eKW9n2F7te15lSWsb67hjy9YzPpF1axqqMLvza/Rc6WcdNqETkTcwDeBK4E2YIeIbDHGjF4C8aNArzFmuYjcDHwZuElEVgI3A6uABuDXInKGMaYg1sNPJg0Hu0Psautj15F+nj/Qw95jA7hdwtvOmMNnrzubPzx7XtE3Kn/59hU01wbYcbCX3W193PtUK3H7C0xtmY9VDZVUlXqtXlav6y3XJV4X6Q5Qji75MfbtpH2nMYakOXmf1bNtz/mwr2MJK1kLj0nMRn9ZC0XiI7GnI1X643W77N5sqyfb6071vLsm7LVOzScZfZ1MXSfNyJwT6+d662OjP4e3fkjgdqd6vq1r96hRAJ/HRYldqvSWciS3i0DJydKygM9NoMRDwD72e63npXrI/alj+/ESjyvvvtjnc9v0lSfeoLUzyFnzKzhzfiVnzqugqabUsVKmZNJwYjDCoe4Qh3uGONIzxOGeIQ71DLH36ACReBKvW1jXXMOnrjyDS1bUs6axCk+RJgYiQqXfSohmyhhDJJ4cGV2Lxk/OQ7Ouk3bnj7FHqqzRKq89MpYqISz1WiN9+fi3pqYun9un8aQWR3lw20H+ZeurnLOwmrVZ6LSNJZJ0DkY41h+moz/MoZ4Qu45YSdzxAWtRFp/bxcqGSm46byHrF9WwYVENDVV+/btQahLpjNBtAvYZY1oBROQh4AZgdKN0A/A5+/inwDfE+su7AXjIGBMBDojIPvv9ts008M7BCFd+9b+pCfioKvVSHfBSXeqlOuCjOmD1WpZ63W8p7Ugdl3hcJO2J28nUf7r2F+lwLMlrHQO8fKSPXUf6GAjHgZNz4e5850quP6eBORUlM/0RCobbJbx3fRPvXd8EWD36r3UM8kpbH7vb+nmtY5D23uG3jAyFY4kpJU7Ticlrz6PweqxkK/UFyW+PIgV8bmoC3pGkxO8dXRplJTSp0qky3+iRJyvp8fus3xv9TyRv5WXbBDAcjfPykT7+3+5jI/eV+dysmFcxktyV+k4mzNbvm3XtdQtWkdup4onkW0eSU+Wv0QShqFWal5o7NTgytypOZzBCdNRoqkugobqU5toAt5zfzKUr6jl/SR1lJVqFP1UiMtL5UV8+e/5fUKeVt+3TRD526RJeae/ne88cJJqw2ov6ch9rm6o5p6malQ2V+L0uu5rkZGmtSyBpIBiO22XBJ+d3DgzH6R2KcnwgzLH+MJ3ByCnzNRfVBbhgaR3nLqzm3IXWefJt7qpS+S6d/70bgSOjbrcB50/0HGNMXET6gTr7/u1jXts47WhHcbuE69c20Dcco28oSncwyv7OIH1D1peYmb73mfMquG5tA+cutObCLZ9TXrS91VPl97pHGt7JxO2VySaabG849Wvr6NzJZd9IlSgKJ0sWNclS5GnbBPDZ61by2etWEozEeeP4IG90DPJaxyCvdwzyxKvH6QlFM3WqET63iwq/x75Y87yaawNU+L3Ul/tYWBtgUV2A5toADdWlWpanVHblbfs0kQVVpTzy8QuJxq2O7V1H+tjV1s+uI308+fqJKS+cU+p1j5QIz6v0c+b8CuZXlbKgys/8Sj/zq/w0VJVSFZj5SLlSs106Cd1435zH/llP9Jx0XouI3AbcBtDc3JxGSFap30RLEscTSfqHY9aSyPGTyySPvu0S3lKulipf87pdLKkvo9SnvUMz5XG7NAlW2ZT1tgmm1z6llJd4WN9cw/rmt664F0uNtI2z0Mxki3W4RUZGn1OlsydH9vRvTak8kpffndLh87hY21TN2qZqPmTfNxiOse9EkLhd2TR2qgBAhd9jz/O05nLqKJtSuZNOQtcGjJ5Z3gQcneA5bSLiAaqAnjRfizHmHuAegI0bN864Ts/jdlGnpS9KFbust02Q+fYJGJlDlYk5XEqpvFRw350mU+H3sq5Zt4JQKl+l06W7A1ghIktExIc1UXfLmOdsATbbxzcCvzXWLqRbgJtFpERElgArgOczE7pSapbTtkkpla+0fVJK5cxpR+jsuu47gMexlt693xjTIiJ3ATuNMVuA7wI/sCfu9mA1XNjPewRrEnAcuL1QVrhUSuU3bZuUUvlK2yelVC6Jmeos1yzbuHGj2blzp9NhKKUyTEReMMZsdDqOmdD2Sanio22TUiofTaVt0ln0SimllFJKKVWgNKFTSimllFJKqQKVdyWXItIJHJrCS+qBriyFM1Uay/g0llPlSxyQu1gWGWPm5OA8WTPF9mk2/hunQ2MZn8YyvlzEom2TczSW8Wks45ttsaTdNuVdQjdVIrIzX2rfNZbxaSz5GwfkVyzFJJ8+V41lfBrL+DSW4pZPn6nGMj6NZXway8S05FIppZRSSimlCpQmdEoppZRSSilVoIohobvH6QBG0VjGp7GcKl/igPyKpZjk0+eqsYxPYxmfxlLc8ukz1VjGp7GMT2OZQMHPoVNKKaWUUkqp2aoYRuiUUkoppZRSalYq2IRORK4WkddFZJ+IfNrhWA6KyCsi8rKI7Mzxue8XkRMismfUfbUi8oSIvGlf1zgYy+dEpN3+bF4WkWtzFMtCEXlSRF4VkRYR+YR9f84/m0liyflnIyJ+EXleRHbZsXzevn+JiDxnfy4Pi4gv27EUM22fRs6t7dOpcWjbNH4s2jblgLZNI+fWtmn8WLR9Gj+W/G+fjDEFdwHcwH5gKeADdgErHYznIFDv0LkvA9YDe0bddzfwafv408CXHYzlc8BfO/C5LADW28cVwBvASic+m0liyflnAwhQbh97geeAC4BHgJvt+/8D+PNc/5sVy0Xbp7ecW9unU+PQtmn8WLRtyv5nrG3TyXNr2zR+LNo+jR9L3rdPhTpCtwnYZ4xpNcZEgYeAGxyOyRHGmKeAnjF33wA8YB8/ALzbwVgcYYw5Zox50T4eBF4FGnHgs5kklpwzlqB902tfDPB24Kf2/Tn7nSlS2j7ZtH0aNw5tm8aPRdum7NO2yaZt0/i0fZowlrxvnwo1oWsEjoy63YZD/8g2A/xKRF4QkdscjCNlnjHmGFh/EMBch+O5Q0R222UFOSlhGE1EFgPrsHpUHP1sxsQCDnw2IuIWkZeBE8ATWD22fcaYuP0Up/+eCp22T5PT9smmbdMpMWjblF3aNk1O26ZRtH06JYa8bp8KNaGTce5zcrnOi40x64FrgNtF5DIHY8k33waWAecCx4B/y+XJRaQc+BnwSWPMQC7PnUYsjnw2xpiEMeZcoAmrx/bs8Z6Wi1iKlLZPhcOx9knbplNp25R12jYVDv3uNHEs2j6No1ATujZg4ajbTcBRh2LBGHPUvj4BPIr1D+2k4yKyAMC+PuFUIMaY4/YfQRK4lxx+NiLixWoEfmSM+bl9tyOfzXixOPnZ2OfvA36HVQdeLSIe+yFH/56KgLZPk5v17ZO2TZPTtilrtG2a3Kxvm0Dbp9PJ1/apUBO6HcAKe3UZH3AzsMWJQESkTEQqUsfAVcCeyV+VdVuAzfbxZuA/nQok1QDY3kOOPhsREeC7wKvGmK+Meijnn81EsTjx2YjIHBGpto9LgT/Eqkt/ErjRfpqjvzNFQNunyc3q9knbpglj0bYp+7Rtmtysbpvs82r7NH4s+d8+TXc1FacvwLVYK97sBz7rYBxLsVaK2gW05DoW4CdYQ84xrN63jwJ1wG+AN+3rWgdj+QHwCrAbq0FYkKNYLsEa+t4NvGxfrnXis5kklpx/NsBa4CX7nHuAO0f9Hj8P7AP+P6Akl7/HxXbR9mnk/No+nRqHtk3jx6JtUw4u2jaNnF/bpvFj0fZp/Fjyvn0SOyCllFJKKaWUUgWmUEsulVJKKaWUUmrW04ROKaWUUkoppQqUJnRKKaWUUkopVaA0oVNKKaWUUkqpAqUJnVJKKaWUUkoVKE3olFJKKaWUUqpAaUKnlFJKKaWUUgVKEzqllFJKKaWUKlD/PzvvGcrw9eVAAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1080x720 with 9 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, axes = plt.subplots(3, 3, figsize=(15, 10))\n",
    "for i,ax in enumerate(axes.flat):\n",
    "    ax.plot(attn[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAEsCAYAAAALni6GAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XucZFV57vHfMyPIHcRBVK6CCBJEkBFFBDWigiciKEZREu/ECAcVb5h41JAYFRM9iqgoEfCO3BTNKAhHEUFkBhhAQJRwEUTFweHOXPs5f+xdTE3T01U1vbp2Vc3z5VOfrr1r97vfnm7qrbXX2mvJNhERESXMaDqBiIgYHSkqERFRTIpKREQUk6ISERHFpKhEREQxKSoREVFMikpERBSTohIREcWkqERERDEpKtF3kl4h6beS7pF0r6T7JN3bdF4RMXXKNC3Rb5JuBF5m+/qmc4mIstJSiSb8KQUlYjSlpRJ9J+kzwOOB7wKLW/ttn9VYUhFRxKOaTiDWSBsBDwIvbttnIEUlYsilpRIREcWkTyX6TtKWks6WdKekP0k6U9KWTecVEVOXohJNOBk4B3gisAXw/XpfRAy5XP6KvpM03/ZunfZFxPBJSyWasEDSYZJm1o/DgLuaTioipi4tleg7SVsDnwP2ohr1dQnwDtu3NppYRExZikpERBST+1SibyS9z/Zxko6naqGsxPZRDaQVEQWlqEQ/taZmmddoFhExbVJUom9sf79++qDt09tfk/SqBlKKiMLSpxJ9J+kK28/otC8ihk9aKtE3kg4AXgpsIemzbS9tBCxrJquIKClFJfrpDqr+lAOBy9v23we8q5GMIqKoXP6KvpO0lu2lTecREeWlpRJN2FbSx4CdgXVaO21v11xKEVFCpmmJJpwMfIGqH+UFwFeBrzWaUUQUkaISTVjX9gVUl19vtf0R4K8bzikiCsjlr2jCIkkzgN9KOhL4PfC4hnOKiALSUokmvBNYDzgK2AP4O+D1jWYUfaHKdyU9telcYnpk9Fc0qm6xbGD73qZziekn6SXAfwGn2X530/lEeWmpRN9J+qakjSStD1wH3CDpvU3nFX3xZuAtwMsk5fL7CEpRiSbsXLdMDgLmAFtTXQKLESZpFvBXtn8EnA8c3HBKMQ1SVKIJa0lai6qofK++ETLXYWuSXiHpt5LukXSvpPskjcLlwb8HvlU/P5mq1RIjJkUlmnAicAuwPvAzSdsAo/CmWcpxwIG2N7a9ke0NbW/UdFIFvJGqmGB7LvAESVs1m1KUlo76GAiSHmU7k0oCki62vXfTeZQkaRPg1bZPbNv3ImCB7SubyyxKS1GJvpF0mO2vSzp6otdtf6rfOQ0iSZ8BHg98F1jc2m/7rMaSWgVJr7d9atN5xODI5a/op/Xrrxuu4hGVjYAHgRcDL6sff9NoRqv2jk4HSHqrpB3q55J0ct1XdLWk3ac/xeinDOmLvrF9oqSZwL22P910PoPK9hubzqEH6uKYdwCn1M8PBXYFngTsDnwW2GdaMotGpKhEX9leLulAIEVlHEnvs32cpOOZYDSc7aMaSKuTbq6fL2tb6uBvgK/avgs4X9Jx05daNCFFJZpwiaTPAacBD7R22r6iuZQGwvX113mNZtGbbloqY5KeACwEXgh8tO21daclq2hMiko04Tn112Pb9pk1fKZi29+vv54KIGnDatP3N5rY5C7u4pgPURXKmcA5tq8FkPQ84KZpzC0akNFfEQNG0i5U68tsStUS+DPw96034z7n8g6qe0vuA06i6gc5xvZ5PcZ5FLCh7YVt+9aneg8a5KIZPcrorzVAPeLmMEkfqre3lrRng/lsLOlTkubVj/+UtHFT+QygLwFH297G9tbAu4EvN5TLm+opdV4MbEZ1A+PHVyPOpsA7JZ0h6XRJ/0I1kWgKyohJURlQkq6ph1yOf1wj6eoew30e2Itq5A1UnzpPKJpwb75S5/C39eNe6jutA4D1bf+ktWH7p6wYjt1vrT6TlwIn276K7vpRVgSQ9gbm1ptfBb5eP/9l/VqMkFz+GlD11CWrZPvWHmJdYfsZkq60vXu97yrbT59qnqtD0nzbu3Xat6aSdDZwBSuWWD4MmG37oAZyORnYgmoI8NOp+kV+anuPHmJcCvzj+DvnJe0GnGj7WQVTjoalpTKg6mV2b20rHjvUz+8E/tJjuKX1/SEGkLQZMFYu2549JOm5rY360+pDDeYzaN5EdanpLOBsYBbVZacmvBk4Bnim7QeBtVcjl40mmorF9nxy0+vIyeivASfprcDhVNektwe2BL5INTSzW5+lenPaXNJHgUOADxZOtRdvA77a1o+ykKz8+LC6M/sogPrDwPpNLWJme0zSlsBrJQFc2Bql1gNJekx7J329c1PywXbk5Bc6+I4A9qaexdf2b+lxPXfb3wDeB/w7cAdwkO3TC+fZi3vrS2+7ArvWl+TuazCfgTJuEbNraXARM0kfp7oj/rr6cZSkj/UY5tPAeZKeJ2nD+vF84IfkJtiRk5bK4Ftse0n9KbE1NHN1OsLWo7oebpq/4exM4BnjPn2fQbVefdSLmEl6HdUiZu8HLgc+2UAuLwV2sz0GIOlU4ErgA90GsP0lSXcA/wr8FdXf4HXAv61GqycGXIrK4LtQ0j8B69ZThb8d6Ol/xHoo8auo3swFnCzpdNv/VjzbyfPYiepNZWNJr2h7aSNgnX7mMuDaFzH7nO2lkpocUbMJK/rxVmvot+0fAD8ollEMrBSVwXcMVWfpNcA/UH1yPanHGIcCu9teBA9f0rgC6GtRAXakmvtpE6qZd1vuA97a51wGWWsRs6tofhGzjwFXSvoJ1QeSfemhlQIg6Tu2/7Z+/gnb72977TzbLy6ZcDQrQ4rXAJJ+CBxq++56exPg67YbmU5d0l62f9HEuYdVk4uY1fN2PZOqqPzS9h97/P72oexX2H7GRK/FaEhLZZpIegrwBWBz27tI2pVqidieWgeSbmbiGWu36yHMYuBaST+uY70I+Lmkz9ax+j377eH1qLaV2H5Tn/MYSJI2pxpU8UTbB0jamerm1f9qIBdRjTTczvaxrdkYbF/WQ5jJPrnmU+2ISVGZPl8G3kt1KQPbV0v6Jr1fcprd9nwdqr6RTXuMcXb9aPlpj99fWvu19XWAg6lGpUXlFKoZBv653v4N1YzOfS8qVLMxjFFN9nks1aXKM6laLt1ar16MawZV3+DuVK0e0fygkSgsRWX6rGf7staorVrPly/qdSfa/V9JP6ea+bVbdwFzWiN4mmb7zPZtSd8Czm8onWLqGzp3sH1yfYPpBrZvXo1Qs2x/R9IHAGwvk7S8aLLde1ZrNoY6l4WS1u4xxh+A1lLRf2x73tqOEZKiMn0WSNqeFXexH0L1P1dPJD2jbXMGVcul17uQXwN8RtKZVPM3Xd/pG8blsG/9dIntS3s8dzd2ALbuIZ+N6iG3E7bYbPc648CUSfow1e9mR6pWxlpUc1ytztxWD0h6LCv+dp4N3FMo1V5NeTYG2y+YjsRiMKWoTJ8jqGab3UnS74GbgdetRpz/ZMV152VUo4Je1UsA24dJ2ohqFNjJ9fDUk4Fv2e7mpsM31jncA0y5qEi6jxU/k4E/Ud2c2a1vUo0iu5yVr8mr3u6lv6mUg6mmhb8CwPYdqtZDWR1HA+cA20m6mGrKlkOKZNm71mwMj5toNoaJ7pSfiKR1gafUE1K29m0NLLf9+/JpR1My+qswSUeP27UuVQvjAQDbn3rEN00e791Ub5St62gr/cJ6iSdpFtXkhO+kWmXwycBnbR/fRQ4Pn3L8673+THXMTalaKK37U2z7ZwXiYPvCXuNMlaTLbO/ZNnnn+sAvbO+6GrHWAY4EXkLVh/EL4PjWkPB+q+8veiHV3+AF7S3d8aO5JomxFvBrqhkUHqj3nQf8k+1hWukyOkhLpbzWp9MdqTozv0f1P+PfAT2/aVLdZd4e52V1nNu6DaBqTfg3Us0d9jVgT9t3SlqPqrhMWlSADeqvrZ/pnHq7lUtPJL2FauqPLYH5wLOp3jh7WvlxFXEuobd50Ur5jqQTgU3qkW1vpvf7iVq+SnVfyr/X24dS/d66aqHWhfZtwCLgpKnOG2b711QFYcLTdRljaT378quBr9StlM1SUEaQ7Tym4QGcR7XSXWt7Q+BHTcShGjm077h9n6i/vrCBn+kaqpbF/Hp7J+C0puKMi/n4KXzvi6imUvkksN8U4lzVzb5Jvv8nVAXpU1Rzh203lX+TDue6oodjdwIuqp9/EDhquvLKo7lHJpScPlsDS9q2lwDbNhRnBz/y0tIBALYv6HMuAIu84u7+R7v6JLxjg3Hazenl4HokXquf6EyqFsLbgLMl3SPpZklv7zGHK+vO+dY5nkV3a8G3PNb2P9k+mqp/5kJVi7u9WNJ3esylmPr307qHq9X6ihGTy1/T52vAZXWT31Qduaf2M46kf6SaK2w7rbxa5Ib09iY15VzGub2+q/+7wI8lLWT17lMpFaddT6sa2n5u/XXCTvl6FNclVPd7TH5i6Rqqf9e1gL+X9Lt6exuqCRi7dZ+kbW3fYvvc+lLTE6mWGLimhzjd6Onfi+pem5OAq91FB38Mn3TUT6N6OPA+9ebPPMFCRdMZR9V6JY+hmr/pmLaX7vNqDrst9TO1xXse1SSFP7K9pNPxfYjzdtsdC0CPMZ9gu+NwchVa7VPSjtXh/k2XKa42SZv28rdU9+P9AXil7aG/NykeKUUlIiKKSZ9KREQUk6LSB5IOT5zhiDNIuSROf+IMUi6jIEWlP0r9sSXO9McZpFwSpz9xBimXoZfRXxERQ2T//ff3ggULujr28ssvP9f2/tOc0krSUT9FanaZ10dYe+3Oq/IuX76cmTNnTnrM5ltu0THO/ffeywYbbbTK15cs6m4Q1oMP3M9662+wytcfuPf+ruIsXbqYtdZ69CpfX7TogY4xxsaWM2PG5P82Y2OdJ5u2zbgZqieIMxCTRkd/LbC92VQCzJ4923Pnzu3q2BkzZlxue3bnI8tJS2VASGWuRD7+8WXmUnzPxz825Ri/u77rmWQm9cvzy0zldcMNvawrtWr331/m9ooHH+xmLs9uDNTnmphcV8PCOxkb4MZAikpExBAxMMhXmFJUIiKGivEAt05TVCIiholh+ViKSkREFGDSp9I4SffbXvXwokce/3yqpXMvmb6sIiJWT/pUhs/zgfupZpeNiBgog1xURuKOeknvk3RU/fzTkv5f/fyFkr5eP/+opKskXSpp83rfyyT9UtKVks6XtLmkbanWw3iXpPmS9pn4rBER/WebsS4fTRiJokK1pG3rzX82sEG9JvZzgYuA9YFLbT+9Pvat9bE/B55te3fg28D7bN8CfBH4tO3dbF80/mSSDpc0T1KWQo2Ivut2FcYmjMrlr8uBPSRtCCwGrqAqLvsAR1GtUPiDtmNfVD/fEjhN0hOAtYGbuzmZ7S8BX4LBu6M+IkabgeW5/DW9bC8FbgHeSNUPchHwAmB74HpgqVeU7eWsKKbHA5+z/TTgH6jWO4+IGGiD3FIZiaJS+xnwnvrrRVT9IvM9+b/sxsDv6+evb9t/H9WSuxERAyd9Kv1xEfAE4Be2/wQsqvdN5iPA6ZIuAtqn/fw+cHA66iNi4HTZSkmfyhTZvgBYq237KW3PN2h7fgZwRv38e8D3Joj1G2DX6cw3ImJ1ZO6viIgoavkAL5uQohIRMVQyoWRERBRiwwDPJ5miMihKXSPdYP1NisR5y8unvgLpe+d+tkAmcN11FxeJc/fddxaJU27FxgF+Z4iBlj6ViIgoZpCLyigNKY6IGHmtqe9L3KciaX9JN0i6UdIxE7z+Bkl/rm+vmC/pLZ1ipqUSETFM7CKjvyTNBE6gmrbqdmCupHNsXzfu0NNsH9lt3LRUIiKGTKGbH/cEbrR9k+0lVJPqvnyquaWoREQMEdMaVNz5vw62AG5r27693jfeKyVdLekMSVt1CjpSRUXSJpLe3nQeERHTaczdPYBZrWU66sfhbWE0Qejxlej7wLa2dwXOB07tlNuo9alsArwd+HzTiURETJceRn8tsD17Fa/dDrS3PLYE7hh3nrvaNr8MfKLTCUeqpQJ8HNi+HqVwuqSHrw9K+oakA+vRDN+T9KN61MOH2445TNJl9fefWHdkRUQMlEJ9KnOBHSQ9SdLawGuAc9oPqNeaajmQaimRSY1aUTkG+B/buwGfo1pfBUkbA88B5tTH7Qm8DtgNeJWk2ZKeCrwa2Lv+/uX1MY+QlR8joimuR3918+gQZxlwJHAuVbH4ju1rJR0r6cD6sKMkXSvpKqoFD9/QKb9Ru/z1MNsXSjpB0uOAVwBn2l4mCeDHrWadpLOolh1eBuxBNawOYF1gwluws/JjRDSp1M2Ptuew4sN2a9+H2p5/APhALzFHtqjUvkbV2ngN8Ka2/eN/I6bqtDq1/keMiBhIrZsfB9WoXf4av2LjKcA7AWxf27b/RZI2lbQucBBwMXABcEjdsqF+fZu+ZB0R0YNCQ4qnxUi1VGzfJeliSb8Cfmj7vZKuB7477tCfU7Vingx80/Y8AEkfBM6TNANYChwB3Nq/nyAiorPMUtxHtl/bei5pPWAH4FvjDrtzomkHbJ8GnDa9GUZErD7bBWfKLm/ULn89TNJ+wK+B423f03Q+ERGllJpQcjqMXEulxfb5wNYT7D+Fqq8lImIoDfLU9yNbVCIiRlWKSnShzB/Jr2/4ZZE4T99l7ynH+M1v5hbIBO6+8+4icc456wtF4lTjOKbuoYfuKxJn6dIlReJkJcrh4AYvbXUjRSUiYsg0NVy4GykqERFDxMDyAR5TnKISETFk0qcSERHFpE8lIiLK6G5a+8aM7M2PUyXpKEnXS/pG07lERLSYYuupTIu0VFbt7cABtm9uOpGIiHa5/DXgJB3NiqnxTwJ2ArYDzpH0Fdufbiy5iIhxUlQGmKQ9qFaIfBbVmiq/BA4D9gdeYHvBBN9zOHB4P/OMiIDBX09ljS8qVKs+nm37AXh4Jch9JvuGrPwYEY0Z8I76FJWqdRIRMTQGuaWS0V/wM+AgSetJWh84GLio4ZwiIiaU0V8DzvYVkk4BLqt3nWT7SikNmIgYTMsHeJGuNb6oANj+FPCpcfu2bSabiIjJNLf+fDdSVCIihohdPQZVikpExJAZ5I76FJWIiCGTIcXRN2OFOvD+/Ofbphxj/q23FsgEnvyMJxeJs86cDYrEWb58aZE4a6316CJxli0rk88gv1HFCrn5MSIiyrGLfXicDikqERHDJi2ViIgoxVlOOCIiShnghkqmaYmIGCbVfSplpmmRtL+kGyTdKOmYSY47RJIlze4Us3hRkXSQpJ3btn/aTSKTxHuipDNW4/s2kfT2qcaJiBg0JYqKpJnACcABwM7Aoe3v3W3HbQgcRbUsSEfT0VI5iCrBKZP0KNt32D5kNb59E6rVGwGYQpyIiAFixpaPdfXoYE/gRts32V4CfBt4+QTH/StwHLCom+y6KiqSvivpcknX1gtUIen+ttcPkXSKpOcABwKflDRf0vb1Ia+SdJmk30jap/6edSSdLOkaSVdKekG9/w2STpf0feA8SdtK+lX92kl13PmS/izpw5I2kHSBpCvqWK1/lI8D29fHfnJcnMnOfZakH0n6raTjuvn3iYjolx4vf82SNK/t0b644BZA+w1pt9f7HiZpd2Ar2z/oNr9uO+rfZPsvktYF5ko6c+If1pdIOgf4ge0z6qQAHmV7T0kvBT4M7AccUX/P0yTtRFVAnlKH2gvYtT7ntm3x31LH3AY4FziFqnoebPteSbOAS+scjgF2sb1b/T0Px+lw7t2A3YHFwA2Sjre90p2AWfkxIprUw42qC2yvqvthoqnYHw4saQbwaeANveTWbVE5StLB9fOtgB16OQlwVv31cmDb+vlzgeMBbP9a0q1A6439x7b/MlEgSesApwNH2r5V0lrAv0vaFxijqrSbd8hnsnNfYPue+lzXAduwcjXPyo8R0awyw79up3o/b9kSuKNte0NgF+CndePg8cA5kg60PW9VQTsWFUnPp2pZ7GX7QUk/BdaBleZeXqdDmMX11+Vt55xswZIHJnnti8BZts+vt18HbAbsYXuppFu6yGeycy9ue96eb0TEQCg0pHgusIOkJwG/B14DvHbFOXwPMKu1Xb/3v2eyggLd9alsDCysC8pOwLPr/X+S9NS6iXRw2/H3UVW4Tn5GVRCoLz1tDdww2TdIOgLY0PbHx+V3Z11QXkDVsuiUR8/njogYCC7TUW97GXAkVVfC9cB3bF8r6VhJB65uet18Cv8R8DZJV1O98V5a7z8G+AHVpaFfAa3Z+r4NfFnSUcBko60+D3xR0jXAMuANthd3WHHxPcBSSfPr7S8C3wC+L2keMB/4NYDtuyRdXHfO/5Bq6NxUzh0R0bjWcsJFYtlzgDnj9n1oFcc+v5uYHYuK7cVU45gn8oj7PmxfzMpDip/f9toC6j4V24uYoAPI9ilUHfCt7Vuoruth+0mryGOvVeT+2nG7WnG6PfffrOJ8ERGNGeQZpdNfEBExZFJUIiKiDBsyoWRERJSSlkr0UZk/tnvu+fOUY+y3x3MKZAI77visInF22WWfInGeuNU2nQ/qwrobrlckzpyzTykSZ+HCPxWJs2TJQ1OOsXz5sgKZjCYDY2mpREREEU5LJSIiCsoiXRERUUh3a6U0JUUlImLIpKgMOEkfAe63/R9N5xIRMRmnTyUiIkry8sEtKmvsGvWS/rlem/l8YMd6326SLpV0taSzJT2m4TQjIh6h1Br102GNLCqS9qCa5nl34BXAM+uXvgq83/auwDVUC4pFRAyOLgtKU0VlTb38tQ9wtu0HAeqVItcHNrF9YX3MqVSLgT1CVn6MiCalT2UwrfZvJSs/RkRTSk59Px3WyMtfVIt0HSxpXUkbAi+jWm1yoaTWXB5/B1y4qgAREY0wePlYV48mrJEtFdtXSDqNalGvW4GL6pdeT7V413rATcAbG0oxImIVcvPjQLL9UeCjE7z07An2RUQMjAGuKWtuUYmIGFZpqURERBF2JpSMiIiC0lKJiIhCzNhYMyO7upGiEhOyp/5He/fddxbIBObPv6BInJkzy/y5b3brVkXizPn5D4vE+d1v/qdInD/88aYicW688Yopx3joofsKZDKiMqFkREQUlT6ViIgoobqjvuksVi1FJSJiyOTyV0RElGEz1tAULN1YU+f+6oqkS5rOISJivEx9P6RsP6fpHCIi2mWW4iEm6f6mc4iIWEmrp76bRweS9q9XwL1R0jETvP42SddImi/p55J27hQzRSUiYqiUWflR0kzgBOAAYGfg0AmKxjdtP832bsBxwKc6ZZeishokHS5pnqR5TecSEWsej3X36GBP4EbbN9leAnwbePlK57Hvbdtcny4WN0yfymrIyo8R0RhTapqWLYDb2rZvB541/iBJRwBHA2sDf90paFoqERFDpNVR3+Xlr1mtqyr14/C2UFpF+JV32CfY3h54P/DBTvmlpRIRMWR6GP21wPbsVbx2O9A+kd2WwB2TxPo28IVOJ0xLZRK2N2g6h4iIlRmPdffoYC6wg6QnSVobeA1wTvsBknZo2/xfwG87BU1LJSJimBSapdj2MklHAucCM4Gv2L5W0rHAPNvnAEdK2g9YCiwEXt8pbopKRMSwKXTzo+05wJxx+z7U9vwdvcZMUYmIGCIGxjL1fayJli9fXiTOokUPFIkjTTTYpXd33FFmUawTPvOtInHWWvvRReLsvufzisR59KPXm3KMX/3qogKZwOLFDxaJU25alAJxskZ9RESU09xkkd1IUYmIGDIpKhERUUyKSkREFGGDB3iRrhSViIghM8ANldxRPxlJt0ia1XQeERErlJn6frqkpRIRMWQGuU9lJFoqkraVdL2kL0u6VtJ5ktaVtL2kH0m6XNJFknaqj99M0pmS5taPvev9j62/90pJJzLxLJ4REc3xYK9RPxJFpbYDcILtvwLuBl5JtebJ/7a9B/Ae4PP1sZ8BPm37mfVxJ9X7Pwz83PbuVBOrbd3H/CMiOjKUmlByWozS5a+bbc+vn18ObAs8Bzi97U7q1q3H+wE7t+3fSNKGwL7AKwBs/7ekhROdqF6T4PCJXouImF7GZRbpmhajVFQWtz1fDmwO3F2vrTzeDGAv2w+176yLTMfynpUfI6IxhWYpni6jdPlrvHuBmyW9CkCVp9evnQcc2TpQUqvw/Ax4Xb3vAOAx/Us3IqI7dnePJoxyUYGqQLxZ0lXAtcDL6/1HAbMlXS3pOuBt9f5/AfaVdAXwYuB3/U44IqKT9KlMM9u3ALu0bf9H28v7T3D8AuDVE+y/i6qYtLyrXJYREVPXWqN+UI1EUYmIWGMMeJ9KikpExFAxYxn9FRERpWSRrpFX4sb7wf0jWX1lfqaxsTIrSJaaIGHcSPTVdubXvlAkznrrbVwkzguffEiROPu98uWdD+rgttuuL5AJLFz4xyJxli9fNjhxqk6VqceZJikqERFDZMBrSopKRMSwSUd9RESUYTOWRboiIqKUtFQiIqKIQb/5cdSnaVktknaSdImkayRdmNUfI2KQZD2V4XSY7acBl7BibrCIiIZ1OZtklhMeHLZ/3ba5DnBXU7lERKzE4MHtp09RmYykl1BNSLnXuP1ZpCsiGpNpWoaQpBnAfwEvsH13+2tZpCsimpKO+uH1ROAe279tOpGIiIe5XEe9pP0l3SDpRknHTPD60ZKuq9eeukDSNp1ipqis2kLg3U0nERGxsu4W6Oo06aSkmcAJwAHAzsChknYed9iVwGzbuwJnAMd1yi5FZdU2Bt7SdBIREY9QZvTXnsCNtm+yvQT4NitWx61P45/YfrDevBTYslPQ9Kmsgu07gDLTtkZEFOTuZwCfJWle2/aX6j5hgC2A29peux141iSx3gz8sNMJU1QiIoaI7V6Wg1hge/YqXptoLYgJq5Wkw4DZwPM6nTBFJSJiyBQa/XU7sFXb9pbAHeMPkrQf8M/A82wv7hQ0RSUiYsgUKipzgR0kPQn4PfAa4LXtB0jaHTgR2N/2nd0ETVGZIkk86lFrTznO0qUdPwDElJVaibLMjWcLFtxeJM7MmX8qEudHrC5AAAAFoklEQVTMb5xQJM7jNtt6yjE22OAxBTIpdz/HsmVLisT5y1/+UCROiZ/L9jJJRwLnAjOBr9i+VtKxwDzb5wCfBDYATpcE8DvbB04WN0UlImKIVPeglPlgY3sOMGfcvg+1Pd+v15gpKhERQ6ZUUZkOKSoREUNmkKdpSVGJiBgyKSoREVFIuT6V6ZCiEhExROy0VCIioqBBLiojN6GkpMMkXSZpvqQT65k4kXRoveb8ryR9ot43U9Ip9b5rJL2r2ewjIjoxHhvr6tGEkWqpSHoq8Gpgb9tLJX0eeJ2k84FPAHtQTWl/nqSDqCZT28L2LvX3b9LlebLyY0Q0xqRPpV9eSFU45tZ3f64L3Ak8E/ip7T8DSPoGsC/wr8B2ko4H/hs4r5uTtK/8OGPGjMFth0bESBrky1+jVlQEnGr7AyvtrFolj2B7oaSnAy8BjgD+FnjTtGcZEbGaBr2jftT6VC4ADpH0OABJm9bLX/4SeJ6kWXUfy6HAhZJmATNsnwn8H+AZTSUeEdGd7pYSbqrwjFRLxfZ1kj5I1WcyA1gKHGH7UkkfAH5C1ZqZY/t7dSvl5PpYgA9MHDkiYnD0sJ5K341UUQGwfRpw2gT7vwl8c9y+q0jrJCKGzCBf/hq5ohIRMdK6W3++MUNbVCQ9lqoPZbwX2r5rCnFPAPYet/sztk9e3ZgREaWYntao77uhLSp14dhtGuIeUTpmRERJmftrhNke6E6zGFzLli0tEqfU9fX777+7SJwV416aV2pF1cF6E29uZFc3UlQiIoZMqSWtp0OKSkTEEKn66VNUIiKiiFz+ioiIklJUIiKilAwpjoiIYnL5KyIiihj02xhSVCIihkxaKiMmKz9GRJNSVEZM+8qPkgb3txsRIylFJSIiCjHk5seIiCjBhrEBLiqDM/NbRER0pdRywpL2l3SDpBslHTPB6/tKukLSMkmHdJNbikpExFAx9lhXj8lImgmcABwA7AwcKmnncYf9DngD41bNnUwuf0VEDJlCHfV7AjfavglA0reBlwPXtZ3nlvq1rq+3paUSETFkCl3+2gK4rW379nrflKSlEhExRKqp77tuqcySNK9t+0v1LREAmij8lJIjRWXKJDFjxswpx1m+fFmBbGJyE/0/NPxK3bMwNlbmb3DRovunHKPU6pGl1h0ZpNUsqz6VrqdpWWB79ipeux3Yqm17S+COqWQGufwVETF0Cl3+mgvsIOlJktYGXgOcM9XcUlQiIoZMiaJiexlwJHAucD3wHdvXSjpW0oEAkp4p6XbgVcCJkq7tlFsuf0VEDJVyKz/angPMGbfvQ23P51JdFutaikpExBDJGvUREVFUJpSMiIhCjMfSUomIiEKyRn1ERBSTPpURk5UfI6IpPd5R33cpKquhfeXHGTNmDO5vNyJGULkhxdMhRSUiYsiMpaM+IiJKSZ9KRESUUXWqNJ3FKqWoREQMEZMhxRERUVA66iMiopj0qURERCHO6K9RJ5VYUbDUqoSD2yyOwVZuBcmuVyVcpRmFVnoq9TNpgP6/ys2PERFRVIpKREQUYkifSkRElJIhxRERUUwuf0VERBG2iwyGmC4pKhERQyYtlYiIKCZFJSIiiklRGTFZ+TEiGpWiMlqy8mNENMU2Y05HfUREFJLLXxERUUyKSkREFOIUlYiIKCfrqURERBGZ+j4iIgpyWioREVFOisoIs71gyZJFt3Y4bBawoMDpEmdKcbq6ZNC3n6nLSQH7FqdLHeMsXbp4oPIZsFy2KXCeXP4aZbY363SMpHm2Z0/1XIkz/XEGKZfE6U+cQcqlS+fantXlsSWKZU9SVCIihojt/ZvOYTIzmk4gIiJGR4pKf3wpcYYmziDlkjj9iTNIuQw9DXKHT0REDJe0VCIiopgUlYiIKCZFJSIiiklRiYiIYlJUIiKimBSViIgo5v8DIHDhsGjTRc8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.ticker as ticker\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "cax = ax.matshow(attn[:12,:15], cmap='bone')\n",
    "fig.colorbar(cax)\n",
    "\n",
    "# axes\n",
    "ax.set_xticklabels([''] + input_sequence.split(' ') + ['<EOS>'], rotation=90)\n",
    "ax.set_yticklabels([''] + output_words)\n",
    "# Show label at every tick\n",
    "ax.xaxis.set_major_locator(ticker.MultipleLocator(1))\n",
    "ax.yaxis.set_major_locator(ticker.MultipleLocator(1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot commands for attention from [here](https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#sphx-glr-intermediate-seq2seq-translation-tutorial-py)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(28, 33)"
      ]
     },
     "execution_count": 219,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "attn.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:deeplearning]",
   "language": "python",
   "name": "conda-env-deeplearning-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}