7. Recurrent Neural Networks II.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning to add using ``seq2seq`` architecture\n",
    "\n",
    "Source: https://github.com/keras-team/keras/blob/master/examples/addition_rnn.py \n",
    "\n",
    "The objective of this code is to train a RNN for adding a sequence of integers. In this case we must process all the sequence in order to produce a result. We will use a ``keras`` implementation of the ``seq2seq`` architecure, originally proposed by Sutskever, Vinyals and Le in 2014. \n",
    "\n",
    "The architecture diagram from their paper is:\n",
    "\n",
    "<center>\n",
    "<img src=\"images/seq2seq.png\" alt=\"\" style=\"width: 800px;\"/> \n",
    "Source: https://arxiv.org/abs/1409.3215\n",
    "<center>\n",
    "\n",
    "Rectangles are recurrent layers. Encoder receives ``[A, B, C]`` sequence as inputs. We don't care about encoder outputs, only about the hidden state it accumulates while reading the sequence. \n",
    "\n",
    "After input sequence ends, encoder passes its final state to decoder, which receives ``[<EOS>, W, X, Y, Z]`` and is trained to output ``[W, X, Y, Z, <EOS>]``. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sequence-to-sequence, or *Seq2Seq*, is a relatively new paradigm made up of two recurrent networks: an *encoder* which takes the model's input sequence as input and encodes it into a fixed-size *context vector*, and a *decoder*, which uses the context vector from above as a *seed* from which to generate an output sequence.\n",
    "\n",
    "For this reason, Seq2Seq models are often referred to as *encoder-decoder* models.\n",
    "\n",
    "The encoder network's job is to read the input sequence and generate a fixed-dimensional context vector $C$ for the sequence. \n",
    "\n",
    "To do so, the encoder will use a recurrent neural network cell -- usually an LSTM -- to read the input tokens one at a time. The final hidden state of the cell will then become $C$. \n",
    "\n",
    "However, because it's so difficult to compress an arbitrary-length sequence into a single fixed-size vector, the encoder will usually consist of *stacked* LSTMs. The *final* layer's LSTM hidden state will be used as $C$.\n",
    "\n",
    "The decoder is also an LSTM network, but its usage is a little more complex than the encoder network. Essentially, we'd like to use it as a language model that's \"aware\" of the words that it's generated so far *and* of the input. \n",
    "\n",
    "To that end, we'll keep the \"stacked\" LSTM architecture from the encoder, but we'll initialize the hidden state of our first layer with the context vector from above; the decoder will literally use the context of the input to generate an output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When both input sequences and output sequences have the same length, you can implement such models simply with a Keras LSTM or GRU layer (or stack thereof).\n",
    "\n",
    "One caveat of this approach is that it assumes that it is possible to generate ``target[...t]`` given ``input[...t]``. That works in some cases (e.g. adding strings of digits) but does not work for most use cases. In the general case, information about the entire input sequence is necessary in order to start generating the target sequence.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating data...\n",
      "Total addition questions: 50000\n",
      "66+57   123 \n"
     ]
    }
   ],
   "source": [
    "'''An implementation of sequence to sequence learning for performing addition\n",
    "Input: \"535+61 \"\n",
    "Output:\"596 \"\n",
    "Padding is handled by using a repeated sentinel character (space)\n",
    "'''\n",
    "\n",
    "from __future__ import print_function\n",
    "from keras.models import Sequential\n",
    "from keras import layers\n",
    "import numpy as np\n",
    "from six.moves import range\n",
    "\n",
    "\n",
    "class CharacterTable(object):\n",
    "    \"\"\"Given a set of characters:\n",
    "    + Encode them to a one hot integer representation\n",
    "    + Decode the one hot integer representation to their character output\n",
    "    + Decode a vector of probabilities to their character output\n",
    "    \"\"\"\n",
    "    def __init__(self, chars):\n",
    "        \"\"\"Initialize character table.\n",
    "        # Arguments\n",
    "            chars: Characters that can appear in the input.\n",
    "        \"\"\"\n",
    "        self.chars = sorted(set(chars))\n",
    "        self.char_indices = dict((c, i) for i, c in enumerate(self.chars))\n",
    "        self.indices_char = dict((i, c) for i, c in enumerate(self.chars))\n",
    "\n",
    "    def encode(self, C, num_rows):\n",
    "        \"\"\"One hot encode given string C.\n",
    "        # Arguments\n",
    "            num_rows: Number of rows in the returned one hot encoding. This is\n",
    "                used to keep the # of rows for each data the same.\n",
    "        \"\"\"\n",
    "        x = np.zeros((num_rows, len(self.chars)))\n",
    "        for i, c in enumerate(C):\n",
    "            x[i, self.char_indices[c]] = 1\n",
    "        return x\n",
    "\n",
    "    def decode(self, x, calc_argmax=True):\n",
    "        if calc_argmax:\n",
    "            x = x.argmax(axis=-1)\n",
    "        return ''.join(self.indices_char[x] for x in x)\n",
    "\n",
    "\n",
    "class colors:\n",
    "    ok = '\\033[92m'\n",
    "    fail = '\\033[91m'\n",
    "    close = '\\033[0m'\n",
    "\n",
    "# Parameters for the model and dataset.\n",
    "TRAINING_SIZE = 50000\n",
    "DIGITS = 3\n",
    "\n",
    "# Maximum length of input is 'int + int' (e.g., '345+678'). Maximum length of\n",
    "# int is DIGITS.\n",
    "MAXLEN = DIGITS + 1 + DIGITS\n",
    "\n",
    "# All the numbers, plus sign and space for padding.\n",
    "chars = '0123456789+ '\n",
    "ctable = CharacterTable(chars)\n",
    "\n",
    "questions = []\n",
    "expected = []\n",
    "seen = set()\n",
    "print('Generating data...')\n",
    "while len(questions) < TRAINING_SIZE:\n",
    "    f = lambda: int(''.join(np.random.choice(list('0123456789'))\n",
    "                    for i in range(np.random.randint(1, DIGITS + 1))))\n",
    "    a, b = f(), f()\n",
    "    # Skip any addition questions we've already seen\n",
    "    # Also skip any such that x+Y == Y+x (hence the sorting).\n",
    "    key = tuple(sorted((a, b)))\n",
    "    if key in seen:\n",
    "        continue\n",
    "    seen.add(key)\n",
    "    # Pad the data with spaces such that it is always MAXLEN.\n",
    "    q = '{}+{}'.format(a, b)\n",
    "    query = q + ' ' * (MAXLEN - len(q))\n",
    "    ans = str(a + b)\n",
    "    # Answers can be of maximum size DIGITS + 1.\n",
    "    ans += ' ' * (DIGITS + 1 - len(ans))\n",
    "    questions.append(query)\n",
    "    expected.append(ans)\n",
    "print('Total addition questions:', len(questions))\n",
    "print(questions[3], expected[3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vectorization...\n",
      "Training Data:\n",
      "(45000, 7, 12)\n",
      "(45000, 4, 12)\n",
      "Validation Data:\n",
      "(5000, 7, 12)\n",
      "(5000, 4, 12)\n",
      "Build model...\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "lstm_1 (LSTM)                (None, 128)               72192     \n",
      "_________________________________________________________________\n",
      "repeat_vector_1 (RepeatVecto (None, 4, 128)            0         \n",
      "_________________________________________________________________\n",
      "lstm_2 (LSTM)                (None, 4, 128)            131584    \n",
      "_________________________________________________________________\n",
      "time_distributed_1 (TimeDist (None, 4, 12)             1548      \n",
      "_________________________________________________________________\n",
      "activation_1 (Activation)    (None, 4, 12)             0         \n",
      "=================================================================\n",
      "Total params: 205,324\n",
      "Trainable params: 205,324\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 1\n",
      "Train on 45000 samples, validate on 5000 samples\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 24s 537us/step - loss: 1.8966 - acc: 0.3204 - val_loss: 1.7978 - val_acc: 0.3412\n",
      "Q 972+672 T 1644 \u001b[91m☒\u001b[0m 1079\n",
      "Q 818+595 T 1413 \u001b[91m☒\u001b[0m 107 \n",
      "Q 337+911 T 1248 \u001b[91m☒\u001b[0m 107 \n",
      "Q 62+208  T 270  \u001b[91m☒\u001b[0m 177 \n",
      "Q 72+45   T 117  \u001b[91m☒\u001b[0m 77  \n",
      "Q 594+3   T 597  \u001b[91m☒\u001b[0m 177 \n",
      "Q 587+8   T 595  \u001b[91m☒\u001b[0m 107 \n",
      "Q 85+77   T 162  \u001b[91m☒\u001b[0m 177 \n",
      "Q 200+0   T 200  \u001b[91m☒\u001b[0m 73  \n",
      "Q 0+703   T 703  \u001b[91m☒\u001b[0m 77  \n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 2\n",
      "Train on 45000 samples, validate on 5000 samples\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 25s 556us/step - loss: 1.7571 - acc: 0.3514 - val_loss: 1.6893 - val_acc: 0.3728\n",
      "Q 46+969  T 1015 \u001b[91m☒\u001b[0m 105 \n",
      "Q 539+511 T 1050 \u001b[91m☒\u001b[0m 112 \n",
      "Q 52+52   T 104  \u001b[91m☒\u001b[0m 56  \n",
      "Q 796+163 T 959  \u001b[91m☒\u001b[0m 1516\n",
      "Q 57+893  T 950  \u001b[91m☒\u001b[0m 102 \n",
      "Q 21+879  T 900  \u001b[91m☒\u001b[0m 902 \n",
      "Q 81+891  T 972  \u001b[91m☒\u001b[0m 102 \n",
      "Q 493+77  T 570  \u001b[91m☒\u001b[0m 902 \n",
      "Q 442+2   T 444  \u001b[91m☒\u001b[0m 35  \n",
      "Q 394+0   T 394  \u001b[91m☒\u001b[0m 100 \n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 3\n",
      "Train on 45000 samples, validate on 5000 samples\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 25s 561us/step - loss: 1.6072 - acc: 0.3978 - val_loss: 1.5368 - val_acc: 0.4244\n",
      "Q 15+88   T 103  \u001b[91m☒\u001b[0m 12  \n",
      "Q 580+98  T 678  \u001b[91m☒\u001b[0m 952 \n",
      "Q 178+826 T 1004 \u001b[91m☒\u001b[0m 100 \n",
      "Q 515+324 T 839  \u001b[91m☒\u001b[0m 609 \n",
      "Q 518+410 T 928  \u001b[91m☒\u001b[0m 700 \n",
      "Q 10+535  T 545  \u001b[91m☒\u001b[0m 252 \n",
      "Q 771+626 T 1397 \u001b[91m☒\u001b[0m 1577\n",
      "Q 96+66   T 162  \u001b[91m☒\u001b[0m 662 \n",
      "Q 273+696 T 969  \u001b[91m☒\u001b[0m 102 \n",
      "Q 406+79  T 485  \u001b[91m☒\u001b[0m 522 \n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 4\n",
      "Train on 45000 samples, validate on 5000 samples\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 25s 560us/step - loss: 1.4611 - acc: 0.4510 - val_loss: 1.3903 - val_acc: 0.4857\n",
      "Q 853+962 T 1815 \u001b[91m☒\u001b[0m 1754\n",
      "Q 99+60   T 159  \u001b[91m☒\u001b[0m 126 \n",
      "Q 137+72  T 209  \u001b[91m☒\u001b[0m 211 \n",
      "Q 69+993  T 1062 \u001b[91m☒\u001b[0m 1066\n",
      "Q 952+29  T 981  \u001b[91m☒\u001b[0m 991 \n",
      "Q 899+159 T 1058 \u001b[91m☒\u001b[0m 1003\n",
      "Q 284+34  T 318  \u001b[91m☒\u001b[0m 351 \n",
      "Q 324+580 T 904  \u001b[91m☒\u001b[0m 811 \n",
      "Q 486+38  T 524  \u001b[91m☒\u001b[0m 416 \n",
      "Q 99+458  T 557  \u001b[91m☒\u001b[0m 533 \n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 5\n",
      "Train on 45000 samples, validate on 5000 samples\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 25s 555us/step - loss: 1.3244 - acc: 0.5060 - val_loss: 1.2649 - val_acc: 0.5288\n",
      "Q 33+74   T 107  \u001b[91m☒\u001b[0m 114 \n",
      "Q 137+880 T 1017 \u001b[91m☒\u001b[0m 1046\n",
      "Q 550+68  T 618  \u001b[91m☒\u001b[0m 621 \n",
      "Q 848+1   T 849  \u001b[91m☒\u001b[0m 844 \n",
      "Q 182+797 T 979  \u001b[91m☒\u001b[0m 904 \n",
      "Q 312+0   T 312  \u001b[91m☒\u001b[0m 211 \n",
      "Q 292+25  T 317  \u001b[91m☒\u001b[0m 214 \n",
      "Q 49+779  T 828  \u001b[91m☒\u001b[0m 832 \n",
      "Q 104+297 T 401  \u001b[91m☒\u001b[0m 114 \n",
      "Q 75+91   T 166  \u001b[91m☒\u001b[0m 176 \n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 6\n",
      "Train on 45000 samples, validate on 5000 samples\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 28s 622us/step - loss: 1.2113 - acc: 0.5481 - val_loss: 1.1692 - val_acc: 0.5606\n",
      "Q 659+38  T 697  \u001b[91m☒\u001b[0m 662 \n",
      "Q 92+146  T 238  \u001b[91m☒\u001b[0m 256 \n",
      "Q 6+415   T 421  \u001b[91m☒\u001b[0m 410 \n",
      "Q 288+302 T 590  \u001b[91m☒\u001b[0m 522 \n",
      "Q 99+842  T 941  \u001b[91m☒\u001b[0m 900 \n",
      "Q 307+73  T 380  \u001b[91m☒\u001b[0m 302 \n",
      "Q 1+952   T 953  \u001b[91m☒\u001b[0m 945 \n",
      "Q 8+502   T 510  \u001b[91m☒\u001b[0m 520 \n",
      "Q 690+766 T 1456 \u001b[91m☒\u001b[0m 1585\n",
      "Q 7+459   T 466  \u001b[91m☒\u001b[0m 464 \n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 7\n",
      "Train on 45000 samples, validate on 5000 samples\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 27s 590us/step - loss: 1.1184 - acc: 0.5810 - val_loss: 1.0802 - val_acc: 0.5954\n",
      "Q 867+66  T 933  \u001b[91m☒\u001b[0m 943 \n",
      "Q 442+360 T 802  \u001b[91m☒\u001b[0m 896 \n",
      "Q 471+168 T 639  \u001b[91m☒\u001b[0m 629 \n",
      "Q 757+234 T 991  \u001b[91m☒\u001b[0m 1009\n",
      "Q 82+873  T 955  \u001b[91m☒\u001b[0m 960 \n",
      "Q 848+7   T 855  \u001b[91m☒\u001b[0m 853 \n",
      "Q 932+60  T 992  \u001b[91m☒\u001b[0m 994 \n",
      "Q 335+806 T 1141 \u001b[91m☒\u001b[0m 1100\n",
      "Q 645+67  T 712  \u001b[91m☒\u001b[0m 720 \n",
      "Q 505+802 T 1307 \u001b[91m☒\u001b[0m 1333\n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 8\n",
      "Train on 45000 samples, validate on 5000 samples\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 28s 621us/step - loss: 1.0237 - acc: 0.6167 - val_loss: 0.9836 - val_acc: 0.6298\n",
      "Q 98+362  T 460  \u001b[91m☒\u001b[0m 461 \n",
      "Q 34+558  T 592  \u001b[91m☒\u001b[0m 591 \n",
      "Q 634+780 T 1414 \u001b[91m☒\u001b[0m 1400\n",
      "Q 9+246   T 255  \u001b[92m☑\u001b[0m 255 \n",
      "Q 30+162  T 192  \u001b[91m☒\u001b[0m 292 \n",
      "Q 276+47  T 323  \u001b[91m☒\u001b[0m 324 \n",
      "Q 401+39  T 440  \u001b[91m☒\u001b[0m 451 \n",
      "Q 34+817  T 851  \u001b[91m☒\u001b[0m 850 \n",
      "Q 89+41   T 130  \u001b[91m☒\u001b[0m 129 \n",
      "Q 157+65  T 222  \u001b[91m☒\u001b[0m 221 \n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 9\n",
      "Train on 45000 samples, validate on 5000 samples\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 27s 602us/step - loss: 0.8806 - acc: 0.6711 - val_loss: 0.8137 - val_acc: 0.6879\n",
      "Q 46+950  T 996  \u001b[91m☒\u001b[0m 907 \n",
      "Q 712+8   T 720  \u001b[91m☒\u001b[0m 729 \n",
      "Q 259+804 T 1063 \u001b[91m☒\u001b[0m 1036\n",
      "Q 74+279  T 353  \u001b[91m☒\u001b[0m 352 \n",
      "Q 467+593 T 1060 \u001b[91m☒\u001b[0m 1052\n",
      "Q 62+699  T 761  \u001b[91m☒\u001b[0m 751 \n",
      "Q 905+226 T 1131 \u001b[91m☒\u001b[0m 1009\n",
      "Q 435+726 T 1161 \u001b[91m☒\u001b[0m 1189\n",
      "Q 701+894 T 1595 \u001b[91m☒\u001b[0m 1596\n",
      "Q 296+25  T 321  \u001b[91m☒\u001b[0m 322 \n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 10\n",
      "Train on 45000 samples, validate on 5000 samples\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 27s 597us/step - loss: 0.6983 - acc: 0.7449 - val_loss: 0.6180 - val_acc: 0.7842\n",
      "Q 9+250   T 259  \u001b[91m☒\u001b[0m 258 \n",
      "Q 524+41  T 565  \u001b[91m☒\u001b[0m 555 \n",
      "Q 270+97  T 367  \u001b[92m☑\u001b[0m 367 \n",
      "Q 273+657 T 930  \u001b[91m☒\u001b[0m 900 \n",
      "Q 86+609  T 695  \u001b[91m☒\u001b[0m 785 \n",
      "Q 740+251 T 991  \u001b[91m☒\u001b[0m 903 \n",
      "Q 9+793   T 802  \u001b[91m☒\u001b[0m 801 \n",
      "Q 329+3   T 332  \u001b[92m☑\u001b[0m 332 \n",
      "Q 46+26   T 72   \u001b[92m☑\u001b[0m 72  \n",
      "Q 354+2   T 356  \u001b[92m☑\u001b[0m 356 \n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 11\n",
      "Train on 45000 samples, validate on 5000 samples\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 27s 594us/step - loss: 0.5361 - acc: 0.8251 - val_loss: 0.4905 - val_acc: 0.8387\n",
      "Q 973+70  T 1043 \u001b[91m☒\u001b[0m 1042\n",
      "Q 261+95  T 356  \u001b[92m☑\u001b[0m 356 \n",
      "Q 67+107  T 174  \u001b[91m☒\u001b[0m 164 \n",
      "Q 922+7   T 929  \u001b[92m☑\u001b[0m 929 \n",
      "Q 763+19  T 782  \u001b[91m☒\u001b[0m 792 \n",
      "Q 577+6   T 583  \u001b[92m☑\u001b[0m 583 \n",
      "Q 19+362  T 381  \u001b[92m☑\u001b[0m 381 \n",
      "Q 322+506 T 828  \u001b[91m☒\u001b[0m 847 \n",
      "Q 119+944 T 1063 \u001b[91m☒\u001b[0m 1033\n",
      "Q 304+654 T 958  \u001b[91m☒\u001b[0m 968 \n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 12\n",
      "Train on 45000 samples, validate on 5000 samples\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 28s 629us/step - loss: 0.4217 - acc: 0.8727 - val_loss: 0.3806 - val_acc: 0.8855\n",
      "Q 27+864  T 891  \u001b[92m☑\u001b[0m 891 \n",
      "Q 253+26  T 279  \u001b[92m☑\u001b[0m 279 \n",
      "Q 27+815  T 842  \u001b[92m☑\u001b[0m 842 \n",
      "Q 18+41   T 59   \u001b[92m☑\u001b[0m 59  \n",
      "Q 841+377 T 1218 \u001b[92m☑\u001b[0m 1218\n",
      "Q 199+165 T 364  \u001b[91m☒\u001b[0m 454 \n",
      "Q 319+24  T 343  \u001b[92m☑\u001b[0m 343 \n",
      "Q 632+9   T 641  \u001b[92m☑\u001b[0m 641 \n",
      "Q 483+631 T 1114 \u001b[92m☑\u001b[0m 1114\n",
      "Q 949+44  T 993  \u001b[91m☒\u001b[0m 994 \n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 13\n",
      "Train on 45000 samples, validate on 5000 samples\n",
      "Epoch 1/1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "24064/45000 [===============>..............] - ETA: 12s - loss: 0.3447 - acc: 0.9080"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-16-476b6bad3208>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     69\u001b[0m               \u001b[0mbatch_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mBATCH_SIZE\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     70\u001b[0m               \u001b[0mepochs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 71\u001b[0;31m               validation_data=(x_val, y_val))\n\u001b[0m\u001b[1;32m     72\u001b[0m     \u001b[0;31m# Select 10 samples from the validation set at random so we can visualize\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     73\u001b[0m     \u001b[0;31m# errors.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/models.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, **kwargs)\u001b[0m\n\u001b[1;32m    963\u001b[0m                               \u001b[0minitial_epoch\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0minitial_epoch\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    964\u001b[0m                               \u001b[0msteps_per_epoch\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msteps_per_epoch\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 965\u001b[0;31m                               validation_steps=validation_steps)\n\u001b[0m\u001b[1;32m    966\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    967\u001b[0m     def evaluate(self, x=None, y=None,\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/engine/training.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, **kwargs)\u001b[0m\n\u001b[1;32m   1667\u001b[0m                               \u001b[0minitial_epoch\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0minitial_epoch\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1668\u001b[0m                               \u001b[0msteps_per_epoch\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msteps_per_epoch\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1669\u001b[0;31m                               validation_steps=validation_steps)\n\u001b[0m\u001b[1;32m   1670\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1671\u001b[0m     def evaluate(self, x=None, y=None,\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/engine/training.py\u001b[0m in \u001b[0;36m_fit_loop\u001b[0;34m(self, f, ins, out_labels, batch_size, epochs, verbose, callbacks, val_f, val_ins, shuffle, callback_metrics, initial_epoch, steps_per_epoch, validation_steps)\u001b[0m\n\u001b[1;32m   1204\u001b[0m                         \u001b[0mins_batch\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mins_batch\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtoarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1205\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1206\u001b[0;31m                     \u001b[0mouts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mins_batch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1207\u001b[0m                     \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mouts\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1208\u001b[0m                         \u001b[0mouts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mouts\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/backend/tensorflow_backend.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, inputs)\u001b[0m\n\u001b[1;32m   2473\u001b[0m         \u001b[0msession\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_session\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2474\u001b[0m         updated = session.run(fetches=fetches, feed_dict=feed_dict,\n\u001b[0;32m-> 2475\u001b[0;31m                               **self.session_kwargs)\n\u001b[0m\u001b[1;32m   2476\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mupdated\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutputs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2477\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    893\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    894\u001b[0m       result = self._run(None, fetches, feed_dict, options_ptr,\n\u001b[0;32m--> 895\u001b[0;31m                          run_metadata_ptr)\n\u001b[0m\u001b[1;32m    896\u001b[0m       \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    897\u001b[0m         \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, handle, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1126\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mfinal_fetches\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mfinal_targets\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mhandle\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mfeed_dict_tensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1127\u001b[0m       results = self._do_run(handle, final_targets, final_fetches,\n\u001b[0;32m-> 1128\u001b[0;31m                              feed_dict_tensor, options, run_metadata)\n\u001b[0m\u001b[1;32m   1129\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1130\u001b[0m       \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_run\u001b[0;34m(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1342\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhandle\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1343\u001b[0m       return self._do_call(_run_fn, self._session, feeds, fetches, targets,\n\u001b[0;32m-> 1344\u001b[0;31m                            options, run_metadata)\n\u001b[0m\u001b[1;32m   1345\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1346\u001b[0m       \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_prun_fn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_session\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeeds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetches\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_call\u001b[0;34m(self, fn, *args)\u001b[0m\n\u001b[1;32m   1348\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1349\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1350\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1351\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mOpError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1352\u001b[0m       \u001b[0mmessage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcompat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_text\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run_fn\u001b[0;34m(session, feed_dict, fetch_list, target_list, options, run_metadata)\u001b[0m\n\u001b[1;32m   1327\u001b[0m           return tf_session.TF_Run(session, options,\n\u001b[1;32m   1328\u001b[0m                                    \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtarget_list\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1329\u001b[0;31m                                    status, run_metadata)\n\u001b[0m\u001b[1;32m   1330\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1331\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_prun_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msession\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "print('Vectorization...')\n",
    "x = np.zeros((len(questions), MAXLEN, len(chars)), dtype=np.bool)\n",
    "y = np.zeros((len(questions), DIGITS + 1, len(chars)), dtype=np.bool)\n",
    "for i, sentence in enumerate(questions):\n",
    "    x[i] = ctable.encode(sentence, MAXLEN)\n",
    "for i, sentence in enumerate(expected):\n",
    "    y[i] = ctable.encode(sentence, DIGITS + 1)\n",
    "\n",
    "# Shuffle (x, y) \n",
    "indices = np.arange(len(y))\n",
    "np.random.shuffle(indices)\n",
    "x = x[indices]\n",
    "y = y[indices]\n",
    "\n",
    "# Explicitly set apart 10% for validation data that we never train over.\n",
    "split_at = len(x) - len(x) // 10\n",
    "(x_train, x_val) = x[:split_at], x[split_at:]\n",
    "(y_train, y_val) = y[:split_at], y[split_at:]\n",
    "\n",
    "print('Training Data:')\n",
    "print(x_train.shape)\n",
    "print(y_train.shape)\n",
    "\n",
    "print('Validation Data:')\n",
    "print(x_val.shape)\n",
    "print(y_val.shape)\n",
    "\n",
    "# Try replacing GRU, or SimpleRNN.\n",
    "RNN = layers.LSTM\n",
    "HIDDEN_SIZE = 128\n",
    "BATCH_SIZE = 128\n",
    "LAYERS = 1\n",
    "\n",
    "print('Build model...')\n",
    "model = Sequential()\n",
    "\n",
    "# \"Encode\" the input sequence using an RNN, producing an output of HIDDEN_SIZE.\n",
    "# Note: In a situation where your input sequences have a variable length,\n",
    "# use input_shape=(None, num_feature).\n",
    "# Returns the last output in the output sequence\n",
    "model.add(RNN(HIDDEN_SIZE, input_shape=(MAXLEN, len(chars))))\n",
    "\n",
    "# As the decoder RNN's input, repeatedly provide with the last hidden state of\n",
    "# RNN for each time step. Repeat 'DIGITS + 1' times as that's the maximum\n",
    "# length of output, e.g., when DIGITS=3, max output is 999+999=1998.\n",
    "model.add(layers.RepeatVector(DIGITS + 1))\n",
    "\n",
    "# The decoder RNN could be multiple layers stacked or a single layer.\n",
    "for _ in range(LAYERS):\n",
    "    # By setting return_sequences to True, return not only the last output but\n",
    "    # all the outputs so far in the form of (num_samples, timesteps,\n",
    "    # output_dim). This is necessary as TimeDistributed in the below expects\n",
    "    # the first dimension to be the timesteps.\n",
    "    model.add(RNN(HIDDEN_SIZE, return_sequences=True))\n",
    "\n",
    "# Apply a dense layer to the every temporal slice of an input. For each of step\n",
    "# of the output sequence, decide which character should be chosen.\n",
    "model.add(layers.TimeDistributed(layers.Dense(len(chars))))\n",
    "model.add(layers.Activation('softmax'))\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=['accuracy'])\n",
    "model.summary()\n",
    "\n",
    "# Train the model each generation and show predictions against the validation\n",
    "# dataset.\n",
    "for iteration in range(1, 200):\n",
    "    print()\n",
    "    print('-' * 50)\n",
    "    print('Iteration', iteration)\n",
    "    model.fit(x_train, y_train,\n",
    "              batch_size=BATCH_SIZE,\n",
    "              epochs=1,\n",
    "              validation_data=(x_val, y_val))\n",
    "    # Select 10 samples from the validation set at random so we can visualize\n",
    "    # errors.\n",
    "    for i in range(10):\n",
    "        ind = np.random.randint(0, len(x_val))\n",
    "        rowx, rowy = x_val[np.array([ind])], y_val[np.array([ind])]\n",
    "        preds = model.predict_classes(rowx, verbose=0)\n",
    "        q = ctable.decode(rowx[0])\n",
    "        correct = ctable.decode(rowy[0])\n",
    "        guess = ctable.decode(preds[0], calc_argmax=False)\n",
    "        print('Q', q[::-1] if REVERSE else q, end=' ')\n",
    "        print('T', correct, end=' ')\n",
    "        if correct == guess:\n",
    "            print(colors.ok + '☑' + colors.close, end=' ')\n",
    "        else:\n",
    "            print(colors.fail + '☒' + colors.close, end=' ')\n",
    "        print(guess)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The general case: canonical sequence-to-sequence\n",
    "\n",
    "In the general case, input sequences and output sequences have different lengths (e.g. machine translation) and the entire input sequence is required in order to start predicting the target. This requires a more advanced setup, which is what people commonly refer to when mentioning \"sequence to sequence models\" with no further context. Here's how it works:\n",
    "\n",
    "+ A RNN layer (or stack thereof) acts as \"encoder\": it processes the input sequence and returns its own internal state. Note that we discard the outputs of the encoder RNN, only recovering the state. This state will serve as the \"context\", or \"conditioning\", of the decoder in the next step.\n",
    "+ Another RNN layer (or stack thereof) acts as \"decoder\": it is trained to predict the next characters of the target sequence, given previous characters of the target sequence. Specifically, it is trained to turn the target sequences into the same sequences but offset by one timestep in the future, a training process called \"teacher forcing\" in this context. Importantly, the encoder uses as initial state the state vectors from the encoder, which is how the decoder obtains information about what it is supposed to generate. Effectively, the decoder learns to generate ``targets[t+1...]`` given ``targets[...t]``, conditioned on the input sequence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In inference mode, i.e. when we want to decode unknown input sequences, we go through a slightly different process:\n",
    "\n",
    "+ Encode the input sequence into state vectors.\n",
    "+ Start with a target sequence of size 1 (just the start-of-sequence character).\n",
    "+ Feed the state vectors and 1-char target sequence to the decoder to produce predictions for the next character.\n",
    "+ Sample the next character using these predictions (we simply use ``argmax``).\n",
    "+ Append the sampled character to the target sequence\n",
    "+ Repeat until we generate the end-of-sequence character or we hit the character limit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For our example implementation, we will use a dataset of pairs of English sentences and their Catalan translation (which can be downloaded from [manythings.org/anki](http://manythings.org/anki)). \n",
    "\n",
    "We will implement a character-level sequence-to-sequence model, processing the input character-by-character and generating the output character-by-character. Another option would be a word-level model, which tends to be more common for machine translation. \n",
    "\n",
    "Here's a summary of our process:\n",
    "\n",
    "1) Turn the sentences into 3 Numpy arrays, ``encoder_input_data``, ``decoder_input_data``, ``decoder_target_data``:\n",
    " + ``encoder_input_data`` is a 3D array of shape ``(num_pairs, max_english_sentence_length, num_english_characters)`` containing a one-hot vectorization of the English sentences.\n",
    " + ``decoder_input_data`` is a 3D array of shape ``(num_pairs, max_cat_sentence_length, num_cat_characters)`` containg a one-hot vectorization of the Catalan sentences.\n",
    " + ``decoder_target_data`` is the same as ``decoder_input_data`` but offset by one timestep. ``decoder_target_data[:, t, :]`` will be the same as ``decoder_input_data[:, t + 1, :]``.\n",
    "\n",
    "2) Train a basic LSTM-based ``Seq2Seq`` model to predict decoder_target_data given ``encoder_input_data`` and ``decoder_input_data``. Our model uses teacher forcing.\n",
    "\n",
    "3) Decode some sentences to check that the model is working (i.e. turn samples from encoder_input_data into corresponding samples from decoder_target_data).\n",
    "\n",
    "Because the training process and inference process (decoding sentences) are quite different, we use different models for both, albeit they all leverage the same inner layers.\n",
    "\n",
    "This is our training model. It leverages three key features of Keras RNNs:\n",
    "\n",
    "+ The ``return_state`` contructor argument, configuring a RNN layer to return a list where the first entry is the outputs and the next entries are the internal RNN states. This is used to recover the states of the encoder.\n",
    "+ The ``inital_state`` call argument, specifying the initial state(s) of a RNN. This is used to pass the encoder states to the decoder as initial states.\n",
    "+ The ``return_sequences`` constructor argument, configuring a RNN to return its full sequence of outputs (instead of just the last output, which the defaults behavior). This is used in the decoder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of samples: 606\n",
      "Number of unique input tokens: 68\n",
      "Number of unique output tokens: 81\n",
      "Max sequence length for inputs: 107\n",
      "Max sequence length for outputs: 106\n",
      "Train on 484 samples, validate on 122 samples\n",
      "Epoch 1/100\n",
      "484/484 [==============================] - 12s 25ms/step - loss: 0.9631 - val_loss: 1.4153\n",
      "Epoch 2/100\n",
      "484/484 [==============================] - 11s 23ms/step - loss: 0.8049 - val_loss: 1.3950\n",
      "Epoch 3/100\n",
      "484/484 [==============================] - 11s 23ms/step - loss: 0.7909 - val_loss: 1.3906\n",
      "Epoch 4/100\n",
      "484/484 [==============================] - 12s 25ms/step - loss: 0.7885 - val_loss: 1.5268\n",
      "Epoch 5/100\n",
      "484/484 [==============================] - 13s 26ms/step - loss: 0.8532 - val_loss: 1.5471\n",
      "Epoch 6/100\n",
      "484/484 [==============================] - 11s 24ms/step - loss: 0.8309 - val_loss: 1.3669\n",
      "Epoch 7/100\n",
      "484/484 [==============================] - 12s 24ms/step - loss: 0.7718 - val_loss: 1.3829\n",
      "Epoch 8/100\n",
      "484/484 [==============================] - 13s 27ms/step - loss: 0.7688 - val_loss: 1.3636\n",
      "Epoch 9/100\n",
      "484/484 [==============================] - 12s 26ms/step - loss: 0.7632 - val_loss: 1.3582\n",
      "Epoch 10/100\n",
      "484/484 [==============================] - 12s 26ms/step - loss: 0.7602 - val_loss: 1.3553\n",
      "Epoch 11/100\n",
      "448/484 [==========================>...] - ETA: 0s - loss: 0.7550"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-8-4304be9a9031>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m    101\u001b[0m           \u001b[0mbatch_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbatch_size\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    102\u001b[0m           \u001b[0mepochs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mepochs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 103\u001b[0;31m           validation_split=0.2)\n\u001b[0m\u001b[1;32m    104\u001b[0m \u001b[0;31m# Save model\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    105\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msave\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m's2s.h5'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/engine/training.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, **kwargs)\u001b[0m\n\u001b[1;32m   1667\u001b[0m                               \u001b[0minitial_epoch\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0minitial_epoch\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1668\u001b[0m                               \u001b[0msteps_per_epoch\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msteps_per_epoch\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1669\u001b[0;31m                               validation_steps=validation_steps)\n\u001b[0m\u001b[1;32m   1670\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1671\u001b[0m     def evaluate(self, x=None, y=None,\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/engine/training.py\u001b[0m in \u001b[0;36m_fit_loop\u001b[0;34m(self, f, ins, out_labels, batch_size, epochs, verbose, callbacks, val_f, val_ins, shuffle, callback_metrics, initial_epoch, steps_per_epoch, validation_steps)\u001b[0m\n\u001b[1;32m   1218\u001b[0m                             val_outs = self._test_loop(val_f, val_ins,\n\u001b[1;32m   1219\u001b[0m                                                        \u001b[0mbatch_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbatch_size\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1220\u001b[0;31m                                                        verbose=0)\n\u001b[0m\u001b[1;32m   1221\u001b[0m                             \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mval_outs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1222\u001b[0m                                 \u001b[0mval_outs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mval_outs\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/engine/training.py\u001b[0m in \u001b[0;36m_test_loop\u001b[0;34m(self, f, ins, batch_size, verbose, steps)\u001b[0m\n\u001b[1;32m   1380\u001b[0m                     \u001b[0mins_batch\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mins_batch\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtoarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1381\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1382\u001b[0;31m                 \u001b[0mbatch_outs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mins_batch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1383\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch_outs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1384\u001b[0m                     \u001b[0;32mif\u001b[0m \u001b[0mbatch_index\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/backend/tensorflow_backend.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, inputs)\u001b[0m\n\u001b[1;32m   2473\u001b[0m         \u001b[0msession\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_session\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2474\u001b[0m         updated = session.run(fetches=fetches, feed_dict=feed_dict,\n\u001b[0;32m-> 2475\u001b[0;31m                               **self.session_kwargs)\n\u001b[0m\u001b[1;32m   2476\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mupdated\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutputs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2477\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    893\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    894\u001b[0m       result = self._run(None, fetches, feed_dict, options_ptr,\n\u001b[0;32m--> 895\u001b[0;31m                          run_metadata_ptr)\n\u001b[0m\u001b[1;32m    896\u001b[0m       \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    897\u001b[0m         \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, handle, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1126\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mfinal_fetches\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mfinal_targets\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mhandle\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mfeed_dict_tensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1127\u001b[0m       results = self._do_run(handle, final_targets, final_fetches,\n\u001b[0;32m-> 1128\u001b[0;31m                              feed_dict_tensor, options, run_metadata)\n\u001b[0m\u001b[1;32m   1129\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1130\u001b[0m       \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_run\u001b[0;34m(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1342\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhandle\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1343\u001b[0m       return self._do_call(_run_fn, self._session, feeds, fetches, targets,\n\u001b[0;32m-> 1344\u001b[0;31m                            options, run_metadata)\n\u001b[0m\u001b[1;32m   1345\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1346\u001b[0m       \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_prun_fn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_session\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeeds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetches\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_call\u001b[0;34m(self, fn, *args)\u001b[0m\n\u001b[1;32m   1348\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1349\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1350\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1351\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mOpError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1352\u001b[0m       \u001b[0mmessage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcompat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_text\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run_fn\u001b[0;34m(session, feed_dict, fetch_list, target_list, options, run_metadata)\u001b[0m\n\u001b[1;32m   1327\u001b[0m           return tf_session.TF_Run(session, options,\n\u001b[1;32m   1328\u001b[0m                                    \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtarget_list\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1329\u001b[0;31m                                    status, run_metadata)\n\u001b[0m\u001b[1;32m   1330\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1331\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_prun_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msession\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "from keras.models import Model\n",
    "from keras.layers import Input, LSTM, Dense\n",
    "import numpy as np\n",
    "\n",
    "batch_size = 64  # Batch size for training.\n",
    "epochs = 100  # Number of epochs to train for.\n",
    "latent_dim = 256  # Latent dimensionality of the encoding space.\n",
    "num_samples = 10000  # Number of samples to train on.\n",
    "# Path to the data txt file on disk.\n",
    "data_path = 'data/cat.txt'\n",
    "\n",
    "# Vectorize the data.\n",
    "input_texts = []\n",
    "target_texts = []\n",
    "input_characters = set()\n",
    "target_characters = set()\n",
    "with open(data_path, 'r', encoding='utf-8') as f:\n",
    "    lines = f.read().split('\\n')\n",
    "for line in lines[: min(num_samples, len(lines) - 1)]:\n",
    "    input_text, target_text = line.split('\\t')\n",
    "    # We use \"tab\" as the \"start sequence\" character\n",
    "    # for the targets, and \"\\n\" as \"end sequence\" character.\n",
    "    target_text = '\\t' + target_text + '\\n'\n",
    "    input_texts.append(input_text)\n",
    "    target_texts.append(target_text)\n",
    "    for char in input_text:\n",
    "        if char not in input_characters:\n",
    "            input_characters.add(char)\n",
    "    for char in target_text:\n",
    "        if char not in target_characters:\n",
    "            target_characters.add(char)\n",
    "\n",
    "input_characters = sorted(list(input_characters))\n",
    "target_characters = sorted(list(target_characters))\n",
    "num_encoder_tokens = len(input_characters)\n",
    "num_decoder_tokens = len(target_characters)\n",
    "max_encoder_seq_length = max([len(txt) for txt in input_texts])\n",
    "max_decoder_seq_length = max([len(txt) for txt in target_texts])\n",
    "\n",
    "print('Number of samples:', len(input_texts))\n",
    "print('Number of unique input tokens:', num_encoder_tokens)\n",
    "print('Number of unique output tokens:', num_decoder_tokens)\n",
    "print('Max sequence length for inputs:', max_encoder_seq_length)\n",
    "print('Max sequence length for outputs:', max_decoder_seq_length)\n",
    "\n",
    "input_token_index = dict(\n",
    "    [(char, i) for i, char in enumerate(input_characters)])\n",
    "target_token_index = dict(\n",
    "    [(char, i) for i, char in enumerate(target_characters)])\n",
    "\n",
    "encoder_input_data = np.zeros(\n",
    "    (len(input_texts), max_encoder_seq_length, num_encoder_tokens),\n",
    "    dtype='float32')\n",
    "decoder_input_data = np.zeros(\n",
    "    (len(input_texts), max_decoder_seq_length, num_decoder_tokens),\n",
    "    dtype='float32')\n",
    "decoder_target_data = np.zeros(\n",
    "    (len(input_texts), max_decoder_seq_length, num_decoder_tokens),\n",
    "    dtype='float32')\n",
    "\n",
    "for i, (input_text, target_text) in enumerate(zip(input_texts, target_texts)):\n",
    "    for t, char in enumerate(input_text):\n",
    "        encoder_input_data[i, t, input_token_index[char]] = 1.\n",
    "    for t, char in enumerate(target_text):\n",
    "        # decoder_target_data is ahead of decoder_input_data by one timestep\n",
    "        decoder_input_data[i, t, target_token_index[char]] = 1.\n",
    "        if t > 0:\n",
    "            # decoder_target_data will be ahead by one timestep\n",
    "            # and will not include the start character.\n",
    "            decoder_target_data[i, t - 1, target_token_index[char]] = 1.\n",
    "\n",
    "# Define an input sequence and process it.\n",
    "encoder_inputs = Input(shape=(None, num_encoder_tokens))\n",
    "encoder = LSTM(latent_dim, return_state=True)\n",
    "encoder_outputs, state_h, state_c = encoder(encoder_inputs)\n",
    "\n",
    "# We discard `encoder_outputs` and only keep the states.\n",
    "encoder_states = [state_h, state_c]\n",
    "\n",
    "# Set up the decoder, using `encoder_states` as initial state.\n",
    "decoder_inputs = Input(shape=(None, num_decoder_tokens))\n",
    "\n",
    "# We set up our decoder to return full output sequences,\n",
    "# and to return internal states as well. We don't use the\n",
    "# return states in the training model, but we will use them in inference.\n",
    "decoder_lstm = LSTM(latent_dim, return_sequences=True, return_state=True)\n",
    "decoder_outputs, _, _ = decoder_lstm(decoder_inputs,\n",
    "                                     initial_state=encoder_states)\n",
    "decoder_dense = Dense(num_decoder_tokens, activation='softmax')\n",
    "decoder_outputs = decoder_dense(decoder_outputs)\n",
    "\n",
    "# Define the model that will turn\n",
    "# `encoder_input_data` & `decoder_input_data` into `decoder_target_data`\n",
    "model = Model([encoder_inputs, decoder_inputs], decoder_outputs)\n",
    "\n",
    "# Run training\n",
    "model.compile(optimizer='rmsprop', loss='categorical_crossentropy')\n",
    "model.fit([encoder_input_data, decoder_input_data], decoder_target_data,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,\n",
    "          validation_split=0.2)\n",
    "# Save model\n",
    "model.save('s2s.h5')\n",
    "\n",
    "# Next: inference mode (sampling).\n",
    "# Here's the drill:\n",
    "# 1) encode input and retrieve initial decoder state\n",
    "# 2) run one step of decoder with this initial state\n",
    "# and a \"start of sequence\" token as target.\n",
    "# Output will be the next target token\n",
    "# 3) Repeat with the current target token and current states\n",
    "\n",
    "# Define sampling models\n",
    "encoder_model = Model(encoder_inputs, encoder_states)\n",
    "\n",
    "decoder_state_input_h = Input(shape=(latent_dim,))\n",
    "decoder_state_input_c = Input(shape=(latent_dim,))\n",
    "decoder_states_inputs = [decoder_state_input_h, decoder_state_input_c]\n",
    "decoder_outputs, state_h, state_c = decoder_lstm(\n",
    "    decoder_inputs, initial_state=decoder_states_inputs)\n",
    "decoder_states = [state_h, state_c]\n",
    "decoder_outputs = decoder_dense(decoder_outputs)\n",
    "decoder_model = Model(\n",
    "    [decoder_inputs] + decoder_states_inputs,\n",
    "    [decoder_outputs] + decoder_states)\n",
    "\n",
    "# Reverse-lookup token index to decode sequences back to\n",
    "# something readable.\n",
    "reverse_input_char_index = dict(\n",
    "    (i, char) for char, i in input_token_index.items())\n",
    "reverse_target_char_index = dict(\n",
    "    (i, char) for char, i in target_token_index.items())\n",
    "\n",
    "\n",
    "def decode_sequence(input_seq):\n",
    "    # Encode the input as state vectors.\n",
    "    states_value = encoder_model.predict(input_seq)\n",
    "\n",
    "    # Generate empty target sequence of length 1.\n",
    "    target_seq = np.zeros((1, 1, num_decoder_tokens))\n",
    "    # Populate the first character of target sequence with the start character.\n",
    "    target_seq[0, 0, target_token_index['\\t']] = 1.\n",
    "\n",
    "    # Sampling loop for a batch of sequences\n",
    "    # (to simplify, here we assume a batch of size 1).\n",
    "    stop_condition = False\n",
    "    decoded_sentence = ''\n",
    "    while not stop_condition:\n",
    "        output_tokens, h, c = decoder_model.predict(\n",
    "            [target_seq] + states_value)\n",
    "\n",
    "        # Sample a token\n",
    "        sampled_token_index = np.argmax(output_tokens[0, -1, :])\n",
    "        sampled_char = reverse_target_char_index[sampled_token_index]\n",
    "        decoded_sentence += sampled_char\n",
    "\n",
    "        # Exit condition: either hit max length\n",
    "        # or find stop character.\n",
    "        if (sampled_char == '\\n' or\n",
    "           len(decoded_sentence) > max_decoder_seq_length):\n",
    "            stop_condition = True\n",
    "\n",
    "        # Update the target sequence (of length 1).\n",
    "        target_seq = np.zeros((1, 1, num_decoder_tokens))\n",
    "        target_seq[0, 0, sampled_token_index] = 1.\n",
    "\n",
    "        # Update states\n",
    "        states_value = [h, c]\n",
    "\n",
    "    return decoded_sentence\n",
    "\n",
    "\n",
    "for seq_index in range(100):\n",
    "    # Take one sequence (part of the training set)\n",
    "    # for trying out decoding.\n",
    "    input_seq = encoder_input_data[seq_index: seq_index + 1]\n",
    "    decoded_sentence = decode_sequence(input_seq)\n",
    "    print('-')\n",
    "    print('Input sentence:', input_texts[seq_index])\n",
    "    print('Decoded sentence:', decoded_sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example\n",
    "\n",
    "A Recurrent Neural Network (LSTM) implementation example using TensorFlow library.\n",
    "\n",
    "This example is using the MNIST database of handwritten digits (http://yann.lecun.com/exdb/mnist/)\n",
    "Long Short Term Memory paper: http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf\n",
    "\n",
    "Author: Aymeric Damien\n",
    "Project: https://github.com/aymericdamien/TensorFlow-Examples/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.5/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.\n",
      "Extracting /tmp/data/train-images-idx3-ubyte.gz\n",
      "Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.\n",
      "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n",
      "Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.\n",
      "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n",
      "Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.\n",
      "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow.contrib import rnn\n",
    "\n",
    "# Import MNIST data\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To classify images using a recurrent neural network, we consider every image row as a sequence of pixels. \n",
    "\n",
    "Because MNIST image shape is $28 \\times 28$ px, we will then handle 28 sequences of 28 steps for every sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Training Parameters\n",
    "learning_rate = 0.001\n",
    "training_steps = 10000\n",
    "batch_size = 128\n",
    "display_step = 200\n",
    "\n",
    "# Network Parameters\n",
    "num_input = 28 # MNIST data input (img shape: 28*28)\n",
    "timesteps = 28 # timesteps\n",
    "num_hidden = 128 # hidden layer num of features\n",
    "num_classes = 10 # MNIST total classes (0-9 digits)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# tf Graph input\n",
    "X = tf.placeholder(\"float\", [None, timesteps, num_input])\n",
    "Y = tf.placeholder(\"float\", [None, num_classes])\n",
    "\n",
    "# Define weights\n",
    "weights = {\n",
    "    'out': tf.Variable(tf.random_normal([num_hidden, num_classes]))\n",
    "}\n",
    "biases = {\n",
    "    'out': tf.Variable(tf.random_normal([num_classes]))\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def RNN(x, weights, biases):\n",
    "\n",
    "    # Prepare data shape to match `rnn` function requirements\n",
    "    # Current data input shape: (batch_size, timesteps, n_input)\n",
    "    # Required shape: 'timesteps' tensors list of shape (batch_size, n_input)\n",
    "\n",
    "    # Unstack to get a list of 'timesteps' tensors of shape (batch_size, n_input)\n",
    "    x = tf.unstack(x, timesteps, 1)\n",
    "\n",
    "    # Define a lstm cell with tensorflow\n",
    "    lstm_cell = rnn.BasicLSTMCell(num_hidden, forget_bias=1.0)\n",
    "\n",
    "    # Get lstm cell output\n",
    "    outputs, states = rnn.static_rnn(lstm_cell, x, dtype=tf.float32)\n",
    "\n",
    "    # Linear activation, using rnn inner loop last output\n",
    "    return tf.matmul(outputs[-1], weights['out']) + biases['out']\n",
    "\n",
    "logits = RNN(X, weights, biases)\n",
    "prediction = tf.nn.softmax(logits)\n",
    "\n",
    "# Define loss and optimizer\n",
    "loss_op = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(\n",
    "    logits=logits, labels=Y))\n",
    "optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)\n",
    "train_op = optimizer.minimize(loss_op)\n",
    "\n",
    "# Evaluate model (with test logits, for dropout to be disabled)\n",
    "correct_pred = tf.equal(tf.argmax(prediction, 1), tf.argmax(Y, 1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))\n",
    "\n",
    "# Initialize the variables (i.e. assign their default value)\n",
    "init = tf.global_variables_initializer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step 1, Minibatch Loss= 2.7822, Training Accuracy= 0.109\n",
      "Step 200, Minibatch Loss= 2.1688, Training Accuracy= 0.250\n",
      "Step 400, Minibatch Loss= 1.9863, Training Accuracy= 0.383\n",
      "Step 600, Minibatch Loss= 1.9121, Training Accuracy= 0.352\n",
      "Step 800, Minibatch Loss= 1.7026, Training Accuracy= 0.492\n",
      "Step 1000, Minibatch Loss= 1.5861, Training Accuracy= 0.531\n",
      "Step 1200, Minibatch Loss= 1.5900, Training Accuracy= 0.461\n",
      "Step 1400, Minibatch Loss= 1.3426, Training Accuracy= 0.633\n",
      "Step 1600, Minibatch Loss= 1.5044, Training Accuracy= 0.492\n",
      "Step 1800, Minibatch Loss= 1.4679, Training Accuracy= 0.547\n",
      "Step 2000, Minibatch Loss= 1.2057, Training Accuracy= 0.641\n",
      "Step 2200, Minibatch Loss= 1.3191, Training Accuracy= 0.594\n",
      "Step 2400, Minibatch Loss= 1.2150, Training Accuracy= 0.570\n",
      "Step 2600, Minibatch Loss= 1.2847, Training Accuracy= 0.562\n",
      "Step 2800, Minibatch Loss= 1.1208, Training Accuracy= 0.633\n",
      "Step 3000, Minibatch Loss= 1.0652, Training Accuracy= 0.688\n",
      "Step 3200, Minibatch Loss= 1.0579, Training Accuracy= 0.656\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-5-49b87c1a1010>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      9\u001b[0m         \u001b[0mbatch_x\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbatch_x\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreshape\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtimesteps\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnum_input\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m         \u001b[0;31m# Run optimization op (backprop)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m         \u001b[0msess\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrain_op\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m{\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbatch_x\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mY\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbatch_y\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     12\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mstep\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0mdisplay_step\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mstep\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     13\u001b[0m             \u001b[0;31m# Calculate batch loss and accuracy\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    893\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    894\u001b[0m       result = self._run(None, fetches, feed_dict, options_ptr,\n\u001b[0;32m--> 895\u001b[0;31m                          run_metadata_ptr)\n\u001b[0m\u001b[1;32m    896\u001b[0m       \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    897\u001b[0m         \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, handle, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1126\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mfinal_fetches\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mfinal_targets\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mhandle\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mfeed_dict_tensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1127\u001b[0m       results = self._do_run(handle, final_targets, final_fetches,\n\u001b[0;32m-> 1128\u001b[0;31m                              feed_dict_tensor, options, run_metadata)\n\u001b[0m\u001b[1;32m   1129\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1130\u001b[0m       \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_run\u001b[0;34m(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1342\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhandle\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1343\u001b[0m       return self._do_call(_run_fn, self._session, feeds, fetches, targets,\n\u001b[0;32m-> 1344\u001b[0;31m                            options, run_metadata)\n\u001b[0m\u001b[1;32m   1345\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1346\u001b[0m       \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_prun_fn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_session\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeeds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetches\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_call\u001b[0;34m(self, fn, *args)\u001b[0m\n\u001b[1;32m   1348\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1349\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1350\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1351\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mOpError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1352\u001b[0m       \u001b[0mmessage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcompat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_text\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run_fn\u001b[0;34m(session, feed_dict, fetch_list, target_list, options, run_metadata)\u001b[0m\n\u001b[1;32m   1327\u001b[0m           return tf_session.TF_Run(session, options,\n\u001b[1;32m   1328\u001b[0m                                    \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtarget_list\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1329\u001b[0;31m                                    status, run_metadata)\n\u001b[0m\u001b[1;32m   1330\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1331\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_prun_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msession\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "\n",
    "    # Run the initializer\n",
    "    sess.run(init)\n",
    "\n",
    "    for step in range(1, training_steps+1):\n",
    "        batch_x, batch_y = mnist.train.next_batch(batch_size)\n",
    "        # Reshape data to get 28 seq of 28 elements\n",
    "        batch_x = batch_x.reshape((batch_size, timesteps, num_input))\n",
    "        # Run optimization op (backprop)\n",
    "        sess.run(train_op, feed_dict={X: batch_x, Y: batch_y})\n",
    "        if step % display_step == 0 or step == 1:\n",
    "            # Calculate batch loss and accuracy\n",
    "            loss, acc = sess.run([loss_op, accuracy], feed_dict={X: batch_x,\n",
    "                                                                 Y: batch_y})\n",
    "            print(\"Step \" + str(step) + \", Minibatch Loss= \" + \\\n",
    "                  \"{:.4f}\".format(loss) + \", Training Accuracy= \" + \\\n",
    "                  \"{:.3f}\".format(acc))\n",
    "\n",
    "    print(\"Optimization Finished!\")\n",
    "\n",
    "    # Calculate accuracy for 128 mnist test images\n",
    "    test_len = 128\n",
    "    test_data = mnist.test.images[:test_len].reshape((-1, timesteps, num_input))\n",
    "    test_label = mnist.test.labels[:test_len]\n",
    "    print(\"Testing Accuracy:\", \\\n",
    "        sess.run(accuracy, feed_dict={X: test_data, Y: test_label}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bidirectional LSTM\n",
    "\n",
    "So  far,  we  have  focused  on  RNNs  that  look  into  the  past    to predict future values in the sequence, but not to  to make predictions based on future values by reading throught the sequence backwards?\n",
    "\n",
    "**Bi-directional  deep  neural networs**,  at each time-step, $t$,  maintain two hidden layers, one for the left-to-right propagation and another for the right-to-left  propagation (hence, consuming twice  as  much  memory  space).   \n",
    "\n",
    "The final classification result, $\\hat{y}$, is generated through combining the score results produced by both RNN hidden layers.\n",
    "\n",
    "<img src = \"./images/t9.png\"  width = \"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The equations are (arrows are for designing left-to-right and right-to-left tensors):\n",
    "\n",
    "$$\n",
    "\t\\overrightarrow{h}_t = f(\\overrightarrow{W} x_t + \\overrightarrow{V} \\overrightarrow{h}_{t-1} + \\overrightarrow{b})\n",
    "$$\n",
    "$$\n",
    "\t\\overleftarrow{h}_t = f(\\overleftarrow{W} x_t + \\overleftarrow{V} \\overleftarrow{h}_{t+1} + \\overleftarrow{b})\n",
    "$$\n",
    "$$\n",
    "\t\\hat{y}_t = g(U h_t + c) = g(U [\\overrightarrow{h}_t; \\overleftarrow{h}_t] + c)\n",
    "$$\n",
    "\n",
    "$[\\overrightarrow{h}_t; \\overleftarrow{h}_t]$ summarizes the past and future of a single element of the sequence.\n",
    "\n",
    "Biderectional RNNs can be stacked as usual!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.5/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting /tmp/data/train-images-idx3-ubyte.gz\n",
      "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n",
      "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n",
      "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow.contrib import rnn\n",
    "\n",
    "# Import MNIST data\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)\n",
    "\n",
    "'''\n",
    "To classify images using a recurrent neural network, we consider every image\n",
    "row as a sequence of pixels. Because MNIST image shape is 28*28px, we will then\n",
    "handle 28 sequences of 28 steps for every sample.\n",
    "'''\n",
    "\n",
    "# Training Parameters\n",
    "learning_rate = 0.001\n",
    "training_steps = 10000\n",
    "batch_size = 128\n",
    "display_step = 200\n",
    "\n",
    "# Network Parameters\n",
    "num_input = 28 # MNIST data input (img shape: 28*28)\n",
    "timesteps = 28 # timesteps\n",
    "num_hidden = 128 # hidden layer num of features\n",
    "num_classes = 10 # MNIST total classes (0-9 digits)\n",
    "\n",
    "# tf Graph input\n",
    "X = tf.placeholder(\"float\", [None, timesteps, num_input])\n",
    "Y = tf.placeholder(\"float\", [None, num_classes])\n",
    "\n",
    "# Define weights\n",
    "weights = {\n",
    "    'out': tf.Variable(tf.random_normal([num_hidden, num_classes]))\n",
    "}\n",
    "biases = {\n",
    "    'out': tf.Variable(tf.random_normal([num_classes]))\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-3-3f42a567fcfd>:24: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "\n",
      "Future major versions of TensorFlow will allow gradients to flow\n",
      "into the labels input on backprop by default.\n",
      "\n",
      "See tf.nn.softmax_cross_entropy_with_logits_v2.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def RNN(x, weights, biases):\n",
    "\n",
    "    # Prepare data shape to match `rnn` function requirements\n",
    "    # Current data input shape: (batch_size, timesteps, n_input)\n",
    "    # Required shape: 'timesteps' tensors list of shape (batch_size, n_input)\n",
    "\n",
    "    # Unstack to get a list of 'timesteps' tensors of shape (batch_size, n_input)\n",
    "    x = tf.unstack(x, timesteps, 1)\n",
    "\n",
    "    # Define a lstm cell with tensorflow\n",
    "    lstm_cell = rnn.BasicLSTMCell(num_hidden, forget_bias=1.0)\n",
    "\n",
    "    # Get lstm cell output\n",
    "    outputs, states = rnn.static_rnn(lstm_cell, x, dtype=tf.float32)\n",
    "\n",
    "    # Linear activation, using rnn inner loop last output\n",
    "    return tf.matmul(outputs[-1], weights['out']) + biases['out']\n",
    "\n",
    "logits = RNN(X, weights, biases)\n",
    "prediction = tf.nn.softmax(logits)\n",
    "\n",
    "# Define loss and optimizer\n",
    "loss_op = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(\n",
    "    logits=logits, labels=Y))\n",
    "optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)\n",
    "train_op = optimizer.minimize(loss_op)\n",
    "\n",
    "# Evaluate model (with test logits, for dropout to be disabled)\n",
    "correct_pred = tf.equal(tf.argmax(prediction, 1), tf.argmax(Y, 1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))\n",
    "\n",
    "# Initialize the variables (i.e. assign their default value)\n",
    "init = tf.global_variables_initializer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step 1, Minibatch Loss= 2.8939, Training Accuracy= 0.070\n",
      "Step 200, Minibatch Loss= 2.1486, Training Accuracy= 0.312\n",
      "Step 400, Minibatch Loss= 1.9229, Training Accuracy= 0.398\n",
      "Step 600, Minibatch Loss= 1.8449, Training Accuracy= 0.414\n",
      "Step 800, Minibatch Loss= 1.7218, Training Accuracy= 0.422\n",
      "Step 1000, Minibatch Loss= 1.5698, Training Accuracy= 0.586\n",
      "Step 1200, Minibatch Loss= 1.6332, Training Accuracy= 0.453\n",
      "Step 1400, Minibatch Loss= 1.4396, Training Accuracy= 0.484\n",
      "Step 1600, Minibatch Loss= 1.4493, Training Accuracy= 0.555\n",
      "Step 1800, Minibatch Loss= 1.3903, Training Accuracy= 0.555\n",
      "Step 2000, Minibatch Loss= 1.3128, Training Accuracy= 0.602\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-4-49b87c1a1010>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      9\u001b[0m         \u001b[0mbatch_x\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbatch_x\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreshape\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtimesteps\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnum_input\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m         \u001b[0;31m# Run optimization op (backprop)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m         \u001b[0msess\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrain_op\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m{\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbatch_x\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mY\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbatch_y\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     12\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mstep\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0mdisplay_step\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mstep\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     13\u001b[0m             \u001b[0;31m# Calculate batch loss and accuracy\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    893\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    894\u001b[0m       result = self._run(None, fetches, feed_dict, options_ptr,\n\u001b[0;32m--> 895\u001b[0;31m                          run_metadata_ptr)\n\u001b[0m\u001b[1;32m    896\u001b[0m       \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    897\u001b[0m         \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, handle, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1126\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mfinal_fetches\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mfinal_targets\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mhandle\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mfeed_dict_tensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1127\u001b[0m       results = self._do_run(handle, final_targets, final_fetches,\n\u001b[0;32m-> 1128\u001b[0;31m                              feed_dict_tensor, options, run_metadata)\n\u001b[0m\u001b[1;32m   1129\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1130\u001b[0m       \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_run\u001b[0;34m(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1342\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhandle\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1343\u001b[0m       return self._do_call(_run_fn, self._session, feeds, fetches, targets,\n\u001b[0;32m-> 1344\u001b[0;31m                            options, run_metadata)\n\u001b[0m\u001b[1;32m   1345\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1346\u001b[0m       \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_prun_fn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_session\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeeds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetches\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_call\u001b[0;34m(self, fn, *args)\u001b[0m\n\u001b[1;32m   1348\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1349\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1350\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1351\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mOpError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1352\u001b[0m       \u001b[0mmessage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcompat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_text\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run_fn\u001b[0;34m(session, feed_dict, fetch_list, target_list, options, run_metadata)\u001b[0m\n\u001b[1;32m   1327\u001b[0m           return tf_session.TF_Run(session, options,\n\u001b[1;32m   1328\u001b[0m                                    \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtarget_list\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1329\u001b[0;31m                                    status, run_metadata)\n\u001b[0m\u001b[1;32m   1330\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1331\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_prun_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msession\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "\n",
    "    # Run the initializer\n",
    "    sess.run(init)\n",
    "\n",
    "    for step in range(1, training_steps+1):\n",
    "        batch_x, batch_y = mnist.train.next_batch(batch_size)\n",
    "        # Reshape data to get 28 seq of 28 elements\n",
    "        batch_x = batch_x.reshape((batch_size, timesteps, num_input))\n",
    "        # Run optimization op (backprop)\n",
    "        sess.run(train_op, feed_dict={X: batch_x, Y: batch_y})\n",
    "        if step % display_step == 0 or step == 1:\n",
    "            # Calculate batch loss and accuracy\n",
    "            loss, acc = sess.run([loss_op, accuracy], feed_dict={X: batch_x,\n",
    "                                                                 Y: batch_y})\n",
    "            print(\"Step \" + str(step) + \", Minibatch Loss= \" + \\\n",
    "                  \"{:.4f}\".format(loss) + \", Training Accuracy= \" + \\\n",
    "                  \"{:.3f}\".format(acc))\n",
    "\n",
    "    print(\"Optimization Finished!\")\n",
    "\n",
    "    # Calculate accuracy for 128 mnist test images\n",
    "    test_len = 128\n",
    "    test_data = mnist.test.images[:test_len].reshape((-1, timesteps, num_input))\n",
    "    test_label = mnist.test.labels[:test_len]\n",
    "    print(\"Testing Accuracy:\", \\\n",
    "        sess.run(accuracy, feed_dict={X: test_data, Y: test_label}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Name generation with LSTM\n",
    "\n",
    "We are going to train RNN \"character-level\" language models. \n",
    "\n",
    "That is, we’ll give the RNN a huge chunk of text and ask it to model the probability distribution of the next character in the sequence given a sequence of previous characters. This will then allow us to generate new text one character at a time.\n",
    "\n",
    "We will encode each character into a vector using ``1-of-k`` encoding (i.e. all zero except for a single one at the index of the character in the vocabulary), and feed them into the RNN one at a time. \n",
    "\n",
    "At test time, we will feed a sequence of characters into the RNN and get a distribution over what characters are likely to come next. We sample from this distribution, and feed it right back in to get the next letter. Repeat this process and you’re sampling text!\n",
    "\n",
    "We can also play with the temperature of the Softmax during sampling. Decreasing the temperature from 1 to some lower number (e.g. 0.5) makes the RNN more confident, but also more conservative in its samples. Conversely, higher temperatures will give more diversity but at cost of more mistakes.\n",
    "\n",
    "> Remider: **Softmax**\n",
    ">$$\n",
    "\tP(y = j | \\mathbf{x}) = \\frac{\\exp(\\mathbf{x}^T \\mathbf{w}_j)/\\tau}{\\sum_{k=1}^K \\exp(\\mathbf{x}^T \\mathbf{w}_k)/\\tau}\n",
    "$$\n",
    "> $\\tau$ is called *temperature*. For high temperatures ( $ \\tau \\to \\infty $ ), all $y$ have nearly the same probability. For a low temperature ( $ \\tau \\to 0^{+}$), the probability of the most probable $y$ tends to 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to process sequences of symbols with RNN we need to represent these symbols by numbers.\n",
    "\n",
    "Let's suppose we have $|V|$ different symbols. The most simple representation is the **one-hot vector**: Represent every word as an $\\mathbb{R}^{|V|\\times1}$ vector with all $0$s and one $1$ at the index of that word in the sorted english language. Symbol vectors in this type of encoding would appear as the following:\n",
    "\n",
    "$$w^{s_1} = \\left[ \\begin{array}{c} 1 \\\\ 0 \\\\ 0 \\\\ \\vdots \\\\ 0 \\end{array} \\right], w^{s_2} = \\left[ \\begin{array}{c} 0 \\\\ 1 \\\\ 0 \\\\ \\vdots \\\\ 0 \\end{array} \\right], w^{s_3} = \\left[ \\begin{array}{c} 0 \\\\ 0 \\\\ 1 \\\\ \\vdots \\\\ 0 \\end{array} \\right], \\cdots \n",
    "w^{s_{|V|}} = \\left[ \\begin{array}{c} 0 \\\\ 0 \\\\ 0 \\\\ \\vdots \\\\ 1 \\end{array} \\right] $$\n",
    "\n",
    "We represent each symbol as a completely independent entity. This symbol representation does not give us directly any notion of similarity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train our model we need text to learn from a large dataset of names. Fortunately we don’t need any labels to train a language model, just raw text.\n",
    "\n",
    "Places names: You can download 52,700 Catalan names from a dataset available on http://territori.gencat.cat/ca/01_departament/11_normativa_i_documentacio/03_documentacio/02_territori_i_mobilitat/cartografia/nomenclator_oficial_de_toponimia_de_catalunya/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "corpus length: 47527\n",
      "total chars: 30\n",
      "nb sequences: 15836\n",
      "Vectorization...\n"
     ]
    }
   ],
   "source": [
    "from __future__ import print_function\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Activation\n",
    "from keras.layers import LSTM\n",
    "from keras.optimizers import RMSprop\n",
    "import numpy as np\n",
    "import random\n",
    "import sys\n",
    "\n",
    "import codecs\n",
    "f = codecs.open('data/NombresMujerBarcelona.txt', \"r\", \"utf-8\")\n",
    "#f = codecs.open('data/toponims.txt', \"r\", \"utf-8\")\n",
    "string = f.read()\n",
    "string.encode('utf-8')\n",
    "text = string.lower()\n",
    "\n",
    "# text = text.replace(\"\\n\", \" \")\n",
    "    \n",
    "print('corpus length:', len(text))\n",
    "\n",
    "chars = sorted(list(set(text)))\n",
    "print('total chars:', len(chars))\n",
    "char_indices = dict((c, i) for i, c in enumerate(chars))\n",
    "indices_char = dict((i, c) for i, c in enumerate(chars))\n",
    "\n",
    "# cut the text in semi-redundant sequences of maxlen characters\n",
    "maxlen = 20\n",
    "step = 3\n",
    "sentences = []\n",
    "next_chars = []\n",
    "for i in range(0, len(text) - maxlen, step):\n",
    "    sentences.append(text[i: i + maxlen])\n",
    "    next_chars.append(text[i + maxlen])\n",
    "print('nb sequences:', len(sentences))\n",
    "\n",
    "print('Vectorization...')\n",
    "X = np.zeros((len(sentences), maxlen, len(chars)), dtype=np.bool)\n",
    "y = np.zeros((len(sentences), len(chars)), dtype=np.bool)\n",
    "for i, sentence in enumerate(sentences):\n",
    "    for t, char in enumerate(sentence):\n",
    "        X[i, t, char_indices[char]] = 1\n",
    "    y[i, char_indices[next_chars[i]]] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Build model...\n"
     ]
    }
   ],
   "source": [
    "# build the model\n",
    "\n",
    "print('Build model...')\n",
    "model = Sequential()\n",
    "model.add(LSTM(64, \n",
    "#               return_sequences=True,\n",
    "               recurrent_dropout=0.2, \n",
    "               dropout=0.2, \n",
    "               input_shape=(maxlen, len(chars))))\n",
    "#model.add(LSTM(64, \n",
    "#               recurrent_dropout=0.2, \n",
    "#               dropout=0.2))\n",
    "model.add(Dense(len(chars)))\n",
    "model.add(Activation('relu'))\n",
    "model.add(Dropout(0.2))\n",
    "model.add(Activation('softmax'))\n",
    "\n",
    "optimizer = RMSprop(lr=0.01)\n",
    "model.compile(loss='categorical_crossentropy', optimizer=optimizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The simplest way to use the Keras LSTM model to make predictions is to first start off with a seed sequence as input, generate the next character then update the seed sequence to add the generated character on the end and trim off the first character. \n",
    "\n",
    "This process is repeated for as long as we want to predict new characters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "--------------------------------------------------\n",
      "Iteration 1\n",
      "Epoch 1/1\n",
      "15836/15836 [==============================] - 7s 439us/step - loss: 2.9351\n",
      "----- Generating with seed: \"mparo amy ana ana al\"\n",
      "\n",
      "----- diversity: 0.5\n",
      "elewela\n",
      "aaqa\n",
      "maria marina\n",
      "siisa ioja ilasela imari\n",
      "\n",
      "----- diversity: 1.0\n",
      "i ebmojda\n",
      "vc cra\n",
      "\n",
      "agdga teqkay esjlotcifa\n",
      "a\n",
      "gptus\n",
      "\n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 2\n",
      "Epoch 1/1\n",
      "15836/15836 [==============================] - 7s 434us/step - loss: 2.7172\n",
      "----- Generating with seed: \"a ione ionela ionica\"\n",
      "\n",
      "----- diversity: 0.5\n",
      "\n",
      "maria haria maria carela\n",
      "maria alia\n",
      "maria caria c\n",
      "\n",
      "----- diversity: 1.0\n",
      "i y\n",
      "hiatakzaawamqçña\n",
      "jbaz\n",
      "gapphria dlkiy exess\n",
      "jvr\n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 3\n",
      "Epoch 1/1\n",
      "15836/15836 [==============================] - 6s 409us/step - loss: 2.6507\n",
      "----- Generating with seed: \"onisa leonor leonor \"\n",
      "\n",
      "----- diversity: 0.5\n",
      "alvevevia\n",
      "ijjafila\n",
      "çoa\n",
      "xiodtk\n",
      "saria clolil\n",
      "agcexa \n",
      "\n",
      "----- diversity: 1.0\n",
      "urelila pjyghpaxmarka mesaflmaqiña\n",
      "margatçñkvxornv\n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 4\n",
      "Epoch 1/1\n",
      "15836/15836 [==============================] - 6s 396us/step - loss: 2.5735\n",
      "----- Generating with seed: \"ndrea nicoleta nicol\"\n",
      "\n",
      "----- diversity: 0.5\n",
      "a\n",
      "vina\n",
      "pafia\n",
      "anda\n",
      "alia\n",
      "abaua\n",
      "a\n",
      "oola\n",
      "any\n",
      "ariva\n",
      "xila\n",
      "\n",
      "----- diversity: 1.0\n",
      " ilpce\n",
      "abntau\n",
      "oçaawace çahzaqfzrnogacupsvca bareta\n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 5\n",
      "Epoch 1/1\n",
      "15836/15836 [==============================] - 7s 467us/step - loss: 2.4820\n",
      "----- Generating with seed: \"a misericordia maria\"\n",
      "\n",
      "----- diversity: 0.5\n",
      " maria\n",
      "maria laxila\n",
      "maria tola\n",
      "maria corela\n",
      "maria \n",
      "\n",
      "----- diversity: 1.0\n",
      "lmens\n",
      "larrxzfnvanchia\n",
      "maria ctofodis\n",
      "varna ymkoysa\n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 6\n",
      "Epoch 1/1\n",
      "15836/15836 [==============================] - 7s 417us/step - loss: 2.4401\n",
      "----- Generating with seed: \"a indira ines ines m\"\n",
      "\n",
      "----- diversity: 0.5\n",
      "arie\n",
      "ferakelia\n",
      "lezjanña\n",
      "eliera\n",
      "riça elen\n",
      "krceyor\n",
      "q\n",
      "\n",
      "----- diversity: 1.0\n",
      "ihrzeqiiuzabena\n",
      "p \n",
      "wremka kwenxexa maria\n",
      "farva \n",
      "ag\n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 7\n",
      "Epoch 1/1\n",
      "15836/15836 [==============================] - 8s 482us/step - loss: 2.4028\n",
      "----- Generating with seed: \"a aguila maria agust\"\n",
      "\n",
      "----- diversity: 0.5\n",
      "er\n",
      "maria lona\n",
      "maria eloria\n",
      "caria exina\n",
      "maria carce\n",
      "\n",
      "----- diversity: 1.0\n",
      "ña\n",
      "mnrwa lana\n",
      "rewa maria\n",
      "ap\n",
      "lsicanme xpctenscañbee\n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 8\n",
      "Epoch 1/1\n",
      "15836/15836 [==============================] - 8s 497us/step - loss: 2.3582\n",
      "----- Generating with seed: \"cruz cruz maria cuih\"\n",
      "\n",
      "----- diversity: 0.5\n",
      "ia maria\n",
      "clicia cancecia\n",
      "ccrncena  elene\n",
      "carman na\n",
      "\n",
      "----- diversity: 1.0\n",
      "oiqa\n",
      "varminieiceçcareugqñistjinjbeudreqgeoreñhsihp\n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 9\n",
      "Epoch 1/1\n",
      "15836/15836 [==============================] - 8s 477us/step - loss: 2.3229\n",
      "----- Generating with seed: \"eopolda leovigilda l\"\n",
      "\n",
      "----- diversity: 0.5\n",
      "iprqiy\n",
      "lilia\n",
      "lira\n",
      "lilia\n",
      "lilia\n",
      "lila\n",
      "lisa\n",
      "liana\n",
      "liub\n",
      "\n",
      "----- diversity: 1.0\n",
      "ña\n",
      "fijgdel\n",
      "flifetna\n",
      "lniçoy\n",
      "yviolaqjina\n",
      "dil ka\n",
      "rikd\n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 10\n",
      "Epoch 1/1\n",
      "15836/15836 [==============================] - 7s 465us/step - loss: 2.2919\n",
      "----- Generating with seed: \"eves blanca patricia\"\n",
      "\n",
      "----- diversity: 0.5\n",
      "\n",
      "maria sacya\n",
      "maria elepina\n",
      "maria conicia\n",
      "maria ane\n",
      "\n",
      "----- diversity: 1.0\n",
      "ya\n",
      "layia katpiobep\n",
      "cwrnenixa bearynbca\n",
      "karia igoli\n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 11\n",
      "Epoch 1/1\n",
      "15836/15836 [==============================] - 7s 469us/step - loss: 2.2725\n",
      "----- Generating with seed: \"eta enya epifania er\"\n",
      "\n",
      "----- diversity: 0.5\n",
      "ica\n",
      "aila\n",
      "aila\n",
      "ala\n",
      "alia\n",
      "alia\n",
      "ala\n",
      "alela\n",
      "ala\n",
      "ala alej\n",
      "\n",
      "----- diversity: 1.0\n",
      "as\n",
      "glaliora\n",
      "lidiia\n",
      "lcapgina\n",
      "lpfia oiela\n",
      "rgkaolaroj\n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 12\n",
      "Epoch 1/1\n",
      "15836/15836 [==============================] - 7s 426us/step - loss: 2.2386\n",
      "----- Generating with seed: \" eduviges eduvigis e\"\n",
      "\n",
      "----- diversity: 0.5\n",
      "te alena\n",
      "çaria maria\n",
      "caria kamia\n",
      "caria harmen\n",
      "mari\n",
      "\n",
      "----- diversity: 1.0\n",
      "a zñsejaveyl\n",
      "marfinwa\n",
      "maribew  maria\n",
      "badria cabjit\n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 13\n",
      "Epoch 1/1\n",
      "15836/15836 [==============================] - 7s 443us/step - loss: 2.2186\n",
      "----- Generating with seed: \"iovanna gisel gisela\"\n",
      "\n",
      "----- diversity: 0.5\n",
      "\n",
      "omia\n",
      "ñomina\n",
      "oolora\n",
      "olona\n",
      "olona\n",
      "uona\n",
      "cona maria\n",
      "co\n",
      "\n",
      "----- diversity: 1.0\n",
      "nna zrnzu\n",
      "poner\n",
      "roneca\n",
      " cdçica\n",
      "qizpika\n",
      "miona\n",
      "imra\n",
      "\n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 14\n",
      "Epoch 1/1\n",
      "15836/15836 [==============================] - 7s 455us/step - loss: 2.2024\n",
      "----- Generating with seed: \"atricia veronica roc\"\n",
      "\n",
      "----- diversity: 0.5\n",
      "ia\n",
      "cermencia alena\n",
      "cercina cribele\n",
      "cermen coneça\n",
      "c\n",
      "\n",
      "----- diversity: 1.0\n",
      "rilaxia cwrapina\n",
      "clerña crenicaqtincubes\n",
      "aezina\n",
      "hi\n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 15\n",
      "Epoch 1/1\n",
      "15836/15836 [==============================] - 7s 445us/step - loss: 2.1538\n",
      "----- Generating with seed: \"uma oana obdulia oct\"\n",
      "\n",
      "----- diversity: 0.5\n",
      "oria\n",
      "ojosa\n",
      "olona\n",
      "oola\n",
      "oolan\n",
      "ooca maria\n",
      "oosa maria\n",
      "\n",
      "\n",
      "----- diversity: 1.0\n",
      "covio a inaily\n",
      "oegiasua\n",
      "onafijafa\n",
      "frjermenviqiqçy\n",
      "\n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 16\n",
      "Epoch 1/1\n",
      "15836/15836 [==============================] - 8s 475us/step - loss: 2.1453\n",
      "----- Generating with seed: \"sa precious presenta\"\n",
      "\n",
      "----- diversity: 0.5\n",
      "\n",
      "remafe\n",
      "risa\n",
      "rima maria\n",
      "qizalina\n",
      "tieñoria\n",
      "olenia\n",
      "e\n",
      "\n",
      "----- diversity: 1.0\n",
      "sxqia\n",
      "euñiepebep\n",
      "eñttwolbej\n",
      "eç\n",
      "eñe na mariajçay\n",
      "er\n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 17\n",
      "Epoch 1/1\n",
      "15836/15836 [==============================] - 8s 497us/step - loss: 2.1251\n",
      "----- Generating with seed: \"ligia ligia elena li\"\n",
      "\n",
      "----- diversity: 0.5\n",
      "saça\n",
      "licia kana\n",
      "licia maria\n",
      "licia licen\n",
      "licia eles\n",
      "\n",
      "----- diversity: 1.0\n",
      "ia\n",
      "filceqey anvelexayxqçaña\n",
      "tbmantqie carmea\n",
      "çakia\n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 18\n",
      "Epoch 1/1\n",
      "15836/15836 [==============================] - 7s 449us/step - loss: 2.1137\n",
      "----- Generating with seed: \"batoul bayan beata b\"\n",
      "\n",
      "----- diversity: 0.5\n",
      "ania\n",
      "sanira\n",
      "sana\n",
      "sanana\n",
      "sandra\n",
      "ianxana\n",
      "sadoniya\n",
      "sa\n",
      "\n",
      "----- diversity: 1.0\n",
      "bzzcalrenwma\n",
      "çajcelisg\n",
      "sajuignamñoritr\n",
      "awauri\n",
      "aqic\n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 19\n",
      "Epoch 1/1\n",
      "15836/15836 [==============================] - 7s 438us/step - loss: 2.1130\n",
      "----- Generating with seed: \"ara balwinder barbar\"\n",
      "\n",
      "----- diversity: 0.5\n",
      "ia\n",
      "warmepna\n",
      "sarana\n",
      "sarsa\n",
      "salyosa\n",
      "sahia\n",
      "sataria\n",
      "bal\n",
      "\n",
      "----- diversity: 1.0\n",
      " kastça\n",
      "patrisvzea\n",
      "cfrospla iaa\n",
      "y\n",
      "waqzaça\n",
      "xizaxvan\n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 20\n",
      "Epoch 1/1\n",
      "15836/15836 [==============================] - 7s 460us/step - loss: 2.0805\n",
      "----- Generating with seed: \"e renee resurreccion\"\n",
      "\n",
      "----- diversity: 0.5\n",
      "\n",
      "resre\n",
      "reslina\n",
      "rene\n",
      "rekaha\n",
      "remen\n",
      "repivia\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-12-38af929f341d>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     30\u001b[0m                 \u001b[0mx\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mchar_indices\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mchar\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m1.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     31\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 32\u001b[0;31m             \u001b[0mpreds\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpredict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mverbose\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     33\u001b[0m             \u001b[0mnext_index\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msample\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpreds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdiversity\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     34\u001b[0m             \u001b[0mnext_char\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mindices_char\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mnext_index\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/models.py\u001b[0m in \u001b[0;36mpredict\u001b[0;34m(self, x, batch_size, verbose, steps)\u001b[0m\n\u001b[1;32m   1025\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuild\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1026\u001b[0m         return self.model.predict(x, batch_size=batch_size, verbose=verbose,\n\u001b[0;32m-> 1027\u001b[0;31m                                   steps=steps)\n\u001b[0m\u001b[1;32m   1028\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1029\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mpredict_on_batch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/engine/training.py\u001b[0m in \u001b[0;36mpredict\u001b[0;34m(self, x, batch_size, verbose, steps)\u001b[0m\n\u001b[1;32m   1798\u001b[0m         \u001b[0mf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpredict_function\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1799\u001b[0m         return self._predict_loop(f, ins, batch_size=batch_size,\n\u001b[0;32m-> 1800\u001b[0;31m                                   verbose=verbose, steps=steps)\n\u001b[0m\u001b[1;32m   1801\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1802\u001b[0m     def train_on_batch(self, x, y,\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/engine/training.py\u001b[0m in \u001b[0;36m_predict_loop\u001b[0;34m(self, f, ins, batch_size, verbose, steps)\u001b[0m\n\u001b[1;32m   1299\u001b[0m                     \u001b[0mins_batch\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mins_batch\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtoarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1300\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1301\u001b[0;31m                 \u001b[0mbatch_outs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mins_batch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1302\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch_outs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1303\u001b[0m                     \u001b[0mbatch_outs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mbatch_outs\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/backend/tensorflow_backend.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, inputs)\u001b[0m\n\u001b[1;32m   2473\u001b[0m         \u001b[0msession\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_session\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2474\u001b[0m         updated = session.run(fetches=fetches, feed_dict=feed_dict,\n\u001b[0;32m-> 2475\u001b[0;31m                               **self.session_kwargs)\n\u001b[0m\u001b[1;32m   2476\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mupdated\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutputs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2477\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    893\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    894\u001b[0m       result = self._run(None, fetches, feed_dict, options_ptr,\n\u001b[0;32m--> 895\u001b[0;31m                          run_metadata_ptr)\n\u001b[0m\u001b[1;32m    896\u001b[0m       \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    897\u001b[0m         \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, handle, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1079\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1080\u001b[0m           \u001b[0msubfeed_dtype\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msubfeed_t\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_numpy_dtype\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1081\u001b[0;31m           if isinstance(subfeed_val, int) and _convert_to_numpy_obj(\n\u001b[0m\u001b[1;32m   1082\u001b[0m               subfeed_dtype, subfeed_val) != subfeed_val:\n\u001b[1;32m   1083\u001b[0m             raise TypeError(\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "def sample(preds, temperature=1.0):\n",
    "    # helper function to sample an index from a probability array\n",
    "    preds = np.asarray(preds).astype('float64')\n",
    "    preds = np.log(preds) / temperature\n",
    "    exp_preds = np.exp(preds)\n",
    "    preds = exp_preds / np.sum(exp_preds)\n",
    "    probas = np.random.multinomial(1, preds, 1)\n",
    "    return np.argmax(probas)\n",
    "\n",
    "# train the model, output generated text after each iteration\n",
    "for iteration in range(1, 60):\n",
    "    print()\n",
    "    print('-' * 50)\n",
    "    print('Iteration', iteration)\n",
    "    model.fit(X, y, batch_size=256, epochs=1)\n",
    "\n",
    "    start_index = random.randint(0, len(text) - maxlen - 1)\n",
    "    generated = ''\n",
    "    sentence = text[start_index: start_index + maxlen]\n",
    "    generated += sentence\n",
    "    print('----- Generating with seed: \"' + sentence.replace(\"\\n\", \" \") + '\"')\n",
    "        \n",
    "    for diversity in [0.5, 1.0]:\n",
    "        print()\n",
    "        print('----- diversity:', diversity)\n",
    "        for i in range(50):\n",
    "            \n",
    "            x = np.zeros((1, maxlen, len(chars)))\n",
    "            for t, char in enumerate(sentence):\n",
    "                x[0, t, char_indices[char]] = 1.\n",
    "\n",
    "            preds = model.predict(x, verbose=0)[0]\n",
    "            next_index = sample(preds, diversity)\n",
    "            next_char = indices_char[next_index]\n",
    "\n",
    "            generated += next_char\n",
    "            sentence = sentence[1:] + next_char\n",
    "\n",
    "            sys.stdout.write(next_char)\n",
    "            sys.stdout.flush()\n",
    "        print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'sarsa' in text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "In the case of places, after several hours you can generate names such as:\n",
    "\n",
    "+ Alzinetes, torrent de les\n",
    "+ Alzinetes, vall de les\n",
    "+ **Alzinó, Mas d'**\n",
    "+ Alzinosa, collada de l'\n",
    "+ Alzinosa, font de l'\n",
    "\n",
    "-\n",
    "\n",
    "+ Benavent, roc de\n",
    "+ Benaviure, Cal\n",
    "+ **Benca**\n",
    "+ Bendiners, pla de\n",
    "+ Benedi, roc del\n",
    "\n",
    "-\n",
    "\n",
    "+ Fiola, la\n",
    "+ Fiola, puig de la\n",
    "+ **Fiper, Granja del**\n",
    "+ Firassa, Finca\n",
    "+ Firell\n",
    "\n",
    "-\n",
    "\n",
    "+ Regueret, lo\n",
    "+ Regueret, lo\n",
    "+ **Regueró**\n",
    "+ Reguerols, els\n",
    "+ Reguerons, els\n",
    "\n",
    "-\n",
    "\n",
    "+ Vallverdú, Mas de\n",
    "+ Vallverdú, serrat de\n",
    "+ **Vallvicamanyà**\n",
    "+ Vallvidrera\n",
    "+ Vallvidrera, riera de\n",
    "\n",
    "-\n",
    "\n",
    "+ Terraubella, Corral de\n",
    "+ Terraubes\n",
    "+ **Terravanca**\n",
    "+ Terrer Nou, Can\n",
    "+ Terrer Roig, lo\n",
    "\n",
    "where names in **bold** are generated and other names are the nearest neighbours (in the training dataset) of the generated name."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise\n",
    "\n",
    "The task here will be to be able to predict values for a timeseries: the history of 2 million minutes of a household's power consumption. We are going to use a **multi-layered LSTM** recurrent neural network to predict the last value of a sequence of values. Put another way, given 49 timesteps of consumption, what will be the 50th value?\n",
    "\n",
    "The initial file contains several different pieces of data. We will here focus on a single value : a house's Global_active_power history, minute by minute for almost 4 years. This means roughly 2 million points. \n",
    "\n",
    "Notes:\n",
    "\n",
    "+ Neural networks usually learn way better when data is pre-processed. However regarding time-series we do not want the network to learn on data too far from the real world. So here we'll keep it simple and simply center the data to have a 0 mean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading data... \n",
      "Data loaded from csv. Formatting...\n",
      "Shift :  1.1030964434554305\n",
      "Data  :  (1024590, 50)\n",
      "\n",
      "Data Loaded.\n"
     ]
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import numpy as np\n",
    "import time\n",
    "import csv\n",
    "from keras.layers.core import Dense, Activation, Dropout\n",
    "from keras.layers.recurrent import LSTM\n",
    "from keras.models import Sequential\n",
    "import zipfile \n",
    "\n",
    "np.random.seed(1234)\n",
    "\n",
    "def data_power_consumption(path_to_dataset,\n",
    "                           sequence_length=50,\n",
    "                           ratio=1.0):\n",
    "\n",
    "    max_values = ratio * 2049280\n",
    "    \n",
    "    zfile = zipfile.ZipFile(path_to_dataset)\n",
    "    zfile.extractall(\"data/\")\n",
    "    \n",
    "    with open('data/household_power_consumption.txt') as f:\n",
    "        data = csv.reader(f, delimiter=\";\")\n",
    "        power = []\n",
    "        nb_of_values = 0\n",
    "        for line in data:\n",
    "            try:\n",
    "                power.append(float(line[2]))\n",
    "                nb_of_values += 1\n",
    "            except ValueError:\n",
    "                pass\n",
    "            # 2049280.0 is the total number of valid values, i.e. ratio = 1.0\n",
    "            if nb_of_values >= max_values:\n",
    "                break\n",
    "\n",
    "    print(\"Data loaded from csv. Formatting...\")\n",
    "\n",
    "    result = []\n",
    "    for index in range(len(power) - sequence_length):\n",
    "        result.append(power[index: index + sequence_length])\n",
    "    result = np.array(result)  # shape (2049230, 50)\n",
    "\n",
    "    result_mean = result.mean()\n",
    "    result -= result_mean\n",
    "    print(\"Shift : \", result_mean)\n",
    "    print(\"Data  : \", result.shape)\n",
    "\n",
    "    row = int(round(0.9 * result.shape[0]))\n",
    "    train = result[:row, :]\n",
    "    np.random.shuffle(train)\n",
    "    X_train = train[:, :-1]\n",
    "    y_train = train[:, -1]\n",
    "    X_test = result[row:, :-1]\n",
    "    y_test = result[row:, -1]\n",
    "\n",
    "    X_train = np.reshape(X_train, (X_train.shape[0], X_train.shape[1], 1))\n",
    "    X_test = np.reshape(X_test, (X_test.shape[0], X_test.shape[1], 1))\n",
    "\n",
    "    return [X_train, y_train, X_test, y_test]\n",
    "\n",
    "global_start_time = time.time()\n",
    "ratio = 0.5\n",
    "sequence_length = 50\n",
    "path_to_dataset = 'data/household_power_consumption.zip'\n",
    "\n",
    "\n",
    "print('Loading data... ')\n",
    "X_train, y_train, X_test, y_test = data_power_consumption(\n",
    "            path_to_dataset, sequence_length, ratio)\n",
    "\n",
    "print('\\nData Loaded.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train=X_train[:20000,:,:]\n",
    "y_train=y_train[:20000]\n",
    "X_test=X_test[:10000]\n",
    "y_test=y_test[:10000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlUAAADFCAYAAAB9/wfYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJztnXl8VNXZx38zGdawJYAhCcqAyiIiSwDLHpCKuNRSt1aUV+xira217Vttq+2Lr13c2r61rVsrIIg7LlUBAWGAgOxLQAHZhi0LAUKSyZ7JvH+cnNwzd+6duXfmbjN5vp9PPjOZuXPPc8/6nOc85zmuM2cqQyAIgiAIgiASwm23AARBEARBEKkAKVUEQRAEQRAGQEoVQRAEQRCEAZBSRRAEQRAEYQCkVBEEQRAEQRgAKVUEQRAEQRAGQEoVQRAEQRCEAZBSRRAEQRAEYQCkVBEEQRAEQRiAx45EhwzpD6/Xa2oa1dXVSE9PNzUNQj9ULs6DysSZULk4DyoTZ2JFuRw7dgwHDvhjXmeLUuX1erF9+3ZT0/D5fMjPzzc1DUI/VC7Og8rEmVC5OA8qE2diRbmMGDFK03W0/EcQBEEQBGEApFQRBEEQBEEYAClVBEEQBEEQBkBKFUEQBEEQhAGQUkUQRHw0NwPr1wP19XZLQhAE4QhIqSIIIj6++AJYswZYvdpuSQiCIBwBKVUEQcRHMMheGxvtlYMgCMIhGKJUVVRcwL333o3x4/MwYcJobNu2xYjbEgSRDIRCdktAEAThCAwJ/vnoo49g2rTpmD9/MRoaGlBbW2PEbQmCcDIul90SEARBOIqELVWVlRXYvHkTZs+eAwBo3749unfvkbBgBEEkCWSpIgiCAGCAper48ePo2bMnHnzwfnzxxT4MHz4Cv//9UxHn8CxatACLFy8EAJSUFMHn8yWadFQCgYDpaRD6oXJxHvGWSfrhw+jt9yPgduNsZqbxgrVxqK04DyoTZ+KkcnGdOVOZ0DRz9+6dmDnzGnz88Urk5Y3Bo48+jK5du+JXv/qt6m9mzpxKZ/+1UahcnEfcZVJYCLz3HnDZZcBddxkuV1uH2orzoDJxJlad/bdq1bqY1yW8/JednYucnFzk5Y0BANx00zdRWLgn0dsSBOF0eHyqw4ftlYMgCMIhJKxUZWVlIScnF4cPHwIArF/vw8CBgxMWjCBiUl8vbesnrKepyW4JCIIgHIUhu//++MdncP/930NDQwP69fPiueeeN+K2BBGdP/0JuPRS4O677ZaEIAiCIIxRqoYNu0rTWiNBGM6RI3ZL0Ha56CL2mpZmrxwEQRAOgSKqEwQRH34/e+3b11YxCIIgnAIpVQRBxMeGDey1utpeOQiCIBwCKVUEQehHDPh59qx9chAEQTgIUqoIgtAPD6dAEARBtEJKFUEQ+qFz/wiCICIgpYogCP3QeX8EQRARkFJFEARBEARhAKRUEQShH7JUtS1WrgQ++shuKQjC8ZBSRRAEQURn0yZgxw67pSAIx0NKFUEQBEEQhAGQUkUQhH5o+Y8gCCICQ87+y8u7El26dIHbnQaPx0PnABIEQRAE0eYwRKkCgPfe+wQ9e/Y06nYEQTgZslQRBEFEQMt/RPJTUmK3BARBEARhjKXK5XLh9tu/CZfLhTlz5mLOnLkR1yxatACLFy8EAJSUFMHn8xmRtCqBQMD0NAj9GFkuXr8fAHDu/fdRNWSIIfdsi8RTJu7aWlzSkv/BDh1wktqa4TipD+Ntze8QeezCSWVCSDipXAxRqj766FNkZ+egrKwMt912My6/fCDGjZsQdo2obM2cORX5+flGJK2Kz+czPQ1CP4aWS0sj8o4YAYwbZ8w92yBxlUkgAGzZwt4PG4ZLqa0ZjqP6MN7WnCKPTTiqTNo6J08CFy4Aw4Y5qlwMWf7Lzs4BAPTu3RvXX38jdu6keCaEhTQ12S1B24b8qwiCsJpXXgGWLrVbiggSVqqqq6sRCFS1vvf51mAILcUQVkJKlfWIitSFC/bJQRAE4SASXv4rKzuDe+6ZDQAIBpvwrW/dhmnTvp6wYARBJAmnTtktAUEQhCNIWKnyevvD59tkhCwEER8ul90StD1oyY8wi6Ym5i/Tv7/dkhCEbiikApH8kFJFEKnDp58Cr74KlJZak96xY8C2bdakRaQ8hgX/JAiCsJRjx4DevYEuXeyWhDCSM2fYa22tNem9+ip7HTPGmvSIlIYsVUTyQ5Yq63HC8t+rrwLz59stBWE0vG5RuyaSEFKqiOSHOl/r2L0bePddu6WQOH/ebgkIs6B2TSQhpFQRyU8qdr4lJcBXX9ktRSQffADs2+cMSxWRmlDdIpIYUqqI5CcVlaoXXwRef91uKQjCerhS5abhiUg+qNYSBEEQzqG5mb2SUkUkIVRrieREDDiZipYqp0NLNIRZcKWK2jWRhJBSRSQnhYXSe+p8CSJ1IKWKSGJIqSKSE7M73MZGssZEg/KGMAtet6iOEVpwWD0hpYpIfoxWsBoagD/8AfjsM2PvSxiHwzpSwkC4pYogtOCwvoCUKoKQU1/PXnfvtlcOgmiLVFTYLQGRTKSqUhUMBjFt2kTMnn2bUbckCHXMXP7ju44c1lgdBeUNYRZNTeyV6hihBYfVE8OUqpdffgEDBw406nYEER1RqTKrUTmssRKEbl58Edi0yW4piGRn3jzgo4/sliIpMESpKio6jdWrP8Xs2f9lxO0IIjZmWqr4vUmpUsfuvLE7/WShpARYudJuKfRBu/6cyY4ddkugjMP6Ao8RN3nssV/hd7/7XwQCAdVrFi1agMWLFwIASkqK4PP5jEhalUAgYHoahH6MKpeMPXvQ3e8HAJzfsQOV3A/KANz19bjE70dzu3Y4YVMd8rY8m9+C9PWUCZfr9MaNyG15X5+ZiWKr86m5OSyPuhw8iF6ff47js2cj1K6dtbKYhBFtxah6ZGV97Of3wxUKoWjjRjT07m16enqera2OK1aWv1a4TMd9PgTq6x1TLgkrVStXLkevXr0wfPhIbNy4QfW6OXPmYs6cuQCAmTOnIj8/P9Gko+Lz+UxPg9CPYeXS2AhUVwMAvKNGAePHJ35PTm0t8PnngMeDAXbVoZYOwmtB+rrKhMs1ahSwaxf7LDcXg6zOp+ZmYP16Jkt+PrBnD+D1wpuXB2RmWiuLSRjSVoyqRxbWR6xbB4RC8I4fD+Tmmp+ejmdrs+OKleWvFS7T5Mnwff65Y8olYaVq69Yt+PTT5fjss1Woq6tDIFCF++//Hl544d9GyEcQyljhUxUMmnPfVOD99+2WIBxask0dqAwJPTgsBEfCPlWPPTYPe/YcwI4d+/DyywswceJkUqiI5IaCD8YmylK/JcjLhvxwUg9qf0QSQnGqiOSHOl/rcZoSw+Vx2KyVIIi2hSGO6pwJEyZhwoRJRt6SIAgn4jSliivWTpOLIIg2BVmqiOTHaEsVWb5i46augzAZaodEEkI9I0EQ+rHbIiQfcMlSRRCEAyClikhOxEGVZrTWQ5YqwmyoXRNacFg9oZ6RIOQ4rJE6EqdahJwqF0EQbQJSqojkhCxV9iJaqpyQ/06QgSCINg8pVURyQoE57cWK4KvRUEuTLFWpAynKRBJCShWRnIjxiKjztR6n+VRRHUhuFi0CPv7YbimIZKSszG4JwnBYz0gQGhEH0YMHgaoqc+5NKCNahIqL7ZODSA2OHgW2b7dbCiIZWbAAHUpL7ZaiFVKqiOREVHyKi4EFC+yTpS3iVEsVKcSpA5UloRGPkZPqBHFYz0gQGpF3uOXl9sjRVhk82N70acAlCMKBGHpMDUFYhhmD6tmzQKdONGBroUsXuyUIh8qMIAgHQJYqo6mtpZ1pVmDGIPqPf7A/giDshxRlQiNO2vObsFJVV1eHGTPykZ8/HpMmjcVTT/3BCLnMpbQUaGw0595PPQV88IE59yYkevY05761tebcVw80mOiHfKoIgnAACStVHTp0wNKlH8Pn24Q1azZi7drV2L59qxGymUNjI/DCC8C775qXxt695t2bYHTvbt69aWCOn1OngOPHzU+Hyohoy4RCbLekEyaBRBgJK1UulwtdWvwrGhsb0djYBJeTA/A1NbFXMzp+6uiJts6//23PTkze9vbupXaYKlA5qlNWxuJ6vfee3ZIQMgxxVA8Gg5g+fTKOHTuKe+/9PvLyxkRcs2jRAixevBAAUFJSBJ/PZ0TSqgQCAcU03PX1uMTvR3P79jhhtAyhELx+PwDAb/LzJStq5aKX9MOH0bslrwEg5HLheIL35WV3csMGXGxnOVpcj/SUCZfr3I4d6CnkP5fTKrldDQ3oJ6R18ZEjSKutBRYuxNlDhxC4/HJT07cCI9qKUeVhdrmK9+fvSzZvRt2JE6akp5Z2LIzqvxLFU1GBvn4/Gs+dw+ncXNPTM6L8u3z1FeqystBk0CqDV+h/0hxSLoBBSlVaWhrWrt2IiooLuOee2di//0sMGXJF2DVz5szFnDlzAQAzZ05Ffn6+EUmr4vP5lNM4dAjwegEAA4yWIRQC1q0DAHhNfr5kRbVc9JKRwZaaOG43+id6X64YTJzYGojQlnJsbra0HukqE55Ho0Yx38QWWuXk35std309sGmTlNa2bUB1Nft/yBBg4kRz07cAQ9qKUeVhdrmK9+fvv/Y1YMAAc9JTSzvmpQb1X4ly/jywaxeQkYHLrZDHiPL3+ZiF7dFHjZCoVSYAqG1sxEgnlAsM3v3XvXsPTJgwCWvWrDbytsZCUXsJIvmJtjR04YJ1chDmQct/6nAXm2TLI7M2iDmIhJWqs2fPoqKCdWK1tbVYt24tLney6d1Mf69kq+DJDOW1s2ixEjkC7jdJEKkK7XZ1LAkv/5WWluAnP/khgsEgQqFmfOMbs3DttTONkI0gtCMesJwo1FHpZ88eYPx4+9KnMtPGvn3ApZeyILdOh8qUSEISVqqGDr0Sa9YUGCELQRDJipN3/BKMykoWSsbrBe65x25pCCMgxdNxUER1IjlJ5c4kGZ/NaqVKnkc1Ndamn8ycO2e3BESi0PKfYyGlykiogifOV18B//d/5BdDEEaTlsZeE2lbRi6zx4L609hQHjEclA+kVBHOYtkytnurqiqx+zz+OPD++/H91kENNGlw0vKfk2RxIonswCovN04OIn7IUhWOg9p8yipV3fbuBd56C/jnP4F586gSphqxyjEUYs7ThDk4rR05ecex00jEUnX+vHFyEMbitDYp4mTZDCZllarMHTuA/ftZsDEAKC62VyAiEiU/GL41v7JS/XcnTwIrVpgjkxKBAFPMDx60Lk2REyecX3/tnil27Sq9t1sWp6I0sB08CASD2u+xZIlx8sSiDQ3EupEbCU6cYNb5kyfNT7u62nH9kZNqSsoqVRFwfwLCGRw8CDz9NCAcNQAAyMxkr9GWKF55BWhoME20iM6cO/Zu3GhemtHSnz8feOkla9JOFuR5JP5PSpU2TpwA3ngDeOKJ8M///vfIIMmdO1snF6EdXu+5MrV8uflpPv+84/ojJ7X4tqNUdehgfho0s9IOP9D69Onwz9u3Z69OGhg7dmSvdCK8c6G2px+1iea5c+ywXhGr4lqZOVlKZbp1Y69FReanFU+g3zbUPtuOUsWxsnBPnAAOH7YuvWRETXkyupy++CK6k21dnfp37pZm0gaOWIgbJynBX3zRpjpxzcjzxGPI0a/G8tRT0nsqQ3Xky39t3JLopJpCSpWZzJ8PvPaadeklE1Z3mO+8w8zWaojnxanJZkeYB1LktCGWWV2dff5vqYpV7VWPf5fRJKMSJ5fZbDeXZMwji2l7ShXhbMxstNEUFLdKU9i7V5IpEDBeplg49QBwuzvXaD5VAPDmm9bJkkrYXa6ENuSWKqvKLd502lC9ajtKlRWF2oYqjulYnZeiUvXZZ9L7pUutl0VMz86Zu16sDAwph9qefpyeZ06XzwnY2TcRiiSsVJ0+fQqzZt2AiRPHYNKksXj55ShLLE7AqkrxwQfWpEMYg6hU7d9vnxxyfD67JdBOogFbCXOhATE6Ts8fv1+yltslq1PzyEE+nQkrVR6PB48//gcUFGzD8uWfYf78f+HgwQNGyGYsVptJd++2Jp1UxU5LlRw7OxKnHtejJJedO7ec2tk7Ba35o3adHfnL06yutmaiY0WMp0RYuBD497/DP0sWS1Ubap8JK1VZWX1w1VUjAABdunTFwIGDUFxswbZOgjCSaDOdNtQhaObIkcjPrFyqjOVTRURy6FD4/8mSZwsXstMxou3QNYJkWGoXN9QAdFKIAzF0T+2JE8exd28h8vJGR3y3aNECLF68EABQUlIEn8nLGjkNDfALgSVPFRSgqVs3ZO3fj04tsTz8BsvgamxEP3kwSxPSSWYCgQB8Ph8y9uxBd78f53ftQqVg4cg+eBAdyspQunkzahVirrQ/exY5CnkMhOezt+Ua/pn8fzlp1dW4WOW+RZs2taZpRVm6mposrUe8TLTA87EmGERnYWZ/bscO1J88aVk+pdXWtpaX3+dD7y++QLosz5K93ekpF0VCIaner10L76JFrV/5fT60P3cusryamxXbSu6hQ2jXsvRkRr56Fer7ma1bUVNaCu+2bQCAU6tXo6lLF9PSLtmyBXUnTkS9NuEySQCxXHjZhVwuHPf50OnUKWT5/Qi53ThuYvkc9/la+yY99cAVDMb1Oy0yAUBdr162lYscw5SqQCCAe++9G0888SS6du0W8f2cOXMxZ85cAMDMmVORn59vVNKK+BcuhNfrbf3fO2EC0LMnix3VEmDSa7QMDQ2KUbcNTyeJ8fl8rOzr64GaGnhHjQLGjZMuOHwYSE+H9+qrgUGDIm8wbx4glKtIWD5zZYp/Jv9fTkUF0NJ5R9x33Di2CzDa742koQEoKGDvO3VqDTpqVtqtZaLtYvY6dGjY9m1vXh7Qty9QWMj+NzufAgFgyxYprTNnIi5J9nanq1yUCIWAdesAAN4pU4D161u/8ubns6NG5PU6GGy9Liz/du9utZJ48/OBlSuBgQNV26JuFAZE75gxwBVXSG13/HjpxAUj4ff/2teA/v1jXJpgmSSC2IfxsnO70T8/n1khDx8G0tLY/2alPW5c6xinq301NgIbNuj/nQaZAKCmqQmjHNLeDdn919jYiHvvvQu33HI7brzxG0bckkh1YvluGGnOTmbTuFMPCnZa4MhkLmMrmD/f2Ptt2sSW5QhrUFvutnr5r2XSpBtRvr172aTaSFLJUT0UCuGhhx7AwIGDcP/9PzZCJnOgtWciXux0BnVQZxGGUp5YmU/kU6UP+XFQAOVZMrFnj/LnVrcDIzbOLF2a0nHkElaqtmzZjHfeeRMbNqzH1KkTMHXqBKxe/akRsiUf1EmZS7xb9hMtFypXZ2NHUNZkIJ5666S6brXC4KRnlyM/b8/Jsiohl/fsWXvksICEbfhf+9o4nDlTaYQsRFtEz9l/Cn4zKQlZqvTx7rvWHJjelkm2QTxeysuBV14B5s5lPrhEOOfPS+9DIe39k7z+pHBMu7YXUb2tdA5OJ55ySE83L61o11CdicRJeeL3O0ueZCHZYg5Zke6+fczyuXOn+WkZiR3xF6nNKeIwb1MLoIqQHDjBEuLx2GMpEp9TnBk6Cbvbkd3pE+ZjRxnzHa1Or19Ol0+O2fI6yKLf9ixVyZ4GYR1Dh7KtwHZGNY8RN8dyeOfF6/rolph03bvbW/+p7RmDlnx8/PHIz5qbgTVrIn1/khWn1SenyQPok8mJ8ptEyipVzU7b8k3oI5FGaJSDK99x8+WX8csSD8nUAQ0ezF6FmFW2kAx51thob9R5M+919CiLb/Xxx8alqVeGRFm0iMWrcyKx+jQ7jxEy+tokJ2WVKpeTzrAi1JGbbZ3o29TYaE+6TkRuqRKxs20ZHffGDP7wB+DFF+2WQsLI8mpuZq9GKo121KeWgLKEBmgsVSRllarWRs4hR3VnYVQ5TJoU+97i/2pLeU6qF06SRQ25jDwCvF3pO/0wXE5Zmd0SRCfeMAw1NcbLopROW8UJlqlESDZ5EyA1lapQSN1SRSQHWi0hHTvaI0sqpacHuaWK+9AcO+ZsuYnEiFa2W7YAH3xgTDpdu0pH35CPnkQsedr48p+TSis1HY+UDlYkR/XUZMQIYNWq6D49Yrkke0BEp8DzhAJvJi9G1Ot588L/T3QXFrW1xLDzVAOjrk1yUtNSNXAge3UrPF4yFG5JSeshuilPor5vHg8wZkx48Eejy5gsVZFwGSlAonPRW4/KytgOPrvRExC4rSA++/79xp/lSBhGaipVubnw33MPcMkldksSHy++CPzrX3ZLYS51dex17Vrtv7ErWGFb7szl8AGP+6ZddhmbvAwa5NxZst00NAALFtgtRWwWLAA2b7ZXBjFKdzKVsZXs22e3BAyyVCmSmst/HHHGk2yO6k4N+mgU3LFVbdeWnnJyuSJncmrEYxmTb3owm2Soo8eOsVeXC7joIntlcSoXLjDrT0MDcPy43dJEIq9nSps4/H7g7be174A1IggjWaoiSVafqnXrWP85cqT58jiEtqNUWUFbbvR2oVTG776b+H1HjgR27WLvrVaqnIw8v10uSaml+h/OP//JlJHbb7dbEm0old/Gjdbs7HMaTqnLa9cChw8Dl14qfeaUcU1LHvGVCL75oA1gyPLfT3/6I1xxxQBMnny1EbczF6c0FkI/Vi7/DRggvSdLlToOOh7CcXDrjl15pLceKV3ftau+e5SW6rteSQZa/pNYtw44fTo8L+SxwBobpQmgU3F6KBEDMUSp+va3Z+PNN98z4lbGorT8Rzgbvcu0oqXEbJkI51iqkqlM1JSqxkbmcHzqlPky9OkT+xqlyYPSZh+999ALLf9FR3501caNwKFD1suhpzzMPknAQZM7Q5SqceMmoEePDCNuZT7UMJ2B0eUQDLLjZGLFt9KjrOn9jVEkQx3NSJL27gTUFJPSUjZALl9uTrpiPVLy0XRiQEnRUuVkv9L9+1k4CatkFH1Px44N/66qyhoZ5OipL59+mtjvkwjLfKoWLVqAxYsXAgBKSorgU4olZSCBQAD79+9Hp6IiAEDRxo1o6N0b2QcPokNLQ/AbLIO7thaX+P0Rn+tNx9tyj7pf/Qrnxo1DY/fuiQungY6nT8MVDKLWxF2TgUAAPp8PWULZiPnDy+fstm0IlJeH/bZDaSmyZfl7fP16ZBQWotuhQ8DTT6Pk2mvRR7jG7/PBXV/fWi7H161DqH37CLk8FRXoK/yubNs2dKmvR6fiYlQDSG/5zug6o4QnEAiThWNW2rxMtHDJ0aNwNzWh8dw5uBsacNLnQ/ZXX6G5QwdUuN2teW92PnmqqhTzSMSKsooGb8cl27ZF1MmOJSUIAcj2+1EfCKBYQVY95aKEq6EB/VTy6MTKleizYgXaC32hl28+EOTM3LsX3WLks0hTly44FYfMPK+CHTqgrroa6SdOAH4//G5363dFGzeioVcv3ffWmrZIZadOON+pU8TnvEx6+3xI9/tR9uGHqO7f33CZImQTZDxfWIhMUWbhfcjlwnGj630opJhHJzdsQFAhj0SUfsfxr12r3xKqcu+6rCzTdQqtWKZUzZkzF3PmzAUAzJw5Ffn5+aam5/P5MGTIEKBlAPWOHw/k5gIHDrA4UAC8RstQXa14dlTMdIJBoKgIuPhi9r9YORobAZPzqhUeyG/OHNOS8Pl8rOxPnpTKRny+gweB4mJ4R49mgT1FTpyI2NnnnTKF5VFLXC/vVVcBX30lfZ+fzxxtP/9cul6MacU5dy7ML8E7diwwYwbw8sthO1cMrzNKlJcD27cDPXqwHWQmp91aJlr4/HM2a87IAOrqcGl+Plt66NQJuPpqVn4mytpKeTmwY0fUSywpq2i0tGPv+PFhyzPeESOA//s/oHdv5sCbk4NBCrLqKhcl6uuBTZsUv/LW1gLdurE/tOSVbFDy5uezPk3nuYqXxSMzT7tjR+DKK1sHW++kSeH5mJOj/95a0xYZNkyx320tkxYfIe/o0exas5DLduml8F56KeuvVOhvdL0PhZhvlwzvxInRfe5CIeW85b/Pz09IqRLvXR0KIc/u9t5CasapsopQCNi2TYq5FItAgCku8i3/r70GvPIK+15uEm1rh/ly65SeA3vNipfCo7TbtftPix+MXTQ3S8s0Zvu0KZFMSwfyaP+8vzDbeVfMo9zc8O+0Bhe2w1dFzK8nnpDeW1nmYv7U1LB+XolElIJ4EJdHrUxTz+eJ3jfJaTtKlRlxqo4fBz75RHsU4jNn2OvWreGf8x0zzc2R8ulpQDU1xhwbUlUF/PWv1u/YKCvTrqByXC5zTpbnDtiAfbv/5B02/3zXLuDNN62ViaO0M6u2lvmWpGgnmTByJ90WS3krZuWbeN9oxzjZjdzn0eWyXlmRI8r07rusnz97Vvl7Kzl61D6n7CuuCP9fzIOvvgrPH/n3ZpNqjur33TcX118/HYcPH8Lw4YOxZMkiI27rfPhsprKSvcbS6HlHIR+kozlS88ry1VdAcXF0ef78Z+DZZ6Nfo4U9e4CKCmDnzsTvpQcjnD6VnG+jNe7aWvV8tUupkqfPOXCAvX74IXsvPldlZaQSXF5uTfTlc+ckC6PTOHRIcenCUuRK1ZEj7DWGP4qhyA8ed4oCfPIk8PjjkZ/bHRvu6FHpPV+Cd8rAbZccWVnKn4dCwOuvAy+9FP55i8+sKk6pgwZjiE/VSy8lwREMToio7mnJbnnk4miycafq119nnfAjj6jf36htq1w+Jd8jMxGf/8MPgcGDwwcepfyRdzB6y3fBAmZBlPtqiJYqs7cDy1F7Bvkun+ZmyQLxl7+wV/GA2w8+YNbUK64wbuYvKpryvDfDYhiLkSOBIUNY+1BiyRL2OmUKey0oYD47PXoYJ8Phw0ypHDNG+Xt5e9+7l72mp5t7xqdYj+IdiM0ewPUo/Vb23WLAU97+7baecZyi3PHy4Aqw3FVlxQptv08xHFJLUhytRwyEQpEzNLEBWXXIshXB97T4TLU4Pcdk0KDo6ShZAkMh5nTNl2SVZlV2Wap4enJFT205UA1+NIoZ8otp8wPMhQ0CpsPT799fSj8WdXXA6tXMSdxIXnuNLQ+pobYkb/bgKJZRVlZ4QFvPox0XAAAgAElEQVStbdtsRVleN+vrnRvB3W7rWcumAtuWctUmsGp1Scyv9PTI7+0KBWEybU+psvOMJP7a0BBe4cTvd+8O/62djup2Ox9r7cRmzNCf1pkzyrFTOHb6VPH0MjPDLU/xWuXMKEfRYfa229ir04+iECclSufcmUXLztNWBg9mr3y51qx2xuvRkCFsJ1vfvtJ3TolRpZRuu3bar7UCpX6Ay2Kl1WjQIKZQOcViFkupimUptdq9xCIcUjomwX0XAGsaZKzKxV/LyoBXX1X+Xj7jtUOpcsqxGtF8zzhud3gnrPQbLcuGSti1/Ke23KDXUqX3Oi0oWTGVfAWdaNpfJPh6yp3FzYDnS5Tt76bCQyEMGRJ/3RGtW1YhnnNnhaLepYu26+y0VM2bx1whmpvZEraVxKoravkSqz9I0Z3tqa1UOQUljZ4vzVy4IFUuJUdtecVTG+AXmbA5QGk2a+YMX56eVmVGVKri+Y0STrBUGaVUmSG/aPVR24BhBXomAGY70y9dqu06q6xEfCct90vs3Vu/DGYrx0ouDWI9t8Iqk54uWQ+VsHvDCsftZuVhl5IuR4+lSinvjJysOsXPDG1JqbLDUZ0HI5M79ImsXi29f+21yEBycqVKbYYt7lZJFDWfqnXrgN//XncwQM3EUqrUyk5UkGINFmrhCpRwmlLlhOU/UQYxTpXLFV5eZudZPM80caLxcojs3ct2znIUIvdbCu87ePu48krJ8hNP3cnMNEw0AGxC+cUXkZ+L9V70H7Lb+mm3UmX38wNsoxR3t9DjU2V33llIaitVRjr0lZVJW9q1wjtVXumUNHP57he+M4jT2Gh9Y+KDZUEBsGyZ9Dn396quTjwNLc+kdSYTq5yjOcXPmAFcfnnk96LyYHWHwK2B8udykqVKjttt7/LftGnALbeofx8IWGP12LlTqreiD5MID8HCMcv6y+/LlSqXS1rO02Kp2rUr7AgUTJtmrHzCaQFhqClVZiH6ByoRzafKSuT97ujR1svQqVNkFHW1/iWWpcpKv0YLSW2l6uc/B775zfDP4m0M//xn7ICL4o6ka6+NtPjEM7g1NMRWLoxemxY7GDFQKe/sjOhQ5PfYvTty+SSeRvfxx9HTkdOxIzB+PHsvzsTLy6V8MMsypwZ3YJbHMXKqpYq/j3fQaWxUH2C1Mnkys/Kq7QScP9+agfD4cakOygPZ8iUmeVw0M5ZzSkqkcBIeIXIOLzN521La/v7hh+H/WxViRW35z+zyGzMGuP569e/lE16rke9yttphXd7/BAKsjqjt1oxlqXKC5c0EUlupSk8H+GHEZiz/8Yoir2xXXskGarlSJd/Zp0XJKi9nS26c06cjr5ErVWfOAO+/b7yFwsgdL3LZRIsYRz4oxVt2SiEVRPr3Z46gomXh1CnpOXmkYKvW7ZcvZ69yB9qKivB8O3dOmyXTKktVQ0N4muvWMZlj8Yc/sDAH8ZSvvEzuuEN6L+6cNCKwrFb4GZLyyZCo3OTkAN/9rnkyfPSR9F7Jf/DkyfD/1Y5hEVHaFp8IYn0Rcbulc1CtUBy4peqGG4CxYyOXbXkd27oV7tpaVrf5qoWZbUveHuRO+1b1R/wZeRvi6e7fz+r6Z58p/07e78qtjim6JJjaShUgVQDui2RkQcayQPG0lyxhjfDLL8O/X7xY+Xff+x7w9a9LAQtFlJQPOW+/zXw7nOLQqITY4MrKlDvYjAzz5YhmeZF3WkZ0Yjt3Aps3q38v5ot8EFu5Evjf/5X+f+UV4MknwyNSv/km6/zEyPp/+Ytxkwm1/HK7wxWo8+eBtWvVg3KK13H0xGFTe55oy0VWzoyDwUiLlLw8L76YDZRGKytAuGLQubP0Pt4ll86dgezsxGSSoxbbKy0NuOsu4Ec/Mja9aIh1+WtfY6/LlgF/+lNYP5p+4kR4/DwrY/m1ayfFqgKsU6r4Bg8+fvF0ubIuLhGLu63ly3/t20vtMz3d0Lxzks0r9ZUqDjdvi7PHRHcf8NnKwYPRg0eeOqW8dHjsmPJ9+/YFJkyI35+AW1bi/X2sxmpEYxaPzVA7yXzHjvjvzwcAtZAKauEZxPdmdFr/+Q+ri2oWHCW5ovlOyOvwgQPM0iUPOGl24Fi1MxtjneUotgujwhxMnmzMfRLh0CHp/R13AL/5jbJCmp5uznE14pEiYlu75BL995ozB3j4YePbg1obSEtjS40XXRSept5zQbUib3N8uX/r1oil/3bnz4f3V2YqVUqTdbWNIvffz96LSpdRcAsrd1Dn9Ykv2YuK+htvsEneM89Elq/LBfz2t8yC3LmzoXlXq+a/aAOpr1TJOwKxolZUMGdssQME2PEea9fGvrdYmV5+OTJNvZ1Qfj7wy19K/ytZb3r10n4/T5ynEKnJbWQHwh1mR4xQv0a+bBEtfflhn1deGT39w4fZq6h8iM89dWrkYBdvfirx179GtxpwPy8AuOwyffeW12fA2N2hWlBbGhcJhaSI9oBxYUHMGnz1IFqJhwxhs3S1AdGMgblnT/b60EPhn196aaSfqRrjx7MBUIxV9d//bYh4UVFrZ2+8YU56ckf1KBHdux08CJSWSh+Yeeh8rPh64vusLNaXmrlcyifp3KVm/372KvZj/FQFpc1M0XwwE6TGQYGHU1+pklcycWYfCLCQBtyhk7N7t7ZDWMW4L0D8EcE5Y8eGLwXwQV0c3JWWxIzulMWBTlw6MMMvzeNRtxjqWW6YOVN6P3x4uD+bkk8VP35DXKIRG316emTdMWInkmgpiDb4i07BRmzNVzsqRS9qnXpubvh1WuqI/JrrrktMHo5aflm5/LdqVeRnSrKapVRxlPyprrySWYHk/Rfn1luBH/yAbbaRozVQZiLYdQwLh8cQlCP2hQBTLkQFy2hiWaqUPjOjLsl9abmlSu9Kj6hoGVzvQ06JMg+DlKo1a1Zh3LhRGDt2OJ577i9G3NI45A1UPEg1Xk25uJg1vFAo+sxA3ghFeAXlSxU33RR5/bhxbKB58EHpMz0VMd5Kyx1tO3TQZxnTgyhb//7h311xBdu2q0f+9HQ2U7vvPmDWrNjX87IX60e0rdw8flh5eWJnk8WKu6O0GaB/f2DSJMnXQ34sjxafHDPO5RPbTyKbCtQG92hEu78dUcD1YralKto9PR7mr/TAA8APf8g+4/6bo0czpUt+7qTRyAOxduggDdZWK1VySxWfxD70kBTbrG9f5ufKGTqUKVVmRgXnZXj11dJnYh9lVxw9cTlZy2rM8OHsVVyi44FME0FtgmczCa9nBINBPPLIL/DOOx8iJycX116bjxkzrsegQVEi1FqJvIEOHiz5bsTjUxUKAS+9xN5PmMAqR15epBM6wBpeYyOrTGvXMmWhsZEFvBs/njmqjxvHBkslBcztlgbSBx8E/vWv6BXx+uvZtuAdO9i27kQrbUaGdn+keAmFJHP/T37COiq3W3k3WLR03e7wZY1Yh0IrKS/8vccjvb/xRuYzx8vnb39jMv7sZ7GfLVq60WST43IB11zD3l93Hftdjx6sXtXUsKWed99l/lR9+7KwAhkZ7GzDAQOAwsLI2DLxIuaXGG9pyJDw4zO0dPT8+S+/3NhllAEDmA9TeTnwwgvss3btzLdUdezI2vihQ9oPizXbUhWLPn2kXZJTp1qXbmFh+P9z50pHdymFgLCSO+5gbal7dzaZCYVYX9+pE85OmADv7bcz36UlS8z1VeS+saLFXswPJZ8lKyxVaWnAj3/MNkONGgW8+CLzPbv1Vua0fvnlQL9+Uh/QqRObCIpjHF/+Cwbj91/t1o3J9sADkedr2kjCStXOndvRv/8AeL3M2jBr1i1YseITZypVR4+Gd97xaPjyQVFp8Oafud2s0gHhW735Z+KW71hkZjKrUbRGw3es8U4p0QaWlqZ+8HOiKCkX7dtLshvZQSilpaRU8Vmg2KmPHs3+li+XfiPvzKqq4lNa4n0+l4spMYCU7re/HXkdt64dORL7WJ5EGTlSWanSc8aikQNC+/bMz2TePLYUx5d79bBnDwtN8otfSPkcDLIB94orlOOGyWfg48ZJ78VJnJrf5Y4dLCDwf/1XdNlKS5l1sF+/6Nc5aAYfhtjGvvENptzF64uaKHJLVWam5NPYoUOYhSpw+eWST1G7dpGBXI2EW3+5fxxPE2CTJ7WQFFbQq5c00fv1r6XP1XxZ5UYDrlQ98QSzxInuG3oYMMC6+GkaSVipKikpRm6uZNbLzs7Bzp3bI65btGgBFi9e2PKbIvjUdnwZRCAQgM/ng6eqCn35lk9xOzqA0s8/R1bLd35BHq/wmaeiAunHjyODf7Z2bev3FZ07o+uxYwi43Uj3+1G8YQP6+v04u307Alri8+ikz8GDCLndKJXlXVptLS72+3Fuxw5UVVcj/fBh9Pb7cWrDBjTp2A3iFbfGAqirroaruRnFLen1PXwYnpoanC4oQKO4jKoDXi7d9u5Fpt+Pqvbt0VBUhJ5+P04WFCDYYnrve+QI6gIBnBXS6XTyZGt5cfwq9ajbl18i0+/HiXXr4K6vb60DJzdsQLBzZ/Q9dAie6mpUeTw415JGRmEhuvv9CHbqhJOy+2YWFqLL0aNwt3RkPN3Ox47honXrUDxzJurFHVcq9DlwAB1bLKWnNmxAk8xHxRUMop/fj/Ldu1FhkFn/4qNHUdPYiHMq/jC8TGLR7sIF5IrHsEDKB09FhdTOAJStXYvefj8az57FaZV782et6NoV3f1+nN+xA5UaA622Ky9Hrt+PM9u2oSaGlSujsBDdjh5FZfv26N4iY/Hnn6M+xrJj1qpV6HT6NEo/+AC1LXGTeuzejR67d+PMtGmoafGP4+2m2eNBAEC634+0luc427cvAi3P32P/fvRoubaupgYlPh967tuHTsXFONVyjXfhQgCAv1+/qOXSet099yh+3/XLL9HT78eJ9evRLC7XGIC8n5Cj1iZFuu/d29qnFh07hobKSvRo1w49/H4c//xzhFqUrl5ffIEuQnpa7q2XvocOoa6yEmc1HMMjlknv/fvRrrwcRSaNZby/K9q8GQ1HjgAAepWWoovfj7PZ2ej8xRfo3BK30M/r0unTrXXJKNpduIBcvx9l27ej2qAwPX1a3BE6lpUBfj/8ceyA7Xv4MOqqq3G2Rw/NfZgVGLidKTpz5szFnDlzAQAzZ05FPj8XzyR8Ph9Lo7JSdWu+d/To1l1RXlEe3sENHMji7AQCrYHXvDk5UhC24cPZ7HP4cCAtDZdOnAhs387uO3Kk8Q917BjgdmOIPO8CAWDLFnjz8tjyX2YmcOoUvBMmhM9yYiGvlP36AfX1GMTT27EDqKpi943HDwZCuXg8bIfUsGFsllpcDO+kSZIj7J49zKlbfNavvmJWFwGvWj3q2BE4cwbeyZPZEtnOnez6SZOY1WHnTlY3rrpKSqOhgV3bowculd+3ro7NrFoGy9Z06+oArxfeSy4Jt0rIWbaMPdugQa0+Cd7qambxEH2AmpqADRvgHTmSLT0YwY4dzCSvkletZRKLefMiAhC25kNZmeSLB8Db0MCuzcjA5Wr3bmwENmxgbaWqCt5Ro8J3PUajtBTYswfesWMjd37KCQaZk+zw4a3+cN5x49SPkeGUlADt2rH2zC2DlZXAhQvwDh3Klv0Bqd20a8fScLtbl4W8eXlSXzB5sjSx69cPg3n/1LkzLuN5xPueKVPgW7eOlcvRo6xuiof+8uvEvD1yhOXL+PFsyYXX/2i+nfEQY/BSbZMibnfrDkkvP15oyhQgFIJX9BsqLw+z8EW9d1UVc8247bbYFjyR3bvZ9RrkDmsrZ88CJSUYaNZYdvAgcOQI62+5j1unTsDy5aze9+3bGrTVy+tSx45SXTKKsjJg9254x4yJvataK7xOt2wI0lRn5OzaxfrO/HztfZgFJOyo3qdPNk6fPtX6f3FxEbKzTXZy1EM0p8dYgfDeeCNy19Rbb0nv1Zb/zEK+JNbYGO4grMV8Xl0d6WhdU6PsX6bmTBjtWQMB/WcDqvk3hUJsd+bzz+u7n3gvPT5VSst/sVC6T1NTpE/N1q3AmjXhn+3fH16f7CIUYgq7mFeFhSzSudZgkb16sdAPXEHkfot1dewoC7Ge7tvH/L04amV15owx4RG0OMWeORPpl6F2sHU0oi1bi/fh/jJA7Da2aFHsI7IA5qO5cqU2Oe0gFGKTGR6VHAiPKeZyJRYSoLiY9T/RguuqyRXPkmMiLgpVVeF1AGD9ubikp9S3jB7NNuIMG2b+AeHR5EgUl0sac7Tct7k50n/NTl/EKCSsVI0cmYejR4/i+HE/Ghoa8P77SzFjRpTzk6wmWoHF2rkRa/uwWqGa5RMgb8QvvMAiZ8vliKZQPPMMi5HECYWAp59mfidKB/hG80dS4tlngZaliVbWrYtUKOT3lMOftaBACvEQbyPS6lMlOqrrvbd4n5deAv78Z22/F5e7QiFpl57RHRiXMxBgA6+gRHfbv585CYtWwGXLIpX2WGncdRcLFClSW8tmlJs2SZ+9+y5TYKJ11qEQU6b/8x9t6ceSTS0ILOf115miJz9qB1Ae6NVklsffUUuTTzzUBmajdnRZ7Z8Uzb/lwgVWnkuXsudzu40/pBnQdlbq73/PNgwlilrgUJHCwvCo4wDrH/7xj/DPnnwS+OMfpf+V6l9aGrOGejySbxcnlpL35ZetFvuoXLjATmvQutkiHtxudaXqxInIcxbffx946inz5DGQhJUqj8eDJ598BnfcMQsTJozGzTfPwuDBQ4yQzRii+RPEUqpidUi8E7XSWiWmcf68siWBy3P0qPIhoOJz8/tt3hy5TOB2Sw27qUmy2vHfNDQoD7pyH5e1a4H169WfI5qlKl74vZ5+Wvm8RKV7R7NU6ZEn3p1sO3eyI4bMZNUqpuAI0fw93P9PPDImkbzn57aJRFMSeFnxYILi9fJdtfX1yg7f0VDaei6PRcRnwbEsBdHyhTuqR3Mglk9c1OpVon2JXbP4oUOl93v3SudYAuFnyPFdX7GIVr5VVbEV2OZmNlERv6usZP3Z6tXS74ywVBUXs2Nt5Mrae+9FTjSVkCuDiViIqqoi6/jbb0dOUkKhyHR37mRnQ/LlfKstVfPnM8VbhI9jDrVOiRgSp2r69BnYvHkXtm0rxM9+9svYP7ASt5uZSpUctmMpVcEg87+59loWriAvD5g+Xfre7uW/WCxbFlk55UTrlNLSpM+Uzil89llm+YoHrqCpdaxG7v5TitGkZHXjZalnV43eDifaM5m1k0jMSz6TFuq+S7yOk8hOz7vu0nadPO9OSW4EqkrYn/7EBgY9cim10VWrlJfOxXSVLAWxrGtKOwJF7r03UjYzlCq7EJ9/6dLwXZfiM3FLVSzU+uizZ5m1R7S8KJXN5s3MCikGc+bXKcV80oO87PhynlJ4nXjQ0rdkZITHz+K8+CKwYEHsNB5/nO3AE+s9zxcz41/pXf5TIl5l2GQsc1S3Fbdb2aKjRam65JJI51k+wwGsLdRYikY8W5KjKVWipUqc9Zw4wZzLE9nSK86O1DoPuTzxDKRA+MyR30Opw+CWOjGifCyMnMWJgVbNWv7Te/94BnelJaBo6SkNrtE69D17om8KUEtbfk+lZSKl9qCkbGpVquTId5il2vJfNMS806pUqS3R8XPn9u+XzsaMVjbRrLCJhDYRf8ufxyiFWItP309/qiyPXr/WpqbYJxEYWZeiLf9Fw6GKlIhzYrubiculHAVbviQFMIWB09wc+9R7cflPTM8MtM5qjVSqlNJbtkz7/bWkq4SRliollO49cCB7VVMKzLYeWBFJWotSLl4X7zPLI3JHq7f8uS+6SPpOqX7EK4uewU7JUqXUnuJVquQDpNr1ybr8Fw2en6EQG/QTcUqPNtCLn/G4TqLCoKRUWe2orgUzlBk1tPgBGkm8liq524gDFay2oVRpcbirqgKKith6LifWur+8gM2ujFobcbyVVI48+KeRiIO2Vp8q+f/y421EYuWBUoPu3p05gcqPgeFEG+iNsPyYdX5VrHoT73fRkC9zRcPlYuEXxFg1Snkdb11Us1TFmqCISoD8e35P0QIifq72v5JSVVsbaZGx+ugRKxB9qvbt03bck56gukrlqbT0a6QV0AqlSmu/kIg8WvoAO3f/yWVxMG1j+U/LcTRKu7UqKqIX4vnzzlj+447GfDlTaRnN5WLr7HLisVQZcS6YOGDF66geK+p0NJTqhMulfm4gd66VY2SHI39+M9C6pJWopcrjYcup0QZOuT+bWBdjLc2Jv4uFFqVKXBaurWVOvXx5e9cuSYGX50dRUfj/8gFQvqTCrXI8fhyX7Z13wi11SvleUMCiT5sdHT8WQ4fGt3MuHmXmhz9k/pe7djH/qMZG9edX60diXWeUpSpaWvGg51SCRIm17G00iSz/ie8daKlqG0pVIhkvLgeq3duqglVLh/t4HTjAzgqUX8eXMXnsIPl3HK1KVTxnJsqJ1dlEy9OhQ2Mfa6D2+2CQ+Q/o7TjUokhz3454l4OsINEZ7Jdfaju0WSldpffivfl3chljWaoSdVRXu0dzc+TW7cJC4FvfYu+5oz+/53bZ6RHy55QHJnW7gf/5H+X8EGPiKcm2ejXz9+OyqKE2UTEKrdZ7OfEoVenp7I8roSUlyrtLY2GXpSpZlCo7LFVK71OAtrv8N2UK29UXy+qidOI979SCQeUO22qfKnm6sRy+RbQ6qqv9Jl60WKoOHgyXjV83eXLsGGIc8TBSAHjuORajhqM18rJaPJ2WiPyaZ6lmLLXFQqw38Thfv/12+E6iSy/Vlq7oLKtXidaqVBllqeLILU8ip09LO0n5vbj/HT+3UpTH44kd40qMa1ReLr0vLmZHIsk3TRQXh/9/8iRTekUrtNlLh3IFe4hCCB2l40wSkYu342hH5Gi1VCnVA7N8quJ95ooK4OOP2Xut/UIikyerLVXyMqqvZ5OGWMdUkaXKIcgj1wLSiexjxjBzP4/5FAiwIIiXX86sO5ddFvlb3lHKrVhO8amSE61hi9/Jd0jq3ZmkRzZ+7a5dyluC5ffy+1uPZNAEb2yZmcAPfsB85U6eDL+mSxftxy6o7YyJRiL1wezOQusMW23598iR6DHg9MgQj6WKoyWf1LaIK6X34YfK91i9Gjh0SPp/5062FBUKsfy45RZWR3bv1iebqHiKCtPrr+MSvz88aCoQGXbjlVciJz/Lluk7nkov11wTHrW8c2d2hNG8eSxfpk8H/v536Xt+4HgiSlWfPuxVvpx8+DDruzMzY08O5O+NWv6rrmZ5oKevjEZBQfgOcz0rA7W1ygptLKy2VImTDbebTaALCmKf4EA+VQ7B41EvrHbt2J8Yx2rEiOj3UwrgZxVarB/xWqrkjZefYaZlK3qsdKKly+MFiXLL03z1Vem9lvzmsrRvz67Py4tUqviZY1qQK1WBQLi1bPt21vGLhyrH2j2qJjNgvKldriRpLSul66ItYcVLvMt/eixV8mepr5e29scKr1JQEP7/8ePsr1s3tsFB7iOlFbEtaZk01NdHHkUjzyvxrFMz+qZ27cL71GHDwr+Xb+ffuZOtDCSiVHEfvWCQKVZiUNHnnmP1Ub5c5vMpn1No9PLfM8+wc/iUwnysWKH/vqJCBWh31ufPLSq0SgrjiRPMwi6elbdqFavH4gTXSkvVe++x97FCQSSBpaptLP8ZfUaSkknfCg2a75jYsCG88nGLGV+KUFKqNmwI/8zvZ0eFRDO38rhN8h1Ozc2SczwgLVHII1bX1YX7cR0+zGRpbg6/VoykLaaRCNzJONqxGXpORpd3bM8+G34e2+nT7Hga8Uw7JUU+Wj0RrRLRlqHiobIy/Bia5maW72fPwiUO7Dz6PP9MXJLiuN36O7OCAmbJUItYzgeoN95g1kvxHL5z59gSZGmp9Jn8HLBo8Gc5cCD88xdekA44jpfKynDFWW++cPcDPUcjya1X0TBrR6m489brZa9XX82sl/JQNevXs7JVqkt6SEtj/dXTT0f2SZWVUp/BX0WFKhRigUjLyyP7ltra+MKZuFxSGz91SlLMv/xS+lwMThoMsrosn5RWV0dO+Djjxye2MUGpD5o/n+WN+N2+fcDGjey9mu+hkQqMUugSIDxvlIIhc2tuMGiMb68JtA1L1eTJMU9W14U86GV1NWtQfIZglvbs8bBG+dln7E8OPwtKfiZUTU3k9T5fdP8EAOjdm72Ks0KAVXbx/MDnnwd++9tw/48FC1gwS3Hp9bXX0KNbNzbAig2CXyPmW7To4loGCu7rxO+tdFSNno704ouB736XdX7cUiAfpIHwCNIbN7KlELGTqqlhVi9eh7h/SigU3rGqdbLx4HKx+y1eLAWfLCtjxwcBcAeDLC9Wr2byiRY8cTmLk4jC+9FH0nu5UsUVSdGXDmDLbl9+GR6p+rXXpN/GQqmtRCMtDbjzTukUgezsSF8mEVHB07vsNmUKMGoUm3BwHxqtTJ4cqcDk5LDJ0OHDzF9Qj7Kmh1GjwpdDATb4NzRELtEFg8wSIj+wWi9ud+TRK5zPPpP8I7/4guWryO7d7G/5cukQ57IyaaKhJbyDHFGpAsKf7/e/B/77v8Ovf+KJyHs8/zxTUMV+QyRRgwCvw+LxQRzhqKpWxLb92WesnM1g+nSW50ePhvs8i5P8v/yFTcREXn0VmDuXKasNDZHKtQNoG0qVfBAeOzax+8kboDiDNhP52XxyuANxr17s+AI+M1RqsGoz/dtuY9u7AWnAFy0cajzxRKSSouDL1u3LL5mpPBbRLGhaBq7x41mnzzvQESMil1f0zuIvvpj9yZdf1CgoYIqruFupvJwtG3KlqrYW+Ne/IjdEfPOb+mSLhngWIe+EWhQqAOhw5gxTHHi9FpUQpSM3EomkL55FKYYxUVOOPB59Ee6VuPde5XAiHHnH/dBDzDL5q1+xOu12Sz5k2dlM+RMd98X2P3iw9F6LwpeWBrShwq0AAA6YSURBVPToIS2h1dQw5eDIEZRnZsI7YQJLu6aGDXDl5UzBS09nlqH27VkdyshgFuvRo1n9+uST8OUdo1FakuK+XUpWVj3WNTVE67icc+fCFZyXX1a/VlREuQVfLDetyMtX3t+99Vbse5w5E33JK1afL6JkueGrGEq72JUOKz92LHyp+5lnJL9TI40FPXqww9flbU8+ln78cWR/o9b2HEJCStV//vM+nnnmT/jqq4P49NO1GDHCJK3WCEaOlA6I1OqcrJVhw8KXucyKI8OdNZXweqWI4ED4jENJqVKrjH37Atddx75XOi8xGhrMse5ojoixZtWDBjEnWS2Nu3//8AarlHd6lv9EvvUtyQcAYAqbkkUHYMsC4pl2QKTv2OnTkZY0MWaRyaTJFVgeJkIJj0d5t5cSDz/MlmpiEW1JtKkpfAlFTmmp8mYSEfH4n1h873uSwiA64ottq18/4PbbpcOvo7VLrXToIB230pJGhdvNfAHz8tR/J1oyxN/ffXfiMkVDycrLLd9al2Z/aeA5sfI2FMvhmcNljif2nrwfkvd/Wq3Neo+UMeo+Srvilc543bcvPnm0IK5mZGVFjkvykCVy4tlAZDIJLbgPHnwFFixYgnHjJhglj3lMmcKc8B57jJ3nlwjyQWXCBODBB6X/4x2sYzF8OHtVCicgN9Nq7VTkdO/OYl3dfHN0pSqar5IcpUHt619nysmtt7L/5QMHf1ZRru98J35lIy2NldtFF7Ggobfeqm8WKCK3tKWnM5P0d76jXRY1Bg1iMurJXyOQOxsDylHRf/IT7e2H7wpTQrSiBALKS6nRGDmSvcqXupXQugSWl6fNigqw+FM/+hF7P2iQ8jVWHDtkF7x/Ea2w4iYNNXr0kH4XT+yzaPdU4oYbpPd33y25NHTvHh4mJZ6+QOvOU+5zBgD33CMp7f36RZ4ry8nJCT/XTwtKvqlqKNXNWHXfjPr8wAPSe9E6L+//OfIQOWYtTyZAQkrVwIGDcNlllxsli7n06MGUHyN8DLp1Y7tNRo5klhOA+apMmADk5koN12hcLqYQ3H0322mSlsZmp+npkZUtlum/fXsmv8fDZvrt2oXPdAHWIYwaxTrAjAz22Z13sqjj3/2udN2sWaxTyshgR7xMm8YUg6ws9vm4ca15EuzQgd1v/HjgqqvYgDR6NDBpUnjas2axPJ43jy3D6O1glLjjDjYQ9u+fmLUyM5PJc//9wI03smfp1489y+TJrFzky3diB3zNNeoD0MSJTEYjHYz5oN+5M1PW+GDWpQtw8cVo6tKFLYmPG8cUi4wM5oNxySXsszFjWL2bNUubEiNHyZ+D77Dt2JHdf8yY8O8zMlhe8GXR/v0li1CPHizff/hD5XsrIe7M4lanb30rfAIUzSKkxEUXsXogb2t8IJ8+Xd/9komcHNZub7tN+mzGDKnvE/3yBg5kE6u+fVm7uPlm9ZMLosHLZ/z4cIVIDLCamyt9164dm0j16sUUmQEDJNePAQNYW+3bl30Xz+oCn3S63eH1cNQoabDv1Qu46Sb2vn9/pmBxhWHoUBYrMTubye1ysSVdgNUh3udqhS9hihNPJYUzO1saL/j3WVnhCvL110dOtMwY11wuVpcyM1leZWWxMWnYMGXjwXXXsfz2eIAf/1hdKbUR15kzlQlvW/vmN6/HvHm/j7r8t2jRAixevBAAUFJShDfFnVMmEAgE0EVrgMhURtx2yt9bEdxNBSoX50Fl4kyoXGIQCsHV2IhQ+/Zw19UhlJaGUIsPnKuhAeD/NzezPi8tDQiFkFZXxyZ3LUeluEIhhDROtuVl4q6tRcjjQahdO5am283SBJhsLQFg3Q0NaE5La5XB3dCAZm6NFo9IMsu6yUOHCCFVXKEQQm43XMEgQi07el3BIJO/ZZxwNTe3Po9pBINwAVI6fJxqbg4P3MvzNRhECAjLKyvaykMP/RyrVq2LeV3MmnTLLd9AWVmk/82vf/07zJx5g8IvlJkzZy7mzJkLAJg5cyryzXSiBODz+UxPg9APlYvzoDJxJlQuzoPKxJk4qVxiKlVLlyrsECAIgiAIgiDCaBvBPwmCIAiCIEwmIaXqk08+wvDhg7F9+1bceedtuP12A2PrEARBEARBJBEJbYW74YabcMMNNxklC0EQBEEQRNJCy38EQRAEQRAGYMsxNceOHTM9+vq5c2fRs6eOSMqEJVC5OA8qE2dC5eI8qEyciRXlcvKkyrmTMgyJU+VEvv71KZpiShDWQuXiPKhMnAmVi/OgMnEmTioXWv4jCIIgCIIwAFKqCIIgCIIgDCDtl7/89Ty7hTCL4cNH2i0CoQCVi/OgMnEmVC7Og8rEmTilXFLWp4ogCIIgCMJKaPmPIAiCIAjCAEipIgiCIAiCMICUVKrWrFmFceNGYezY4Xjuub/YLU5Kc/r0KcyadQMmThyDSZPG4uWXnwcAlJefx6233oyrrx6BW2+9GRculAMAQqEQfvObX2Ls2OGYMmUcCgt3t97rzTeX4OqrR+Dqq0fgzTeX2PI8qUQwGMS0aRMxe/ZtAIDjx/247rqpGDt2OL7//XvQ0NAAAKivr8f3v38Pxo4djuuum4oTJ6R4LH/7258xduxwjBs3CmvWrLblOVKJiooLuPfeuzF+fB4mTBiNbdu2UFuxmRdf/AcmTRqLyZOvxn33zUVdXR21FRv46U9/hCuuGIDJk69u/czItrFnzy5MmfI1jB07HL/5zS8RCpnj+ZRySlUwGMQjj/wCb7yxFAUF2/Dee+/i4MEDdouVsng8Hjz++B9QULANy5d/hvnz/4WDBw/guef+ismTp2DLlt2YPHkKnnvurwCAzz5biaNHj2DLlt3485//hocf/hkA1nieffYprFixBp9+uhbPPvtUawMi4uPll1/AwIEDW/9/4on/wX33PYCtW/ege/ceWLJkEQBgyZJF6N69B7Zu3YP77nsATzzxPwCAgwcP4P33l2LDhq1488338MgjP0cwGLTlWVKFRx99BNOmTcemTTuwdu0mDBw4iNqKjRQXF+Hf/34JK1euw/r1WxAMNuODD5ZSW7GBb397Nt58872wz4xsGw8//DP8+c/PYcuW3Th69AjWrFllynOknFK1c+d29O8/AF5vf7Rv3x6zZt2CFSs+sVuslCUrqw+uumoEAKBLl64YOHAQiouLsGLFJ7jjjjsBAHfccSeWL/8YALB8+TLcfvt34HK5MHr0WFRUVKC0tARr136GKVOmIiMjEz16ZGDKlKk020uAoqLTWL36U8ye/V8A2MyuoGAdbrqJHXp+xx3faS0TVlbfAQDcdNM3sWGDD6FQCCtWfIJZs25Bhw4d0K+fF/37D8DOndvteaAUoLKyAps3b8Ls2XMAAO3bt0f37j2ordhMU1MT6upq0dTUhNraGlx0URa1FRsYN24CevTICPvMqLZRWlqCqqoqjB49Fi6XC7ff/h0sW2aOXpBySlVJSTFyc/u2/p+dnYPi4iIbJWo7nDhxHHv3FiIvbzTKysqQldUHAHDRRVkoKysDAJSUFCEnRyqfnJxcFBcXobi4GLm5ucLnOSguLrb2AVKIxx77FX73u/+F282a+Pnz59GtW3d4POxkqpycXJSUsPwV24zH40HXrt1w/vx5FBcXISdHLBPpN4R+jh8/jp49e+LBB+/HtGkT8bOf/RjV1dXUVmwkOzsHP/rRTzBy5FAMG3Y5unbthuHDR1JbcQhGtY3i4iJkZ8vLxxy9IOWUKsIeAoEA7r33bjzxxJPo2rVb2Hculwsul8smydoeK1cuR69evRwTt4VgBINNKCzcg3vu+S7WrClA586d8fe/h/t8UluxlgsXyrFixTJs374XhYVfoaamxrRlISIxkqVtpJxS1adPNk6fPtX6P9NQc2yUKPVpbGzEvffehVtuuR033vgNAEDv3r1RWloCACgtLUGvXuywyz59clBUJJVPUdFpZGfnIDs7G6dPnxY+L0J2draFT5E6bN26BZ9+uhx5eVfiBz+Yi4KC9Xj00YdRWVmBpqYmACzf+/Rh+Su2maamJlRVVSIzMxPZ2TkoKhLLRPoNoZ/s7Fzk5OQiL28MALZ8VFi4h9qKjaxf78Mll/RDr1690K5dO9xww03YunULtRWHYFTbYCtW8vIxRy9IOaVq5Mg8HD16FMeP+9HQ0ID331+KGTOut1uslCUUCuGhhx7AwIGDcP/9P279fMaM6/HWW68DAN5663Vcd90NAIDrrpuJt99+A6FQCNu3b0W3bt2QldUHU6deg3Xr1uDChXJcuFCOdevWYOrUa2x5pmTnscfmYc+eA9ixYx9efnkBJk6cjBdffAUTJkzGRx99AAB46603WsuEldUbAICPPvoAEydOgcvlwowZ1+P995eivr4ex4/7cfToUYwaNdq250p2srKykJOTi8OHDwFgA/rAgYOprdhIbm5f7NixDTU1NQiFQtiwYR0GDRpEbcUhGNU2srL6oGvXrti+fStCoRDefvsNzJxpjl7gMeWuNuLxePDkk8/gjjtmIRgM4s4778bgwUPsFitl2bJlM955500MGTIUU6dOAAA8+ujv8OCDP8P3v38PlixZhL59L8G//70QADB9+gysXr0SY8cOR+fOnfG3v7EQDBkZmfj5zx/GtdfmAwB+8YtHkJGRaccjpSy//e3juO++ufjTn57AsGHDWx2mZ8+egwce+AHGjh2OjIwMvPTSAgDA4MFDcPPNszBx4hh4PB489dSzSEtLs/MRkp4//vEZ3H//99DQ0IB+/bx47rnn0dzcTG3FJvLyxuDGG2/G9OmT4PF4cOWVV+Huu+di+vQZ1FYs5r775mLjxgKcP38Ow4cPxsMP/8bQceSpp/6CBx+8H7W1tbjmmq/jmmuuNeU56JgagiAIgiAIA0i55T+CIAiCIAg7IKWKIAiCIAjCAEipIgiCIAiCMABSqgiCIAiCIAyAlCqCIAiCIAgDIKWKIAiCIAjCAEipIgiCIAiCMID/BwMDg9gwH2ATAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f9bab9de588>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "ax.plot(y_test, 'r-', alpha=0.5)\n",
    "plt.gcf().set_size_inches((10,3))\n",
    "fig.set_facecolor('#EAEAF2')\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = Sequential()    \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# your model here\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "start = time.time()\n",
    "model.compile(loss=\"mse\", optimizer=\"rmsprop\")\n",
    "print(\"Compilation Time : \", time.time() - start)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# change epochs, batch_size...\n",
    "\n",
    "\n",
    "epochs = 1\n",
    "model.fit(X_train, y_train,\n",
    "          batch_size=10, \n",
    "          nb_epoch=epochs, \n",
    "          validation_split=0.05)\n",
    "predicted = model.predict(X_test)\n",
    "predicted = np.reshape(predicted, (predicted.size,))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "ax.plot(y_test[250:450], 'r')\n",
    "plt.plot(predicted[250:450], 'g')\n",
    "plt.gcf().set_size_inches((10,3))\n",
    "fig.set_facecolor('#EAEAF2')\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "ax.plot(y_test, 'r', alpha=0.5)\n",
    "plt.plot(predicted, 'g', alpha=0.5)\n",
    "plt.gcf().set_size_inches((10,3))\n",
    "fig.set_facecolor('#EAEAF2')\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}