CharRNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Four flavours of a CharRNN implementation in Lasagne\n",
    "==="
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "import sys\n",
    "import os\n",
    "import time\n",
    "\n",
    "import numpy as np\n",
    "import theano\n",
    "import theano.tensor as T\n",
    "import lasagne"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8.0.dev-a34dec55bfd6bd84e92a97346b5665f685b83a44\n",
      "0.2.dev1\n"
     ]
    }
   ],
   "source": [
    "print(theano.__version__)  #should be at least 0.8.0.dev\n",
    "print(lasagne.__version__) #should be at least 0.2.dev1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A note on RNN model building in lasagne\n",
    "#### Input layer\n",
    "The convention used in lasagne is that sequential data is presented in the shape `(batch_size, n_time_steps, n_features_1, n_features_2, ...)`. Because not all sequences in each minibatch will always have the same length, all recurrent layers in lasagne accept a separate mask input which has shape `(batch_size, n_time_steps)`, which is populated such that `mask[i, j] = 1` when `j <= (length of sequence i)` and `mask[i, j] = 0` when `j > (length of sequence i)`. When no mask is provided, it is assumed that all sequences in the minibatch are of length `n_time_steps`. Finally, as is true of the first `(batch_size)` dimension, the `n_time_steps` dimension can be set to None, which means that it can vary from batch to batch. This means that the network can take in minibatches which have an arbitrary number of sequences which are of arbitrary length - very convenient!\n",
    "\n",
    "#### LSTM layer\n",
    "The de facto method for training recurrent networks is backpropagation through time, which simply unrolls the network across timesteps and treats it as a network which is repeated for each time step. This can result in an incredibly \"deep\" and computationally expensive network, so it's common practice to truncate the number of unrolled sequence steps. This can be controlled with the `gradient_steps` argument; when it's -1 (the default), this means \"don't truncate\". A common method for mitigating the exponentially growing gradients commonly found when \"unrolling\" recurrent networks through time and backpropagating is to simply preventing them from being larger than a pre-set value. In recurrent layers, this can be achieved by passing in a float (rather than False) to `grad_clipping`. Some of the dot products computed in recurrent layers are non-recursive, which means they can be computed ahead of time in one big dot product. Since one big dot product is more efficient than lots of little dot products, lasagne does it by default. However, it imposes an additional memory requirement, so if you're running out of memory, set `precompute_input` to False.\n",
    "\n",
    "> ##### LSTM params\n",
    "The GRULayer and LSTMLayer utilize the Gate class, which is essentially just a container for parameter initializers. The LSTMLayer initializer accepts four Gate instances - one for the input gate, one for the forget gate, one for the cell, and one for the output gate. From the lasagne code base: the bias of the forget gate is often initialized to a large positive value to encourage the layer initially remember the cell value, see e.g. _\"Learning to forget: Continual prediction with LSTM.\"_ page 15.\n",
    "\n",
    "#### Fully-connected layer\n",
    "As mentioned above, recurrent layers and feed-forward layers expect different input shapes. The output of l_sum will be of shape `(n_batch, n_time_steps, N_HIDDEN)`. If we fed this into a non-recurrent layer, it would think that the n_time_steps dimension was a \"feature\" dimension, when in fact it's a \"sample\" dimension. That is, each index in the second dimension should be treated as a different sample, and a non-recurrent lasagne layer would instead treat them as different feature values, which would be incorrect. Fortunately, the ReshapeLayer makes combining these conventions very convenient - we just combine the first and second dimension so that there are essentially `n_batch*n_time_steps` individual samples before using any non-recurrent layers, then (optionally) reshape the output back to the original shape. Note that because we will only be using the output of the network at the end of the sequence, this could also be done using a SliceLayer (as in the recurrent.py example included with lasagne) which is a bit more efficient. In this tutorial, we'll do it with the ReshapeLayer for illustration."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Flavour 1 - seq2seq training, seq2sample prediction\n",
    "\n",
    "In the first flavour of the CharRNN we adopt the following procedures. In the training phase we employ a sequence 2 sequence procedure. That is, for every input character, we predict the subsequent character. At test time, to sample from our model, we start with a bootstrap text sequence for which we predict the single next character. After that, the input for the next time step is `bootstrap_text[1:]+predicted_char`, for which we predict the following character etc.\n",
    "\n",
    "In lasagne every input sequence is treated seperately by default. That is, the hidden and cell states get reset for every new input (so no state is transfered to the following input; this will be the subject of one of the following CharRNN flavours). We will use an input sequence length of 25 chars and we backpropagate the gradient for 20 timesteps. So five timesteps are used to \"bootstrap\" the hidden and cell states. We will divide our data in sequences as follows: data[0:25], data[5:30], data[10:35] etc. So there will be overlap in the input sequences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Hold options as static element in the opts class\n",
    "class opts():\n",
    "    hidden_size = 50\n",
    "    seq_len = 25         # Data sequence length\n",
    "    gradient_steps = 20  # Truncated BPTT length\n",
    "    data_offset = 15     # Offset for every new input sequence\n",
    "    batch_size = 50\n",
    "    n_epochs = 100\n",
    "    lr = 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def build_rnn_1(input_var, dim):\n",
    "    # By setting the first and second dimensions to None, we allow arbitrary minibatch\n",
    "    # sizes with arbitrary sequence lengths.\n",
    "    # In this case, we know the sequence length beforehand.\n",
    "    # We leave the batch size undetermined, so that the model can be used for single value prediction.\n",
    "    # The dim variable will be equal to the size of the vocabulary (one-hot char representation).\n",
    "    # ----- INPUT LAYER -----\n",
    "    l_in = lasagne.layers.InputLayer(shape=(None, opts.seq_len, dim), input_var=input_var)\n",
    "    \n",
    "    # The following input can be used to provide the network with masks.\n",
    "    # Masks are expected to be matrices of shape (n_batch, n_time_steps);\n",
    "    # Since all our sequences will be of equal length, we don't need the masks here.\n",
    "    # l_mask = lasagne.layers.InputLayer(shape=(None, None))\n",
    "    \n",
    "    # The convention is that gates use the standard sigmoid nonlinearity,\n",
    "    # which is the default for the Gate class.\n",
    "    io_gate_parameters = lasagne.layers.recurrent.Gate(\n",
    "        W_in=lasagne.init.Orthogonal(), W_hid=lasagne.init.Orthogonal(),\n",
    "        b=lasagne.init.Constant(0.))\n",
    "    \n",
    "    forget_gate_parameters = lasagne.layers.recurrent.Gate(\n",
    "        W_in=lasagne.init.Orthogonal(), W_hid=lasagne.init.Orthogonal(),\n",
    "        b=lasagne.init.Constant(5.))\n",
    "    \n",
    "    cell_parameters = lasagne.layers.recurrent.Gate(\n",
    "        W_in=lasagne.init.Orthogonal(), W_hid=lasagne.init.Orthogonal(),\n",
    "        # Setting W_cell to None denotes that no cell connection will be used.\n",
    "        W_cell=None, b=lasagne.init.Constant(0.),\n",
    "        # By convention, the cell nonlinearity is tanh in an LSTM.\n",
    "        nonlinearity=lasagne.nonlinearities.tanh)\n",
    "    \n",
    "    # ----- LSTM LAYER -----\n",
    "    l_lstm = lasagne.layers.recurrent.LSTMLayer(\n",
    "        l_in, opts.hidden_size,\n",
    "        # We need to specify a separate input for masks (not needed here)\n",
    "        # mask_input=None,\n",
    "        # Here, we supply the gate parameters for each gate\n",
    "        ingate=io_gate_parameters, forgetgate=forget_gate_parameters,\n",
    "        cell=cell_parameters, outgate=io_gate_parameters,\n",
    "        # We'll learn the initialization and use gradient clipping\n",
    "        learn_init=True, grad_clipping=50., gradient_steps=opts.gradient_steps)\n",
    "    \n",
    "    # ----- FC LAYER -----\n",
    "    # First reshape so that output at every time step is treated as separate sample\n",
    "    # Since batch size is unknown, we have to determine it first.\n",
    "    batch_size, _, _ = l_in.input_var.shape\n",
    "    l_reshape = lasagne.layers.ReshapeLayer(l_lstm, (batch_size * opts.seq_len, opts.hidden_size))\n",
    "    l_dense = lasagne.layers.DenseLayer(\n",
    "        l_reshape, num_units=dim, nonlinearity=lasagne.nonlinearities.softmax)\n",
    "    # The output size of the network is thus (batch_size * opts.seq_len, dim)\n",
    "    \n",
    "    return l_dense"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data processing.\n",
    "Remember, the convention used in lasagne is that sequential data is presented in the shape `(batch_size, n_time_steps, n_features_1, n_features_2, ...)`. Also remember how we will construct our input sequences with the proper offset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary size = 65; total data size = 1115394\n"
     ]
    }
   ],
   "source": [
    "data = open('tinyshakespeare.txt', 'r').read()\n",
    "chars = list(set(data))\n",
    "data_size, vocab_size = len(data), len(chars)\n",
    "print ('Vocabulary size = ' + str(vocab_size) + '; total data size = ' + str(data_size))\n",
    "char_to_ix = { ch:i for i,ch in enumerate(chars) }\n",
    "ix_to_char = { i:ch for i,ch in enumerate(chars) }\n",
    "\n",
    "# Define function to get batches of preprocessed data.\n",
    "def get_batch(b):\n",
    "    if (b+1)*opts.batch_size*opts.data_offset - opts.data_offset + opts.seq_len + 1 >= len(data):\n",
    "        return None, None\n",
    "    X = np.zeros((opts.batch_size, opts.seq_len, vocab_size), dtype=theano.config.floatX)\n",
    "    y = np.zeros((opts.batch_size, opts.seq_len, vocab_size), dtype=np.int8)\n",
    "    \n",
    "    for i in xrange(opts.batch_size):\n",
    "        c = b*opts.data_offset*opts.batch_size + opts.data_offset*i\n",
    "        for j in xrange(opts.seq_len):\n",
    "            X[i, j, char_to_ix[data[c]]] = 1.0\n",
    "            y[i, j, char_to_ix[data[c+1]]] = 1.0\n",
    "            c += 1\n",
    "    \n",
    "    return X, y.reshape((opts.batch_size*opts.seq_len, vocab_size))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's test the `get_batch()` function. Following code should give twice the same output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " you curs,\n",
      "--\n",
      " you curs,"
     ]
    }
   ],
   "source": [
    "X,y = get_batch(10)\n",
    "for i in xrange(opts.data_offset, opts.seq_len):\n",
    "    sys.stdout.write(ix_to_char[np.argmax(X[4][i])])\n",
    "print(\"\\n--\")\n",
    "for i in xrange(opts.data_offset, opts.seq_len):\n",
    "    sys.stdout.write(ix_to_char[np.argmax(X[5][i-opts.data_offset])])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Main program."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Saving and reading model parameter functions\n",
    "import cPickle\n",
    "def save_params(network, filename):\n",
    "    params = lasagne.layers.get_all_param_values(network)\n",
    "    with open(filename, 'wb') as f:\n",
    "        cPickle.dump(params, f)\n",
    "\n",
    "def load_params(network, filename):\n",
    "    params = None\n",
    "    with open(filename, 'rb') as f:\n",
    "        params = pickle.load(f)\n",
    "    lasagne.layers.set_all_param_values(network, params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the necessary tensor variables, create the network, define the loss function, define the parameters to be learned, and compile the train and sample functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "input_var = T.tensor3('inputs')\n",
    "output_var = T.bmatrix('outputs') # the outputs will be flattened over 1st and 2nd dimension to reflect\n",
    "                                 # dense layer output\n",
    "\n",
    "network = build_rnn_1(input_var, vocab_size)\n",
    "\n",
    "# Now predict the cost of batch in terms of a loss function\n",
    "network_output = lasagne.layers.get_output(network)\n",
    "loss = lasagne.objectives.categorical_crossentropy(network_output, output_var).mean()\n",
    "\n",
    "# Retrieve all network params\n",
    "all_params = lasagne.layers.get_all_params(network)\n",
    "\n",
    "# Compute adam updates for training\n",
    "updates = lasagne.updates.adam(loss, all_params)\n",
    "\n",
    "# Theano function for training and computing cost\n",
    "train = theano.function(\n",
    "    [input_var, output_var],\n",
    "    loss, updates=updates)\n",
    "\n",
    "# Theano function to sample from the RNN; we only keep the final output prediction for the first batch\n",
    "sample = theano.function(\n",
    "    [input_var], network_output[-1,:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also need a function to sample text from the RNN in order to babysit the learning process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def sample_text(length=200):\n",
    "    # First take a random piece of bootstrap text\n",
    "    start = np.random.randint(0, len(data)-opts.seq_len)\n",
    "    s = data[start:start+opts.seq_len]\n",
    "    \n",
    "    # Convert to proper input data shape (here, batch size = 1)\n",
    "    s_np = np.zeros((1, opts.seq_len, vocab_size), dtype=theano.config.floatX)\n",
    "    for i in xrange(opts.seq_len):\n",
    "        s_np[0, i, char_to_ix[s[i]]] = 1.0\n",
    "    \n",
    "    # Start sampling loop\n",
    "    res = ''\n",
    "    for k in xrange(length):\n",
    "        # Predict next character\n",
    "        predict = sample(s_np)\n",
    "        predict_i = np.random.choice(range(vocab_size), p=predict.ravel())\n",
    "        res += ix_to_char[predict_i]\n",
    "        \n",
    "        # Update s_np\n",
    "        s_np[0, 0:-1, :] = s_np[0, 1:, :]\n",
    "        s_np[0, -1, :] = 0.0\n",
    "        s_np[0, -1, predict_i] = 1.0\n",
    "    \n",
    "    return res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hxjb.R,;Q;elhTNqI.NT,ErKprJADOM'LDuRdD ySbOZl$Qh-ZdZRDTEFl$'Dixp?3QDZ;O:OTQgqmGjV!WQv?cPbvl:hu3p?lapjKYMJ-&LY&VUdknGIyYqTnJSb-Gy&bEOmt-DBcZBd?U:3zA\n",
      "iklhkOArFPsl!eTRcE;$EQ:uPbxSnspkwsPOKAO:a;hXMoEa:x?d\n"
     ]
    }
   ],
   "source": [
    "# Let's test the function on an untrained RNN\n",
    "print(sample_text(200))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Time to train the RNN. After every epoch, we will also sample a text from the RNN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start training RNN...\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-84-d75f0f5a218a>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      8\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mX\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0mNone\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0my\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      9\u001b[0m             \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 10\u001b[0;31m         \u001b[0mcost\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     11\u001b[0m         \u001b[0mb\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1.0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     12\u001b[0m     \u001b[0;32mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"\\nEpoch {} with {} batches, cost = {}\\n\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mepoch\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcost\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0mb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/cedricdeboom/Documents/UGent/PhD/Onderzoek/Theano/theano/compile/function_module.pyc\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    857\u001b[0m         \u001b[0mt0_fn\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtime\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    858\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 859\u001b[0;31m             \u001b[0moutputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    860\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    861\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'position_of_error'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/cedricdeboom/Documents/UGent/PhD/Onderzoek/Theano/theano/gof/op.pyc\u001b[0m in \u001b[0;36mrval\u001b[0;34m(p, i, o, n)\u001b[0m\n\u001b[1;32m    912\u001b[0m             \u001b[0;31m# default arguments are stored in the closure of `rval`\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    913\u001b[0m             \u001b[0;32mdef\u001b[0m \u001b[0mrval\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mp\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnode_input_storage\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mo\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnode_output_storage\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 914\u001b[0;31m                 \u001b[0mr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mx\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mo\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    915\u001b[0m                 \u001b[0;32mfor\u001b[0m \u001b[0mo\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mnode\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutputs\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    916\u001b[0m                     \u001b[0mcompute_map\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mo\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/cedricdeboom/Documents/UGent/PhD/Onderzoek/Theano/theano/tensor/blas.pyc\u001b[0m in \u001b[0;36mperform\u001b[0;34m(self, node, inp, out)\u001b[0m\n\u001b[1;32m   1550\u001b[0m         \u001b[0mz\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mout\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1551\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1552\u001b[0;31m             \u001b[0mz\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnumpy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnumpy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1553\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mValueError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1554\u001b[0m             \u001b[0;31m# The error raised by numpy has no shape information, we mean to\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "# Train procedure\n",
    "print(\"Start training RNN...\")\n",
    "for epoch in range(opts.n_epochs):\n",
    "    cost  = 0.0\n",
    "    b = 0.0\n",
    "    while True:\n",
    "        X, y = get_batch(int(b))\n",
    "        if X is None or y is None:\n",
    "            break\n",
    "        cost += train(X, y)\n",
    "        b += 1.0\n",
    "    print(\"\\nEpoch {} with {} batches, cost = {}\\n\".format(epoch + 1, int(b), cost / b))\n",
    "    #print(\"Saving...\")\n",
    "    #save_params(network, \"params_\"+str(epoch + 1)+\".dump\")\n",
    "    # Sampling\n",
    "    print(sample_text(100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Flavour 2 - seq2sample training, seq2sample prediction\n",
    "\n",
    "In the second flavour of the CharRNN we use a sequence 2 sample training procedure. In this case, we predict only the final next character for a given input sequence. At test time, we proceed in the same way as in the first flavour.\n",
    "\n",
    "We keep the same options as before, but in this case it is reasonable to set the data_offset to 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Hold options as static element in the opts class\n",
    "class opts():\n",
    "    hidden_size = 50\n",
    "    seq_len = 25         # Data sequence length\n",
    "    gradient_steps = 20  # Truncated BPTT length\n",
    "    data_offset = 1      # Offset for every new input sequence\n",
    "    batch_size = 50\n",
    "    n_epochs = 100\n",
    "    lr = 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def build_rnn_2(input_var, dim):\n",
    "    # ----- INPUT LAYER -----\n",
    "    l_in = lasagne.layers.InputLayer(shape=(None, opts.seq_len, dim), input_var=input_var)\n",
    "\n",
    "    io_gate_parameters = lasagne.layers.recurrent.Gate(\n",
    "        W_in=lasagne.init.Orthogonal(), W_hid=lasagne.init.Orthogonal(),\n",
    "        b=lasagne.init.Constant(0.))\n",
    "    \n",
    "    forget_gate_parameters = lasagne.layers.recurrent.Gate(\n",
    "        W_in=lasagne.init.Orthogonal(), W_hid=lasagne.init.Orthogonal(),\n",
    "        b=lasagne.init.Constant(5.))\n",
    "    \n",
    "    cell_parameters = lasagne.layers.recurrent.Gate(\n",
    "        W_in=lasagne.init.Orthogonal(), W_hid=lasagne.init.Orthogonal(),\n",
    "        W_cell=None, b=lasagne.init.Constant(0.),\n",
    "        nonlinearity=lasagne.nonlinearities.tanh)\n",
    "    \n",
    "    # ----- LSTM LAYER -----\n",
    "    l_lstm = lasagne.layers.recurrent.LSTMLayer(\n",
    "        l_in, opts.hidden_size,\n",
    "        ingate=io_gate_parameters, forgetgate=forget_gate_parameters,\n",
    "        cell=cell_parameters, outgate=io_gate_parameters,\n",
    "        learn_init=True, grad_clipping=50., gradient_steps=opts.gradient_steps)\n",
    "    \n",
    "    # ----- SLICE LAYER -----\n",
    "    # We only need the final output of the LSTM layer\n",
    "    # Output of this layer now has shape (batch_size, opts.hidden_size)\n",
    "    l_slice = lasagne.layers.SliceLayer(l_lstm, indices=-1, axis=1)\n",
    "    \n",
    "    # ----- FC LAYER -----\n",
    "    l_dense = lasagne.layers.DenseLayer(\n",
    "        l_slice, num_units=dim, nonlinearity=lasagne.nonlinearities.softmax)\n",
    "    # The output size of the network is thus (batch_size, dim)\n",
    "    \n",
    "    return l_dense"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary size = 65; total data size = 1115394\n"
     ]
    }
   ],
   "source": [
    "data = open('tinyshakespeare.txt', 'r').read()\n",
    "chars = list(set(data))\n",
    "data_size, vocab_size = len(data), len(chars)\n",
    "print ('Vocabulary size = ' + str(vocab_size) + '; total data size = ' + str(data_size))\n",
    "char_to_ix = { ch:i for i,ch in enumerate(chars) }\n",
    "ix_to_char = { i:ch for i,ch in enumerate(chars) }\n",
    "\n",
    "# Define function to get batches of preprocessed data.\n",
    "def get_batch(data, b, b_size):\n",
    "    if (b+1)*b_size*opts.data_offset - opts.data_offset + opts.seq_len + 1 >= len(data):\n",
    "        return None, None\n",
    "    X = np.zeros((b_size, opts.seq_len, vocab_size), dtype=theano.config.floatX)\n",
    "    y = np.zeros((b_size, vocab_size), dtype=np.int8)\n",
    "    \n",
    "    for i in xrange(b_size):\n",
    "        c = b*opts.data_offset*b_size + opts.data_offset*i\n",
    "        for j in xrange(opts.seq_len):\n",
    "            X[i, j, char_to_ix[data[c]]] = 1.0\n",
    "            c += 1\n",
    "        y[i, char_to_ix[data[c]]] = 1.0\n",
    "    \n",
    "    return X, y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test the `get_batch()` function, which seems OK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "izens, the patricians goo\n",
      "d\n",
      "--\n",
      "zens, the patricians good\n",
      ".\n"
     ]
    }
   ],
   "source": [
    "X,y = get_batch(data, 10, opts.batch_size)\n",
    "for i in xrange(opts.seq_len):\n",
    "    sys.stdout.write(ix_to_char[np.argmax(X[4][i])])\n",
    "print(\"\")\n",
    "print(ix_to_char[np.argmax(y[4])])\n",
    "print(\"--\")\n",
    "for i in xrange(opts.seq_len):\n",
    "    sys.stdout.write(ix_to_char[np.argmax(X[5][i])])\n",
    "print(\"\")\n",
    "print(ix_to_char[np.argmax(y[5])])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Main program."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "input_var = T.tensor3('inputs')\n",
    "output_var = T.bmatrix('outputs') # the outputs will be flattened over 1st and 2nd dimension to reflect\n",
    "                                 # dense layer output\n",
    "\n",
    "network = build_rnn_2(input_var, vocab_size)\n",
    "\n",
    "# Now predict the cost of batch in terms of a loss function\n",
    "network_output = lasagne.layers.get_output(network)\n",
    "loss = lasagne.objectives.categorical_crossentropy(network_output, output_var).mean()\n",
    "\n",
    "# Retrieve all network params\n",
    "all_params = lasagne.layers.get_all_params(network)\n",
    "\n",
    "# Compute adam updates for training\n",
    "updates = lasagne.updates.adam(loss, all_params)\n",
    "\n",
    "# Theano function for training and computing cost\n",
    "train = theano.function(\n",
    "    [input_var, output_var],\n",
    "    loss, updates=updates)\n",
    "\n",
    "# Theano function to sample from the RNN; we only keep the final output prediction for the first batch\n",
    "sample = theano.function(\n",
    "    [input_var], network_output[-1,:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def sample_text(data, length=200):\n",
    "    # First take a random piece of bootstrap text\n",
    "    start = np.random.randint(0, len(data)-opts.seq_len)\n",
    "    s = data[start:start+opts.seq_len]\n",
    "    \n",
    "    # Convert to proper input data shape (here, batch size = 1)\n",
    "    s_np = np.zeros((1, opts.seq_len, vocab_size), dtype=theano.config.floatX)\n",
    "    for i in xrange(opts.seq_len):\n",
    "        s_np[0, i, char_to_ix[s[i]]] = 1.0\n",
    "    \n",
    "    # Start sampling loop\n",
    "    res = ''\n",
    "    for k in xrange(length):\n",
    "        # Predict next character\n",
    "        predict = sample(s_np)\n",
    "        predict_i = np.random.choice(range(vocab_size), p=predict.ravel())\n",
    "        res += ix_to_char[predict_i]\n",
    "        \n",
    "        # Update s_np\n",
    "        s_np[0, 0:-1, :] = s_np[0, 1:, :]\n",
    "        s_np[0, -1, :] = 0.0\n",
    "        s_np[0, -1, predict_i] = 1.0\n",
    "    \n",
    "    return res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "had de andes wrat wy he fnavert mard kinhe gsend is I ubmes yat ay to s fot cond dat mill hinc\n",
      "OQ an mr thans, sheaperof suifor porell.\n",
      "\n",
      "BKOANLARTA:\n",
      "AZd dency thul thas !y, rold in af Greard to thult \n"
     ]
    }
   ],
   "source": [
    "# Let's test again the sample_text function on this very untrained RNN\n",
    "print(sample_text(data, 200))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now train the flavour-2 RNN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start training RNN...\n",
      ",\n",
      "Dhit ipsendvime indese?\n",
      "\n",
      "Th ttaout, syold fi, hin, tho'r ons hompcone\n",
      "or fo su wars;\n",
      "kne!e, the!;\n",
      "\n",
      "dt porefou woven as,\n",
      "MI I bace coand sfity amyou: heronl,,\n",
      "CCalergollI hafom.\n",
      "\n",
      "CORIOLANUNh:\n",
      "Mef\n",
      "Oy a\n",
      "sgt:\n",
      "Fer ceanr poristins wor omn;inhalg fise briplne,\n",
      "Wo besir:\n",
      "Fa niss-iend hadr comet tan toos sol\n",
      "d afand baclat lact ford por:\n",
      "indest farer lstere lave npuellom hurner costhou mothe.\n",
      "For tree fore \n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-18-36247eaf6e7f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      9\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mX\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0mNone\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0my\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m             \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m         \u001b[0mcost\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     12\u001b[0m         \u001b[0mb\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1.0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     13\u001b[0m         \u001b[0mcounter\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/cedricdeboom/Documents/UGent/PhD/Onderzoek/Theano/theano/compile/function_module.pyc\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    857\u001b[0m         \u001b[0mt0_fn\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtime\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    858\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 859\u001b[0;31m             \u001b[0moutputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    860\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    861\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'position_of_error'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/cedricdeboom/Documents/UGent/PhD/Onderzoek/Theano/theano/gof/op.pyc\u001b[0m in \u001b[0;36mrval\u001b[0;34m(p, i, o, n)\u001b[0m\n\u001b[1;32m    912\u001b[0m             \u001b[0;31m# default arguments are stored in the closure of `rval`\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    913\u001b[0m             \u001b[0;32mdef\u001b[0m \u001b[0mrval\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mp\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnode_input_storage\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mo\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnode_output_storage\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 914\u001b[0;31m                 \u001b[0mr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mx\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mo\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    915\u001b[0m                 \u001b[0;32mfor\u001b[0m \u001b[0mo\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mnode\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutputs\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    916\u001b[0m                     \u001b[0mcompute_map\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mo\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/cedricdeboom/Documents/UGent/PhD/Onderzoek/Theano/theano/tensor/blas.pyc\u001b[0m in \u001b[0;36mperform\u001b[0;34m(self, node, inp, out)\u001b[0m\n\u001b[1;32m   1550\u001b[0m         \u001b[0mz\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mout\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1551\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1552\u001b[0;31m             \u001b[0mz\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnumpy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnumpy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1553\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mValueError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1554\u001b[0m             \u001b[0;31m# The error raised by numpy has no shape information, we mean to\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "# Train procedure\n",
    "print(\"Start training RNN...\")\n",
    "counter = 0\n",
    "for epoch in range(opts.n_epochs):\n",
    "    cost  = 0.0\n",
    "    b = 0.0\n",
    "    while True:\n",
    "        X, y = get_batch(data, int(b), opts.batch_size)\n",
    "        if X is None or y is None:\n",
    "            break\n",
    "        cost += train(X, y)\n",
    "        b += 1.0\n",
    "        counter += 1   \n",
    "        if counter % 1000 == 0:\n",
    "            print(sample_text(data, 100))\n",
    "    print(\"\\nEpoch {} with {} batches, cost = {}\\n\".format(epoch + 1, int(b), cost / b))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Perplexity\n",
    "\n",
    "Before moving on to the third flavour, let's evaluate our model.\n",
    "Language models are often evaluated through a measure called perplexity. For a word language model, perplexity is defined as follows:\n",
    "$$\n",
    "\\mathrm{Perplexity} = \\exp\\left(\\frac{-\\sum_{k=1}^{N}\\log \\mathrm{P}(w_k|w_{1:k-1})}{N} \\right)\n",
    "$$\n",
    "... in which $N$ is the sequence length. In our case we don't use words, but characters. However, perplexity can still be calculated on a separate test set. In the case of sequence2sample prediction, we can simply use the Theano function `sample()` we compiled before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def raw_perplexity(input_data, input_targets):\n",
    "    # Assume input_data.shape = (batch_size, seq_len, vocab_size)\n",
    "    # Assume input_targets.shape = (batch_size, vocab_size)\n",
    "    batch_size = input_data.shape[0]\n",
    "    \n",
    "    num = 0\n",
    "    counter = 0\n",
    "    for b in xrange(batch_size):\n",
    "        sample_output = sample(np.expand_dims(input_data[b], axis=0))\n",
    "        p = np.sum(sample_output * input_targets[b])\n",
    "        num -= np.log(p)\n",
    "        counter += 1\n",
    "    \n",
    "    return num, counter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's test it for a sample batch. Lower perplexity is always better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11.050741546\n"
     ]
    }
   ],
   "source": [
    "ba_X, ba_y = get_batch(data, 10, opts.batch_size)\n",
    "num, den = raw_perplexity(ba_X, ba_y)\n",
    "print(np.exp(num / den))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now add perplexity evaluation during model training. We will train on a specified train set, and evaluate perplexity on the test set. Every 500 batches we see that the perplexity on the test set indeed decreases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start training RNN...\n",
      "le quy beand prascirsely thoupc; ints pavinsd were, linor omemud\n",
      "Thice poor butuut cruc!\n",
      "Thu efos 'p\n",
      "Perplexity: 10.6966303961\n",
      "\n",
      "g borfons wer.\n",
      "\n",
      "HENENIUS:\n",
      "Hot shane baod torst! comelforisef tho reTporgis you han, ard be lut to pf\n",
      "Perplexity: 10.4050989616\n",
      "\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-78-6e171c71f900>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     13\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mX\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0mNone\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0my\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     14\u001b[0m             \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 15\u001b[0;31m         \u001b[0mcost\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     16\u001b[0m         \u001b[0mb\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1.0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     17\u001b[0m         \u001b[0mcounter\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/cedricdeboom/Documents/UGent/PhD/Onderzoek/Theano/theano/compile/function_module.pyc\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    857\u001b[0m         \u001b[0mt0_fn\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtime\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    858\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 859\u001b[0;31m             \u001b[0moutputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    860\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    861\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'position_of_error'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/cedricdeboom/Documents/UGent/PhD/Onderzoek/Theano/theano/scan_module/scan_op.pyc\u001b[0m in \u001b[0;36mrval\u001b[0;34m(p, i, o, n, allow_gc)\u001b[0m\n\u001b[1;32m    947\u001b[0m         \u001b[0mallow_gc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mconfig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mallow_gc\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mallow_gc\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    948\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 949\u001b[0;31m         def rval(p=p, i=node_input_storage, o=node_output_storage, n=node,\n\u001b[0m\u001b[1;32m    950\u001b[0m                  allow_gc=allow_gc):\n\u001b[1;32m    951\u001b[0m             \u001b[0mr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mx\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mo\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "train_portion = 0.99\n",
    "train_data = data[0:int(train_portion*len(data))]\n",
    "test_data = data[int(train_portion*len(data)):]\n",
    "\n",
    "# Train procedure\n",
    "print(\"Start training RNN...\")\n",
    "counter = 0\n",
    "for epoch in range(opts.n_epochs):\n",
    "    cost  = 0.0\n",
    "    b = 0.0\n",
    "    while True:\n",
    "        X, y = get_batch(train_data, int(b), opts.batch_size)\n",
    "        if X is None or y is None:\n",
    "            break\n",
    "        cost += train(X, y)\n",
    "        b += 1.0\n",
    "        counter += 1   \n",
    "        if counter % 500 == 0:\n",
    "            # Sample using test_data\n",
    "            print(sample_text(test_data, 100))\n",
    "            # Evaluate perplexity\n",
    "            num, den = 0.0, 0.0\n",
    "            tb = 0\n",
    "            while True:\n",
    "                Xt, yt = get_batch(test_data, tb, 1000)\n",
    "                if Xt is None or yt is None:\n",
    "                    break\n",
    "                n2, d2 = raw_perplexity(Xt, yt)\n",
    "                num += n2\n",
    "                den += d2\n",
    "                tb += 1\n",
    "            print(\"Perplexity: \" + str(np.exp(num/den)) + '\\n')\n",
    "    print(\"\\nEpoch {} with {} batches, cost = {}\\n\".format(epoch + 1, int(b), cost / b))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Flavour 3 - seq2seq training, sample2sample prediction\n",
    "\n",
    "In the third flavour of the CharRNN we use again the sequence 2 sequence training procedure. To sample text from the model, we will do this sample by sample, while keeping track of the hidden and cell states of the LSTM layer in between samples. This is probably the most logical method to sample from an RNN."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implementing a sample2sample prediction in lasagne is not straightforward. The standard implementation of the `get_output_for()` function in the `LSTMLayer` starts always starts with the initial (learnt or given) hidden and cell states and only returns the final hidden state to be used in the next layer. This final hidden state is however not stored, nor is the final cell state. In other words, if we sample from the model, we will always start with initial hidden and cell states. To mitigate this, we will use extra input layers for these initial states. Furthermore, the `get_output_for()` function only return the final hidden state, while we also need the final cell state. For this purpose, I created an adapted version of the `LSTMLayer`, which I called `LSTMLayer_v2`, and in which I copied the `get_output_for()` function so that now also the cell state is returned. The consequence is that we cannot use the helper function `get_output()` anymore, so we will have to go through the network manually..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import LSTMLayer_v2\n",
    "LSTMLayer_v2 = reload(LSTMLayer_v2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Hold options as static element in the opts class\n",
    "class opts():\n",
    "    hidden_size = 50\n",
    "    seq_len = 25         # Data sequence length\n",
    "    gradient_steps = 20  # Truncated BPTT length\n",
    "    data_offset = 15     # Offset for every new input sequence\n",
    "    batch_size = 50\n",
    "    n_epochs = 100\n",
    "    lr = 0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train the RNN we will use the `build_rnn_1()` function from before. To sample from the RNN, we will use a different model with the `LSTMLayer_v2` (and we will load the parameters from the train model at prediction time). We explicitly turn off the `learn_init` argument, and we will use extra input layers to provide the initial hidden and cell states."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def build_test_rnn_1(input_var, hid_var, cell_var, dim):\n",
    "    # ----- INPUT LAYER -----\n",
    "    l_in = lasagne.layers.InputLayer(shape=(1, 1, dim), input_var=input_var)\n",
    "    \n",
    "    io_gate_parameters = lasagne.layers.recurrent.Gate(\n",
    "        W_in=lasagne.init.Orthogonal(), W_hid=lasagne.init.Orthogonal(),\n",
    "        b=lasagne.init.Constant(0.))\n",
    "    \n",
    "    forget_gate_parameters = lasagne.layers.recurrent.Gate(\n",
    "        W_in=lasagne.init.Orthogonal(), W_hid=lasagne.init.Orthogonal(),\n",
    "        b=lasagne.init.Constant(5.))\n",
    "    \n",
    "    cell_parameters = lasagne.layers.recurrent.Gate(\n",
    "        W_in=lasagne.init.Orthogonal(), W_hid=lasagne.init.Orthogonal(),\n",
    "        W_cell=None, b=lasagne.init.Constant(0.),\n",
    "        nonlinearity=lasagne.nonlinearities.tanh)\n",
    "    \n",
    "    # ----- LSTM LAYER -----\n",
    "    # Here we initialize extra input layers for the initial hidden and cell states\n",
    "    l_hid = lasagne.layers.InputLayer(shape=(1, opts.hidden_size), input_var=hid_var)\n",
    "    l_cell = lasagne.layers.InputLayer(shape=(1, opts.hidden_size), input_var=cell_var)\n",
    "    l_lstm = LSTMLayer_v2.LSTMLayer_v2(\n",
    "        l_in, opts.hidden_size,\n",
    "        ingate=io_gate_parameters, forgetgate=forget_gate_parameters,\n",
    "        cell=cell_parameters, outgate=io_gate_parameters,\n",
    "        hid_init=hid_var, cell_init=cell_var,\n",
    "        learn_init=False, grad_clipping=50., gradient_steps=opts.gradient_steps)\n",
    "    \n",
    "    # ----- SLICE LAYER -----\n",
    "    l_slice = lasagne.layers.SliceLayer(l_lstm, indices=0, axis=1)\n",
    "    \n",
    "    # ----- FC LAYER -----\n",
    "    l_dense = lasagne.layers.DenseLayer(\n",
    "        l_slice, num_units=dim, nonlinearity=lasagne.nonlinearities.softmax)\n",
    "    \n",
    "    return l_dense, l_lstm, l_hid, l_cell"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary size = 65; total data size = 1115394\n"
     ]
    }
   ],
   "source": [
    "data = open('tinyshakespeare.txt', 'r').read()\n",
    "chars = list(set(data))\n",
    "data_size, vocab_size = len(data), len(chars)\n",
    "print ('Vocabulary size = ' + str(vocab_size) + '; total data size = ' + str(data_size))\n",
    "char_to_ix = { ch:i for i,ch in enumerate(chars) }\n",
    "ix_to_char = { i:ch for i,ch in enumerate(chars) }\n",
    "\n",
    "# Define function to get batches of preprocessed data.\n",
    "def get_batch(data, b, b_size):\n",
    "    if (b+1)*b_size*opts.data_offset - opts.data_offset + opts.seq_len + 1 >= len(data):\n",
    "        return None, None\n",
    "    X = np.zeros((b_size, opts.seq_len, vocab_size), dtype=theano.config.floatX)\n",
    "    y = np.zeros((b_size, opts.seq_len, vocab_size), dtype=np.int8)\n",
    "    \n",
    "    for i in xrange(b_size):\n",
    "        c = b*opts.data_offset*b_size + opts.data_offset*i\n",
    "        for j in xrange(opts.seq_len):\n",
    "            X[i, j, char_to_ix[data[c]]] = 1.0\n",
    "            y[i, j, char_to_ix[data[c+1]]] = 1.0\n",
    "            c += 1\n",
    "    \n",
    "    return X, y.reshape((b_size*opts.seq_len, vocab_size))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Main program."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "input_var = T.tensor3('inputs')\n",
    "output_var = T.bmatrix('outputs') # the outputs will be flattened over 1st and 2nd dimension to reflect\n",
    "                                 # dense layer output\n",
    "\n",
    "network = build_rnn_1(input_var, vocab_size)\n",
    "# Now predict the cost of batch in terms of a loss function\n",
    "network_output = lasagne.layers.get_output(network)\n",
    "loss = lasagne.objectives.categorical_crossentropy(network_output, output_var).mean()\n",
    "# Retrieve all network params\n",
    "all_params = lasagne.layers.get_all_params(network)\n",
    "# Compute adam updates for training\n",
    "updates = lasagne.updates.adam(loss, all_params)\n",
    "# Theano function for training and computing cost\n",
    "train = theano.function(\n",
    "    [input_var, output_var],\n",
    "    loss, updates=updates)\n",
    "\n",
    "# hold the initial hidden state of LSTM layer\n",
    "hid_var = theano.shared(np.zeros((1,opts.hidden_size), dtype=theano.config.floatX))\n",
    "# hold the initial cell state of LSTM layer\n",
    "cell_var = theano.shared(np.zeros((1,opts.hidden_size), dtype=theano.config.floatX))\n",
    "\n",
    "test_network, test_lstm_layer, test_hid, test_cell = build_test_rnn_1(input_var, hid_var, cell_var, vocab_size)\n",
    "test_network_output = lasagne.layers.get_output(test_network)\n",
    "\n",
    "# Theano function to sample from the RNN at test/prediction time\n",
    "# Make sure that also the final hidden and cell states are returned\n",
    "from collections import OrderedDict\n",
    "hid_out, cell_out = test_lstm_layer.get_hid_cell_for([input_var, test_hid.input_var, test_cell.input_var])\n",
    "# The following is essential, as it updates the hidden and cell state variables\n",
    "# to the hidden and cell outputs after the prediction.\n",
    "sample_updates = OrderedDict()\n",
    "sample_updates[hid_var] = hid_out[-1, :, :]\n",
    "sample_updates[cell_var] = cell_out[-1, :, :]\n",
    "sample = theano.function(\n",
    "    [input_var],\n",
    "    test_network_output[-1,:],\n",
    "    updates=sample_updates)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def sample_text(data, length=200):\n",
    "    # First take a random piece of bootstrap text\n",
    "    start = np.random.randint(0, len(data)-opts.seq_len)\n",
    "    s = data[start:start+opts.seq_len]\n",
    "    \n",
    "    # Convert to proper input data shape (here, batch size = 1)\n",
    "    s_np = np.zeros((1, opts.seq_len, vocab_size), dtype=theano.config.floatX)\n",
    "    for i in xrange(opts.seq_len):\n",
    "        s_np[0, i, char_to_ix[s[i]]] = 1.0\n",
    "        \n",
    "    # Feed into network sequentially and ignore all but last sample (bootstrapping hidden/cell states)\n",
    "    for char in xrange(opts.seq_len):\n",
    "        predict = sample(s_np[:, [char], :])\n",
    "    \n",
    "    # Start sampling loop\n",
    "    res = ''\n",
    "    predict_i = np.random.choice(range(vocab_size), p=predict.ravel())\n",
    "    s_np = np.zeros((1, 1, vocab_size), dtype=theano.config.floatX)\n",
    "    s_np[0, 0, predict_i] = 1.0\n",
    "    for k in xrange(length):\n",
    "        # Predict next character\n",
    "        predict = sample(s_np)\n",
    "        predict_i = np.random.choice(range(vocab_size), p=predict.ravel())\n",
    "        res += ix_to_char[predict_i]\n",
    "        \n",
    "        # Update s_np\n",
    "        s_np[0, 0, :] = 0.0\n",
    "        s_np[0, 0, predict_i] = 1.0\n",
    "    \n",
    "    return res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iddsb$wYsJOG!J!bLjTni;3phGdCc Y-sqeQomoJ;jXvSuZvSvv'ezT-;ir3g MDla\n",
      "RfWaY3iIw:eItmgv vYtzKALtTR3Ysi dSCCMd iDmGsyCmG\n",
      "$SOosKq: UDDCm:T.r$symVOidDo:z'u!mOs-sTzVXjov:ZDgC 'pFFiTMa\n",
      "GijstoMDMWSD.TQ?!!$'FmD \n"
     ]
    }
   ],
   "source": [
    "# Let's test again the sample_text function on an untrained RNN\n",
    "print(sample_text(data, 200))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "We now have two networks, one to train and one to sample from. Prior to sampling it is important to set the parameters of the test network to the parameter values of the train network. This can be done through `lasagne.layers.set_all_param_values(test_network, lasagne.layers.get_all_param_values(network))`. This will also set the `hid_init` and `cell_init` values - which are learnt in the train network - correctly in the test network! This is demonstrated below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 5.  5.  5.  5.  5.  5.  5.  5.  5.  5.  5.  5.  5.  5.  5.  5.  5.  5.\n",
      "   5.  5.  5.  5.  5.  5.  5.  5.  5.  5.  5.  5.  5.  5.  5.  5.  5.  5.\n",
      "   5.  5.  5.  5.  5.  5.  5.  5.  5.  5.  5.  5.  5.  5.]]\n",
      "[[ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.\n",
      "   0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.\n",
      "   0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]]\n"
     ]
    }
   ],
   "source": [
    "hid_var.set_value(np.ones((1,50))*5.0)\n",
    "print(hid_var.get_value())\n",
    "lasagne.layers.set_all_param_values(test_network, lasagne.layers.get_all_param_values(network))\n",
    "print(hid_var.get_value())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's combine training and sampling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start training RNN...\n",
      "\n",
      "Epoch 1 with 1472 batches, cost = 2.66165176926\n",
      "\n",
      "owI AIO:\n",
      "euh gherethes fwodh sasthi'tong;\n",
      "Bus andl'btd.\n",
      "\n",
      "TPrORI:\n",
      "y what noo wirint, nor wiwe the tha\n",
      "\n",
      "Epoch 2 with 1472 batches, cost = 2.27936997562\n",
      "\n",
      "ald t.\n",
      "NKVAnITLE:\n",
      "Soot hhat mave gith:\n",
      "Whind hasl, o cime whas ald steparet you teppoirtn tobe sinnt\n",
      "\n",
      "Epoch 3 with 1472 batches, cost = 2.15341372483\n",
      "\n",
      "urd.\n",
      "\n",
      "LUTENTBANO:\n",
      "Why lith the wipine pen's tase lothat will bot;\n",
      "What ship me brast belt rammel som\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-97-89ad410bdee2>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     12\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mX\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0mNone\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0my\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     13\u001b[0m             \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 14\u001b[0;31m         \u001b[0mcost\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     15\u001b[0m         \u001b[0mb\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1.0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     16\u001b[0m     \u001b[0;32mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"\\nEpoch {} with {} batches, cost = {}\\n\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mepoch\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcost\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0mb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/cedricdeboom/Documents/UGent/PhD/Onderzoek/Theano/theano/compile/function_module.pyc\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    857\u001b[0m         \u001b[0mt0_fn\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtime\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    858\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 859\u001b[0;31m             \u001b[0moutputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    860\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    861\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'position_of_error'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/cedricdeboom/Documents/UGent/PhD/Onderzoek/Theano/theano/gof/op.pyc\u001b[0m in \u001b[0;36mrval\u001b[0;34m(p, i, o, n)\u001b[0m\n\u001b[1;32m    912\u001b[0m             \u001b[0;31m# default arguments are stored in the closure of `rval`\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    913\u001b[0m             \u001b[0;32mdef\u001b[0m \u001b[0mrval\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mp\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnode_input_storage\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mo\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnode_output_storage\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 914\u001b[0;31m                 \u001b[0mr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mx\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mo\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    915\u001b[0m                 \u001b[0;32mfor\u001b[0m \u001b[0mo\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mnode\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutputs\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    916\u001b[0m                     \u001b[0mcompute_map\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mo\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/cedricdeboom/Documents/UGent/PhD/Onderzoek/Theano/theano/tensor/blas.pyc\u001b[0m in \u001b[0;36mperform\u001b[0;34m(self, node, inp, out)\u001b[0m\n\u001b[1;32m   1550\u001b[0m         \u001b[0mz\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mout\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1551\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1552\u001b[0;31m             \u001b[0mz\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnumpy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnumpy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1553\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mValueError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1554\u001b[0m             \u001b[0;31m# The error raised by numpy has no shape information, we mean to\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Python/2.7/site-packages/numpy/core/numeric.pyc\u001b[0m in \u001b[0;36masarray\u001b[0;34m(a, dtype, order)\u001b[0m\n\u001b[1;32m    404\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    405\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 406\u001b[0;31m \u001b[0;32mdef\u001b[0m \u001b[0masarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0morder\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    407\u001b[0m     \"\"\"Convert the input to an array.\n\u001b[1;32m    408\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "train_portion = 0.99\n",
    "train_data = data[0:int(train_portion*len(data))]\n",
    "test_data = data[int(train_portion*len(data)):]\n",
    "\n",
    "# Train procedure\n",
    "print(\"Start training RNN...\")\n",
    "for epoch in range(opts.n_epochs):\n",
    "    cost  = 0.0\n",
    "    b = 0.0\n",
    "    while True:\n",
    "        X, y = get_batch(train_data, int(b), opts.batch_size)\n",
    "        if X is None or y is None:\n",
    "            break\n",
    "        cost += train(X, y)\n",
    "        b += 1.0\n",
    "    print(\"\\nEpoch {} with {} batches, cost = {}\\n\".format(epoch + 1, int(b), cost / b))\n",
    "    # Update parameter values\n",
    "    lasagne.layers.set_all_param_values(test_network, lasagne.layers.get_all_param_values(network))\n",
    "    # Sample using test_data\n",
    "    print(sample_text(test_data, 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Flavour 4 - seq2seq training with state remembrance, sample2sample prediction\n",
    "\n",
    "In the final flavour of the CharRNN we will again use the sample 2 sample prediction from above; that is, keeping track of the hidden and cell states while sampling a new character. In the training procedure we will now adopt a same kind of logic. We will train sequence 2 sequence, e.g. data[0:25] -> data[1:26], but now we will rememeber the hidden state at time step 1. Next, using this hidden state, we train data[1:26] -> data[2:27] and we remember hidden state 2, etc. We can of course always increase the data offset in this scheme. Note that this training procedure is generally only applicable in a context of batch size 1. This procedure is the most closely related one to Karpathy's charRNN gist. The advantage is that we can propagate the gradient through the entire input sequence safely since we do not need to bootstrap the hidden states."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will again use our LSTMLayer_v2 to be able to update the hidden and cell state vectors, but this time also in the train network. This time we will therefore not learn the initial states."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import LSTMLayer_v2\n",
    "LSTMLayer_v2 = reload(LSTMLayer_v2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Hold options as static element in the opts class\n",
    "class opts():\n",
    "    hidden_size = 50\n",
    "    seq_len = 25         # Data sequence length\n",
    "    gradient_steps = 25  # Truncated BPTT length\n",
    "    data_offset = 25     # Offset for every new input sequence\n",
    "    batch_size = 1\n",
    "    n_epochs = 100\n",
    "    lr = 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def build_rnn_4(input_var, hid_var, cell_var, dim):\n",
    "    # ----- INPUT LAYER -----\n",
    "    l_in = lasagne.layers.InputLayer(shape=(1, opts.seq_len, dim), input_var=input_var)\n",
    "    \n",
    "    io_gate_parameters = lasagne.layers.recurrent.Gate(\n",
    "        W_in=lasagne.init.Orthogonal(), W_hid=lasagne.init.Orthogonal(),\n",
    "        b=lasagne.init.Constant(0.))\n",
    "    \n",
    "    forget_gate_parameters = lasagne.layers.recurrent.Gate(\n",
    "        W_in=lasagne.init.Orthogonal(), W_hid=lasagne.init.Orthogonal(),\n",
    "        b=lasagne.init.Constant(5.))\n",
    "    \n",
    "    cell_parameters = lasagne.layers.recurrent.Gate(\n",
    "        W_in=lasagne.init.Orthogonal(), W_hid=lasagne.init.Orthogonal(),\n",
    "        # Setting W_cell to None denotes that no cell connection will be used.\n",
    "        W_cell=None, b=lasagne.init.Constant(0.),\n",
    "        # By convention, the cell nonlinearity is tanh in an LSTM.\n",
    "        nonlinearity=lasagne.nonlinearities.tanh)\n",
    "    \n",
    "    # ----- LSTM LAYER -----\n",
    "    l_hid = lasagne.layers.InputLayer(shape=(1, opts.hidden_size), input_var=hid_var)\n",
    "    l_cell = lasagne.layers.InputLayer(shape=(1, opts.hidden_size), input_var=cell_var)\n",
    "    l_lstm = LSTMLayer_v2.LSTMLayer_v2(\n",
    "        l_in, opts.hidden_size,\n",
    "        ingate=io_gate_parameters, forgetgate=forget_gate_parameters,\n",
    "        cell=cell_parameters, outgate=io_gate_parameters,\n",
    "        cell_init=l_cell, hid_init=l_hid,\n",
    "        learn_init=False, grad_clipping=50., gradient_steps=opts.gradient_steps)\n",
    "    \n",
    "    # ----- FC LAYER -----\n",
    "    l_reshape = lasagne.layers.ReshapeLayer(l_lstm, (1 * opts.seq_len, opts.hidden_size))\n",
    "    l_dense = lasagne.layers.DenseLayer(\n",
    "        l_reshape, num_units=dim, nonlinearity=lasagne.nonlinearities.softmax)\n",
    "    # The output size of the network is thus (opts.seq_len, dim)\n",
    "    \n",
    "    return l_dense, l_lstm, l_hid, l_cell"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary size = 65; total data size = 1115394\n"
     ]
    }
   ],
   "source": [
    "data = open('tinyshakespeare.txt', 'r').read()\n",
    "chars = list(set(data))\n",
    "data_size, vocab_size = len(data), len(chars)\n",
    "print ('Vocabulary size = ' + str(vocab_size) + '; total data size = ' + str(data_size))\n",
    "char_to_ix = { ch:i for i,ch in enumerate(chars) }\n",
    "ix_to_char = { i:ch for i,ch in enumerate(chars) }\n",
    "\n",
    "# Define function to get batches of preprocessed data.\n",
    "def get_batch(data, b, b_size):\n",
    "    if (b+1)*b_size*opts.data_offset - opts.data_offset + opts.seq_len + 1 >= len(data):\n",
    "        return None, None\n",
    "    X = np.zeros((b_size, opts.seq_len, vocab_size), dtype=theano.config.floatX)\n",
    "    y = np.zeros((b_size, opts.seq_len, vocab_size), dtype=np.int8)\n",
    "    \n",
    "    for i in xrange(b_size):\n",
    "        c = b*opts.data_offset*b_size + opts.data_offset*i\n",
    "        for j in xrange(opts.seq_len):\n",
    "            X[i, j, char_to_ix[data[c]]] = 1.0\n",
    "            y[i, j, char_to_ix[data[c+1]]] = 1.0\n",
    "            c += 1\n",
    "    \n",
    "    return X, y.reshape((b_size*opts.seq_len, vocab_size))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Main program."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "input_var = T.tensor3('inputs')\n",
    "output_var = T.bmatrix('outputs') # the outputs will be flattened over 1st and 2nd dimension to reflect\n",
    "                                 # dense layer output\n",
    "\n",
    "# hold the initial hidden state of LSTM layer\n",
    "hid_var = theano.shared(np.zeros((1,opts.hidden_size), dtype=theano.config.floatX))\n",
    "# hold the initial cell state of LSTM layer\n",
    "cell_var = theano.shared(np.zeros((1,opts.hidden_size), dtype=theano.config.floatX))\n",
    "\n",
    "network, l_lstm, l_hid, l_cell = build_rnn_4(input_var, hid_var, cell_var, vocab_size)\n",
    "# Now predict the cost of batch in terms of a loss function\n",
    "network_output = lasagne.layers.get_output(network)\n",
    "loss = lasagne.objectives.categorical_crossentropy(network_output, output_var).mean()\n",
    "# Retrieve all network params\n",
    "all_params = lasagne.layers.get_all_params(network, trainable=True)\n",
    "# Compute adam updates for training\n",
    "updates = lasagne.updates.adam(loss, all_params)\n",
    "# Add extra state updates\n",
    "hid_out, cell_out = l_lstm.get_hid_cell_for([input_var, l_hid.input_var, l_cell.input_var])\n",
    "updates[hid_var] = hid_out[-1, [opts.data_offset-1], :]\n",
    "updates[cell_var] = cell_out[-1, [opts.data_offset-1], :]\n",
    "# Theano function for training and computing cost\n",
    "train = theano.function(\n",
    "    [input_var, output_var],\n",
    "    loss, updates=updates)\n",
    "\n",
    "test_network, test_lstm_layer, test_hid, test_cell = build_test_rnn_1(input_var, hid_var, cell_var, vocab_size)\n",
    "test_network_output = lasagne.layers.get_output(test_network)\n",
    "\n",
    "# Theano function to sample from the RNN at test/prediction time\n",
    "# Make sure that also the final hidden and cell states are returned\n",
    "from collections import OrderedDict\n",
    "hid_out, cell_out = test_lstm_layer.get_hid_cell_for([input_var, test_hid.input_var, test_cell.input_var])\n",
    "# The following is essential, as it updates the hidden and cell state variables\n",
    "# to the hidden and cell outputs after the prediction.\n",
    "sample_updates = OrderedDict()\n",
    "sample_updates[hid_var] = hid_out[-1, :, :]\n",
    "sample_updates[cell_var] = cell_out[-1, :, :]\n",
    "sample = theano.function(\n",
    "    [input_var],\n",
    "    test_network_output[-1,:],\n",
    "    updates=sample_updates)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the sample `sample_text()` function as above. Let's train and sample. _Do not forget to set the initial hidden and cell states to zero after each epoch. Also do not forget to store the states before sampling and restoring them after sampling (sampling adjusts these states)!!_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start training RNN...\n",
      "'ek:\n",
      "ME: orme, hore'loe con het lall 'nsine clond bdi bee.\n",
      "\n",
      "Holevicold,\n",
      "He asy lende I athe the. ur \n",
      "\n",
      " in ge;\n",
      "Wicher konet Cosed anoud,y\n",
      "Hatet the ko I Snenge\n",
      "And ang lowstato Cheilgrasces hivce Richarg\n",
      "\n",
      "BaWDY:\n",
      "rt sist Ons wans thin colloug, wind ad frar, coulen, amcne that plore cotith to cort firel wa\n",
      "\n",
      "t wisk higho\n",
      "n the the menit menblen, kals the a wrtor, whaslled, Lall worayow,\n",
      "And the then you dhe\n",
      "\n",
      "\n",
      "Anks; pnod my nilis Qake not four the knig, the an tis my lobke we elle wiar, thy me Hank mus, that\n",
      "\n",
      "The, for, what for,\n",
      "I nouls, quene shees?\n",
      "Comss.\n",
      "\n",
      "A LARLEPLAULIULIO:\n",
      "Gor.\n",
      "\n",
      "PEONWAN:\n",
      "\n",
      "TAULEO:\n",
      "Hangore\n",
      "\n",
      "hermare no nof ad, stay your us dowht,\n",
      "Phate wink beiten, and ary, if thoood tell youruesrad?\n",
      "Firy y\n",
      "\n",
      "\n",
      "ANAneeas to no weatio haong in.\n",
      "\n",
      "FRIMIO:\n",
      "Tiby tol,\n",
      "Ad it af ar at a To to ksar.\n",
      "\n",
      "ETUMIO:\n",
      "Bear\n",
      "As? I\n",
      "\n",
      "\n",
      "Epoch 1 with 44169 batches, cost = 2.16966171264\n",
      "\n",
      " he cool ablod\n",
      "Themale nare,\n",
      "Nome beepadly :\n",
      "I, all\n",
      "Threm on he sharler'tn spean: be.\n",
      "\n",
      "SINIANIUS:\n",
      "No\n",
      "\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-129-c634cd04bc1d>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     15\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mX\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0mNone\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0my\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     16\u001b[0m             \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 17\u001b[0;31m         \u001b[0mcost\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     18\u001b[0m         \u001b[0mb\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1.0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     19\u001b[0m         \u001b[0mcounter\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/cedricdeboom/Documents/UGent/PhD/Onderzoek/Theano/theano/compile/function_module.pyc\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    857\u001b[0m         \u001b[0mt0_fn\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtime\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    858\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 859\u001b[0;31m             \u001b[0moutputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    860\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    861\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'position_of_error'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/cedricdeboom/Documents/UGent/PhD/Onderzoek/Theano/theano/gof/op.pyc\u001b[0m in \u001b[0;36mrval\u001b[0;34m(p, i, o, n)\u001b[0m\n\u001b[1;32m    912\u001b[0m             \u001b[0;31m# default arguments are stored in the closure of `rval`\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    913\u001b[0m             \u001b[0;32mdef\u001b[0m \u001b[0mrval\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mp\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnode_input_storage\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mo\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnode_output_storage\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 914\u001b[0;31m                 \u001b[0mr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mx\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mo\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    915\u001b[0m                 \u001b[0;32mfor\u001b[0m \u001b[0mo\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mnode\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutputs\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    916\u001b[0m                     \u001b[0mcompute_map\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mo\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/cedricdeboom/Documents/UGent/PhD/Onderzoek/Theano/theano/tensor/blas.pyc\u001b[0m in \u001b[0;36mperform\u001b[0;34m(self, node, inp, out)\u001b[0m\n\u001b[1;32m   1550\u001b[0m         \u001b[0mz\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mout\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1551\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1552\u001b[0;31m             \u001b[0mz\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnumpy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnumpy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1553\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mValueError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1554\u001b[0m             \u001b[0;31m# The error raised by numpy has no shape information, we mean to\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "train_portion = 0.99\n",
    "train_data = data[0:int(train_portion*len(data))]\n",
    "test_data = data[int(train_portion*len(data)):]\n",
    "\n",
    "# Train procedure\n",
    "print(\"Start training RNN...\")\n",
    "for epoch in range(opts.n_epochs):\n",
    "    hid_var.set_value(np.zeros((1,opts.hidden_size)))\n",
    "    cell_var.set_value(np.zeros((1,opts.hidden_size)))\n",
    "    cost  = 0.0\n",
    "    b = 0.0\n",
    "    counter = 0\n",
    "    while True:\n",
    "        X, y = get_batch(train_data, int(b), 1)\n",
    "        if X is None or y is None:\n",
    "            break\n",
    "        cost += train(X, y)\n",
    "        b += 1.0\n",
    "        counter += 1\n",
    "        \n",
    "        if counter % 5000 == 0:\n",
    "            # Store hidden and cell states\n",
    "            states = [hid_var.get_value(), cell_var.get_value()]\n",
    "            # Update parameter values\n",
    "            lasagne.layers.set_all_param_values(test_network,\n",
    "                                                lasagne.layers.get_all_param_values(network, trainable=True),\n",
    "                                               trainable=True)\n",
    "            # Sample using test_data\n",
    "            print(sample_text(test_data, 100) + '\\n')\n",
    "            # Restore states\n",
    "            hid_var.set_value(states[0])\n",
    "            cell_var.set_value(states[1])\n",
    "            \n",
    "    print(\"\\nEpoch {} with {} batches, cost = {}\\n\".format(epoch + 1, int(b), cost / b))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}