3-simple_lstm_network_using_PyTorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LSTM Network\n",
    "\n",
    "We'll be using a Long-Short Term Memory network (LSTM) as they are commonly used in sequences modeling. An RNN takes in sequence of words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the dataset\n",
    "\n",
    "I previously do some preprocess on IMDb dataset and dump into a pickle file. I will load this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "imdb_data = pd.read_pickle('pickles/cleaned_data2.pkl')\n",
    "\n",
    "reviews = list(imdb_data.review)\n",
    "sentiments = np.array(imdb_data.sentiment)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Pre-Processing\n",
    "The first step when building a neural network model is getting your data into the proper form to feed into the network. Since we're using embedding layers, we'll need to encode each word with an integer. We'll also want to clean it up a bit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "reviews = [review.replace('\\n', ' ') for review in reviews]\n",
    "reviews = '\\n'.join([review for review in reviews])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "from string import punctuation\n",
    "\n",
    "# get rid of punctuation\n",
    "reviews = reviews.lower() # lowercase, standardize\n",
    "all_text = ''.join([c for c in reviews if c not in punctuation])\n",
    "\n",
    "# split by new lines and spaces\n",
    "reviews_split = all_text.split('\\n')\n",
    "all_text = ' '.join(reviews_split)\n",
    "\n",
    "# create a list of words\n",
    "words = all_text.split()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Encoding the words\n",
    "The embedding lookup requires that we pass in integers to our network. The easiest way to do this is to create dictionaries that map the words in the vocabulary to integers. Then we can convert each of our reviews into integers so they can be passed into the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "# feel free to use this import \n",
    "from collections import Counter\n",
    "\n",
    "\n",
    "## Build a dictionary that maps words to integers\n",
    "counts = Counter(words)\n",
    "vocab = sorted(counts, key=counts.get, reverse=True)\n",
    "vocab_to_int = {word: ii for ii, word in enumerate(vocab, 1)}\n",
    "\n",
    "## use the dict to tokenize each review in reviews_split\n",
    "## store the tokenized reviews in reviews_ints\n",
    "reviews_ints = []\n",
    "for review in reviews_split:\n",
    "    reviews_ints.append([vocab_to_int[word] for word in review.split()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test your code\n",
    "\n",
    "As a text that you've implemented the dictionary correctly, print out the number of unique words in your vocabulary and the contents of the first, tokenized review."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unique words:  108519\n",
      "\n",
      "Tokenized review: \n",
      " [[5, 234, 355, 13, 408, 2827, 192, 162, 1600, 117, 556, 107, 32, 36, 2833, 2827, 1083, 11283, 18, 519, 94, 117, 298, 29, 1338, 20, 5028, 400, 8445, 20, 581, 1702, 1060, 648, 326, 519, 2954, 237, 65, 298, 170, 2827, 6247, 303, 11080, 5455, 1810, 517, 38409, 1116, 1279, 16848, 440, 3392, 1920, 764, 1912, 1785, 872, 236, 17465, 12374, 268, 3662, 3064, 440, 282, 50259, 3468, 9565, 5904, 1073, 917, 2189, 5085, 14921, 239, 2036, 5940, 415, 6818, 6445, 50, 156, 171, 14, 39, 208, 791, 20, 603, 103, 201, 20, 14, 4, 1482, 662, 105, 280, 1038, 2216, 183, 662, 614, 662, 748, 2827, 4, 714, 111, 32, 192, 56, 132, 2833, 1383, 1735, 24, 4, 39, 1432, 13, 368, 920, 2827, 104, 7742, 268, 472, 1186, 519, 519, 5154, 2907, 1723, 162, 2604, 17466, 4346, 162, 102, 470, 12, 171, 19, 989, 692, 646, 4346, 98, 764, 4068, 603, 296, 609, 982, 764, 345, 13, 2827, 123, 138, 1792, 2609, 233, 1262, 12, 445, 3403, 389]]\n"
     ]
    }
   ],
   "source": [
    "# stats about vocabulary\n",
    "print('Unique words: ', len((vocab_to_int)))  # should ~ 74000+\n",
    "print()\n",
    "\n",
    "# print tokens in first review\n",
    "print('Tokenized review: \\n', reviews_ints[:1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Removing Outlier\n",
    "\n",
    "As an additional pre-processing step, we want to make sure that our reviews are in good shape for standard processing. That is, our network will expect a standard input text size, and so, we'll want to shape our reviews into a specific length. We'll approach this task in two main steps:\n",
    "\n",
    "1. Getting rid of extremely long or short reviews; the outliers\n",
    "1. Padding/truncating the remaining data so that we have reviews of the same length.\n",
    "\n",
    "Before we pad our review text, we should check for reviews of extremely short or long lengths; outliers that may mess with our training.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Zero-length reviews: 0\n",
      "Maximum review length: 1458\n"
     ]
    }
   ],
   "source": [
    "# outlier review stats\n",
    "review_lens = Counter([len(x) for x in reviews_ints])\n",
    "print(\"Zero-length reviews: {}\".format(review_lens[0]))\n",
    "print(\"Maximum review length: {}\".format(max(review_lens)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Padding sequences\n",
    "To deal with both short and very long reviews, we'll pad or truncate all our reviews to a specific length. For reviews shorter than some seq_length, we'll pad with 0s. For reviews longer than seq_length, we can truncate them to the first seq_length words. A good seq_length, in this case, is 200."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pad_features(reviews_ints, seq_length):\n",
    "    ''' Return features of review_ints, where each review is padded with 0's \n",
    "        or truncated to the input seq_length.\n",
    "    '''\n",
    "    \n",
    "    # getting the correct rows x cols shape\n",
    "    features = np.zeros((len(reviews_ints), seq_length), dtype=int)\n",
    "\n",
    "    # for each review, I grab that review and \n",
    "    for i, row in enumerate(reviews_ints):\n",
    "        features[i, -len(row):] = np.array(row)[:seq_length]\n",
    "    \n",
    "    return features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test your implementation!\n",
    "\n",
    "seq_length = 200\n",
    "\n",
    "features = pad_features(reviews_ints, seq_length=seq_length)\n",
    "\n",
    "## test statements - do not change - ##\n",
    "assert len(features)==len(reviews_ints), \"Your features should have as many rows as reviews.\"\n",
    "assert len(features[0])==seq_length, \"Each feature row should contain seq_length values.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training, Validation, Test\n",
    "With our data in nice shape, we'll split it into training, validation, and test sets.\n",
    "\n",
    "- We'll need to create sets for the features and the labels, train_x and train_y, for example.\n",
    "- Define a split fraction, split_frac as the fraction of data to keep in the training set. Usually this is set to 0.8 or 0.9.\n",
    "- Whatever data is left will be split in half to create the validation and testing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t\t\tFeature Shapes:\n",
      "Train set: \t\t(40000, 200) \n",
      "Validation set: \t(5000, 200) \n",
      "Test set: \t\t(5000, 200)\n"
     ]
    }
   ],
   "source": [
    "split_frac = 0.8\n",
    "\n",
    "## split data into training, validation, and test data (features and labels, x and y)\n",
    "\n",
    "split_idx = int(len(features)*split_frac)\n",
    "train_x, remaining_x = features[:split_idx], features[split_idx:]\n",
    "train_y, remaining_y = sentiments[:split_idx], sentiments[split_idx:]\n",
    "\n",
    "test_idx = int(len(remaining_x)*0.5)\n",
    "val_x, test_x = remaining_x[:test_idx], remaining_x[test_idx:]\n",
    "val_y, test_y = remaining_y[:test_idx], remaining_y[test_idx:]\n",
    "\n",
    "## print out the shapes of your resultant feature data\n",
    "print(\"\\t\\t\\tFeature Shapes:\")\n",
    "print(\"Train set: \\t\\t{}\".format(train_x.shape), \n",
    "      \"\\nValidation set: \\t{}\".format(val_x.shape),\n",
    "      \"\\nTest set: \\t\\t{}\".format(test_x.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DataLoader and Batching\n",
    "\n",
    "After creating training, test, validation data, we can create DataLoaders for this data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.utils.data import TensorDataset, DataLoader\n",
    "\n",
    "# create Tensor datasets\n",
    "train_data = TensorDataset(torch.from_numpy(train_x), torch.from_numpy(train_y))\n",
    "valid_data = TensorDataset(torch.from_numpy(val_x), torch.from_numpy(val_y))\n",
    "test_data = TensorDataset(torch.from_numpy(test_x), torch.from_numpy(test_y))\n",
    "\n",
    "# dataloaders\n",
    "batch_size = 50\n",
    "\n",
    "# make sure the SHUFFLE your training data\n",
    "train_loader = DataLoader(train_data, shuffle=True, batch_size=batch_size)\n",
    "valid_loader = DataLoader(valid_data, shuffle=True, batch_size=batch_size)\n",
    "test_loader = DataLoader(test_data, shuffle=True, batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample input size:  torch.Size([50, 200])\n",
      "Sample input: \n",
      " tensor([[    0,     0,     0,  ...,  1104,   982,   289],\n",
      "        [    0,     0,     0,  ...,   493,   480,   180],\n",
      "        [    0,     0,     0,  ...,   205,  1995,  3090],\n",
      "        ...,\n",
      "        [    0,     0,     0,  ...,   205,     7,   272],\n",
      "        [38625,  1898,   603,  ...,   789,  1275,  1531],\n",
      "        [    0,     0,     0,  ...,  6527,   257,  2053]])\n",
      "\n",
      "Sample label size:  torch.Size([50])\n",
      "Sample label: \n",
      " tensor([0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 0, 0,\n",
      "        1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 1, 1, 1,\n",
      "        0, 1])\n"
     ]
    }
   ],
   "source": [
    "# obtain one batch of training data\n",
    "dataiter = iter(train_loader)\n",
    "sample_x, sample_y = dataiter.next()\n",
    "\n",
    "print('Sample input size: ', sample_x.size()) # batch_size, seq_length\n",
    "print('Sample input: \\n', sample_x)\n",
    "print()\n",
    "print('Sample label size: ', sample_y.size()) # batch_size\n",
    "print('Sample label: \\n', sample_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Sentiment Network with PyTorch\n",
    "\n",
    "The network will have the following layers:\n",
    "\n",
    "1. An embedding layer that converts our word tokens (integers) into embeddings of a specific size.\n",
    "2. An LSTM layer defined by a hidden_state size and number of layers\n",
    "3. A fully-connected output layer that maps the LSTM layer outputs to a desired output_size\n",
    "4. A sigmoid activation layer which turns all outputs into a value 0-1; return only the last sigmoid output as the output of this network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No GPU available, training on CPU.\n"
     ]
    }
   ],
   "source": [
    "# First checking if GPU is available\n",
    "train_on_gpu = torch.cuda.is_available()\n",
    "\n",
    "if(train_on_gpu):\n",
    "    print('Training on GPU.')\n",
    "else:\n",
    "    print('No GPU available, training on CPU.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "\n",
    "class SentimentRNN(nn.Module):\n",
    "    \"\"\"\n",
    "    The RNN model that will be used to perform Sentiment analysis.\n",
    "    \"\"\"\n",
    "    def __init__(self, vocab_size, output_size, embedding_dim, hidden_dim, n_layers, drop_prob=0.5):\n",
    "        super(SentimentRNN, self).__init__()\n",
    "\n",
    "        self.output_size = output_size\n",
    "        self.n_layers = n_layers\n",
    "        self.hidden_dim = hidden_dim\n",
    "        \n",
    "        # embedding and LSTM layers\n",
    "        self.embedding = nn.Embedding(vocab_size, embedding_dim)\n",
    "        self.lstm = nn.LSTM(embedding_dim, hidden_dim, n_layers, \n",
    "                            dropout=drop_prob, batch_first=True)\n",
    "        \n",
    "        # dropout layer\n",
    "        self.dropout = nn.Dropout(0.3)\n",
    "        \n",
    "        # linear and sigmoid layers\n",
    "        self.fc = nn.Linear(hidden_dim, output_size)\n",
    "        self.sig = nn.Sigmoid()\n",
    "        \n",
    "\n",
    "    def forward(self, x, hidden):\n",
    "        \"\"\"\n",
    "        Perform a forward pass of our model on some input and hidden state.\n",
    "        \"\"\"\n",
    "        batch_size = x.size(0)\n",
    "\n",
    "        # embeddings and lstm_out\n",
    "        x = x.long()\n",
    "        embeds = self.embedding(x)\n",
    "        lstm_out, hidden = self.lstm(embeds, hidden)\n",
    "    \n",
    "        # stack up lstm outputs\n",
    "        lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)\n",
    "        \n",
    "        # dropout and fully-connected layer\n",
    "        out = self.dropout(lstm_out)\n",
    "        out = self.fc(out)\n",
    "        # sigmoid function\n",
    "        sig_out = self.sig(out)\n",
    "        \n",
    "        # reshape to be batch_size first\n",
    "        sig_out = sig_out.view(batch_size, -1)\n",
    "        sig_out = sig_out[:, -1] # get last batch of labels\n",
    "        \n",
    "        # return last sigmoid output and hidden state\n",
    "        return sig_out, hidden\n",
    "    \n",
    "    \n",
    "    def init_hidden(self, batch_size):\n",
    "        ''' Initializes hidden state '''\n",
    "        # Create two new tensors with sizes n_layers x batch_size x hidden_dim,\n",
    "        # initialized to zero, for hidden state and cell state of LSTM\n",
    "        weight = next(self.parameters()).data\n",
    "        \n",
    "        if (train_on_gpu):\n",
    "            hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().cuda(),\n",
    "                  weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().cuda())\n",
    "        else:\n",
    "            hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_(),\n",
    "                      weight.new(self.n_layers, batch_size, self.hidden_dim).zero_())\n",
    "        \n",
    "        return hidden"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Instantiate the network\n",
    "Here, we'll instantiate the network. First up, defining the hyperparameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SentimentRNN(\n",
      "  (embedding): Embedding(108520, 400)\n",
      "  (lstm): LSTM(400, 256, num_layers=2, batch_first=True, dropout=0.5)\n",
      "  (dropout): Dropout(p=0.3, inplace=False)\n",
      "  (fc): Linear(in_features=256, out_features=1, bias=True)\n",
      "  (sig): Sigmoid()\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# Instantiate the model w/ hyperparams\n",
    "vocab_size = len(vocab_to_int)+1 # +1 for the 0 padding + our word tokens\n",
    "output_size = 1\n",
    "embedding_dim = 400\n",
    "hidden_dim = 256\n",
    "n_layers = 2\n",
    "\n",
    "net = SentimentRNN(vocab_size, output_size, embedding_dim, hidden_dim, n_layers)\n",
    "\n",
    "print(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Model\n",
    "\n",
    "I'll train this model for 1 epochs since I don't have GPU. and I'll also be using a new kind of cross entropy loss, which is designed to work with a single Sigmoid output. BCELoss, or `Binary Cross Entropy Loss`, applies cross entropy loss to a single value between 0 and 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "# loss and optimization functions\n",
    "lr = 0.001\n",
    "\n",
    "criterion = nn.BCELoss()\n",
    "optimizer = torch.optim.Adam(net.parameters(), lr=lr)\n",
    "\n",
    "# training params\n",
    "\n",
    "epochs = 1\n",
    "\n",
    "counter = 0\n",
    "print_every = 100\n",
    "clip = 5 # gradient clipping\n",
    "\n",
    "# move model to GPU, if available\n",
    "if(train_on_gpu):\n",
    "    net.cuda()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1/1... Step: 100... Loss: 0.266906... Val Loss: 0.350315\n",
      "Epoch: 1/1... Step: 200... Loss: 0.356343... Val Loss: 0.330677\n",
      "Epoch: 1/1... Step: 300... Loss: 0.391675... Val Loss: 0.337420\n",
      "Epoch: 1/1... Step: 400... Loss: 0.110327... Val Loss: 0.327143\n",
      "Epoch: 1/1... Step: 500... Loss: 0.236646... Val Loss: 0.319984\n",
      "Epoch: 1/1... Step: 600... Loss: 0.299508... Val Loss: 0.306203\n",
      "Epoch: 1/1... Step: 700... Loss: 0.496564... Val Loss: 0.308981\n",
      "Epoch: 1/1... Step: 800... Loss: 0.245462... Val Loss: 0.290510\n"
     ]
    }
   ],
   "source": [
    "# Training start------------\n",
    "net.train()\n",
    "# train for some number of epochs\n",
    "for e in range(epochs):\n",
    "    # initialize hidden state\n",
    "    h = net.init_hidden(batch_size)\n",
    "\n",
    "    # batch loop\n",
    "    for inputs, labels in train_loader:\n",
    "        counter += 1\n",
    "\n",
    "        if(train_on_gpu):\n",
    "            inputs, labels = inputs.cuda(), labels.cuda()\n",
    "\n",
    "        # Creating new variables for the hidden state, otherwise\n",
    "        # we'd backprop through the entire training history\n",
    "        h = tuple([each.data for each in h])\n",
    "\n",
    "        # zero accumulated gradients\n",
    "        net.zero_grad()\n",
    "\n",
    "        # get the output from the model\n",
    "        output, h = net(inputs, h)\n",
    "\n",
    "        # calculate the loss and perform backprop\n",
    "        loss = criterion(output.squeeze(), labels.float())\n",
    "        loss.backward()\n",
    "        # `clip_grad_norm` helps prevent the exploding gradient problem in RNNs / LSTMs.\n",
    "        nn.utils.clip_grad_norm_(net.parameters(), clip)\n",
    "        optimizer.step()\n",
    "\n",
    "        # loss stats\n",
    "        if counter % print_every == 0:\n",
    "            # Get validation loss\n",
    "            val_h = net.init_hidden(batch_size)\n",
    "            val_losses = []\n",
    "            net.eval()\n",
    "            for inputs, labels in valid_loader:\n",
    "\n",
    "                # Creating new variables for the hidden state, otherwise\n",
    "                # we'd backprop through the entire training history\n",
    "                val_h = tuple([each.data for each in val_h])\n",
    "\n",
    "                if(train_on_gpu):\n",
    "                    inputs, labels = inputs.cuda(), labels.cuda()\n",
    "\n",
    "                output, val_h = net(inputs, val_h)\n",
    "                val_loss = criterion(output.squeeze(), labels.float())\n",
    "\n",
    "                val_losses.append(val_loss.item())\n",
    "\n",
    "            net.train()\n",
    "            print(\"Epoch: {}/{}...\".format(e+1, epochs),\n",
    "                  \"Step: {}...\".format(counter),\n",
    "                  \"Loss: {:.6f}...\".format(loss.item()),\n",
    "                  \"Val Loss: {:.6f}\".format(np.mean(val_losses)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing\n",
    "There are a few ways to test your network.\n",
    "\n",
    "- Test data performance\n",
    "- Inference on user-generated data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test loss: 0.293\n",
      "Test accuracy: 0.878\n"
     ]
    }
   ],
   "source": [
    "# Get test data loss and accuracy\n",
    "\n",
    "test_losses = [] # track loss\n",
    "num_correct = 0\n",
    "\n",
    "# init hidden state\n",
    "h = net.init_hidden(batch_size)\n",
    "\n",
    "net.eval()\n",
    "# iterate over test data\n",
    "for inputs, labels in test_loader:\n",
    "\n",
    "    # Creating new variables for the hidden state, otherwise\n",
    "    # we'd backprop through the entire training history\n",
    "    h = tuple([each.data for each in h])\n",
    "\n",
    "    if(train_on_gpu):\n",
    "        inputs, labels = inputs.cuda(), labels.cuda()\n",
    "    \n",
    "    # get predicted outputs\n",
    "    output, h = net(inputs, h)\n",
    "    \n",
    "    # calculate loss\n",
    "    test_loss = criterion(output.squeeze(), labels.float())\n",
    "    test_losses.append(test_loss.item())\n",
    "    \n",
    "    # convert output probabilities to predicted class (0 or 1)\n",
    "    pred = torch.round(output.squeeze())  # rounds to the nearest integer\n",
    "    \n",
    "    # compare predictions to true label\n",
    "    correct_tensor = pred.eq(labels.float().view_as(pred))\n",
    "    correct = np.squeeze(correct_tensor.numpy()) if not train_on_gpu else np.squeeze(correct_tensor.cpu().numpy())\n",
    "    num_correct += np.sum(correct)\n",
    "\n",
    "\n",
    "# -- stats! -- ##\n",
    "# avg test loss\n",
    "print(\"Test loss: {:.3f}\".format(np.mean(test_losses)))\n",
    "\n",
    "# accuracy over all test data\n",
    "test_acc = num_correct/len(test_loader.dataset)\n",
    "print(\"Test accuracy: {:.3f}\".format(test_acc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inference on a test review\n",
    "\n",
    "New text should go through the same process as training data. So I'll write a predict function for new inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "from string import punctuation\n",
    "\n",
    "def tokenize_review(test_review):\n",
    "    test_review = test_review.lower() # lowercase\n",
    "    # get rid of punctuation\n",
    "    test_text = ''.join([c for c in test_review if c not in punctuation])\n",
    "\n",
    "    # splitting by spaces\n",
    "    test_words = test_text.split()\n",
    "\n",
    "    # tokens\n",
    "    test_ints = []\n",
    "    test_ints.append([vocab_to_int[word] for word in test_words])\n",
    "\n",
    "    return test_ints\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(net, test_review, sequence_length=200):\n",
    "    \n",
    "    net.eval()\n",
    "    \n",
    "    # tokenize review\n",
    "    test_ints = tokenize_review(test_review)\n",
    "    \n",
    "    # pad tokenized sequence\n",
    "    seq_length=sequence_length\n",
    "    features = pad_features(test_ints, seq_length)\n",
    "    \n",
    "    # convert to tensor to pass into your model\n",
    "    feature_tensor = torch.from_numpy(features)\n",
    "    \n",
    "    batch_size = feature_tensor.size(0)\n",
    "    \n",
    "    # initialize hidden state\n",
    "    h = net.init_hidden(batch_size)\n",
    "    \n",
    "    if(train_on_gpu):\n",
    "        feature_tensor = feature_tensor.cuda()\n",
    "    \n",
    "    # get the output from the model\n",
    "    output, h = net(feature_tensor, h)\n",
    "    \n",
    "    # convert output probabilities to predicted class (0 or 1)\n",
    "    pred = torch.round(output.squeeze()) \n",
    "    # printing output value, before rounding\n",
    "    print('Prediction value, pre-rounding: {:.6f}'.format(output.item()))\n",
    "    \n",
    "    # print custom response\n",
    "    if(pred.item()==1):\n",
    "        print(\"Positive review detected!\")\n",
    "    else:\n",
    "        print(\"Negative review detected.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction value, pre-rounding: 0.985588\n",
      "Positive review detected!\n"
     ]
    }
   ],
   "source": [
    "# positive test review\n",
    "test_review_pos = 'This movie had the best acting and the dialogue was so good. I loved it.'\n",
    "\n",
    "# call function\n",
    "seq_length=200 # good to use the length that was trained on\n",
    "\n",
    "predict(net, test_review_pos, seq_length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction value, pre-rounding: 0.016109\n",
      "Negative review detected.\n"
     ]
    }
   ],
   "source": [
    "# negative test review\n",
    "test_review_neg = 'The worst movie I have seen; acting was terrible and I want my money back. This movie had bad acting and the dialogue was slow.'\n",
    "\n",
    "# call function\n",
    "seq_length=200 # good to use the length that was trained on\n",
    "\n",
    "predict(net, test_review_neg, seq_length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}