9-Probabilistic-Word-Translation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Save2Drive](https://raw.githubusercontent.com/alahnala/AI4All2020-Michigan-NLP/master/slides/save2drive.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Word Level Translation\n",
    "\n",
    "In this notebook, you will learn how to use some simple probabilities and a corpus of parallel text from two languages to create your own word translator."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run the cell below to get setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os, sys\n",
    "IN_COLAB = 'google.colab' in sys.modules\n",
    "if IN_COLAB:\n",
    "  !rm -r AI4All2020-Michigan-NLP\n",
    "  !git clone https://github.com/alahnala/AI4All2020-Michigan-NLP.git\n",
    "  !cp -r AI4All2020-Michigan-NLP/utils/ .\n",
    "  !cp -r AI4All2020-Michigan-NLP/Data/ .\n",
    "  !cp -r AI4All2020-Michigan-NLP/slides/ .\n",
    "  !cp -r AI4All2020-Michigan-NLP/Experiment-Report-Templates/ .\n",
    "  !echo \"=== Files Copied ===\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data - we're going to start with a parallel corpus of English and Spanish sentences\n",
    "\n",
    "A parallel corpus is one that has the same text in two different languages.\n",
    "Let's load the data and take a look at the first time in the list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"Data/mt-data/eng-spa.txt\") as f:\n",
    "    english_spanish = f.read().split('\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "english_spanish[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Now let's parse the data to make a list of spanish sentences and a list of english sentences\n",
    "\n",
    "Fill in code where the comments are in the `parse_lines` function. The function should return two lists, one that has just the English sentences, and one that has just the Spanish sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm import tqdm\n",
    "\n",
    "def parse_lines(lines):\n",
    "    language1_sentences = []\n",
    "    language2_sentences = []\n",
    "    for line in tqdm(lines):\n",
    "        try:\n",
    "            '''\n",
    "            Add code here to separate the english part from the spanish part of the line\n",
    "            Then save the english and spanish part into two separate variables to add to the lists\n",
    "            Hint: try using the split function to split on tabs, which are represented with the string '\\t'\n",
    "            We only need the first two list items after splitting on tabs as the last item is extra information about\n",
    "            the source of the sentences.\n",
    "            '''\n",
    "            \n",
    "            \n",
    "        except:\n",
    "            continue\n",
    "        language1_sentences.append('''TODO''')\n",
    "        language2_sentences.append('''TODO''')\n",
    "\n",
    "    return language1_sentences, language2_sentences\n",
    "\n",
    "\n",
    "english, spanish = parse_lines(english_spanish)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# We are going to implement a function that compute pointwise mutual information\n",
    "\n",
    "\n",
    "\n",
    "Take a brief moment to read the definition of [Pointwise Mutual Information](https://en.wikipedia.org/wiki/Pointwise_mutual_information) from Wikipedia below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Save2Drive](https://raw.githubusercontent.com/alahnala/AI4All2020-Michigan-NLP/master/slides/PMI.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The main idea of PMI is that it compute the probability that two things are associated with eachother. \n",
    "In the context of translation, we want to know the probability that two words from different languages appear in the same line of the parallel corpus in their respective languages.\n",
    "\n",
    "For example, if the word \"Stop\" shows up in the English corpus and the word \"Parad\" shows up in the Spanish corpus, we want to know how often those two words appear in parallel translations in our corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "english_spanish[20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's begin by understanding what components we need are in order to compute the PMI of `token_A` and of `token_B`, where `token_A` is a word from English and `token_B` is a word from Spanish. Take a look at the cell below to understand the input to a function called `pointwise_mutual_information`, which we will write to compute the PMI of `token_A` and `token_B`. The cell below the function explains each component."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pointwise_mutual_information(token_A, token_B, A_B_probabilities, A_probabilities, B_Probabilities):\n",
    "    return"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- We need to keep track of the probabilities of A, and the probabilties of B, so these are really just unigram dictionaries for each language. We do that with `A_probabilities` and `B_Probabilities` which are dictionaries of unigram probabilities. We create these the same way we did in the Language Identification project ([see a walkthrough of that project here](https://colab.research.google.com/drive/1Pj8mL_amPoYJbb8nTpcX-EMNimQS8IN9?usp=sharing)). \n",
    "- We also need to keep track of the probabilities of token_A AND token_B, meaning the probability that token_A shows up in a parallel translation of token_B. We do this with a dictionary called `A_B_probabilities`. Computing these probabilities will be explained deeper later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Let's prepare a preprocessing function to obtain tokens\n",
    "\n",
    "- First step is to preprocessing the sentences into tokens. We've written a function for you that uses the nltk `word_tokenize` function, but feel free to modify this if you like.\n",
    "- Run the cell below to load the preprocess function and to show its output on an English test string and a Spanish test string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "nltk.download('punkt')\n",
    "from nltk.tokenize import word_tokenize\n",
    "\n",
    "def preprocess(string, language):\n",
    "    tokens = word_tokenize(string.lower(), language=language)\n",
    "    return tokens\n",
    "\n",
    "test_string = english[0]\n",
    "tokens = preprocess(test_string, 'english')\n",
    "print(test_string)\n",
    "print(tokens)\n",
    "\n",
    "test_string = spanish[0]\n",
    "tokens = preprocess(test_string, 'spanish')\n",
    "print(test_string)\n",
    "print(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "english[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Count the english unigrams and spanish unigrams\n",
    "\n",
    "- Fill in the code as detailed in the comments below to count the unigram tokens for each language.\n",
    "- If you need help, remember we did something very similar in the Language Identification project. Find where did that in the [Language Identification Walkthrough](https://colab.research.google.com/drive/1Pj8mL_amPoYJbb8nTpcX-EMNimQS8IN9?usp=sharing) for reference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "\n",
    "english_unigrams = defaultdict(lambda:0)\n",
    "\n",
    "for sentence in tqdm(english):\n",
    "    tokens = preprocess(sentence, 'english')\n",
    "    '''\n",
    "    Iterate through the tokens to add them to the english_unigrams dictionary and count them\n",
    "    '''\n",
    "        \n",
    "spanish_unigrams = defaultdict(lambda:0)\n",
    "\n",
    "for sentence in tqdm(spanish):\n",
    "    tokens = preprocess(sentence, 'spanish')\n",
    "    '''\n",
    "    Iterate through the tokens to add them to the spanish_unigrams dictionary and count them\n",
    "    '''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Compute English and Spanish unigram probabilities\n",
    "\n",
    "- Here we compute the probabilities of each token, which is the token count divided by the total tokens.\n",
    "- We've written these steps for you, so you just need to run the cell (try to understand each line of code however)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "english_unigram_probabilities = {}\n",
    "total_english_unigrams = sum(english_unigrams.values())\n",
    "for token in tqdm(english_unigrams):\n",
    "    token_count = english_unigrams[token]\n",
    "    english_unigram_probabilities[token] = token_count / total_english_unigrams\n",
    "    \n",
    "spanish_unigram_probabilities = {}\n",
    "total_spanish_unigrams = sum(spanish_unigrams.values())\n",
    "for token in tqdm(spanish_unigrams):\n",
    "    token_count = spanish_unigrams[token]\n",
    "    spanish_unigram_probabilities[token] = token_count / total_spanish_unigrams\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# To compute probabilities of encountering english unigram and spanish unigram in parallel sentences, first count the co-occurrences\n",
    "\n",
    "- Here we are creating a dictionary of dictionaries that counts the number of times that `token_A` occurs in parallel translations of sentences that have `token_B`.\n",
    "- This is the first step in building the data structure needed for the `A_B_probabilities` described earlier. \n",
    "- We are going to name this data structure `en_es_cooc_count`\n",
    "- Walk through the cell below reading the comments to understand what each line is doing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initializing a dictionary of dictionaries, with lambda:0 so that each new key automatically maps to 0.\n",
    "en_es_cooc_count = defaultdict(lambda:defaultdict(lambda:0))\n",
    "\n",
    "\n",
    "# We zip up our list of english and spanish sentences and walk through each parallel pair of english and spanish sentences\n",
    "for english_sentence, spanish_sentence in tqdm(zip(english, spanish)):\n",
    "    \n",
    "    # first we preprocess the English and Spanish sentences to obtain tokens\n",
    "    english_tokens = preprocess(english_sentence, 'english')\n",
    "    spanish_tokens = preprocess(spanish_sentence, 'spanish')\n",
    "    \n",
    "    # let's create a set so that so that we get just one instance of each token in case any tokens appear multiple times in the sentences\n",
    "    english_token_set = set(english_tokens)\n",
    "    spanish_token_set = set(spanish_tokens)\n",
    "    \n",
    "    # For each individual English token, we count the number of times it appears with each individual Spanish token in this particular parallel translation\n",
    "    for english_token in list(english_token_set):\n",
    "        for spanish_token in spanish_token_set:\n",
    "            en_es_cooc_count[english_token][spanish_token] += 1\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Now compute the probabilities\n",
    "\n",
    "- Here, we compute the probabilities from the counts we just computed\n",
    "- Walk through the lines of code reading the comments to understand how this works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initializing a dictionary of dictionaries, with lambda:0 so that each new key automatically maps to 0.\n",
    "en_es_cooc_probabilities = defaultdict(lambda:defaultdict(lambda:0))\n",
    "\n",
    "# first compute the total pairs. We iterate through each English word and add the sum of it's dictionary of Spanish token's values\n",
    "total_pairs = 0\n",
    "for en_tok in tqdm(en_es_cooc_count):\n",
    "    total_pairs += sum(en_es_cooc_count[en_tok].values())\n",
    "\n",
    "# Then we go through all of the pairs, and get the count of each pair for the top, and divide it by the total_pairs to get the probability\n",
    "for en_tok in tqdm(en_es_cooc_count):\n",
    "    for es_tok in en_es_cooc_count[en_tok]:\n",
    "        top = en_es_cooc_count[en_tok][es_tok]\n",
    "        bottom = total_pairs\n",
    "        \n",
    "        pr = top / bottom\n",
    "        en_es_cooc_probabilities[en_tok][es_tok] = pr\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Now we have all of the components we need for the PMI function\n",
    "- Now we have all of the components we need for the PMI function. \n",
    "- Add code below to complete the pointwise mutual information function to compute the PMI of token A and token B (refer back to the equation).\n",
    "- We imported a function to compute the log of a number. It's syntax is simply log(some number)\n",
    "- We included a mini test case. If your function is working correctly, then the cell should output:\n",
    "``PMI of \"go\" and \"vaya\": 18.698165913571383``\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from math import log\n",
    "\n",
    "def pointwise_mutual_information(token_A, token_B, A_B_probabilities, A_probabilities, B_Probabilities):\n",
    "\n",
    "    '''\n",
    "    Add code here to complete the function\n",
    "    '''\n",
    "    return pmi\n",
    "\n",
    "\n",
    "test_A_B_Probabilities = {'go':{'vaya':79}}\n",
    "test_A_Probabilities = {'go': 0.003110732656094365}\n",
    "test_B_Probabilities = {'vaya': 0.0001924217344287494}\n",
    "test_token_A = 'go'\n",
    "test_token_B = 'vaya'\n",
    "\n",
    "test_pmi = pointwise_mutual_information('go', 'vaya', test_A_B_Probabilities, test_A_Probabilities, test_B_Probabilities)\n",
    "print('PMI of \"go\" and \"vaya\":', test_pmi)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Run the cell below to compute the PMIs for all of the token pairs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "en_es_pmis = {}\n",
    "\n",
    "for en_tok in tqdm(en_es_cooc_count):\n",
    "    for es_tok in en_es_cooc_count[en_tok]:\n",
    "        pmi = pointwise_mutual_information(en_tok, es_tok, en_es_cooc_probabilities, english_unigram_probabilities, spanish_unigram_probabilities)\n",
    "        pair = en_tok + '\\t' + es_tok\n",
    "        en_es_pmis[pair] = pmi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Run the cell below to sort our dictionary of PMI's to see the pairs with high pmi scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import operator\n",
    "\n",
    "sorted_pmis = sorted(en_es_pmis.items(), key=operator.itemgetter(1), reverse=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Run the cell below to print out the words with the top ten highest PMI's"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for pair, pmi in sorted_pmis[:10]:\n",
    "    parts = pair.split('\\t')\n",
    "    en = parts[0]\n",
    "    es = parts[1]\n",
    "    print(en, '--->', es)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Now that you have learn how to compute PMI's, can you create a system that translate a full sentence using just these PMI's? \n",
    "\n",
    "1. Expand upon what you learned throughout all of the NLP project notebooks. \n",
    "2. Then, if you are interested, try coming up with a way to use translation in a Chatbot [open a fresh Chatbot starter here](https://colab.research.google.com/github/alahnala/AI4All2020-Michigan-NLP/blob/master/Chatbot-Starter.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}