exercise_12.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyP67Strdiz3oAGhypUIC5M3",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/TurkuNLP/intro-to-nlp/blob/master/exercise_12.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Task 12: analogy evaluation\n",
        "\n",
        "In the lecture, we touched upon *Mikolov's analogy dataset* which was one of the first analogy evaluation datasets for word embeddings. It consists of 9+5=14 sets of word analogies. You can find it for example here: https://github.com/nicholas-leonard/word2vec/blob/master/questions-words.txt\n",
        "\n",
        "It might be interesting to know how well our embeddings fare on each of these 14 tasks. And that will be our exercise. The steps are as follows:\n",
        "\n",
        "1. Read in the analogy tuples from the file above, for each task separately (the format of the file is kinda self-explanatory once you open it)\n",
        "2. Write a function `eval_analogy(tuples,embeddings,K)` which will return the top-K accuracy of the `embeddings` (Gensim's KeyedVectors) on `tuples`, which are the analogy 4-tuples. For instance for the tuple (\"Athens\",\"Greece\",\"Havana\",\"Cuba\") will be counted as correct if the analogy on the first three words results in K nearest neighbors that contain also \"Cuba\". Hope this is clear. :)\n",
        "3. Run this function on the 14 tasks you read in step (1) and see if you see any interesting differences\n",
        "\n",
        "Below is the relevant embedding-loading and analogy example code from the lecture that you can reuse.\n",
        "\n",
        "**Tip:** these do take a while to compute, so you might want to debug your code on a small sample and when happy, run the whole thing only once. I also like to use `tqdm` to get a progress bar, so I see how long I need to wait to see some output."
      ],
      "metadata": {
        "id": "AeYKb5I5bAeJ"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c3-FPp5MZjAS"
      },
      "outputs": [],
      "source": [
        "import gensim"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# I found this link in the NLPL repository\n",
        "# It refers to English model trained on the Gigaword corpus of news\n",
        "##!wget http://vectors.nlpl.eu/repository/20/12.zip\n",
        "\n",
        "## Try these if the download above is too slow, I mirrored these:\n",
        "!wget http://dl.turkunlp.org/TKO_7095_2023/12.zip\n",
        "#!wget http://dl.turkunlp.org/TKO_7095_2023/42.zip\n"
      ],
      "metadata": {
        "id": "QRbTPYZGZuiJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "626b165a-799c-458c-dd68-d9a1854ceb77"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2023-04-18 18:14:08--  http://dl.turkunlp.org/TKO_7095_2023/12.zip\n",
            "Resolving dl.turkunlp.org (dl.turkunlp.org)... 195.148.30.23\n",
            "Connecting to dl.turkunlp.org (dl.turkunlp.org)|195.148.30.23|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 613577258 (585M) [application/zip]\n",
            "Saving to: ‘12.zip’\n",
            "\n",
            "12.zip              100%[===================>] 585.15M  22.0MB/s    in 26s     \n",
            "\n",
            "2023-04-18 18:14:35 (22.1 MB/s) - ‘12.zip’ saved [613577258/613577258]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Somewhat awkwardly, these are numbered files and both\n",
        "# .zip files contain \"model.bin\"\n",
        "# Let's unzip and rename\n",
        "# -o means \"do not ask, overwrite by default\"\n",
        "!unzip -o 12.zip\n",
        "!mv model.bin en.bin\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AiV_3crkzBtI",
        "outputId": "9485e221-ba03-4c1f-d3e7-cde3fc06edc9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Archive:  12.zip\n",
            "  inflating: meta.json               \n",
            "  inflating: model.bin               \n",
            "  inflating: model.txt               \n",
            "  inflating: README                  \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "*   Now we can load the embeddings\n",
        "*   These are huge, but they are sorted by frequency, so we can easily limit ourselves to the top 100,000 words, which will be plenty enough for us\n",
        "*   This is maybe good to note, now we enter the territory of NLP models which count in the gigabytes in size\n",
        "\n"
      ],
      "metadata": {
        "id": "lL1rXODZOCYi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# This is how you load the trained embeddings\n",
        "# check the documentation\n",
        "# w2v embeddings are traditionlly distributed in one of two formats: a text form, and a binary form\n",
        "# The embeddings we downloaded above are in the binary form, so we need to indicate that when loading\n",
        "\n",
        "from gensim.models import KeyedVectors\n",
        "\n",
        "wv_emb_en=KeyedVectors.load_word2vec_format(\"en.bin\", limit=100000, binary=True)\n"
      ],
      "metadata": {
        "id": "4DbFt1VOaDCu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "`KeyedVectors` documentation is here: https://radimrehurek.com/gensim/models/keyedvectors.html"
      ],
      "metadata": {
        "id": "mEsD37OxeP2C"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Basic operations with the embeddings\n",
        "\n",
        "* The KeyedVectors object allows for all the basic operations with embeddings which we saw in the lecture\n"
      ],
      "metadata": {
        "id": "FT9CdAlCOW3q"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Word analogy\n",
        "\n",
        "* \"A is to B as C is to D\"\n",
        "* Can be implemented as D=B-A+C, where (A,B,C) are word embeddings\n",
        "* Then list words nearest to the computed embedding D\n",
        "* In the library, the implementation lets us list words with \"+\" sign, and words with \"-\" sign\n"
      ],
      "metadata": {
        "id": "XtyrScWuOokr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# B     A      C\n",
        "# Paris-France+Sweden= ___?\n",
        "#\n",
        "# i.e. France is to Paris as Sweden is to X\n",
        "wv_emb_en.most_similar(positive=[\"Paris\",\"Sweden\"],negative=[\"France\"])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7PaVLPxsOujq",
        "outputId": "0ef680e9-6244-46de-d618-ea5f4682430c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('Stockholm', 0.7338932752609253),\n",
              " ('Malmo', 0.5458161234855652),\n",
              " ('Helsinki', 0.5444940328598022),\n",
              " ('Goteborg', 0.5421050190925598),\n",
              " ('Swedish', 0.5309098362922668),\n",
              " ('Malmoe', 0.5198634266853333),\n",
              " ('Oslo', 0.5004472732543945),\n",
              " ('Gothenburg', 0.4957912266254425),\n",
              " ('STOCKHOLM', 0.48791587352752686),\n",
              " ('Copenhagen', 0.47769418358802795)]"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "triples=[(\"cow\",\"milk\",\"hen\"),\n",
        "         (\"Paris\",\"France\",\"Helsinki\"),\n",
        "         (\"car\",\"wheel\",\"airplane\"),\n",
        "         (\"airplane\",\"propeller\",\"ship\"),\n",
        "         (\"king\",\"queen\",\"man\"),\n",
        "         (\"man\",\"doctor\",\"woman\"),\n",
        "         (\"man\",\"boss\",\"woman\")\n",
        "         ]\n",
        "for what,is_to_what,as_this_is in triples:\n",
        "    # is_to_what-what+as_this_is\n",
        "    to_what=wv_emb_en.most_similar(positive=[is_to_what,as_this_is],negative=[what])[0][0]\n",
        "    print(f\"{what} is to {is_to_what} as {as_this_is} is to: {to_what}\")\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-85ZWiCmPVv9",
        "outputId": "e47ab8bc-12f3-4fd6-da3c-04989d7a6697"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "cow is to milk as hen is to: sauce\n",
            "Paris is to France as Helsinki is to: Finland\n",
            "car is to wheel as airplane is to: rudder\n",
            "airplane is to propeller as ship is to: vessel\n",
            "king is to queen as man is to: woman\n",
            "man is to doctor as woman is to: physician\n",
            "man is to boss as woman is to: bosses\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# The exercise code starts below\n",
        "\n",
        "* I will donate you a function for reading Mikolov's data, but I recommend you delete it and write your own as a further exercise\n",
        "* Reading annoying file formats is an integral part of NLP"
      ],
      "metadata": {
        "id": "BuMn_21DlLTx"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Remember you always need to download the \"raw\" link from GitHub, or else you get an HTML with the pretty-printed data, not the data itself\n",
        "!wget https://raw.githubusercontent.com/nicholas-leonard/word2vec/master/questions-words.txt"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CBx6TuWtliB-",
        "outputId": "2bfff2ae-a098-4e1e-cdbf-b5d0f1ce06f8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2023-04-18 18:24:36--  https://raw.githubusercontent.com/nicholas-leonard/word2vec/master/questions-words.txt\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 603955 (590K) [text/plain]\n",
            "Saving to: ‘questions-words.txt’\n",
            "\n",
            "\rquestions-words.txt   0%[                    ]       0  --.-KB/s               \rquestions-words.txt 100%[===================>] 589.80K  --.-KB/s    in 0.009s  \n",
            "\n",
            "2023-04-18 18:24:36 (61.2 MB/s) - ‘questions-words.txt’ saved [603955/603955]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tasks={} #A dictionary with taskname as key, and value will then be a list of 4-tuples with the analogy data\n",
        "\n",
        "with open(\"questions-words.txt\") as f:\n",
        "    for line in f:\n",
        "        line=line.rstrip(\"\\n\")\n",
        "        if not line:\n",
        "            continue #skip possible empty lines\n",
        "        if line.startswith(\": \"): #All tasks seem to start with a line like \": task-name\"\n",
        "            taskname=line[2:] #get rid of \": \"\n",
        "            tuple_list=[] #let's make a new list for the tuples and store it into the tasks dictionary\n",
        "            #then we keep filling it, until a new task starts, when a new list is created, the previous\n",
        "            #of course remains in the `tasks` dictionary\n",
        "            tasks[taskname]=tuple_list\n",
        "        else: #not a task line, so this should be a 4-word line, with words space-separated it seems\n",
        "            w1,w2,w3,w4=line.split()\n",
        "            tuple_list.append((w1,w2,w3,w4)) #let's append it and we're done\n",
        "\n",
        "print(f\"We have {len(tasks)} tasks.\")\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iT4EFRMUlwoS",
        "outputId": "df546852-b190-4f38-97ca-27549729a688"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "We have 14 tasks.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import tqdm\n",
        "\n",
        "def eval_analogy(tuples,embeddings,K):\n",
        "    #### YOUR CODE GOES HERE ########\n",
        "\n",
        "### MY results are\n",
        "# Task *gram9-plural-verbs* has top-3 accuracy of 83.45%\n",
        "# Task *capital-common-countries* has top-3 accuracy of 93.68%\n",
        "# Task *capital-world* has top-3 accuracy of 95.97%\n",
        "# Task *currency* has top-3 accuracy of 40.54%\n",
        "# Task *city-in-state* has top-3 accuracy of 63.76%\n",
        "# Task *family* has top-3 accuracy of 93.16%\n",
        "# Task *gram1-adjective-to-adverb* has top-3 accuracy of 49.25%\n",
        "# Task *gram2-opposite* has top-3 accuracy of 49.62%\n",
        "# Task *gram3-comparative* has top-3 accuracy of 95.50%\n",
        "# Task *gram4-superlative* has top-3 accuracy of 86.32%\n",
        "# Task *gram5-present-participle* has top-3 accuracy of 83.97%\n",
        "# Task *gram6-nationality-adjective* has top-3 accuracy of 95.45%\n",
        "# Task *gram7-past-tense* has top-3 accuracy of 90.45%\n",
        "# Task *gram8-plural* has top-3 accuracy of 89.04%\n",
        "# Task *gram9-plural-verbs* has top-3 accuracy of 83.45%"
      ],
      "metadata": {
        "id": "ZkD_Ky24nvqn"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}