sequence_labelling.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.5"
    },
    "colab": {
      "name": "sequence_labelling.ipynb",
      "provenance": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/TurkuNLP/intro-to-nlp/blob/master/sequence_labelling.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B-Cv4hpadhCb"
      },
      "source": [
        "# Sequence labeling\n",
        "\n",
        "* Many classification tasks produce a sequence of predictions, rather than a single prediction\n",
        "* In this lecture we have a look at these tasks:\n",
        "  * understand how this setting differs from basic text classification\n",
        "  * how it affects our modelling\n",
        "  * test sequence classification on an example problem\n",
        "  * when done, you will be able to apply a sequence classification model to a problem\n",
        "  * you will have the necessary background to move to more complex models at a later point\n",
        "  \n",
        "* Sequence classification is best explained through several example problems:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KR59yZkKdhCj"
      },
      "source": [
        "### POS Tagging\n",
        "\n",
        "![posfig](https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/pos_voita.png?raw=1)\n",
        "\n",
        "![posfig](https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/pos_house.png?raw=1)\n",
        "\n",
        "* Every word is assigned to its part-of-speech category\n",
        "* The number of categories is potentially quite large, in this case less than 20 though (You can see them [here](https://universaldependencies.org/u/pos/index.html) by the way)\n",
        "* POS tagging is often used as a pre-processing step\n",
        "* You can also use it to pick important words as features (nouns, verbs, etc)\n",
        "* Note the context-dependence of the tags\n",
        "  * `voita` can be a verb also, `voi` can be a noun also\n",
        "  * `house` can be a noun or a verb\n",
        "  * ...\n",
        "* The tags also have a dependence among each other\n",
        "  * Many sequences are impossible or at least highly unlikely, regardless of the input\n",
        "  * In English, having seen a determiner, the likely next tag is a noun or an adjective, and e.g. a verb is extremely unlikely\n",
        "  \n",
        " * The figs come from this demo: https://turkunlp.org/finnish_nlp.html#parser\n",
        " \n",
        "  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8Vb0wBdYdhCk"
      },
      "source": [
        "### Named entity recognition\n",
        "\n",
        "![nerfig](https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/ner_demo.png?raw=1)\n",
        "\n",
        "![nerfig](https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/ner_demo_en.png?raw=1)\n",
        "\n",
        "\n",
        "* NER is usually cast as a sequence labeling problem\n",
        "* Entities are (typically) sequences of words, like `Turun Yliopisto` or `British Airways`\n",
        "* The type tells what kind of an entity we have. The list of types is usually quite restricted: `Person, Organization, Location, Product, Event, Date, Other` would be a typical list\n",
        "\n",
        "* These figs come from https://demo.allennlp.org/named-entity-recognition and a [temporary Finnish demo](http://86.50.253.19:8001/tagdemo/)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ffDYH7rMdhCl"
      },
      "source": [
        "### BIO-coding\n",
        "\n",
        "* NER and other similar tasks that involve locating multi-word entities are cast as classification of individual tokens into three groups of classes:\n",
        "\n",
        "* **B-category**: The token begins an entity of type `category`. For example `B-Person` or `B-Location`\n",
        "* **I-category**: The token continues an entity that is already started (with a `B-category`)\n",
        "* **O**: The token is not a part of any entity\n",
        "\n",
        "Here is an example from our [Finnish NER training data](https://github.com/TurkuNLP/turku-ner-corpus):\n",
        "\n",
        "```\n",
        "The\tB-PRO\n",
        "Garden\tI-PRO\n",
        "Collection\tI-PRO\n",
        "by\tO\n",
        "H&M\tB-ORG\n",
        "\n",
        "Viikonlopun\tO\n",
        "pyöritys\tO\n",
        "alkoi\tO\n",
        "H&M:n\tB-ORG\n",
        "järjestämällä\tO\n",
        "bloggaajabrunssilla\tO\n",
        "Helsingissä\tB-LOC\n",
        ".\tO\n",
        "```\n",
        "\n",
        "And here is an example from the [CoNLL 2003 English data](https://raw.githubusercontent.com/davidsbatista/NER-datasets/master/CONLL2003/train.txt)\n",
        "\n",
        "```\n",
        "-DOCSTART- -X- -X- O\n",
        "\n",
        "EU NNP B-NP B-ORG\n",
        "rejects VBZ B-VP O\n",
        "German JJ B-NP B-MISC\n",
        "call NN I-NP O\n",
        "to TO B-VP O\n",
        "boycott VB I-VP O\n",
        "British JJ B-NP B-MISC\n",
        "lamb NN I-NP O\n",
        ". . O O\n",
        "\n",
        "Peter NNP B-NP B-PER\n",
        "Blackburn NNP I-NP I-PER\n",
        "\n",
        "BRUSSELS NNP B-NP B-LOC\n",
        "1996-08-22 CD I-NP O\n",
        "```\n",
        "\n",
        "* `BIO-coding` is suitable for cases where you do not have entity nesting and overlaps\n",
        "* There are, once again, quite clear dependencies between labels regardless of the input:\n",
        "  * Exmaples of legal: `O B-Person O O`, `B-Person I-Person O O`, `B-Person B-Person`\n",
        "  * Examples of illegal: `B-Person O I-Person O`, `O O I-Person O O`, `O B-Person I-Event O`\n",
        "* Preferably, the classifier should be prevented from producing illegal BIO sequences"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hbAtJ46ldhCn"
      },
      "source": [
        "### Text segmentation\n",
        "\n",
        "* Text segmentation (splitting into tokens and sentences) is often carried out as sequence labeling\n",
        "* One would label every individual character as one of:\n",
        "  * token ends after this character\n",
        "  * sentence ends after this character\n",
        "  * inside token\n",
        "\n",
        "Example:\n",
        "\n",
        "```\n",
        "Is it you?\n",
        "\n",
        "I     inside\n",
        "s     token-break\n",
        "      token-break\n",
        "i     inside\n",
        "t     token-break\n",
        "      token-break\n",
        "y     inside\n",
        "o     inside\n",
        "u     token-break\n",
        "?     sentence-break\n",
        "```\n",
        "\n",
        "* **Note:** what, precisely, happens at spaces is somewhat implementation-dependent and you can do it in various ways, this is only one of the possibilities"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kdHt9ub4dhCp"
      },
      "source": [
        "### Zoning\n",
        "\n",
        "* In many applications, one may want to separate text into zones\n",
        "    * scientific papers may need to be separated into backround, methods, results, citations\n",
        "    * patents can be separated into background and claims\n",
        "    * ...\n",
        "* This allows for focused information retrieval, etc.\n",
        "   * e.g. when mining scientific literature for new factual statements, you may want to focus on the *Results* section\n",
        "* The BIO coding is applicable also here\n",
        "    * perhaps the unit of classification are the whole sentences or even paragraphs, not words\n",
        "    * depends on task, ie can you expect a zone to change half-way through the sentence\n",
        "\n",
        "![zoningfig](https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/zones.gif?raw=1)\n",
        "\n",
        "Figure from: https://www.cl.cam.ac.uk/~sht25/az.html"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3houVcusdhCr"
      },
      "source": [
        "# Modelling considerations in sequence classification\n",
        "\n",
        "![posfig](https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/pos_house.png?raw=1)\n",
        "\n",
        "* **Context** is of crucial importance\n",
        "* *house* has two different labels in the above sequence\n",
        "* The label depends on the context of the occurrence\n",
        "  * *in my ______ .* is quite likely a noun\n",
        "  * *can ______ you* is quite likely a verb\n",
        "* NLP methods differ in how they model the context\n",
        "  * Anything from simple left/right bag of features, perhaps marked for position...\n",
        "  * ...to complex recurrent networks like LSTM or attention-based models like the Transformer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cuCl68RTdhCt"
      },
      "source": [
        "# CoNLL-03 POS and NER data\n",
        "\n",
        "* You can get this data easily from this repository on github: https://github.com/davidsbatista/NER-datasets\n",
        "* For some reason, this got deleted in January 2022, but we can go back in commit history on GitHub and get the data anyway\n",
        "* (Remember, when you remove something from your Git repository, it still stays in the commit history!)\n",
        "* The data is here: https://github.com/davidsbatista/NER-datasets/tree/dcb6c7439a7de43abc2448bad5b1d81a47f26c0d/CONLL2003\n",
        "* Look at CONLL2003/valid.txt\n",
        "* Let us try to learn a POS tagget based on this data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zG_VzzOiehDC",
        "outputId": "28ac2fb3-0e38-4655-a47f-7c8aeb780df7"
      },
      "source": [
        "!mkdir -p CONLL2003\n",
        "!wget -nc -O CONLL2003/train.txt https://github.com/davidsbatista/NER-datasets/raw/dcb6c7439a7de43abc2448bad5b1d81a47f26c0d/CONLL2003/train.txt\n",
        "!wget -nc -O CONLL2003/valid.txt https://github.com/davidsbatista/NER-datasets/raw/dcb6c7439a7de43abc2448bad5b1d81a47f26c0d/CONLL2003/valid.txt"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2022-02-09 12:47:41--  https://github.com/davidsbatista/NER-datasets/raw/dcb6c7439a7de43abc2448bad5b1d81a47f26c0d/CONLL2003/train.txt\n",
            "Resolving github.com (github.com)... 140.82.114.4\n",
            "Connecting to github.com (github.com)|140.82.114.4|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://raw.githubusercontent.com/davidsbatista/NER-datasets/dcb6c7439a7de43abc2448bad5b1d81a47f26c0d/CONLL2003/train.txt [following]\n",
            "--2022-02-09 12:47:41--  https://raw.githubusercontent.com/davidsbatista/NER-datasets/dcb6c7439a7de43abc2448bad5b1d81a47f26c0d/CONLL2003/train.txt\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 3283418 (3.1M) [text/plain]\n",
            "Saving to: ‘CONLL2003/train.txt’\n",
            "\n",
            "CONLL2003/train.txt 100%[===================>]   3.13M  --.-KB/s    in 0.07s   \n",
            "\n",
            "2022-02-09 12:47:42 (46.0 MB/s) - ‘CONLL2003/train.txt’ saved [3283418/3283418]\n",
            "\n",
            "--2022-02-09 12:47:42--  https://github.com/davidsbatista/NER-datasets/raw/dcb6c7439a7de43abc2448bad5b1d81a47f26c0d/CONLL2003/valid.txt\n",
            "Resolving github.com (github.com)... 140.82.114.3\n",
            "Connecting to github.com (github.com)|140.82.114.3|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://raw.githubusercontent.com/davidsbatista/NER-datasets/dcb6c7439a7de43abc2448bad5b1d81a47f26c0d/CONLL2003/valid.txt [following]\n",
            "--2022-02-09 12:47:42--  https://raw.githubusercontent.com/davidsbatista/NER-datasets/dcb6c7439a7de43abc2448bad5b1d81a47f26c0d/CONLL2003/valid.txt\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.110.133, 185.199.108.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 827441 (808K) [text/plain]\n",
            "Saving to: ‘CONLL2003/valid.txt’\n",
            "\n",
            "CONLL2003/valid.txt 100%[===================>] 808.05K  --.-KB/s    in 0.05s   \n",
            "\n",
            "2022-02-09 12:47:42 (17.5 MB/s) - ‘CONLL2003/valid.txt’ saved [827441/827441]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DyE6UlWZdhCu",
        "outputId": "9aa5f44b-1e63-4337-c4fd-8b484e7f8628"
      },
      "source": [
        "# this is how you read a file of this kind\n",
        "# one item per line, empty lines between sequences\n",
        "\n",
        "from collections import namedtuple\n",
        "\n",
        "#Same as tuple but the fields are named for convenience\n",
        "#this says we have four fields\n",
        "OneWord=namedtuple(\"OneWord\",[\"word\",\"pos_label\",\"chunk_label\",\"entity_label\"])\n",
        "\n",
        "def read_conll2003(f_name):\n",
        "    \"\"\"Yield complete sentences\"\"\"\n",
        "    current_sentence=[] #This will be a list of (word,label), which we accumulate for each sentence\n",
        "    with open(f_name) as f:\n",
        "        for line in f:\n",
        "            line=line.strip() #drop whitespace\n",
        "            if line.startswith(\"-DOCSTART-\"): #let's not worry about these for the time being\n",
        "                continue\n",
        "            if not line: #sentence break\n",
        "                if current_sentence: #if we gathered a sentence, we should yield it, because a new one starts\n",
        "                    yield current_sentence #much like return, but continues past this line once the element has been consumed\n",
        "                    current_sentence=[] #...and start a new one\n",
        "                continue\n",
        "            #if we made it here, we are on a normal line\n",
        "            columns=line.split() #an actual word line\n",
        "            assert len(columns)==4 #we should have four columns, looking at the data\n",
        "            current_sentence.append(OneWord(*columns)) #* expands columns as arguments to OneWord constructor\n",
        "        else: #for ... else -> the else part is executed once, when \"for\" runs out of elements\n",
        "            if current_sentence: #yield also the last one!\n",
        "                yield current_sentence\n",
        "\n",
        "#Now just read the data in\n",
        "sentences_train=list(read_conll2003(\"CONLL2003/train.txt\"))\n",
        "sentences_dev=list(read_conll2003(\"CONLL2003/valid.txt\"))\n",
        "\n",
        "print(\"First three sentences\")\n",
        "for sent in sentences_dev[:3]:\n",
        "    print(sent)\n",
        "    print()"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "First three sentences\n",
            "[OneWord(word='CRICKET', pos_label='NNP', chunk_label='B-NP', entity_label='O'), OneWord(word='-', pos_label=':', chunk_label='O', entity_label='O'), OneWord(word='LEICESTERSHIRE', pos_label='NNP', chunk_label='B-NP', entity_label='B-ORG'), OneWord(word='TAKE', pos_label='NNP', chunk_label='I-NP', entity_label='O'), OneWord(word='OVER', pos_label='IN', chunk_label='B-PP', entity_label='O'), OneWord(word='AT', pos_label='NNP', chunk_label='B-NP', entity_label='O'), OneWord(word='TOP', pos_label='NNP', chunk_label='I-NP', entity_label='O'), OneWord(word='AFTER', pos_label='NNP', chunk_label='I-NP', entity_label='O'), OneWord(word='INNINGS', pos_label='NNP', chunk_label='I-NP', entity_label='O'), OneWord(word='VICTORY', pos_label='NN', chunk_label='I-NP', entity_label='O'), OneWord(word='.', pos_label='.', chunk_label='O', entity_label='O')]\n",
            "\n",
            "[OneWord(word='LONDON', pos_label='NNP', chunk_label='B-NP', entity_label='B-LOC'), OneWord(word='1996-08-30', pos_label='CD', chunk_label='I-NP', entity_label='O')]\n",
            "\n",
            "[OneWord(word='West', pos_label='NNP', chunk_label='B-NP', entity_label='B-MISC'), OneWord(word='Indian', pos_label='NNP', chunk_label='I-NP', entity_label='I-MISC'), OneWord(word='all-rounder', pos_label='NN', chunk_label='I-NP', entity_label='O'), OneWord(word='Phil', pos_label='NNP', chunk_label='I-NP', entity_label='B-PER'), OneWord(word='Simmons', pos_label='NNP', chunk_label='I-NP', entity_label='I-PER'), OneWord(word='took', pos_label='VBD', chunk_label='B-VP', entity_label='O'), OneWord(word='four', pos_label='CD', chunk_label='B-NP', entity_label='O'), OneWord(word='for', pos_label='IN', chunk_label='B-PP', entity_label='O'), OneWord(word='38', pos_label='CD', chunk_label='B-NP', entity_label='O'), OneWord(word='on', pos_label='IN', chunk_label='B-PP', entity_label='O'), OneWord(word='Friday', pos_label='NNP', chunk_label='B-NP', entity_label='O'), OneWord(word='as', pos_label='IN', chunk_label='B-PP', entity_label='O'), OneWord(word='Leicestershire', pos_label='NNP', chunk_label='B-NP', entity_label='B-ORG'), OneWord(word='beat', pos_label='VBD', chunk_label='B-VP', entity_label='O'), OneWord(word='Somerset', pos_label='NNP', chunk_label='B-NP', entity_label='B-ORG'), OneWord(word='by', pos_label='IN', chunk_label='B-PP', entity_label='O'), OneWord(word='an', pos_label='DT', chunk_label='B-NP', entity_label='O'), OneWord(word='innings', pos_label='NN', chunk_label='I-NP', entity_label='O'), OneWord(word='and', pos_label='CC', chunk_label='O', entity_label='O'), OneWord(word='39', pos_label='CD', chunk_label='B-NP', entity_label='O'), OneWord(word='runs', pos_label='NNS', chunk_label='I-NP', entity_label='O'), OneWord(word='in', pos_label='IN', chunk_label='B-PP', entity_label='O'), OneWord(word='two', pos_label='CD', chunk_label='B-NP', entity_label='O'), OneWord(word='days', pos_label='NNS', chunk_label='I-NP', entity_label='O'), OneWord(word='to', pos_label='TO', chunk_label='B-VP', entity_label='O'), OneWord(word='take', pos_label='VB', chunk_label='I-VP', entity_label='O'), OneWord(word='over', pos_label='IN', chunk_label='B-PP', entity_label='O'), OneWord(word='at', pos_label='IN', chunk_label='B-PP', entity_label='O'), OneWord(word='the', pos_label='DT', chunk_label='B-NP', entity_label='O'), OneWord(word='head', pos_label='NN', chunk_label='I-NP', entity_label='O'), OneWord(word='of', pos_label='IN', chunk_label='B-PP', entity_label='O'), OneWord(word='the', pos_label='DT', chunk_label='B-NP', entity_label='O'), OneWord(word='county', pos_label='NN', chunk_label='I-NP', entity_label='O'), OneWord(word='championship', pos_label='NN', chunk_label='I-NP', entity_label='O'), OneWord(word='.', pos_label='.', chunk_label='O', entity_label='O')]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qM_pu7mjdhCz"
      },
      "source": [
        "* Now we have the input data\n",
        "* Next we generate features for each word\n",
        "* These will be used to predict its POS\n",
        "* Let's start simple, the feature will be the word itself and nothing else"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LT1ASGBOdhC0",
        "outputId": "35a26f1f-2f0c-400f-dfe8-a735baadbbd7"
      },
      "source": [
        "def generate_sentence_features(sent):\n",
        "    #Given a sentence as a list of (word, label) pairs\n",
        "    #generate the features for every word\n",
        "    #The result should be a list of same length as the sentence\n",
        "    #Each item is a dictionary of {\"feature name\"->feature value} mappings, holding all features of the word at that position\n",
        "    \n",
        "    sent_features=[] #this will be the result\n",
        "    for one_word in sent:\n",
        "        #We must do nothing with label\n",
        "        #it just happens to be around\n",
        "        word_features={}\n",
        "        word_features[\"word_\"+one_word.word]=1 #the word itself is a feature\n",
        "        sent_features.append(word_features)\n",
        "    return sent_features\n",
        "\n",
        "print(generate_sentence_features(sentences_dev[0])  )"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[{'word_CRICKET': 1}, {'word_-': 1}, {'word_LEICESTERSHIRE': 1}, {'word_TAKE': 1}, {'word_OVER': 1}, {'word_AT': 1}, {'word_TOP': 1}, {'word_AFTER': 1}, {'word_INNINGS': 1}, {'word_VICTORY': 1}, {'word_.': 1}]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ePYrYsRudhC1"
      },
      "source": [
        "* The code above takes care of basic feature generation\n",
        "* For the simple classifier we will be building, we only need the sentence boundaries when generating the features\n",
        "* After that, we can flatten the data into a single stream of words"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p72UoLvxdhC2"
      },
      "source": [
        "#...now we can generate the training examples\n",
        "def prep_data(sentences):\n",
        "    all_labels=[] #here we gather labels for all words in all sentences\n",
        "    all_features=[] #here we gather features for all words in all sentences\n",
        "    for sentence in sentences:\n",
        "        sent_features=generate_sentence_features(sentence)\n",
        "        assert len(sent_features)==len(sentence)\n",
        "        #Now we can get, for every position its label and its features\n",
        "        for one_word,features in zip(sentence,sent_features):\n",
        "            all_labels.append(one_word.pos_label) #label\n",
        "            all_features.append(features)         #and features to go with it\n",
        "    return all_labels, all_features\n",
        "\n",
        "train_labels,train_features=prep_data(sentences_train)\n",
        "dev_labels,dev_features=prep_data(sentences_dev)"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6d2_CVK5dhC3"
      },
      "source": [
        "* Now we have the data in the usual form\n",
        "* We yet need to get the actual feature vectors\n",
        "* sklearn's DictVectorizer is a useful tool here - turns a dictionary of {feature_name -> value} into the corresponding feature vector\n",
        "* ...this gives us the freedom to build the dictionaries any way we like"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WjTN99bDdhC3",
        "outputId": "0c48c01a-f2af-4972-cef3-012dac755243"
      },
      "source": [
        "from sklearn.feature_extraction import DictVectorizer\n",
        "vectorizer=DictVectorizer()\n",
        "vectorizer.fit(train_features)\n",
        "print(\"Vectorizer vocab size:\",len(vectorizer.vocabulary_))\n",
        "\n",
        "feature_vectors_train=vectorizer.transform(train_features)\n",
        "feature_vectors_dev=vectorizer.transform(dev_features)\n",
        "\n",
        "print(\"Train shape\",feature_vectors_train.shape)\n",
        "print(\"Dev shape\",feature_vectors_dev.shape)"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Vectorizer vocab size: 23623\n",
            "Train shape (203621, 23623)\n",
            "Dev shape (51362, 23623)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zxTnRmZpdhC4"
      },
      "source": [
        "* And now we can train the classifier as usual\n",
        "* How well can we do?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Fe-9GVFVdhC4",
        "outputId": "40dcc8e0-b060-4e19-c10f-114472b5ab5a"
      },
      "source": [
        "import sklearn.svm\n",
        "\n",
        "classifier=sklearn.svm.LinearSVC(C=0.05,verbose=1)\n",
        "classifier.fit(feature_vectors_train, train_labels)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[LibLinear]"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "LinearSVC(C=0.05, verbose=1)"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mNWt3SKpdhC5",
        "outputId": "7760c5eb-f0aa-44df-8ff0-e792c41a2533"
      },
      "source": [
        "classifier.score(feature_vectors_dev,dev_labels)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.8655426190568903"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LKErE0esdhC5"
      },
      "source": [
        "* Oh my, that is a pretty good score for such a simple classifier!\n",
        "* The features are simply the words themselves, there is no context\n",
        "* Then again, is 86% a good POS tagger accuracy?\n",
        "* Can we do better?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VME8QrqndhC5",
        "outputId": "4a44f192-6c70-47eb-ba2d-1cae73ddb749"
      },
      "source": [
        "def generate_sentence_features(sent):\n",
        "    #Given a sentence as a list of (word, label) pairs\n",
        "    #generate the features for every word\n",
        "    #The result should be a list of same length as the sentence\n",
        "    #Each item is a dictionary of {\"feature name\"->feature value} mappings, holding all features of the word at that position\n",
        "    \n",
        "    sent_features=[] #this will be the result\n",
        "    for word_idx, one_word in enumerate(sent):\n",
        "        #We do nothing with label\n",
        "        #it just happens to be around\n",
        "        word_features={}\n",
        "        word_features[\"word_\"+one_word.word]=1 #the word itself is a feature\n",
        "        if word_idx!=0:\n",
        "            word_features[\"left_word_\"+sent[word_idx-1].word]=1\n",
        "        if word_idx!=len(sent)-1:\n",
        "            word_features[\"right_word_\"+sent[word_idx+1].word]=1\n",
        "        sent_features.append(word_features)\n",
        "    return sent_features\n",
        "\n",
        "train_labels,train_features=prep_data(sentences_train)\n",
        "dev_labels,dev_features=prep_data(sentences_dev)\n",
        "vectorizer=DictVectorizer()\n",
        "vectorizer.fit(train_features)\n",
        "feature_vectors_train=vectorizer.transform(train_features)\n",
        "feature_vectors_dev=vectorizer.transform(dev_features)\n",
        "\n",
        "print(\"Train shape\",feature_vectors_train.shape)\n",
        "print(\"Dev shape\",feature_vectors_dev.shape)\n",
        "\n",
        "classifier=sklearn.svm.LinearSVC(C=1,verbose=1)\n",
        "classifier.fit(feature_vectors_train, train_labels)\n",
        "classifier.score(feature_vectors_dev,dev_labels)"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Train shape (203621, 68467)\n",
            "Dev shape (51362, 68467)\n",
            "[LibLinear]"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9292862427475566"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "shJ9c-t7dhC6",
        "outputId": "dffec8c8-53b2-4ef6-eea1-8561c8a8b88e"
      },
      "source": [
        "# Let us try to look at some predictions\n",
        "sentence=\"I can house you in my house .\".split()\n",
        "\n",
        "sentence_data=[OneWord(w,\"XXX\",\"XXX\",\"XXX\") for w in sentence] #we need to fake this a bit, to get data in the correct format\n",
        "_,sentence_features=prep_data([sentence_data])\n",
        "sentence_vectors=vectorizer.transform(sentence_features)\n",
        "predictions=classifier.predict(sentence_vectors)\n",
        "for word,label in zip(sentence,predictions):\n",
        "    print(word,label)\n"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "I PRP\n",
            "can MD\n",
            "house VB\n",
            "you PRP\n",
            "in IN\n",
            "my PRP$\n",
            "house NN\n",
            ". .\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_f5PO9Z2dhC6"
      },
      "source": [
        "* PRP - personal pronoun\n",
        "* MD - modal verb\n",
        "* VB - verb\n",
        "* IN - preposition\n",
        "* my - possessive pronoun\n",
        "* NN - noun\n",
        "\n",
        "* Happily, we can see that the classifier was able to distinguish between the two occurences of `house`\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UVQR1dizdhC7"
      },
      "source": [
        "# What has the classifier learned?\n",
        "\n",
        "* We can use the same approach to introspecting the classifier as before\n",
        "* The classifier learns one decision hyperplane for each class\n",
        "* Otherwise, the code is *exactly* the same as in feature_interpretation, so let's use it"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3pRv4Z-GdhC8",
        "outputId": "f54d79a3-e021-4fa8-91ee-349836ca46d2"
      },
      "source": [
        "print(\"Learned coefficients:\",classifier.coef_.shape)\n",
        "print(\"Classes in the data:\",classifier.classes_)\n"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Learned coefficients: (45, 68467)\n",
            "Classes in the data: ['\"' '$' \"''\" '(' ')' ',' '.' ':' 'CC' 'CD' 'DT' 'EX' 'FW' 'IN' 'JJ' 'JJR'\n",
            " 'JJS' 'LS' 'MD' 'NN' 'NNP' 'NNPS' 'NNS' 'NN|SYM' 'PDT' 'POS' 'PRP' 'PRP$'\n",
            " 'RB' 'RBR' 'RBS' 'RP' 'SYM' 'TO' 'UH' 'VB' 'VBD' 'VBG' 'VBN' 'VBP' 'VBZ'\n",
            " 'WDT' 'WP' 'WP$' 'WRB']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Z7NYUApYdhC-",
        "outputId": "7458683b-177f-4f2b-9809-3d088a40c83d"
      },
      "source": [
        "import numpy\n",
        "\n",
        "#Reverse the dictionary\n",
        "index2feature={}\n",
        "for feature,idx in vectorizer.vocabulary_.items():\n",
        "    assert idx not in index2feature #This really should hold\n",
        "    index2feature[idx]=feature\n",
        "#Now we can query index2feature to get the feature names as we need\n",
        "\n",
        "i=list(classifier.classes_).index(\"NN\") #which of the coefficients corresponds to nouns?\n",
        "indices=numpy.argsort(classifier.coef_[i])\n",
        "print(\"Negative features\")\n",
        "for idx in indices[:30]:\n",
        "    print(index2feature[idx])\n",
        "print(\"-------------------------------\")\n",
        "print(\"Positive features\")\n",
        "for idx in indices[::-1][:30]: #you can also do it the other way round, reverse, then pick\n",
        "    print(index2feature[idx])"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Negative features\n",
            "left_word_will\n",
            "left_word_Sale\n",
            "word_,\n",
            "left_word_going\n",
            "left_word_could\n",
            "left_word_would\n",
            "left_word_goals\n",
            "right_word_A-rated\n",
            "left_word_We\n",
            "word_and\n",
            "left_word_At\n",
            "word_in\n",
            "left_word_still\n",
            "right_word_announcement\n",
            "left_word_mixer\n",
            "left_word_can\n",
            "left_word_I\n",
            "left_word_should\n",
            "left_word_kms\n",
            "left_word_might\n",
            "left_word_8:00\n",
            "left_word_prices\n",
            "left_word_Mike\n",
            "right_word_SCOREBOARD\n",
            "left_word_must\n",
            "left_word_n't\n",
            "left_word_overs\n",
            "right_word_effect\n",
            "left_word_Services\n",
            "word_two\n",
            "-------------------------------\n",
            "Positive features\n",
            "word_world\n",
            "word_power\n",
            "word_consumer\n",
            "word_peace\n",
            "word_number\n",
            "word_hospital\n",
            "word_vouch\n",
            "word_cricket\n",
            "word_procure\n",
            "word_soccer\n",
            "word_victory\n",
            "word_championship\n",
            "word_staff\n",
            "word_motor\n",
            "word_value\n",
            "word_cabinet\n",
            "word_lunch\n",
            "word_rain\n",
            "word_injury\n",
            "word_league\n",
            "word_anyone\n",
            "word_UNION\n",
            "word_weekend\n",
            "word_edge\n",
            "word_parliament\n",
            "word_shutdown\n",
            "word_division\n",
            "word_cash\n",
            "word_tournament\n",
            "word_race\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gDhaF6q1dhC_"
      },
      "source": [
        "# What have we learned?\n",
        "\n",
        "* The (English) POS tagging task has a surprisingly high trivial baseline\n",
        "* We can move the accuracy up by including features based on the context\n",
        "* Introspecting the classifier shows that these are in fact picked up by the classifier very strongly\n",
        "* Even then, we are left far behind the state-of-the-art which is typically in the 97-99% range for a vast number of languages\n",
        "* It is only a tiny change to the code to e.g. predict named entities\n",
        "\n",
        "# What we have not covered\n",
        "\n",
        "* All predictions are independent of each other\n",
        "* We did not treat the sequence as a sequence\n",
        "* We have failed to directly account for dependencies among the output labels\n",
        "* This is best done by a different class of machine learing models\n",
        "* Classically this is the domain of Conditional Random Fields (CRF)\n",
        "* These models take into account also tag-to-tag dependencies\n",
        "* The code above is actually not that far from being able to be used with the CRF\n",
        "* A fully worked example is here: https://sklearn-crfsuite.readthedocs.io/en/latest/tutorial.html#let-s-use-conll-2002-data-to-build-a-ner-system\n",
        "\n"
      ]
    }
  ]
}