basic_nlp_v2.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/TurkuNLP/intro-to-nlp/blob/master/basic_nlp_v2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ziutcOnEWFMO"
      },
      "source": [
        "## Text Segmentation\n",
        "\n",
        "* **Segmentation:** Divide bigger units into smaller ones\n",
        "* In many cases text needs to be segmented into sentences and/or words\n",
        "* Why?\n",
        "\n",
        "\n",
        "* **Tokenization / word segmentation:** Segment text into individual tokens\n",
        "* **Sentence splitting / sentence segmentation:** Segment text into individual sentences"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9TXhAZEGWFMQ"
      },
      "source": [
        "```\n",
        "Extremely bad customer service\n",
        "\n",
        "Do not go to this salon, especially if you have to get your hair straightened. They did a very bad job with my hair \n",
        "and were extremely rude when I went back to ask them why it didn't work for my hair. Rude, insensitive, discourteous people!!!!!\n",
        "```\n",
        "(*Text source: https://github.com/UniversalDependencies/UD_English-EWT*)\n",
        "\n",
        "\n",
        "**Tokenized:**\n",
        "```\n",
        "Extremely bad customer service\n",
        "\n",
        "Do not go to this salon , especially if you have to get your hair straightened . They did a very bad job with my hair \n",
        "and were extremely rude when I went back to ask them why it did n't work for my hair . Rude , insensitive , discourteous people !!!!!\n",
        "```\n",
        "\n",
        "**Sentence splitted:**\n",
        "```\n",
        "Extremely bad customer service\n",
        "\n",
        "Do not go to this salon, especially if you have to get your hair straightened.\n",
        "\n",
        "They did a very bad job with my hair and were extremely rude when I went back to ask them why it didn't work for my hair.\n",
        "\n",
        "Rude, insensitive, discourteous people!!!!!\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E4dd60DpWFMR"
      },
      "source": [
        "### Tokenization: HOW?\n",
        "\n",
        "* **Naive method 1:** Split from whitespace characters"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tdfax4N4WFMR",
        "outputId": "0f520416-6ec5-4731-8323-f51a1926dffd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Extremely\n",
            "bad\n",
            "customer\n",
            "service\n",
            "Do\n",
            "not\n",
            "go\n",
            "to\n",
            "this\n",
            "salon,\n",
            "especially\n",
            "if\n",
            "you\n",
            "have\n",
            "to\n",
            "get\n",
            "your\n",
            "hair\n",
            "straightened.\n",
            "They\n",
            "did\n",
            "a\n",
            "very\n",
            "bad\n",
            "job\n",
            "with\n",
            "my\n",
            "hair\n",
            "and\n",
            "were\n",
            "extremely\n",
            "rude\n",
            "when\n",
            "I\n",
            "went\n",
            "back\n",
            "to\n",
            "ask\n",
            "them\n",
            "why\n",
            "it\n",
            "didn't\n",
            "work\n",
            "for\n",
            "my\n",
            "hair.\n",
            "Rude,\n",
            "insensitive,\n",
            "discourteous\n",
            "people!!!!!\n"
          ]
        }
      ],
      "source": [
        "text=\"\"\"Extremely bad customer service\n",
        "\n",
        "Do not go to this salon, especially if you have to get your hair straightened. \\\n",
        "They did a very bad job with my hair and were extremely rude when I went back to \\\n",
        "ask them why it didn't work for my hair. Rude, insensitive, discourteous people!!!!!\"\"\"\n",
        "\n",
        "tokenized_text = text.split() # split(): Return a list of the words in the string, using whitespace as the delimiter string.\n",
        "\n",
        "for w in tokenized_text:\n",
        "    print(w)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cXT_iPR2WFMS"
      },
      "source": [
        "* **Naive method 2:** Split from whitespace characters, take into account punctuation\n",
        "* Regular expressions:\n",
        "    * Define search patters\n",
        "    * Find these patterns from raw text, or find-and-replace if needed\n",
        "* Find all punctuation characters, and replace with whitespace+punctuation character\n",
        "    * *book.* --> *book .*\n",
        "    * *people!!!!!* --> *people !!!!!*\n",
        "    * How about clitics in English? [don't, can't, cannot?]\n",
        "    * 2-(14-hydroxypentadecyl)-4-methyl-5-oxo-2,5-dihydrofuran-3-carboxylic acid ???\n",
        "    * Usually it's not that important how exactly you do it, just be consistent!\n",
        "        * consistent = always do it the same way\n",
        "        * If you download two datasets which are already tokenized, the tokenization may differ and you need to be aware of it!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b0omWr9QWFMT",
        "outputId": "3307154d-499a-4270-8cc9-8f7f01352ffc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Extremely bad customer service\n",
            "\n",
            "Do not go to this salon , especially if you have to get your hair straightened . They did a very bad job with my hair and were extremely rude when I went back to ask them why it did n't work for my hair . Rude , insensitive , discourteous people !!!!!\n"
          ]
        }
      ],
      "source": [
        "import re\n",
        "\n",
        "tokenized = re.sub(r'([.,!?]+)', r' \\1', text) # replace . , ! ? with whitespace+character(s), '+' means one or more\n",
        "tokenized = re.sub(r\"(n't)\", r\" \\1\", tokenized) # clitics\n",
        "\n",
        "print(tokenized) # Note: this is still string, apply simple whitespace splitting to get a list of tokens"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WvF1NWsNWFMX"
      },
      "source": [
        "* **Naive method 2** works quite well for English, Finnish, Swedish...\n",
        "    * Approx. 97-99% correct on clean text\n",
        "    * Many tokenizers are just a large number (in the hundreds) regular expressions\n",
        "\n",
        "\n",
        "* How about other languages, does it work for all?\n",
        "\n",
        ".\n",
        "\n",
        ".\n",
        "\n",
        ".\n",
        "\n",
        ".\n",
        "\n",
        ".\n",
        "\n",
        ".\n",
        "\n",
        ".\n",
        "\n",
        ".\n",
        "\n",
        "**Nope! Why not?**\n",
        "\n",
        ".\n",
        "\n",
        ".\n",
        "\n",
        ".\n",
        "\n",
        ".\n",
        "\n",
        ".\n",
        "\n",
        ".\n",
        "\n",
        ".\n",
        "\n",
        "* All languages do not use whitespace or punctuation, or the meaning of those may be different.\n",
        "* Chinese, Thai, Vietnamese\n",
        "\n",
        "![tokenization.png](https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/tokenization.png?raw=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XfQmIvngWFMY"
      },
      "source": [
        "* **Naive algorithm:**\n",
        "    1) Build a vocabulary for the language\n",
        "    2) Start from the beginning of the text, and find the longest matching word\n",
        "    3) Split the matching word and continue from the next remaining character\n",
        "* *the table down there* --> *thetabledownthere* --> *theta bled own there*\n",
        "    * Does not work well for English, but in Chinese words are usually 2-4 characters long, so the simple algorithm works better\n",
        "    * Where to get the dictionary?\n",
        "    \n",
        "**Tokenization: State-of-the-art**\n",
        "* State-of-the-art = The best existing method currently known\n",
        "* Machine learning\n",
        "    * Collect raw (untokenized) text for the language you are interested in, and manually tokenize it.\n",
        "    * Train a classifier\n",
        "    * The trained classifier can be used to tokenize new text"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5xMDiYq9WFMY"
      },
      "source": [
        "#### Sentence splitting: HOW?\n",
        "\n",
        "* **Naive method 1:** What kind of punctuation characters end the sentence?\n",
        "    * yes: . ! ?\n",
        "    * no: ,\n",
        "* Define a list of sentence-final punctuation, and always split on those.\n",
        "* Problems?\n",
        "\n",
        "![sentence_splitting.png](https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/sentence_splitting.png?raw=1)\n",
        "\n",
        "\n",
        "* **Solution 1:** Define a list of rules to identify when punctuation does not end a sentence\n",
        "    * List of known abbreviations, list of regular expression to regocnize numbers etc. (*The cost was approx. 1.5 million euros.*)\n",
        "    * How about missing punctuation? Other languages?\n",
        "    \n",
        "**Sentence splitting: State-of-the-art**\n",
        "* Machine learning\n",
        "    * Collect raw text for the language you are interested in, and manually sentence segment it.\n",
        "    * Train a classifier\n",
        "    * The trained classifier can be used to sentence segment new text\n",
        "    \n",
        "## Try UDPipe machine learned tokenizer and sentence splitter"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ce6O2NqgWFMY",
        "outputId": "a75c87d4-d2e8-409b-8ec6-c6dbda37bec2"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2023-03-13 19:51:56--  https://github.com/TurkuNLP/intro-to-nlp/raw/master/Data/en.segmenter.udpipe\n",
            "Resolving github.com (github.com)... 20.27.177.113\n",
            "Connecting to github.com (github.com)|20.27.177.113|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://raw.githubusercontent.com/TurkuNLP/intro-to-nlp/master/Data/en.segmenter.udpipe [following]\n",
            "--2023-03-13 19:51:57--  https://raw.githubusercontent.com/TurkuNLP/intro-to-nlp/master/Data/en.segmenter.udpipe\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 17394186 (17M) [application/octet-stream]\n",
            "Saving to: ‘en.segmenter.udpipe’\n",
            "\n",
            "en.segmenter.udpipe 100%[===================>]  16.59M  --.-KB/s    in 0.1s    \n",
            "\n",
            "2023-03-13 19:51:57 (140 MB/s) - ‘en.segmenter.udpipe’ saved [17394186/17394186]\n",
            "\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting ufal.udpipe\n",
            "  Downloading ufal.udpipe-1.3.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (936 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m936.9/936.9 KB\u001b[0m \u001b[31m13.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: ufal.udpipe\n",
            "Successfully installed ufal.udpipe-1.3.0.1\n"
          ]
        }
      ],
      "source": [
        "# Let's try to tokenize and sentence split a small dataset with UDPipe machine learned segmenter!\n",
        "# Documentation: https://ufal.mff.cuni.cz/udpipe/users-manual\n",
        "# Training data: \n",
        "# Finnish (intro-to-nlp/Data/fi.segmenter.udpipe): https://github.com/UniversalDependencies/UD_Finnish-TDT v.2.2\n",
        "# English (intro-to-nlp/Data/en.segmenter.udpipe): https://github.com/UniversalDependencies/UD_English-EWT v.2.2\n",
        "# Swedish (intro-to-nlp/Data/sv.segmenter.udpipe): https://github.com/UniversalDependencies/UD_Swedish-Talbanken v.2.2\n",
        "\n",
        "!wget -nc https://github.com/TurkuNLP/intro-to-nlp/raw/master/Data/en.segmenter.udpipe\n",
        "\n",
        "!pip3 install ufal.udpipe\n",
        "\n",
        "import ufal.udpipe as udpipe\n",
        "\n",
        "model = udpipe.Model.load(\"en.segmenter.udpipe\")\n",
        "pipeline = udpipe.Pipeline(model,\"tokenize\",\"none\",\"none\",\"horizontal\") # horizontal: returns one sentence per line, with words separated by a single space\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "document=\"\"\"\n",
        "The North American X-15 is a hypersonic rocket-powered aircraft. It was operated by the United States \n",
        "Air Force and the National Aeronautics and Space Administration as part of the X-plane series of \n",
        "experimental aircraft. The X-15 set speed and altitude records in the 1960s, reaching \n",
        "the edge of outer space and returning with valuable data used in aircraft and spacecraft \n",
        "design. The X-15's highest speed, 4,520 miles per hour (7,274 km/h; 2,021 m/s),[1] was \n",
        "achieved on 3 October 1967,[2] when William J. Knight flew at Mach 6.7 at an altitude of \n",
        "102,100 feet (31,120 m), or 19.34 miles. This set the official world record for the highest \n",
        "speed ever recorded by a crewed, powered aircraft, which remains unbroken.[3][4]\n",
        "\n",
        "During the X-15 program, 12 pilots flew a combined 199 flights.[1] Of these, \n",
        "8 pilots flew a combined 13 flights which met the Air Force spaceflight criterion \n",
        "by exceeding the altitude of 50 miles (80 km), thus qualifying these pilots as being \n",
        "astronauts; of those 13 flights, two (flown by the same civilian pilot) met the FAI \n",
        "definition (100 kilometres (62 mi)) of outer space. The 5 Air Force pilots qualified \n",
        "for military astronaut wings immediately, while the 3 civilian pilots were eventually \n",
        "awarded NASA astronaut wings in 2005, 35 years after the last X-15 flight.[5][6]\n",
        "\"\"\"\n",
        "\n",
        "segmented_document = pipeline.process(document)\n",
        "\n",
        "print(segmented_document)"
      ],
      "metadata": {
        "id": "f_b3xiM2cghl",
        "outputId": "ace24adc-f364-4303-de33-9895e54b256b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The North American X - 15 is a hypersonic rocket - powered aircraft .\n",
            "It was operated by the United States Air Force and the National Aeronautics and Space Administration as part of the X - plane series of experimental aircraft .\n",
            "The X - 15 set speed and altitude records in the 1960s , reaching the edge of outer space and returning with valuable data used in aircraft and spacecraft design .\n",
            "The X - 15 's highest speed , 4,520 miles per hour ( 7,274 km/h ; 2,021 m/s ) ,[ 1 ] was achieved on 3 October 1967 , [ 2 ] when William J. Knight flew at Mach 6.7 at an altitude of 102,100 feet ( 31,120 m ) , or 19.34 miles .\n",
            "This set the official world record for the highest speed ever recorded by a crewed , powered aircraft , which remains unbroken .\n",
            "[ 3 ] [ 4 ]\n",
            "During the X - 15 program , 12 pilots flew a combined 199 flights .\n",
            "[ 1 ]\n",
            "Of these , 8 pilots flew a combined 13 flights which met the Air Force spaceflight criterion by exceeding the altitude of 50 miles ( 80 km ) , thus qualifying these pilots as being astronauts ; of those 13 flights , two ( flown by the same civilian pilot ) met the FAI definition ( 100 kilometres ( 62 mi ) ) of outer space .\n",
            "The 5 Air Force pilots qualified for military astronaut wings immediately , while the 3 civilian pilots were eventually awarded NASA astronaut wings in 2005 , 35 years after the last X - 15 flight .\n",
            "[ 5 ] [ 6 ]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M9BtnHDeWFMZ"
      },
      "source": [
        "## 4. Word frequencies\n",
        "\n",
        "* How many times each word appears in the corpus?\n",
        "* How many unique words the corpus has?\n",
        "    * vocabulary size"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!wget -nc https://github.com/TurkuNLP/intro-to-nlp/raw/master/Data/imdb_train.json\n",
        "\n",
        "import json # JSON encoder and decoder: store python data structures (e.g. lists and dictionaries) as strings\n",
        "\n",
        "with open(\"imdb_train.json\", \"rt\", encoding=\"utf-8\") as f:\n",
        "    data = json.load(f)\n",
        "    \n",
        "print(\"Data type:\", type(data))\n",
        "print(\"First item type:\", type(data[0]))\n",
        "print(\"First item:\", data[0])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5mOLvytzhfr_",
        "outputId": "6a77168c-4175-47c9-e5e6-9f438b2bc3fb"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "File ‘imdb_train.json’ already there; not retrieving.\n",
            "\n",
            "Data type: <class 'list'>\n",
            "First item type: <class 'dict'>\n",
            "First item: {'class': 'pos', 'text': \"With all this stuff going down at the moment with MJ i've started listening to his music, watching the odd documentary here and there, watched The Wiz and watched Moonwalker again. Maybe i just want to get a certain insight into this guy who i thought was really cool in the eighties just to maybe make up my mind whether he is guilty or innocent. Moonwalker is part biography, part feature film which i remember going to see at the cinema when it was originally released. Some of it has subtle messages about MJ's feeling towards the press and also the obvious message of drugs are bad m'kay.  Visually impressive but of course this is all about Michael Jackson so unless you remotely like MJ in anyway then you are going to hate this and find it boring. Some may call MJ an egotist for consenting to the making of this movie BUT MJ and most of his fans would say that he made it for the fans which if true is really nice of him.  The actual feature film bit when it finally starts is only on for 20 minutes or so excluding the Smooth Criminal sequence and Joe Pesci is convincing as a psychopathic all powerful drug lord. Why he wants MJ dead so bad is beyond me. Because MJ overheard his plans? Nah, Joe Pesci's character ranted that he wanted people to know it is he who is supplying drugs etc so i dunno, maybe he just hates MJ's music.  Lots of cool things in this like MJ turning into a car and a robot and the whole Speed Demon sequence. Also, the director must have had the patience of a saint when it came to filming the kiddy Bad sequence as usually directors hate working with one kid let alone a whole bunch of them performing a complex dance scene.  Bottom line, this movie is for people who like MJ on one level or another (which i think is most people). If not, then stay away. It does try and give off a wholesome message and ironically MJ's bestest buddy in this movie is a girl! Michael Jackson is truly one of the most talented people ever to grace this planet but is he guilty? Well, with all the attention i've gave this subject....hmmm well i don't know because people can be different behind closed doors, i know this for a fact. He is either an extremely nice but stupid guy or one of the most sickest liars. I hope he is not the latter.\"}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "KaPTwF3NWFMa",
        "outputId": "595e6fbb-7a09-41a9-dad1-b48ebcb1a2fe",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 1000/1000 [00:53<00:00, 18.67it/s]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Most common tokens: [('the', 11464), (',', 10973), ('.', 10515), ('a', 6291), ('and', 6269), ('of', 5723), ('to', 5221), ('is', 4310), ('in', 3421), ('I', 3115), ('it', 3101), ('that', 2723), ('\"', 2633), (\"'s\", 2432), ('this', 2287), ('\\\\', 2249), ('was', 2013), ('-', 1980), ('with', 1812), ('as', 1711)]\n",
            "Vocabulary size: 21940\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\n"
          ]
        }
      ],
      "source": [
        "from collections import Counter\n",
        "import tqdm\n",
        "\n",
        "token_counter = Counter()\n",
        "for doc in tqdm.tqdm(data[:1000]): # IMDB documents\n",
        "    tokenized = pipeline.process(doc[\"text\"])\n",
        "    tokens = tokenized.split() # after segmenter, we can do whitespace splitting\n",
        "    token_counter.update(tokens)\n",
        "\n",
        "print(\"Most common tokens:\", token_counter.most_common(20))\n",
        "print(\"Vocabulary size:\", len(token_counter))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UI4Tme-MWFMb"
      },
      "source": [
        "### Stop words\n",
        "\n",
        "* Commonly used functional words with little semantic meaning\n",
        "* Typically the most frequent words in the corpus\n",
        "* The idea is to densify the data by removing these \"meaningless\" words"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "cXQF3pzdWFMb",
        "outputId": "36881327-3d6b-4ac6-95cf-6b67b074f95b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of tokens: 47\n",
            "Tokens: [(\"'s\", 2432), ('film', 1630), ('movie', 1596), (\"n't\", 1237), ('one', 1004), ('like', 729), (\"'\", 685), ('good', 634), ('would', 527), ('time', 488), ('really', 445), ('even', 430), ('story', 425), ('see', 397), ('could', 383), ('get', 364), ('people', 361), ('much', 345), ('bad', 340), ('well', 334), ('great', 326), ('made', 311), ('first', 310), ('way', 307), ('make', 305), ('also', 299), ('think', 279), ('movies', 278), ('films', 275), ('characters', 275), ('many', 268), ('character', 267), ('show', 266), ('acting', 250), ('ever', 246), ('watch', 241), ('seen', 240), ('plot', 240), ('love', 229), ('never', 225), ('little', 220), ('best', 218), ('say', 217), ('two', 216), ('know', 214), ('life', 213), ('end', 206)]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n"
          ]
        }
      ],
      "source": [
        "import nltk\n",
        "nltk.download('stopwords') # download the stopwords dataset\n",
        "\n",
        "from nltk.corpus import stopwords\n",
        "\n",
        "# take 150 most common words from the IMDB corpus and filter out stop words and punctuation\n",
        "filtered_tokens = []\n",
        "punctuation_chars = '. , : ( ) ! ? \" = & - ; ... \\\\ '.split() # list of punctuation symbols to ignore\n",
        "for word, count in token_counter.most_common(150):\n",
        "    if word.lower() in stopwords.words(\"english\") or word in punctuation_chars:\n",
        "        continue\n",
        "    filtered_tokens.append((word, count))\n",
        "print(\"Number of tokens:\", len(filtered_tokens))\n",
        "print(\"Tokens:\", filtered_tokens)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dPqSB58sWFMc"
      },
      "source": [
        "Quotes from the internet search:\n",
        "\n",
        "* *A stop word is a commonly used word (such as “the”, “a”, “an”, “in”) that a search engine has been programmed to ignore.* (geeksforgeeks.org)\n",
        "* *Stop words are words which are filtered out before processing of natural language data (text).* (Wikipedia)\n",
        "\n",
        "**Not necessarily true with modern machine learning techniques!**\n",
        "\n",
        "* Another approach: Do not remove anything but give a higher importance to more meaningful words\n",
        "\n",
        "\n",
        "### tf-idf weighting\n",
        "\n",
        "* TF = term frequency *tf(t, d)*, how many times the term *t* appears in the **document** *d*\n",
        "* DF = document frequency *df(t)*, in how many documents (out of all documents) the term *t* appears\n",
        "* IDF = inverse document frequency, *m/df(t)*, where *m* is the total number of documents in your collection\n",
        "* TF-IDF = **tf(t, d) * idf(t)**\n",
        "    * Usually calculated using logaritmic scale --> tf(t, d) * log(idf(t)) or log(1 + tf(t,d)) * log(idf(t))\n",
        "    \n",
        "| ![log.png](https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/log.png?raw=1) |\n",
        "|:--:|\n",
        "| *Source: Wikipedia* |\n",
        "    \n",
        "* common in information retrieval, also used in document classification\n",
        "* scale down the impact of tokens that occur very frequently in many documents and are hence empirically less informative than words that occur in a small fraction of the documents\n",
        "\n",
        "### Examples of idf-weights calculated using natural logarithm (ln) and a Finnish corpus\n",
        "\n",
        "![idf.png](https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/idf.png?raw=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8Rv5xWXSWFMc"
      },
      "source": [
        "## 5. Text Normalization\n",
        "\n",
        "* Remove certain \"randomness\" from the data\n",
        "* Try to reduce uncommon cases\n",
        "* Normalization techniques involve:\n",
        "  * Tokenization\n",
        "  * Punctuation removal\n",
        "  * Capitalization / Lowercasing\n",
        "  * Accent removal\n",
        "  * Stemming / Lemmatization\n",
        "  * ...\n",
        "\n",
        "### Stemming and lemmatization\n",
        "\n",
        "* Densify data by removing inflectional variation\n",
        "\n",
        "* **Stemming:** Determine the word root by removing inflectional affixes \n",
        "    * play, plays, playing, played --> play\n",
        "    * activate, active, activated, activation --> activ\n",
        "    * koira, koiran, koiralla, koirilla --> koir\n",
        "    * koirasta --> koir\n",
        "* Risk of overstemming or understemming: two separate inflected words are stemmed to the same root, or inflections of the same word are stemmed to different roots\n",
        "* Does not take into account the context (lives --> live / life, koirasta --> koira / koiras)\n",
        "\n",
        "\n",
        "* **Lemmatization:** Determine the base (dictionary) form of the word\n",
        "    * play, plays, playing, played --> play\n",
        "    * activate, active, activated, activation --> activate, active, activate, activation\n",
        "    * koira, koiran, koiralla, koirilla --> koira\n",
        "    * koirasta --> koira / koiras\n",
        "* Generally better, but also computationally heavier and more complex method"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "x2yfKSNkWFMd",
        "outputId": "0daebf68-30d6-43f1-bb17-5490449419ec",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "with all this stuff go down at the moment with mj i'v start listen to his music, watch the odd documentari here and there, watch the wiz and watch moonwalk again. mayb i just want to get a certain insight into this guy who i thought was realli cool in the eighti just to mayb make up my mind whether he is guilti or innocent. moonwalk is part biography, part featur film which i rememb go to see at the cinema when it was origin released. some of it has subtl messag about mj feel toward the press and also the obvious messag of drug are bad m'kay. visual impress but of cours this is all about michael jackson so unless you remot like mj in anyway then you are go to hate this and find it boring. some may call mj an egotist for consent to the make of this movi but mj and most of his fan would say that he made it for the fan which if true is realli nice of him. the actual featur film bit when it final start is onli on for 20 minut or so exclud the smooth crimin sequenc and joe pesci is convinc as a psychopath all power drug lord. whi he want mj dead so bad is beyond me. becaus mj overheard his plans? nah, joe pesci charact rant that he want peopl to know it is he who is suppli drug etc so i dunno, mayb he just hate mj music. lot of cool thing in this like mj turn into a car and a robot and the whole speed demon sequence. also, the director must have had the patienc of a saint when it came to film the kiddi bad sequenc as usual director hate work with one kid let alon a whole bunch of them perform a complex danc scene. bottom line, this movi is for peopl who like mj on one level or anoth (which i think is most people). if not, then stay away. it doe tri and give off a wholesom messag and iron mj bestest buddi in this movi is a girl! michael jackson is truli one of the most talent peopl ever to grace this planet but is he guilty? well, with all the attent i'v gave this subject....hmmm well i don't know becaus peopl can be differ behind close doors, i know this for a fact. he is either an extrem nice but stupid guy or one of the most sickest liars. i hope he is not the latter.\n"
          ]
        }
      ],
      "source": [
        "from nltk.stem.snowball import SnowballStemmer\n",
        "\n",
        "stemmer = SnowballStemmer(\"english\")\n",
        "\n",
        "print(\" \".join(stemmer.stem(w) for w in data[0][\"text\"].split()))\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "jAHseuydj21Z"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Full parsing\n",
        "\n",
        "* The most complex text preprocessing\n",
        "* Computationally heavy but gives most information\n",
        "* POS+morphological information\n",
        "* Lemma\n",
        "* Dependency relations (syntactic tree)\n"
      ],
      "metadata": {
        "id": "LlFA9fY-j3WZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install trankit transformers"
      ],
      "metadata": {
        "id": "efC_iOA_1Awx",
        "outputId": "22c3dfd1-542b-4415-889f-3c607342816f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting trankit\n",
            "  Downloading trankit-1.1.1-py3-none-any.whl (773 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m773.4/773.4 KB\u001b[0m \u001b[31m12.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting transformers\n",
            "  Downloading transformers-4.26.1-py3-none-any.whl (6.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.3/6.3 MB\u001b[0m \u001b[31m61.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting tokenizers>=0.7.0\n",
            "  Downloading tokenizers-0.13.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.6/7.6 MB\u001b[0m \u001b[31m68.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: requests in /usr/local/lib/python3.9/dist-packages (from trankit) (2.25.1)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.9/dist-packages (from trankit) (3.9.0)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.9/dist-packages (from trankit) (23.0)\n",
            "Collecting sacremoses\n",
            "  Downloading sacremoses-0.0.53.tar.gz (880 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m880.6/880.6 KB\u001b[0m \u001b[31m53.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.9/dist-packages (from trankit) (1.22.4)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.9/dist-packages (from trankit) (4.65.0)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.9/dist-packages (from trankit) (2022.6.2)\n",
            "Requirement already satisfied: protobuf in /usr/local/lib/python3.9/dist-packages (from trankit) (3.19.6)\n",
            "Collecting sentencepiece\n",
            "  Downloading sentencepiece-0.1.97-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m59.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting langid==1.1.6\n",
            "  Downloading langid-1.1.6.tar.gz (1.9 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.9/1.9 MB\u001b[0m \u001b[31m54.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: torch>=1.6.0 in /usr/local/lib/python3.9/dist-packages (from trankit) (1.13.1+cu116)\n",
            "Collecting huggingface-hub<1.0,>=0.11.0\n",
            "  Downloading huggingface_hub-0.13.2-py3-none-any.whl (199 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m199.2/199.2 KB\u001b[0m \u001b[31m17.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.9/dist-packages (from transformers) (6.0)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.9/dist-packages (from huggingface-hub<1.0,>=0.11.0->transformers) (4.5.0)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/dist-packages (from requests->trankit) (2022.12.7)\n",
            "Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.9/dist-packages (from requests->trankit) (4.0.0)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.9/dist-packages (from requests->trankit) (2.10)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.9/dist-packages (from requests->trankit) (1.26.14)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.9/dist-packages (from sacremoses->trankit) (1.15.0)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.9/dist-packages (from sacremoses->trankit) (8.1.3)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.9/dist-packages (from sacremoses->trankit) (1.1.1)\n",
            "Building wheels for collected packages: langid, sacremoses\n",
            "  Building wheel for langid (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for langid: filename=langid-1.1.6-py3-none-any.whl size=1941188 sha256=76a6d60a4162d41879f9669f999df8f4dca1b5f168c1f0dc26a5ea262ee626dd\n",
            "  Stored in directory: /root/.cache/pip/wheels/93/95/a9/c292c9dd8cadb8f2359f1670ff198a40d47167b0be3236e1c8\n",
            "  Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sacremoses: filename=sacremoses-0.0.53-py3-none-any.whl size=895260 sha256=05ffc8089f72463871a3b4a8701f677995a603a0dc2d5cb9467081ad5e687178\n",
            "  Stored in directory: /root/.cache/pip/wheels/12/1c/3d/46cf06718d63a32ff798a89594b61e7f345ab6b36d909ce033\n",
            "Successfully built langid sacremoses\n",
            "Installing collected packages: tokenizers, sentencepiece, sacremoses, langid, trankit, huggingface-hub, transformers\n",
            "Successfully installed huggingface-hub-0.13.2 langid-1.1.6 sacremoses-0.0.53 sentencepiece-0.1.97 tokenizers-0.13.2 trankit-1.1.1 transformers-4.26.1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import trankit\n",
        "p = trankit.Pipeline('english', gpu=False)"
      ],
      "metadata": {
        "id": "VhmIpLVp_y2X",
        "outputId": "bd01d6c8-d2a2-440b-a464-713733363ee3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 287,
          "referenced_widgets": [
            "903d9cd8e1ca40408c157862adc67d0b",
            "2173dec9fc184ee4b9b9940317b27d3d",
            "17cd0abfa49442758876c5fe51a525c7",
            "1a86784a1b7f4e93afe82873711ac175",
            "81cffceacb7d4428901ef0fb73148ede",
            "5864d0d2008c4483b34c418bbc462247",
            "e8e957ada4974cd189e8d1552f026e23",
            "f634c9ddd16d4f2e8102622067c191ea",
            "9e48b37cef0841be8049508b63516c09",
            "62d3255feff0434986c9a9f598139409",
            "2397d5806c9c4d0c960c0f3d2164c7f2",
            "5f86257e5cb44002bc1a7a8d36c74c30",
            "0512e158325a496d948e37789d4d03a1",
            "c5dfcd88259c49dea07f6136bcc1bd8c",
            "cb375ccae6b4417dbf6c31797536afb0",
            "8f11afe96cac4bf2b7c533af35316d14",
            "9035ca3f2bca48cda23fe0e468e214fd",
            "662b0f43b0824a979559c00fc4d1fc47",
            "6f691646a44140b7ab25acd5c3e1c513",
            "a1973fcfa62545418ceb8f1b67c28317",
            "0ead3426a1644e8d919c404da4a83df0",
            "c0143d2244b041218fb97537c9e3abf4",
            "ced73a1b7f4745d0b16c609acb5400ea",
            "2303befd57ce4f579eef2d63f3e6c393",
            "53a34f960a0e4c6b9c032c165f878806",
            "b202c4c95b4c44aaa1558e461440b2a5",
            "ad2d094e1fad4d08a464aa462320d19f",
            "0d7123b9eba54e2a97c410da9884de99",
            "63737186945f4ce799b726a97043efb8",
            "083f7a444ab34af7a11ea284f9b38dcc",
            "94ad2cd15000434fa06f705cdd50e98c",
            "de911cfe24ac468fbd7c2953b7fb280f",
            "2a5ec0a082fc4f28a6d7fa69d509daca"
          ]
        }
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/5.07M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "903d9cd8e1ca40408c157862adc67d0b"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "http://nlp.uoregon.edu/download/trankit/v1.0.0/xlm-roberta-base/english.zip\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Downloading: 100%|██████████| 47.9M/47.9M [00:02<00:00, 22.4MiB/s]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Loading pretrained XLM-Roberta, this may take a while...\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/512 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "5f86257e5cb44002bc1a7a8d36c74c30"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/1.12G [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "ced73a1b7f4745d0b16c609acb5400ea"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Loading tokenizer for english\n",
            "Loading tagger for english\n",
            "Loading lemmatizer for english\n",
            "Loading NER tagger for english\n",
            "==================================================\n",
            "Active language: english\n",
            "==================================================\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "p(\"The children ate the cakes with spoons.\")"
      ],
      "metadata": {
        "id": "YG89BMqX_5mT",
        "outputId": "3e24ced9-0e36-41bb-f310-6a8254a03635",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'text': 'The children ate the cakes with spoons.',\n",
              " 'sentences': [{'id': 1,\n",
              "   'text': 'The children ate the cakes with spoons.',\n",
              "   'tokens': [{'id': 1,\n",
              "     'text': 'The',\n",
              "     'upos': 'DET',\n",
              "     'xpos': 'DT',\n",
              "     'feats': 'Definite=Def|PronType=Art',\n",
              "     'head': 2,\n",
              "     'deprel': 'det',\n",
              "     'dspan': (0, 3),\n",
              "     'span': (0, 3),\n",
              "     'lemma': 'the',\n",
              "     'ner': 'O'},\n",
              "    {'id': 2,\n",
              "     'text': 'children',\n",
              "     'upos': 'NOUN',\n",
              "     'xpos': 'NNS',\n",
              "     'feats': 'Number=Plur',\n",
              "     'head': 3,\n",
              "     'deprel': 'nsubj',\n",
              "     'dspan': (4, 12),\n",
              "     'span': (4, 12),\n",
              "     'lemma': 'child',\n",
              "     'ner': 'O'},\n",
              "    {'id': 3,\n",
              "     'text': 'ate',\n",
              "     'upos': 'VERB',\n",
              "     'xpos': 'VBD',\n",
              "     'feats': 'Mood=Ind|Tense=Past|VerbForm=Fin',\n",
              "     'head': 0,\n",
              "     'deprel': 'root',\n",
              "     'dspan': (13, 16),\n",
              "     'span': (13, 16),\n",
              "     'lemma': 'eat',\n",
              "     'ner': 'O'},\n",
              "    {'id': 4,\n",
              "     'text': 'the',\n",
              "     'upos': 'DET',\n",
              "     'xpos': 'DT',\n",
              "     'feats': 'Definite=Def|PronType=Art',\n",
              "     'head': 5,\n",
              "     'deprel': 'det',\n",
              "     'dspan': (17, 20),\n",
              "     'span': (17, 20),\n",
              "     'lemma': 'the',\n",
              "     'ner': 'O'},\n",
              "    {'id': 5,\n",
              "     'text': 'cakes',\n",
              "     'upos': 'NOUN',\n",
              "     'xpos': 'NNS',\n",
              "     'feats': 'Number=Plur',\n",
              "     'head': 3,\n",
              "     'deprel': 'obj',\n",
              "     'dspan': (21, 26),\n",
              "     'span': (21, 26),\n",
              "     'lemma': 'cake',\n",
              "     'ner': 'O'},\n",
              "    {'id': 6,\n",
              "     'text': 'with',\n",
              "     'upos': 'ADP',\n",
              "     'xpos': 'IN',\n",
              "     'head': 7,\n",
              "     'deprel': 'case',\n",
              "     'dspan': (27, 31),\n",
              "     'span': (27, 31),\n",
              "     'lemma': 'with',\n",
              "     'ner': 'O'},\n",
              "    {'id': 7,\n",
              "     'text': 'spoons',\n",
              "     'upos': 'NOUN',\n",
              "     'xpos': 'NNS',\n",
              "     'feats': 'Number=Plur',\n",
              "     'head': 3,\n",
              "     'deprel': 'obl',\n",
              "     'dspan': (32, 38),\n",
              "     'span': (32, 38),\n",
              "     'lemma': 'spoon',\n",
              "     'ner': 'O'},\n",
              "    {'id': 8,\n",
              "     'text': '.',\n",
              "     'upos': 'PUNCT',\n",
              "     'xpos': '.',\n",
              "     'head': 3,\n",
              "     'deprel': 'punct',\n",
              "     'dspan': (38, 39),\n",
              "     'span': (38, 39),\n",
              "     'lemma': '.',\n",
              "     'ner': 'O'}],\n",
              "   'dspan': (0, 39)}],\n",
              " 'lang': 'english'}"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "p = trankit.Pipeline('finnish', gpu=False)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cSGBStMPkxkI",
        "outputId": "5cc01110-3b2c-4572-cefb-2a1d078a299b"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Loading pretrained XLM-Roberta, this may take a while...\n",
            "Loading tokenizer for finnish\n",
            "Loading tagger for finnish\n",
            "Loading multi-word expander for finnish\n",
            "Loading lemmatizer for finnish\n",
            "==================================================\n",
            "Active language: finnish\n",
            "==================================================\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "p(\"Puhu retkesta pappisi kanssa.\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JIUt8bUblvNm",
        "outputId": "e74688d2-c2d6-43e7-eabd-a84b0689983f"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'text': 'Puhu retkesta pappisi kanssa.',\n",
              " 'sentences': [{'id': 1,\n",
              "   'text': 'Puhu retkesta pappisi kanssa.',\n",
              "   'dspan': (0, 29),\n",
              "   'tokens': [{'id': 1,\n",
              "     'text': 'Puhu',\n",
              "     'upos': 'VERB',\n",
              "     'xpos': 'V',\n",
              "     'feats': 'Mood=Imp|Number=Sing|Person=2|VerbForm=Fin|Voice=Act',\n",
              "     'head': 0,\n",
              "     'deprel': 'root',\n",
              "     'dspan': (0, 4),\n",
              "     'span': (0, 4),\n",
              "     'lemma': 'puhua'},\n",
              "    {'id': 2,\n",
              "     'text': 'retkesta',\n",
              "     'upos': 'NOUN',\n",
              "     'xpos': 'N',\n",
              "     'feats': 'Case=Ela|Number=Sing',\n",
              "     'head': 1,\n",
              "     'deprel': 'obl',\n",
              "     'dspan': (5, 13),\n",
              "     'span': (5, 13),\n",
              "     'lemma': 'retki'},\n",
              "    {'id': 3,\n",
              "     'text': 'pappisi',\n",
              "     'upos': 'NOUN',\n",
              "     'xpos': 'N',\n",
              "     'feats': 'Case=Gen|Number=Sing|Number[psor]=Sing|Person[psor]=2',\n",
              "     'head': 1,\n",
              "     'deprel': 'obl',\n",
              "     'dspan': (14, 21),\n",
              "     'span': (14, 21),\n",
              "     'lemma': 'pappi'},\n",
              "    {'id': 4,\n",
              "     'text': 'kanssa',\n",
              "     'upos': 'ADP',\n",
              "     'xpos': 'Adp',\n",
              "     'feats': 'AdpType=Post',\n",
              "     'head': 3,\n",
              "     'deprel': 'case',\n",
              "     'dspan': (22, 28),\n",
              "     'span': (22, 28),\n",
              "     'lemma': 'kanssa'},\n",
              "    {'id': 5,\n",
              "     'text': '.',\n",
              "     'upos': 'PUNCT',\n",
              "     'xpos': 'Punct',\n",
              "     'head': 1,\n",
              "     'deprel': 'punct',\n",
              "     'dspan': (28, 29),\n",
              "     'span': (28, 29),\n",
              "     'lemma': '.'}]}],\n",
              " 'lang': 'finnish'}"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "basic_nlp.ipynb",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.5"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "903d9cd8e1ca40408c157862adc67d0b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_2173dec9fc184ee4b9b9940317b27d3d",
              "IPY_MODEL_17cd0abfa49442758876c5fe51a525c7",
              "IPY_MODEL_1a86784a1b7f4e93afe82873711ac175"
            ],
            "layout": "IPY_MODEL_81cffceacb7d4428901ef0fb73148ede"
          }
        },
        "2173dec9fc184ee4b9b9940317b27d3d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_5864d0d2008c4483b34c418bbc462247",
            "placeholder": "​",
            "style": "IPY_MODEL_e8e957ada4974cd189e8d1552f026e23",
            "value": "Downloading: 100%"
          }
        },
        "17cd0abfa49442758876c5fe51a525c7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f634c9ddd16d4f2e8102622067c191ea",
            "max": 5069051,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_9e48b37cef0841be8049508b63516c09",
            "value": 5069051
          }
        },
        "1a86784a1b7f4e93afe82873711ac175": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_62d3255feff0434986c9a9f598139409",
            "placeholder": "​",
            "style": "IPY_MODEL_2397d5806c9c4d0c960c0f3d2164c7f2",
            "value": " 5.07M/5.07M [00:00&lt;00:00, 26.0MB/s]"
          }
        },
        "81cffceacb7d4428901ef0fb73148ede": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5864d0d2008c4483b34c418bbc462247": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e8e957ada4974cd189e8d1552f026e23": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f634c9ddd16d4f2e8102622067c191ea": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9e48b37cef0841be8049508b63516c09": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "62d3255feff0434986c9a9f598139409": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2397d5806c9c4d0c960c0f3d2164c7f2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "5f86257e5cb44002bc1a7a8d36c74c30": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_0512e158325a496d948e37789d4d03a1",
              "IPY_MODEL_c5dfcd88259c49dea07f6136bcc1bd8c",
              "IPY_MODEL_cb375ccae6b4417dbf6c31797536afb0"
            ],
            "layout": "IPY_MODEL_8f11afe96cac4bf2b7c533af35316d14"
          }
        },
        "0512e158325a496d948e37789d4d03a1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_9035ca3f2bca48cda23fe0e468e214fd",
            "placeholder": "​",
            "style": "IPY_MODEL_662b0f43b0824a979559c00fc4d1fc47",
            "value": "Downloading: 100%"
          }
        },
        "c5dfcd88259c49dea07f6136bcc1bd8c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6f691646a44140b7ab25acd5c3e1c513",
            "max": 512,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_a1973fcfa62545418ceb8f1b67c28317",
            "value": 512
          }
        },
        "cb375ccae6b4417dbf6c31797536afb0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0ead3426a1644e8d919c404da4a83df0",
            "placeholder": "​",
            "style": "IPY_MODEL_c0143d2244b041218fb97537c9e3abf4",
            "value": " 512/512 [00:00&lt;00:00, 16.1kB/s]"
          }
        },
        "8f11afe96cac4bf2b7c533af35316d14": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9035ca3f2bca48cda23fe0e468e214fd": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "662b0f43b0824a979559c00fc4d1fc47": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "6f691646a44140b7ab25acd5c3e1c513": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a1973fcfa62545418ceb8f1b67c28317": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "0ead3426a1644e8d919c404da4a83df0": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c0143d2244b041218fb97537c9e3abf4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ced73a1b7f4745d0b16c609acb5400ea": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_2303befd57ce4f579eef2d63f3e6c393",
              "IPY_MODEL_53a34f960a0e4c6b9c032c165f878806",
              "IPY_MODEL_b202c4c95b4c44aaa1558e461440b2a5"
            ],
            "layout": "IPY_MODEL_ad2d094e1fad4d08a464aa462320d19f"
          }
        },
        "2303befd57ce4f579eef2d63f3e6c393": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0d7123b9eba54e2a97c410da9884de99",
            "placeholder": "​",
            "style": "IPY_MODEL_63737186945f4ce799b726a97043efb8",
            "value": "Downloading: 100%"
          }
        },
        "53a34f960a0e4c6b9c032c165f878806": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_083f7a444ab34af7a11ea284f9b38dcc",
            "max": 1115590446,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_94ad2cd15000434fa06f705cdd50e98c",
            "value": 1115590446
          }
        },
        "b202c4c95b4c44aaa1558e461440b2a5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_de911cfe24ac468fbd7c2953b7fb280f",
            "placeholder": "​",
            "style": "IPY_MODEL_2a5ec0a082fc4f28a6d7fa69d509daca",
            "value": " 1.12G/1.12G [00:22&lt;00:00, 58.5MB/s]"
          }
        },
        "ad2d094e1fad4d08a464aa462320d19f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0d7123b9eba54e2a97c410da9884de99": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "63737186945f4ce799b726a97043efb8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "083f7a444ab34af7a11ea284f9b38dcc": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "94ad2cd15000434fa06f705cdd50e98c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "de911cfe24ac468fbd7c2953b7fb280f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2a5ec0a082fc4f28a6d7fa69d509daca": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}