demo_basic_nlp.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.9"
    },
    "colab": {
      "name": "demo_basic_nlp.ipynb",
      "provenance": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/TurkuNLP/intro-to-nlp/blob/master/demo_basic_nlp.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UbdnU7HznK0m"
      },
      "source": [
        "# Basic NLP exercises\n",
        "\n",
        "* During these exercises, you will learn basic Python skills required in NLP, for example\n",
        "  * Reading and processing language data\n",
        "  * Segmenting text\n",
        "  * Calculating word frequencies and idf weights\n",
        "\n",
        "* Exercises are based on tweets downloaded using Twitter API. Both Finnish and English tweets are available, you are free to choose which language you want to work with.\n",
        "\n",
        "\n",
        "> Finnish: http://dl.turkunlp.org/intro-to-nlp/finnish-tweets-sample.jsonl.gz\n",
        "\n",
        "> English: http://dl.turkunlp.org/intro-to-nlp/english-tweets-sample.jsonl.gz\n",
        "\n",
        "\n",
        "* Both files include 10,000 tweets. If processing the whole file takes too much time, you can also read just a subset of the data, for example only 1,000 tweets.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KmlbS9GLm596"
      },
      "source": [
        "## 1) Read tweets in Python\n",
        "\n",
        "* Download the file, and read the data in Python\n",
        "* **The outcome of this exercise** should be a list of tweets, where each tweet is a dictionary including different (key, value) pairs"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F1CZr0f4m5-E",
        "outputId": "1f24bd5d-017f-4a9c-bab5-5261815a3ca4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# write your code here and run the cell to see your output\n",
        "print(\"Hello world!\")\n",
        "print(\"A\")"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Hello world!\n",
            "A\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XhehNYtz06ce"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X8mHlc_9m5-H"
      },
      "source": [
        "## 2) Extract texts from the tweet jsons\n",
        "\n",
        "* During these exercises we need only the actual tweet text. Inspect the dictionary and extract the actual text field for each tweet.\n",
        "* When carefully inspecting the dictionary keys and values, you may see the old Twitter character limit causing unexpect behavior for text. In these cases, are you able to extract the full text?\n",
        "* **The outcome of this exercise** should be a list of tweets, where each tweet is a string."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H1hCGcbcm5-J"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "04OtJSi8m5-O"
      },
      "source": [
        "## 3) Segment tweets\n",
        "\n",
        "* Segment tweets using the UDPipe machine learned model, remember to select the correct language.\n",
        "\n",
        "> English model: https://github.com/TurkuNLP/intro-to-nlp/raw/master/Data/en.segmenter.udpipe\n",
        "\n",
        "> Finnish model: https://github.com/TurkuNLP/intro-to-nlp/raw/master/Data/fi.segmenter.udpipe\n",
        "\n",
        "* Note that the segmentation model was not trained on tweets, so it may have difficulties in some cases. Inspect the output to get an idea how well it performs on tweets.\n",
        "* Note: In case of the notebook cell dies while trying to load/run the model, the most typical reason is wrong file path or name, or incorrectly downloaded model.\n",
        "* **The output of this excercise** should be a list of segmented tweets, where each tweet is a string."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yRmL8lThm5-P"
      },
      "source": [
        "\n",
        "    "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vgFEubVym5-S"
      },
      "source": [
        "## 4) Calculate word frequencies\n",
        "\n",
        "* Calculate a word frequency list (how many times each word appears) based on the tweets. Which are the most common words appearing in the data?\n",
        "* Calculate the size of your vocabulary (how many unique words there are).\n",
        "* **The output of this excercise** should be a sorted list of X most common words and their frequencies, and the number of unique words in the data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9L-qLoLum5-S"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aaC9e4Rjm5-T"
      },
      "source": [
        "## 5) Calculate idf weights\n",
        "\n",
        "* Calculate idf weight for each word appearing in the data (one tweet = one document), and print top 20 words with lowest and highest idf values.\n",
        "* Can you think of a reason why someone could claim that tf does not have a high impact when processing tweets?\n",
        "* **The output of this excercise** should be a list of words sorted by their idf weights.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1pjNY3H9m5-U"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EwxJM9Rdm5-U"
      },
      "source": [
        "## 6) Duplicates or near duplicates\n",
        "\n",
        "* Check whether we have duplicate tweets (in terms of text field only) in our dataset. Duplicate tweet means here that the exactly same tweet text appears more than once in our dataset.\n",
        "* Note: It makes sense to check the duplicates using original tweet texts as the texts were before segmentation. I would also recommend using the full 10,000 dataset here in order to get higher chance of seeing duplicates (this does not require heavy computing).\n",
        "* Try to check whether tweets have additional near-duplicates. Near duplicate means here that tweet text is almost the same in two or more tweets. Ponder what kind of near duplicates there could be and how to find those. Start by considering for example different normalization techniques. Implement some of the techniques you considered.\n",
        "* **The outcome of this exercise** should be a number of unique tweets in our dataset (with possibly counting also which are the most common duplicates) as well as the number of unique tweets after removing also near duplicates."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1Jt0sZrSm5-V"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}