Notebook_9_2024.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/TurkuNLP/ATP_kurssi/blob/master/Notebook_9_2024.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Please use the server to do these exercises. If you are unfamiliar with scripts, you can also start with the time-out exercises listed on Notebook8**\n",
        "\n",
        "\n",
        "### Tips\n",
        "\n",
        "**About perl**\n",
        "\n",
        "A perl script can be useful for modifying strings. This example would replace all instances of REPLACED with ORIGINAL. You place the character(s) you want to replace between the first and second / and the replacement between the second and the third /\n",
        "\n",
        "```\n",
        "cat inputfile.txt | perl -pe 's/ORIGINAL/REPLACED/g'\n",
        "```\n",
        "\n",
        "E.g.,\n",
        "\n",
        "\n",
        "`perl -pe 's/ /\\n/g'`\n",
        "\n",
        "replaces whitespace with a new line\n",
        "\n",
        "**NOTE 1**\n",
        "\n",
        "You can use perl with **regular expressions** as well, e.g.,\n",
        "\n",
        "\n",
        "`perl -pe 's/[A-Z]//g'`\n",
        "\n",
        "replaces any capital letter with nothing\n",
        "\n",
        "\n",
        "`perl -pe 's/[[:punct:]]$//g'`\n",
        "\n",
        "replaces any punctuation in the end of line with nothing\n",
        "\n",
        "\n",
        "`perl -pe 's/^[0-9]/#/g'`\n",
        "\n",
        "replaces any one number in the beginning of a line with #\n",
        "\n",
        "\n",
        "`perl -pe 's/^[0-9]+/#/g'`\n",
        "\n",
        "replaces any number that occurs once or more in the beginning of a line with #\n",
        "\n",
        "**NOTE 2**\n",
        "\n",
        "This script supports also **back reference**, so you can refer to expressions in `ORIGINAL` and refer to those with `$` in `REPLACED`. E.g., here any punctuation is replaced by a newline and that punctuation\n",
        "\n",
        "```\n",
        " perl -pe 's/([[:punct:]])/\\n$1/g'\n",
        "```\n",
        "\n",
        "Note that the expression you refer back to must be within ()\n",
        "\n",
        "----------------\n",
        "\n",
        "**About scripts**\n",
        "\n",
        "Note that in a script, you can have various commands, and all the commands will be executed one by one (with the exception of pipes, where the commands are executed all at once).\n",
        "\n",
        "For instance if a script has the two commands below, it will first print \"kukkuu\" to a file called delme.txt, and then, undepending of the first one, print the file and egrep lines correspomding to \"kuu\".\n",
        "\n",
        "```\n",
        "echo \"kukkuu\" > delme.txt\n",
        "\n",
        "cat delme.txt | egrep \"kuu\"\n",
        "```\n",
        "\n",
        "----------------\n",
        "\n",
        "**About arguments**\n",
        "\n",
        "Arguments can be used to refer to also other parts in the script than just the file.\n",
        "\n",
        "E.g., if you want to give the line of characters you need to grep from the command line, you refer to the line of characters within the script with $ and running numbering:\n",
        "\n",
        "in the script\n",
        "\n",
        "`egrep $1`\n",
        "\n",
        "when you run the script\n",
        "\n",
        "`cat file.txt | ./script.sh whatyouwanttogrep`\n",
        "\n",
        "E.g.,\n",
        "\n",
        "`cat file.txt | ./script.sh snow`\n",
        "\n",
        "\n",
        "\n",
        "----------------\n",
        "\n",
        "## Exercise 1\n",
        "\n",
        "The Github repo https://github.com/MarkHershey/CompleteTrumpTweetsArchive has all the tweets published by Donald Trump when he was at office. Clone the repo to your home folder on the server.\n",
        "\n",
        "### 1.1\n",
        "Count the most frequent hashtags and / or handles of the dataset covering Tweets when Trump was in office. Make sure to ignore possible punctuations, such as\n",
        "```\n",
        "@realDonaldTrump:\n",
        "@realDonaldTrump\n",
        "```\n",
        "### 1.2\n",
        "\n",
        "Make a script that takes as argument a handle and prints out its distribution over time month by month. Run the script on a couple of interesting handles / hashtags. Do you see any trends?\n",
        "\n",
        "**Extra:** sort the output so that you have the tweets ordered from older to newer, followed by the number of tweets for that time stamp, like\n",
        "\n",
        "```\n",
        "YEAR-MONTH1 NUM-OF-TWEETS\n",
        "YEAR-MONTH2 NUM-OF-TWEETS\n",
        "```\n",
        "If you get a permission denied error, you have forgotten to add execution rights to your script. This can be done with `chmod a+rwx file.txt`\n",
        "\n",
        "### 1.3\n",
        "Make another script that takes as argument a handle and prints out a cleaned and normalized frequency list of the words that occur in the tweets with the handle.\n",
        "\n",
        "You can try out different ways of cleaning the data. Does it make sense to include tokens with numbers and / or punctuation at all? Or is it better to just, e.g., delete tokens and numbers and otherwise otherwise keep the strings there?\n",
        "\n",
        "### 1.4 Advanced\n",
        "\n",
        "The output of 1.3 would be much better if it excluded function words, don't you think?\n",
        "\n",
        "Trankit is super slow and cannot be run on the server. But UdPipe can - the performance is lower, but maybe it would be enough. Let's try!\n",
        "\n",
        "UdPipe documentation can be found at https://lindat.mff.cuni.cz/services/udpipe/api-reference.php\n",
        "\n",
        "With the following call, UdPipe prints to standard output the conllu analysis of the input file, here called delme.csv\n",
        "\n",
        "`curl -F data=@delme.csv -F model=english -F tokenizer= -F tagger= -F parser= http://lindat.mff.cuni.cz/services/udpipe/api/process | PYTHONIOENCODING=utf-8 python -c \"import sys,json; sys.stdout.write(json.load(sys.stdin)['result'])\" `\n",
        "\n",
        "If you have time, try to implement a script that is similar to 1.3 but outputs only content POS classes, such as nouns, verbs, etc, instead of all most frequent words, and lemmas instead of running words.\n",
        "\n",
        "You can try with different POS classes to figure out the most useful output in your opinion. What happens if you include only nouns? or proper nouns? or adjetives? or all these classes?\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "2YaAm2zJ0jGn"
      }
    }
  ]
}