task_8_transitions.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyNBJIw+BbMrMJ9M7L4ohBx5",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/TurkuNLP/intro-to-nlp/blob/master/task_8_transitions.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Task 8: POS transition probabilities\n",
        "\n",
        "In the lecture, we briefly saw the concept of hidden markov models and transition probabilies, as applied to POS tags. In the simplest case, these probabilities model the sequences of POS tag pairs such that e.g. probability of DET -> NOUN will be the probability of seeing a NOUN, having seen a DET (determiner), i.e. more formally P(NOUN|DET). We also had the intuition that for example DET -> NOUN should be much larger than, say DET -> VERB. And of course, since these are probabilities, sum of P(x|y) over all x should sum up\n",
        "to 1 for any given y. These probabilities can be easily estimated by counting from the data, i.e. the probability of DET -> NOUN transition, i.e. P(NOUN|DET) is simply the count of how many times you saw NOUN following a DET, divided by how many times you saw DET.\n",
        "\n",
        "Your task is to pick a Universal Dependencies dataset of your choice, e.g. UD_English-EWT training data, calculate these transition probabilities, pretty-print them if you can, and check that our intuitions hold, i.e. that for example DET -> NOUN is substantially more likely than, say, DET -> VERB.\n",
        "\n"
      ],
      "metadata": {
        "id": "-OJBvK7u7PT6"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MyPHNDWjtwB8"
      },
      "outputs": [],
      "source": [
        "# Grab the data\n",
        "\n",
        "!wget https://github.com/UniversalDependencies/UD_English-EWT/archive/refs/heads/master.zip\n",
        "!unzip master.zip"
      ]
    }
  ]
}