sentence_splitting_and_tokenization.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyOndzZYFs9ZsjB4nTuT2Gvb",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/TurkuNLP/intro-to-nlp/blob/master/sentence_splitting_and_tokenization.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Sentence splitting and tokenization example\n",
        "\n",
        "This short notebook illustrates one comparatively fast way to do sentence splitting and tokenization in Python. It's not particularly accurate at either, but should do the job in cases where the details don't matter too much.\n",
        "\n",
        "We'll use the [sentence-splitter](https://pypi.org/project/sentence-splitter/) and [regex](https://pypi.org/project/regex/) packages. "
      ],
      "metadata": {
        "id": "OZ16vxvAqdKV"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "id": "FdjkVj9DoO5a"
      },
      "outputs": [],
      "source": [
        "!pip install --quiet sentence-splitter regex "
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Grab some example data"
      ],
      "metadata": {
        "id": "cVWL_xRdrK6G"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!wget -nc http://dl.turkunlp.org/TKO_7095_2023/fiwiki-20221120-sample.txt"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zGXFZ2ZNoSTk",
        "outputId": "d9bbde43-38b1-4352-9b78-25f46f5c9839"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "File ‘fiwiki-20221120-sample.txt’ already there; not retrieving.\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Read in example data in paragraph-per-line format"
      ],
      "metadata": {
        "id": "tZhPmsM9rOpj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "paragraphs = open('fiwiki-20221120-sample.txt').readlines()"
      ],
      "metadata": {
        "id": "fx2506pGoqCW"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Instantiate sentence splitter. Note that you need to provide the language, and not all languages are supported."
      ],
      "metadata": {
        "id": "7pqTKikyrWKc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sentence_splitter import SentenceSplitter\n",
        "\n",
        "splitter = SentenceSplitter(language='fi')"
      ],
      "metadata": {
        "id": "UeK5rcDNoxgM"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Run sentence splitting and log runtime. Just take some paragraphs from the start to keep things reasonably fast."
      ],
      "metadata": {
        "id": "WEFqCEbLrd_5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "\n",
        "sentences = [s for p in paragraphs[:100000] for s in splitter.split(p)]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "alhd_yIgpAbO",
        "outputId": "e365a26e-02e6-41ea-dbec-c82900a108e8"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 1min 5s, sys: 344 ms, total: 1min 5s\n",
            "Wall time: 1min 7s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Split into tokens using a regular expression. Here the regular expression defines as a token any sequence of alphanumeric characters or any (other) single non-space character. "
      ],
      "metadata": {
        "id": "xN6m8TE_riXM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import regex\n",
        "\n",
        "TOKENIZE_RE = regex.compile(r'([[:alnum:]]+|\\S)')"
      ],
      "metadata": {
        "id": "t9V-nxRIpGxK"
      },
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Tokenize and log runtime"
      ],
      "metadata": {
        "id": "7i9iVu8wrwoi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "\n",
        "tokenized = [TOKENIZE_RE.findall(s) for s in sentences]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "h9pRUUYDp3w6",
        "outputId": "e7568e94-a01e-46ad-d165-c6d7facb253a"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 6.11 s, sys: 742 ms, total: 6.85 s\n",
            "Wall time: 6.95 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Check a few examples"
      ],
      "metadata": {
        "id": "nCscgzWsr5oy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for t in tokenized[:10]:\n",
        "    print(t)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LmUGB8eDqAyl",
        "outputId": "a7deca0b-1cb7-4217-c0fa-d3c7d86f915f"
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['Patrick', 'Joseph', 'Leahy', '(', 's', '.']\n",
            "['31', '.', 'maaliskuuta', '1940', 'Montpelier', ',', 'Vermont', ')', 'on', 'yhdysvaltalainen', 'demokraattisen', 'puolueen', 'poliitikko', '.']\n",
            "['Leahy', 'toimii', 'Yhdysvaltain', 'senaatin', 'president', 'pro', 'temporena', 'eli', 'de', 'facto', 'senaatin', 'varapresidenttinä', '.']\n",
            "['Hän', 'on', 'toiminut', 'Vermontin', 'osavaltion', 'senaattorina', 'vuodesta', '1975', '.']\n",
            "['Grassley', 'myös', 'toimi', 'senaatin', 'president', 'pro', 'temporena', 'joulukuusta', '2012', 'tammikuuhun', '2015', '.']\n",
            "['Hän', 'on', 'ollut', 'myös', 'senaatin', 'oikeusvaliokunnan', 'puheenjohtaja', '.']\n",
            "['Elävä', 'kuollut', 'eli', 'epäkuollut', 'tarkoittaa', 'yleisesti', 'erilaisia', 'taruolentoja', ',', 'jotka', 'ovat', 'heränneet', 'kuolleista', 'takaisin', 'elävien', 'maailmaan', '.']\n",
            "['Populaarikulttuurissa', 'tunnetuimpia', 'eläviä', 'kuolleita', 'ovat', 'vampyyrit', 'ja', 'zombit', '.']\n",
            "['Sanan', \"'\", 'epäkuollut', \"'\", 'kehitti', 'kääntäjä', 'Kersti', 'Juva', 'englannin', 'sanan', '\"', 'undead', '\"', 'vastineeksi', 'kääntäessään', 'J', '.', 'R', '.', 'R', '.', 'Tolkienin', '\"', 'Tarua', 'sormusten', 'herrasta', '\"', '.']\n",
            "['Isku', 'pelasi', 'ensimmäisen', 'kautensa', 'mestaruussarjassa', 'vuonna', '1962', ',', 'kun', 'vuoden', '1961', 'Suomen', '-', 'mestari', 'Järvensivun', 'Kisa', 'luopui', 'sarjapaikastaan', 'Iskun', 'hyväksi', '.']\n"
          ]
        }
      ]
    }
  ]
}