notebooks/TextExtraction.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "<a href=\"https://colab.research.google.com/github/LUMII-AILab/NLP_Course/blob/main/notebooks/TextExtraction.ipynb\" target=\"_new\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
      ],
      "metadata": {
        "id": "s09Svv85NZfD"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Teksta izgūšana | Text extraction\n",
        "\n",
        "**LV**: Teksta korpusa izveide ir viens no priekšnoteikumiem daudzos valodas modelēšanas uzdevumos. Tā kā vienkāršs teksts (*plain-text*) bieži vien nav pieejams, nākas nodarboties ar vienkārša teksta izguvi no dokumentiem, kas pieejami citos formātos.\n",
        "\n",
        "Šajā piezīmju grāmatiņā ir apskatīti trīs vienkāršoti gadījumi teksta izguvei no dažāda formāta dokumentiem: HTML, PDF un VERT.\n",
        "\n",
        "Populārajiem dokumentu formātiem (PDF, DOCX, HTML u.c.) eksistē dažādas *Python* bibliotēkas, kuras var izmantot vienkāršā teksta izguves uzdevumam, kā tas ir nodemonstrēts šajā nodarbībā.\n",
        "\n",
        "---\n",
        "\n",
        "**EN**: Creating a text corpus is one of the prerequisites for many language modeling tasks. Since plain-text is often not available, one has to deal with plain-text extraction from documents that are available in other formats.\n",
        "\n",
        "This notebook covers three simplified cases for extracting text from documents in various formats: HTML, PDF and VERT.\n",
        "\n",
        "For the popular document formats (PDF, DOCX, HTML, etc.), there are various Python libraries that can be used for the plain-text extraction task as demonstrated in this session."
      ],
      "metadata": {
        "id": "BbtrO9w3zVaO"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## HTML-to-Text\n",
        "\n",
        "**LV**: Viena no populārākajām un vienkāršāk izmantojamajām *Python* bibliotēkām HTML formāta dokumentu parsēšanai un satura \"noskrāpēšanai\" (*web scraping*) ir *BeautifulSoup* (https://pypi.org/project/beautifulsoup4/).\n",
        "\n",
        "*BeautifulSoup* savukārt izmanto zemāka līmeņa HTML/XML parsētāju: var tikt izmantots gan *Python* iebūvētais `html.parser`, gan ārējas bibliotēkas (piem., `lxml`, `html5lib`), kas nodrošina dažādas priekšrocības (piem., ātrdarbību un papildu funkcionalitāti). Šajā demonstrācijā ir izmantots iebūvētais HTML parsētājs.\n",
        "\n",
        "---\n",
        "\n",
        "**EN**: One of the most popular and easy-to-use Python libraries for parsing HTML documents and web scraping is BeautifulSoup (https://pypi.org/project/beautifulsoup4/).\n",
        "\n",
        "BeautifulSoup uses a lower-level HTML/XML parser - both Python's built-in `html.parser` and external libraries (e.g. `lxml`, `html5lib`) can be used. While the external libraries can provide various advantages w.r.t. performance and functionality, this demo uses the built-in HTML parser."
      ],
      "metadata": {
        "id": "mi-kondA0Eqd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install beautifulsoup4"
      ],
      "metadata": {
        "id": "IGKT3JxM9etu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!wget -O \"sample_article.html\" https://www.vti.lu.lv/par-mums/zinas/zina/t/82316/"
      ],
      "metadata": {
        "id": "TtXR-Hof8K2y"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "r6U6v5VezUNd"
      },
      "outputs": [],
      "source": [
        "from bs4 import BeautifulSoup\n",
        "import html\n",
        "import re\n",
        "\n",
        "\n",
        "# Removes specific HTML elements from a webpage to get rid of needless content.\n",
        "# For instance: header and footer blocks, menus, etc.\n",
        "# Needs to be adapted for each website to get the best results.\n",
        "def remove_html_elements(text):\n",
        "    soup = BeautifulSoup(text, \"html.parser\")\n",
        "\n",
        "    # Filtering by specific HTML elements\n",
        "    for element in soup.find_all([\"header\", \"footer\", \"button\"]):\n",
        "        element.decompose() # Removes an element from the tree\n",
        "\n",
        "    # Filtering by HTML elements having specific attributes\n",
        "    for element in soup.find_all([\"div\"], attrs={\"class\": re.compile(\".*([Mm]enu|share|backlink).*\")}):\n",
        "        element.decompose()\n",
        "\n",
        "    return str(soup)\n",
        "\n",
        "\n",
        "# (1) Unescapes HTML entities.\n",
        "# (2) Removes HTML tags while keeping the content.\n",
        "# For instance: &amp; => &, <p>content</p> => content.\n",
        "# This function is universal - it can be applied to any webpage from any website.\n",
        "def convert_to_plaintext(text):\n",
        "    text = html.unescape(text)                     # 1\n",
        "    text = BeautifulSoup(text, \"html.parser\").text # 2\n",
        "    return text\n",
        "\n",
        "\n",
        "# Normalizes spaces and line breaks in the plain-text.\n",
        "def normalize_white_spaces(text):\n",
        "    text = re.sub(\"[ ]+\", \" \", text)\n",
        "    text = re.sub(\"[ ]?\\n+\", \"\\n\", text) # Try to comment out this line\n",
        "    return text\n",
        "\n",
        "\n",
        "# (1) Removal of needless HTML elements.\n",
        "# (2) Unescaping HTML entities and removal of HTML tags.\n",
        "# (3) Normalization of whitespaces in the plain-text.\n",
        "def html_to_txt(html_file, txt_file):\n",
        "    text = \"\"\n",
        "\n",
        "    with open(html_file, \"r\", encoding=\"utf-8\") as input_file:\n",
        "        text = input_file.read()\n",
        "\n",
        "    text = remove_html_elements(text)   # 1\n",
        "    text = convert_to_plaintext(text)   # 2\n",
        "    text = normalize_white_spaces(text) # 3\n",
        "\n",
        "    with open(txt_file, \"w\", encoding=\"utf-8\") as output_file:\n",
        "        output_file.write(text)\n",
        "\n",
        "\n",
        "html_to_txt(\"sample_article.html\", \"sample_article.txt\")"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## PDF-to-Text\n",
        "\n",
        "**LV**: Saistīta teksta izguve no PDF dokumentiem lielākoties ir sarežģīta un ķēpīga: informācija par teksta struktūru un noformējumu bieži vien nav viennozīmīgi izgūstama, un teksta segmentēšana teikumos un rindkopās ir apgrūtināta, jo PDF formāts ir veidots satura drukāšanas nevis mašīnlasīšanas vajadzībām.\n",
        "\n",
        "Arī PDF dokumentu apstrādei ir pieejamas dažādas ārējās *Python* bibliotēkas, piemēram, `pypdf`, `PyPDF2`, `PDFMiner`, `PyMuPDF`, `tabula-py`. Demonstrācijas nolūkiem izmantosim `pypdf` (https://pypi.org/project/pypdf/).\n",
        "\n",
        "Plašāk par PDF dokumentu mašīnlasīšanas problemātiku aprakstīts `pypdf` [dokumentācijā](https://pypdf.readthedocs.io/en/stable/user/extract-text.html).\n",
        "\n",
        "Eksperimentēšanas vērta alternatīva pieeja: konvertēt PDF uz HTML un tālāk strādāt ar HTML dokumentiem. Konvertēšanas funkcionalitāti nodrošina, piemēram, `PDFMiner` (https://pypi.org/project/pdfminer/).\n",
        "\n",
        "---\n",
        "\n",
        "**EN**: Retrieving continuous text from PDF documents is generally difficult and cumbersome: information about the text structure and presentation often cannot be unambiguously retrieved. Segmenting such text into sentences and paragraphs is difficult, since the PDF format is designed for printing rather than machine-reading the content.\n",
        "\n",
        "Various external Python libraries are available for processing PDF documents, such as `pypdf`, `PyPDF2`, `PDFMiner`, `PyMuPDF`, `tabula-py`. For demonstration purposes, we will use `pypdf` (https://pypi.org/project/pypdf/).\n",
        "\n",
        "The problem of text extraction from PDF documents is described in more detail [here](https://pypdf.readthedocs.io/en/stable/user/extract-text.html).\n",
        "\n",
        "An alternative approach worth experimenting is to convert PDF to HTML and then work with HTML documents. Conversion functionality is provided, for example, by `PDFMiner` (https://pypi.org/project/pdfminer/)."
      ],
      "metadata": {
        "id": "y2b4ioRTQBj1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install pypdf"
      ],
      "metadata": {
        "id": "fNOYOjaAP91t"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!wget -O \"sample_paper.pdf\" https://www.apgads.lu.lv/fileadmin/user_upload/lu_portal/apgads/PDF/Valoda-nozime-forma/VNF-10/vnf_10-16_Nespore_Saulite_Rituma.pdf"
      ],
      "metadata": {
        "id": "bD4oVHMhAxXJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from pypdf import PdfReader\n",
        "\n",
        "\n",
        "# Provides a very basic text extraction functionality\n",
        "def pdf_to_txt(pdf_file, txt_file):\n",
        "    text = \"\"\n",
        "\n",
        "    with open(pdf_file, \"rb\") as input_file:\n",
        "        reader = PdfReader(input_file)\n",
        "\n",
        "        # Reads the document page by page\n",
        "        for page in reader.pages:\n",
        "            text += page.extract_text() + \"\\n\"\n",
        "\n",
        "    with open(txt_file, \"w\", encoding=\"utf-8\") as output_file:\n",
        "        output_file.write(normalize_white_spaces(text))\n",
        "\n",
        "    print(\"Total number of lines in the text:\", text.count(\"\\n\"))\n",
        "\n",
        "\n",
        "pdf_to_txt(\"sample_paper.pdf\", \"sample_paper.txt\")"
      ],
      "metadata": {
        "id": "YrxiI6npQSbM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**LV**: Papētot iegūto rezultātu (`sample_paper.txt`), redzams, ka ar tik vienkāršiem soļiem nepietiek, lai no PDF dokumenta izgūtu kvalitatīvu tekstu.\n",
        "\n",
        "Izteiktākā problēma ir tā, ka izgūtajā tekstā ir saglabāts teksta dalījums rindās un lappusēs tā, kā tas drukas vajadzībām ir izkārtots PDF dokumentā, bet mums ir nepieciešams teksts, kas būtu strukturēts atbilstoši teikumu un rindkopu robežām, nevis dokumenta vizuālajam noformējumam.\n",
        "\n",
        "Lai iegūtu vēlamo rezultātu (t.i., tuvinātos tam), ir jāveic papildu soļi teksta noformējuma analīzē un atbilstošā pēcapstrādē.\n",
        "\n",
        "---\n",
        "\n",
        "**EN**: By examining the obtained result (`sample_paper.txt`), we can see that such a simple approach is not enough to extract quality text from a PDF document.\n",
        "\n",
        "The most obvious problem is that the retrieved text preserves the line and page split which follows the layout of the PDF document. However, we need the text to be structured according to sentence and paragraph boundaries, not the visual layout of the document.\n",
        "\n",
        "To obtain the desired result (i.e. to approximate it), additional steps are required to analyze the text layout and formatting and to adjust the post-processing accordingly."
      ],
      "metadata": {
        "id": "pbJPfX0hgDmk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Analyzes (heuristically) line breaks and merges lines if necessary.\n",
        "def merge_lines(text):\n",
        "    # If a line ends with a lower case letter followed by a hyphen,\n",
        "    # we *assume* this is a hyhenation of a word.\n",
        "    text = re.sub(r\"(?<=[a-zāčēģīķļņšūž])[--]\\n(?=[a-zāčēģīķļņšūž])\", \"\", text)\n",
        "\n",
        "    # If a line begins with a lower case letter,\n",
        "    # we *assume* this is a continuation of a sentence.\n",
        "    text = re.sub(r\"(\\n)+(?=[a-zāčēģīķļņšūž])\", \" \", text) # FIXME: \\p{Ll}\n",
        "\n",
        "    return text\n",
        "\n",
        "\n",
        "# A more elaborate implementation of the basic text extractor\n",
        "def pdf_to_txt_2(pdf_file, txt_file):\n",
        "    text = \"\"\n",
        "\n",
        "    with open(pdf_file, \"rb\") as input_file:\n",
        "        reader = PdfReader(input_file)\n",
        "\n",
        "        for page in reader.pages:\n",
        "            text += page.extract_text() + \"\\n\"\n",
        "\n",
        "    text = merge_lines(normalize_white_spaces(text))\n",
        "\n",
        "    with open(txt_file, \"w\", encoding=\"utf-8\") as output_file:\n",
        "        output_file.write(text)\n",
        "\n",
        "    print(\"Total number of lines in the text:\", text.count(\"\\n\"))\n",
        "\n",
        "\n",
        "pdf_to_txt_2(\"sample_paper.pdf\", \"sample_paper_2.txt\")\n",
        "\n",
        "# TODO: a potential mini-project - develop an advanced PDF-to-Text extractor"
      ],
      "metadata": {
        "id": "1kTb92qOCe_w"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## VERT-to-Text\n",
        "\n",
        "**LV**: Dažkārt nākas saskarties ar failu formātiem, kas nav plaši izplatīti vai tiek izmantoti pamatā tikai valodu tehnoloģiju jomā, un to apstrādei nav pieejamas jau gatavas bibliotēkas, vai arī pastāv vairāki varianti, kā attiecīgais datu formāts var tikt realizēts vai interpretēts.\n",
        "\n",
        "Daži piemēri: CoNLL, VERT, VRT, TSV3. Tie ir specifiski *tab-separated* datu apmaiņas formāti, kas valodu tehnoloģiju jomā tiek izmantoti dažāda veida anotētiem valodas resursiem. Lai izgūtu saistītu tekstu, nepieciešams analizēt faila struktūru, nolasīt nepieciešamos datu laukus un rekonstruēt tekstu.\n",
        "\n",
        "Demonstrācijas nolūkā apskatīsim VERT formātu, kuru izmanto populārā teksta korpusu platforma [SketchEngine](https://www.sketchengine.eu/my_keywords/vertical/) un tās atvērtā pirmkoda versija [NoSketchEngine](https://nlp.fi.muni.cz/trac/noske). Šis formāts tiek izmantots arī latviešu valodas korpusu kolekcijā [Korpuss.lv](https://korpuss.lv/).\n",
        "\n",
        "Lai no VERT faila iegūtu vienkāršu, saistītu tekstu ir nepieciešams ievērot teksta segmentēšanu teikumos un teikumu segmentēšanu tekstvienībās atbilstoši VERT failā lietotajam strukturālajam marķējumam.\n",
        "\n",
        "Demonstrācijai izmantosim atvērto Raiņa tekstu korpusu (2,3 milj. tekstvienību), kas pieejams CLARIN-LV repozitorijā: http://hdl.handle.net/20.500.12574/41\n",
        "\n",
        "---\n",
        "\n",
        "**EN**: Sometimes you have to deal with file formats that are not widely used or are used only in the field of language technology, thus, there might be no ready-made libraries available for processing these formats. Also, there might several alternatives how such data formats are implemented or interpreted.\n",
        "\n",
        "Some examples: CoNLL, VERT, VRT, TSV3. These are specific tab-separated file formats used to encode and exchange annotated language resources. To retrieve continuous text, it is necessary to analyze the file structure, read the necessary data fields and reconstruct the text.\n",
        "\n",
        "For demonstration purposes, we will consider the VERT format used by the popular text corpora platform [SketchEngine](https://www.sketchengine.eu/my_keywords/vertical/) and its open source version [NoSketchEngine](https://nlp.fi.muni.cz/trac/noske). This format is used also by the the Latvian corpora collection [Korpuss.lv](https://korpuss.lv/).\n",
        "\n",
        "In order to obtain plain and continuous text from a VERT file, it is necessary to follow the segmentation of the text into sentences and the segmentation of sentences into tokens according to the structural markup used in the VERT file.\n",
        "\n",
        "We will use the open text corpus of Rainis (a Latvian poet; 2.3M tokens), which can be downloaded from the CLARIN-LV repository: http://hdl.handle.net/20.500.12574/41"
      ],
      "metadata": {
        "id": "ayvqvSpIDB_z"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!wget -O \"sample_corpus.vert\" https://repository.clarin.lv/repository/xmlui/bitstream/handle/20.500.12574/41/rainis_v20180716.vert"
      ],
      "metadata": {
        "id": "9CR8SGBtICnT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Reads a VERT file, line by line, and reconstructs sentences and paragraphs.\n",
        "def vert_to_txt(vert_file, txt_file):\n",
        "    input_file = open(vert_file, \"r\", encoding=\"utf-8\")\n",
        "    output_file = open(txt_file, \"w\", encoding=\"utf-8\")\n",
        "\n",
        "    text = \"\"\n",
        "\n",
        "    while True:\n",
        "        line = input_file.readline()\n",
        "\n",
        "        if not line: break\n",
        "\n",
        "        if line == \"\\n\":\n",
        "            if text != \"\":\n",
        "                output_file.write(text + \"\\n\")\n",
        "            text = \"\"\n",
        "\n",
        "        # If a line contains a tag, it has to be processed accordingly\n",
        "        elif line[0] == \"<\" and line[1] != \"\\t\":\n",
        "\n",
        "            # </doc>, </p>, </s> - end of a document, paragraph, sentence:\n",
        "            # outputs the so far constructed text segment and begins a new one\n",
        "            if line == \"</doc>\\n\":\n",
        "                if text[:-1] == \" \": text = text[:-1]\n",
        "                output_file.write(text + \"\\n\\n\")\n",
        "                text = \"\"\n",
        "            elif line == \"</p>\\n\":\n",
        "                if text[:-1] == \" \": text = text[:-1]\n",
        "                output_file.write(text + \"\\n\")\n",
        "                text = \"\"\n",
        "            elif line == \"</s>\\n\":\n",
        "                if text[:-1] == \" \": text = text[:-1]\n",
        "                output_file.write(text)\n",
        "                text = \"\"\n",
        "\n",
        "            # <g/> - \"glue\" tag - there should be no space between the consecutive tokens,\n",
        "            # e.g. a word and the following punctuation mark\n",
        "            elif line == \"<g />\\n\" and len(text) > 1:\n",
        "                if text[-1] == \" \":\n",
        "                    text = text[:-1]\n",
        "\n",
        "            # Ignores the opening <doc>, <p>, <s> tags (as well as other tags)\n",
        "            else:\n",
        "                continue\n",
        "\n",
        "        # If a line does not contain a structural tag, it contains a text token,\n",
        "        # which is the first element of the tab-separated line\n",
        "        else:\n",
        "            text = text + line.split(\"\\t\")[0] + \" \"\n",
        "\n",
        "    input_file.close()\n",
        "    output_file.close()\n",
        "\n",
        "\n",
        "vert_to_txt(\"sample_corpus.vert\", \"sample_corpus.txt\")\n",
        "\n",
        "# TODO: create a CoNLL-U to plain-text converter to extract a multilingual corpus\n",
        "# from the https://universaldependencies.org 200+ treebanks!"
      ],
      "metadata": {
        "id": "uYnTQQW6FM7O"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}