Introduction_Information_Extraction.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "0BtcW_CoGDbE"
      ]
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0BtcW_CoGDbE"
      },
      "source": [
        "## Setup\n",
        "\n",
        "The code in this cell prepares the files and libraries needed. You can run it without expanding its contents (but of course you can peek into it if you're curious!)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WlpSDgIdtvIB",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ef034bff-5af8-4dd1-a978-5eb26f9b2846"
      },
      "source": [
        "%%bash\n",
        "# Transformers installation\n",
        "pip install transformers\n",
        "# To install from source instead of the last release, comment the command above and uncomment the following one.\n",
        "#pip install git+https://github.com/huggingface/transformers.git\n",
        "pip install Wikipedia-API\n",
        "pip install pyspotlight\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting transformers\n",
            "  Downloading transformers-4.26.1-py3-none-any.whl (6.3 MB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.3/6.3 MB 85.5 MB/s eta 0:00:00\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.8/dist-packages (from transformers) (4.64.1)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from transformers) (3.9.0)\n",
            "Collecting tokenizers!=0.11.3,<0.14,>=0.11.1\n",
            "  Downloading tokenizers-0.13.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.6 MB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.6/7.6 MB 112.1 MB/s eta 0:00:00\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.8/dist-packages (from transformers) (2022.6.2)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.8/dist-packages (from transformers) (1.21.6)\n",
            "Collecting huggingface-hub<1.0,>=0.11.0\n",
            "  Downloading huggingface_hub-0.12.1-py3-none-any.whl (190 kB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 190.3/190.3 KB 26.6 MB/s eta 0:00:00\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.8/dist-packages (from transformers) (23.0)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from transformers) (2.25.1)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.8/dist-packages (from transformers) (6.0)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.8/dist-packages (from huggingface-hub<1.0,>=0.11.0->transformers) (4.5.0)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (2022.12.7)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (2.10)\n",
            "Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (4.0.0)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (1.24.3)\n",
            "Installing collected packages: tokenizers, huggingface-hub, transformers\n",
            "Successfully installed huggingface-hub-0.12.1 tokenizers-0.13.2 transformers-4.26.1\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting Wikipedia-API\n",
            "  Downloading Wikipedia_API-0.5.8-py3-none-any.whl (13 kB)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from Wikipedia-API) (2.25.1)\n",
            "Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests->Wikipedia-API) (4.0.0)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests->Wikipedia-API) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests->Wikipedia-API) (2022.12.7)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->Wikipedia-API) (1.24.3)\n",
            "Installing collected packages: Wikipedia-API\n",
            "Successfully installed Wikipedia-API-0.5.8\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting pyspotlight\n",
            "  Downloading pyspotlight-0.7.2-py2.py3-none-any.whl (15 kB)\n",
            "Requirement already satisfied: requests~=2.10 in /usr/local/lib/python3.8/dist-packages (from pyspotlight) (2.25.1)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests~=2.10->pyspotlight) (2.10)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests~=2.10->pyspotlight) (1.24.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests~=2.10->pyspotlight) (2022.12.7)\n",
            "Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests~=2.10->pyspotlight) (4.0.0)\n",
            "Installing collected packages: pyspotlight\n",
            "Successfully installed pyspotlight-0.7.2\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import requests\n",
        "from bs4 import BeautifulSoup\n",
        "from pprint import pprint\n",
        "\n",
        "from transformers import pipeline, AutoTokenizer, AutoModelForSeq2SeqLM\n",
        "import os, random, json, logging, csv\n",
        "import wikipediaapi\n",
        "import spotlight\n",
        "\n",
        "import torch"
      ],
      "metadata": {
        "id": "KWw61ntA3-q6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6XL2cZTyR6sR"
      },
      "source": [
        "# Entity linking methods\n",
        "\n",
        "We are first looking at entity linking on natural language text by using a popular online tool: DBpedia Spotlight.\n",
        "It takes sentences as an input and returns entity URI's from the Wikipedia-based knowledge graph DBpedia.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "idGyTpMXTwTO"
      },
      "source": [
        "## DBpedia Spotlight 🔦\n",
        "\n",
        "[DBpedia Spotlight](https://www.dbpedia-spotlight.org/) is a tool for annotating text with metadata about entities. It is based on a pipeline that performs named entity recognition, candidate generation, and entity linking.\n",
        "\n",
        "The following code defines a function that takes as input some text and returns the annotated response from the Spotlight API."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hV29u83jR-R-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8c59be00-8a27-435c-f9c5-68416796ad23"
      },
      "source": [
        "\n",
        "\n",
        "def spotlight_link(sentence):\n",
        "    parameters = {'text' : sentence}\n",
        "    r = requests.get(\"https://api.dbpedia-spotlight.org/en/annotate\", params=parameters)\n",
        "    soup = BeautifulSoup(r.text)\n",
        "    body = soup.find('body')\n",
        "    return body.prettify()\n",
        "\n",
        "pprint(spotlight_link('The president of the United States visited Vietnam.'))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<body>\n",
            "<div>\n",
            "The president of the <a href=\"http://dbpedia.org/resource/United_States\" target=\"_blank\" title=\"http://dbpedia.org/resource/United_States\">United States</a> visited <a href=\"http://dbpedia.org/resource/Vietnam\" target=\"_blank\" title=\"http://dbpedia.org/resource/Vietnam\">Vietnam</a>.\n",
            "</div>\n",
            "</body>\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Prompt-based Relation Extraction\n",
        "\n",
        "Instead of fine-tuning a relation extraction model, which often takes several GPU hours/days for training, existing pre-trained language models can be directly used for relation extraction.\n",
        "\n",
        "Here, we only demonstrate a very basic approach for prompt-based relation extraction:\n",
        "\n",
        "We prompt the language model with the input sentence that we want to extract the triple from and the subject/object entities. The goal of the model is to find a word which best fits between the subject and the object entity. \n",
        "\n",
        "#### Problems: 🙅\n",
        "\n",
        "We still need to map back from the predicted word to the relation in the knowledge graph.\n",
        "\n",
        "*Try out the code from the celle below*\n",
        "\n",
        "As you can see, the model correctly predicts the word *directed*. \n"
      ],
      "metadata": {
        "id": "fF5iYy4UJbs5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "generator = pipeline(model='facebook/opt-1.3b')\n",
        "\n",
        "input_sentence = \"Inception is a 2010 science fiction action film directed by Christopher Nolan.\"\n",
        "subj = \"Inception\"\n",
        "obj = \"Christopher Nolan\"\n",
        "generator(f\"{input_sentence} What is the relation between {subj} and {obj}?\")"
      ],
      "metadata": {
        "id": "bQCLCJ4hJ2S_",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "90ad474c-8d20-4067-efbd-9cff13a0b346"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Input length of input_ids is 26, but `max_length` is set to 21. This can lead to unexpected behavior. You should consider increasing `max_new_tokens`.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'generated_text': 'Inception is a 2010 science fiction action film directed by Christopher Nolan. What is the relation between Inception and Christopher Nolan?\\n'}]"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "##End-to-End Information Extraction\n",
        "\n",
        "In this part, we will have a explore end-to-end information extractionwith the important NLP library 🤗  Transformers. \n",
        "Instead of performing entity recognition, entity linking, and relation extraction separately, recent Transformer models can be used to perform all tasks in a single step.\n"
      ],
      "metadata": {
        "id": "gU7XauyIglrN"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "###  Extract Information with the REBEL Model\n",
        "\n",
        "Instead of training our own machine learning model here, we download a model, which is trained on the distantly supervised training dataset from Wikipedia and WIkidata. The model that we are working with is a fine-tuned generative language model for information extraction. It is based on the Transformer model 🤖 [REBEL](https://huggingface.co/Babelscape/rebel-large).\n",
        "\n",
        "The Huggingface library offers lots of existing pre-trained models for a variety of tasks that can be easily downloaded and used for various NLP tasks. "
      ],
      "metadata": {
        "id": "4ZR7XUxP7TLF"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Helper methods"
      ],
      "metadata": {
        "id": "z7unntSE_X0t"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def call_wiki_api(item):\n",
        "  try:\n",
        "    url = f\"https://www.wikidata.org/w/api.php?action=wbsearchentities&search={item}&language=en&format=json\"\n",
        "    data = requests.get(url).json()\n",
        "    # Return the first id (Could upgrade this in the future)\n",
        "    return data['search'][0]['id']\n",
        "  except:\n",
        "    return item\n",
        "\n",
        "\n",
        "def write2csv(file_name, triples):\n",
        "  with open(file_name, \"w\", newline=\"\") as f:\n",
        "      writer = csv.writer(f)\n",
        "      writer.writerows(triples)\n",
        "\n",
        "def annotate(text):\n",
        "  try:\n",
        "    spotlight_results = spotlight.annotate('https://api.dbpedia-spotlight.org/en/annotate',text)\n",
        "    urls = []\n",
        "    for r in spotlight_results:\n",
        "      urls.append(r['URI'])\n",
        "    return urls\n",
        "  except:\n",
        "    print(f'No entity found for {text}')\n",
        "\n",
        "#Return the Wikipedia abstract\n",
        "def get_wikipedia_abstract(url):\n",
        "    wiki_wiki = wikipediaapi.Wikipedia('en')\n",
        "    page_name = url.replace('http://dbpedia.org/resource/','')\n",
        "    page_py = wiki_wiki.page(page_name)\n",
        "    return page_py.summary\n",
        "\n",
        "def extract_triplets(text):\n",
        "    triplets = []\n",
        "    relation, subject, relation, object_ = '', '', '', ''\n",
        "    text = text.strip()\n",
        "    current = 'x'\n",
        "    for token in text.replace(\"<s>\", \"\").replace(\"<pad>\", \"\").replace(\"</s>\", \"\").split():\n",
        "        if token == \"<triplet>\":\n",
        "            current = 't'\n",
        "            if relation != '':\n",
        "                triplets.append([subject.strip(), relation.strip(), object_.strip()])\n",
        "                relation = ''\n",
        "            subject = ''\n",
        "        elif token == \"<subj>\":\n",
        "            current = 's'\n",
        "            if relation != '':\n",
        "                triplets.append([subject.strip(), relation.strip(), object_.strip()])\n",
        "            object_ = ''\n",
        "        elif token == \"<obj>\":\n",
        "            current = 'o'\n",
        "            relation = ''\n",
        "        else:\n",
        "            if current == 't':\n",
        "                subject += ' ' + token\n",
        "            elif current == 's':\n",
        "                object_ += ' ' + token\n",
        "            elif current == 'o':\n",
        "                relation += ' ' + token\n",
        "    if subject != '' and relation != '' and object_ != '':\n",
        "        triplets.append([subject.strip(), relation.strip(), object_.strip()])\n",
        "    return triplets"
      ],
      "metadata": {
        "id": "2WJjPPZy-cNY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Extraction\n",
        "\n",
        "This code extracts triples and writes them to a .csv file."
      ],
      "metadata": {
        "id": "BNwhSraJ_ejF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Text to extract triplets from\n",
        "text = 'Punta Cana is a resort town in the municipality of Higüey, in La Altagracia Province, the easternmost province of the Dominican Republic.'\n",
        "\n",
        "\n",
        "# Use GPU if available\n",
        "device = \"cuda:0\" if torch.cuda.is_available() else \"cpu\"\n",
        "\n",
        "\n",
        "\n",
        "#Load model and tokenizer\n",
        "tokenizer = AutoTokenizer.from_pretrained(\"Babelscape/rebel-large\")\n",
        "model = AutoModelForSeq2SeqLM.from_pretrained(\"Babelscape/rebel-large\").to(device)\n",
        "gen_kwargs = {\n",
        "    \"max_length\": 256,\n",
        "    \"length_penalty\": 0,\n",
        "    \"num_beams\": 3,\n",
        "    \"num_return_sequences\": 1,\n",
        "}\n",
        "\n",
        "\n",
        "\n",
        "# Tokenizer text\n",
        "model_inputs = tokenizer(text, max_length=256, padding=True, truncation=True, return_tensors = 'pt').to(device)\n",
        "\n",
        "# Generate\n",
        "generated_tokens = model.generate(\n",
        "    model_inputs[\"input_ids\"].to(device),\n",
        "    attention_mask=model_inputs[\"attention_mask\"].to(device),\n",
        "    **gen_kwargs,\n",
        ")\n",
        "\n",
        "# Extract text\n",
        "decoded_preds = tokenizer.batch_decode(generated_tokens, skip_special_tokens=False)\n",
        "\n",
        "# Extract triplets\n",
        "for idx, sentence in enumerate(decoded_preds):\n",
        "  print(f'Prediction triplets sentence {idx}')\n",
        "  triples = extract_triplets(sentence)   \n",
        "  print(triples)\n",
        "  write2csv('output.csv', triples)"
      ],
      "metadata": {
        "id": "rWX0ly8JttBz",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b1d88823-abc0-4243-8a57-4a897c2e70ed"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Prediction triplets sentence 0\n",
            "[['Punta Cana', 'located in the administrative territorial entity', 'La Altagracia Province'], ['Punta Cana', 'country', 'Dominican Republic'], ['Higüey', 'located in the administrative territorial entity', 'La Altagracia Province'], ['Higüey', 'country', 'Dominican Republic'], ['La Altagracia Province', 'country', 'Dominican Republic'], ['Dominican Republic', 'contains administrative territorial entity', 'La Altagracia Province']]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Hands-On 💻\n",
        "\n",
        "Until here, we have seen on how to use DBpedia Spotlight for extracting entities, using prompt-based relation extraction, and REBEL as an end-to-end model for relation extraction from text.\n",
        "\n",
        "We will now try to combine your knowledge with the previous hands-on exercises by extracting additional triples from Wikipedia to enrich your ontology.\n",
        "1. Link artist names to Wikipedia with Dbpedia Spotlight.\n",
        "2. Get the Wikipedia Abstracts for the artists.\n",
        "3. Perform relation extraction with REBEL on these Wikipedia Abstracts and save the triples to a .csv file.\n",
        "\n",
        "Below you find some first Python code for the usage of two APIs that we will use.\n",
        "\n"
      ],
      "metadata": {
        "id": "cr0Ki7Ydj52C"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Example code for usage of DBpedia Spotlight and for getting the respective Wikipedia Abstract.\n",
        "\n",
        "#Read the artist names from .csv\n",
        "artists = []\n",
        "with open('artists.csv', newline='') as csvfile:\n",
        "    artistreader = csv.reader(csvfile, delimiter='\\t')\n",
        "    #skip header\n",
        "    next(artistreader)\n",
        "    for row in artistreader:\n",
        "      artists.append(row[1])\n",
        "\n",
        "\n",
        "#Use Dbpedia Spotlight API to link entity\n",
        "url = annotate('Travis Scott')\n",
        "print(url)\n",
        "abstract = get_wikipedia_abstract(url[0])\n",
        "print(abstract)\n",
        "\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vpgSW2jUlyKw",
        "outputId": "31c4df50-f557-494c-b688-f07f0f1b0abc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['http://dbpedia.org/resource/Travis_Scott']\n",
            "Jacques Bermon Webster II (born April 30, 1991), better known by his stage name Travis Scott (formerly stylized as Travi$ Scott), is an American rapper, singer, songwriter, and record producer. His stage name is the namesake of a favorite uncle combined with the first name of one of his inspirations, Kid Cudi (whose real name is Scott Mescudi).In 2012, Scott signed his first major-label contract with Epic Records and a publishing deal with Kanye West's GOOD Music. In April 2013, he signed a joint-recording contract with Epic and T.I.'s Grand Hustle imprint. Scott's first full-length project, the mixtape Owl Pharaoh, was self-released in 2013. It was followed with a second mixtape, Days Before Rodeo, in 2014. His debut studio album, Rodeo (2015), was led by the hit single \"Antidote\". His second album, Birds in the Trap Sing McKnight (2016), became his first number one album on the Billboard 200. The following year, Scott released a collaborative album with Quavo titled Huncho Jack, Jack Huncho under the group name Huncho Jack.\n",
            "In 2018, his third studio album, Astroworld, was released to critical acclaim and produced his first Billboard Hot 100 number one single, \"Sicko Mode\" (featuring Drake). In late 2019, Scott's record label Cactus Jack Records released the compilation album JackBoys, which topped the Billboard 200. After the release of his single \"Franchise\" (featuring Young Thug and M.I.A.) in 2020, Scott became the first artist on the Hot 100 to have three songs debut at number one in less than a year.Over the course of his career, Scott has become a globally recognized artist and pop culture figure. Along with his highly publicized relationship with American media personality Kylie Jenner, Scott has collaborated with organizations including Nike, Dior, and McDonald's. In 2017, he founded the record label Cactus Jack Records. Throughout his career, Scott has achieved four number-one hits on the Billboard Hot 100, along with 80 total charted songs. Additionally, he has been nominated for eight Grammy Awards and won a Billboard Music Award, Latin Grammy Award, MTV Video Music Award, and multiple BET Hip Hop Awards.\n",
            "Scott has also gained notoriety for controversies and legal issues regarding safety at his concerts. In November 2021, a mass-casualty crowd crush occurred during Scott's performance at the Astroworld Festival in his hometown of Houston, Texas, resulting in widespread condemnation of the artist.\n"
          ]
        }
      ]
    }
  ]
}