sequence_labeling_crf.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyPWaZs5xjk0gEx2grp09zpo",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/TurkuNLP/intro-to-nlp/blob/master/sequence_labeling_crf.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Sequence labeling with CRF\n",
        "\n",
        "This notebook provides an example of how to perform sequence labeling with a Conditional Random Field (CRF) model.\n",
        "\n",
        "(This notebook is based on this [notebook for CoNLL 2002 Spanish tagging](https://github.com/TeamHG-Memex/sklearn-crfsuite/blob/master/docs/CoNLL2002.ipynb))"
      ],
      "metadata": {
        "id": "RFGEAWhFZoYy"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "## Setup\n",
        "\n",
        "We'll be using the [sklearn-crfsuite](https://sklearn-crfsuite.readthedocs.io/en/latest/) Python package, which wraps the [CRFsuite](http://www.chokkan.org/software/crfsuite/) CRF implementation.\n",
        "\n",
        "(Specifically, we'll install a fork that fixes a [versioning incompatibility issue](https://github.com/TeamHG-Memex/sklearn-crfsuite/issues/60) with the integration)"
      ],
      "metadata": {
        "id": "Klope-YOaLmm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install --quiet git+https://github.com/MeMartijn/updated-sklearn-crfsuite.git#egg=sklearn_crfsuite"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uhyqgRJHo5Rv",
        "outputId": "4bc8f451-f211-4bcd-9e21-174e665388fb"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "To take advantage of the integration of CRFsuite into the [scikit-learn](https://scikit-learn.org/stable/) machine learning library, we'll here be using parts of that library to support data loading and evaluation. (The basic functionality provided by this library is similar to what you could get from the `datasets` and `evaluate` libraries that you should know by now, and there's no need to try to memorize any of the specific details of these libraries.) "
      ],
      "metadata": {
        "id": "5tor3S1madzC"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "VbMIi8zaojau"
      },
      "outputs": [],
      "source": [
        "from itertools import chain\n",
        "\n",
        "import nltk\n",
        "import sklearn\n",
        "import scipy.stats\n",
        "import sklearn.metrics\n",
        "\n",
        "from nltk.corpus.reader import ConllCorpusReader\n",
        "\n",
        "from sklearn.metrics import make_scorer\n",
        "from sklearn.model_selection import cross_val_score\n",
        "from sklearn.model_selection import RandomizedSearchCV\n",
        "\n",
        "import sklearn_crfsuite\n",
        "from sklearn_crfsuite import scorers\n",
        "from sklearn_crfsuite import metrics\n",
        "from sklearn_crfsuite.utils import flatten"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "## Load data\n",
        "\n",
        "We'll use the CoNLL'03 English dataset, which you should be familiar with. We've placed the original data on dl.turkunlp.org, and can download with `wget` as usual: "
      ],
      "metadata": {
        "id": "0ctbbmzIbY6t"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!wget -nc --quiet http://dl.turkunlp.org/TKO_7095_2023/eng.train\n",
        "!wget -nc --quiet http://dl.turkunlp.org/TKO_7095_2023/eng.testa\n",
        "!wget -nc --quiet http://dl.turkunlp.org/TKO_7095_2023/eng.testb"
      ],
      "metadata": {
        "id": "xMHcKOGgb1Em"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We'll use the NLTK `ConllCorpusReader` to read in the words, parts of speech and named entity \"chunks\""
      ],
      "metadata": {
        "id": "OxzH2EyoelQ4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "train_data = ConllCorpusReader('', 'eng.train', ['words', 'pos', 'ignore', 'chunk'])\n",
        "devel_data = ConllCorpusReader('', 'eng.testa', ['words', 'pos', 'ignore', 'chunk'])\n",
        "test_data = ConllCorpusReader('', 'eng.testa', ['words', 'pos', 'ignore', 'chunk'])\n",
        "\n",
        "# The comprehension here drops empty sentences used to denote document boundaries \n",
        "train_sentences = [s for s in train_data.iob_sents() if s]\n",
        "devel_sentences = [s for s in devel_data.iob_sents() if s]\n",
        "test_sentences = [s for s in devel_data.iob_sents() if s]"
      ],
      "metadata": {
        "id": "x-3Jt5QRdPuw"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "This reader returns a simple Python list representing the sentences, each of which is represented as a list of tuples of token features (form, POS tag, NE tag).\n",
        "\n",
        "Note that unlike in `datasets`, the data is here in \"human-readable\" string form rather than as integer IDs."
      ],
      "metadata": {
        "id": "KtI0SxUblWeK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "train_sentences[0]"
      ],
      "metadata": {
        "id": "uVVNBV5yplN5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "da4677c0-a177-421d-b89a-73894f54f3fb"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('EU', 'NNP', 'I-ORG'),\n",
              " ('rejects', 'VBZ', 'O'),\n",
              " ('German', 'JJ', 'I-MISC'),\n",
              " ('call', 'NN', 'O'),\n",
              " ('to', 'TO', 'O'),\n",
              " ('boycott', 'VB', 'O'),\n",
              " ('British', 'JJ', 'I-MISC'),\n",
              " ('lamb', 'NN', 'O'),\n",
              " ('.', '.', 'O')]"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "## Create features\n",
        "\n",
        "Here a very basic function that returns explicitly defined features for a token"
      ],
      "metadata": {
        "id": "dGrCQrSkh9zS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def token_features(tokens, index):\n",
        "    token = tokens[index][0]\n",
        "    pos_tag = tokens[index][1]\n",
        "    \n",
        "    features = {\n",
        "        'bias': 1.0,\n",
        "        'token.text()': token,\n",
        "        'token.istitle()': token.istitle(),\n",
        "        'token.isdigit()': token.isdigit(),\n",
        "        'pos_tag': pos_tag,\n",
        "    }\n",
        "               \n",
        "    return features"
      ],
      "metadata": {
        "id": "orTF5hBlpo0z"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Some convenience functions to get features for all tokens in a sentence as well as all labels and all token texts."
      ],
      "metadata": {
        "id": "vggsScuniQ8m"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def sentence_features(tokens):\n",
        "    return [token_features(tokens, i) for i in range(len(tokens))]\n",
        "\n",
        "def sentence_labels(tokens):\n",
        "    return [label for token, postag, label in tokens]\n",
        "\n",
        "def sentence_tokens(tokens):\n",
        "    return [token for token, postag, label in tokens]"
      ],
      "metadata": {
        "id": "7asTewAKiNDW"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Check features for a single token"
      ],
      "metadata": {
        "id": "cBjEVBS7l6dR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "sentence_features(train_sentences[0])[0]"
      ],
      "metadata": {
        "id": "dk5c3rKcqBu0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e75d1c59-a547-4732-8985-31bc9be334df"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'bias': 1.0,\n",
              " 'token.text()': 'EU',\n",
              " 'token.istitle()': False,\n",
              " 'token.isdigit()': False,\n",
              " 'pos_tag': 'NNP'}"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Get features and labels for training and development data"
      ],
      "metadata": {
        "id": "qMa3tJbHjThA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "X_train = [sentence_features(s) for s in train_sentences]\n",
        "y_train = [sentence_labels(s) for s in train_sentences]\n",
        "\n",
        "X_devel = [sentence_features(s) for s in devel_sentences]\n",
        "y_devel = [sentence_labels(s) for s in devel_sentences]"
      ],
      "metadata": {
        "id": "THQte3hTqDbR"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "## Train and evaluate model\n",
        "\n",
        "Instantiate CRF. For details on the model hyperparameters, see the [CRFsuite documentation](https://www.chokkan.org/software/crfsuite/)."
      ],
      "metadata": {
        "id": "Su2nWOQ6jZUs"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "crf = sklearn_crfsuite.CRF(\n",
        "    algorithm='lbfgs', \n",
        "    c1=0.1, \n",
        "    c2=0.1, \n",
        "    max_iterations=100, \n",
        "    all_possible_transitions=True\n",
        ")"
      ],
      "metadata": {
        "id": "v-Jhj_qkqUzM"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Train the model"
      ],
      "metadata": {
        "id": "f2vguONKjnLJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "crf.fit(X_train, y_train)"
      ],
      "metadata": {
        "id": "ZSUtNbIGjQI-",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 92
        },
        "outputId": "777734e9-604b-4c2f-e1b1-6357cce88c1b"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "CRF(algorithm='lbfgs', all_possible_transitions=True, c1=0.1, c2=0.1,\n",
              "    max_iterations=100)"
            ],
            "text/html": [
              "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>CRF(algorithm=&#x27;lbfgs&#x27;, all_possible_transitions=True, c1=0.1, c2=0.1,\n",
              "    max_iterations=100)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">CRF</label><div class=\"sk-toggleable__content\"><pre>CRF(algorithm=&#x27;lbfgs&#x27;, all_possible_transitions=True, c1=0.1, c2=0.1,\n",
              "    max_iterations=100)</pre></div></div></div></div></div>"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Get set of POS tags for evaluation, excluding the \"Out\" tag `O`"
      ],
      "metadata": {
        "id": "SFIp-jnEjuUi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "labels = list(crf.classes_)\n",
        "labels.remove('O')\n",
        "labels"
      ],
      "metadata": {
        "id": "xrPY6i55qXaJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "20e761c8-67e1-44f6-d7c1-24f66bab6380"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['I-ORG', 'I-MISC', 'I-PER', 'I-LOC', 'B-LOC', 'B-MISC', 'B-ORG']"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Predict labels for development set"
      ],
      "metadata": {
        "id": "OBzTAFh4j4bF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "y_pred = crf.predict(X_devel)"
      ],
      "metadata": {
        "id": "-m0c_Lw8qbNJ"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Evaluate predictions against gold standard development set labels"
      ],
      "metadata": {
        "id": "cjRLXER_j9mu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "sorted_labels = sorted(labels, key=lambda name: (name[1:], name[0]))\n",
        "\n",
        "print(sklearn.metrics.classification_report(\n",
        "    flatten(y_devel),\n",
        "    flatten(y_pred),\n",
        "    labels=sorted_labels\n",
        "))"
      ],
      "metadata": {
        "id": "NSEiOFvlqeKL",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "35ca9f9a-903c-4256-aa70-271c031d1dfd"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.9/dist-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n",
            "/usr/local/lib/python3.9/dist-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "       B-LOC       0.00      0.00      0.00         0\n",
            "       I-LOC       0.87      0.82      0.85      2094\n",
            "      B-MISC       0.00      0.00      0.00         4\n",
            "      I-MISC       0.93      0.76      0.83      1264\n",
            "       B-ORG       0.00      0.00      0.00         0\n",
            "       I-ORG       0.81      0.76      0.78      2092\n",
            "       I-PER       0.87      0.91      0.89      3149\n",
            "\n",
            "   micro avg       0.86      0.83      0.84      8603\n",
            "   macro avg       0.50      0.46      0.48      8603\n",
            "weighted avg       0.86      0.83      0.84      8603\n",
            "\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.9/dist-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n",
            "/usr/local/lib/python3.9/dist-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n",
            "/usr/local/lib/python3.9/dist-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n",
            "/usr/local/lib/python3.9/dist-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "## Analyze classifier\n",
        "\n",
        "We can look at `transition_features_` to analyze the transition weights learned by the model:"
      ],
      "metadata": {
        "id": "U7dSD_gUobjY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from collections import Counter\n",
        "\n",
        "def print_transitions(trans_features):\n",
        "    for (label_from, label_to), weight in trans_features:\n",
        "        print(\"%-6s -> %-7s %0.6f\" % (label_from, label_to, weight))\n",
        "\n",
        "print(\"Most likely transitions:\")\n",
        "print_transitions(Counter(crf.transition_features_).most_common(10))\n",
        "\n",
        "print(\"\\nLeast likely transitions:\")\n",
        "print_transitions(Counter(crf.transition_features_).most_common()[-10:])"
      ],
      "metadata": {
        "id": "bjtNiBFrqhyZ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "27ed5719-2f68-4327-e44f-78bf7a9bca59"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Most likely transitions:\n",
            "B-ORG  -> B-ORG   5.427859\n",
            "I-MISC -> B-MISC  4.697286\n",
            "I-ORG  -> I-ORG   3.499751\n",
            "I-MISC -> I-MISC  3.493192\n",
            "I-PER  -> I-PER   3.042392\n",
            "B-MISC -> I-MISC  3.006700\n",
            "I-LOC  -> B-LOC   2.983642\n",
            "I-LOC  -> I-LOC   1.332836\n",
            "O      -> I-MISC  1.299373\n",
            "O      -> O       1.245926\n",
            "\n",
            "Least likely transitions:\n",
            "I-MISC -> I-ORG   -2.181237\n",
            "B-MISC -> I-ORG   -2.267659\n",
            "B-ORG  -> O       -2.469499\n",
            "I-MISC -> I-LOC   -2.534070\n",
            "I-PER  -> I-MISC  -3.184330\n",
            "I-ORG  -> I-LOC   -3.394631\n",
            "I-PER  -> I-LOC   -3.760992\n",
            "I-LOC  -> I-PER   -4.260426\n",
            "I-LOC  -> I-ORG   -4.329928\n",
            "I-PER  -> I-ORG   -5.490100\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Similarly, we can look at `state_features_` to look at the probabilities of the explicit features we introduced:"
      ],
      "metadata": {
        "id": "bKYDOb4pol0x"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def print_state_features(state_features):\n",
        "    for (attr, label), weight in state_features:\n",
        "        print(\"%0.6f %-8s %s\" % (weight, label, attr))    \n",
        "\n",
        "print(\"Top positive:\")\n",
        "print_state_features(Counter(crf.state_features_).most_common(10))\n",
        "\n",
        "print(\"\\nTop negative:\")\n",
        "print_state_features(Counter(crf.state_features_).most_common()[-10:])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CeOjznEsm5-G",
        "outputId": "aad771d4-ae59-439e-ee44-ebc8220028e2"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Top positive:\n",
            "9.048645 O        token.text():Minister\n",
            "8.736580 I-MISC   token.text():GMT\n",
            "8.716987 I-MISC   token.text():DUTCH\n",
            "8.449610 O        token.text():President\n",
            "8.298575 I-LOC    token.text():AMSTERDAM\n",
            "8.130931 I-ORG    token.text():1860\n",
            "8.032552 I-LOC    token.text():BONN\n",
            "7.993696 I-ORG    token.text():OSCE\n",
            "7.771414 I-PER    token.text():Inzamam-ul-Haq\n",
            "7.749895 I-LOC    token.text():ATHENS\n",
            "\n",
            "Top negative:\n",
            "-2.377139 O        token.text():serie\n",
            "-2.407187 O        token.text():de\n",
            "-2.409392 O        pos_tag:NNPS\n",
            "-2.496532 O        token.text():154\n",
            "-2.781862 O        token.text():95\n",
            "-2.858090 O        token.text():04\n",
            "-3.306775 B-MISC   bias\n",
            "-3.451022 O        token.text():NEW\n",
            "-3.565962 O        token.text():ST\n",
            "-4.336524 B-LOC    bias\n"
          ]
        }
      ]
    }
  ]
}