unsupervised_nlp.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/TurkuNLP/intro-to-nlp/blob/master/unsupervised_nlp.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "32xQBnkw5wsp"
      },
      "source": [
        "# Unsupervised methods in NLP\n",
        "\n",
        "\n",
        "\n",
        "### Themes:\n",
        "- Part I: Analyzing topics (unsupervised NLP basics)\n",
        "    - Term weighting with TF-IDF\n",
        "    - Document clustering\n",
        "- Part II: Transfer learning (modern NLP)\n",
        "    - Word vectors (word2vec)\n",
        "    - Pre-trained language models (ELMo, BERT)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "meZuXD275wst"
      },
      "source": [
        "## Overview\n",
        "\n",
        "- Goal: identify meaningful relationships in raw text *internally*\n",
        "    - No need for manual annotation (as in supervised learning)\n",
        "- Unsupervised methods involve:\n",
        "    - Counting frequencies of word occurence in different contexts\n",
        "    - Counting/estimating frequencies of word co-occurrence in different contexts\n",
        "    - Machine learning methods for clustering objects (e.g. documents)\n",
        "    - Machine learning methods for modeling word sequences (language modeling)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eHePwQzM5wst"
      },
      "source": [
        "# Part I: Analyzing topics (unsupervised NLP basics)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rmhbDofw5wsu"
      },
      "source": [
        "## Term weighting with TF-IDF\n",
        "\n",
        "### Recap from \"Basic NLP\" lecture: _tf-idf weighting_\n",
        "\n",
        "* TF = term frequency *tf(t, d)*, how many times the term _t_ appears in document *d*\n",
        "* DF = document frequency *df(t)*, in how many documents the term *t* appears\n",
        "* IDF = inverse document frequency, *m/df(t)*, where *m* is the total number of documents in your collection\n",
        "* TF-IDF = **tf(t, d) * idf(t)**\n",
        "    * Usually calculated using logaritmic (sublinear) scale --> tf(t, d) * log(idf(t))\n",
        "    \n",
        "* common in information retrieval, also used in document classification and clustering\n",
        "* scale down the impact of tokens that occur very frequently in many documents and are hence empirically less informative/distinctive than words that occur in a small fraction of the documents"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "21Ek2aZl5wsv"
      },
      "source": [
        "More reading on [tf-idf](https://nlp.stanford.edu/IR-book/html/htmledition/term-frequency-and-weighting-1.html) and [scaling and normalization variations](https://nlp.stanford.edu/IR-book/html/htmledition/variant-tf-idf-functions-1.html) (in the context of information retrieval)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ibtk6kaY5wsv",
        "outputId": "f8349216-7344-4130-f512-7dfd9dbf8cc9"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2022-02-16 09:45:32--  https://github.com/TurkuNLP/intro-to-nlp/raw/master/Data/patents.txt\n",
            "Resolving github.com (github.com)... 140.82.113.4\n",
            "Connecting to github.com (github.com)|140.82.113.4|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://raw.githubusercontent.com/TurkuNLP/intro-to-nlp/master/Data/patents.txt [following]\n",
            "--2022-02-16 09:45:32--  https://raw.githubusercontent.com/TurkuNLP/intro-to-nlp/master/Data/patents.txt\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 10851473 (10M) [text/plain]\n",
            "Saving to: ‘patents.txt’\n",
            "\n",
            "patents.txt         100%[===================>]  10.35M  --.-KB/s    in 0.1s    \n",
            "\n",
            "2022-02-16 09:45:33 (102 MB/s) - ‘patents.txt’ saved [10851473/10851473]\n",
            "\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(13990, 5000)"
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ],
      "source": [
        "### Calculate TF-IDF scores on a corpus\n",
        "\n",
        "import json\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "\n",
        "!wget https://github.com/TurkuNLP/intro-to-nlp/raw/master/Data/patents.txt\n",
        "with open(\"patents.txt\", \"rt\", encoding=\"utf-8\") as f:\n",
        "    documents = [line.strip() for line in f.readlines()]\n",
        "\n",
        "# Set parameters and initialize\n",
        "tfidf_vectorizer = TfidfVectorizer(min_df=2, use_idf=True, sublinear_tf=True, max_df=1.0, max_features=5000)\n",
        "# Tip: the vectorizer also supports extracting n-gram features (common short sequences of words), which may be more descriptive but also much less frequent\n",
        "\n",
        "# Calcualate term-document matrix with tf-idf scores\n",
        "tfidf_matrix = tfidf_vectorizer.fit_transform(documents)\n",
        "\n",
        "# Check matrix shape\n",
        "tfidf_matrix.toarray().shape # N_docs x N_terms"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 127
        },
        "id": "clmIZqpf5ws0",
        "outputId": "6425526e-cb7d-494e-ce0a-b74967c4435d"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "\"A fund-raising computer network and method for simplifying and inducing the giving of monetary contributions by a multiplicity of prospective contributors consisting of a computer with communication circuitry for communicating over a computer network data sets comprised of requests soliciting the making of monetary contributions to a plurality of remote prospective contributors' computers and, a means for receiving data sets comprised of contributions from the remote contributors' respective computers communicated over a computer network. The network and method induces and simplifies the making of contributions. Contributions include political contributions, charitable contributions, and religious contributions. Requests for contributions include multi-media presentations communicated to prospective contributors' computers which are networked to computer networks.\""
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ],
      "source": [
        "documents[0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YeaV0Tay5ws1",
        "outputId": "ff564c22-beb6-4628-8293-0a40c13363e3"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['00', '000', '10', '100', '101', '102', '103', '104', '105', '106']\n",
            "['yen', 'yet', 'yield', 'yields', 'you', 'your', 'zero', 'zip', 'zone', 'zones']\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function get_feature_names is deprecated; get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use get_feature_names_out instead.\n",
            "  warnings.warn(msg, category=FutureWarning)\n"
          ]
        }
      ],
      "source": [
        "# Inspect terms in vocabulary\n",
        "print(tfidf_vectorizer.get_feature_names()[:10])\n",
        "print(tfidf_vectorizer.get_feature_names()[-10:])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "txavg8BF5ws2",
        "outputId": "87169496-3ae0-489a-bead-2f8ba93c20ca"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function get_feature_names is deprecated; get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use get_feature_names_out instead.\n",
            "  warnings.warn(msg, category=FutureWarning)\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.        ],\n",
              "       [0.        ],\n",
              "       [0.        ],\n",
              "       ...,\n",
              "       [0.16807321],\n",
              "       [0.        ],\n",
              "       [0.        ]])"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ],
      "source": [
        "tfidf_matrix[:,tfidf_vectorizer.get_feature_names().index('record')].toarray() # Get doc vector for term"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_DMS0s795ws3",
        "outputId": "5c03bc34-ac5d-4dbc-aaa5-0dea35825b26"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "13656\tthe\n",
            "13433\tand\n",
            "12903\tto\n",
            "12876\tof\n",
            "11967\tfor\n",
            "9739\tin\n",
            "9380\tis\n",
            "9316\tan\n",
            "7418\tor\n",
            "7404\tsystem\n",
            "7262\twith\n",
            "7213\tmethod\n",
            "6626\tby\n",
            "6495\ton\n",
            "6194\tthat\n",
            "6152\tfrom\n",
            "5997\tare\n",
            "5601\tbe\n",
            "5312\tone\n",
            "5000\tat\n"
          ]
        }
      ],
      "source": [
        "## Inspect document frequencies (counts) of terms\n",
        "\n",
        "from collections import Counter\n",
        "terms_in_docs = tfidf_vectorizer.inverse_transform(tfidf_matrix)\n",
        "token_counter = Counter()\n",
        "for terms in terms_in_docs:\n",
        "    token_counter.update(terms)\n",
        "\n",
        "for term, count in token_counter.most_common(20):\n",
        "    print(\"%d\\t%s\" % (count, term))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dkwieSM25ws3",
        "outputId": "8641bd96-2690-4aba-f75f-d7f51aac2a2e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[('the', 1.0241620550548614), ('and', 1.040625458906476), ('to', 1.080876924928241), ('of', 1.0829714914595021), ('for', 1.1561778445190274), ('in', 1.3621731480967052), ('is', 1.399727898607335), ('an', 1.406573577282861), ('or', 1.6343699885437601), ('system', 1.6362588180313868), ('with', 1.6556212991271178), ('method', 1.6623906832698014), ('by', 1.747262052645059), ('on', 1.767227662891772), ('that', 1.8146717506700432), ('from', 1.8214744979927957), ('are', 1.8469881854243315), ('be', 1.9152905889132321), ('one', 1.9682576182823426), ('at', 2.028776373314382)]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function get_feature_names is deprecated; get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use get_feature_names_out instead.\n",
            "  warnings.warn(msg, category=FutureWarning)\n"
          ]
        }
      ],
      "source": [
        "# Alternatively: Inspect IDF values directly\n",
        "features = tfidf_vectorizer.get_feature_names()\n",
        "print(sorted(zip(features,tfidf_vectorizer._tfidf.idf_),key=lambda x:x[1])[:20])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0t99bMe75ws4",
        "outputId": "07105a75-fdab-4f2b-9b89-19eb7fcda2fe"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function get_feature_names is deprecated; get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use get_feature_names_out instead.\n",
            "  warnings.warn(msg, category=FutureWarning)\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Document 0, top terms by TF-IDF\n",
            "0.41\tcontributions\n",
            "0.36\tcontributors\n",
            "0.23\tprospective\n",
            "0.23\tcomputers\n",
            "0.22\tcomprised\n",
            "\n",
            "Document 1, top terms by TF-IDF\n",
            "0.40\tchecking\n",
            "0.37\tname\n",
            "0.24\tcode\n",
            "0.24\treal\n",
            "0.23\tbank\n",
            "\n",
            "Document 2, top terms by TF-IDF\n",
            "0.32\tadditional\n",
            "0.26\tbasic\n",
            "0.25\tserver\n",
            "0.25\tdepending\n",
            "0.24\tselected\n",
            "\n",
            "Document 3, top terms by TF-IDF\n",
            "0.37\ttaxable\n",
            "0.28\tperson\n",
            "0.28\tsum\n",
            "0.25\tperiodic\n",
            "0.24\tmoney\n",
            "\n",
            "Document 4, top terms by TF-IDF\n",
            "0.32\ttranslation\n",
            "0.29\tcurrency\n",
            "0.17\toptimal\n",
            "0.16\ttarget\n",
            "0.16\tdynamic\n"
          ]
        }
      ],
      "source": [
        "## Inspect top terms per document\n",
        "\n",
        "features = tfidf_vectorizer.get_feature_names()\n",
        "for doc_i in range(5):\n",
        "    print(\"\\nDocument %d, top terms by TF-IDF\" % doc_i)\n",
        "    for term, score in sorted(list(zip(features,tfidf_matrix.toarray()[doc_i])), key=lambda x:-x[1])[:5]:\n",
        "        print(\"%.2f\\t%s\" % (score, term))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "66YbQYaC5ws4"
      },
      "source": [
        "## Document clustering\n",
        "\n",
        "- Clustering: grouping of similar objects in order to structure a set\n",
        "    - E.g., clustering documents in a corpus -> reveal common topics and provide overview\n",
        "- TF-IDF weighting for extracting meaningful *features* of documents, highlighting descriptive terms\n",
        "- Goal: Given features, calculate similarities between documents and assign them into clusters"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ac0nCJk15ws5"
      },
      "source": [
        "- The term-document matrix (`tfidf_matrix`) represents the corpus as tf-idf weights (features)\n",
        "    - Each document represented as a (row) vector\n",
        "    - Document vectors are sparse:\n",
        "        - Dimensionality equal to the number of terms extracted from the corpus\n",
        "        - Non-zero values only for the fraction of terms in document"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "C-zLseAH5ws5",
        "outputId": "b4a18973-7c19-42cd-b051-ad6a1e15f85f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[0. 0. 0. ... 0. 0. 0.]\n",
            " [0. 0. 0. ... 0. 0. 0.]\n",
            " [0. 0. 0. ... 0. 0. 0.]\n",
            " ...\n",
            " [0. 0. 0. ... 0. 0. 0.]\n",
            " [0. 0. 0. ... 0. 0. 0.]\n",
            " [0. 0. 0. ... 0. 0. 0.]]\n"
          ]
        }
      ],
      "source": [
        "print(tfidf_matrix.toarray())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SO-q4Tcl5ws5",
        "outputId": "abd7f954-257c-4736-e820-5badaff313eb"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Document vector length: 5000\n",
            "Non-zero dimensions for document 0: 46\n",
            "Non-zero dimensions for document 1: 45\n",
            "Non-zero dimensions for document 2: 41\n",
            "Non-zero dimensions for document 3: 56\n",
            "Non-zero dimensions for document 4: 94\n"
          ]
        }
      ],
      "source": [
        "print(\"Document vector length:\", tfidf_matrix.shape[1])\n",
        "for i in range(5):\n",
        "    print(\"Non-zero dimensions for document %d: %d\" % (i, len([x for x in tfidf_matrix.toarray()[i] if x > 0])))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1ZNJIEVr5ws6",
        "outputId": "d3c25c1e-9d32-42f5-f324-6f6b5871867d"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Sample word: confirmation\n",
            "Occurs in 80 documents\n",
            "out of 13990 documents\n"
          ]
        }
      ],
      "source": [
        "print(\"Sample word:\", features[1000])\n",
        "print(\"Occurs in %d documents\" % len([x for x in tfidf_matrix.toarray()[:][1000] if x > 0]))\n",
        "print(\"out of %d documents\" % len(tfidf_matrix.toarray()))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z_2PXvWj5ws6"
      },
      "source": [
        "### Vector space model\n",
        "\n",
        "- A vector space model encodes meaning as coordinates in a high-dimensional space\n",
        "    - E.g., tf-idf term vectors describe the topic of a document\n",
        "- Semantic similarities are measured as distances in the space:\n",
        "    - For vectors *x* and _y_:\n",
        "        - Euclidean distance: $$d = \\sqrt{ (x_1 - y_1)^2 + (x_2 - y_2)^2 + ... + (x_n - y_n)^2 }$$\n",
        "        - Cosine similarity (distance = 1-similarity): $$s = cos(\\sigma) = \\frac{x \\cdot y}{|x||y|}$$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lbrtwBKX5ws6"
      },
      "source": [
        "- We use vector space notion of semantic similarity to group documents into clusters\n",
        "    - Find common topics, go from many documents to fewer clusters "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aDN36jWp5ws7"
      },
      "source": [
        "More reading on the [vector space model](https://nlp.stanford.edu/IR-book/html/htmledition/the-vector-space-model-for-scoring-1.html)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n1KUwyYb5ws7"
      },
      "source": [
        "### k-means clustering\n",
        "\n",
        "- We will cluster documents based on their tf-idf vectors \n",
        "- and the relatively simple and widely used *k-means* method\n",
        "    - Takes as argument *k* number of clusters to be found\n",
        "    - Uses a random initialization (i.e., results can vary between runs)\n",
        "    - Works according to this pseudo code (from the [Mining Massive Datasets book](http://infolab.stanford.edu/~ullman/mmds/ch7.pdf)):\n",
        "\n",
        "<code>Initially choose k points that are likely to be in different clusters;\n",
        "Make these points the centroids of their clusters;\n",
        "FOR each remaining point p DO \n",
        "    find the centroid to which p is closest;\n",
        "    Add p to the cluster of that centroid;\n",
        "    Adjust the centroid of that cluster to account for p;\n",
        "END;</code>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "LguU_p8s5ws7"
      },
      "outputs": [],
      "source": [
        "# We use a subset of the documents in clustering for faster calculation and easier interpretation of results\n",
        "matrix_sample = tfidf_matrix[:1000]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SwRwFOBM5ws7",
        "outputId": "9dab72fe-8bc4-4e6d-9c2f-82a5c7f8a8b2"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "KMeans(n_clusters=30, random_state=123)"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ],
      "source": [
        "from sklearn.cluster import KMeans\n",
        "\n",
        "# Do clustering\n",
        "km = KMeans(n_clusters=30, random_state=123, verbose=0)\n",
        "km.fit(matrix_sample)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "ZcxLR5wi5ws8"
      },
      "outputs": [],
      "source": [
        "import heapq, numpy as np\n",
        "\n",
        "# Custom function to print top keywords for each cluster\n",
        "def print_clusters(matrix, clusters, n_keywords=10):\n",
        "    for cluster in range(min(clusters), max(clusters)+1):\n",
        "        cluster_docs = [i for i, c in enumerate(clusters) if c == cluster]\n",
        "        print(\"Cluster: %d (%d docs)\" % (cluster, len(cluster_docs)))\n",
        "        \n",
        "        # Keep scores for top n terms\n",
        "        new_matrix = np.zeros((len(cluster_docs), matrix.shape[1]))\n",
        "        for cluster_i, doc_vec in enumerate(matrix[cluster_docs].toarray()):\n",
        "            for idx, score in heapq.nlargest(n_keywords, enumerate(doc_vec), key=lambda x:x[1]):\n",
        "                new_matrix[cluster_i][idx] = score\n",
        "\n",
        "        # Aggregate scores for kept top terms\n",
        "        keywords = heapq.nlargest(n_keywords, zip(new_matrix.sum(axis=0), features))\n",
        "        print(', '.join([w for s,w in keywords]))\n",
        "        print()\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kWkLvNZz5ws8",
        "outputId": "6347a4ec-b0d3-43c7-8cb3-f786c620030f"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([27, 27, 22, 14, 21, 14,  3, 10,  5,  1,  0, 24, 21, 26,  0,  8, 25,\n",
              "       20, 15, 15, 17, 21, 21,  2, 28, 21,  7, 12, 19,  2, 19, 15, 21, 12,\n",
              "       20,  6,  8, 10, 11, 27, 15, 17,  5,  4, 14,  0, 28,  4, 29, 24, 27,\n",
              "        1, 17,  6, 12, 27, 21,  6, 18, 20, 21,  9,  9,  6,  9, 12,  7,  0,\n",
              "       13, 19, 15, 21, 21,  4, 10,  0,  9,  9,  1,  9, 18, 12,  0,  3,  0,\n",
              "       10, 18,  6, 12,  0, 15,  6, 20,  0, 10, 10, 10, 18, 15,  6, 25,  9,\n",
              "       23,  8,  0, 14, 26, 12,  5, 24,  0, 27, 26,  6, 10, 21,  3, 15, 26,\n",
              "        9,  9,  0, 20, 27, 18, 19, 27,  7, 20, 22,  8,  9, 17, 24, 15, 27,\n",
              "       23,  0, 18, 14,  7,  8,  1,  8, 29,  7, 24,  0,  5,  1,  4,  6,  1,\n",
              "       13, 18,  0, 15, 20,  3, 10, 22, 27, 12,  4,  8, 23,  9, 18,  0,  0,\n",
              "       17,  6, 15,  5,  6,  6,  6,  5,  3, 17,  8,  8,  7,  6, 25, 15,  0,\n",
              "       26, 23, 15, 24,  4, 20, 27, 17,  7, 18,  8, 14, 25, 22, 14,  8, 16,\n",
              "       17,  8,  0, 28, 24, 19,  7,  4, 19, 18, 14, 17, 26,  0,  6, 18, 22,\n",
              "       15,  8,  4, 27,  0, 10,  0, 12, 28, 27,  8, 25,  6, 27,  6, 29, 20,\n",
              "       14,  8, 27, 26, 19, 29, 17, 15,  4, 25, 11, 26, 25, 26,  6,  0,  0,\n",
              "        2, 28,  0, 29,  8, 25, 21, 11, 11,  8,  0, 15,  1,  1,  0, 28,  8,\n",
              "       20, 21, 10, 11,  2, 20, 12, 18, 12, 14,  9,  0, 17,  9, 20,  0, 24,\n",
              "        0, 17, 12, 10,  4, 27, 25, 14,  6, 28, 25,  6, 20, 21, 28,  1, 23,\n",
              "       26, 22,  3,  7, 25, 18, 28,  7, 12, 23,  4, 14, 25,  7, 21,  7, 29,\n",
              "        0,  5, 15, 25,  8, 23, 20, 19,  4, 10,  8, 10,  7, 25, 18,  8, 19,\n",
              "        0, 26, 17,  6,  1, 20,  4, 22, 17,  1,  4, 17, 21, 21, 12, 12, 20,\n",
              "       11,  8, 21, 23, 12, 26,  0,  4, 26,  8, 26,  4, 29, 18, 19, 15,  7,\n",
              "       28, 26, 12, 25, 10,  9, 13, 14, 27, 20, 15,  6, 12, 23, 27, 26, 26,\n",
              "        2, 16, 22,  1,  6,  6, 13, 16,  6, 26,  4,  6, 20, 12, 10,  8, 10,\n",
              "       12, 25, 20,  8, 12,  4,  1,  8,  6, 25, 20, 27, 10, 21,  4,  0, 15,\n",
              "        3, 10, 21, 10, 22,  0,  6, 22, 14, 12, 13, 13,  4,  9,  8, 10, 25,\n",
              "       15, 13,  3,  9, 26, 23, 15, 26, 15, 27,  4, 22, 24, 13,  1, 12, 21,\n",
              "       22,  0, 11,  0, 18, 21, 15, 10,  8, 19, 28, 12, 18,  0,  5, 23, 18,\n",
              "       14,  1,  4, 23,  4, 23, 27, 15,  6, 18, 21,  9,  6, 15, 26, 17,  9,\n",
              "       13, 25, 14, 19,  0, 15,  9, 25, 10,  7,  2, 26, 17,  3, 19, 27, 18,\n",
              "       25,  0,  0,  8, 21,  4, 20, 27, 16, 20,  0, 29, 10,  1, 20, 20,  1,\n",
              "        1,  5, 18, 25,  4, 25, 22, 23, 13, 20,  5, 19, 20, 11, 29, 17, 23,\n",
              "        1,  6, 20, 25, 18, 24, 29, 25,  1, 16,  9,  6,  5, 14, 21, 29, 19,\n",
              "        2, 24, 15,  0, 16, 19, 10, 26,  7,  7,  7, 12, 20, 29, 26,  5, 27,\n",
              "        6,  8, 26, 27,  8,  1, 21,  3,  0,  5, 26, 27, 13,  4, 21,  9, 11,\n",
              "       20,  4, 20,  1, 18,  0, 22,  6,  4,  4, 23,  2, 25,  8, 15, 21, 27,\n",
              "       24,  7,  1, 27, 12,  0, 17, 15, 14, 25,  3, 27,  6,  1,  7,  4, 21,\n",
              "       15, 23, 25, 20, 27, 28, 11,  8, 12,  2, 25, 26, 14, 15, 29, 13, 17,\n",
              "       29, 24, 26, 10, 15,  5, 25,  6,  6, 12,  9,  7, 29, 27, 25, 14, 15,\n",
              "        8,  0,  7,  4, 26, 16, 20,  9, 25, 25, 18, 26, 18,  4, 24,  0,  2,\n",
              "       21, 11, 27, 12,  6,  4, 21, 17, 10, 20, 17,  0,  2, 15, 25, 13,  2,\n",
              "       13,  3, 24, 16,  4, 25, 15, 17, 14, 20,  1, 25,  2, 23,  4, 24, 16,\n",
              "       16, 16,  4,  6, 13, 16, 10, 13,  3, 21, 11,  0, 14, 25,  3, 19, 10,\n",
              "       12,  4,  1, 20, 27, 24, 20, 16, 14,  0,  6,  9, 19, 11, 23, 23, 15,\n",
              "        6, 24, 14,  0,  7, 14,  6, 20,  0, 27,  3, 14,  8, 26, 10, 16, 20,\n",
              "        0, 29,  3, 22,  4,  2,  5, 25,  5,  4, 10, 27, 20, 18, 12, 26,  0,\n",
              "       27, 17,  3, 23,  2, 28, 24, 26,  8, 15,  0, 19, 24, 27,  9, 17, 21,\n",
              "       21, 25, 27,  1,  8, 29, 13, 26, 12, 20,  0, 28,  2, 26, 28, 27,  6,\n",
              "       11, 25, 20, 25,  7, 17,  2,  8,  1, 12, 26,  5, 25, 11, 27,  1, 11,\n",
              "        8, 13, 27,  2, 23,  2,  2,  2,  2,  2, 12, 24,  5, 11,  1, 29, 17,\n",
              "        5, 14, 29,  6, 13,  8,  4,  8,  4,  6, 17, 11,  6, 12,  6, 28, 26,\n",
              "       26, 14,  6,  8,  5, 26, 26,  1, 23, 18, 29, 24,  0, 20, 14, 26,  6,\n",
              "       27, 23,  0, 15, 12,  4, 15, 17, 17, 15, 12, 20, 23, 20, 17, 21, 22,\n",
              "       15, 21, 22, 24,  6, 14,  5, 18,  8,  4, 21,  3,  4,  2,  8, 21, 26,\n",
              "       12, 27, 20, 17, 14, 27,  3, 15, 13,  0,  7,  0,  6, 10, 23,  9, 28,\n",
              "       25, 27,  2, 13,  3, 25, 21, 27,  8, 20, 23, 21, 26,  8, 12,  6, 28,\n",
              "        4, 17, 23, 27, 18, 20, 11, 21, 15, 15, 17,  1, 10, 19,  6,  6,  6,\n",
              "       13, 22,  8, 20, 14,  9, 16,  3,  3,  8,  8, 24,  0,  0,  8, 16,  1,\n",
              "        1,  6, 13,  7,  4,  6,  1,  6, 25, 10, 12,  6, 25, 10],\n",
              "      dtype=int32)"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ],
      "source": [
        "km.labels_"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DHDegXEp5ws9",
        "outputId": "ecff3823-c436-4a8f-8065-33a3b74d88f7"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cluster: 0 (63 docs)\n",
            "network, content, maintaining, users, lease, process, portal, server, probability, devices\n",
            "\n",
            "Cluster: 1 (35 docs)\n",
            "group, master, accounts, account, dependent, strategy, level, scope, parameter, linking\n",
            "\n",
            "Cluster: 2 (25 docs)\n",
            "firm, fee, organization, patent, charge, pocket, trademark, client, separate, bills\n",
            "\n",
            "Cluster: 3 (22 docs)\n",
            "note, stock, forward, holder, unit, contract, remarketing, structuring, trustee, obligation\n",
            "\n",
            "Cluster: 4 (46 docs)\n",
            "capital, business, tax, cost, company, intellectual, equity, profit, enterprise, evaluation\n",
            "\n",
            "Cluster: 5 (21 docs)\n",
            "goods, term, products, working, unit, attributes, services, retrieving, upgraded, reached\n",
            "\n",
            "Cluster: 6 (59 docs)\n",
            "card, account, credit, transaction, customer, rebate, financial, second, device, revolve\n",
            "\n",
            "Cluster: 7 (26 docs)\n",
            "member, valid, telephone, non, members, validity, number, companies, details, identifier\n",
            "\n",
            "Cluster: 8 (50 docs)\n",
            "instrument, company, agreement, party, order, financial, vehicle, parameters, date, underwriter\n",
            "\n",
            "Cluster: 9 (27 docs)\n",
            "asset, assets, valuation, ip, values, license, technology, classification, patent, title\n",
            "\n",
            "Cluster: 10 (35 docs)\n",
            "management, remote, deal, business, machine, investment, client, databases, entitlement, application\n",
            "\n",
            "Cluster: 11 (19 docs)\n",
            "pool, investments, annuity, assets, beneficiary, investment, annuitant, income, term, liabilities\n",
            "\n",
            "Cluster: 12 (40 docs)\n",
            "plan, retirement, benefit, provider, participant, employer, employee, service, health, corporate\n",
            "\n",
            "Cluster: 13 (23 docs)\n",
            "marketing, datasets, privacy, ah, shari, collection, correlations, analysis, television, box\n",
            "\n",
            "Cluster: 14 (32 docs)\n",
            "life, plan, contracts, invested, insured, assets, insurance, plans, policy, interest\n",
            "\n",
            "Cluster: 15 (45 docs)\n",
            "collection, compliance, receipt, code, terminal, status, specific, reports, display, source\n",
            "\n",
            "Cluster: 16 (16 docs)\n",
            "receivable, accounts, server, module, electrically, business, database, computers, vouchers, journal\n",
            "\n",
            "Cluster: 17 (34 docs)\n",
            "module, storage, reconciler, terminal, server, integration, mobile, synchronization, layer, bank\n",
            "\n",
            "Cluster: 18 (29 docs)\n",
            "page, web, rule, pages, statement, trend, intelligence, file, customer, appraisal\n",
            "\n",
            "Cluster: 19 (20 docs)\n",
            "bonds, bond, interests, private, loss, proposed, without, dda, limit, subsidiary\n",
            "\n",
            "Cluster: 20 (48 docs)\n",
            "product, products, supply, financial, flows, chain, institution, end, customers, front\n",
            "\n",
            "Cluster: 21 (41 docs)\n",
            "currency, target, objects, numeric, rate, conversion, currencies, dynamically, exchange, fluctuations\n",
            "\n",
            "Cluster: 22 (18 docs)\n",
            "annuity, contract, death, additional, individual, vehicle, quality, delinquent, selected, annuities\n",
            "\n",
            "Cluster: 23 (28 docs)\n",
            "instruments, exchangeable, transaction, servicing, convertible, quasi, themselves, financial, networks, parties\n",
            "\n",
            "Cluster: 24 (24 docs)\n",
            "check, image, electronic, responsive, batch, checks, adjustment, institution, customer, digital\n",
            "\n",
            "Cluster: 25 (46 docs)\n",
            "merchant, flow, cash, recipient, gift, income, fuel, model, equity, expression\n",
            "\n",
            "Cluster: 26 (43 docs)\n",
            "entity, entities, sub, activities, records, identity, identification, tag, secret, shares\n",
            "\n",
            "Cluster: 27 (47 docs)\n",
            "unit, monetary, risk, real, debt, application, amount, program, business, bank\n",
            "\n",
            "Cluster: 28 (18 docs)\n",
            "fund, mutual, open, end, securities, funds, investors, security, investment, linked\n",
            "\n",
            "Cluster: 29 (20 docs)\n",
            "factors, risk, value, performance, said, elements, business, measuring, enterprise, tangible\n",
            "\n"
          ]
        }
      ],
      "source": [
        "print_clusters(matrix_sample, km.labels_)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 254
        },
        "id": "8RzT2s-e5ws9",
        "outputId": "dc2cfe76-32b3-4f66-a7c4-bc88b68bddbc"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAADtCAYAAACvfY5sAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAdVklEQVR4nO3dX4gd130H8O9Pq24ionQLdRGLJVuuK0HVptkkQinEsBuSBzkP9kNLsYyhBRMRqEtLTI1LiynuU1K7hYLaaCEhbUBx3TwU0ar4oY1sXOpYClm5tYWF7P6x7EVq/mjRxko2Wv/6MHt6zz17zsyZmTN37j33+wGh3Xvnzpw5c87v/Jkzd0VVQUREk29H3wkgIqI0GNCJiDLBgE5ElAkGdCKiTDCgExFlYmdfB77tttt0//79fR2eiGgiffvb3/6uqv6c773eAvr+/ftx/vz5vg5PRDSRROS/Q+9xyoWIKBMM6EREmWBAJyLKBAM6EVEmGNCJiDLBgE5ElInKgC4iXxGRayLyH4H3RUT+QkQui8grIvLR9MkkIqIqMT30rwI4WvL+vQAObP07DuCv2ieLiIjqqnywSFVfEJH9JZvcD+BvtPhi9ZdE5GdEZF5VV+skZHkZOHWqzieA1VXg6tXh19bX6+3DtXt3u8+79uwB5ufT7nOS+K5RHRsbxb8yqa+Zz8JC98eo8uCDwPHjfaeCxlmKJ0VvB/CW9fuVrde2BXQROY6iF4877rhjKIivrABra8CuXUUF3r27OhhevVoE8BQVWhV47732DYJtc7PYX5uAFjIpDUXZNYoJ1pub1cdYWwNmZsLvz84W/5ra2CjKp8stK22PU2ZtDXjxReCxx7rZf0ioIWvbUDc5ZhvT0hiO9NF/VV0GsAwAd955WB97rCioBw8OLuLKyqCSz88DZ8+G97e0VPxftk1sz980KDGNQ9tg2rYyrK0Nfm6yn7IKE0pb6DNlFWV5ucjX3buBL36x2M5txG/eBObmit/L8nVlpbzxXljwp31trQiybvrrVPClpeL47j7sIL++7j+Oj5vOmIZhZmbQQTCdD59Qw5a6sUnZmaqSor6srAzHgSb7rNvQ9NGIpAjobwPYZ/2+d+u1Ut///qD3dfUq8OijxckvLQ0ye2WlCAChTDHbudv4ev4maAD+wGEuVtWFNsE0FHiqAtypU8X+TZpiGgc3TeZc3nijyMMUldgcY319sE/382463IrinrtdgczPbiNu9unm64MPbt/H7t2D6+TL56WlYl92xbPLUijdhrtPc73sz5ptlpeHP+vrwYfKgpvOl18edGI2N4ufTb675WN1tbjuQDGata+P/Tmb3ai517AsP41Qp8i+HlXnXKaq02XXF2D7yM5uVHz1yVcG7HJuN5CmLvnqTllD7G5vytjTT/vjyZ4921+387JpY5AioJ8G8IiIPAPg4wDWYufP7Uw4dWpwAvaJ2q+75ueBS5cGhcEXxPfsGQ7Wly4NPmtXTvNZt/DUCbgxAW5lpdgnMBzE7G3dAh6bJrvA2b14oLqwmCBzzz2DfJqdHe5Zm2Bs5+vq6iC/TYWx920f15zT3Nxg9GWOu7g42M6ueKZnbCqAfaxTp7afy8LC8IjNF+R9+Xbp0vZ9mutljx7N+bnv2ekD/I2GaaTMNnv3AleuFPl882bRyM3P+xs5kyZzPva5mvLy8svD008m0Jn9Av6ete+6hTpFpuz5OkChc/bV3zqdLt/IyEzL2u+vrAyP6n0dKLNve5/26G9hYXBd3ZF/qCG2G2E73abx9XWQykY4vusRTVVL/wH4Oor58J+gmB9/GMDnAHxu630BcALAGwD+HcDhqn2qKnbv/pjOzakuLqoePKg6N6d68mTx+9zc4DWzzcmTus3ioipQ/G8+Z362f7e3N6/Z79m/28cFyo/v26/5vPms+dzJk4P9mm1i0ulL065d29Nl9g8U25t9zMwMti87J3Ms8/OuXYN9xeTdwYPb921fC9+52NsYbj75rlFZvpnP2NvYnwuVI/ccfGlz86jsmrn/3Ott/ysrj773fHlofp+ZGc4/txyW5bdbHtwy6O4rtF1M/fHll1t33PLtu6a++mWXv1C9tD9rlw17/740h655KK991zeU5/bx7Trrll0A5zUQVyuXLarqMVWdV9WfUtW9qvplVf2Sqn5p631V1d9W1btV9UOqWvs7cefnixbU7ZWaYdXzzxe9Q3eY6zKtqukJAUVrt7Q0+OzCwuD91dXtn7V7BYuLRYtrjm/vJ3Rs83nT43RHD+bczPZnz/p7j/b7bppmZ4vW3e4N+Xq/CwtFj/vIkeHPm15I2TDX9CTskYCdd2Y6zLw+Pz/owZn8WvWM09zRl8vNp1CehPLNHVovLAz34Nxjmnl+07Myvbi6K6586TP/7DxbXR1OU1meuuXYvOfLQ7PPe+4Brl8v/rnnY87V3pdvWsx3Pr68MfsDirTG1J+HHhrck7A/59Yd87mnnx4c03zGnKuv/Luv++plqPy7o66yMuCee6js+K6vL8/NCPj554tzN3XcqKqvRm/fhx7DBJXFxcEJxQxDQkNlMzxcXS0akIWF4QtjVyYz3DLDLFPAYoZD5iK6lcdNj4+bHnMcN02h4/r28773Ff+789J22tw8CKVpdbVoaG/eLAqgb2hsAs7Vq/6poaqbS24+2cdfWto+deHeQ7Hzyq1ApoEHiv2cOjUoC2Vp85UT33smfaF7CXaemNUzdsPnVnZTjldWgKeeAq5dKz5nGul33ileC03D2VNB7nHsslV2rqHtTp0aLgsvvDAc5OwpInMOdlD21Wlz7ZaWimkwt0PhTqfZ7Hq3uuq/TxZTh8zxy/iCsj0FaZeR0La+/ZkOmUl7VXpdYx3QgfgMtm+Qms+ZCgsMemHu3HJZD8VOAzCYizNLx8qCesx+ge2Vx/e5UOByzxkALlzYfowf/3h4BGQHCV8vpexcrl7dPhdo/2/PT5Y1XHW4vS/f+yZfTJA0eWUHRfdein0/IzYN7s/272WNvi9PZmcH5dEOvG4jacq/b6752rXhBsll9+ZNx8RecAAMz4P7ymBZmbPLwj/8g/9z7jn4OjxNbqT6Gli3gQ591sx/V436y/iO4d7XK9s2Zn91TcR3udhDxdC0h5m2sYf6bk89tHrFrexuC2tvt7i4fXrIZj7vDq1tdtAJtd6+YZ97TN9UlRluu/txz+Ps2aJyV6U19L79uxs0TB5UrTGvwx26htJsgqQ77LWnQOzhsVsmTNrdHq3ZT6jS2dNOQP0pGzcPUwpdK3shgPta6Fx9Zc6dQooJTLEdHqD+VFHMsc3NzKee2n6tfcdLzS5nVceok57eeug3b25fFmSGj+7yK18PzW7RTa9mbm778je3d1DFbWHNZ0JDWrenYD5vp8HlBh1fBTA9IfeOfIivZ16lKq2+/Ayxe6D2qgK792d6w/a5le3XNHxmO9/UhW+UYpRNhYRGEW5grfu8gdsDNscz6fQ1cmau9J57/GnyKdsfMDh33/uhRr6OCxeAD3+43meqjum7ljFTRVXTTj67dxdl1B7h2FOyvuPVYdIE+NMVupfkqxt1Gq/eAvrm5vaAbjLXHlIaoXlpO+ikFpr7DG3T5EZaiB1szfEBf+AyQ3GXr9KXBcAy9r7K1ra7a4R9PU8T8ENBxLf80p2nN0HPbYDNsVdXqzsCIW2mjOx02091ukvnbO7TsG5DBhQB1L5JZu/v5s3iNfvamqkH3/FiuNMwLjMKTMm9lnbdDs1PA9XTTnWPD4Qb5lgmmAPDscPutJr7IKE0uPcPYkzElIvRVfAEhqdKbDFDSHfKBkg73VC2MqSsQTOV3g7AviFzbBrcffnMzm4PULHD8LrH8rGfMjbH9q0iid1Xk2F3zNRcGTOCs6/79evVX4PQ9Noadh2oWo0EhEcAZdOWbbZten5NplBizj8UMwxfuTeNj12233mnfD91jOVNUdMTMYvu7WGIaTl9d7HbcHvEddgVwBTw1N/pkeKGSVOpvt/Gva6j0rQj4JuPB4aH+O6yR9fKSrHKKOV3BPnETIGY0ZNbd+wRgd05qCpzoRGA78agPQXkTsO1aYTKhKZQYlWdf9VI05cWXwOYaoQBjGlANz2R9fVBQbWDbaqTBwZzz25lqDNHaLfmTXrmbgHv60u37IbTFvMFWTH7bbsfe1918zk0X2v2U+d6mwoIFOukQ+XRjJ7m5tKce5mYKRDzRKo7fWivtknJbvjMFNCuXcXTk2Z1TJsbwGVlwW2g+uwQAf4psjpCddM1lgHd2NysN1dnLmydyh7af905QlNgXnxx+HW7wQgVwJgbcbE3EttoM0opYz/G3sT6erHMbHZ2sO65am445qaUXcHqXu8u7t245aPplE9X6n6hla/naRoV890ybZa2hoJkk2vjW8Jadh/BvaeRmnvt7dFAWR2dqDn0KnZFaLLqw7WyAly86J8rvHChfCWBYS8jLJsbrppnNp9VTTff5qoz/59iaaKZO6wKWpubRYWtM6/u3pRKZXW1qMh1zju24rvlIzTl05bvgaY66UudFqB+WnwBNeacTJmzr59pANxyUjaPbt/TCB23TR0x194sq4zdR1YB3ajbsw9ZWxs8lAMMX7jr1wfDyJTz5VWVzfR62vbSfQ2eKcAx59PmxqWpiHbQbRK0TIUB/OfjayR9FbqO+fnBlzG5LlwAbtzYfv3qTrdUjTTtBsXXWNh5sb6+vSxtbNQrQ1V51iZwuTexfXzn6N4kNkH5jTfKG1DfTckyVR2tUGMAVNeRsnyz60id9GYZ0GPVGe75Vh6kZAenLo9jlDV46+vdDiftimivTa/L7jHGNuBlFWRlZTgA1u09Xr9efA1r2+tXdmPdfC+6/XW7vnQAg7n7umlxr39VULGvQxejxzoN4uZm+/sV6+vF+XR9I7ss4Dc9h7GYQ+9iqZ/NNzXS5fp1oH5haHvTxOVr+e155TJd38Czla0UqFseVlaKKSlgEJibjGTMfQxzfDPNVfUHOEahybUxoz43P915Y1NmzTHq3oDu6uZqCnXOxZy/ndd23RmXexo+Y9FDjx3q20GyztC5i6mRKmUVr27Pb2OjfgPhO197dYYttO8bN/zbdtVzcRveugF9bQ0QGfRi7Z5pzLTAwoJ/SWXVNFeqzoEvv4Fm94PcfHTLg2+Kwi2zbabVuuQ2njFl0pxL2fWvmqoxUtzTqHsfxqjqOIxFDz2WXeDsp0pDPdrQUp9R/emskJh5Q5tZGdBWqMBubvqDne/PnHXZe7efbqw635i/R2pLPQKqYv4kXQzTAIX+rFyT+0Gx5zrK0VgKvsYz9hyqGl7ffuyeve/JzpBQXVtfH6SjbkCP6Tj02kPveo7KPFnmziGOSyE2o4x33+0mL+x9+tZC2619WQEr+wPMVcfv6hr7nkgdpapj+77aYlyEburmqmlvGIgfpbiLGUJz+V2X2V576G1Orm2g6LoxcZnlbjZ7lOG2vCnSV5W/KaYJytLZR8CNPeaor/8ojru6OujtV2kS4HxpTzXarfNVxnWNonE1z0c0kbJMjMUcehNtg0Xo811VOLPcLZb9tGwdvgduul6xsrnZzVfm+tQ5l9DNQKC/3r19XPumawpdrowKPe3aNh/NuVetDDLl69132x2vLXs1mq1pPlQ9RVy3Pk1EQLczsGxuqk3FsIdKTacYXClWPqQIPKMIXlVD0zrf+peq1++7GRhzLNMQ2Ctmmqj6nKmsIvX3naoRSKHqPO2pCPeGd+y5m/Jl7jOkqqN1mWncLviWfNYdWfR+U9Q3FeGyM9Ct0PYdevci11kPG9u7qVO5Qxe+ywDbpodcJ12jWqfe9fF9xzLDZ1Oeml6v0OfMjbY2QblJI9CVsvxxpxO7+vK6STczk6ah6D2gt/3SprLPphiCuoHDd7w+5mNffhm4dcv/Xdqj0OcNSTdIxM4bjwvT22zbYLjajlJTHD9X9jdSdqltHvYe0EdRCNocI6ayxWyT8um5suVLMWmZtADY1KSf4/o6sGNH/PJRd5Sa8muPY3rUk57fZcxDU1UdprYdqrZ52Pscun0CXQT3Lr+69Ny56m1MBUt1w6rrx/L7UueR8TbnPw55V5UG+0nFplMrqRptU39SjfyapqnJ8UMPatXhXquqdPQ9ldR7D902aS18H3fcxzWP2lSeuk9atllR0GaeMtWQe1yvoWscGj/3u2vqCD2oBcSX16pr1eYLybqYHhurgN7WpFQUo68K00U+lVWevp/MjWVu0PsqWhff/TPu5TWUvia9/3FoHGxl5TVGm4am6eqmmGP1PuXi6nI50igKVZO15rkrW/M/qooe+30fdac5ur4JPcllxM7vrp4bqZOWsmuVKu7ElIcmPfO1tbgHl7LqoVeZhMoxbj2ZLqW6HqluXMesaHIrbN9zpnWMumxV5fmFC6Opk1W96fX1dOkoC7pV39mTwlQF9HGVsicTkuIGUYxJ/o6QJo/MT8p0EjC6Dk1sw+F+6ZjJ24sXiz8sM45LcJvk4SjLyNhNuaQ0CcElZhVOip5Vl70CW5s5xUm4Xq5JGPWlUGeaoO1KFvNXwiZh9JNqdJhK1gG9ToHoajia4k527kFjkp4ezP1ahNgdgq7yYFrzFkh37pxy2WJnaJPMDf1F9KobbNM0Zx4yzRW5K+OSp2XfvTQOmqZjXPLXlXUPfZRCfxE95nuziUZtVOUudJxxKffjko6Qug1OVEAXkaMi8rqIXBaRxz3v3yEi3xSR74jIKyLymXrJ6E9f39o2bSZpfnzcK/k0a3JtzLRn399zUzftTdb7VwZ0EZkBcALAvQAOATgmIoeczf4IwLOq+hEADwD4y3rJ6E9shrVdJTIuQ8yuhfIzJqDHLBuk6XHjRpoykGJBQNv6O6qyHNNDPwLgsqq+qaobAJ4BcL+zjQL46a2f5wC8ky6J46FtoWBwqsY8ai+nPCyrc016223qcBf5urmZvqMXE9BvB/CW9fuVrddsfwzgIRG5AuAMgN/x7UhEjovIeRE5HzpYTgUyF7wm0zPCmhSjWobbtdR1K9VN0WMAvqqqewF8BsDXRGTbvlV1WVUPq+rhRMelCTFOf2GnCTZqNAliAvrbAPZZv+/des32MIBnAUBV/w3A+wHcliKBlIc+/8IOe9c0LWIC+jkAB0TkLhGZRXHT87Szzf8A+BQAiMgvogjo/5syoU1N0uoK6gZ71zQtKgO6qt4C8AiA5wBcRLGa5VUReVJE7tva7FEAnxWRCwC+DuC3VMdjkM2ATtOODdr0iHpSVFXPoLjZab/2hPXzawA+kTZpRDSNOEXWHJ8UJaKxwhFFcwzoRESZYEAnIsoEAzoRUSYY0ImIMsGATiPBG11E3WNAJyLKBAM6EdEYajKqZUAnIsoEAzoRUSYY0ImIMsGATkSUCQZ0IqJMMKATEWWCAZ2IKBMM6Ims81HIHo3F31Ih6h0DeiKbd9/oOwlENOUY0ImIMsGATkRTaceO/KZJGdCJiDLBgE7Em6qUCQZ0opY29L2+k0AEgAGdxtQO3MIMbvWdjCizB37YdxJojL1/x+jKMQM6EY05TonFYkAnoimVX0PBgF7pet8JIJoaHxjhNNuu2ZsjO9aoMKATEWWCAZ2IKBMM6EREmWBAJyLKxBgG9PzuPBMRjcIYBnQiImoiKqCLyFEReV1ELovI44FtfkNEXhORV0XkVNpkEo2vPZf46D+Nh8qALiIzAE4AuBfAIQDHROSQs80BAH8A4BOq+ksAfq+DtBKNpZM3vtN3EogAxPXQjwC4rKpvquoGgGcA3O9s81kAJ1T1BwCgqtfSJpOIiKrEBPTbAbxl/X5l6zXbQQAHReRfReQlETmaKoFpne07AUREndmZcD8HACwB2AvgBRH5kKoOPTcvIscBHC9++1iiQxMRERDXQ38bwD7r971br9muADitqj9R1f8EcAlFgB+iqsuqelhVDzdNMBER+cUE9HMADojIXSIyC+ABAKedbf4eRe8cInIbiimYNxOms5HNzXUAXIFARNOhMqCr6i0AjwB4DsBFAM+q6qsi8qSI3Le12XMAvicirwH4JoDfV9XvdZXoWHfffR67dr3bdzKIiEYiag5dVc8AOOO89oT1swL4/NY/IiLqAZ8UJSLKBAM6EVEmGNCJiDLBgE5ElAkGdCKiTDCgExFlggGdiCgTDOhERJlgQCciygQDOhFRJhjQs8I/sE00zRjQiYgywYBORJQJBnQiokwwoBMRZYIBnYgoEwzoRESZYEAnIsoEAzoRUSYY0ImIMsGATkSUCQZ0IqJMMKATEWWCAZ2IKBMM6EREmWBAJyLKBAM6EVEmGNCJiDLBgE5EY+OuHet9J2GiMaCPwI4dt/pOAhFNAQZ0IqJMMKDn5gMcDRClM1l/eJ0BnYgoE1EBXUSOisjrInJZRB4v2e7XRERF5HC6JBIRUYzKgC4iMwBOALgXwCEAx0TkkGe7DwL4XQDfSp1IIiKqFtNDPwLgsqq+qaobAJ4BcL9nuz8B8AUAP0qYPiIiihQT0G8H8Jb1+5Wt1/6fiHwUwD5V/ceyHYnIcRE5LyLna6eUiIhKtb4pKiI7APwZgEertlXVZVU9rKqcYyciSiwmoL8NYJ/1+96t14wPAvhlAGdF5L8A/CqA07wxSkQ0WjEB/RyAAyJyl4jMAngAwGnzpqquqeptqrpfVfcDeAnAfarKaRUiohGqDOiqegvAIwCeA3ARwLOq+qqIPCki93WdQCIiirMzZiNVPQPgjPPaE4Ftl9oni4imkUzWg5ljh0+KEtHYYEBvhwGdiCgTDOhERJlgQCciygQDOhFRJhjQiYgywYBORJQJBnQiokwwoBMRZYIBnYgoEwzoRESZYEAnIsoEAzoRUSYY0ImIMsGATkSUCQZ0IqJMMKATEWWCAZ2IKBMM6EREmWBAJyLKBAM6EVEmGNCJiDLBgE5ElAkGdCKiTDCgExFlggGdiCgTDOhUk/adACIKYEAnIsoEAzoRUSYY0ImIMsGATkSUCQZ0IqJMRAV0ETkqIq+LyGURedzz/udF5DUReUVE/llE7kyfVCIiKlMZ0EVkBsAJAPcCOATgmIgccjb7DoDDqvorAL4B4IupE0rULy7XpPEX00M/AuCyqr6pqhsAngFwv72Bqn5TVd/d+vUlAHvTJpOIiKrEBPTbAbxl/X5l67WQhwH8k+8NETkuIudF5Hx8EomIKMbOlDsTkYcAHAaw6HtfVZcBLBfbHuYYlogooZiA/jaAfdbve7deGyIinwbwhwAWVfXHaZJHRESxYqZczgE4ICJ3icgsgAcAnLY3EJGPADgJ4D5VvZY+mUREVKUyoKvqLQCPAHgOwEUAz6rqqyLypIjct7XZnwLYDeDvRGRFRE4Hdpetn3+j7xQQ0bSLmkNX1TMAzjivPWH9/OnE6Zo4X15fwSex1HcyiGiK8UlRIqJMMKATEWUi+4Cuutl3EoiIRiLrgC6yE3xkm4imRdYBnYhomjCgExFlggGdiCgTDOhERJnIPKC/13cCmpFbfaeAiCZQ1gFdlStciGh6ZB3QiYimCQM6EVEmGNCJiDLBgE5ElInMAzpvihLR9Mg8oBMRTQ8GdCKiTDCgExFlggGdiCgTDOhERJlgQCciygQDOhFRJhjQiYgywYBORJQJBnQiokwwoBMRZYIBnYgoEwzoRESZYEAnIsoEAzoRUSYY0ImIMsGATkSUCQZ0IqJMRAV0ETkqIq+LyGURedzz/vtE5G+33v+WiOxPnVAiIipXGdBFZAbACQD3AjgE4JiIHHI2exjAD1T1FwD8OYAvpE4oERGVi+mhHwFwWVXfVNUNAM8AuN/Z5n4Af7318zcAfEpEJF0yiShnM7jVdxKysDNim9sBvGX9fgXAx0PbqOotEVkD8LMAvmtvJCLHARzf+nUdkNebJDrWhQtd7n3YJ0vee++9mjvTFgn5YYvPjpG6WdansmtPcTa3/n9lhMf84Y9Gc5wfpS/Md4beiAnoyajqMoDlUR6TiGhaxEy5vA1gn/X73q3XvNuIyE4AcwC+lyKBREQUJyagnwNwQETuEpFZAA8AOO1scxrAb279/OsA/kVV20wcEBFRTZVTLltz4o8AeA7ADICvqOqrIvIkgPOqehrAlwF8TUQuA/g+iqBPREQjJOxIExHlgU+KEhFlggGdiCgTDOhERJlgQCciygQDOhFRJhjQiYgywYBORJSJ/wNPDjedvvNz1gAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "### Hierarchical clustering (alternative approach)\n",
        "\n",
        "from scipy.cluster.hierarchy import dendrogram, linkage, fcluster\n",
        "\n",
        "Z = linkage(matrix_sample.todense(), metric='cosine', method='complete')\n",
        "_ = dendrogram(Z, no_labels=True) # Plot dentrogram chart"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 254
        },
        "id": "IznMBTYc5ws9",
        "outputId": "376a0f31-377f-47df-c3ea-f2f99b462a53"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAADtCAYAAACvfY5sAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAMbElEQVR4nO3dX6ic+V3H8feniasX1i6YI0gSmwVTMIhky2Er9MKAFbK9SC4U2ZSCXZbmphGlRVhR1rLeWAv+w/gnalktbOPaCwkYyYV2FcQtOYuxmCwrh/gniWv3dLssSNF14etFJjJ7cs6ZJydz5sx8835BYJ7n+Z15vkye+cx3fs8zM6kqJEmL7z27XYAkaToMdElqwkCXpCYMdElqwkCXpCb27taO9+3bV4cOHdqt3UvSQnr55Ze/UVVLG23btUA/dOgQKysru7V7SVpISf5ts21OuUhSEwa6JDVhoEtSExMDPckXkrye5J822Z4kv5VkNcnXknxw+mVKkiYZ0qE/BxzfYvvjwOHRv9PA795/WZKkezUx0Kvqb4FvbjHkJPAnddtLwMNJvndaBUqShpnGHPp+4MbY8s3RurskOZ1kJcnK2traFHYtSbpjpidFq+pcVS1X1fLS0obXxUuStmkaHyy6BRwcWz4wWqcZO3cOnn9+t6uQ4GMfg9Ond7uKB880Av0CcCbJeeBDwFtV9doU7lf36Pnn4coVOHp0tyvRTnrtNfj613e7is299dbt43Cem4uuLzgTAz3Jl4BjwL4kN4FfAr4NoKp+D7gIfBRYBb4FPLlTxc7aonW8V67sdgXb0/XJtVOOHbsd6L5wb8+d50nHY25ioFfVqQnbC/jU1CqaI4vW8c6izml3hzvRzT0ILxBHj8KLL+52FYvp2LHdrmDn7NqXcy0KnzjvNu/dYefuS5pkYQN9FtMhd8Jhp1/RF62jnOcXuXnsvqZ9rO7EcTnPx6CP33AL+10ud6ZDdtLRozvfic77ySPdv2kfq9M+Luf9GPTxG25hO3SY705xqHnsKOfR0C7tXrqvWXZV83ysLsIx6OM3zEIH+rzZzlvD7b79m5e3eLMy9AT10M7LuXZtZNGfwwb6FG3nqpjtvPV7UMNoml3aPHVVmh+L/hw20KdsFm8NDSNp5yzyc3hhT4pKkt7NDl1tbTUfutW856Ken9ju5X3zNAes+zN3gb7oVzMsqo6P+1bzoZvNey7y+YntfrJ5nuaAdX/mLtC9mmF3dH3c73U+dNHPT8zq8r5Ff5y6mrtAB69muGOzrnlSl7zdztjHXVpscxnoum2zrnmrLnleOmMtvu2eg4Ddn257UBnoc+5BmzLQ/NjOOQjYXlPhi8d0GOiSNrWdabjtNBWzfPHozECXNBdm9eIxS7M+D+YHiyRph2z2TZFbfePj/Xx7ox26JO2gWZ4Hs0OXpCYMdElqwkCXpCYMdElqwkCXpCa8yuUePWhfySp10/k5bId+j7b6BfLNri2dp18Flx50nZ/Ddujb4PeraL3OXV9HXZ/DdujSFHTu+rQ47NClKena9Wlx2KFLUhN26NKCmfU3+Glx2KFLC2bW3+CnxWGHLi0g5+u1ETt0SWpiUIee5Djwm8Ae4A+r6lfWbf8+4I+Bh0djnq6qi1OudVP+HqEkDejQk+wBzgKPA0eAU0mOrBv2i8ALVfUo8ATwO9MudCvbuQYYnFeU1MuQDv0xYLWqrgMkOQ+cBK6NjSngu0a33wf8xzSLHKLj7xFK0r0YMoe+H7gxtnxztG7cZ4GPJ7kJXAR+eqM7SnI6yUqSlbW1tW2UK0nazLROip4CnquqA8BHgS8mueu+q+pcVS1X1fLS0tKUdi1JgmGBfgs4OLZ8YLRu3FPACwBV9ffAdwD7plGgJGmYIYF+GTic5JEkD3H7pOeFdWP+HfhRgCQ/wO1Ad05FkmZoYqBX1TvAGeAS8Aq3r2a5muTZJCdGwz4DfDLJPwJfAj5RVbVTRUuS7jboOvTRNeUX1617Zuz2NeDD0y1NknQv/KSoJDVhoEtSEwa6JDVhoEtSEwa6JDVhoEtSEwa6JDVhoEtSEwa6JDVhoEtSEwa6JDVhoEtSEwa6JDVhoEtSEwa6JDVhoEtSEwa6JDVhoEtSEwa6JDVhoEtSEwa6JDVhoEtSEwa6JDVhoEtSEwa6JDVhoEtSEwa6JDVhoEtSEwa6JDVhoEtSEwa6JDVhoEtSEwa6JDUxKNCTHE/yapLVJE9vMuYnk1xLcjXJ89MtU5I0yd5JA5LsAc4CPwbcBC4nuVBV18bGHAZ+HvhwVb2Z5Ht2qmBJ0saGdOiPAatVdb2q3gbOAyfXjfkkcLaq3gSoqtenW6YkaZIhgb4fuDG2fHO0btwHgA8k+bskLyU5vtEdJTmdZCXJytra2vYqliRtaFonRfcCh4FjwCngD5I8vH5QVZ2rquWqWl5aWprSriVJMCzQbwEHx5YPjNaNuwlcqKr/rap/Af6Z2wEvSZqRIYF+GTic5JEkDwFPABfWjflzbnfnJNnH7SmY61OsU5I0wcRAr6p3gDPAJeAV4IWquprk2SQnRsMuAW8kuQZ8Bfi5qnpjp4qWJN1t4mWLAFV1Ebi4bt0zY7cL+PTonyRpF/hJUUlqwkCXpCYMdElqwkCXpCYMdElqwkCXpCYMdElqwkCXpCYMdElqwkCXpCYMdElqwkCXpCYMdElqwkCXpCYMdElqwkCXpCYMdElqwkCXpCYMdElqwkCXpCYMdElqwkCXpCYMdElqwkCXpCYMdElqwkCXpCYMdElqwkCXpCYMdElqwkCXpCYMdElqwkCXpCYMdElqYlCgJzme5NUkq0me3mLcjyepJMvTK1GSNMTEQE+yBzgLPA4cAU4lObLBuPcCPwN8ddpFSpImG9KhPwasVtX1qnobOA+c3GDcLwOfA/57ivVJkgYaEuj7gRtjyzdH6/5fkg8CB6vqL7a6oySnk6wkWVlbW7vnYiVJm7vvk6JJ3gP8GvCZSWOr6lxVLVfV8tLS0v3uWpI0Zkig3wIOji0fGK27473ADwIvJvlX4IeBC54YlaTZGhLol4HDSR5J8hDwBHDhzsaqequq9lXVoao6BLwEnKiqlR2pWJK0oYmBXlXvAGeAS8ArwAtVdTXJs0lO7HSBkqRh9g4ZVFUXgYvr1j2zydhj91+WJOle+UlRSWrCQJekJgx0SWrCQJekJgx0SWrCQJekJgx0SWrCQJekJgx0SWrCQJekJgx0SWrCQJekJgx0SWrCQJekJgx0SWrCQJekJgx0SWrCQJekJgx0SWrCQJekJgx0SWrCQJekJgx0SWrCQJekJgx0SWrCQJekJgx0SWrCQJekJgx0SWrCQJekJgx0SWrCQJekJgx0SWpiUKAnOZ7k1SSrSZ7eYPunk1xL8rUkf5Xk/dMvVZK0lYmBnmQPcBZ4HDgCnEpyZN2wfwCWq+qHgC8DvzrtQiVJWxvSoT8GrFbV9ap6GzgPnBwfUFVfqapvjRZfAg5Mt0xJ0iRDAn0/cGNs+eZo3WaeAv5yow1JTidZSbKytrY2vEpJ0kRTPSma5OPAMvD5jbZX1bmqWq6q5aWlpWnuWpIeeHsHjLkFHBxbPjBa9y5JPgL8AvAjVfU/0ylPkjTUkA79MnA4ySNJHgKeAC6MD0jyKPD7wImqen36ZUqSJpkY6FX1DnAGuAS8ArxQVVeTPJvkxGjY54HvBP4syZUkFza5O0nSDhky5UJVXQQurlv3zNjtj0y5LknSPfKTopLUhIEuSU0Y6JLUhIEuSU0Y6JLUhIEuSU0Y6JLUhIEuSU0Y6JLUhIEuSU0Y6JLUhIEuSU0Y6JLUhIEuSU0Y6JLUhIEuSU0Y6JLUhIEuSU0Y6JLUhIEuSU0Y6JLUhIEuSU0Y6JLUhIEuSU0Y6JLUhIEuSU0Y6JLUhIEuSU0Y6JLUhIEuSU0Y6JLUhIEuSU0Y6JLUxKBAT3I8yatJVpM8vcH2b0/yp6PtX01yaNqFSpK2NjHQk+wBzgKPA0eAU0mOrBv2FPBmVX0/8OvA56ZdqCRpa3sHjHkMWK2q6wBJzgMngWtjY04Cnx3d/jLw20lSVTXFWjWHjj13bMP1V/7zN0bbf/aubS9+4sUdrEh6cA0J9P3AjbHlm8CHNhtTVe8keQv4buAb44OSnAZOjxb/K8mrm+00GVDZFP5mlvua9/ru5+/u9igAf/PkBvt4cvs78f9q9vua9/pmua85qe/9m20YEuhTU1XngHOz3KckPSiGnBS9BRwcWz4wWrfhmCR7gfcBb0yjQEnSMEMC/TJwOMkjSR4CngAurBtzAfip0e2fAP7a+XNJmq2JUy6jOfEzwCVgD/CFqrqa5FlgpaouAH8EfDHJKvBNboe+JGmGYiMtST34SVFJasJAl6QmDHRJasJAl6QmDHRJasJAl6QmDHRJauL/AF/XaMFqz33tAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "# View dendrogram for subset\n",
        "Z_ = linkage(matrix_sample.todense()[:25], metric='cosine', method='complete')\n",
        "_ = dendrogram(Z_, no_labels=True) # Plot dentrogram chart"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "scrolled": false,
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "44BhzWnA5ws-",
        "outputId": "7428468e-6afc-4676-8fb1-e4b8cd06c429"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cluster: 1 (17 docs)\n",
            "records, organization, presentation, instrument, entity, future, employee, financial, maintaining, predictions\n",
            "\n",
            "Cluster: 2 (6 docs)\n",
            "health, savings, care, expenses, premium, techniques, modeling, icon, modeled, methodologies\n",
            "\n",
            "Cluster: 3 (27 docs)\n",
            "business, entity, object, case, inter, measuring, value, strategic, performance, model\n",
            "\n",
            "Cluster: 4 (10 docs)\n",
            "content, debt, collection, node, debtors, nodes, debtor, strategies, behavioral, deals\n",
            "\n",
            "Cluster: 5 (120 docs)\n",
            "capital, instrument, firm, money, investment, intellectual, added, issuer, securities, unit\n",
            "\n",
            "Cluster: 6 (30 docs)\n",
            "account, funds, assessment, bank, eps, virtual, day, intelligence, customer, delinquent\n",
            "\n",
            "Cluster: 7 (153 docs)\n",
            "asset, estate, parameter, agreement, real, investment, aggregated, charitable, risk, budget\n",
            "\n",
            "Cluster: 8 (3 docs)\n",
            "structuring, index, obligation, credit, convertible, bearing, method, complex, liquidity, operating\n",
            "\n",
            "Cluster: 9 (10 docs)\n",
            "rebate, card, revolve, expenditures, category, spending, billing, member, status, said\n",
            "\n",
            "Cluster: 10 (35 docs)\n",
            "plan, retirement, stable, participant, investments, housing, investment, benefit, reward, value\n",
            "\n",
            "Cluster: 11 (33 docs)\n",
            "server, terminal, processor, projects, customer, equity, kiosk, content, 11, service\n",
            "\n",
            "Cluster: 12 (62 docs)\n",
            "property, lease, objects, patent, customer, right, numeric, leasing, valid, telephone\n",
            "\n",
            "Cluster: 13 (19 docs)\n",
            "unit, project, risk, factors, equity, forward, estimating, calculating, contract, base\n",
            "\n",
            "Cluster: 14 (44 docs)\n",
            "instruments, software, financial, web, exchangeable, servicing, convertible, transaction, product, commercial\n",
            "\n",
            "Cluster: 15 (45 docs)\n",
            "annuity, buyer, seller, accounting, targets, initial, material, financing, contract, consent\n",
            "\n",
            "Cluster: 16 (77 docs)\n",
            "investor, assets, products, retirement, probability, clearing, score, securities, utility, dealer\n",
            "\n",
            "Cluster: 17 (69 docs)\n",
            "check, terminal, vehicle, image, transaction, portal, equipment, jurisdiction, synchronization, business\n",
            "\n",
            "Cluster: 18 (79 docs)\n",
            "module, layer, customer, recipient, datasets, analysis, receivable, accounts, server, donor\n",
            "\n",
            "Cluster: 19 (161 docs)\n",
            "currency, entity, merchant, service, documents, expression, certificate, memory, stock, savings\n",
            "\n"
          ]
        }
      ],
      "source": [
        "## Get flat clusters from cluster hierarchy\n",
        "\n",
        "#clusters = fcluster(Z, 50, criterion='maxclust') # Create fix number of flat clusters\n",
        "clusters = fcluster(Z, 0.99, criterion='distance') # Create flat clusters by distance threshold\n",
        "\n",
        "print_clusters(matrix_sample, clusters)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dchTDsUD5ws-"
      },
      "source": [
        "### Probabilistic topic modeling\n",
        "\n",
        "- Various machine learning methods for *topic modeling* extend the idea of topic clustering\n",
        "- The most popular: [Latent Dirichlet Allocation (LDA) method](https://dl.acm.org/doi/pdf/10.1145/2133806.2133826) (see also [gensim library](https://radimrehurek.com/gensim/models/ldamodel.html))\n",
        "- LDA performs a type of fuzzy clustering into *k* topics:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ivLzg-iz5ws-"
      },
      "source": [
        "- \n",
        "    - A document can be assigned to *serveral topics*, according to a probability distribution\n",
        "        - E.g., 68% Topic 6, 21% Topic 5, 11% Topic 2"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W6dA2rIg5ws-"
      },
      "source": [
        "-     - A topic is described by a *weighting of terms* (probability distribution)\n",
        "        - The terms reflect term frequencies in documents assigned to the topic\n",
        "        - The top-10 are often inspected, after some filtering/reranking"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "id": "FndJK9mA5ws-"
      },
      "outputs": [],
      "source": [
        "## Topic modeling demo\n",
        "#!pip3 install gensim\n",
        "\n",
        "# Fast and simple tokenization\n",
        "new_vectorizer = TfidfVectorizer()\n",
        "word_tokenizer = new_vectorizer.build_tokenizer()\n",
        "tokenized_text = [word_tokenizer(doc) for doc in documents]\n",
        "\n",
        "# Train LDA model\n",
        "from gensim import corpora, models\n",
        "dictionary = corpora.Dictionary(tokenized_text)\n",
        "lda_corpus = [dictionary.doc2bow(text) for text in tokenized_text]\n",
        "lda_model = models.LdaModel(lda_corpus, id2word=dictionary, num_topics=10)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OsV9YU825ws_",
        "outputId": "11cc927e-de83-4ffb-806f-5c44b8db7757"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Topic 0\n",
            "0.1036\tthe\n",
            "0.0533\tof\n",
            "0.0277\tand\n",
            "0.0267\tto\n",
            "0.0223\tfor\n",
            "0.0200\tfirst\n",
            "0.0175\tis\n",
            "0.0141\tsecond\n",
            "0.0129\tloan\n",
            "0.0128\tan\n",
            "\n",
            "Topic 1\n",
            "0.0866\tthe\n",
            "0.0381\tto\n",
            "0.0298\tand\n",
            "0.0227\tof\n",
            "0.0216\taccount\n",
            "0.0205\ttransaction\n",
            "0.0176\tfor\n",
            "0.0174\tis\n",
            "0.0155\tThe\n",
            "0.0149\tpayment\n",
            "\n",
            "Topic 2\n",
            "0.0916\tthe\n",
            "0.0547\tof\n",
            "0.0247\tto\n",
            "0.0193\tfor\n",
            "0.0182\tone\n",
            "0.0159\tand\n",
            "0.0157\tat\n",
            "0.0154\tis\n",
            "0.0148\tan\n",
            "0.0135\tThe\n",
            "\n",
            "Topic 3\n",
            "0.0533\tof\n",
            "0.0482\tand\n",
            "0.0440\tthe\n",
            "0.0222\tto\n",
            "0.0197\tfor\n",
            "0.0128\trisk\n",
            "0.0107\tin\n",
            "0.0102\ton\n",
            "0.0098\tis\n",
            "0.0095\tare\n",
            "\n",
            "Topic 4\n",
            "0.0728\tthe\n",
            "0.0396\tto\n",
            "0.0276\tand\n",
            "0.0243\tauction\n",
            "0.0206\tfor\n",
            "0.0186\tis\n",
            "0.0176\tof\n",
            "0.0172\tan\n",
            "0.0133\tThe\n",
            "0.0128\tin\n",
            "\n",
            "Topic 5\n",
            "0.0638\tthe\n",
            "0.0455\tof\n",
            "0.0424\tand\n",
            "0.0317\tto\n",
            "0.0200\tfor\n",
            "0.0183\tor\n",
            "0.0114\tis\n",
            "0.0112\tin\n",
            "0.0111\tThe\n",
            "0.0095\tan\n",
            "\n",
            "Topic 6\n",
            "0.0544\tthe\n",
            "0.0401\tof\n",
            "0.0394\tand\n",
            "0.0331\tto\n",
            "0.0301\tdata\n",
            "0.0229\tfor\n",
            "0.0173\tsystem\n",
            "0.0149\tThe\n",
            "0.0135\tin\n",
            "0.0107\tan\n",
            "\n",
            "Topic 7\n",
            "0.0418\tthe\n",
            "0.0286\tand\n",
            "0.0257\tto\n",
            "0.0207\ttrade\n",
            "0.0172\tis\n",
            "0.0161\tfor\n",
            "0.0160\tof\n",
            "0.0148\tare\n",
            "0.0135\tthat\n",
            "0.0120\tThe\n",
            "\n",
            "Topic 8\n",
            "0.0588\tthe\n",
            "0.0370\tand\n",
            "0.0345\tto\n",
            "0.0288\tfinancial\n",
            "0.0282\tof\n",
            "0.0184\tfor\n",
            "0.0176\tThe\n",
            "0.0136\twith\n",
            "0.0098\tdata\n",
            "0.0097\tbe\n",
            "\n",
            "Topic 9\n",
            "0.0457\tthe\n",
            "0.0404\tto\n",
            "0.0335\tor\n",
            "0.0265\tand\n",
            "0.0231\tof\n",
            "0.0195\tbe\n",
            "0.0186\tmay\n",
            "0.0183\tin\n",
            "0.0115\torder\n",
            "0.0115\tare\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Inspect topics\n",
        "for i, topic in lda_model.show_topics(num_words=10, formatted=False):\n",
        "    print(\"Topic\", i)\n",
        "    for term, score in topic:\n",
        "        print(\"%.4f\\t%s\" % (score,term))\n",
        "    print()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nfBTw4Yq5ws_",
        "outputId": "27a4a3e7-c311-4ec8-916a-c4b9d726a549"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['fund',\n",
              " 'raising',\n",
              " 'computer',\n",
              " 'network',\n",
              " 'and',\n",
              " 'method',\n",
              " 'for',\n",
              " 'simplifying',\n",
              " 'and',\n",
              " 'inducing',\n",
              " 'the',\n",
              " 'giving',\n",
              " 'of',\n",
              " 'monetary',\n",
              " 'contributions',\n",
              " 'by',\n",
              " 'multiplicity',\n",
              " 'of',\n",
              " 'prospective',\n",
              " 'contributors',\n",
              " 'consisting',\n",
              " 'of',\n",
              " 'computer',\n",
              " 'with',\n",
              " 'communication',\n",
              " 'circuitry',\n",
              " 'for',\n",
              " 'communicating',\n",
              " 'over',\n",
              " 'computer',\n",
              " 'network',\n",
              " 'data',\n",
              " 'sets',\n",
              " 'comprised',\n",
              " 'of',\n",
              " 'requests',\n",
              " 'soliciting',\n",
              " 'the',\n",
              " 'making',\n",
              " 'of',\n",
              " 'monetary',\n",
              " 'contributions',\n",
              " 'to',\n",
              " 'plurality',\n",
              " 'of',\n",
              " 'remote',\n",
              " 'prospective',\n",
              " 'contributors',\n",
              " 'computers',\n",
              " 'and',\n",
              " 'means',\n",
              " 'for',\n",
              " 'receiving',\n",
              " 'data',\n",
              " 'sets',\n",
              " 'comprised',\n",
              " 'of',\n",
              " 'contributions',\n",
              " 'from',\n",
              " 'the',\n",
              " 'remote',\n",
              " 'contributors',\n",
              " 'respective',\n",
              " 'computers',\n",
              " 'communicated',\n",
              " 'over',\n",
              " 'computer',\n",
              " 'network',\n",
              " 'The',\n",
              " 'network',\n",
              " 'and',\n",
              " 'method',\n",
              " 'induces',\n",
              " 'and',\n",
              " 'simplifies',\n",
              " 'the',\n",
              " 'making',\n",
              " 'of',\n",
              " 'contributions',\n",
              " 'Contributions',\n",
              " 'include',\n",
              " 'political',\n",
              " 'contributions',\n",
              " 'charitable',\n",
              " 'contributions',\n",
              " 'and',\n",
              " 'religious',\n",
              " 'contributions',\n",
              " 'Requests',\n",
              " 'for',\n",
              " 'contributions',\n",
              " 'include',\n",
              " 'multi',\n",
              " 'media',\n",
              " 'presentations',\n",
              " 'communicated',\n",
              " 'to',\n",
              " 'prospective',\n",
              " 'contributors',\n",
              " 'computers',\n",
              " 'which',\n",
              " 'are',\n",
              " 'networked',\n",
              " 'to',\n",
              " 'computer',\n",
              " 'networks']"
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ],
      "source": [
        "tokenized_text[0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "scrolled": false,
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NIeezrUX5ws_",
        "outputId": "f0c8d481-46a4-4df5-83a0-cd681a7588ba"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Topic 0\n",
            "0.0200\tfirst\n",
            "0.0141\tsecond\n",
            "0.0129\tloan\n",
            "0.0110\tvalue\n",
            "0.0108\tmethod\n",
            "0.0070\tsystem\n",
            "0.0065\torder\n",
            "0.0065\tone\n",
            "0.0064\tamount\n",
            "0.0060\tleast\n",
            "\n",
            "Topic 1\n",
            "0.0216\taccount\n",
            "0.0205\ttransaction\n",
            "0.0149\tpayment\n",
            "0.0132\tinformation\n",
            "0.0128\tcard\n",
            "0.0123\tsystem\n",
            "0.0119\tcredit\n",
            "0.0085\tuser\n",
            "0.0084\tcustomer\n",
            "0.0073\tdevice\n",
            "\n",
            "Topic 2\n",
            "0.0182\tone\n",
            "0.0110\tleast\n",
            "0.0100\tcontract\n",
            "0.0080\tmore\n",
            "0.0078\tpayment\n",
            "0.0076\tmethod\n",
            "0.0070\tamount\n",
            "0.0061\tinsurance\n",
            "0.0055\tbased\n",
            "0.0054\tperiod\n",
            "\n",
            "Topic 3\n",
            "0.0128\trisk\n",
            "0.0073\tone\n",
            "0.0066\tasset\n",
            "0.0065\tsystem\n",
            "0.0064\tbased\n",
            "0.0060\teach\n",
            "0.0059\tthat\n",
            "0.0057\tmethod\n",
            "0.0055\tdata\n",
            "0.0051\tassets\n",
            "\n",
            "Topic 4\n",
            "0.0243\tauction\n",
            "0.0123\tsystem\n",
            "0.0112\tseller\n",
            "0.0107\tbid\n",
            "0.0105\tbuyer\n",
            "0.0085\tservice\n",
            "0.0082\tinformation\n",
            "0.0074\tthat\n",
            "0.0064\tmethod\n",
            "0.0062\twhich\n",
            "\n",
            "Topic 5\n",
            "0.0091\tsystem\n",
            "0.0076\tmethod\n",
            "0.0073\tthat\n",
            "0.0065\tone\n",
            "0.0055\tfinancial\n",
            "0.0052\tcredit\n",
            "0.0049\tinvention\n",
            "0.0042\tvirtual\n",
            "0.0040\ttransactions\n",
            "0.0037\tproduct\n",
            "\n",
            "Topic 6\n",
            "0.0301\tdata\n",
            "0.0173\tsystem\n",
            "0.0106\tinformation\n",
            "0.0094\tuser\n",
            "0.0092\ttrading\n",
            "0.0084\tone\n",
            "0.0072\tplurality\n",
            "0.0062\tmethod\n",
            "0.0060\tinterface\n",
            "0.0058\tmarket\n",
            "\n",
            "Topic 7\n",
            "0.0207\ttrade\n",
            "0.0135\tthat\n",
            "0.0080\ttrading\n",
            "0.0066\tsystem\n",
            "0.0057\ttime\n",
            "0.0056\tcredit\n",
            "0.0052\torder\n",
            "0.0051\tprice\n",
            "0.0048\trisk\n",
            "0.0047\twhich\n",
            "\n",
            "Topic 8\n",
            "0.0288\tfinancial\n",
            "0.0098\tdata\n",
            "0.0094\tinformation\n",
            "0.0091\tsystem\n",
            "0.0085\tuser\n",
            "0.0076\tthat\n",
            "0.0074\tcomputer\n",
            "0.0061\tbased\n",
            "0.0059\ttransaction\n",
            "0.0057\tmethod\n",
            "\n",
            "Topic 9\n",
            "0.0115\torder\n",
            "0.0093\tthat\n",
            "0.0081\tproperty\n",
            "0.0080\tsuch\n",
            "0.0077\torders\n",
            "0.0068\tmarket\n",
            "0.0055\tprice\n",
            "0.0054\texchange\n",
            "0.0053\ttrading\n",
            "0.0052\tother\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Inspect topics\n",
        "for i, topic in lda_model.show_topics(num_words=50, formatted=False):\n",
        "    print(\"Topic\", i)\n",
        "    printed_terms = 0\n",
        "    for term, score in topic:\n",
        "        if printed_terms >= 10:\n",
        "            break\n",
        "        elif term in \"the of and to for in or The is be may an a with at are on by as from can\".split():\n",
        "            continue\n",
        "        printed_terms += 1\n",
        "        print(\"%.4f\\t%s\" % (score,term))\n",
        "    print()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w0cRKEvZ5wtA"
      },
      "source": [
        "For visualization of topic models, see, for instance, [pyLDAvis](https://github.com/bmabey/pyLDAvis)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k_RjbN6v5wtA"
      },
      "source": [
        "# Part II: Transfer learning (modern NLP)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2L0yqYls5wtA"
      },
      "source": [
        "# Word vectors\n",
        "\n",
        "- Recap: We used the term-document matrix to understand semantic similarities *between documents* based on overlapping keywords\n",
        "- Now, we study similarities *between words*, by comparing the close contexts in which they appear\n",
        "- The comparison is also done in a vector space, see illustration (with only 3 dimensions):\n",
        "\n",
        "![semantic_space](figs/semspace.svg \"Illustration of words embedded in a semantic vector space\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WcB6fWsQ5wtA"
      },
      "source": [
        "**Simple demo, document context:**\n",
        "- We can also inspect the term-document matrix vertically, by extracting *column vectors* for terms\n",
        "- Meaning by context: A term is described by the documents in which it occurs\n",
        "- The tf-idf scores of the term vector (or word vector) define coordinates in a high-dimensional space\n",
        "    - Distances can be measured, commonly by the cosine similarity function\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VErtSMDg5wtA",
        "outputId": "41a575af-8db9-4eff-8496-7c6141f4a3ab"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Similar terms to: confirmation\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/sklearn/utils/validation.py:598: FutureWarning: np.matrix usage is deprecated in 1.0 and will raise a TypeError in 1.2. Please convert to a numpy array with np.asarray. For more information see: https://numpy.org/doc/stable/reference/generated/numpy.matrix.html\n",
            "  FutureWarning,\n",
            "/usr/local/lib/python3.7/dist-packages/sklearn/utils/validation.py:598: FutureWarning: np.matrix usage is deprecated in 1.0 and will raise a TypeError in 1.2. Please convert to a numpy array with np.asarray. For more information see: https://numpy.org/doc/stable/reference/generated/numpy.matrix.html\n",
            "  FutureWarning,\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[(1.0000000000000004, 'confirmation'),\n",
              " (0.1457608865754832, 'consequence'),\n",
              " (0.14410381882972184, 'confirm'),\n",
              " (0.12744202494028473, 'confirming'),\n",
              " (0.12693387313222515, 'stp'),\n",
              " (0.12125411540432149, 'confirmed'),\n",
              " (0.11702088765632153, 'message'),\n",
              " (0.10537137289117164, 'patron'),\n",
              " (0.0952692856901657, 'to'),\n",
              " (0.09432629698865568, 'the')]"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ],
      "source": [
        "from sklearn.metrics.pairwise import cosine_similarity\n",
        "\n",
        "term_index = 1000\n",
        "print(\"Similar terms to:\", features[term_index])\n",
        "# Get most similar terms according to the cosine similarity of their vectors (columns in the term-document matrix)\n",
        "heapq.nlargest(10, zip(cosine_similarity(tfidf_matrix[:,term_index].todense().T, tfidf_matrix.todense().T)[0], features))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yOMle8JZ5wtA"
      },
      "source": [
        "### Better approach: Word co-occurrence matrix\n",
        "\n",
        "- Look at a more immediate context of word use, e.g., +/- 5 words\n",
        "- Count word co-occurrence statistics (--> word-word matrix)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xt65VpfH5wtB"
      },
      "source": [
        "**Analyzing word combinations (first-order co-occurrence):**\n",
        "- Strongly associated pairs of words can be identified using the *pointwise mutual information* (PMI) score:\n",
        "    - For words x and y: $$pmi(x,y) = \\log \\frac{p(x,y)}{p(x)p(y)} $$\n",
        "\n",
        "    - where p(x,y) is estimated as the co-occurrence frequency and p(x) and p(y) as the independent occurence frequencies\n",
        "- PMI can highlight common combinations of words and topics (e.g., \"poor\"-\"service\"; \"handsome\"-\"man\" / \"beautiful\"-\"woman\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1P2LnPWd5wtB"
      },
      "source": [
        "More reading on: [PMI](https://en.wikipedia.org/wiki/Pointwise_mutual_information) and [other association measures](http://collocations.de/AM/index.html)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FpcRyRgF5wtB"
      },
      "source": [
        "**Analyzing word similarity (second-order co-occurrence):**\n",
        "\n",
        "- \"You shall know a word by the company it keeps\" (J.R. Firth)\n",
        "    - The meaning of a word can be understood by the distribution of words it tends to appear in (= distributional hypothesis)\n",
        "    - Words may be similar although they to not directly co-occur (but share context words) --> 2nd order co-occurrence\n",
        "- Cosine function between word vectors gives word pair similarity"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NGRno_k05wtB"
      },
      "source": [
        "- Vectors are very sparse: \n",
        "    - Vocabulary (dimensionality) grows with corpus size, while context size is small and fixed\n",
        "    - Similarity can be estimated only when there is an overlap in terms\n",
        "    - Many rare terms / sparse vectors --> very big corpora needed for good models --> huge memory usage and scalability issues"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "97uqzLAi5wtB"
      },
      "source": [
        "### Learning dense vectors\n",
        "\n",
        "More useful vectors through:\n",
        "- Matrix decomposition approach:\n",
        "    - Reduce the number of dimensions of the semantic vector space, for instance, using Singular Value Decomposition (SVD)\n",
        "        - Dimensionality reduction, e.g., from 10k or 100k to 100\n",
        "    - Also used in Latent Semantic Analysis (LSA), applied to term-document matrix"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lf81QY7N5wtC"
      },
      "source": [
        "- Neural network approach:\n",
        "    - Learning *distributed representations* of concepts explored first by Hinton (1986)\n",
        "    - word2vec, from 2013, is an important milestone in NLP:\n",
        "        - Efficient estimation of dense word vectors, enabling fast training\n",
        "        - Provides high precision vectors when applied to large corpora\n",
        "\n",
        "- word2vec skip-gram model:\n",
        "<img src=\"https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/skipgram_model.svg?raw=1\" alt=\"skip-gram model\" width=\"800\"/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WZJtCkqZ5wtC"
      },
      "source": [
        "More reading on [counting vs. predicting word vectors](https://www.aclweb.org/anthology/P14-1023.pdf)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KSLcTdL45wtC",
        "outputId": "14bcde0a-be90-48f8-fbff-2d410064aaad"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "2022-02-16 09:50:39,129 : INFO : collecting all words and their counts\n",
            "2022-02-16 09:50:39,131 : INFO : PROGRESS: at sentence #0, processed 0 words, keeping 0 word types\n",
            "2022-02-16 09:50:39,331 : INFO : PROGRESS: at sentence #10000, processed 1138498 words, keeping 16039 word types\n",
            "2022-02-16 09:50:39,414 : INFO : collected 18113 word types from a corpus of 1590394 raw words and 13990 sentences\n",
            "2022-02-16 09:50:39,416 : INFO : Loading a fresh vocabulary\n",
            "2022-02-16 09:50:39,597 : INFO : effective_min_count=5 retains 8519 unique words (47% of original 18113, drops 9594)\n",
            "2022-02-16 09:50:39,599 : INFO : effective_min_count=5 leaves 1572573 word corpus (98% of original 1590394, drops 17821)\n",
            "2022-02-16 09:50:39,628 : INFO : deleting the raw counts dictionary of 18113 items\n",
            "2022-02-16 09:50:39,631 : INFO : sample=0.001 downsamples 48 most-common words\n",
            "2022-02-16 09:50:39,633 : INFO : downsampling leaves estimated 1156208 word corpus (73.5% of prior 1572573)\n",
            "2022-02-16 09:50:39,664 : INFO : estimated required memory for 8519 words and 100 dimensions: 11074700 bytes\n",
            "2022-02-16 09:50:39,666 : INFO : resetting layer weights\n",
            "2022-02-16 09:50:41,264 : INFO : training model with 4 workers on 8519 vocabulary and 100 features, using sg=0 hs=0 sample=0.001 negative=5 window=5\n",
            "2022-02-16 09:50:42,275 : INFO : EPOCH 1 - PROGRESS: at 51.34% examples, 596820 words/s, in_qsize 8, out_qsize 0\n",
            "2022-02-16 09:50:43,157 : INFO : worker thread finished; awaiting finish of 3 more threads\n",
            "2022-02-16 09:50:43,161 : INFO : worker thread finished; awaiting finish of 2 more threads\n",
            "2022-02-16 09:50:43,174 : INFO : worker thread finished; awaiting finish of 1 more threads\n",
            "2022-02-16 09:50:43,183 : INFO : worker thread finished; awaiting finish of 0 more threads\n",
            "2022-02-16 09:50:43,188 : INFO : EPOCH - 1 : training on 1590394 raw words (1156237 effective words) took 1.9s, 603226 effective words/s\n",
            "2022-02-16 09:50:44,213 : INFO : EPOCH 2 - PROGRESS: at 50.09% examples, 576984 words/s, in_qsize 7, out_qsize 0\n",
            "2022-02-16 09:50:45,143 : INFO : worker thread finished; awaiting finish of 3 more threads\n",
            "2022-02-16 09:50:45,154 : INFO : worker thread finished; awaiting finish of 2 more threads\n",
            "2022-02-16 09:50:45,162 : INFO : worker thread finished; awaiting finish of 1 more threads\n",
            "2022-02-16 09:50:45,167 : INFO : worker thread finished; awaiting finish of 0 more threads\n",
            "2022-02-16 09:50:45,169 : INFO : EPOCH - 2 : training on 1590394 raw words (1155719 effective words) took 2.0s, 585459 effective words/s\n",
            "2022-02-16 09:50:46,185 : INFO : EPOCH 3 - PROGRESS: at 51.99% examples, 600697 words/s, in_qsize 7, out_qsize 0\n",
            "2022-02-16 09:50:47,019 : INFO : worker thread finished; awaiting finish of 3 more threads\n",
            "2022-02-16 09:50:47,026 : INFO : worker thread finished; awaiting finish of 2 more threads\n",
            "2022-02-16 09:50:47,044 : INFO : worker thread finished; awaiting finish of 1 more threads\n",
            "2022-02-16 09:50:47,049 : INFO : worker thread finished; awaiting finish of 0 more threads\n",
            "2022-02-16 09:50:47,051 : INFO : EPOCH - 3 : training on 1590394 raw words (1156625 effective words) took 1.9s, 616699 effective words/s\n",
            "2022-02-16 09:50:48,083 : INFO : EPOCH 4 - PROGRESS: at 51.99% examples, 594278 words/s, in_qsize 7, out_qsize 0\n",
            "2022-02-16 09:50:48,912 : INFO : worker thread finished; awaiting finish of 3 more threads\n",
            "2022-02-16 09:50:48,920 : INFO : worker thread finished; awaiting finish of 2 more threads\n",
            "2022-02-16 09:50:48,936 : INFO : worker thread finished; awaiting finish of 1 more threads\n",
            "2022-02-16 09:50:48,944 : INFO : worker thread finished; awaiting finish of 0 more threads\n",
            "2022-02-16 09:50:48,946 : INFO : EPOCH - 4 : training on 1590394 raw words (1155819 effective words) took 1.9s, 612102 effective words/s\n",
            "2022-02-16 09:50:49,956 : INFO : EPOCH 5 - PROGRESS: at 52.56% examples, 611108 words/s, in_qsize 7, out_qsize 0\n",
            "2022-02-16 09:50:50,810 : INFO : worker thread finished; awaiting finish of 3 more threads\n",
            "2022-02-16 09:50:50,824 : INFO : worker thread finished; awaiting finish of 2 more threads\n",
            "2022-02-16 09:50:50,833 : INFO : worker thread finished; awaiting finish of 1 more threads\n",
            "2022-02-16 09:50:50,834 : INFO : worker thread finished; awaiting finish of 0 more threads\n",
            "2022-02-16 09:50:50,835 : INFO : EPOCH - 5 : training on 1590394 raw words (1155899 effective words) took 1.9s, 613715 effective words/s\n",
            "2022-02-16 09:50:50,841 : INFO : training on a 7951970 raw words (5780299 effective words) took 9.6s, 603851 effective words/s\n"
          ]
        }
      ],
      "source": [
        "### Train word vectors\n",
        "\n",
        "import gensim # Make sure you also have cython installed to accelerate computation!\n",
        "\n",
        "import logging\n",
        "logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)\n",
        "\n",
        "# Train word2vec model\n",
        "vectors = gensim.models.Word2Vec(tokenized_text, size=100, window=5, min_count=5, workers=4)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qTk6t6h75wtC",
        "outputId": "1b8c3ef7-e914-40b6-92ca-762d9328e581"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([-0.20422484,  0.31763935, -0.4086411 ,  0.40712512,  0.61239105,\n",
              "       -0.36136898, -0.36744946,  0.48536384, -0.77636147, -0.8818024 ,\n",
              "       -0.9706582 , -0.14553961, -0.9953826 ,  0.31411788, -0.48825204,\n",
              "       -0.98364806, -0.4044525 , -0.7629718 ,  0.07662179,  0.6668267 ,\n",
              "        1.1169783 , -0.9381234 , -0.47327662, -0.99468535,  0.7775386 ,\n",
              "       -0.89725596, -1.2869796 ,  0.59936583,  0.38795128, -0.21560162,\n",
              "        0.28929687, -1.1843922 ,  0.8098827 , -2.1943617 , -1.2991003 ,\n",
              "        1.7470335 ,  0.1376872 , -0.21452804,  0.89608234, -0.32440564,\n",
              "       -1.0077158 ,  0.80983657,  0.2833914 , -0.46517956, -1.0963019 ,\n",
              "        0.43595776,  0.8079535 , -1.2955928 , -0.41100898,  1.0340084 ,\n",
              "       -1.1356119 , -0.7746373 ,  0.9427416 , -0.69869095,  1.3082514 ,\n",
              "       -0.27267987,  0.85908425,  0.26002538, -0.17557463,  1.4576957 ,\n",
              "       -0.23210119, -0.39438847,  1.7945946 , -0.48386508,  1.2646602 ,\n",
              "        0.6007769 ,  0.228463  ,  0.10584885,  1.3912407 , -0.9092588 ,\n",
              "        0.6572006 , -0.9569288 ,  1.3129162 , -0.658416  , -1.2860945 ,\n",
              "       -1.2104768 ,  0.6322934 ,  0.22245555, -0.5003167 ,  1.0814933 ,\n",
              "       -0.38750708,  1.1751286 , -1.5327952 , -0.4788207 ,  1.1389452 ,\n",
              "        0.45741686,  0.94077396, -1.4665413 ,  0.2923041 ,  0.7282923 ,\n",
              "       -0.42117268, -1.779236  , -0.43622863, -1.3860005 ,  0.22563277,\n",
              "        1.1983752 , -0.8612148 ,  1.2455028 , -0.08006074, -0.40060958],\n",
              "      dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 26
        }
      ],
      "source": [
        "# Inspect a word vector\n",
        "vectors.wv['process']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "r5exY-7w5wtC",
        "outputId": "4b5eb5d4-768f-4214-cdcd-45c3c9c12745"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "2022-02-16 09:50:58,139 : INFO : precomputing L2-norms of word weight vectors\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Most similar to: process\n",
            "[('workflow', 0.6910364627838135), ('procedure', 0.6733125448226929), ('logic', 0.6215505599975586), ('processes', 0.6087464690208435), ('operation', 0.6058263778686523), ('decision', 0.6044843196868896), ('engine', 0.5916510820388794), ('negotiation', 0.5634622573852539), ('audit', 0.5561366677284241), ('methodology', 0.5537846088409424)]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Inspect words with vectors most similar to a given word\n",
        "print(\"Most similar to:\", 'process')\n",
        "print(vectors.wv.most_similar('process'))\n",
        "print()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ndI7KM615wtC",
        "outputId": "5ce6807b-80b2-42c5-ed33-e51efd3f43aa"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Most similar to: ['process', 'payment']\n",
            "[('processing', 0.6326549053192139), ('transaction', 0.6238937377929688), ('negotiation', 0.6085587739944458), ('authorization', 0.5986728668212891), ('approval', 0.5673232674598694), ('procedure', 0.563011884689331), ('settlement', 0.5614774227142334), ('operation', 0.5446236729621887), ('request', 0.5320792198181152), ('payments', 0.5281507968902588)]\n"
          ]
        }
      ],
      "source": [
        "# ...or combination or words (average vector)\n",
        "print(\"Most similar to:\", ['process', 'payment'])\n",
        "print(vectors.wv.most_similar(['process', 'payment']))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ygSUkMgD5wtD",
        "outputId": "6a88d919-f82b-4568-e9c7-aea08aa0b7c0"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.7070455\n",
            "0.40349627\n",
            "0.15130822\n",
            "0.70881736\n"
          ]
        }
      ],
      "source": [
        "# Inspect cosine similarities between specific words\n",
        "print(vectors.wv.similarity('technology', 'infrastructure'))\n",
        "print(vectors.wv.similarity('technology', 'system'))\n",
        "print(vectors.wv.similarity('technology', 'consumer'))\n",
        "print(vectors.wv.similarity('technology', 'advanced'))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 72
        },
        "id": "D_psVY655wtD",
        "outputId": "8f8752ef-3ca7-43b3-af6a-3ab723411da1"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "\"\\n# Man is to King as Woman is to ...\\n# vec(woman)+vec(king)-vec(man)\\nprint(vectors.most_similar(['woman','king'], negative=['man']))\\nprint()\\n\\n# Fall is to Fell as Watch is to ...\\nprint(vectors.most_similar(['wear','fell'], negative=['fall']))\\nprint()\\n\\nprint(vectors.most_similar(['Moscow','France'], negative=['Paris']))\\nprint()\\n\\nprint(vectors.most_similar(['London','France'], negative=['Paris']))\\n\""
            ]
          },
          "metadata": {},
          "execution_count": 30
        }
      ],
      "source": [
        "## Test for analogies and syntactic regularities (when trained on a large and suitable corpus)\n",
        "\n",
        "\"\"\"\n",
        "# Man is to King as Woman is to ...\n",
        "# vec(woman)+vec(king)-vec(man)\n",
        "print(vectors.most_similar(['woman','king'], negative=['man']))\n",
        "print()\n",
        "\n",
        "# Fall is to Fell as Watch is to ...\n",
        "print(vectors.most_similar(['wear','fell'], negative=['fall']))\n",
        "print()\n",
        "\n",
        "print(vectors.most_similar(['Moscow','France'], negative=['Paris']))\n",
        "print()\n",
        "\n",
        "print(vectors.most_similar(['London','France'], negative=['Paris']))\n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "agtLbh8O5wtD"
      },
      "source": [
        "### Pre-trained word vectors\n",
        "- Word vectors trained on large corpora achieve more accurate results\n",
        "- For instance, word2vec vectors pre-trained on 100B words of Google News text can be [downloaded](https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing) (1.5GB) and used in the same way as self-trained vectors\n",
        "- word2vec is also efficient for training your own vectors on mid-sized corpora (e.g., 1B words)\n",
        "    - Online demo [available here](http://bionlp-www.utu.fi/wv_demo/)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_Q7EUKJ95wtD"
      },
      "source": [
        "### Vectors as features\n",
        "\n",
        "- Dense vectors are practical to use especially in neural network models for downstream tasks (e.g., text classification)\n",
        "- Word vectors (often _word embeddings_) estimated on large amounts of raw text:\n",
        "    - Embedding layer can serve as an abstraction of text input\n",
        "    - Reduces reliance on (limited) annotated data in supervised tasks\n",
        "    - Pre-trained word vectors can make NLP models more accurate and robust\n",
        "    - This transfer of knowledge about language is called *transfer learning*\n",
        "\n",
        "<img src=\"https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/simple_lstm.svg?raw=1\" alt=\"Neural network with word embeddings\" width=\"400\"/>\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CbUXrIdy5wtD"
      },
      "source": [
        "### Limitations\n",
        "\n",
        "- Problem: Word vectors produced by word2vec are static with respect to the word\n",
        "    - I.e., a term always has the same vector representation regardless of how it is used\n",
        "    - Multiple senses of a word are not distinguished\n",
        "    - Difficult to capture meaning of sentences (unless fine-tuned on some task)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bCd9hbe05wtE",
        "outputId": "356d8781-7f2c-4cdc-c041-bded0cda4789"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Most similar to: data\n",
            "[('information', 0.7078695297241211), ('Data', 0.67232346534729), ('records', 0.6146502494812012), ('repository', 0.5902501344680786), ('record', 0.5762346386909485), ('files', 0.5615368485450745), ('database', 0.5484235882759094), ('fields', 0.5438851118087769), ('summary', 0.5435164570808411), ('table', 0.5370356440544128)]\n",
            "\n",
            "Most similar to: ['data', 'transfer']\n",
            "[('source', 0.6565860509872437), ('remittance', 0.6117523908615112), ('file', 0.6052581667900085), ('electronically', 0.5856301784515381), ('sender', 0.566631555557251), ('store', 0.5646321773529053), ('instruction', 0.5534605979919434), ('document', 0.5481635928153992), ('money', 0.5404837131500244), ('request', 0.53480464220047)]\n",
            "\n",
            "Most similar to: ['data', 'storage']\n",
            "[('repository', 0.7675380706787109), ('memory', 0.7538337707519531), ('recording', 0.691872775554657), ('database', 0.6729086637496948), ('store', 0.67164546251297), ('storing', 0.6642770767211914), ('databases', 0.6425287127494812), ('file', 0.6393442749977112), ('table', 0.63494873046875), ('record', 0.6338653564453125)]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Ambiguous word vector examples\n",
        "\n",
        "print(\"Most similar to:\", 'data')\n",
        "print(vectors.wv.most_similar('data'))\n",
        "print()\n",
        "\n",
        "# Are the below average vectors less ambiguous?\n",
        "print(\"Most similar to:\", ['data','transfer'])\n",
        "print(vectors.wv.most_similar(['data','transfer']))\n",
        "print()\n",
        "\n",
        "print(\"Most similar to:\", ['data','storage'])\n",
        "print(vectors.wv.most_similar(['data','storage']))\n",
        "print()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "49OPtb0-5wtE"
      },
      "source": [
        "## Contextualized word vectors with language modeling\n",
        "\n",
        "- Word meaning can be disambiguated from the context in an instance of use\n",
        "    - Compare:\n",
        "        - \"He wrote the **play** himself.\"\n",
        "        - \"He did **play** with the kids.\"\n",
        "        \n",
        "- Recent methods build on *language modeling* to provide *contextualized* word vectors"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NX7gviNW5wtE"
      },
      "source": [
        "- Recap: A language model estimates the probability *P(x)* of seeing a word _x_ following a context, such as:\n",
        "\n",
        "$$P(\\ x\\ |\\ the\\ students\\ opened\\ their\\ )$$\n",
        "\n",
        "![language modeling (source: mc.ai)](figs/language_modeling.png \"Language modeling illustration\")\n",
        "The above suggestions would be deemed much more likely than, e.g., *P(x=keyboard)*, _P(x=follows)_ or *P(x=the)*."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hz8Ft8Cg5wtE"
      },
      "source": [
        "<font color=\"grey\">Lanugage modeling recap: Consider a simple statistical model that estimates the probability of a word given a number of previous words in a text. For instance, given the words *the dog loved to*, the model would deem it much more likely that the next word is _play_ than any word that does not fit semantically or syntactically (e.g., *green*, _played_ or *code*). The probability _p(play | the dog loved to)_ is much higher than for most other words. Such a model can be implemented as a machine learning model, typically a neural network that can take into account a long context, and be trained over large amounts of text. Language models that have a large number of internal trainable parameters can learn complicated regularities of natural language, by learning to accurately predict the next word in a sequence.</font>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g0bxAasK5wtE"
      },
      "source": [
        "For instance, [ELMo](https://allennlp.org/elmo) (released early 2018) uses recurrent neural networks (LSTM) for learning contextualized word vectors through language modeling.\n",
        "\n",
        "<img src=\"https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/elmo-language-modeling.png?raw=1\" width=\"400\" alt=\"Source: https://jalammar.github.io/illustrated-bert/\"/>\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T2B32Wao5wtF"
      },
      "source": [
        "- ELMo holds internal vector representations as it traverses the word sequence, which can be aggregated to:\n",
        "    - Contextualized word vectors\n",
        "    - A sentence vector, reflecting composite meaning\n",
        "\n",
        "<img src=\"https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/elmo-embedding.png?raw=1\" width=\"550\" alt=\"Source: https://jalammar.github.io/illustrated-bert/\"/>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 487
        },
        "id": "aGxdR6IP5wtF",
        "outputId": "76221548-5775-4204-dbcd-42d9dab901a1"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "2022-02-16 09:51:39,925 : INFO : Using /tmp/tfhub_modules to cache modules.\n",
            "2022-02-16 09:51:39,933 : INFO : Downloading TF-Hub Module 'https://tfhub.dev/google/elmo/3'.\n",
            "2022-02-16 09:51:44,814 : INFO : Downloaded https://tfhub.dev/google/elmo/3, Total size: 357.40MB\n",
            "2022-02-16 09:51:44,816 : INFO : Downloaded TF-Hub Module 'https://tfhub.dev/google/elmo/3'.\n"
          ]
        },
        {
          "output_type": "error",
          "ename": "RuntimeError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-32-cd331230c3c9>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m \u001b[0;31m# Load ELMo model (takes a little while)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 9\u001b[0;31m \u001b[0melmo\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhub\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mModule\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"https://tfhub.dev/google/elmo/3\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrainable\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     10\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     11\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0melmo_vectors\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msents\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow_hub/module.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, spec, trainable, name, tags)\u001b[0m\n\u001b[1;32m    177\u001b[0m           \u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_name\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    178\u001b[0m           \u001b[0mtrainable\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_trainable\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 179\u001b[0;31m           tags=self._tags)\n\u001b[0m\u001b[1;32m    180\u001b[0m       \u001b[0;31m# pylint: enable=protected-access\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    181\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow_hub/native_module.py\u001b[0m in \u001b[0;36m_create_impl\u001b[0;34m(self, name, trainable, tags)\u001b[0m\n\u001b[1;32m    390\u001b[0m         \u001b[0mtrainable\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtrainable\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    391\u001b[0m         \u001b[0mcheckpoint_path\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_checkpoint_variables_path\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 392\u001b[0;31m         name=name)\n\u001b[0m\u001b[1;32m    393\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    394\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_export\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpath\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvariables_saver\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow_hub/native_module.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, spec, meta_graph, trainable, checkpoint_path, name)\u001b[0m\n\u001b[1;32m    449\u001b[0m     \u001b[0;31m# TPU training code.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    450\u001b[0m     \u001b[0;32mwith\u001b[0m \u001b[0mscope_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 451\u001b[0;31m       \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_init_state\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    452\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    453\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_init_state\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow_hub/native_module.py\u001b[0m in \u001b[0;36m_init_state\u001b[0;34m(self, name)\u001b[0m\n\u001b[1;32m    452\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    453\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_init_state\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 454\u001b[0;31m     \u001b[0mvariable_tensor_map\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_state_map\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_create_state_graph\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    455\u001b[0m     self._variable_map = recover_partitioned_variable_map(\n\u001b[1;32m    456\u001b[0m         get_node_map_from_tensor_map(variable_tensor_map))\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow_hub/native_module.py\u001b[0m in \u001b[0;36m_create_state_graph\u001b[0;34m(self, name)\u001b[0m\n\u001b[1;32m    510\u001b[0m         \u001b[0mmeta_graph\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    511\u001b[0m         \u001b[0minput_map\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 512\u001b[0;31m         import_scope=relative_scope_name)\n\u001b[0m\u001b[1;32m    513\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    514\u001b[0m     \u001b[0;31m# Build a list from the variable name in the module definition to the actual\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow/python/training/saver.py\u001b[0m in \u001b[0;36mimport_meta_graph\u001b[0;34m(meta_graph_or_file, clear_devices, import_scope, **kwargs)\u001b[0m\n\u001b[1;32m   1565\u001b[0m   return _import_meta_graph_with_return_elements(meta_graph_or_file,\n\u001b[1;32m   1566\u001b[0m                                                  \u001b[0mclear_devices\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mimport_scope\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1567\u001b[0;31m                                                  **kwargs)[0]\n\u001b[0m\u001b[1;32m   1568\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1569\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow/python/training/saver.py\u001b[0m in \u001b[0;36m_import_meta_graph_with_return_elements\u001b[0;34m(meta_graph_or_file, clear_devices, import_scope, return_elements, **kwargs)\u001b[0m\n\u001b[1;32m   1575\u001b[0m   \u001b[0;34m\"\"\"Import MetaGraph, and return both a saver and returned elements.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1576\u001b[0m   \u001b[0;32mif\u001b[0m \u001b[0mcontext\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexecuting_eagerly\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1577\u001b[0;31m     raise RuntimeError(\"Exporting/importing meta graphs is not supported when \"\n\u001b[0m\u001b[1;32m   1578\u001b[0m                        \u001b[0;34m\"eager execution is enabled. No graph exists when eager \"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1579\u001b[0m                        \"execution is enabled.\")\n",
            "\u001b[0;31mRuntimeError\u001b[0m: Exporting/importing meta graphs is not supported when eager execution is enabled. No graph exists when eager execution is enabled."
          ]
        }
      ],
      "source": [
        "#!pip install tensorflow==1.15\n",
        "#!pip install \"tensorflow_hub>=0.6.0\"\n",
        "#!pip3 install tensorflow_text==1.15\n",
        "\n",
        "import tensorflow_hub as hub\n",
        "import tensorflow as tf\n",
        "\n",
        "# Load ELMo model (takes a little while)\n",
        "elmo = hub.Module(\"https://tfhub.dev/google/elmo/3\", trainable=True)\n",
        "\n",
        "def elmo_vectors(sents):\n",
        "    embeddings = elmo(sents, signature=\"default\", as_dict=True)[\"elmo\"]\n",
        "    with tf.Session() as sess:\n",
        "        sess.run(tf.global_variables_initializer())\n",
        "        return sess.run(embeddings)\n",
        "        #sess.run(tf.tables_initializer())\n",
        "        # return average of ELMo features as sentence vector\n",
        "        #return sess.run(tf.reduce_mean(embeddings,1))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PlWsvJFi5wtF"
      },
      "outputs": [],
      "source": [
        "# Get contextualized vectors for target word in different sentences\n",
        "\n",
        "sents = \"\"\"He wrote the play himself .\n",
        "He did play with the kids .\n",
        "His excuse didn't play well .\n",
        "I saw the play with him .\n",
        "He can play it by ear .\"\"\".split('\\n')\n",
        "\n",
        "target = 'play'\n",
        "\n",
        "elmo_vecs = elmo_vectors(sents)\n",
        "word_vecs = []\n",
        "for i, sent in enumerate(sents):\n",
        "    word_vecs.append(elmo_vecs[i][sent.split().index(target)])\n",
        "    print(\"Sentence:\", sent)\n",
        "    print(\"Vector for '%s':\" % target, word_vecs[-1])\n",
        "    print()\n",
        "\n",
        "print(\"Word vector size:\", word_vecs[0].shape)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RTxDTb5L5wtF"
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics.pairwise import cosine_similarity\n",
        "\n",
        "vec_size = word_vecs[0].shape[0]\n",
        "print(\"Similarities between '%s' vector in sentences:\" % target)\n",
        "for i in range(1, len(sents)):\n",
        "    print(\"Sent 0-%d:\" % i, cosine_similarity(word_vecs[0].reshape((1,vec_size)), \n",
        "                                              word_vecs[i].reshape((1,vec_size)))[0][0])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vZVyMKJ75wtF"
      },
      "source": [
        "### Even bigger pre-trained models: BERT etc.\n",
        "\n",
        "- In late 2018, a new, much larger pre-trained language model was released called *BERT* (Bidirectional Encoder Representations from Transformers)\n",
        "- Transformers can be better parallelized than recurrent network architectures\n",
        "    - --> Enables training of much larger models than before (on GPU/TPU)\n",
        "- BERT Large model: 340M parameters ~ 24 Transformer layers x 1024 hidden units x 16 attention heads\n",
        "    - Compared with ELMo (93.6M parameters), BERT Base (110M)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "naaebaUn5wtG"
      },
      "source": [
        "- Language modeling by learning to predict individual masked words\n",
        "    - Tricky task that benefits from sophisticated \"understanding\" of sentences\n",
        "    - Transformers are good at modeling long-range dependencies"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z-hLEfbG5wtG"
      },
      "source": [
        "- Computationally very demaning to train (compute cost ~$10k per run, trained on 3.3B words)\n",
        "- But, fine-tuning model for downstream task (e.g., classification) is quick\n",
        "- BERT was able to beat the best performing models in a range of NLP tasks at once"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mx2WtP1b5wtG"
      },
      "source": [
        "Simple BERT model illustration. Each token position is connected to each position in the next layer.\n",
        "<img src=\"https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/bert.png?raw=1\" width=\"400\"/>\n",
        "More illustrated reading on the [Transformer architecture](https://jalammar.github.io/illustrated-transformer/)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ky2nlW2F5wtG"
      },
      "source": [
        "Visualization of token-token activations for a sentence per layer and attention head.\n",
        "<img src=\"https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/bertvis.png?raw=1\" source=\"Jesse Vig\"/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MjpwQZEr5wtG"
      },
      "source": [
        "- BERT has been particularly successful at *natural language understanding (NLU)* tasks, e.g.:\n",
        "    - Sentence acceptability\n",
        "    - Sentiment\n",
        "    - Paraphrase identification\n",
        "    - Natural language inference\n",
        "    - Question answering\n",
        "    \n",
        "\n",
        "- Note, however, these models risk learning spurious statistical patterns, rather than real inference (cf. [[1]](https://www.technologyreview.com/s/615126/ai-common-sense-reads-human-language-ai2/)[[2]](https://www.aclweb.org/anthology/P19-1459.pdf))!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TmwM5JAY5wtG"
      },
      "source": [
        "BERT has inspired a never-ending stream of successors. See [GLUE (NLU tasks) leaderboard](https://gluebenchmark.com/leaderboard).\n",
        "<img alt=\"Map of the BERTology landscape\" src=\"https://github.com/TurkuNLP/intro-to-nlp/blob/master/figs/bertology_map.png?raw=1\"/>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PV3ojJuT5wtH"
      },
      "source": [
        "Getting started with Transformer-based language models (GPU highly recommended): [huggingface transformers library](https://huggingface.co/transformers/pretrained_models.html)\n",
        "\n",
        "<code>import torch\n",
        "from transformers import BertTokenizer, BertModel\n",
        "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')\n",
        "model = BertModel.from_pretrained('bert-base-uncased')\n",
        "...</code>\n",
        "\n",
        "or with `bert-base-finnish-cased-v1` model from TurkuNLP!\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tdViYiPA5wtH"
      },
      "source": [
        "Multilingual models include Multilingual BERT `bert-base-multilingual-cased` and the more recent XLM-RoBERTa (XLM-R) `xlm-roberta-base` or `xlm-roberta-large`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ji_vG3-S5wtH"
      },
      "outputs": [],
      "source": [
        ""
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.9"
    },
    "colab": {
      "name": "unsupervised_nlp.ipynb",
      "provenance": [],
      "toc_visible": true,
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}