learn/rag/project_simple_rag_chroma_paper_titles/rag-paper-titles.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qLiDBaSrzWu8"
      },
      "source": [
        "# Getting Started with RAG using Fireworks Fast Inference LLMs\n",
        "\n",
        "<a href=\"https://colab.research.google.com/drive/1v_Snr3bcPD-hYF-sk3C-4FaRBjOXjo2T?usp=sharing\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
        "\n",
        "While large language models (LLMs) show powerful capabilities that power advanced use cases, they suffer from issues such as factual inconsistency and hallucination. Retrieval-augmented generation (RAG) is a powerful approach to enrich LLM capabilities and improve their reliability. RAG involves combining LLMs with external knowledge by enriching the prompt context with relevant information that helps accomplish a task.\n",
        "\n",
        "This tutorial shows how to getting started with RAG by leveraging vector store and open-source LLMs. To showcase the power of RAG, this use case will cover building a RAG system that suggests short and easy to read ML paper titles from original ML paper titles. Paper tiles can be too technical for a general audience so using RAG to generate short titles based on previously created short titles can make research paper titles more accessible and used for science communication such as in the form of newsletters or blogs."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UsChkJxn2CSZ"
      },
      "source": [
        "Before getting started, let's first install the libraries we will use:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "9gy2ijb5zWu-"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "!pip install chromadb tqdm fireworks-ai python-dotenv pandas\n",
        "!pip install sentence-transformers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LM3QD4orAPR1"
      },
      "source": [
        "Let's download the dataset we will use:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hKODF7sRAVhS",
        "outputId": "a815994f-431a-4a13-d7d8-6076ba85e63f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "--2023-11-10 15:04:22--  https://raw.githubusercontent.com/dair-ai/ML-Papers-of-the-Week/main/research/ml-potw-10232023.csv\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.108.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 664158 (649K) [text/plain]\n",
            "Saving to: ‘ml-potw-10232023.csv’\n",
            "\n",
            "ml-potw-10232023.cs 100%[===================>] 648.59K  --.-KB/s    in 0.06s   \n",
            "\n",
            "2023-11-10 15:04:22 (10.7 MB/s) - ‘ml-potw-10232023.csv’ saved [664158/664158]\n",
            "\n",
            "mkdir: cannot create directory ‘data’: File exists\n"
          ]
        }
      ],
      "source": [
        "!wget https://raw.githubusercontent.com/dair-ai/ML-Papers-of-the-Week/main/research/ml-potw-10232023.csv\n",
        "!mkdir data\n",
        "!mv ml-potw-10232023.csv data/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-3iYAReMAe1q"
      },
      "source": [
        "Before continuing, you need to obtain a Fireworks API Key to use the Mistral 7B model.\n",
        "\n",
        "Checkout this quick guide to obtain your Fireworks API Key: https://readme.fireworks.ai/docs"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "pBSEMYFszWu_"
      },
      "outputs": [],
      "source": [
        "import fireworks.client\n",
        "import os\n",
        "import dotenv\n",
        "import chromadb\n",
        "import json\n",
        "from tqdm.auto import tqdm\n",
        "import pandas as pd\n",
        "import random\n",
        "from google.colab import userdata\n",
        "\n",
        "# you can set envs using Colab secrets\n",
        "fireworks.client.api_key = userdata.get('FIREWORKS_API_KEY')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q9v_0IEDtgov"
      },
      "source": [
        "## Getting Started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J8wYyXMizWu_"
      },
      "source": [
        "Let's define a function to get completions from the Fireworks inference platform."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 114,
      "metadata": {
        "id": "1hZldHjmzWvA"
      },
      "outputs": [],
      "source": [
        "def get_completion(prompt, model=None, max_tokens=50):\n",
        "\n",
        "    fw_model_dir = \"accounts/fireworks/models/\"\n",
        "\n",
        "    if model is None:\n",
        "        model = fw_model_dir + \"llama-v2-7b\"\n",
        "    else:\n",
        "        model = fw_model_dir + model\n",
        "\n",
        "    completion = fireworks.client.Completion.create(\n",
        "        model=model,\n",
        "        prompt=prompt,\n",
        "        max_tokens=max_tokens,\n",
        "        temperature=0\n",
        "    )\n",
        "\n",
        "    return completion.choices[0].text"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ys59WgrGzWvA"
      },
      "source": [
        "Let's first try the function with a simple prompt:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "id": "edQeSLODzWvA",
        "outputId": "fb0174c2-1490-424c-98a5-fd954a833d40"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "\" Robert and I'm Italian living in Mexico. I have a passion for photography and movies. I hope you will enjoy them too. You can also notice that I'm a huge fan of James Bond 007, Michael Bay\""
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "get_completion(\"Hello, my name is\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CwDjmi8EzWvB"
      },
      "source": [
        "Now let's test with Mistral-7B-Instruct:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "id": "O9TwL-2DzWvB",
        "outputId": "affb05c1-6a61-4a31-8dee-f2ac6a769fd9"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "' Jane. Today, I would like to share with you some interesting facts about London.\\n\\nLondon is the capital of England and the United Kingdom, and it is one of the oldest cities in the world. The city has a rich history, dating'"
            ]
          },
          "execution_count": 6,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "mistral_llm = \"mistral-7b-instruct-4k\"\n",
        "\n",
        "get_completion(\"Hello, my name is\", model=mistral_llm)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LZdlBtx-zWvB"
      },
      "source": [
        "The Mistral 7B Instruct model needs to be instructed using special instruction tokens `[INST] <instruction> [/INST]` to get the right behavior. You can find more instructions on how to prompt Mistral 7B Instruct here: https://docs.mistral.ai/llm/mistral-instruct-v0.1"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "id": "ITURzGa9zWvC",
        "outputId": "5ba3d395-9887-438a-d2a8-808c1598cda6"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'!\\n1: Why don’t scientists trust atoms? Because they make up everything! \\n\\n2: Did you hear they are republishing that Simple Mathematics study guide? It’s the revised addition.'"
            ]
          },
          "execution_count": 7,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "mistral_llm = \"mistral-7b-instruct-4k\"\n",
        "\n",
        "get_completion(\"Tell me 2 jokes\", model=mistral_llm)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "id": "QN6Y2y1GzWvC",
        "outputId": "4181f5e5-cb73-47d4-c1f3-1ea1e2f8eb8b"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "\" Sure, here are two jokes for you:\\n\\n1. Why don't scientists trust atoms? Because they make up everything!\\n2. What do you call a fake noodle? An impasta!\""
            ]
          },
          "execution_count": 8,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "mistral_llm = \"mistral-7b-instruct-4k\"\n",
        "\n",
        "get_completion(\"[INST]Tell me 2 jokes[/INST]\", model=mistral_llm)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jZHyn0tJzWvC"
      },
      "source": [
        "Now let's try with a more complex prompt that involves instructions:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "id": "dVwL--2kzWvC",
        "outputId": "1e1d37cd-ba24-4b0a-ab2f-62b516d515f1"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "\" Dear John Doe,\\n\\nWe, Tom and Mary, are writing to thank you for attending our wedding. We were thrilled to have you there and share such a special occasion with us. We couldn't help but be impressed by your amazing accomplishment of climbing Mount Everest in 2020, and we are grateful for the insight and experiences you brought to our wedding. Thank you again for being a wonderful guest and for being a part of our special day.\\n\\nSincerely,\\nTom and Mary\""
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "prompt = \"\"\"[INST]\n",
        "Given the following wedding guest data, write a very short 3-sentences thank you letter:\n",
        "\n",
        "{\n",
        "  \"name\": \"John Doe\",\n",
        "  \"relationship\": \"Bride's cousin\",\n",
        "  \"hometown\": \"New York, NY\",\n",
        "  \"fun_fact\": \"Climbed Mount Everest in 2020\",\n",
        "  \"attending_with\": \"Sophia Smith\",\n",
        "  \"bride_groom_name\": \"Tom and Mary\"\n",
        "}\n",
        "\n",
        "Use only the data provided in the JSON object above.\n",
        "\n",
        "The senders of the letter is the bride and groom, Tom and Mary.\n",
        "[/INST]\"\"\"\n",
        "\n",
        "get_completion(prompt, model=mistral_llm, max_tokens=150)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9SROezW6zWvD"
      },
      "source": [
        "## RAG Use Case: Generating Short Paper Titles\n",
        "\n",
        "For the RAG use case, we will be using [a dataset](https://github.com/dair-ai/ML-Papers-of-the-Week/tree/main/research) that contains a list of weekly top trending ML papers.\n",
        "\n",
        "The user will provide an original paper title. We will then take that input and then use the dataset to generate a context of short and catchy papers titles that will help generate catchy title for the original input title.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0sw7Uk6qzWvD"
      },
      "source": [
        "### Step 1: Load the Dataset\n",
        "\n",
        "Let's first load the dataset we will use:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "mv1z1LLczWvD"
      },
      "outputs": [],
      "source": [
        "# load dataset from data/ folder to pandas dataframe\n",
        "# dataset contains column names\n",
        "\n",
        "ml_papers = pd.read_csv(\"data/ml-potw-10232023.csv\", header=0)\n",
        "\n",
        "# remove rows with empty titles or descriptions\n",
        "ml_papers = ml_papers.dropna(subset=[\"Title\", \"Description\"])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 467
        },
        "id": "ErNNQRPqzWvD",
        "outputId": "817472e8-8b94-4fd6-c207-b26c62d2babd"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-3dc47341-6df3-43a9-9cb2-859c68d247b4\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Title</th>\n",
              "      <th>Description</th>\n",
              "      <th>PaperURL</th>\n",
              "      <th>TweetURL</th>\n",
              "      <th>Abstract</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Llemma</td>\n",
              "      <td>an LLM for mathematics which is based on conti...</td>\n",
              "      <td>https://arxiv.org/abs/2310.10631</td>\n",
              "      <td>https://x.com/zhangir_azerbay/status/171409802...</td>\n",
              "      <td>We present Llemma, a large language model for ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>LLMs for Software Engineering</td>\n",
              "      <td>a comprehensive survey of LLMs for software en...</td>\n",
              "      <td>https://arxiv.org/abs/2310.03533</td>\n",
              "      <td>https://x.com/omarsar0/status/1713940983199506...</td>\n",
              "      <td>This paper provides a survey of the emerging a...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Self-RAG</td>\n",
              "      <td>presents a new retrieval-augmented framework t...</td>\n",
              "      <td>https://arxiv.org/abs/2310.11511</td>\n",
              "      <td>https://x.com/AkariAsai/status/171511027707796...</td>\n",
              "      <td>Despite their remarkable capabilities, large l...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Retrieval-Augmentation for Long-form Question ...</td>\n",
              "      <td>explores retrieval-augmented language models o...</td>\n",
              "      <td>https://arxiv.org/abs/2310.12150</td>\n",
              "      <td>https://x.com/omarsar0/status/1714986431859282...</td>\n",
              "      <td>We present a study of retrieval-augmented lang...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>GenBench</td>\n",
              "      <td>presents a framework for characterizing and un...</td>\n",
              "      <td>https://www.nature.com/articles/s42256-023-007...</td>\n",
              "      <td>https://x.com/AIatMeta/status/1715041427283902...</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-3dc47341-6df3-43a9-9cb2-859c68d247b4')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-3dc47341-6df3-43a9-9cb2-859c68d247b4 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-3dc47341-6df3-43a9-9cb2-859c68d247b4');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-955efae2-3381-4f2c-9566-3fbc5c06b068\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-955efae2-3381-4f2c-9566-3fbc5c06b068')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-955efae2-3381-4f2c-9566-3fbc5c06b068 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "                                               Title  \\\n",
              "0                                             Llemma   \n",
              "1                      LLMs for Software Engineering   \n",
              "2                                           Self-RAG   \n",
              "3  Retrieval-Augmentation for Long-form Question ...   \n",
              "4                                           GenBench   \n",
              "\n",
              "                                         Description  \\\n",
              "0  an LLM for mathematics which is based on conti...   \n",
              "1  a comprehensive survey of LLMs for software en...   \n",
              "2  presents a new retrieval-augmented framework t...   \n",
              "3  explores retrieval-augmented language models o...   \n",
              "4  presents a framework for characterizing and un...   \n",
              "\n",
              "                                            PaperURL  \\\n",
              "0                   https://arxiv.org/abs/2310.10631   \n",
              "1                   https://arxiv.org/abs/2310.03533   \n",
              "2                   https://arxiv.org/abs/2310.11511   \n",
              "3                   https://arxiv.org/abs/2310.12150   \n",
              "4  https://www.nature.com/articles/s42256-023-007...   \n",
              "\n",
              "                                            TweetURL  \\\n",
              "0  https://x.com/zhangir_azerbay/status/171409802...   \n",
              "1  https://x.com/omarsar0/status/1713940983199506...   \n",
              "2  https://x.com/AkariAsai/status/171511027707796...   \n",
              "3  https://x.com/omarsar0/status/1714986431859282...   \n",
              "4  https://x.com/AIatMeta/status/1715041427283902...   \n",
              "\n",
              "                                            Abstract  \n",
              "0  We present Llemma, a large language model for ...  \n",
              "1  This paper provides a survey of the emerging a...  \n",
              "2  Despite their remarkable capabilities, large l...  \n",
              "3  We present a study of retrieval-augmented lang...  \n",
              "4                                                NaN  "
            ]
          },
          "execution_count": 11,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "ml_papers.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "KzyvzYcNzWvD"
      },
      "outputs": [],
      "source": [
        "# convert dataframe to list of dicts with Title and Description columns only\n",
        "\n",
        "ml_papers_dict = ml_papers.to_dict(orient=\"records\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "F3LUGNHIzWvE",
        "outputId": "3b1aa123-e316-488f-d0a4-0369bb2f75dd"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'Title': 'Llemma',\n",
              " 'Description': 'an LLM for mathematics which is based on continued pretraining from Code Llama on the Proof-Pile-2 dataset; the dataset involves scientific paper, web data containing mathematics, and mathematical code; Llemma outperforms open base models and the unreleased Minerva on the MATH benchmark; the model is released, including dataset and code to replicate experiments.',\n",
              " 'PaperURL': 'https://arxiv.org/abs/2310.10631',\n",
              " 'TweetURL': 'https://x.com/zhangir_azerbay/status/1714098025956864031?s=20',\n",
              " 'Abstract': 'We present Llemma, a large language model for mathematics. We continue pretraining Code Llama on the Proof-Pile-2, a mixture of scientific papers, web data containing mathematics, and mathematical code, yielding Llemma. On the MATH benchmark Llemma outperforms all known open base models, as well as the unreleased Minerva model suite on an equi-parameter basis. Moreover, Llemma is capable of tool use and formal theorem proving without any further finetuning. We openly release all artifacts, including 7 billion and 34 billion parameter models, the Proof-Pile-2, and code to replicate our experiments.'}"
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "ml_papers_dict[0]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WwfW0XoxzWvE"
      },
      "source": [
        "We will be using SentenceTransformer for generating embeddings that we will store to a chroma document store."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "1zFDOicHzWvE"
      },
      "outputs": [],
      "source": [
        "from chromadb import Documents, EmbeddingFunction, Embeddings\n",
        "from sentence_transformers import SentenceTransformer\n",
        "embedding_model = SentenceTransformer('all-MiniLM-L6-v2')\n",
        "\n",
        "class MyEmbeddingFunction(EmbeddingFunction):\n",
        "    def __call__(self, input: Documents) -> Embeddings:\n",
        "        batch_embeddings = embedding_model.encode(input)\n",
        "        return batch_embeddings.tolist()\n",
        "\n",
        "embed_fn = MyEmbeddingFunction()\n",
        "\n",
        "# Initialize the chromadb directory, and client.\n",
        "client = chromadb.PersistentClient(path=\"./chromadb\")\n",
        "\n",
        "# create collection\n",
        "collection = client.get_or_create_collection(\n",
        "    name=f\"ml-papers-nov-2023\"\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eu0_-PREzWvE"
      },
      "source": [
        "We will now generate embeddings for batches:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 49,
          "referenced_widgets": [
            "b9da413d4f84436ab5dc0fd10d237b0a",
            "c269f70baec246288519dbb2517c05c0",
            "b568c3c04efb49acb00e44aadc247735",
            "868c943d8a82435a8c3df6f32a3cc433",
            "51ad257305a0438fbed46c613d2d59fb",
            "cfbba792e6054178b504c2e2bbc23b2f",
            "54c8ecbae313483e82879016cc49bd25",
            "8e06dc7bf2f94d63a69b651f594ecf74",
            "bacb31eccd5c4a2dba61503a658333f2",
            "b9545ef27bf24e0c86d713ae8a3c0d2c",
            "0285ca3156854ca09a252540ad6a43ff"
          ]
        },
        "id": "kUauose2zWvE",
        "outputId": "18b7bf6a-0341-4843-8168-9875d78a6de9"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "b9da413d4f84436ab5dc0fd10d237b0a",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "  0%|          | 0/9 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Generate embeddings, and index titles in batches\n",
        "batch_size = 50\n",
        "\n",
        "# loop through batches and generated + store embeddings\n",
        "for i in tqdm(range(0, len(ml_papers_dict), batch_size)):\n",
        "\n",
        "    i_end = min(i + batch_size, len(ml_papers_dict))\n",
        "    batch = ml_papers_dict[i : i + batch_size]\n",
        "\n",
        "    # Replace title with \"No Title\" if empty string\n",
        "    batch_titles = [str(paper[\"Title\"]) if str(paper[\"Title\"]) != \"\" else \"No Title\" for paper in batch]\n",
        "    batch_ids = [str(sum(ord(c) + random.randint(1, 10000) for c in paper[\"Title\"])) for paper in batch]\n",
        "    batch_metadata = [dict(url=paper[\"PaperURL\"],\n",
        "                           abstract=paper['Abstract'])\n",
        "                           for paper in batch]\n",
        "\n",
        "    # generate embeddings\n",
        "    batch_embeddings = embedding_model.encode(batch_titles)\n",
        "\n",
        "    # upsert to chromadb\n",
        "    collection.upsert(\n",
        "        ids=batch_ids,\n",
        "        metadatas=batch_metadata,\n",
        "        documents=batch_titles,\n",
        "        embeddings=batch_embeddings.tolist(),\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1xrbURsMzWvF"
      },
      "source": [
        "Now we can test the retriever:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YoDlxtZhzWvF",
        "outputId": "c258fabb-452d-4740-9073-3d3cf7791bb7"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[['LLMs for Software Engineering', 'Communicative Agents for Software Development']]\n"
          ]
        }
      ],
      "source": [
        "collection = client.get_or_create_collection(\n",
        "    name=f\"ml-papers-nov-2023\",\n",
        "    embedding_function=embed_fn\n",
        ")\n",
        "\n",
        "retriever_results = collection.query(\n",
        "    query_texts=[\"Software Engineering\"],\n",
        "    n_results=2,\n",
        ")\n",
        "\n",
        "print(retriever_results[\"documents\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NUHeag1XzWvF"
      },
      "source": [
        "Now let's put together our final prompt:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 121,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "x_A0VZ8YzWvF",
        "outputId": "2b3074dc-381e-4cc0-9ee8-ea90673e0da9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model Suggestions:\n",
            "\n",
            "1. S3Eval: A Comprehensive Evaluation Suite for Large Language Models\n",
            "2. Synthetic and Scalable Evaluation for Large Language Models\n",
            "3. Systematic Evaluation of Large Language Models with S3Eval\n",
            "4. S3Eval: A Synthetic and Scalable Approach to Evaluating Large Language Models\n",
            "5. S3Eval: A Synthetic and Scalable Evaluation Suite for Large Language Models\n",
            "\n",
            "\n",
            "\n",
            "Prompt Template:\n",
            "[INST]\n",
            "\n",
            "Your main task is to generate 5 SUGGESTED_TITLES based for the PAPER_TITLE \n",
            "\n",
            "You should mimic a similar style and length as SHORT_TITLES but PLEASE DO NOT include titles from SHORT_TITLES in the SUGGESTED_TITLES, only generate versions of the PAPER_TILE.\n",
            "\n",
            "PAPER_TITLE: S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models\n",
            "\n",
            "SHORT_TITLES: Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling\n",
            "ChemCrow: Augmenting large-language models with chemistry tools\n",
            "A Survey of Large Language Models\n",
            "LLaMA: Open and Efficient Foundation Language Models\n",
            "SparseGPT: Massive Language Models Can Be Accurately Pruned In One-Shot\n",
            "REPLUG: Retrieval-Augmented Black-Box Language Models\n",
            "LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention\n",
            "Auditing large language models: a three-layered approach\n",
            "Fine-Tuning Language Models with Just Forward Passes\n",
            "DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving Agents\n",
            "\n",
            "SUGGESTED_TITLES:\n",
            "\n",
            "[/INST]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# user query\n",
        "user_query = \"S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models\"\n",
        "\n",
        "# query for user query\n",
        "results = collection.query(\n",
        "    query_texts=[user_query],\n",
        "    n_results=10,\n",
        ")\n",
        "\n",
        "# concatenate titles into a single string\n",
        "short_titles = '\\n'.join(results['documents'][0])\n",
        "\n",
        "prompt_template = f'''[INST]\n",
        "\n",
        "Your main task is to generate 5 SUGGESTED_TITLES based for the PAPER_TITLE\n",
        "\n",
        "You should mimic a similar style and length as SHORT_TITLES but PLEASE DO NOT include titles from SHORT_TITLES in the SUGGESTED_TITLES, only generate versions of the PAPER_TILE.\n",
        "\n",
        "PAPER_TITLE: {user_query}\n",
        "\n",
        "SHORT_TITLES: {short_titles}\n",
        "\n",
        "SUGGESTED_TITLES:\n",
        "\n",
        "[/INST]\n",
        "'''\n",
        "\n",
        "responses = get_completion(prompt_template, model=mistral_llm, max_tokens=2000)\n",
        "suggested_titles = ''.join([str(r) for r in responses])\n",
        "\n",
        "# Print the suggestions.\n",
        "print(\"Model Suggestions:\")\n",
        "print(suggested_titles)\n",
        "print(\"\\n\\n\\nPrompt Template:\")\n",
        "print(prompt_template)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cSAJcQ6Y2cNt"
      },
      "source": [
        "As you can see, the short titles generated by the LLM are somewhat okay. This use case still needs a lot more work and could potentially benefit from finetuning as well. For the purpose of this tutorial, we have provided a simple application of RAG using open-source models from Firework's blazing-fast models.\n",
        "\n",
        "Try out other open-source models here: https://app.fireworks.ai/models\n",
        "\n",
        "Read more about the Fireworks APIs here: https://readme.fireworks.ai/reference/createchatcompletion\n"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "rag",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.18"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "0285ca3156854ca09a252540ad6a43ff": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "51ad257305a0438fbed46c613d2d59fb": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "54c8ecbae313483e82879016cc49bd25": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "868c943d8a82435a8c3df6f32a3cc433": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b9545ef27bf24e0c86d713ae8a3c0d2c",
            "placeholder": "​",
            "style": "IPY_MODEL_0285ca3156854ca09a252540ad6a43ff",
            "value": " 9/9 [00:04&lt;00:00,  2.22it/s]"
          }
        },
        "8e06dc7bf2f94d63a69b651f594ecf74": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b568c3c04efb49acb00e44aadc247735": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8e06dc7bf2f94d63a69b651f594ecf74",
            "max": 9,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_bacb31eccd5c4a2dba61503a658333f2",
            "value": 9
          }
        },
        "b9545ef27bf24e0c86d713ae8a3c0d2c": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b9da413d4f84436ab5dc0fd10d237b0a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_c269f70baec246288519dbb2517c05c0",
              "IPY_MODEL_b568c3c04efb49acb00e44aadc247735",
              "IPY_MODEL_868c943d8a82435a8c3df6f32a3cc433"
            ],
            "layout": "IPY_MODEL_51ad257305a0438fbed46c613d2d59fb"
          }
        },
        "bacb31eccd5c4a2dba61503a658333f2": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "c269f70baec246288519dbb2517c05c0": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_cfbba792e6054178b504c2e2bbc23b2f",
            "placeholder": "​",
            "style": "IPY_MODEL_54c8ecbae313483e82879016cc49bd25",
            "value": "100%"
          }
        },
        "cfbba792e6054178b504c2e2bbc23b2f": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}