learn/function-calling/notebooks_firefunction_openai/fireworks_firefunction_openai_qa.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "f5OOOJoXB-O3",
      "metadata": {
        "id": "f5OOOJoXB-O3"
      },
      "source": [
        "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1IF3tJX3eqfB14doiegE_9eJ1Vme0vvtn?usp=drive_link)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "71a43144",
      "metadata": {
        "id": "71a43144"
      },
      "source": [
        "# Structured answers with Fireworks functions\n",
        "\n",
        "Several real world applications of LLM require them to respond in a structured manner. This structured response could look like `JSON` or `YAML`. For e.g. answering research questions using arxiv along with citations. Instead of parsing the entire LLM response and trying to figure out the actual answer of the LLM vs the citations provided by the LLM, we can use function calling ability of the LLMs to answer questions in a structured way.\n",
        "\n",
        "In this notebook, we demonstrate structured response generation ability of the Fireworks function calling model. We will build an application that can answer questions (along with citations) regarding the State of the Union speech of 2022."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "18320b01",
      "metadata": {},
      "source": [
        "# 🚀 Running This Notebook\n",
        "This notebook is designed to be run in Google Colab for a seamless experience. If you prefer to run it locally, please follow the setup instructions below.\n",
        "\n",
        "## To Running Locally\n",
        "To run this notebook locally, make sure to:\n",
        "1. Set up a Python virtual environment.\n",
        "2. Install the required libraries (`openai`, `jupyter`, and `python-dotenv`).\n",
        "3. Configure your API key and launch the Jupyter Notebook server.\n",
        "\n",
        "You can find detailed setup instructions in the following cells."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "dfce55b2",
      "metadata": {},
      "source": [
        "## Local Setup Instructions\n",
        "\n",
        "To run this notebook locally, follow these steps:\n",
        "\n",
        "### Step 1: Create a Virtual Environment\n",
        "In your terminal, navigate to the directory where this notebook is located and run:\n",
        "```bash\n",
        "python3 -m venv venv\n",
        "source venv/bin/activate  # On macOS/Linux\n",
        ".\u000benv\\Scripts\u0007ctivate   # On Windows"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "753f3c68",
      "metadata": {},
      "source": [
        "### Step 2: Install Required Libraries\n",
        "Install the necessary packages:\n",
        "```bash\n",
        "pip install jupyter openai python-dotenv\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "903bce74",
      "metadata": {},
      "source": [
        "### Step 3: Set Up Your API Key\n",
        "You can set your API key in the terminal:\n",
        "- **On macOS/Linux**:\n",
        "  ```bash\n",
        "  export FIREWORKS_API_KEY=<YOUR_FIREWORKS_API_KEY>\n",
        "  ```\n",
        "- **On Windows**:\n",
        "  ```bash\n",
        "  set FIREWORKS_API_KEY=<YOUR_FIREWORKS_API_KEY>\n",
        "  ```\n",
        "\n",
        "Alternatively, create a `.env` file in the project directory with:\n",
        "```\n",
        "FIREWORKS_API_KEY=<YOUR_FIREWORKS_API_KEY>\n",
        "```\n",
        "\n",
        "Load the `.env` file in your Python code with:\n",
        "```python\n",
        "from dotenv import load_dotenv\n",
        "load_dotenv()\n",
        "```\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "205ef68d",
      "metadata": {},
      "source": [
        "### Step 4: Launch Jupyter Notebook\n",
        "Start the Jupyter Notebook server:\n",
        "```bash\n",
        "jupyter notebook\n",
        "```\n",
        "Open this notebook file (`fireworks_demo.ipynb`) and proceed to run the cells."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5a787203",
      "metadata": {},
      "source": [
        "## How Function Calling Works\n",
        "\n",
        "The function-calling process involves the following steps:\n",
        "\n",
        "1. **Define User Query and Tools**: Specify the user query and the available tools using the `messages` and `tools` arguments.\n",
        "2. **Model Decision**: The model determines whether to respond directly or generate a tool call based on the user query.\n",
        "3. **User Executes Tool Call**: If the model generates a tool call, the user must execute the function manually and provide the result back to the model.\n",
        "4. **Response Generation**: The model uses the tool call result to generate a final response.\n",
        "\n",
        "For more details, refer to:\n",
        "- [Fireworks Blog Post on FireFunction-v2](https://fireworks.ai/blog/firefunction-v2-launch-post)\n",
        "- [OpenAI Function Calling Guide](https://platform.openai.com/docs/guides/function-calling)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "-7tAxHrBp4IQ",
      "metadata": {
        "id": "-7tAxHrBp4IQ"
      },
      "source": [
        "# Setup\n",
        "\n",
        "Install all the dependencies and import the required python modules."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "abd69237",
      "metadata": {},
      "outputs": [],
      "source": [
        "!pip3 install openai"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "id": "f059012e",
      "metadata": {
        "id": "f059012e"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import requests\n",
        "import re\n",
        "import openai"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "tgbH6j3Lp-_x",
      "metadata": {
        "id": "tgbH6j3Lp-_x"
      },
      "source": [
        "##  Download & Clean the Content\n",
        "\n",
        "We are going to download the content using the python package `requests` and perform minor cleanup by removing several newlines. Even minimal cleanup should be good enough to obtain good results with the model."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "da5655ef",
      "metadata": {},
      "source": [
        "### **Downloading the Document**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "id": "IcIybYoE35ro",
      "metadata": {
        "id": "IcIybYoE35ro"
      },
      "outputs": [],
      "source": [
        "url = \"https://raw.githubusercontent.com/hwchase17/chat-your-data/master/state_of_the_union.txt\"\n",
        "content = requests.get(url).content\n",
        "content = str(content, \"utf-8\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "33f8bae9",
      "metadata": {},
      "source": [
        "### **Cleaning Up the Content**\n",
        "Minor cleanup is performed by removing extra newlines:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "id": "xTeisbO_4UI7",
      "metadata": {
        "id": "xTeisbO_4UI7"
      },
      "outputs": [],
      "source": [
        "# Some clean up\n",
        "clean_content = content.replace(\"\\n\\n\", \"\\n\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "id": "8508e1e1",
      "metadata": {},
      "outputs": [],
      "source": [
        "clean_content = clean_content[:5000]  # Use only the first 5000 characters"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "XBfEwDuiqQMT",
      "metadata": {
        "id": "XBfEwDuiqQMT"
      },
      "source": [
        "## Setup your API Key\n",
        "\n",
        "In order to use the Fireworks AI function calling model, you must first obtain Fireworks API Keys. If you don't already have one, you can one by following the instructions [here](https://readme.fireworks.ai/docs/quickstart)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "id": "ZlTFlhtB5baq",
      "metadata": {
        "id": "ZlTFlhtB5baq"
      },
      "outputs": [],
      "source": [
        "client = openai.OpenAI(\n",
        "    base_url = \"https://api.fireworks.ai/inference/v1\",\n",
        "    api_key = \"YOUR_FW_API_KEY\",\n",
        ")\n",
        "model_name = \"accounts/fireworks/models/firefunction-v2\""
      ]
    },
    {
      "cell_type": "markdown",
      "id": "JoHfdVFlqbjN",
      "metadata": {
        "id": "JoHfdVFlqbjN"
      },
      "source": [
        "## Define the Structure\n",
        "\n",
        "Let's define the strucutre in which we want our model to responsd. The JSON structure for function calling follows the conventions of [JSON Schema](https://json-schema.org/). Here we define a structure with `answer` and `citations` field."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "id": "Zj-9l4m283b4",
      "metadata": {
        "id": "Zj-9l4m283b4"
      },
      "outputs": [],
      "source": [
        "tools = [\n",
        "    {\n",
        "        \"type\": \"function\",\n",
        "        \"function\": {\n",
        "            \"name\": \"get_answer_with_sources\",\n",
        "            \"description\": \"Answer questions from the user while quoting sources.\",\n",
        "            \"parameters\": {\n",
        "                \"type\": \"object\",\n",
        "                \"properties\": {\n",
        "                  \"answer\": {\n",
        "                      \"type\": \"string\",\n",
        "                      \"description\": \"Answer to the question that was asked.\"\n",
        "                  },\n",
        "                  \"sources\": {\n",
        "                      \"type\": \"array\",\n",
        "                      \"items\": {\n",
        "                          \"type\": \"string\",\n",
        "                          \"description\": \"Source used to answer the question\"\n",
        "                      }\n",
        "                  }\n",
        "                },\n",
        "                \"required\": [\"answer\", \"sources\"],\n",
        "            }\n",
        "        }\n",
        "    }\n",
        "]\n",
        "tool_choice = {\"type\": \"function\", \"function\": {\"name\":\"get_answer_with_sources\"}}"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4tz7bwV-qset",
      "metadata": {
        "id": "4tz7bwV-qset"
      },
      "source": [
        "## Perform Sanity Test\n",
        "\n",
        "Let's perform a sanity test by querying the speech for some basic information. This would ensure that our model setup is working correctly and the document is being processed correctly."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "id": "LcnDoz7H8jjE",
      "metadata": {
        "id": "LcnDoz7H8jjE"
      },
      "outputs": [],
      "source": [
        "messages = [\n",
        "    {\"role\": \"system\", \"content\": \"You are a helpful assistant with access to a summary of the 2022 State of the Union speech.\"},\n",
        "    {\"role\": \"user\", \"content\": \"What did the president say about Ketanji Brown Jackson?\"}\n",
        "]\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "id": "ENX3Fgcd_JfZ",
      "metadata": {
        "id": "ENX3Fgcd_JfZ"
      },
      "outputs": [],
      "source": [
        "chat_completion = client.chat.completions.create(\n",
        "    model=model_name,\n",
        "    messages=messages,\n",
        "    tools=tools,\n",
        "    tool_choice=tool_choice,\n",
        "    temperature=0.1\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "id": "0WzRJ5PgFAXc",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0WzRJ5PgFAXc",
        "outputId": "ee5a8472-167e-4167-f716-94378d8bd333"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{\n",
            "    \"content\": null,\n",
            "    \"refusal\": null,\n",
            "    \"role\": \"assistant\",\n",
            "    \"function_call\": null,\n",
            "    \"tool_calls\": [\n",
            "        {\n",
            "            \"id\": \"call_kIpcwu7koMRkUtkhVvxuuQQu\",\n",
            "            \"function\": {\n",
            "                \"arguments\": \"{\\\"answer\\\": \\\"The President mentioned Ketanji Brown Jackson as the first Black woman to serve on the Supreme Court.\\\", \\\"sources\\\": [\\\"2022 State of the Union speech\\\"]}\",\n",
            "                \"name\": \"get_answer_with_sources\"\n",
            "            },\n",
            "            \"type\": \"function\",\n",
            "            \"index\": 0\n",
            "        }\n",
            "    ]\n",
            "}\n"
          ]
        }
      ],
      "source": [
        "print(chat_completion.choices[0].message.model_dump_json(indent=4))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "id": "bF-o87oxD05g",
      "metadata": {
        "id": "bF-o87oxD05g"
      },
      "outputs": [],
      "source": [
        "agent_response = chat_completion.choices[0].message\n",
        "\n",
        "messages.append(\n",
        "    {\n",
        "        \"role\": agent_response.role,\n",
        "        \"content\": \"\",\n",
        "        \"tool_calls\": [\n",
        "            tool_call.model_dump()\n",
        "            for tool_call in agent_response.tool_calls\n",
        "        ]\n",
        "    }\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "oC2FSQjDAyL8",
      "metadata": {
        "id": "oC2FSQjDAyL8"
      },
      "source": [
        "## Using Function Calling in Conversation\n",
        "\n",
        "Our model currently support multi-turn conversation when using function calling. You can reference previous completions generated by the model to ask more clarifying questions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "id": "wYGPiSXfAysM",
      "metadata": {
        "id": "wYGPiSXfAysM"
      },
      "outputs": [],
      "source": [
        "messages.append(\n",
        "    {\n",
        "        \"role\": \"user\",\n",
        "        \"content\": \"What did he say about her predecessor?\"\n",
        "    }\n",
        ")\n",
        "next_chat_completion = client.chat.completions.create(\n",
        "    model=model_name,\n",
        "    messages=messages,\n",
        "    tools=tools,\n",
        "    tool_choice=tool_choice,\n",
        "    temperature=0.1\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "id": "UkWhQ4hPFMc_",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UkWhQ4hPFMc_",
        "outputId": "dcce465f-8d89-4937-f17c-4906dc142dcd"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{\n",
            "    \"content\": null,\n",
            "    \"refusal\": null,\n",
            "    \"role\": \"assistant\",\n",
            "    \"function_call\": null,\n",
            "    \"tool_calls\": [\n",
            "        {\n",
            "            \"id\": \"call_tdAos9CIlEKaq1ym4y6WvlL0\",\n",
            "            \"function\": {\n",
            "                \"arguments\": \"{\\\"answer\\\": \\\"The President mentioned Justice Stephen Breyer, who retired from the Supreme Court, making way for Ketanji Brown Jackson to take his seat.\\\", \\\"sources\\\": [\\\"2022 State of the Union speech\\\"]}\",\n",
            "                \"name\": \"get_answer_with_sources\"\n",
            "            },\n",
            "            \"type\": \"function\",\n",
            "            \"index\": 0\n",
            "        }\n",
            "    ]\n",
            "}\n"
          ]
        }
      ],
      "source": [
        "print(next_chat_completion.choices[0].message.model_dump_json(indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "EU_U5Bm9GJNa",
      "metadata": {
        "id": "EU_U5Bm9GJNa"
      },
      "source": [
        "## Modifying the output format to more specific one\n",
        "\n",
        "During the conversation, some questions might need a more flexible response format. We have flexibility to change that during the conversation.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "id": "JxsGWpUcIGan",
      "metadata": {
        "id": "JxsGWpUcIGan"
      },
      "outputs": [],
      "source": [
        "tools = [\n",
        "    {\n",
        "        \"type\": \"function\",\n",
        "        \"function\": {\n",
        "            \"name\": \"get_answer_with_countries\",\n",
        "            \"description\": \"Answer questions from the user while quoting sources.\",\n",
        "            \"parameters\": {\n",
        "                \"type\": \"object\",\n",
        "                \"properties\": {\n",
        "                  \"answer\": {\n",
        "                      \"type\": \"string\",\n",
        "                      \"description\": \"Answer to the question that was asked.\"\n",
        "                  },\n",
        "                  \"countries\": {\n",
        "                      \"type\": \"array\",\n",
        "                      \"items\": {\n",
        "                          \"type\": \"string\",\n",
        "                      },\n",
        "                      \"description\": \"countries mentioned in the sources\"\n",
        "                  }\n",
        "                },\n",
        "                \"required\": [\"answer\", \"countries\"],\n",
        "            }\n",
        "        }\n",
        "    }\n",
        "]\n",
        "tool_choice = {\"type\": \"function\", \"function\": {\"name\":\"get_answer_with_countries\"}}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "id": "g-CYqzXIIUIl",
      "metadata": {
        "id": "g-CYqzXIIUIl"
      },
      "outputs": [],
      "source": [
        "agent_response = next_chat_completion.choices[0].message\n",
        "\n",
        "messages.append(\n",
        "    {\n",
        "        \"role\": agent_response.role,\n",
        "        \"content\": \"\",\n",
        "        \"tool_calls\": [\n",
        "            tool_call.model_dump()\n",
        "            for tool_call in agent_response.tool_calls\n",
        "        ]\n",
        "    }\n",
        ")\n",
        "\n",
        "messages.append(\n",
        "    {\n",
        "        \"role\": \"user\",\n",
        "        \"content\": \"What did he say about human traffickers?\"\n",
        "    }\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "id": "RylJZ8BiIewx",
      "metadata": {
        "id": "RylJZ8BiIewx"
      },
      "outputs": [],
      "source": [
        "chat_completion = client.chat.completions.create(\n",
        "    model=model_name,\n",
        "    messages=messages,\n",
        "    tools=tools,\n",
        "    tool_choice=tool_choice,\n",
        "    temperature=0.1\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "id": "qi_gNf-qI-CG",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qi_gNf-qI-CG",
        "outputId": "d3018c86-21dd-4b40-9a38-7f14bbec05cd"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{\n",
            "    \"content\": null,\n",
            "    \"refusal\": null,\n",
            "    \"role\": \"assistant\",\n",
            "    \"function_call\": null,\n",
            "    \"tool_calls\": [\n",
            "        {\n",
            "            \"id\": \"call_Meyq8T6lEcoLTHmteAJvhRp2\",\n",
            "            \"function\": {\n",
            "                \"arguments\": \"{\\\"answer\\\": \\\"The President mentioned that human traffickers are being brought to justice and that the US is working with other countries to combat human trafficking.\\\", \\\"countries\\\": [\\\"United States\\\"]}\",\n",
            "                \"name\": \"get_answer_with_countries\"\n",
            "            },\n",
            "            \"type\": \"function\",\n",
            "            \"index\": 0\n",
            "        }\n",
            "    ]\n",
            "}\n"
          ]
        }
      ],
      "source": [
        "print(chat_completion.choices[0].message.model_dump_json(indent=4))"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.12.6"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}