tools/evaluation/03_eval_ragas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "60d208cc-206a-4526-a061-489b0af46adc",
   "metadata": {},
   "source": [
    "# Notebook 3: Evaluation with Ragas\n",
    "\n",
    "\n",
    "Leveraging a strong LLM for reference-free evaluation is an upcoming solution that has shown a lot of promise. They correlate better with human judgment than traditional metrics and also require less human annotation. Papers like G-Eval have experimented with this and given promising results but there are certain shortcomings too.\n",
    "\n",
    "LLM prefers their own outputs and when asked to compare between different outputs the relative position of those outputs matters more. LLMs can also have a bias toward a value when asked to score given a range and they also prefer longer responses.\n",
    "\n",
    "[Ragas](https://docs.ragas.io/en/latest/) aims to work around these limitations of using LLMs to evaluate your QA pipelines while also providing actionable metrics using as little annotated data as possible, cheaper, and faster.\n",
    "\n",
    "In this notebook, we will use NVIDIA AI playground's  Llama 70B LLM as a judge and eval model. **NVIDIA AI Playground** on NGC allows developers to experience state of the art LLMs accelerated on NVIDIA DGX Cloud with NVIDIA TensorRT nd Triton Inference Server. Developers get **free credits for 10K requests** to any of the available models. Sign up process is easy. Follow the instructions [here.](../docs/rag/aiplayground.md)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b765c93-e4bc-4918-b389-63b413912e0b",
   "metadata": {},
   "source": [
    "### Step 1: Set NVIDIA AI Playground API key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00e39b77-8e68-458e-88f6-72b50f24e9ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ['NVIDIA_API_KEY'] = \"nvapi-*\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "174ff59e-cd9e-4f42-833b-de4aec02c0e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_nvidia_ai_endpoints import ChatNVIDIA, NVIDIAEmbeddings\n",
    "llm = ChatNVIDIA(\n",
    "    model=\"llama2_70b\",\n",
    "    temperature=0.2,\n",
    "    max_tokens=300,\n",
    ")\n",
    "embeddings = NVIDIAEmbeddings(model=\"nvolveqa_40k\", model_type=\"passage\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f057f67e-a0c0-4d8d-971c-b470a6df6475",
   "metadata": {},
   "source": [
    "### Bring your own LLMs¶\n",
    "Ragas uses langchain under the hood for connecting to LLMs for metrices that require them. This means you can swap out the default LLM (gpt-3.5) with llama2 70B from AI playground."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ab053ae-f7a5-476a-be30-b590d3f4484e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ragas.llms import LangchainLLMWrapper\n",
    "from ragas.embeddings import LangchainEmbeddingsWrapper\n",
    "nvpl_llm = LangchainLLMWrapper(langchain_llm=llm)\n",
    "nvpl_embeddings = LangchainEmbeddingsWrapper(embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc5adab9-cdc1-4ae7-ba83-a74afc734b36",
   "metadata": {},
   "source": [
    "### Step 2: Import Eval Data and Reformat It"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fee3b485-3ace-466c-92fc-2a5347cebaf7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "with open('eval.json', 'r') as file:\n",
    "    json_data = json.load(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40269987-bfba-49c2-bb75-726ce96d8e5a",
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_questions = []\n",
    "eval_answers = []\n",
    "ground_truths = []\n",
    "vdb_contexts = []\n",
    "counter = 0\n",
    "for entry in json_data:\n",
    "    eval_questions.append(entry[\"question\"])\n",
    "    eval_answers.append(entry[\"answer\"])\n",
    "    vdb_contexts.append(entry[\"contexts\"])\n",
    "    ground_truths.append([entry[\"gt_answer\"]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e175a36-c26b-43b9-9879-be4bb5c7048b",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_samples = {\n",
    "    'question': eval_questions,\n",
    "    'answer': eval_answers,\n",
    "    'contexts' : vdb_contexts,\n",
    "    'ground_truths': ground_truths\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11d4c1dd-0e3f-4c3c-afc5-75005a424ba2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ragas import evaluate\n",
    "from datasets import Dataset\n",
    "\n",
    "dataset = Dataset.from_dict(data_samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb8b6495-a66b-4120-a6bf-4c344e6513bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall\n",
    "evaluate(dataset, llm=nvpl_llm, embeddings=nvpl_embeddings, metrics=[faithfulness, answer_relevancy, context_precision, context_recall])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "221e56fb-3794-48b2-a4ce-8900ee633f13",
   "metadata": {},
   "source": [
    "### Step 3: View and Interpret Results\n",
    "\n",
    "A Ragas score is comprised of the following:\n",
    "![ragas](imgs/ragas.png)\n",
    "\n",
    "#### Metrics explained \n",
    "1. **Faithfulness**: measures the factual accuracy of the generated answer with the context provided. This is done in 2 steps. First, given a question and generated answer, Ragas uses an LLM to figure out the statements that the generated answer makes. This gives a list of statements whose validity we have we have to check. In step 2, given the list of statements and the context returned, Ragas uses an LLM to check if the statements provided are supported by the context. The number of correct statements is summed up and divided by the total number of statements in the generated answer to obtain the score for a given example.\n",
    "   \n",
    "2. **Answer Relevancy**: measures how relevant and to the point the answer is to the question. For a given generated answer Ragas uses an LLM to find out the probable questions that the generated answer would be an answer to and computes similarity to the actual question asked.\n",
    "   \n",
    "3. **Context Precision**: measures the precision of the retrieved context in providing relevant information for generating answer. Given a question, answer and retrieved context, Ragas calls LLM to check sentences from the ground truth answer against a retrieved context. It is the ratio between the relevant sentences from retrieved context and the total sentence from ground truth answer.\n",
    "\n",
    "4. **Context Recall**: measures the ability of the retriever to retrieve all the necessary information needed to answer the question. Ragas calculates this by using the provided ground_truth answer and using an LLM to check if each statement from it can be found in the retrieved context. If it is not found that means the retriever was not able to retrieve the information needed to support that statement.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}