samples/extraction/text-based/document-intelligence-phi.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Extraction - Azure AI Document Intelligence + Phi-3.5 MoE\n",
    "\n",
    "This sample demonstrates how to extract structured data from any document using Azure AI Document Intelligence and small language models, such as Microsoft's Phi-3.5 MoE.\n",
    "\n",
    "![Data Extraction](../../../images/extraction-document-intelligence-phi.png)\n",
    "\n",
    "This is achieved by the following process:\n",
    "\n",
    "- Analyze a document using Azure AI Document Intelligence's `prebuilt-layout` model to extract the structure as Markdown.\n",
    "- Construct a system prompt that defines the instruction for extracting structured data from documents.\n",
    "- Construct a user prompt that includes specific extraction instruction for the type of document, the expected JSON schema, and the Markdown content of the document\n",
    "- Use the chat completions API with the Phi-3.5 MoE model to generate a structured output from the content.\n",
    "\n",
    "## Objectives\n",
    "\n",
    "By the end of this sample, you will have learned how to:\n",
    "\n",
    "- Convert a document to Markdown format using Azure AI Document Intelligence.\n",
    "- Use prompt engineering techniques to instruct Phi-3.5 MoE to extract structured data from a type of document using example JSON structures.\n",
    "- Use the analysis result from Azure AI Document Intelligence to determine the confidence of the extracted structured output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import modules\n",
    "\n",
    "This sample takes advantage of the following Python dependencies:\n",
    "\n",
    "- **azure-ai-documentintelligence** to interface with the Azure AI Document Intelligence API for analyzing documents.\n",
    "- **openai** to use the default OpenAI client implementation to interface with a deployed Phi-3.5 MoE serverless endpoint in Azure AI Studio. _Note: The serverless endpoint for Phi-3.5 MoE uses the same API schema as the OpenAI API._\n",
    "- **azure-identity** to securely authenticate with deployed Azure Services using Microsoft Entra ID credentials.\n",
    "\n",
    "The following local modules are also used:\n",
    "\n",
    "- **modules.app_settings** to access environment variables from the `.env` file.\n",
    "- **modules.comparison** to compare the output of the extraction process with expected results.\n",
    "- **modules.document_intelligence_confidence** to evaluate the confidence of the extraction process based on the extracted structured output and the analysis result from Azure AI Document Intelligence.\n",
    "- **modules.document_processing_result** to store the results of the extraction process as a file.\n",
    "- **modules.invoice** to provide the expected structured output JSON schema for invoice documents.\n",
    "- **modules.stopwatch** to measure the end-to-end execution time for the classification process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append('../../') # Import local modules\n",
    "\n",
    "from IPython.display import display, Markdown\n",
    "import os\n",
    "import pandas as pd\n",
    "import json\n",
    "from dotenv import dotenv_values\n",
    "from azure.ai.documentintelligence import DocumentIntelligenceClient\n",
    "from azure.ai.documentintelligence.models import AnalyzeResult, ContentFormat\n",
    "from openai import OpenAI\n",
    "from azure.identity import DefaultAzureCredential\n",
    "\n",
    "from modules.app_settings import AppSettings\n",
    "from modules.comparison import extraction_comparison\n",
    "from modules.document_intelligence_confidence import evaluate_confidence\n",
    "from modules.document_processing_result import DataExtractionResult\n",
    "from modules.invoice import Invoice, InvoiceProduct, InvoiceSignature, InvoiceEvaluator\n",
    "from modules.stopwatch import Stopwatch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure the Azure services\n",
    "\n",
    "To use Azure AI Document Intelligence and a serverless model endpoint deployment, their SDKs are used to create client instances using a deployed endpoint and authentication credentials.\n",
    "\n",
    "For this sample, the credentials of the Azure CLI are used to authenticate with Azure AI Document Intelligence. The OpenAI client for the Phi model uses an API key to authenticate with the serverless endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the working directory to the root of the repo\n",
    "working_dir = os.path.abspath('../../../')\n",
    "settings = AppSettings(dotenv_values(f\"{working_dir}/.env\"))\n",
    "\n",
    "# Configure the default credential for accessing Azure services using Azure CLI credentials\n",
    "credential = DefaultAzureCredential(\n",
    "    exclude_workload_identity_credential=True,\n",
    "    exclude_developer_cli_credential=True,\n",
    "    exclude_environment_credential=True,\n",
    "    exclude_managed_identity_credential=True,\n",
    "    exclude_powershell_credential=True,\n",
    "    exclude_shared_token_cache_credential=True,\n",
    "    exclude_interactive_browser_credential=True\n",
    ")\n",
    "\n",
    "# The serverless endpoint for Phi models uses the same API as OpenAI so we can use the OpenAI client\n",
    "openai_client = OpenAI(\n",
    "    base_url=settings.phi3_endpoint,\n",
    "    api_key=settings.phi3_primary_key,\n",
    ")\n",
    "\n",
    "document_intelligence_client = DocumentIntelligenceClient(\n",
    "    endpoint=settings.ai_services_endpoint,\n",
    "    credential=credential\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Establish the expected output\n",
    "\n",
    "To compare the accuracy of the extraction process, the expected output of the extraction process has been defined in the following code block based on the details of the [Invoice](../../assets/Invoice.pdf).\n",
    "\n",
    "The expected output has been defined by a human evaluating the document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "pdf_path = f\"{working_dir}/samples/assets/\"\n",
    "pdf_file_name = \"Invoice.pdf\"\n",
    "fname = f\"{pdf_path}{pdf_file_name}\"\n",
    "\n",
    "expected = Invoice(\n",
    "    invoice_number='3847193',\n",
    "    purchase_order_number='15931',\n",
    "    customer_name='Sharp Consulting',\n",
    "    customer_address='73 Regal Way, Leeds, LS1 5AB, UK',\n",
    "    delivery_date='2024-05-16',\n",
    "    payable_by='2024-05-24',\n",
    "    products=[\n",
    "        InvoiceProduct(\n",
    "            id='MA197',\n",
    "            description='STRETCHWRAP ROLL',\n",
    "            unit_price=16.62,\n",
    "            quantity=5,\n",
    "            total=83.10,\n",
    "            reason=None\n",
    "        ),\n",
    "        InvoiceProduct(\n",
    "            id='ST4086',\n",
    "            description='BALLPOINT PEN MED.',\n",
    "            unit_price=2.49,\n",
    "            quantity=10,\n",
    "            total=24.90,\n",
    "            reason=None\n",
    "        ),\n",
    "        InvoiceProduct(\n",
    "            id='JF9912413BF',\n",
    "            description='BUBBLE FILM ROLL CL.',\n",
    "            unit_price=15.46,\n",
    "            quantity=12,\n",
    "            total=185.52,\n",
    "            reason=None\n",
    "        ),\n",
    "    ],\n",
    "    returns=[\n",
    "        InvoiceProduct(\n",
    "            id='MA145',\n",
    "            description='POSTAL TUBE BROWN',\n",
    "            unit_price=None,\n",
    "            quantity=1,\n",
    "            total=None,\n",
    "            reason='This item was provided in previous order as a replacement'\n",
    "        ),\n",
    "        InvoiceProduct(\n",
    "            id='JF7902',\n",
    "            description='MAILBOX 25PK',\n",
    "            unit_price=None,\n",
    "            quantity=1,\n",
    "            total=None,\n",
    "            reason='Not required'\n",
    "        ),\n",
    "    ],\n",
    "    total_product_quantity=27,\n",
    "    total_product_price=293.52,\n",
    "    product_signatures=[\n",
    "        InvoiceSignature(\n",
    "            type='Customer',\n",
    "            name='Sarah H',\n",
    "            is_signed=True\n",
    "        ),\n",
    "        InvoiceSignature(\n",
    "            type='Driver',\n",
    "            name='James T',\n",
    "            is_signed=True\n",
    "        )\n",
    "    ],\n",
    "    returns_signatures=[\n",
    "        InvoiceSignature(\n",
    "            type='Customer',\n",
    "            name='Sarah H',\n",
    "            is_signed=True\n",
    "        ),\n",
    "        InvoiceSignature(\n",
    "            type='Driver',\n",
    "            name='James T',\n",
    "            is_signed=True\n",
    "        )\n",
    "    ]   \n",
    ")\n",
    "\n",
    "invoice_evaluator = InvoiceEvaluator(expected)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extract data from the document\n",
    "\n",
    "The following code block executes the data extraction process using Azure AI Document Intelligence and Phi-3.5 MoE.\n",
    "\n",
    "It performs the following steps:\n",
    "\n",
    "1. Get the document bytes from the provided file path. _Note: In this example, we are processing a local document, however, you can use any document storage location of your choice, such as Azure Blob Storage._\n",
    "2. Use Azure AI Document Intelligence to analyze the structure of the document and convert it to Markdown format using the pre-built layout model.\n",
    "3. Using a Phi-3.5 MoE Serverless Endpoint model deployment in Azure AI Studio and prompt engineering techniques, extract a structured data transfer object (DTO) from the content of the Markdown."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "with Stopwatch() as di_stopwatch:\n",
    "    with open(fname, \"rb\") as f:\n",
    "        poller = document_intelligence_client.begin_analyze_document(\n",
    "            \"prebuilt-layout\",\n",
    "            analyze_request=f,\n",
    "            output_content_format=ContentFormat.MARKDOWN,\n",
    "            content_type=\"application/pdf\"\n",
    "        )\n",
    "        \n",
    "    result: AnalyzeResult = poller.result()\n",
    "\n",
    "markdown = result.content\n",
    "\n",
    "with Stopwatch() as oai_stopwatch:\n",
    "    completion = openai_client.chat.completions.create(\n",
    "        model='phi-3',\n",
    "        messages=[\n",
    "            {\n",
    "                \"role\": \"system\",\n",
    "                \"content\": \"You are an AI assistant that extracts data from documents and returns them as structured JSON objects. Do not return as a JSON code block.\",\n",
    "            },\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": f\"\"\"Extract the data from this invoice. \n",
    "                - If a value is not present, provide null.\n",
    "                - Use the following structure: {json.dumps(Invoice.example().to_dict())}\"\"\",\n",
    "            },\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": markdown,\n",
    "            }\n",
    "        ],\n",
    "        max_tokens=4096,\n",
    "        temperature=0.1,\n",
    "        top_p=0.1\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize the outputs\n",
    "\n",
    "To provide context for the execution of the code, the following code blocks visualize the outputs of the data extraction process.\n",
    "\n",
    "This includes:\n",
    "\n",
    "- The Markdown representation of the document structure as determined by Azure AI Document Intelligence.\n",
    "- The accuracy of the structured data extraction comparing the expected output with the output generated by Phi-3.5 MoE.\n",
    "- The confidence score of the structured data extraction by comparing against the Azure AI Document Intelligence analysis.\n",
    "- The execution time of the end-to-end process.\n",
    "- The total number of tokens consumed by the Phi-3.5 MoE model.\n",
    "- The side-by-side comparison of the expected output and the output generated by Phi-3.5 MoE.\n",
    "\n",
    "### Understanding Accuracy vs Confidence\n",
    "\n",
    "When using AI to extract structured data, both confidence and accuracy are essential for different but complementary reasons.\n",
    "\n",
    "- **Accuracy** measures how close the AI model's output is to a ground truth or expected output. It reflects how well the model's predictions align with reality.\n",
    "  - Accuracy ensures consistency in the extraction process, which is crucial for downstream tasks using the data.\n",
    "- **Confidence** represents the AI model's internal assessment of how certain it is about its predictions.\n",
    "  - Confidence indicates that the model is certain about its predictions, which can be a useful indicator for human reviewers to step in for manual verification.\n",
    "\n",
    "High accuracy and high confidence are ideal, but in practice, there is often a trade-off between the two. While accuracy cannot always be self-assessed, confidence scores can and should be used to prioritize manual verification of low-confidence predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "NEXGEN\n",
       "\n",
       "Innovation drives progress\n",
       "\n",
       "\n",
       "# INVOICE\n",
       "\n",
       "\n",
       "<table>\n",
       "<tr>\n",
       "<th>Customer:</th>\n",
       "<th colspan=\"2\">Sharp Consulting</th>\n",
       "<th colspan=\"2\">Address:</th>\n",
       "<th colspan=\"3\">73 Regal Way, Leeds, LS1 5AB, UK</th>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Delv. Date:</td>\n",
       "<td>5/16/2024</td>\n",
       "<td colspan=\"2\">Invoice Number:</td>\n",
       "<td colspan=\"2\">3847193</td>\n",
       "<td>Purchase Order:</td>\n",
       "<td>15931</td>\n",
       "</tr>\n",
       "</table>\n",
       "\n",
       "\n",
       "<table>\n",
       "<tr>\n",
       "<th>Item Code</th>\n",
       "<th>Item Desc.</th>\n",
       "<th>Unit Price</th>\n",
       "<th>Quantity</th>\n",
       "<th>Total</th>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>MA197</td>\n",
       "<td>STRETCHWRAP ROLL</td>\n",
       "<td>16.62</td>\n",
       "<td>5</td>\n",
       "<td>83.10</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>ST4086</td>\n",
       "<td>BALLPOINT PEN MED.</td>\n",
       "<td>2.49</td>\n",
       "<td>10</td>\n",
       "<td>24.90</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>JF9912413BF</td>\n",
       "<td>BUBBLE FILM ROLL CL.</td>\n",
       "<td>15.46</td>\n",
       "<td>12</td>\n",
       "<td>185.52</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "</tr>\n",
       "</table>\n",
       "\n",
       "\n",
       "NOTES\n",
       "\n",
       "\n",
       "<table>\n",
       "<tr>\n",
       "<th>Total Pcs .:</th>\n",
       "<th>27</th>\n",
       "<th>Total Price:</th>\n",
       "<th>293.52</th>\n",
       "</tr>\n",
       "<tr>\n",
       "<td colspan=\"4\">Payable on or before 5/24/2024</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Cust. Sig.</td>\n",
       "<td>Sfr</td>\n",
       "<td>Drivr. Sig.</td>\n",
       "<td></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Cust. Name:</td>\n",
       "<td>Sarah H</td>\n",
       "<td>Drivr. Name:</td>\n",
       "<td>James T</td>\n",
       "</tr>\n",
       "</table>\n",
       "\n",
       "\n",
       "<!-- PageBreak -->\n",
       "\n",
       "\n",
       "<table>\n",
       "<tr>\n",
       "<td colspan=\"6\">NEXGEN Innovation drives progress</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td></td>\n",
       "<td colspan=\"5\">RETURNS</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Customer:</td>\n",
       "<td>Sharp Consulting</td>\n",
       "<td colspan=\"2\">Address:</td>\n",
       "<td colspan=\"2\">73 Regal Way, Leeds, LS1 5AB, UK</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Coll. Date:</td>\n",
       "<td>5/16/2024</td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td colspan=\"2\"></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Item Code</td>\n",
       "<td>Item Desc.</td>\n",
       "<td>Pcs.</td>\n",
       "<td colspan=\"3\">Reason</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>MA145</td>\n",
       "<td>POSTAL TUBE BROWN</td>\n",
       "<td>1</td>\n",
       "<td colspan=\"3\">This item was provided in previous order</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>JF7902</td>\n",
       "<td>MAILBOX 25PK</td>\n",
       "<td>1 7</td>\n",
       "<td colspan=\"3\">as replacement</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td colspan=\"3\">Not required</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td colspan=\"3\"></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td colspan=\"6\"></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>NOTES</td>\n",
       "<td></td>\n",
       "<td colspan=\"4\"></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td colspan=\"4\"></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td></td>\n",
       "<td colspan=\"5\"></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td></td>\n",
       "<td colspan=\"5\"></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td colspan=\"2\"></td>\n",
       "<td></td>\n",
       "<td colspan=\"3\"></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td colspan=\"4\"></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Total Pcs .:</td>\n",
       "<td>2</td>\n",
       "<td></td>\n",
       "<td colspan=\"3\"></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td colspan=\"6\"></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Cust. Sig.</td>\n",
       "<td>8</td>\n",
       "<td></td>\n",
       "<td colspan=\"2\">Drivr. Sig.</td>\n",
       "<td>&amp;</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Cust. Name:</td>\n",
       "<td>Sarah H</td>\n",
       "<td></td>\n",
       "<td colspan=\"2\">Drivr. Name:</td>\n",
       "<td>James T</td>\n",
       "</tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Displays the output of the Azure AI Document Intelligence pre-built layout analysis in Markdown format.\n",
    "display(Markdown(markdown))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<table>\n",
       "<thead>\n",
       "<tr><th>Accuracy  </th><th>Confidence  </th><th>Execution Time  </th><th>Document Intelligence Execution Time  </th><th>Phi-3.5 Execution Time  </th><th style=\"text-align: right;\">  Prompt Tokens</th><th style=\"text-align: right;\">  Completion Tokens</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td>98.61%    </td><td>96.13%      </td><td>38.43 seconds   </td><td>4.79 seconds                          </td><td>33.64 seconds           </td><td style=\"text-align: right;\">           2019</td><td style=\"text-align: right;\">                684</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<table>\n",
       "<thead>\n",
       "<tr><th>Field                         </th><th>Expected                                                 </th><th>Extracted                                                               </th><th style=\"text-align: right;\">  Confidence</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td>invoice_number                </td><td>3847193                                                  </td><td><span style='color: green'>3847193</span>                               </td><td style=\"text-align: right;\">       0.998</td></tr>\n",
       "<tr><td>purchase_order_number         </td><td>15931                                                    </td><td><span style='color: green'>15931</span>                                 </td><td style=\"text-align: right;\">       0.995</td></tr>\n",
       "<tr><td>customer_name                 </td><td>Sharp Consulting                                         </td><td><span style='color: green'>Sharp Consulting</span>                      </td><td style=\"text-align: right;\">       0.995</td></tr>\n",
       "<tr><td>customer_address              </td><td>73 Regal Way, Leeds, LS1 5AB, UK                         </td><td><span style='color: green'>73 Regal Way, Leeds, LS1 5AB, UK</span>      </td><td style=\"text-align: right;\">       0.987</td></tr>\n",
       "<tr><td>delivery_date                 </td><td>2024-05-16                                               </td><td><span style='color: green'>2024-05-16</span>                            </td><td style=\"text-align: right;\">       0    </td></tr>\n",
       "<tr><td>payable_by                    </td><td>2024-05-24                                               </td><td><span style='color: green'>2024-05-24</span>                            </td><td style=\"text-align: right;\">       0    </td></tr>\n",
       "<tr><td>products_0_id                 </td><td>MA197                                                    </td><td><span style='color: green'>MA197</span>                                 </td><td style=\"text-align: right;\">       0.997</td></tr>\n",
       "<tr><td>products_0_description        </td><td>STRETCHWRAP ROLL                                         </td><td><span style='color: green'>STRETCHWRAP ROLL</span>                      </td><td style=\"text-align: right;\">       0.989</td></tr>\n",
       "<tr><td>products_0_unit_price         </td><td>16.62                                                    </td><td><span style='color: green'>16.62</span>                                 </td><td style=\"text-align: right;\">       0.997</td></tr>\n",
       "<tr><td>products_0_quantity           </td><td>5                                                        </td><td><span style='color: green'>5</span>                                     </td><td style=\"text-align: right;\">       0.996</td></tr>\n",
       "<tr><td>products_0_total              </td><td>83.1                                                     </td><td><span style='color: green'>83.1</span>                                  </td><td style=\"text-align: right;\">       0.998</td></tr>\n",
       "<tr><td>products_0_reason             </td><td>                                                         </td><td><span style='color: red'></span>                                        </td><td style=\"text-align: right;\">       0    </td></tr>\n",
       "<tr><td>products_1_id                 </td><td>ST4086                                                   </td><td><span style='color: green'>ST4086</span>                                </td><td style=\"text-align: right;\">       0.997</td></tr>\n",
       "<tr><td>products_1_description        </td><td>BALLPOINT PEN MED.                                       </td><td><span style='color: green'>BALLPOINT PEN MED.</span>                    </td><td style=\"text-align: right;\">       0.992</td></tr>\n",
       "<tr><td>products_1_unit_price         </td><td>2.49                                                     </td><td><span style='color: green'>2.49</span>                                  </td><td style=\"text-align: right;\">       0.992</td></tr>\n",
       "<tr><td>products_1_quantity           </td><td>10                                                       </td><td><span style='color: green'>10</span>                                    </td><td style=\"text-align: right;\">       0.998</td></tr>\n",
       "<tr><td>products_1_total              </td><td>24.9                                                     </td><td><span style='color: green'>24.9</span>                                  </td><td style=\"text-align: right;\">       0.996</td></tr>\n",
       "<tr><td>products_1_reason             </td><td>                                                         </td><td><span style='color: red'></span>                                        </td><td style=\"text-align: right;\">       0    </td></tr>\n",
       "<tr><td>products_2_id                 </td><td>JF9912413BF                                              </td><td><span style='color: green'>JF9912413BF</span>                           </td><td style=\"text-align: right;\">       0.995</td></tr>\n",
       "<tr><td>products_2_description        </td><td>BUBBLE FILM ROLL CL.                                     </td><td><span style='color: green'>BUBBLE FILM ROLL CL.</span>                  </td><td style=\"text-align: right;\">       0.986</td></tr>\n",
       "<tr><td>products_2_unit_price         </td><td>15.46                                                    </td><td><span style='color: green'>15.46</span>                                 </td><td style=\"text-align: right;\">       0.998</td></tr>\n",
       "<tr><td>products_2_quantity           </td><td>12                                                       </td><td><span style='color: green'>12</span>                                    </td><td style=\"text-align: right;\">       0.998</td></tr>\n",
       "<tr><td>products_2_total              </td><td>185.52                                                   </td><td><span style='color: green'>185.52</span>                                </td><td style=\"text-align: right;\">       0.996</td></tr>\n",
       "<tr><td>products_2_reason             </td><td>                                                         </td><td><span style='color: red'></span>                                        </td><td style=\"text-align: right;\">       0    </td></tr>\n",
       "<tr><td>returns_0_id                  </td><td>MA145                                                    </td><td><span style='color: green'>MA145</span>                                 </td><td style=\"text-align: right;\">       0.996</td></tr>\n",
       "<tr><td>returns_0_description         </td><td>POSTAL TUBE BROWN                                        </td><td><span style='color: green'>POSTAL TUBE BROWN</span>                     </td><td style=\"text-align: right;\">       0.992</td></tr>\n",
       "<tr><td>returns_0_unit_price          </td><td>                                                         </td><td><span style='color: green'>None</span>                                  </td><td style=\"text-align: right;\">       0    </td></tr>\n",
       "<tr><td>returns_0_quantity            </td><td>1                                                        </td><td><span style='color: green'>1</span>                                     </td><td style=\"text-align: right;\">       0.995</td></tr>\n",
       "<tr><td>returns_0_total               </td><td>                                                         </td><td><span style='color: green'>None</span>                                  </td><td style=\"text-align: right;\">       0    </td></tr>\n",
       "<tr><td>returns_0_reason              </td><td>This item was provided in previous order as a replacement</td><td><span style='color: red'>This item was provided in previous order</span></td><td style=\"text-align: right;\">       0.662</td></tr>\n",
       "<tr><td>returns_1_id                  </td><td>JF7902                                                   </td><td><span style='color: green'>JF7902</span>                                </td><td style=\"text-align: right;\">       0.995</td></tr>\n",
       "<tr><td>returns_1_description         </td><td>MAILBOX 25PK                                             </td><td><span style='color: green'>MAILBOX 25PK</span>                          </td><td style=\"text-align: right;\">       0.992</td></tr>\n",
       "<tr><td>returns_1_unit_price          </td><td>                                                         </td><td><span style='color: green'>None</span>                                  </td><td style=\"text-align: right;\">       0    </td></tr>\n",
       "<tr><td>returns_1_quantity            </td><td>1                                                        </td><td><span style='color: green'>1</span>                                     </td><td style=\"text-align: right;\">       0.995</td></tr>\n",
       "<tr><td>returns_1_total               </td><td>                                                         </td><td><span style='color: green'>None</span>                                  </td><td style=\"text-align: right;\">       0    </td></tr>\n",
       "<tr><td>returns_1_reason              </td><td>Not required                                             </td><td><span style='color: red'>as replacement</span>                          </td><td style=\"text-align: right;\">       0.834</td></tr>\n",
       "<tr><td>total_product_quantity        </td><td>27.0                                                     </td><td><span style='color: green'>27.0</span>                                  </td><td style=\"text-align: right;\">       0    </td></tr>\n",
       "<tr><td>total_product_price           </td><td>293.52                                                   </td><td><span style='color: green'>293.52</span>                                </td><td style=\"text-align: right;\">       0.994</td></tr>\n",
       "<tr><td>product_signatures_0_type     </td><td>Customer                                                 </td><td><span style='color: green'>Customer</span>                              </td><td style=\"text-align: right;\">       0.998</td></tr>\n",
       "<tr><td>product_signatures_0_name     </td><td>Sarah H                                                  </td><td><span style='color: green'>Sarah H</span>                               </td><td style=\"text-align: right;\">       0.683</td></tr>\n",
       "<tr><td>product_signatures_0_is_signed</td><td>True                                                     </td><td><span style='color: green'>True</span>                                  </td><td style=\"text-align: right;\">       0    </td></tr>\n",
       "<tr><td>product_signatures_1_type     </td><td>Driver                                                   </td><td><span style='color: green'>Driver</span>                                </td><td style=\"text-align: right;\">       0    </td></tr>\n",
       "<tr><td>product_signatures_1_name     </td><td>James T                                                  </td><td><span style='color: green'>James T</span>                               </td><td style=\"text-align: right;\">       0.983</td></tr>\n",
       "<tr><td>product_signatures_1_is_signed</td><td>True                                                     </td><td><span style='color: green'>True</span>                                  </td><td style=\"text-align: right;\">       0    </td></tr>\n",
       "<tr><td>returns_signatures_0_type     </td><td>Customer                                                 </td><td><span style='color: green'>Customer</span>                              </td><td style=\"text-align: right;\">       0.998</td></tr>\n",
       "<tr><td>returns_signatures_0_name     </td><td>Sarah H                                                  </td><td><span style='color: green'>Sarah H</span>                               </td><td style=\"text-align: right;\">       0.683</td></tr>\n",
       "<tr><td>returns_signatures_0_is_signed</td><td>True                                                     </td><td><span style='color: green'>True</span>                                  </td><td style=\"text-align: right;\">       0    </td></tr>\n",
       "<tr><td>returns_signatures_1_type     </td><td>Driver                                                   </td><td><span style='color: green'>Driver</span>                                </td><td style=\"text-align: right;\">       0    </td></tr>\n",
       "<tr><td>returns_signatures_1_name     </td><td>James T                                                  </td><td><span style='color: green'>James T</span>                               </td><td style=\"text-align: right;\">       0.983</td></tr>\n",
       "<tr><td>returns_signatures_1_is_signed</td><td>True                                                     </td><td><span style='color: green'>True</span>                                  </td><td style=\"text-align: right;\">       0    </td></tr>\n",
       "</tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Gets the JSON response from the completion and converts it to an Invoice object.\n",
    "response_json = completion.choices[0].message.content\n",
    "invoice = Invoice.from_json(response_json)\n",
    "\n",
    "# Determines the accuracy of the extracted data against the expected values.\n",
    "accuracy = invoice_evaluator.evaluate(invoice)\n",
    "\n",
    "# Determines the confidence of the extracted data against the expected values using the result of the Azure AI Document Intelligence pre-built layout analysis.\n",
    "confidence = evaluate_confidence(invoice.to_dict(), result)\n",
    "\n",
    "# Gets the total execution time of the data extraction process.\n",
    "total_elapsed = di_stopwatch.elapsed + oai_stopwatch.elapsed\n",
    "\n",
    "# Gets the prompt tokens and completion tokens from the completion response.\n",
    "prompt_tokens = completion.usage.prompt_tokens\n",
    "completion_tokens = completion.usage.completion_tokens\n",
    "\n",
    "# Save the output of the data extraction result.\n",
    "extraction_result = DataExtractionResult(invoice.to_dict(), confidence, accuracy, prompt_tokens, completion_tokens, total_elapsed)\n",
    "\n",
    "with open(f\"{working_dir}/samples/extraction/text-based/document-intelligence-phi.{pdf_file_name}.json\", \"w\") as f:\n",
    "    f.write(extraction_result.to_json(indent=4))\n",
    "    \n",
    "# Display the outputs of the data extraction process.\n",
    "df = pd.DataFrame([\n",
    "    {\n",
    "        \"Accuracy\": f\"{accuracy['overall'] * 100:.2f}%\",\n",
    "        \"Confidence\": f\"{confidence['_overall'] * 100:.2f}%\",\n",
    "        \"Execution Time\": f\"{total_elapsed:.2f} seconds\",\n",
    "        \"Document Intelligence Execution Time\": f\"{di_stopwatch.elapsed:.2f} seconds\",\n",
    "        \"Phi-3.5 Execution Time\": f\"{oai_stopwatch.elapsed:.2f} seconds\",\n",
    "        \"Prompt Tokens\": prompt_tokens,\n",
    "        \"Completion Tokens\": completion_tokens\n",
    "    }\n",
    "])\n",
    "\n",
    "display(Markdown(df.to_markdown(index=False, tablefmt='unsafehtml')))\n",
    "display(Markdown(extraction_comparison(expected.to_dict(), invoice.to_dict(), confidence)))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}