Model Selection Guide minor formatting fixes (#1824)

Author: kashyapm-tribe

examples/partners/model_selection_guide/model_selection_guide.ipynb

@@ -101,7 +101,7 @@
     "\n",
     "## 1\\. Scenario Snapshot\n",
     "\n",
-    "* **Corpus:** The primary document is the [Trademark Trial and Appeal Board Manual of Procedure (TBMP, 2019 version)](https://www.uspto.gov/sites/default/files/documents/tbmp-2019.pdf). This manual contains detailed procedural rules and guidelines, coming to 1194 pages total.  \n",
+    "* **Corpus:** The primary document is the [Trademark Trial and Appeal Board Manual of Procedure (TBMP, 2024 version)](https://www.uspto.gov/sites/default/files/documents/tbmp-Master-June2024.pdf). This manual contains detailed procedural rules and guidelines, coming to 1194 pages total.  \n",
     "* **Users:** The target users are intellectual property (IP) litigation associates and paralegals who need quick, accurate answers to procedural questions based *only* on the TBMP.  \n",
     "* **Typical Asks:** Users pose questions requiring synthesis and citation, such as:  \n",
     "  1. \"What are the requirements for filing a motion to compel discovery according to the TBMP?\"  \n",
@@ -2440,7 +2440,7 @@
     "\n",
     "![](../../../images/3C_insurance_task_card.png)\n",
     "\n",
-    "Many businesses are faced with the task of digitizing hand filled forms. In this section, we will demonstrate how OpenAI can be used to digitize and validate a hand filled insurance form. While this is a common problem for insurance, the same techniques can be applied to a variety of other industries and forms, for example tax forms, invoices, and more.\n",
+    "Many businesses are faced with the task of digitizing hand-filled forms. In this section, we will demonstrate how OpenAI can be used to digitize and validate a hand-filled insurance form. While this is a common problem for insurance, the same techniques can be applied to a variety of other industries and forms, for example tax forms, invoices, and more.\n",
     "\n",
     "## 🗂️ TL;DR Matrix\n",
     "\n",
@@ -2489,7 +2489,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 20,
+   "execution_count": 6,
    "id": "923344db",
    "metadata": {},
    "outputs": [
@@ -2519,7 +2519,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 21,
+   "execution_count": 7,
    "id": "7ccd93f6",
    "metadata": {},
    "outputs": [],
@@ -2592,14 +2592,14 @@
    "source": [
     "**Flow Explanation: Stage 1**\n",
     "\n",
-    "1. **Image:** The image of the form taken from the user's smartphone is passed to the model. OpenAI's models can accept a variety of image formats, but we typically use a PNG format to keep the text crisp and reduce artifacts. For this example, we pass the image to the model from a publically available content URL. In a production environment, you likely would pass the image as a signed URL to an image hosted in your own cloud storage bucket.  \n",
+    "1. **Image:** The image of the form taken from the user's smartphone is passed to the model. OpenAI's models can accept a variety of image formats, but we typically use a PNG format to keep the text crisp and reduce artifacts. For this example, we pass the image to the model from a publicly available content URL. In a production environment, you likely would pass the image as a signed URL to an image hosted in your own cloud storage bucket.  \n",
     "     \n",
     "2. **Structured Output Schema:** We define a Pydantic model that sets the structure of the output data. The model includes all of the fields that we need to extract from the form, along with the appropriate types for each field. Our model is broken into several subcomponents, each of which is a Pydantic model itself and referenced by the parent model."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 22,
+   "execution_count": 8,
    "id": "59263ec9",
    "metadata": {},
    "outputs": [],
@@ -2620,7 +2620,7 @@
     "\n",
     "class DwellingDetails(BaseModel):\n",
     "    coverage_a_limit: str\n",
-    "    compantion_policy_expiration_date: str\n",
+    "    companion_policy_expiration_date: str\n",
     "    occupancy_of_dwelling: str\n",
     "    type_of_policy: str\n",
     "    unrepaired_structural_damage: bool\n",
@@ -2654,22 +2654,15 @@
    "id": "70e746a3",
    "metadata": {},
    "source": [
-    "3. **Run OCR:** Using the vision capabilities of GPT-4.1, we run the first stage of our pipeline to extract the text from the document in a structured format. This initial stage aims to achieve high accuracy while passing through uncertainty to the second stage. Our prompt explicitly instructs the model to avoid inferring inputs and instead to fill out the details as exact as possible. For the image input, we set image input detail to `auto` to infer a detail level that's appropriate to the image. We found in our experiments that `auto` worked well, but if you are seeing quality issues in your OCR processing considering using `high`."
+    "3. **Run OCR:** Using the vision capabilities of GPT-4.1, we run the first stage of our pipeline to extract the text from the document in a structured format. This initial stage aims to achieve high accuracy while passing through uncertainty to the second stage. Our prompt explicitly instructs the model to avoid inferring inputs and instead to fill out the details as exact as possible. For the image input, we set image input detail to `auto` to infer a detail level that's appropriate to the image. We found in our experiments that `auto` worked well, but if you are seeing quality issues in your OCR processing consider using `high`."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 23,
+   "execution_count": 9,
    "id": "1537dad2",
    "metadata": {},
    "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "HTTP Request: POST https://api.openai.com/v1/responses \"HTTP/1.1 200 OK\"\n"
-     ]
-    },
     {
      "name": "stdout",
      "output_type": "stream",
@@ -2707,7 +2700,7 @@
       "  \"companion_policy_number\": \"81265919\",\n",
       "  \"dwelling_details\": {\n",
       "    \"coverage_a_limit\": \"$900,000\",\n",
-      "    \"compantion_policy_expiration_date\": \"5/31/27\",\n",
+      "    \"companion_policy_expiration_date\": \"5/31/27\",\n",
       "    \"occupancy_of_dwelling\": \"Owner\",\n",
       "    \"type_of_policy\": \"Homeowners\",\n",
       "    \"unrepaired_structural_damage\": false,\n",
@@ -2734,7 +2727,7 @@
     "OCR_PROMPT = \"\"\"You are a helpful assistant who excels at processing insurance forms.\n",
     "\n",
     "You will be given an image of a hand-filled insurance form. Your job is to OCR the data into the given structured format.\n",
-    "Fill out the fields as exactly as possible. If a written character could possibly be ambigious (i.e. l or 1, o or 0), include all possiblities in the field separated by \"OR\", especially for email addresses.\n",
+    "Fill out the fields as exactly as possible. If a written character could possibly be ambiguous (i.e. l or 1, o or 0), include all possiblities in the field separated by \"OR\", especially for email addresses.\n",
     "\"\"\"\n",
     "\n",
     "user_content = [\n",
@@ -2777,7 +2770,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 24,
+   "execution_count": 10,
    "id": "72dc150e",
    "metadata": {},
    "outputs": [],
@@ -2830,91 +2823,36 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 25,
+   "execution_count": 11,
    "id": "ae8fcf6d",
    "metadata": {},
    "outputs": [],
    "source": [
     "PROMPT = \"\"\"You are a helpful assistant who excels at processing insurance forms.\n",
     "\n",
-    "You will be given a javascript representation of an OCR'd document. Consider at which fields are ambigious reason about how to fill them in. Fill any missing fields that are possible to infer from existing data, or search the web. If you cannot fill a field, reason about why.\n",
+    "You will be given a javascript representation of an OCR'd document. Consider at which fields are ambiguous reason about how to fill them in. Fill any missing fields that are possible to infer from existing data, or search the web. If you cannot fill a field, reason about why.\n",
     "\n",
     "Use the tools provided if necessary to clarify the results. If the OCR system has provided two possibilities, do your best to definitely pick which option is correct.\n",
     "\"\"\""
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 26,
+   "execution_count": 12,
    "id": "1d2b77ee",
    "metadata": {},
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "Requesting completion from model 'o4-mini-2025-04-16' (messages=2)\n"
-     ]
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "HTTP Request: POST https://api.openai.com/v1/responses \"HTTP/1.1 200 OK\"\n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Assistant requested tool calls, resolving ...\n",
-      "Tool call search_web complete, result: 855 Brannan St, San Francisco, 94103, San Francisco County\n",
-      "Requesting completion from model 'o4-mini-2025-04-16' (messages=5)\n"
-     ]
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "HTTP Request: POST https://api.openai.com/v1/responses \"HTTP/1.1 200 OK\"\n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
+      "Requesting completion from model 'o4-mini-2025-04-16' (messages=2)\n",
       "Assistant requested tool calls, resolving ...\n",
       "Tool call validate_email complete, result: True\n",
-      "Requesting completion from model 'o4-mini-2025-04-16' (messages=8)\n"
-     ]
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "HTTP Request: POST https://api.openai.com/v1/responses \"HTTP/1.1 200 OK\"\n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
+      "Requesting completion from model 'o4-mini-2025-04-16' (messages=5)\n",
       "Assistant requested tool calls, resolving ...\n",
       "Tool call validate_email complete, result: False\n",
-      "Requesting completion from model 'o4-mini-2025-04-16' (messages=11)\n"
-     ]
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "HTTP Request: POST https://api.openai.com/v1/responses \"HTTP/1.1 200 OK\"\n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
+      "Requesting completion from model 'o4-mini-2025-04-16' (messages=8)\n",
       "Received parsed result from model\n",
       "{\n",
       "  \"applicant\": {\n",
@@ -2935,21 +2873,21 @@
       "    \"street\": \"855 Brannan St\",\n",
       "    \"city\": \"San Francisco\",\n",
       "    \"state\": \"CA\",\n",
-      "    \"zip\": \"94103\",\n",
+      "    \"zip\": \"94107\",\n",
       "    \"county\": \"San Francisco\"\n",
       "  },\n",
       "  \"mailing_address_if_different_than_risk_address\": {\n",
-      "    \"street\": \"\",\n",
-      "    \"city\": \"\",\n",
-      "    \"state\": \"\",\n",
-      "    \"zip\": \"\",\n",
-      "    \"county\": \"\"\n",
+      "    \"street\": \"855 Brannan St\",\n",
+      "    \"city\": \"San Francisco\",\n",
+      "    \"state\": \"CA\",\n",
+      "    \"zip\": \"94107\",\n",
+      "    \"county\": \"San Francisco\"\n",
       "  },\n",
       "  \"participating_insurer\": \"Acme Insurance Co\",\n",
       "  \"companion_policy_number\": \"81265919\",\n",
       "  \"dwelling_details\": {\n",
       "    \"coverage_a_limit\": \"$900,000\",\n",
-      "    \"compantion_policy_expiration_date\": \"5/31/27\",\n",
+      "    \"companion_policy_expiration_date\": \"5/31/27\",\n",
       "    \"occupancy_of_dwelling\": \"Owner\",\n",
       "    \"type_of_policy\": \"Homeowners\",\n",
       "    \"unrepaired_structural_damage\": false,\n",
@@ -3015,30 +2953,32 @@
     "\n",
     "To help us understand and debug the model, we can also print the summary chain-of-thought reasoning produced by the model. This can help expose common failure modes, points where the model is unclear, or incorrect upstream details.\n",
     "\n",
-    "While developing this solution the chain-of-thought summaries exposed some incorrectly named and typed schema values."
+    "While developing this solution, the chain-of-thought summaries exposed some incorrectly named and typed schema values."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 27,
+   "execution_count": 13,
    "id": "ab1d4fbc",
    "metadata": {},
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "**Filling in missing fields**\n",
+      "**Determining insurance form details**\n",
+      "\n",
+      "I have a JSON representation of a partially filled insurance form, and there are a few missing or ambiguous fields that I need to address.\n",
       "\n",
-      "The user has given me an OCR result map, and I need to address the missing fields. For the applicant's email, we have two options: \"[email protected]\" or \"[email protected].\" I should validate which one is correct. For the address, 855 Brannan St, SF, it's likely in the 94107 zip code rather than 94103, given its location. San Francisco County is the correct county. For the mailing address, I could either note it as \"Same as risk address\" or leave it empty if they're the same.\n",
+      "For the email address, I see two options. I can validate which one is correct by checking both with the tool.\n",
       "\n",
-      "**Considering mailing address input**\n",
+      "The risk address fields for zip code and county are empty. Based on the address \"855 Brannan St, San Francisco, CA,\" I can determine the correct zip code is 94107, as that area corresponds to South Beach. Lastly, since the mailing address is empty, I assume it's the same as the risk address.\n",
       "\n",
-      "I'm looking at the applicant's email and the mailing address input. The field for \"mailing_address_if_different_than_risk_address\" can be tricky. If the mailing address is the same as the risk address, I think it’s best to leave it blank. The schema requires empty strings for optional fields, so filling it with \"same\" might lead to confusion about whether the addresses differ. I’ll confirm that it should ideally just remain empty to avoid any misleading implications.\n",
+      "**Filling insurance form details**\n",
       "\n",
-      "**Clarifying mailing address input**\n",
+      "I think it’s best to set the mailing address to be the same as the risk address or clarify that a blank one implies the same. Since it’s an explicit instruction to fill missing fields, I’ll fill in the mailing address with the risk address to avoid confusion.\n",
       "\n",
-      "I’m looking at the \"mailing_address_if_different_than_risk_address.\" If the addresses are the same, I think it’s best to leave those fields empty since they're only necessary if they differ. The instruction is to fill missing fields that can be inferred, but this mailing address isn’t technically missing. I should focus on filling in the zip and county for the risk address. So, the final output will include these details with other fields remaining as they are.\n",
+      "All co-applicant fields are present, and dwelling details are complete. The effective and expiration dates are also provided. I plan to validate both email options by checking each one separately. Let's begin with validating the first email.\n",
       "\n"
      ]
     }
@@ -3091,7 +3031,7 @@
     "| :---- | :---- | :---- | :---- |\n",
     "| Input | 2,000 | $1.00 | $0.002 |\n",
     "| Output | 1,500 | $4.00 | $0.006 |\n",
-    "| **Total (Stage 1\\)** |  |  | **$8.00** |\n",
+    "| **Total for 1,000 pages (Stage 1\\)** |  |  | **$8.00** |\n",
     "\n",
     "#### **Stage 2: Reasoning**\n",
     "\n",
@@ -3101,7 +3041,7 @@
     "| :---- | :---- | :---- | :---- |\n",
     "| Input | 2,000 | $0.55 | $0.0011 |\n",
     "| Output | 3,000 | $2.20 | $0.0066 |\n",
-    "| **Total (Stage 2\\)** |  |  | **$7.70** |\n",
+    "| **Total for 1,000 pages (Stage 2\\)** |  |  | **$7.70** |\n",
     "\n",
     "#### Grand Total (per 1,000 pages): **$15.70**\n",
     "\n",