{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "*Module 10 Self-Driven Lab*\n", "\n", "# Escalation & Human Review: Triggers, Handoffs, and Confidence Calibration\n", "\n", "**Objective:** design a deterministic escalation policy, build a standalone handoff payload, and calibrate a human-review workflow with per-field confidence.\n", "\n", "## Challenge Outline\n", "\n", "Build a complete notebook that demonstrates the following outcomes:\n", "\n", "- **Escalation criteria:** add the three hard triggers — explicit human request, policy gap, and inability to make meaningful progress — to the system prompt, along with few-shot examples demonstrating escalate-vs-resolve decisions. Test a frustrated-but-simple case (acknowledge the frustration and offer to resolve) against a policy-gap case such as a competitor price match (escalate rather than invent a discount).\n", "- **Multiple-match handling:** make `get_customer` return two matching records and verify the agent asks for an additional identifier, such as an order number or billing zip, instead of picking one by recency or name similarity.\n", "- **Handoff payload:** implement the `escalate_to_human` tool with a strict schema including `customer_id`, `root_cause`, `actions_attempted`, `amounts`, and `recommended_next_step`, and confirm the payload stands alone for a reader who never saw the transcript.\n", "- **Confidence output:** extend an extraction tool to emit a per-field confidence score, and route any field below your threshold to a review queue while the rest auto-approve.\n", "- **Calibration check:** using the labeled validation set from the module, compute accuracy by document type and identify the segment that needs continued human review despite a strong aggregate number.\n", "\n", "Your solution should include enough code, output, or written observations to prove each outcome worked. Keep the notebook focused on final behavior and evidence rather than a guided walkthrough." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Student Workspace\n", "\n", "Use the sections below to build your solution. Each section maps to one required outcome from the challenge outline." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Part 1: Escalation criteria\n", "\n", "Add the three hard triggers — explicit human request, policy gap, and inability to make meaningful progress — to the system prompt, along with few-shot examples demonstrating escalate-vs-resolve decisions. Test with a frustrated-but-simple case (the agent should acknowledge the frustration and offer to resolve) and a policy-gap case such as a competitor price match (the agent should escalate rather than invent a discount)." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "# Part 1: Escalation criteria\n", "# Add your implementation, outputs, or notes here." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Part 2: Multiple-match handling\n", "\n", "Make `get_customer` return two matching records and verify the agent asks for an additional identifier, such as an order number or billing zip, instead of picking one by recency or name similarity. A heuristic pick that lands on the wrong customer is a data-exposure incident." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "# Part 2: Multiple-match handling\n", "# Add your implementation, outputs, or notes here." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Part 3: Handoff payload\n", "\n", "Implement the `escalate_to_human` tool with a strict schema including `customer_id`, `root_cause`, `actions_attempted`, `amounts`, and `recommended_next_step`. Confirm the payload makes sense to a reader who has never seen the transcript." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "# Part 3: Handoff payload\n", "# Add your implementation, outputs, or notes here." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Part 4: Confidence output\n", "\n", "Extend an extraction tool to emit a per-field confidence score, and route any field below your threshold to a review queue while the rest auto-approve." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "# Part 4: Confidence output\n", "# Add your implementation, outputs, or notes here." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Part 5: Calibration check\n", "\n", "Using the labeled validation set below, compute accuracy by document type and identify the segment that needs continued human review despite the strong aggregate number.\n", "\n", "```csv\n", "doc_id,doc_type,field,model_value,label_value,correct\n", "D01,invoice,total,4820.50,4820.50,true\n", "D02,invoice,total,1130.00,1130.00,true\n", "D03,invoice,total,905.75,905.75,true\n", "D04,invoice,total,2210.10,2210.10,true\n", "D05,receipt,total,88.20,88.20,true\n", "D06,receipt,total,41.99,41.99,true\n", "D07,receipt,total,17.35,17.35,true\n", "D08,purchase_order,total,15200.00,12500.00,false\n", "D09,purchase_order,total,730.00,7300.00,false\n", "D10,purchase_order,total,2650.00,2650.00,true\n", "```" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "# Part 5: Calibration check\n", "# Add your implementation, outputs, or notes here." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Verification Notes\n", "\n", "Summarize the evidence that each part worked. Capture API signals, validation outcomes, errors, recovery behavior, cost observations, or comparisons required by this lab." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "# Verification notes\n", "# Record the evidence that proves each lab outcome worked." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "## Answer Key\n", "\n", "The cells below contain the completed reference implementation/content for this module. Use this section only after attempting the self-driven lab." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Escalation & Human Review: Triggers, Handoffs, and Confidence Calibration\n", "\n", "*Module 10*\n", "\n", "This module covers the human side of reliable agent systems: which triggers should move a case from an autonomous agent to a human operator, how to package a handoff so the human can act without the transcript, and how to design human-review workflows that spend limited reviewer capacity where the model is least reliable.\n", "\n", "**Lab notebook:** [Module10_Complete.ipynb](lab_files/Module10_Complete.ipynb)\n", "\n", "## 1. Explicit Escalation: Triggers as Policy\n", "\n", "Escalation should be driven by written policy, not model vibes or sentiment alone.\n", "\n", "### 1a. Hard Triggers\n", "\n", "- **Customer requests:** any explicit request for a human must be honored immediately without re-routing or attempting further autonomous resolution.\n", "- **Policy gaps or silence:** if a request touches an area where the provided policy is ambiguous or silent, such as a refund-window exception or competitor price match, the agent must escalate rather than improvising a new policy.\n", "- **Inability to make meaningful progress:** if the agent has looped through several turns or tool calls without advancing the case — repeating the same questions, retrying the same failing action, or circling the same diagnosis — it must escalate rather than continue burning the customer's time.\n", "- **High-stakes actions:** use the PreToolUse hooks from Module 9 to catch and escalate irreversible actions before they execute.\n", "\n", "```system prompt policy fragment\n", "Escalate immediately when:\n", "- The customer asks for a human, manager, representative, or supervisor.\n", "- The policy does not explicitly cover the requested exception.\n", "- You are not making meaningful progress on the case: you have repeated\n", " the same question, retried the same failing action, or gone several\n", " turns without moving the case forward.\n", "- The requested action is irreversible, regulated, or above the\n", " configured approval threshold.\n", "```\n", "\n", "### 1b. Unreliable Triggers the Exam Expects You to Reject\n", "\n", "Two commonly proposed triggers fail in practice, and the exam expects you to reject both:\n", "\n", "- **Sentiment-based escalation:** customer frustration does not correlate with case complexity. A furious customer with a standard damage claim is still a simple case the agent can resolve in one turn. Routing on sentiment sends easy cases to humans while hard-but-polite cases stay with the agent. When a customer expresses frustration but the issue is within the agent's capability, the correct behavior is to acknowledge the frustration and *offer* to resolve the issue — escalating only if the customer reiterates the request for a human.\n", "- **Self-reported confidence scores:** LLM confidence is poorly calibrated. The agent is already wrongly confident on exactly the hard cases, so a \"escalate when confidence < 0.7\" threshold routes the wrong cases: the model reports high confidence on the cases it is getting wrong.\n", "\n", "The proportionate fix for mis-calibrated escalation behavior is **explicit criteria plus few-shot examples** in the system prompt demonstrating escalate-vs-resolve decisions — before reaching for classifiers, fine-tuning, or ML routing infrastructure. Show the model a frustrated-but-simple case being resolved and a policy-gap case being escalated, and most mis-routing disappears.\n", "\n", "### 1c. Ambiguous Identity: Multiple Matches\n", "\n", "Identity resolution has its own hard rule. When `get_customer` returns multiple matches, the agent must ask the customer for an additional identifier — an order number, a billing zip code — and re-query. It must **never** select a record by heuristic, such as picking the most recent account or the closest name match. A heuristic pick that lands on the wrong customer turns a lookup into a data-exposure incident.\n", "\n", "## 2. The Structured Handoff Protocol\n", "\n", "A blind handoff frustrates users. A structured payload ensures the human operator can resolve the issue quickly.\n", "\n", "The final act of the agent is to call an `escalate_to_human` tool with a strict schema. Remember: the human receiving the handoff does **not** have the conversation transcript — the payload must stand alone. The exam's handoff checklist is: customer ID, root cause, amounts, and recommended action.\n", "\n", "*strict handoff schema*\n", "```json\n", "{\n", " \"name\": \"escalate_to_human\",\n", " \"description\": \"Transfer control to a human operator with enough context to resolve the issue without repeating failed steps.\",\n", " \"strict\": true,\n", " \"input_schema\": {\n", " \"type\": \"object\",\n", " \"properties\": {\n", " \"customer_id\": { \"type\": \"string\" },\n", " \"root_cause\": { \"type\": \"string\" },\n", " \"actions_attempted\": {\n", " \"type\": \"array\",\n", " \"items\": { \"type\": \"string\" }\n", " },\n", " \"amounts\": {\n", " \"type\": \"array\",\n", " \"items\": {\n", " \"type\": \"object\",\n", " \"properties\": {\n", " \"description\": { \"type\": \"string\" },\n", " \"amount_usd\": { \"type\": \"number\" }\n", " },\n", " \"required\": [\"description\", \"amount_usd\"],\n", " \"additionalProperties\": false\n", " }\n", " },\n", " \"recommended_next_step\": { \"type\": \"string\" }\n", " },\n", " \"required\": [\n", " \"customer_id\",\n", " \"root_cause\",\n", " \"actions_attempted\",\n", " \"amounts\",\n", " \"recommended_next_step\"\n", " ],\n", " \"additionalProperties\": false\n", " }\n", "}\n", "```\n", "\n", "## 3. Human Review Workflows & Confidence Calibration\n", "\n", "Escalation covers live conversations. Batch workloads — document extraction, classification, claims intake — need a different human-in-the-loop design: a review workflow that decides *which* outputs a human checks.\n", "\n", "### 3a. Aggregate Accuracy Masks Segment Failures\n", "\n", "A headline metric like \"97% extraction accuracy\" can hide a segment that is failing badly: 60% accuracy on one document type, or one field that is systematically wrong across every document. Before reducing human review, validate accuracy **by document type** and **by field**. The decision to pull humans out of the loop must be made per segment, never on the aggregate number.\n", "\n", "### 3b. Field-Level Confidence, Calibrated Against Labels\n", "\n", "Have the model output a confidence score for each extracted field, then **calibrate** review thresholds using a labeled validation set: for each confidence band, measure the actual error rate against ground truth, and set the review threshold where the measured error rate crosses your tolerance. Raw self-reported confidence is not trustworthy until it has been calibrated against labels — it becomes useful only as an empirically validated routing signal.\n", "\n", "*extraction output with per-field confidence*\n", "```json\n", "{\n", " \"document_type\": \"invoice\",\n", " \"fields\": {\n", " \"vendor_name\": { \"value\": \"Acme Industrial Supply\", \"confidence\": 0.98 },\n", " \"invoice_total\": { \"value\": 4820.50, \"confidence\": 0.95 },\n", " \"due_date\": { \"value\": \"2026-08-01\", \"confidence\": 0.62 },\n", " \"po_number\": { \"value\": \"PO-77813\", \"confidence\": 0.41 }\n", " }\n", "}\n", "```\n", "\n", "With a calibrated threshold of 0.85, `due_date` and `po_number` route to the human review queue while the other two fields flow straight through.\n", "\n", "### 3c. Stratified Random Sampling of High-Confidence Output\n", "\n", "Never stop sampling just because confidence is high. Continuously pull a stratified random sample of high-confidence extractions for human review. This serves two purposes: it measures the true error rate in the auto-approved stream, and it detects **novel error patterns** — a new document layout or vendor format that the model confidently misreads — before they accumulate.\n", "\n", "### 3d. Routing: Spend Reviewers Where the Model Is Least Reliable\n", "\n", "Reviewer capacity is the scarce resource. Route to human review: (1) low-confidence extractions below the calibrated threshold, (2) source documents that are ambiguous or internally contradictory — a scan where two totals disagree cannot be resolved by any extraction model, and (3) the ongoing stratified sample from the high-confidence stream. Everything else auto-approves.\n", "\n", "> **Tip.** **Exam tip:** distinguish calibrated field-level confidence from self-reported confidence. Field-level scores validated against a labeled set are a legitimate routing signal for review queues. Asking the model \"how confident are you, 1–10?\" and acting on the raw answer is the same uncalibrated signal rejected as an escalation trigger in section 1.\n", "\n", "## Lab Exercise: Escalation Policy, Handoffs, and Review Calibration\n", "\n", "**Lab notebook:** [Module10_Self_Driven_Lab.ipynb](lab_files/Module10_Self_Driven_Lab.ipynb)\n", "\n", "**Objective:** design a deterministic escalation policy, build a standalone handoff payload, and calibrate a human-review workflow with per-field confidence.\n", "\n", "1. **Escalation criteria:** add the three hard triggers — explicit human request, policy gap, and inability to make meaningful progress — to the system prompt, along with few-shot examples demonstrating escalate-vs-resolve decisions. Test with a frustrated-but-simple case (the agent should acknowledge the frustration and offer to resolve) and a policy-gap case such as a competitor price match (the agent should escalate rather than invent a discount).\n", "2. **Multiple-match handling:** make `get_customer` return two matching records and verify the agent asks for an additional identifier, such as an order number or billing zip, instead of picking one by recency or name similarity.\n", "3. **Handoff payload:** implement the `escalate_to_human` tool with a strict schema including `customer_id`, `root_cause`, `actions_attempted`, `amounts`, and `recommended_next_step`. Confirm the payload makes sense to a reader who has never seen the transcript.\n", "4. **Confidence output:** extend an extraction tool to emit a per-field confidence score, and route any field below your threshold to a review queue while the rest auto-approve.\n", "5. **Calibration check:** using the labeled set below, compute accuracy by document type and identify the segment that needs continued human review despite the strong aggregate number.\n", "\n", "*labeled validation set*\n", "```csv\n", "doc_id,doc_type,field,model_value,label_value,correct\n", "D01,invoice,total,4820.50,4820.50,true\n", "D02,invoice,total,1130.00,1130.00,true\n", "D03,invoice,total,905.75,905.75,true\n", "D04,invoice,total,2210.10,2210.10,true\n", "D05,receipt,total,88.20,88.20,true\n", "D06,receipt,total,41.99,41.99,true\n", "D07,receipt,total,17.35,17.35,true\n", "D08,purchase_order,total,15200.00,12500.00,false\n", "D09,purchase_order,total,730.00,7300.00,false\n", "D10,purchase_order,total,2650.00,2650.00,true\n", "```\n", "\n", "> **Tip.** **Exam tip:** escalation triggers, handoff completeness, and review calibration are separate design decisions. A system can have perfect handoffs and still escalate the wrong cases.\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.0" } }, "nbformat": 4, "nbformat_minor": 5 }