{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Module 9 Self-Driven Lab*\r\n",
    "\r\n",
    "# Reliability & Deterministic Enforcement\r\n",
    "\r\n",
    "**Objective:** combine `tool_choice`, PreToolUse hooks, structured errors, and PostToolUse normalization to build a safe financial agent.\r\n",
    "\r\n",
    "## Challenge Outline\r\n",
    "\r\n",
    "Build a complete notebook that demonstrates the following outcomes:\r\n",
    "\r\n",
    "- **Prerequisite forcing:** use `tool_choice` to force the agent to call an `identity_verification` tool on its first turn.\r\n",
    "- **Safety hook:** implement a PreToolUse hook that blocks `issue_refund` when `amount_usd` is over `$500`, even if the model's reasoning says it is authorized. Deny with a reason string that instructs the model to call `escalate_to_human`, then verify the model responds by escalating on its next turn.\r\n",
    "- **Structured error handling:** create a tool handler for a \"System Offline\" error that returns `isError: true`, `errorCategory: TRANSIENT`, and `isRetryable: true`; observe the agent attempting a recovery retry.\r\n",
    "- **Normalization:** implement a PostToolUse hook that converts a Unix timestamp from an MCP database tool into a human-readable ISO string.\r\n",
    "- **Validation vs. access:** test a \"User Not Found\" response as `isError: false` because it is a valid empty result, then compare it with an actual \"Access Denied\" error marked as `PERMISSION`.\r\n",
    "\r\n",
    "Your solution should include enough code, output, or written observations to prove each outcome worked. Keep the notebook focused on final behavior and evidence rather than a guided walkthrough.\r\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Student Workspace\n",
    "\n",
    "Use the sections below to build your solution. Each section maps to one required outcome from the challenge outline.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 1: Prerequisite forcing\n",
    "\n",
    "use `tool_choice` to force the agent to call an `identity_verification` tool on its first turn.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Part 1: Prerequisite forcing\n",
    "# Add your implementation, outputs, or notes here.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 2: Safety hook\r\n",
    "\r\n",
    "implement a PreToolUse hook that blocks `issue_refund` when `amount_usd` is over `$500`, even if the model's reasoning says it is authorized. Deny with a reason string that instructs the model to call `escalate_to_human`, then verify the model responds by escalating on its next turn.\r\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Part 2: Safety hook\n",
    "# Add your implementation, outputs, or notes here.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 3: Structured error handling\n",
    "\n",
    "create a tool handler for a \"System Offline\" error that returns `isError: true`, `errorCategory: TRANSIENT`, and `isRetryable: true`; observe the agent attempting a recovery retry.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Part 3: Structured error handling\n",
    "# Add your implementation, outputs, or notes here.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 4: Normalization\n",
    "\n",
    "implement a PostToolUse hook that converts a Unix timestamp from an MCP database tool into a human-readable ISO string.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Part 4: Normalization\n",
    "# Add your implementation, outputs, or notes here.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 5: Validation vs. access\n",
    "\n",
    "test a \"User Not Found\" response as `isError: false` because it is a valid empty result, then compare it with an actual \"Access Denied\" error marked as `PERMISSION`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Part 5: Validation vs. access\n",
    "# Add your implementation, outputs, or notes here.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verification Notes\n",
    "\n",
    "Summarize the evidence that each part worked. Capture API signals, validation outcomes, errors, recovery behavior, cost observations, or comparisons required by this lab.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Verification notes\n",
    "# Record the evidence that proves each lab outcome worked.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Answer Key\n",
    "\n",
    "The cells below contain the completed reference implementation/content for this module. Use this section only after attempting the self-driven lab."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reliability & Deterministic Enforcement\r\n",
    "\r\n",
    "This module focuses on Agent SDK hooks and structured error contracts that guarantee compliance and normalization even when the model's reasoning deviates from its instructions.\r\n",
    "\r\n",
    "Prompts are probabilistic. Hooks, gates, and validators are deterministic. Use prompts to guide model behavior, but use runtime controls to enforce policy, prerequisites, and normalized tool results.\r\n",
    "\r\n",
    "## 1. Deterministic vs. Probabilistic Enforcement\r\n",
    "\r\n",
    "Prompts can ask the model not to issue a high-risk refund. A **PreToolUse** hook can block the refund before it executes. This distinction matters for financial, legal, privacy, safety, and irreversible operations.\r\n",
    "\r\n",
    "### 1a. PreToolUse Interception\r\n",
    "\r\n",
    "Use **PreToolUse** to intercept outgoing tool calls before execution. This is the only way to guarantee that high-value actions, such as refunds over $500, are blocked regardless of the model's intent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "REFUND_LIMIT = 500\r\n",
    "\r\n",
    "def pre_tool_use(tool_call):\r\n",
    "    if tool_call[\"name\"] != \"issue_refund\":\r\n",
    "        return {\"decision\": \"allow\"}\r\n",
    "\r\n",
    "    amount = tool_call[\"arguments\"].get(\"amount_usd\", 0)\r\n",
    "    if amount > REFUND_LIMIT:\r\n",
    "        return {\r\n",
    "            \"decision\": \"deny\",\r\n",
    "            \"reason\": (\r\n",
    "                \"Refunds above $500 require human approval. \"\r\n",
    "                \"Call escalate_to_human with a structured \"\r\n",
    "                \"handoff summary.\"\r\n",
    "            ),\r\n",
    "        }\r\n",
    "\r\n",
    "    return {\"decision\": \"allow\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the limits of the mechanism: a hook cannot substitute a different tool call. It can only allow the call or deny it with a reason string that is fed back to the model. Write the denial reason so it tells the model what to do next — here, the model reads the reason and chooses to call `escalate_to_human` itself on its next turn.\r\n",
    "\r\n",
    "### 1b. Programmatic Prerequisite Gates\r\n",
    "\r\n",
    "`tool_choice` can force the model to call an identification tool first, but a programmatic gate is what blocks downstream tools until the prerequisite has actually succeeded.\r\n",
    "\r\n",
    "For example, block `process_refund` until `get_customer` or `identity_verification` has returned a verified success status. A model can ignore a prompt; it cannot bypass a runtime gate.\r\n",
    "\r\n",
    "## 2. Standardized Structured Error Contracts\r\n",
    "\r\n",
    "Uniform prose errors like \"Operation failed\" are anti-patterns because they prevent the agent from making informed recovery decisions. Every tool should return the same structured fields when it fails.\r\n",
    "\r\n",
    "### 2a. The `errorCategory` Enum\r\n",
    "\r\n",
    "Standardize tool errors into four categories:\r\n",
    "\r\n",
    "- `TRANSIENT`: timeouts or service unavailability; Claude should attempt a retry.\r\n",
    "- `VALIDATION`: invalid input, such as a bad email format; the agent should clarify with the user.\r\n",
    "- `PERMISSION`: authorization issues; the agent should inform the user of the access gap.\r\n",
    "- `BUSINESS`: policy violations, such as refund exceeds limit; the agent should communicate the rule.\r\n",
    "\r\n",
    "### 2b. Metadata Over Prose\r\n",
    "\r\n",
    "Always return an `isRetryable` boolean. This prevents the agent from wasting tokens on repeated attempts for non-retryable permission, validation, or business-rule failures.\r\n",
    "\r\n",
    "```json\r\n",
    "{\r\n",
    "  \"isError\": true,\r\n",
    "  \"errorCategory\": \"TRANSIENT\",\r\n",
    "  \"isRetryable\": true,\r\n",
    "  \"message\": \"Refund system is temporarily offline.\"\r\n",
    "}\r\n",
    "```\r\n",
    "\r\n",
    "```json\r\n",
    "{\r\n",
    "  \"isError\": true,\r\n",
    "  \"errorCategory\": \"BUSINESS\",\r\n",
    "  \"isRetryable\": false,\r\n",
    "  \"message\": \"Refunds over $500 require human approval.\"\r\n",
    "}\r\n",
    "```\r\n",
    "\r\n",
    "## 3. PostToolUse Normalization for MCP\r\n",
    "\r\n",
    "Different MCP servers often return inconsistent data formats: Unix timestamps, local date strings, ISO strings, numeric status codes, or provider-specific labels.\r\n",
    "\r\n",
    "Implement **PostToolUse** hooks to rewrite heterogeneous tool results into a homogeneous shape before they reach the model. This reduces thinking-budget consumption because the model does not have to reconcile formats manually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime, timezone\n",
    "\n",
    "def post_tool_use(tool_name, result):\n",
    "    if tool_name == \"crm_database_lookup\" and \"created_at_unix\" in result:\n",
    "        created_at = datetime.fromtimestamp(\n",
    "            result[\"created_at_unix\"],\n",
    "            tz=timezone.utc,\n",
    "        ).isoformat()\n",
    "        result[\"created_at\"] = created_at\n",
    "        del result[\"created_at_unix\"]\n",
    "\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Forced Prerequisite Strategy\r\n",
    "\r\n",
    "Use forced tool selection to implement deterministic multi-step sequences.\r\n",
    "\r\n",
    "On the first turn, set:\r\n",
    "\r\n",
    "```json\r\n",
    "{\r\n",
    "  \"tool_choice\": {\r\n",
    "    \"type\": \"tool\",\r\n",
    "    \"name\": \"get_customer\"\r\n",
    "  }\r\n",
    "}\r\n",
    "```\r\n",
    "\r\n",
    "This makes the identification step mandatory. On subsequent turns, switch back to `\"auto\"` or `\"any\"` so the agent can reason flexibly after the prerequisite evidence exists.\r\n",
    "\r\n",
    "Forced selection and hooks work together:\r\n",
    "\r\n",
    "- `tool_choice` ensures the first step is attempted.\r\n",
    "- `PreToolUse` gates ensure unsafe downstream tools cannot execute without verified prerequisite state.\r\n",
    "- Structured errors tell the model whether to retry, clarify, inform, or escalate.\r\n",
    "- `PostToolUse` normalization ensures the model reasons over stable result shapes.\r\n",
    "\r\n",
    "## Lab Exercise: The Defense-in-Depth Refund Workflow\r\n",
    "\r\n",
    "**Objective:** combine `tool_choice`, PreToolUse hooks, structured errors, and PostToolUse normalization to build a safe financial agent.\r\n",
    "\r\n",
    "1. **Prerequisite forcing:** use `tool_choice` to force the agent to call an `identity_verification` tool on its first turn.\r\n",
    "2. **Safety hook:** implement a PreToolUse hook that blocks `issue_refund` when `amount_usd` is over `$500`, even if the model's reasoning says it is authorized. Deny with a reason string that instructs the model to call `escalate_to_human`, then verify the model responds by escalating on its next turn.\r\n",
    "3. **Structured error handling:** create a tool handler for a \"System Offline\" error that returns `isError: true`, `errorCategory: TRANSIENT`, and `isRetryable: true`; observe the agent attempting a recovery retry.\r\n",
    "4. **Normalization:** implement a PostToolUse hook that converts a Unix timestamp from an MCP database tool into a human-readable ISO string.\r\n",
    "5. **Validation vs. access:** test a \"User Not Found\" response as `isError: false` because it is a valid empty result, then compare it with an actual \"Access Denied\" error marked as `PERMISSION`.\r\n",
    "\r\n",
    "> **Exam tip:** errors talk to the model; hooks talk to the runtime. Use structured errors to guide recovery, but use hooks and gates to enforce policy."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}