{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "*Lab 3 Self-Driven Lab*\n", "\n", "# Claude Models, Prompting & Context Engineering\n", "\n", "**Objective:** work the caching numbers and the model routing for a 40,000-request-per-day support system.\n", "\n", "## Challenge Outline\n", "\n", "Build a complete notebook that demonstrates the following outcomes:\n", "\n", "- **Confirm the shape matches Sample 2:** recognize the caching case (large stable prefix, short varying suffix, cost and latency both in play) and write the correctly ordered call with the breakpoint on the policy block.\n", "- **Check the minimum cacheable prefix:** confirm the 8,000-token prefix clears Opus 4.8's 4096-token minimum, and describe the silent failure below it.\n", "- **Compute the cache economics:** compute the daily prefix cost uncached versus cached at 40k requests/day, the percentage reduction, and the annualized saving.\n", "- **Choose models per stage:** route a cheap Haiku triage stage in front of Opus synthesis for the hard subset, and compute the blended cost against all-Opus.\n", "- **Watch the context ceiling:** state why a 300K-token retrieved corpus disqualifies Haiku on capacity (200K window) regardless of price, while Sonnet and Opus carry it.\n", "- **Write the escalation rule, then enforce it at egress:** write the five-part system prompt's escalation clause, then implement the code-level validator that guarantees what the prompt only requests.\n", "\n", "Your solution should include enough code, output, or written observations to prove each outcome worked. Keep the notebook focused on final behavior and evidence rather than a guided walkthrough.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Student Workspace\n", "\n", "Use the sections below to build your solution. Each section maps to one required outcome from the challenge outline.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Part 1: Confirm the shape matches Sample 2\n", "\n", "recognize the caching case (large stable prefix, short varying suffix, cost and latency both in play) and write the correctly ordered call with the breakpoint on the policy block.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Part 1: Confirm the shape matches Sample 2\n", "# Add your implementation, outputs, or notes here.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Part 2: Check the minimum cacheable prefix\n", "\n", "confirm the 8,000-token prefix clears Opus 4.8's 4096-token minimum, and describe the silent failure below it.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Part 2: Check the minimum cacheable prefix\n", "# Add your implementation, outputs, or notes here.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Part 3: Compute the cache economics\n", "\n", "compute the daily prefix cost uncached versus cached at 40k requests/day, the percentage reduction, and the annualized saving.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Part 3: Compute the cache economics\n", "# Add your implementation, outputs, or notes here.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Part 4: Choose models per stage\n", "\n", "route a cheap Haiku triage stage in front of Opus synthesis for the hard subset, and compute the blended cost against all-Opus.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Part 4: Choose models per stage\n", "# Add your implementation, outputs, or notes here.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Part 5: Watch the context ceiling\n", "\n", "state why a 300K-token retrieved corpus disqualifies Haiku on capacity (200K window) regardless of price, while Sonnet and Opus carry it.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Part 5: Watch the context ceiling\n", "# Add your implementation, outputs, or notes here.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Part 6: Write the escalation rule, then enforce it at egress\n", "\n", "write the five-part system prompt's escalation clause, then implement the code-level validator that guarantees what the prompt only requests.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Part 6: Write the escalation rule, then enforce it at egress\n", "# Add your implementation, outputs, or notes here.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Verification Notes\n", "\n", "Summarize the evidence that each part worked. Capture API signals, validation outcomes, errors, recovery behavior, cost observations, or comparisons required by this lab.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Verification notes\n", "# Record the evidence that proves each lab outcome worked.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "## Answer Key\n", "\n", "The cells below contain the completed reference implementation/content for this lab. Use this section only after attempting the self-driven lab.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Lab 3 Answer Key*\n", "\n", "# Claude Models, Prompting & Context Engineering\n", "\n", "**Objective:** work the caching numbers and the model routing for a 40,000-request-per-day support system.\n", "\n", "This lab has real numbers behind every decision: a token price, a cache multiplier, a context ceiling. The code cells compute the cost models; the markdown cells carry the judgment.\n", "\n", "**Scenario.** A support platform sends the same 8,000-token policy prefix on every request, followed by a short varying customer question. Volume is 40,000 requests per day. Leadership wants both lower cost per ticket and faster first response.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 1: Confirm the shape matches Sample 2\n", "\n", "Large stable prefix, short varying suffix, both cost and latency in play: this is the guide's Sample 2, and the answer is to place the static content before the dynamic content and enable prompt caching. Caching is a prefix match, so any change anywhere in the prefix invalidates everything after it.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import anthropic\n", "\n", "client = anthropic.Anthropic()\n", "\n", "POLICY_DOCUMENT = open(\"support_policy.txt\").read() # ~8,000 stable tokens\n", "customer_question = \"Can I return an opened item after 30 days?\"\n", "\n", "response = client.messages.create(\n", " model=\"claude-opus-4-8\",\n", " max_tokens=1024,\n", " system=[\n", " {\n", " \"type\": \"text\",\n", " \"text\": POLICY_DOCUMENT, # stable content FIRST\n", " \"cache_control\": {\"type\": \"ephemeral\"}, # breakpoint on the stable block\n", " },\n", " ],\n", " messages=[\n", " # The varying question comes AFTER the cached prefix, never before it.\n", " {\"role\": \"user\", \"content\": customer_question},\n", " ],\n", ")\n", "\n", "# Verify the cache actually hit. input_tokens is the UNCACHED REMAINDER ONLY;\n", "# the cached portion is reported separately.\n", "print(\"cache_read:\", response.usage.cache_read_input_tokens) # ~8000 on a hit\n", "print(\"uncached remainder:\", response.usage.input_tokens)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The distractor version of this call puts the customer question in the system array before the policy. The question changes on every request, so every request has a different prefix, `cache_read_input_tokens` stays zero forever, and the team concludes caching does not work. Order is the whole mechanism: `tools`, then `system`, then `messages`, stable before varying.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 2: Check the minimum cacheable prefix\n", "\n", "On Opus 4.8 the minimum cacheable prefix is 4096 tokens, and a breakpoint on a shorter prefix does nothing, silently. Confirm the 8k policy clears it, and know the diagnostic shortlist when reads are zero.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "8,000 tokens clears Opus 4.8's 4096-token minimum comfortably, so the breakpoint will actually take. The failure below the minimum is worth internalizing because it is silent: no error, `cache_creation_input_tokens` comes back zero, and you keep paying full price while believing you are cached. A 3,000-token prompt that caches on a lower-minimum model will not cache on Opus.\n", "\n", "The diagnostic shortlist when caching is enabled but `cache_read_input_tokens` is zero:\n", "\n", "1. prefix under the model's minimum (silent no-op)\n", "2. a silent invalidator in the prefix: a timestamp, session ID, or user name interpolated into the system prompt\n", "3. dynamic-before-static ordering (Part 1's distractor)\n", "\n", "Anything that varies goes last.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 3: Compute the cache economics\n", "\n", "Without caching, each request pays for 8,000 prefix tokens at input price. With caching, the first request of each 5-minute window writes at 1.25x and every subsequent request reads at 0.1x. At 40k/day the prefix is re-read constantly, so compute what almost-always-0.1x is worth.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Cache economics on Opus 4.8 ($5 per 1M input tokens).\n", "OPUS_IN = 5.00 / 1_000_000\n", "CACHE_READ, CACHE_WRITE = 0.1, 1.25\n", "\n", "REQS_PER_DAY = 40_000\n", "PREFIX = 8_000\n", "\n", "# 40k/day is ~28 requests/minute, so the 5-minute entry never expires in\n", "# practice; the worst case is one write per 5-minute window all day.\n", "writes = 24 * 12\n", "reads = REQS_PER_DAY - writes\n", "\n", "uncached = REQS_PER_DAY * PREFIX * OPUS_IN\n", "cached = (writes * CACHE_WRITE + reads * CACHE_READ) * PREFIX * OPUS_IN\n", "\n", "print(f\"prefix cost/day uncached: ${uncached:8,.2f}\")\n", "print(f\"prefix cost/day cached: ${cached:8,.2f}\")\n", "print(f\"reduction on the prefix: {1 - cached / uncached:.0%}\")\n", "print(f\"annualized saving: ${(uncached - cached) * 365:,.0f}\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Roughly a 90% reduction on the 8k prefix cost (about $1,600/day down to about $175/day, on the order of half a million dollars a year), and it arrives with a latency improvement rather than a latency price, because the cached prefix is not reprocessed and time-to-first-token drops. This is why caching is the exam's favorite lever: it is the rare change that moves cost and latency in the same direction. That annualized figure is the number for the CFO, stated with the assumption that makes it honest: the prefix must stay byte-stable.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 4: Choose models per stage\n", "\n", "Route a cheap first stage (classify the ticket's intent and whether it is in scope) to Haiku 4.5, and send only tickets needing genuine policy synthesis to Opus 4.8. Do not route the whole volume to Opus, and do not route the hard synthesis to Haiku to save money; that is the always-wrong smallest-model answer.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Routed pipeline vs all-Opus, per day.\n", "HAIKU_IN, HAIKU_OUT = 1.00 / 1e6, 5.00 / 1e6\n", "OPUS_IN, OPUS_OUT = 5.00 / 1e6, 25.00 / 1e6\n", "CACHE_READ = 0.1\n", "\n", "REQS = 40_000\n", "HARD_RATE = 0.30 # tickets that need genuine policy synthesis\n", "\n", "# Stage 1: Haiku triages EVERY ticket. Intent + in-scope classification\n", "# needs the question, not the 8k policy corpus.\n", "stage1 = REQS * (600 * HAIKU_IN + 30 * HAIKU_OUT)\n", "\n", "# Stage 2: Opus answers the hard subset over the cached policy prefix.\n", "hard = int(REQS * HARD_RATE)\n", "per_hard = (8_000 * CACHE_READ * OPUS_IN # cached policy read\n", " + 400 * OPUS_IN # the question itself\n", " + 500 * OPUS_OUT) # the grounded answer\n", "stage2 = hard * per_hard\n", "\n", "routed = stage1 + stage2\n", "all_opus = REQS * per_hard\n", "\n", "print(f\"routed (Haiku triage + Opus on the hard {HARD_RATE:.0%}): ${routed:7,.2f}/day\")\n", "print(f\"all-Opus: ${all_opus:7,.2f}/day\")\n", "print(f\"routing saves {1 - routed / all_opus:.0%} on top of caching\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The routing discipline, not a blanket cost rule: Haiku takes the well-specified high-volume work because nothing cheaper exists and the task is fully specified; Opus takes the synthesis because a wrong policy answer gets retried, escalated, or shipped and corrected, and cost per *correct* output is the metric. The two always-wrong answers are the mirror images: all-Opus overpays on 70% of the volume, and all-Haiku produces weak resolutions whose cost per correct output is worse than what it saved. Easy tickets Haiku can answer from its triage context close in stage 1 and never pay for the policy prefix at all.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 5: Watch the context ceiling\n", "\n", "If any stage must load a much larger policy corpus (say 300K tokens of retrieved regulation), check the context ceiling before the price sheet. Haiku's window is the odd one out.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "| Model | Context window | Carries a 300K-token stage? |\n", "|---|---|---|\n", "| Haiku 4.5 | **200K** | No. The request fails; price is irrelevant. |\n", "| Sonnet 5 | 1M | Yes |\n", "| Opus 4.8 | 1M | Yes |\n", "\n", "The trap in exam form: a scenario feeds a stage 300K or 400K tokens and offers Haiku as the cost-optimized answer. Haiku is not the weaker choice there; it is **disqualified on capacity**, because 300K does not fit a 200K window. Check the ceiling before the price sheet, and remember the architect's counter-question: should any stage be carrying 300K tokens at all, or should retrieval be narrowing it first? The 1M window is a budget, not a target.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 6: Write the escalation rule, then enforce it at egress\n", "\n", "The five-part prompt needs its escalation clause, and the clause lives in a soft prompt: the model usually follows it and can be talked out of it. The guarantee lives in a hard validator at egress. Write both.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The system prompt, all five parts, with the escalation clause as the fifth:\n", "\n", "> **Role:** You are a support agent for Acme's customer-service team.\n", "> **Task:** Answer the customer's question using only the policy document above.\n", "> **Constraints:** Never quote internal policy section numbers; never promise actions the policy does not authorize.\n", "> **Output contract:** Return JSON: `{\"answer\": str}` or `{\"escalate\": true, \"reason\": str}`. No other keys, no prose outside the JSON.\n", "> **Escalation rule:** If the question is not answerable from the policy above, or asks for a refund over $500, do not answer; return `{\"escalate\": true, \"reason\": ...}` so a human agent takes it.\n", "\n", "That clause is advisory. The enforcement is code the model cannot argue with:\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import json\n", "\n", "ALLOWED_KEYS = {\"answer\", \"escalate\", \"reason\"}\n", "\n", "\n", "def enforce_egress(raw: str) -> dict:\n", " \"\"\"The prompt asks; this validator guarantees.\n", "\n", " Nothing malformed, and no unescalated out-of-scope answer, can reach\n", " the customer, no matter what the model was talked into.\n", " \"\"\"\n", " reply = json.loads(raw) # not JSON at all -> raises -> rejected\n", " if not isinstance(reply, dict) or not set(reply) <= ALLOWED_KEYS:\n", " raise ValueError(\"output contract violation: unexpected shape or keys\")\n", " if reply.get(\"escalate\"):\n", " return {\"route\": \"human_agent\",\n", " \"reason\": reply.get(\"reason\", \"unspecified\")}\n", " if \"answer\" not in reply:\n", " raise ValueError(\"neither an answer nor an escalation\")\n", " return {\"route\": \"customer\", \"answer\": reply[\"answer\"]}\n", "\n", "\n", "# The soft/hard split in action:\n", "print(enforce_egress('{\"escalate\": true, \"reason\": \"refund over $500\"}'))\n", "print(enforce_egress('{\"answer\": \"Opened items may be returned within 60 days.\"}'))\n", "try:\n", " enforce_egress('{\"answer\": \"...\", \"account_balance\": 1042.17}')\n", "except ValueError as e:\n", " print(\"rejected at egress:\", e)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Rejected outputs are retried once and then escalated to a human by default: fail closed, never fail open. In production the validator would be a real JSON Schema check, paired with `output_config={\"format\": {\"type\": \"json_schema\", ...}}` on the request so the model is constrained toward the contract and the validator still verifies it. The prompt asks; the schema constrains; the validator guarantees. That layering, soft instruction backed by hard egress control, is the item this domain tests hardest.\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.0" } }, "nbformat": 4, "nbformat_minor": 5 }