{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "*Lab 11 Answer Key*\n", "\n", "# Capstone: End-to-End Architecture Review\n", "\n", "**Objective:** audit the deliberately flawed Regulatory Change Assistant, name one defect per exam domain, and commit to each repair before reading the annotated findings.\n", "\n", "The system under review: a global bank's Regulatory Change Assistant. Analysts track publications across 14 jurisdictions (~300 documents/day); every assertion in an impact memo must be traceable to a source; retrieval must resolve exact clauses like 'Article 17(2)'; GDPR and a data-residency mandate apply; p95 must be under 30 seconds; and a wrong 'no impact' verdict is a regulatory finding. The team proposes: an autonomous agent with a broad tool set (including delete_policy), one union-permission service account, dense-vector retrieval, auto-filed 'no impact' verdicts, unredacted PII, average-latency plus uncalibrated-LLM-judge evaluation with no held-out set, a 99% live-traffic accuracy SLA, and secrets in CLAUDE.md.\n", "\n", "This notebook is the review performed. Each part is one domain: the finding, why it fails against the brief's own requirements, and the repair. Write your own finding for each domain before reading the worked answer; score one point per defect found AND correctly repaired.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 1: D1 Solution Design: agent vs. workflow\n", "\n", "The brief's own task description enumerates the steps: retrieve affected policies, draft a memo, route it. Review the control-flow decision against that fact before anything else.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Finding.** The design hands control flow to the model: an autonomous agent decides per turn which tools to call, on a path that never changes.\n", "\n", "**Why it fails.** The steps are fixed and enumerable (map change -> affected policies -> memo -> approval), which is the tell for a workflow. An agent here buys non-determinism, higher cost, higher latency, and an audit trail nobody can reconstruct, all to rediscover a sequence the team already wrote down. In a regulated process the un-reconstructable audit trail is the worst part: you cannot tell a regulator why a given memo was produced.\n", "\n", "**Repair.** A **workflow** whose classification step is a single augmented-LLM call over the retrieved policy terms, with code owning the sequence. One-line defense: when the path is known in advance, code owns the path and the model owns only the judgment inside each step. Reserve agents for the genuinely-unknown-path case, which this is not. (Labs 1 and 2.)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 2: D2 Prompting & Context: the cache can never hit\n", "\n", "The proposed call puts a datetime.now() preamble at the very top and the varying publication before the static 60K-token corpus, with the cache breakpoint downstream of both. Caching is a prefix match: any byte change before a breakpoint invalidates everything after it.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Finding.** Two invalidators sit in front of the corpus: a per-request timestamp at the top of `system`, and the varying publication placed before the static corpus. The `cache_control` breakpoint is downstream of both, so it never gets reused: the bank re-processes 60K tokens at full price 300 times a day and blows the p95 SLA that a warm cache would have protected.\n", "\n", "**Repair.** Stable content physically first (render order is tools -> system -> messages): the corpus behind the breakpoint, and the date and today's publication in the user turn, after it. The repaired call also makes the regulator's traceability rule a schema, which D5 will rely on:\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import anthropic\n", "\n", "client = anthropic.Anthropic()\n", "\n", "# Every assertion must carry a source. The schema makes that a hard contract:\n", "# an uncited claim cannot validate, so the rule is enforced by shape.\n", "IMPACT_SCHEMA = {\n", " \"type\": \"json_schema\",\n", " \"schema\": {\n", " \"type\": \"object\",\n", " \"properties\": {\n", " \"verdict\": {\"type\": \"string\", \"enum\": [\"impact\", \"no_impact\"]},\n", " \"affected_policies\": {\"type\": \"array\", \"items\": {\"type\": \"string\"}},\n", " \"cited_sources\": {\n", " \"type\": \"array\",\n", " \"items\": {\n", " \"type\": \"object\",\n", " \"properties\": {\n", " \"clause\": {\"type\": \"string\"}, # e.g. \"Article 17(2)\"\n", " \"quote\": {\"type\": \"string\"},\n", " },\n", " \"required\": [\"clause\", \"quote\"],\n", " \"additionalProperties\": False,\n", " },\n", " },\n", " },\n", " \"required\": [\"verdict\", \"affected_policies\", \"cited_sources\"],\n", " \"additionalProperties\": False,\n", " },\n", "}\n", "\n", "response = client.messages.create(\n", " model=\"claude-opus-4-8\",\n", " max_tokens=8000,\n", " thinking={\"type\": \"adaptive\"},\n", " output_config={\"effort\": \"high\", \"format\": IMPACT_SCHEMA},\n", " system=[\n", " # STABLE prefix first: the 60K corpus, cached across all 300 docs/day.\n", " {\"type\": \"text\", \"text\": POLICY_CORPUS,\n", " \"cache_control\": {\"type\": \"ephemeral\"}},\n", " ],\n", " messages=[\n", " # Everything volatile AFTER the breakpoint: the date and today's doc.\n", " {\"role\": \"user\",\n", " \"content\": build_user_turn(today_iso, incoming_publication)},\n", " ],\n", ")\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Prove the fix rather than assume it: read the usage back. input_tokens is\n", "# only the uncached remainder; the corpus must show up under the cache fields.\n", "u = response.usage\n", "print(\"written to cache :\", u.cache_creation_input_tokens) # ~1.25x, cold write\n", "print(\"served from cache:\", u.cache_read_input_tokens) # ~0.1x, warm hits\n", "print(\"uncached remainder:\", u.input_tokens) # today's doc only\n", "\n", "assert u.cache_read_input_tokens > 0, \"prefix still has a silent invalidator\"\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 3: D3 Integration: tool bloat, confused deputy, wrong retrieval\n", "\n", "Three defects hide in one row of the component table: the tool set, the identity model, and the retrieval choice. Judge each against the brief's requirement that queries like 'Article 17(2)' resolve exactly.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Finding 1: capability bloat.** The agent holds a `delete_policy` tool it never needs. An unused destructive tool is pure blast radius: it can only ever hurt you. **Repair:** delete it from the schema. The model cannot call what it was never given.\n", "\n", "**Finding 2: confused deputy.** One service account holds the union of every jurisdiction's permissions. A prompt-injected publication could steer the assistant to act with authority far beyond the task, across jurisdictions it should never touch. **Repair:** scope credentials to the jurisdiction of the document being processed, so even a fully hijacked turn cannot reach beyond its own lane.\n", "\n", "**Finding 3: wrong retrieval.** Dense-vector similarity answers 'what is *similar* to this?', but 'Article 17(2)' is a key, not a concept. Embedding it is architecture theatre that silently returns near-miss clauses. **Repair:** an exact clause lookup (a keyed index) for citation queries, keeping semantic search only for the genuinely fuzzy 'what else might this touch?' pass. (Labs 4 and 5.)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 4: D4 Evaluation: the wrong metrics and an uncalibrated judge\n", "\n", "Success is measured by average answer latency and an LLM-judge rubric never checked against human graders, with no held-out set. Review each instrument against what it is supposed to prove.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Finding.** Three evaluation defects compound:\n", "\n", "1. **Average latency hides the tail.** The SLA is p95 under 30 seconds; an average can look healthy while one memo in twenty times out.\n", "2. **The judge is uncalibrated.** A rubric scored by an LLM that was never checked against human graders is an opinion of unknown correlation with ground truth: a high score proves nothing about memo quality.\n", "3. **No held-out set.** Every prompt change is tuned against the same data it is scored on, so reported improvements are indistinguishable from overfitting.\n", "\n", "**Repair.** Track **p95**, not the mean. Calibrate the judge against a human-labelled sample and report the agreement rate before trusting it. Freeze a held-out regression set that no prompt change is allowed to see, and gate deploys on it. The citation requirement gives the evaluation seam for free: cited-span-exists is a deterministic check on every memo. (Labs 6 and 7.)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 5: D5 Governance: the auto-filed 'no impact' verdict\n", "\n", "The brief spends its words on one fact: a wrong 'no impact' verdict is a regulatory finding. Check which branch the design chose to automate, then review the PII and retention rows.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Finding.** The design automates precisely the decision that most needs a human: 'no impact' verdicts auto-file with no gate, on the exact branch whose wrong answer is a regulatory finding. Alongside it, PII goes to the model unredacted (widening the GDPR surface for no benefit) and outputs are retained indefinitely (a standing liability under residency and right-to-erasure rules).\n", "\n", "**Repair.**\n", "\n", "- **A named human on the 'no impact' path.** Auto-file, if anything, only the low-risk 'impact found and routed' cases, which analysts will see anyway. The gate is a governance constraint, not a UX preference, and no confidence score moves it.\n", "- **Redact PII before the call** where it is not needed for the decision, so it never enters the trace at all.\n", "- **Bound retention** to the residency requirement, with a data-flow inventory covering traces, caches, and the eval set so an erasure request is honorable.\n", "- The D2 schema is also a governance control: a required `cited_sources` field means an uncited memo cannot validate, enforcing the regulator's traceability rule by shape rather than by a polite instruction. (Lab 8.)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 6: D6 Stakeholder & Lifecycle: the unhonourable SLA, and the handover\n", "\n", "The team promised the regulator 99% accuracy on live traffic. Test the promise for measurability, then assemble the deliverable set a finished review hands over: the review is not done when the code is right.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Finding.** 'Accuracy on live traffic' has no denominator you control: the true label on a novel publication is unknown until an analyst rules on it, so the promised number cannot be measured in real time, let alone guaranteed. It also anchors the relationship to a figure one hard month will breach, converting a capability into a liability. A commitment made without an evaluation methodology that can support it is a lifecycle failure, not a marketing choice.\n", "\n", "**Repair.** Commit to what you can evidence: a measured accuracy on a frozen, human-labelled benchmark; a documented human gate on the high-risk branch; a monitored p95. Promise the process, not a live-traffic percentage. (Lab 9.)\n", "\n", "**The handover set.** The review is finished when the next engineer, the regulator, and the on-call responder can each do their job from what you leave behind. A missing row is a defect:\n", "\n", "| Artifact | Answers |\n", "|---|---|\n", "| Architecture diagram (five planes) | What the system does and where each control lives |\n", "| ADR log | Why the agent was rejected; why the keyed lookup replaced vectors |\n", "| Eval suite + frozen regression set | How we know a change did not regress accuracy |\n", "| Runbook | How on-call recovers when a memo fails to file |\n", "| Cost model (Part 8) | What 300 docs/day costs, cached vs. uncached |\n", "| Risk register | The residual risks after every repair, and who owns each |\n", "| On-call rotation | Who responds, and within what window |\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 7: D7 Developer Productivity: secrets in CLAUDE.md\n", "\n", "API keys and credentials are pasted into the project CLAUDE.md so the team shares one setup. Review that choice for a bank.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Finding.** `CLAUDE.md` is committed to the repository and loaded into context on every run; it is documentation, not a vault. Secrets there are exposed to everyone with repo access, leak into logs and traces on every request, and are near-impossible to rotate cleanly. For a bank this is an audit finding by itself, before any incident occurs.\n", "\n", "**Repair.** Move secrets to environment variables or a managed secret store, referenced from code and never from context. Keep `CLAUDE.md` for the operational knowledge it is good at: how to run the eval suite, where the runbook lives. (Lab 10.)\n", "\n", "**The through-line across all seven findings:** every repair *subtracts*. Delete the agent, delete the tool, narrow the credentials, gate the automated branch, drop the live-traffic promise, take the secrets out of context. When two options differ by a capability, the one that removes it is usually right.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 8: The cost model\n", "\n", "Do the arithmetic, because 'caching saves money' is not a number and a CFO does not fund adjectives. Illustrative rates: Opus 4.8 at $5.00/M input and $25.00/M output, cache reads ~0.1x input, cold writes 1.25x (5-minute TTL).\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Illustrative model, not a price quote. 60K-token corpus, 4K publication,\n", "# 2K memo, 300 docs/day; corpus re-warmed ~12x/day on a 5-minute TTL.\n", "OPUS_IN = 5.00 / 1_000_000\n", "OPUS_OUT = 25.00 / 1_000_000\n", "CORPUS, PUB, MEMO, DOCS = 60_000, 4_000, 2_000, 300\n", "\n", "uncached = CORPUS * OPUS_IN + PUB * OPUS_IN + MEMO * OPUS_OUT\n", "cached = CORPUS * OPUS_IN * 0.10 + PUB * OPUS_IN + MEMO * OPUS_OUT\n", "warm_writes = 12 * CORPUS * OPUS_IN * 1.25\n", "\n", "daily_uncached = uncached * DOCS\n", "daily_cached = cached * DOCS + warm_writes\n", "\n", "print(f\"per request, uncached (proposed): ${uncached:.3f}\")\n", "print(f\"per request, cached (repaired): ${cached:.3f}\")\n", "print(f\"daily, uncached: ${daily_uncached:.2f}\")\n", "print(f\"daily, cached : ${daily_cached:.2f} (incl. ${warm_writes:.2f} cache writes)\")\n", "print(f\"the D2 ordering defect was costing ${daily_uncached - daily_cached:.2f}/day,\")\n", "print(f\"about ${(daily_uncached - daily_cached) * 365 / 1000:.0f}K/year, for nothing\")\n", "\n", "# Cost per request is not cost per RESOLVED task. With 20% of memos needing a\n", "# second pass, ~360 calls resolve 300 business tasks:\n", "calls = DOCS * 1.2\n", "per_resolved = cached * calls / DOCS\n", "print(f\"cost per request : ${cached:.3f}\")\n", "print(f\"cost per resolved task: ${per_resolved:.3f}\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Roughly a 3x reduction, driven almost entirely by the corpus moving from full price to a tenth of it, and the same repair protects the 30-second p95 because the warm prefix is faster to first token: caching is the rare lever that moves cost and the SLA in the same direction. The CFO cares about the last number, cost per **resolved** task, because it is the one tied to a completed unit of work; report that, not cost per request.\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.0" } }, "nbformat": 4, "nbformat_minor": 5 }