{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "*Lab 5 Self-Driven Lab*\n", "\n", "# Integration Protocols, Auth & Observability\n", "\n", "**Objective:** harden a multi-tenant agent platform against the four failure modes of Domain 3: capability bloat, broken authorization, the wrong integration protocol, and blind observability.\n", "\n", "## Challenge Outline\n", "\n", "Build a complete notebook that demonstrates the following outcomes:\n", "\n", "- **Capability-bloat audit:** list every tool on the agent, name the tenant task each one serves, delete the ones you cannot name, and write the one-sentence justification that kills the rest as least privilege.\n", "- **Per-tenant authorization:** replace the shared service account with per-tenant identity, authorize each tool call in code on the real arguments against that tenant's grants, and state exactly where the check lives for the invoice tool.\n", "- **Protocol per integration:** choose MCP, direct API/CLI, or agent-to-agent for the CRM read, the calendar write, and invoice issuance, defending each choice with reuse, ownership, trust boundary, and discoverability.\n", "- **Reconcile the SLO:** assign a lever to the interactive invoice path and the nightly CRM sync, and name the lever that is 50% cheaper and completely wrong for the interactive path.\n", "- **Trace schema:** write the exact per-turn trace record, including request_id, the three-way token split, stop reason, per-tenant cost, and a tool-error flag, and map every field to up, correct, or affordable.\n", "- **Sampling policy:** define what you sample, redact, and aggregate, name the exemplars you always keep, and state what becomes unanswerable and why that trade is acceptable.\n", "\n", "Your solution should include enough code, output, or written observations to prove each outcome worked. Keep the notebook focused on final behavior and evidence rather than a guided walkthrough.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Student Workspace\n", "\n", "Use the sections below to build your solution. Each section maps to one required outcome from the challenge outline.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Part 1: Capability-bloat audit\n", "\n", "list every tool on the agent, name the tenant task each one serves, delete the ones you cannot name, and write the one-sentence justification that kills the rest as least privilege.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Part 1: Capability-bloat audit\n", "# Add your implementation, outputs, or notes here.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Part 2: Per-tenant authorization\n", "\n", "replace the shared service account with per-tenant identity, authorize each tool call in code on the real arguments against that tenant's grants, and state exactly where the check lives for the invoice tool.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Part 2: Per-tenant authorization\n", "# Add your implementation, outputs, or notes here.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Part 3: Protocol per integration\n", "\n", "choose MCP, direct API/CLI, or agent-to-agent for the CRM read, the calendar write, and invoice issuance, defending each choice with reuse, ownership, trust boundary, and discoverability.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Part 3: Protocol per integration\n", "# Add your implementation, outputs, or notes here.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Part 4: Reconcile the SLO\n", "\n", "assign a lever to the interactive invoice path and the nightly CRM sync, and name the lever that is 50% cheaper and completely wrong for the interactive path.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Part 4: Reconcile the SLO\n", "# Add your implementation, outputs, or notes here.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Part 5: Trace schema\n", "\n", "write the exact per-turn trace record, including request_id, the three-way token split, stop reason, per-tenant cost, and a tool-error flag, and map every field to up, correct, or affordable.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Part 5: Trace schema\n", "# Add your implementation, outputs, or notes here.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Part 6: Sampling policy\n", "\n", "define what you sample, redact, and aggregate, name the exemplars you always keep, and state what becomes unanswerable and why that trade is acceptable.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Part 6: Sampling policy\n", "# Add your implementation, outputs, or notes here.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Verification Notes\n", "\n", "Summarize the evidence that each part worked. Capture API signals, validation outcomes, errors, recovery behavior, cost observations, or comparisons required by this lab.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Verification notes\n", "# Record the evidence that proves each lab outcome worked.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "## Answer Key\n", "\n", "The cells below contain the completed reference implementation/content for this lab. Use this section only after attempting the self-driven lab.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Lab 5 Answer Key*\n", "\n", "# Integration Protocols, Auth & Observability\n", "\n", "**Objective:** harden a multi-tenant agent platform against the four failure modes of Domain 3: capability bloat, broken authorization, the wrong integration protocol, and blind observability.\n", "\n", "This is a design lab. The scenario: a multi-tenant agent platform where each tenant's agent needs three integrations (read the tenant's CRM, write to the tenant's calendar, issue invoices against the tenant's billing account). Today every agent runs as one shared service account, holds those three capabilities plus a dozen more inherited from a template, authenticates once at session start, and logs nothing but HTTP status codes. Volume is outgrowing full-prompt logging. Most parts are worked design artifacts; code appears only where the lab itself shows it (the authorization check and the trace record).\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 1: Capability-bloat audit\n", "\n", "List every tool on the agent. For each, name the tenant task it serves. Delete the ones you cannot name, and justify each deletion as least privilege, not cleanup. Remember the second reason to prune: more tools measurably degrade tool-selection accuracy, so bloat is a correctness problem as well as a security one.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Artifact: the capability audit.** Fifteen tools are wired in; the tenant workflow names exactly three tasks.\n", "\n", "| Tool | Tenant task it serves | Verdict |\n", "| --- | --- | --- |\n", "| `crm.read_records` | \"Answer questions from my CRM\" | Keep |\n", "| `calendar.create_event` | \"Book the follow-up meeting\" | Keep |\n", "| `billing.issue_invoice` | \"Issue an invoice against my account\" | Keep |\n", "| `crm.write_records`, `crm.delete_records` | None: agents answer from the CRM, they do not edit it | Delete |\n", "| `calendar.delete_event`, `calendar.read_all` | None: the task is to add events, not manage or read every calendar | Delete |\n", "| `billing.refund`, `billing.void_invoice` | None: no tenant workflow refunds or voids through the agent | Delete |\n", "| `email.send`, `sms.send` | None: notifications are the platform's job, not the agent's | Delete |\n", "| `files.delete`, `files.share_external` | None: inherited from the template, never invoked | Delete |\n", "| `admin.manage_users`, `admin.rotate_keys` | None: administrative surface an agent must never hold | Delete |\n", "\n", "**The sentence that kills the twelve:** if no tenant task names this tool, the tool is not a capability; it is attack surface plus a tool-selection distractor, and least privilege deletes it rather than logging it, prompting around it, or upgrading the model behind it.\n", "\n", "**Why deletion and not monitoring:** logging is detective (you learn after the account is gone), a confirmation prompt is compensating (the capability is still one injection away from firing), and a smarter model is unrelated to scope. Only removal changes what the agent can do. The bonus is correctness: a three-tool agent chooses among three schemas instead of fifteen, so it picks the wrong action less often.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 2: Per-tenant authorization\n", "\n", "The shared service account is a confused deputy: any tenant's agent can reach any tenant's data, because the model, not the identity layer, is the only thing deciding who sees what. Redesign so each agent acts with the requesting tenant's permissions, checked in code on the real arguments, never in the prompt.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**The redesign, as principles.**\n", "\n", "1. **Propagate the tenant's identity.** The agent authenticates and acts as the requesting tenant, never as an omnipotent service account. The identity is bound at request-assembly time; the model cannot supply, forge, or override it.\n", "2. **Authorize every call, not just the session.** Access checked once at login and never again is a gap; the check runs inside each tool, on the actual arguments, on every invocation.\n", "3. **Scope the tokens.** Short-lived, audience-scoped tokens per tenant per integration; verify audience on receipt; rotate. A stolen unbounded token works forever, anywhere.\n", "4. **Keep secrets out of the context.** Credentials are injected at the transport layer. Prompts are logged, cached, and replayed; a key pasted into one is already leaked.\n", "\n", "**Where the check lives for the invoice tool:** inside the tool implementation, at the line that compares the billing account's owner to the requesting tenant, before any money moves.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import anthropic\n", "from anthropic import beta_tool\n", "\n", "client = anthropic.Anthropic()\n", "\n", "# The requesting tenant's identity is bound at request-assembly time,\n", "# NOT supplied by the model. The model cannot forge or override it.\n", "def make_tools(tenant):\n", "\n", " @beta_tool\n", " def issue_invoice(account_id: str, amount_cents: int) -> str:\n", " \"\"\"Issue an invoice against a billing account.\n", "\n", " Args:\n", " account_id: The billing account to invoice.\n", " amount_cents: Invoice amount in cents.\n", " \"\"\"\n", " account = db.billing_accounts.get(account_id)\n", " # Authorization is enforced HERE, in code, on the real\n", " # arguments, against the requesting tenant's grants.\n", " if account is None or account.tenant_id != tenant.id:\n", " # Do not leak existence. Same answer for \"missing\" and \"forbidden\".\n", " return \"No billing account with that ID is available to you.\"\n", " return billing.issue(account, amount_cents)\n", "\n", " return [issue_invoice] # plus the scoped CRM-read and calendar-write tools\n", "\n", "# The agent ACTS AS `tenant`, never as a shared service account.\n", "# The model chooses which tool and what arguments; this code decides\n", "# whether this tenant may have that result. Never the other way around.\n", "runner = client.beta.messages.tool_runner(\n", " model=\"claude-opus-4-8\",\n", " max_tokens=4096,\n", " tools=make_tools(current_tenant),\n", " messages=[{\"role\": \"user\", \"content\": user_request}],\n", ")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 3: Protocol per integration\n", "\n", "Pick a mechanism per integration and defend it with the four criteria: reuse across clients, who owns the tool, whether a trust boundary is crossed, and whether discoverability is needed. Say which integration, if any, actually crosses a trust boundary, and which one an MCP server would over-engineer.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Artifact: the protocol decision table.**\n", "\n", "| Integration | Mechanism | Reuse | Ownership | Trust boundary | Discoverability |\n", "| --- | --- | --- | --- | --- | --- |\n", "| CRM read | **MCP** | Every tenant's agent, plus internal reporting clients, consume the same CRM tool surface: N-to-M | The platform team owns one server that many clients consume | None crossed | Needed: new clients discover the tools instead of re-integrating by hand |\n", "| Calendar write | **Direct API** | One consumer, one call shape | We own this integration end to end | None crossed | Not needed: discovery machinery is pure overhead here |\n", "| Invoice issuance | **Direct API, tightly scoped** | One consumer | We own the client side; the billing provider owns the far side | The closest thing to one: money moves against the tenant's billing account | Not needed |\n", "\n", "**The two named answers.** The calendar write is the bespoke integration an MCP server would over-engineer: a single consumer, a single owner, no reuse story, so wrapping it in a protocol plus a server to run and secure adds cost with no payoff. Invoice issuance is the one that approaches a trust boundary, because money moves against an account the billing provider ultimately controls, but the far side is a plain API, not an agent. Agent-to-agent would add a coordination protocol and a second audit trail with no agent to coordinate with; it becomes the right answer only if the billing side ships its own agentic system. When the scenario is one team, one system, one owner, agent-to-agent is the distractor.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 4: Reconcile the SLO\n", "\n", "Invoice issuance must feel instant to the user but need not be cheap; the nightly CRM sync must be cheap but no one is waiting on it. Justify each configuration as a three-part sentence: the SLO, the lever, and the cost of the lever.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Artifact: one justified sentence per path.**\n", "\n", "| Path | SLO | Lever | The cost of the lever |\n", "| --- | --- | --- | --- |\n", "| Invoice issuance (interactive) | Feels instant: low time-to-first-token; correctness over cheapness | **Streaming** for perceived latency, plus **prompt caching** on the stable tenant-policy prefix | Streaming moves perceived latency only, not cost or total latency; caching requires the stable prefix to sit before anything that varies, or it caches nothing |\n", "| Nightly CRM sync (offline) | Cheap; complete by morning; nobody is waiting | **Batches API** (50% of standard token cost) on a cheap, fast tier (`claude-haiku-4-5`) | Latency destroyed (up to a 24-hour window) and single-turn only, which this workload does not care about; results return UNORDERED, so key every request by `custom_id`, never by position |\n", "\n", "**The lever I will not use on the interactive path, and why:** the Batches API. It is 50% cheaper, which makes it the tempting answer, and it is completely wrong here because a user is waiting: the batch window is up to 24 hours, and no invoice screen survives that. Batching is the classic trap of a true fact (\"half price\") attached to the wrong requirement (anything interactive).\n", "\n", "**The written-out justification for the invoice path:** the SLO is instant feel at invoice time; the levers are streaming (moves time-to-first-token) and prompt caching on the tenant policy prefix (moves cost and latency together, the rare two-for-one); the cost is prefix discipline, because anything variable placed before the cached corpus silently voids the cache.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 5: Trace schema\n", "\n", "Write the exact fields you emit per turn, and state which of the three observability questions (is it up, is it correct, is it affordable) each field serves. The token usage must be split three ways: input_tokens is the uncached remainder only, so naive cost math undercounts unless you sum all three.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Artifact: field-to-question mapping.**\n", "\n", "| Field | Question it serves |\n", "| --- | --- |\n", "| `request_id` | Up and correct: the join key for tracing any failure with Anthropic |\n", "| `stop_reason` | Correct: a `max_tokens` spike means truncation; a `refusal` spike means a content or prompt problem |\n", "| `tool_error` flag | Correct: a tool failing 5% of calls is invisible in aggregate latency but corrodes correctness |\n", "| `input_tokens` + `cache_read_input_tokens` + `cache_creation_input_tokens` | Affordable: only the three-way sum is the true prompt size; `input_tokens` alone is the uncached remainder |\n", "| `output_tokens` | Affordable |\n", "| `tenant_id` (keys per-tenant cost) | Affordable: the number the CFO asks for, and the only way to catch one tenant's runaway loop |\n", "| `latency_ms` | Up: feeds the p95/p99 alerting |\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import json, logging\n", "\n", "logger = logging.getLogger(\"agent.trace\")\n", "\n", "def emit_trace(response, tenant_id, turn_id, latency_ms, tool_error=False):\n", " u = response.usage\n", " record = {\n", " \"turn_id\": turn_id,\n", " \"tenant_id\": tenant_id, # affordable (per-tenant cost)\n", " # Public despite the underscore: this is what Anthropic traces on.\n", " \"request_id\": response._request_id, # up + correct\n", " \"model\": response.model,\n", " \"stop_reason\": response.stop_reason, # correct\n", " \"tool_error\": tool_error, # correct\n", " \"latency_ms\": latency_ms, # up\n", " # Split the usage out. input_tokens is the UNCACHED remainder only;\n", " # summing all three gives the true prompt size and the true cost.\n", " \"input_tokens\": u.input_tokens, # affordable\n", " \"cache_read_input_tokens\": u.cache_read_input_tokens,\n", " \"cache_creation_input_tokens\": u.cache_creation_input_tokens,\n", " \"output_tokens\": u.output_tokens, # affordable\n", " }\n", " # A metric record (cheap, aggregate): NOT the full prompt (PII + volume).\n", " logger.info(json.dumps(record))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 6: Sampling policy\n", "\n", "You cannot log full prompts at this volume: that is PII you must protect and storage you cannot afford. Define what you sample, what you redact, what you aggregate, and which exemplars you always keep. Then state what you will be unable to answer as a result, and why that is an acceptable trade.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Artifact: the three-tier sampling policy.**\n", "\n", "| Tier | Coverage | Policy |\n", "| --- | --- | --- |\n", "| Metrics | 100% of requests | Aggregate the Part 5 trace record for every turn: latency percentiles, stop-reason counts, tool error rates, three-way token sums per tenant. Cheap, PII-free, and the alerting layer |\n", "| Traces | Sampled | Full redacted traces for 2% of ordinary traffic, chosen randomly per tenant so no tenant is unobserved. Redaction: strip user content, keep structure, token counts, tool names, and argument shapes (never argument values) |\n", "| Exemplars | 100% of the interesting classes | Always keep, redacted: every `refusal`, every tool error, every authorization denial from Part 2, and every `max_tokens` truncation. These are the cases debugging and the eval pipeline (Lab 6) feed on |\n", "\n", "**What becomes unanswerable:** for the ~98% of ordinary requests outside the sample, we cannot replay the exact prompt, so a one-off question like \"why did this specific benign turn phrase its answer oddly last Tuesday\" has no full-fidelity answer.\n", "\n", "**Why the trade is acceptable:** that question belongs to the correctness tier, and correctness is measured by evals over labelled datasets, not by warehousing every prompt. Every failure class that indicates a real defect (refusals, tool errors, authz denials, truncations) is retained at 100%, aggregate metrics still see every request, and the alternative is a PII store we must secure, govern, and eventually explain to a regulator. We keep the signal, we drop the liability.\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.0" } }, "nbformat": 4, "nbformat_minor": 5 }