{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Lab 1 Self-Driven Lab*\n",
    "\n",
    "# Solution Design & Architecture\n",
    "\n",
    "**Objective:** produce a one-page architecture for the discharge-summary scenario and defend the pattern choice against the two alternatives.\n",
    "\n",
    "## Challenge Outline\n",
    "\n",
    "Build a complete notebook that demonstrates the following outcomes:\n",
    "\n",
    "- **Run the five-step sequence:** write one sentence per step (decision, value pillar, ground truth, determinism, blast radius) and do not proceed until step 3 has an answer.\n",
    "- **Choose the pattern and kill the other two:** name the pattern, then write the sentence that kills each alternative: 'this is not agentic because ___' and 'this is not a bare augmented LLM call because ___'.\n",
    "- **Draw the five planes:** for each of ingress, context, reasoning, egress, and feedback, name one concrete decision, including what happens to patient identifiers and what the physician actually sees.\n",
    "- **Reconcile the SLA with the model choice:** rank streaming, caching, model tier, and effort against the 20-second bound, note which degrade quality, and name the one lever that moves latency and cost together.\n",
    "- **Design the feedback loop before you build:** specify exactly what is persisted from the physician's pre-signature edit, and what must not be persisted under HIPAA without a BAA and a retention policy.\n",
    "- **Defend the cost:** estimate input, output, and cache economics for 900 drafts a day, state the pillar and the number you would put in front of a CFO, and state what would falsify it.\n",
    "\n",
    "Your solution should include enough code, output, or written observations to prove each outcome worked. Keep the notebook focused on final behavior and evidence rather than a guided walkthrough.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Student Workspace\n",
    "\n",
    "Use the sections below to build your solution. Each section maps to one required outcome from the challenge outline.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 1: Run the five-step sequence\n",
    "\n",
    "write one sentence per step (decision, value pillar, ground truth, determinism, blast radius) and do not proceed until step 3 has an answer.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Part 1: Run the five-step sequence\n",
    "# Add your implementation, outputs, or notes here.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 2: Choose the pattern and kill the other two\n",
    "\n",
    "name the pattern, then write the sentence that kills each alternative: 'this is not agentic because ___' and 'this is not a bare augmented LLM call because ___'.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Part 2: Choose the pattern and kill the other two\n",
    "# Add your implementation, outputs, or notes here.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 3: Draw the five planes\n",
    "\n",
    "for each of ingress, context, reasoning, egress, and feedback, name one concrete decision, including what happens to patient identifiers and what the physician actually sees.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Part 3: Draw the five planes\n",
    "# Add your implementation, outputs, or notes here.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 4: Reconcile the SLA with the model choice\n",
    "\n",
    "rank streaming, caching, model tier, and effort against the 20-second bound, note which degrade quality, and name the one lever that moves latency and cost together.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Part 4: Reconcile the SLA with the model choice\n",
    "# Add your implementation, outputs, or notes here.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 5: Design the feedback loop before you build\n",
    "\n",
    "specify exactly what is persisted from the physician's pre-signature edit, and what must not be persisted under HIPAA without a BAA and a retention policy.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Part 5: Design the feedback loop before you build\n",
    "# Add your implementation, outputs, or notes here.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 6: Defend the cost\n",
    "\n",
    "estimate input, output, and cache economics for 900 drafts a day, state the pillar and the number you would put in front of a CFO, and state what would falsify it.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Part 6: Defend the cost\n",
    "# Add your implementation, outputs, or notes here.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verification Notes\n",
    "\n",
    "Summarize the evidence that each part worked. Capture API signals, validation outcomes, errors, recovery behavior, cost observations, or comparisons required by this lab.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Verification notes\n",
    "# Record the evidence that proves each lab outcome worked.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Answer Key\n",
    "\n",
    "The cells below contain the completed reference implementation/content for this lab. Use this section only after attempting the self-driven lab.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Lab 1 Answer Key*\n",
    "\n",
    "# Solution Design & Architecture\n",
    "\n",
    "**Objective:** produce a one-page architecture for the discharge-summary scenario and defend the pattern choice against the two alternatives.\n",
    "\n",
    "This is a design lab, not a coding lab: the deliverable is a defensible architecture, and most parts produce written artifacts rather than code.\n",
    "\n",
    "**Scenario.** A hospital network wants to reduce the time clinicians spend writing discharge summaries. Today a physician dictates notes; a transcriptionist produces a draft; the physician edits and signs. Volume is 900 discharges a day across 11 facilities. Summaries must be signed by the attending physician. Records are PHI under HIPAA. Physicians will abandon any tool that takes more than 20 seconds to produce a draft.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: Run the five-step sequence\n",
    "\n",
    "Start at the decision the business wants changed, not at the model. Name the decision, the pillar, the ground truth, whether the steps are enumerable, and the blast radius. If you cannot say how you would know a summary was good, you cannot build this.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Step | Answer for the discharge-summary system |\n",
    "|---|---|\n",
    "| 1. Decision | Produce the first draft of a discharge summary from the chart and the dictated notes, so the attending physician edits and signs instead of waiting on a transcriptionist. 900 times a day across 11 facilities. |\n",
    "| 2. Value pillar | Primarily **efficiency**: transcriptionist hours removed and physician minutes per summary reduced. Secondarily a **performance SLA**: the draft must appear in under 20 seconds or physicians abandon the tool. Not transformation; discharge summaries already exist. |\n",
    "| 3. Ground truth | **The signed summary.** The physician edits the draft before signing, so every edit is a labelled correction. Edit distance between draft and signed version is a measurable quality signal, and a summary the physician signs with near-zero edits was a good draft. Ground truth exists, so the project is evaluable. Proceed. |\n",
    "| 4. Determinism | The steps are enumerable and identical for every discharge: gather the chart and the dictation, assemble the prompt, generate the draft, present it for edit and signature. No step's existence depends on a previous step's output. |\n",
    "| 5. Blast radius | A clinical error that survives the physician's review and lands in a signed record can harm a patient; a PHI leak is a HIPAA event. The signature requirement bounds the first; ingress handling and the BAA bound the second. |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2: Choose the pattern and kill the other two\n",
    "\n",
    "Force yourself to say the kill sentences out loud. This is the step the exam tests: the ladder is augmented LLM, then workflow, then agentic, and you climb only when the rung below cannot hold the requirement.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Pattern: a workflow whose generation step is one augmented LLM call.** Application code owns the fixed sequence (fetch chart, assemble context, generate, present for signature); the model owns exactly one step inside it.\n",
    "\n",
    "The kill sentences:\n",
    "\n",
    "- **This is not agentic because** the path is identical for every discharge: gather, generate, present. Nothing about the sequence is unknown until the work begins, so handing control flow to the model buys non-determinism, cost, and latency against a 20-second SLA, and returns nothing. It also fails the four-part agent test at the first criterion (complexity: the task is fully specifiable in advance).\n",
    "- **This is not a bare augmented LLM call because** the system has fixed pre- and post-steps that belong in application code: chart retrieval by encounter ID, PHI handling at ingress, and above all the signature gate at egress. The regulation makes the gate an architectural requirement, and a bare call has nowhere to put it.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3: Draw the five planes\n",
    "\n",
    "A production architecture has five planes: ingress, context, reasoning, egress, feedback. If your diagram is missing one, an exam item is waiting for it. Name one concrete decision per plane for this system.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Plane | Decision | Because |\n",
    "|---|---|---|\n",
    "| Ingress | Chart sections and dictation enter through the EHR integration under a BAA. Direct identifiers not needed for drafting (SSN, address, insurance ID) are stripped or tokenized before the model call; clinical content stays. | Minimizes the PHI surface without degrading the draft. HIPAA compliance is an ingress property, not a prompt instruction. |\n",
    "| Context | The encounter's chart is fetched **by encounter ID**: a lookup, not a vector search. Stable instructions and the format contract render first; the varying chart and dictation render last. | The query pattern is an exact key, and stable-first ordering is what makes the prefix cacheable (Lab 3). |\n",
    "| Reasoning | One model call with structured output: required sections (diagnoses, hospital course, medications, follow-ups) as a schema. Start on Sonnet 5; promote to Opus only if the edit-distance eval says Sonnet does not clear the bar. | The schema makes the draft checkable at egress, and routing starts at the cheapest model that clears the quality bar. |\n",
    "| Egress | The draft is **never auto-filed**. It renders in the editor for the attending physician, who edits and signs; the signature is the gate, on every summary. Schema validation rejects malformed drafts before a physician ever sees them. | The regulation says signed by the attending. That is an architectural constraint, not a UX preference. |\n",
    "| Feedback | Persist the draft, the signed final, the structured diff between them, and the model and prompt version, inside the covered environment. | The edit is free labelled data, and it is unrecoverable if not captured on day one. |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 4: Reconcile the SLA with the model choice\n",
    "\n",
    "Twenty seconds is a hard bound. Which of streaming, caching, model tier, and effort move it, which degrade quality, and which one moves latency and cost together?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Lever | Moves the 20-second bound? | Quality cost |\n",
    "|---|---|---|\n",
    "| Streaming | Yes, decisively: the physician sees the draft forming within a second or two, and perceived latency is time-to-first-token, not time-to-last. | None. First lever to pull. |\n",
    "| Prompt caching | Yes: the stable instruction prefix is not reprocessed, cutting time-to-first-token **and** per-request cost. | None. Requires stable-first prompt assembly (Part 3). |\n",
    "| Model tier | Down-tiering to Haiku is fast but a discharge summary is nuanced synthesis over a long chart, and long charts can also pinch Haiku's 200K window. Sonnet 5 is the defensible middle. | Real. Route down only if the edit-distance eval stays flat. |\n",
    "| Effort | Lowering `output_config.effort` cuts tokens and latency. Start at `medium` and tune against the eval. | Some depth loss; measurable, so measure it. |\n",
    "\n",
    "The lever that moves **latency and cost together** is caching. Streaming moves perceived latency only; a smaller model and lower effort trade quality for both. The design therefore leads with streaming plus caching, which cost nothing in quality, before touching tier or effort.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 5: Design the feedback loop before you build\n",
    "\n",
    "The physician edits the draft before signing, and that edit is a labelled correction. Write down exactly what you persist, and note what you must not persist under HIPAA without a BAA and a retention policy.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Persist (inside the covered environment, under the BAA, with a retention policy):**\n",
    "\n",
    "- the model draft and the signed final, keyed by encounter ID\n",
    "- the structured diff between them (the labelled correction)\n",
    "- model ID, prompt version, effort level, and timestamps, so a drift in edit distance can be attributed to a change\n",
    "- time-to-draft and time-to-signature, the SLA and adoption telemetry\n",
    "\n",
    "**Do not persist** duplicate copies of the chart outside the source EHR, any of this data in a vendor or analytics system not covered by the BAA, or anything beyond the retention window. De-identify before any aggregate analysis leaves the covered environment.\n",
    "\n",
    "The loop this buys: mean edit distance per facility per week is the quality metric; heavily edited drafts flow into the eval set with their corrections; a rising edit distance is the early warning of drift and fires before physicians start abandoning the tool. Design this on day one, because the diff cannot be reconstructed later.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 6: Defend the cost\n",
    "\n",
    "900 drafts a day. Estimate input tokens (the chart), output tokens (the summary), and the cache hit rate, then state the pillar you are claiming, the number you would put in front of a CFO, and what would falsify it.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cost model: 900 drafts/day on Sonnet 5 ($3 in / $15 out per 1M tokens).\n",
    "SONNET_IN = 3.00 / 1_000_000\n",
    "SONNET_OUT = 15.00 / 1_000_000\n",
    "CACHE_READ, CACHE_WRITE = 0.1, 1.25   # multipliers on input price\n",
    "\n",
    "DRAFTS_PER_DAY = 900\n",
    "CHART_TOKENS = 15_000    # varying chart + dictated notes (never cacheable)\n",
    "PREFIX_TOKENS = 5_000    # stable instructions + format contract (cached)\n",
    "OUTPUT_TOKENS = 1_200    # the draft summary\n",
    "\n",
    "# 900/day over a ~12h clinical day is >1 request per 5-minute TTL window,\n",
    "# so the cache stays warm; worst case is one write per window.\n",
    "writes = 12 * 12\n",
    "prefix_uncached = DRAFTS_PER_DAY * PREFIX_TOKENS * SONNET_IN\n",
    "prefix_cached = (writes * CACHE_WRITE\n",
    "                 + (DRAFTS_PER_DAY - writes) * CACHE_READ) * PREFIX_TOKENS * SONNET_IN\n",
    "\n",
    "chart_cost = DRAFTS_PER_DAY * CHART_TOKENS * SONNET_IN\n",
    "output_cost = DRAFTS_PER_DAY * OUTPUT_TOKENS * SONNET_OUT\n",
    "\n",
    "daily = prefix_cached + chart_cost + output_cost\n",
    "print(f\"prefix/day:  ${prefix_uncached:6.2f} uncached -> ${prefix_cached:5.2f} cached\")\n",
    "print(f\"chart/day:   ${chart_cost:6.2f}   output/day: ${output_cost:6.2f}\")\n",
    "print(f\"total/day:   ${daily:6.2f}   per draft: ${daily / DRAFTS_PER_DAY:.4f}\")\n",
    "\n",
    "# The baseline being displaced: transcription at roughly $10 per summary.\n",
    "baseline = DRAFTS_PER_DAY * 10.00\n",
    "print(f\"transcription baseline/day: ${baseline:,.0f}\")\n",
    "print(f\"annualized model spend: ${daily * 365:,.0f} vs baseline ${baseline * 365:,.0f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**The pillar claimed is efficiency**, with the 20-second SLA as a constraint, not the pillar. The CFO number: model spend on the order of $60 to $70 a day (about $25k a year) against a transcription baseline near $9,000 a day (about $3.3M a year), plus recovered physician minutes per discharge. Even if the token estimates are off by 3x, the claim survives.\n",
    "\n",
    "**What would falsify it:** the edit-distance eval showing physicians spend as long correcting the draft as they previously spent editing the transcriptionist's (efficiency claim dies), draft latency exceeding 20 seconds at p95 (adoption dies), or the diff telemetry showing rising edits over time (quality drift). Each falsifier is measurable because Part 5 captured the data.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}