{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Lab 7 Self-Driven Lab*\n",
    "\n",
    "# Diagnosis & Optimization\n",
    "\n",
    "**Objective:** run a real on-call differential end to end, from symptom to rollback decision, without proposing a fix before the evidence isolates a culprit.\n",
    "\n",
    "## Challenge Outline\n",
    "\n",
    "Build a complete notebook that demonstrates the following outcomes:\n",
    "\n",
    "- **List what changed:** produce the change inventory for the incident and write the one sentence that uses the held-constant latency to rule out an infrastructure or model-tier cause.\n",
    "- **Read the cost drop as evidence:** explain the single mechanism that produces a quality drop and a cost drop together, and name the two deploys that could each cause it.\n",
    "- **Design the ablation:** write the two cross-runs (last night's prompt + yesterday's index, and yesterday's prompt + last night's index) and state which result convicts which deploy.\n",
    "- **Name the metric that reveals the culprit:** commit to the single logged number that would have caught the incident before the eval score moved, and justify it from the cost drop.\n",
    "- **Write the rollback decision rule:** commit to a single conditional rollback sentence of the form 'roll back X if the ablation shows Y; otherwise roll back Z.'\n",
    "- **Close the feedback loop:** specify what the deploy pipeline gains so a quality-down / cost-down pair pages automatically next time, instead of a human noticing at 08:00.\n",
    "\n",
    "Your solution should include enough code, output, or written observations to prove each outcome worked. Keep the notebook focused on final behavior and evidence rather than a guided walkthrough.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Student Workspace\n",
    "\n",
    "Use the sections below to build your solution. Each section maps to one required outcome from the challenge outline.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 1: List what changed\n",
    "\n",
    "produce the change inventory for the incident and write the one sentence that uses the held-constant latency to rule out an infrastructure or model-tier cause.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Part 1: List what changed\n",
    "# Add your implementation, outputs, or notes here.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 2: Read the cost drop as evidence\n",
    "\n",
    "explain the single mechanism that produces a quality drop and a cost drop together, and name the two deploys that could each cause it.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Part 2: Read the cost drop as evidence\n",
    "# Add your implementation, outputs, or notes here.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 3: Design the ablation\n",
    "\n",
    "write the two cross-runs (last night's prompt + yesterday's index, and yesterday's prompt + last night's index) and state which result convicts which deploy.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Part 3: Design the ablation\n",
    "# Add your implementation, outputs, or notes here.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 4: Name the metric that reveals the culprit\n",
    "\n",
    "commit to the single logged number that would have caught the incident before the eval score moved, and justify it from the cost drop.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Part 4: Name the metric that reveals the culprit\n",
    "# Add your implementation, outputs, or notes here.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 5: Write the rollback decision rule\n",
    "\n",
    "commit to a single conditional rollback sentence of the form 'roll back X if the ablation shows Y; otherwise roll back Z.'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Part 5: Write the rollback decision rule\n",
    "# Add your implementation, outputs, or notes here.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 6: Close the feedback loop\n",
    "\n",
    "specify what the deploy pipeline gains so a quality-down / cost-down pair pages automatically next time, instead of a human noticing at 08:00.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Part 6: Close the feedback loop\n",
    "# Add your implementation, outputs, or notes here.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verification Notes\n",
    "\n",
    "Summarize the evidence that each part worked. Capture API signals, validation outcomes, errors, recovery behavior, cost observations, or comparisons required by this lab.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Verification notes\n",
    "# Record the evidence that proves each lab outcome worked.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Answer Key\n",
    "\n",
    "The cells below contain the completed reference implementation/content for this lab. Use this section only after attempting the self-driven lab.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Lab 7 Answer Key*\n",
    "\n",
    "# Diagnosis & Optimization\n",
    "\n",
    "**Objective:** run a real on-call differential end to end, from symptom to rollback decision, without proposing a fix before the evidence isolates a culprit.\n",
    "\n",
    "This is a design exercise, not a coding exercise. You are paged at 08:00. Overnight: support-answer quality dropped 12% (measured against your eval set), cost per request fell 30%, and p95 latency is unchanged. Two things deployed last night within the same hour: a prompt change and a corpus re-index. Leadership wants to know what to roll back. Work each part on paper (or in the markdown cells) before reading the worked answer. Do not propose a fix until Part 4.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: List what changed\n",
    "\n",
    "The thing that changed is the thing that broke. Before you form any theory, list what moved and what was explicitly held constant; the constant is there to point you at the variables that did move.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Change inventory.**\n",
    "\n",
    "| Variable | Moved overnight? | Evidence |\n",
    "|---|---|---|\n",
    "| Prompt template | **Yes** (deployed last night) | Deploy log |\n",
    "| Corpus / index | **Yes** (re-indexed last night) | Deploy log |\n",
    "| Model version | No | No model deploy in the window |\n",
    "| Traffic mix | No (assume verified against request logs) | Ingress volume flat |\n",
    "| p95 latency | No, explicitly unchanged | Dashboard |\n",
    "\n",
    "**The ruling-out sentence.** p95 latency is unchanged, so neither the serving infrastructure nor a model-tier change degraded overnight; whatever broke walked in with one of the two deploys, because those are the only variables that moved.\n",
    "\n",
    "Note also that the quality drop is measured against the eval set, so it is a real regression, not a perception problem. Two suspects remain: the prompt change and the re-index. Everything after this is about separating them.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2: Read the cost drop as evidence\n",
    "\n",
    "Quality fell and cost fell together. The cost drop is the clue, not the reassurance: ask what single mechanism sends fewer tokens to the model and starves the answer at the same time.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**The mechanism: a shorter effective prompt.** Fewer input tokens reach the model, so the bill falls; less grounding reaches the model, so quality falls. One cause, both symptoms, same moment.\n",
    "\n",
    "Either deploy can produce it:\n",
    "\n",
    "- **The prompt change** could have truncated or dropped a context block: a template bug, a removed section, or a reordered prefix that changed what actually renders into the request.\n",
    "- **The re-index** could be returning fewer or emptier chunks: a broken re-index, an embedding/dimension mismatch between query and corpus encoders, or a filter that now excludes most documents. Retrieval that returns less text also means fewer input tokens per call.\n",
    "\n",
    "A falling bill with flat traffic is never free money during an incident. Do the arithmetic to see how much prompt must have vanished:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Output length is flat (the answers are the same size, just worse), so the\n",
    "# 30% cost drop must come from input tokens alone. Solve for the missing text.\n",
    "OPUS_IN = 5.00 / 1_000_000    # $/token, input\n",
    "OPUS_OUT = 25.00 / 1_000_000  # $/token, output\n",
    "\n",
    "baseline_in, baseline_out = 12_000, 800\n",
    "baseline_cost = baseline_in * OPUS_IN + baseline_out * OPUS_OUT\n",
    "\n",
    "observed_cost = baseline_cost * 0.70  # the 30% drop\n",
    "implied_in = (observed_cost - baseline_out * OPUS_OUT) / OPUS_IN\n",
    "\n",
    "print(f\"baseline input tokens: {baseline_in:,}\")\n",
    "print(f\"implied input tokens : {implied_in:,.0f}\")\n",
    "print(f\"tokens that vanished : {baseline_in - implied_in:,.0f}\")\n",
    "# ~4,800 tokens: roughly the size of a retrieved-context block. The model is\n",
    "# answering with far less grounding, which is why quality fell 12% at once.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3: Design the ablation\n",
    "\n",
    "You have two suspects and need to isolate one. Each ablation run holds everything constant except a single variable, exactly like the ablation ladder from the lab.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**The two runs.** Take the failing slice of the eval set and run it through both cross-combinations. Everything else (model, effort, eval harness) is pinned.\n",
    "\n",
    "| Run | Prompt | Index | If this run FAILS | If this run PASSES |\n",
    "|---|---|---|---|---|\n",
    "| A | Last night's (new) | Yesterday's (old) | The prompt carries the bug even on the good index | The new prompt is fine on good data |\n",
    "| B | Yesterday's (old) | Last night's (new) | The re-index carries the bug even under the good prompt | The new index is fine under the good prompt |\n",
    "\n",
    "**Reading the grid:**\n",
    "\n",
    "- A fails, B passes: **the prompt is convicted.**\n",
    "- A passes, B fails: **the re-index is convicted.**\n",
    "- Both fail: both deploys are defective (roll back both; the deploys should never have shared a window).\n",
    "- Both pass: the incident is not reproducible on the eval slice; go back to Part 1 and re-check the change inventory (traffic mix, ingress) before touching either deploy.\n",
    "\n",
    "This is two eval runs, not a week of prompt archaeology. Run them before answering leadership.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 4: Name the metric that reveals the culprit\n",
    "\n",
    "Which single number (retrieved-chunk relevance, average input tokens per call, cache-hit rate, cited-span-exists rate) would have caught this before the eval score moved? The cost drop tells you which one.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Average input tokens per call.** The cost drop is the tell: cost per request on flat traffic and flat output length can only fall because fewer input tokens are billed. Input tokens per call comes straight off `usage` on every response, requires no labels and no judge, and would have shown a step change the minute the deploy landed: hours before the nightly eval run moved.\n",
    "\n",
    "Why not the others:\n",
    "\n",
    "- **Retrieved-chunk relevance** would localize the bug precisely, but it needs labelled judgments; it is a diagnosis tool, not a cheap always-on alarm.\n",
    "- **Cache-hit rate** is the right leading indicator for a poisoned prefix, but a cache miss makes cost go **up**, not down; the direction of this incident clears it.\n",
    "- **Cited-span-exists rate** is a strong quality alarm and worth having, but it fires when answers are already ungrounded; input tokens per call fires when the grounding disappears from the request itself, one step earlier in the causal chain.\n",
    "\n",
    "Rule of thumb: alert on the cheapest metric that sits closest to the mechanism. Here the mechanism is a shrinking prompt, and the prompt's size is logged for free on every call.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 5: Write the rollback decision rule\n",
    "\n",
    "Force yourself to commit to one rollback, not both. A rule that hedges ('roll back both to be safe') destroys the information the ablation just bought you.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**The rule.** Roll back the **prompt** if the failing slice fails on run A (new prompt + old index) and passes on run B (old prompt + new index); otherwise roll back the **re-index**.\n",
    "\n",
    "Why one and not both: rolling back both restores service but erases the diagnosis, and the defective deploy will be re-shipped next sprint with the same bug. The ablation already paid for the answer; the rule spends it. The only both-back case is the both-fail cell of the grid, and then the rule still commits: both, explicitly, because the evidence convicted both.\n",
    "\n",
    "Notice the rule is written **before** the ablation results arrive. Deciding the decision rule first is what keeps the 09:00 leadership call from turning into a debate about what the numbers mean.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 6: Close the feedback loop\n",
    "\n",
    "The incident was found by a human reading a dashboard the morning after. Add the machinery that makes the same signature page on its own, at deploy time.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Three additions, in order of leverage:\n",
    "\n",
    "1. **A canary eval gate on every deploy.** The failing slice plus a smoke slice of the eval set runs against the new build before full rollout; a pass-rate drop beyond threshold blocks the deploy. This catches the regression at 23:00, not 08:00.\n",
    "2. **A paired-signal alert.** Quality-down together with cost-down is an incident signature (a shrinking effective prompt), not two independent graphs. Page when both move in the same window.\n",
    "3. **A per-deploy metrics diff.** Emit average input tokens per call, cache-hit rate, and eval pass rate as a before/after diff attached to every deploy record, so the on-call's Part 1 change inventory is already written when the page fires.\n",
    "\n",
    "And one process fix: the prompt change and the re-index landed in the same hour. Separate deploy windows for independent subsystems is what makes the Part 3 ablation unnecessary next time.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The pager rule as code: a quality-down / cost-down pair is one signature.\n",
    "def deploy_health_alert(before, after):\n",
    "    q_drop = (before[\"eval_pass_rate\"] - after[\"eval_pass_rate\"]) / before[\"eval_pass_rate\"]\n",
    "    c_drop = (before[\"cost_per_request\"] - after[\"cost_per_request\"]) / before[\"cost_per_request\"]\n",
    "    in_drop = (before[\"avg_input_tokens\"] - after[\"avg_input_tokens\"]) / before[\"avg_input_tokens\"]\n",
    "\n",
    "    if q_drop > 0.05 and c_drop > 0.10:\n",
    "        return (\"PAGE\", \"quality and cost fell together: suspect a shorter effective prompt\")\n",
    "    if in_drop > 0.15:\n",
    "        return (\"WARN\", \"avg input tokens stepped down: a context block may have been dropped\")\n",
    "    return (\"OK\", \"no deploy-shaped regression\")\n",
    "\n",
    "\n",
    "before = {\"eval_pass_rate\": 0.91, \"cost_per_request\": 0.080, \"avg_input_tokens\": 12_000}\n",
    "after  = {\"eval_pass_rate\": 0.80, \"cost_per_request\": 0.056, \"avg_input_tokens\": 7_200}\n",
    "print(deploy_health_alert(before, after))\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}