{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Lab 6 Answer Key*\n",
    "\n",
    "# Evaluation Frameworks & Datasets\n",
    "\n",
    "**Objective:** build the evaluation program for the Lab 1 claims-triage system: one metric per family, an honest target, a dataset built from overrides, a grader funnel, a CI regression gate, and an A/B design for a model swap.\n",
    "\n",
    "This is a design lab with a real harness at its core. The system under evaluation is the claims-triage pipeline from Lab 1: 4,000 claims a day routed into four queues (auto-approve, standard review, fraud review, request-information) by adjusters who agree with each other about 85% of the time. Write each artifact as if handing it to the team that will run the program. Code appears where the lab itself builds it: the regression harness and the pre-registered A/B decision rule.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: One metric per family\n",
    "\n",
    "Do not proceed until every family has exactly one number. Each family hides something: an aggregate accuracy hides which segment regressed, average latency hides the tail, cost per request hides retries, one-sided safety invites the other side to rot.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Artifact: the metric sheet, one number per family.**\n",
    "\n",
    "| Family | The one number | Why this one, and what the naive alternative hides |\n",
    "| --- | --- | --- |\n",
    "| Accuracy | **Cost-weighted queue accuracy**, reported per queue segment: a fraud claim routed to auto-approve is weighted far heavier than a standard claim routed to request-information | Plain exact-match treats all misroutes as equal, but the business does not; and a single aggregate hides a 60% fraud-queue collapse behind a 95% overall |\n",
    "| Latency | **p95 end-to-end triage time**, gated at 5 minutes against the one-hour SLA | The mean makes the dashboard look calm; the tail generates the support ticket. Gating well inside the SLA leaves headroom for retries and spikes |\n",
    "| Cost | **Cost per correctly routed claim** | Cost per request falls when a worse model is swapped in, while retries push the real bill up; the denominator must be the unit the business buys |\n",
    "| Safety | **False-denial rate**: claims wrongly routed away from approval or wrongly toward denial | Harmful-output rate alone rewards a system that refuses or escalates everything; safety is two-sided |\n",
    "| Security | **Cross-claim data-leak rate** under an injection probe suite: does claimant A's PII ever surface in claimant B's packet or output | Passing only your own probes hides the probe you did not write, so the suite is refreshed from new red-team findings, not frozen |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2: The honest target\n",
    "\n",
    "Adjusters agree with each other about 85% of the time. Write the sentence that states what your accuracy ceiling therefore is, and why a proposal promising 97% should make you suspicious rather than pleased.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**The ceiling sentence.** Adjusters agree on the queue label only 85% of the time, so 85% is the achievable accuracy ceiling for this task: the remaining 15% of claims have no ground truth, only opinions, and no model can be \"right\" against a label humans cannot agree on.\n",
    "\n",
    "**Why 97% is a red flag.** A vendor or team promising 97% on this task is not describing excellence; they are describing measurement error. There are only two ways to score above the agreement ceiling: overfit one annotator's habits (the model has learned Dave, not claims triage) or leak labels between the tuning set and the test set. Either way the number will not survive contact with a second annotator or with production. The correct reading of \"more accurate than the annotators agree with each other\" is \"broken measurement,\" not \"impressive model.\"\n",
    "\n",
    "**The target this program actually sets:** within a few points of the ceiling on the unambiguous stratum (cases where adjusters do agree), measured against multi-annotator consensus labels; and for the genuinely ambiguous 15%, the goal is not a correct label but a defensible route to human review. Measure agreement first: it tells you what a perfect score would even mean.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3: Dataset design from the overrides\n",
    "\n",
    "Every adjuster override from Lab 1's feedback plane is a labelled case: the system's answer, the human's correction, and the packet that produced both. Describe the stratification, which failure modes you over-sample, and what goes in the frozen hold-out.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Artifact: the dataset specification.**\n",
    "\n",
    "**Sources, in priority order:** adjuster overrides (free labels on exactly the cases the system got wrong), production traces sampled for distribution realism, red-team injection packets for the security metric, and synthetic edge cases only for combinations production has not produced yet (rare claim types crossed with rare document formats).\n",
    "\n",
    "**Stratification: by queue crossed with claim-value band.** Four queues times three value bands gives twelve strata, each reported separately, so a regression in high-value fraud routing cannot hide behind strength in low-value standard review. A dataset that mirrors production is 95% cases the system already passes, which means 95% of the eval spend measures nothing; stratification plus over-sampling is the correction.\n",
    "\n",
    "**Over-sampled failure modes:** fraud claims routed to auto-approve (the costliest miss), overrides where the adjuster reversed the model with high confidence, request-information cases the system answered anyway, and packets containing quoted instructions (injection-shaped inputs).\n",
    "\n",
    "**The frozen hold-out:** roughly 500 cases, stratified like the working set, with consensus labels from at least two adjusters (cases where they disagree go to the ambiguity stratum, not the hold-out). Nobody tunes against it, nobody reads it, and it never changes: a dataset you tune against stops being a test and becomes training data. The working set refreshes weekly from new overrides; the frozen suite answers \"did anything that worked stop working\" with a stable yardstick.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 4: The grader funnel\n",
    "\n",
    "Three graders cost wildly different amounts and catch different failures. Use the cheapest one that can answer the question, and reserve the expensive one for the question only it can answer. An uncalibrated judge is a metric you invented.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Artifact: criterion-to-grader assignment.**\n",
    "\n",
    "| Criterion | Grader | Why |\n",
    "| --- | --- | --- |\n",
    "| Output parses against the triage schema | Programmatic | Nearly free, exact; a broken schema needs no judge |\n",
    "| Queue label matches the consensus label | Programmatic | Exact match on an enum |\n",
    "| Cited clause actually exists in the policy manual | Programmatic | Set membership; deterministic |\n",
    "| Rationale is a faithful reading of the cited clause | LLM judge | Requires judgment; too voluminous for humans at 4,000 claims a day |\n",
    "| A genuinely ambiguous claim was routed defensibly | Human | The 15% with no ground truth; only humans can adjudicate opinion |\n",
    "\n",
    "**The funnel.** Programmatic gates run first on everything and catch most regressions for free. The LLM judge runs only on cases that survive the gates, scores a rubric (faithful_to_clause, answers_the_packet, no_fabricated_clause) with a structured verdict, and randomizes candidate order to defeat position bias. Humans see two slices: a weekly calibration sample that re-measures judge-versus-human agreement, and the ambiguous stratum.\n",
    "\n",
    "**The calibration discipline:** the judge is a measurement instrument only while its agreement number is current. Re-measure it whenever the rubric or the judge model changes; a judge calibrated on last month's rubric is a metric that silently expired. The judge model must be at least as capable as the generator for this rubric (`claude-opus-4-8` judging faithfulness), because a cheaper judge is safe only for narrow, well-specified criteria.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 5: The regression gate\n",
    "\n",
    "The frozen suite runs in CI on every change. State the pass condition that blocks a merge as a pass rate over the set plus a p95 guardrail. Reproducibility comes from pinning the model ID and the prompt version, not from sampling parameters: individual runs vary, which is exactly why the gate is a rate.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import anthropic, asyncio, json, time\n",
    "from statistics import quantiles\n",
    "\n",
    "client = anthropic.AsyncAnthropic()\n",
    "\n",
    "TRIAGE_SCHEMA = {\n",
    "    \"type\": \"json_schema\",\n",
    "    \"schema\": {\n",
    "        \"type\": \"object\",\n",
    "        \"properties\": {\n",
    "            \"queue\": {\"type\": \"string\",\n",
    "                      \"enum\": [\"auto_approve\", \"standard_review\",\n",
    "                               \"fraud_review\", \"request_information\"]},\n",
    "            \"cited_clause\": {\"type\": \"string\"},\n",
    "        },\n",
    "        \"required\": [\"queue\", \"cited_clause\"],\n",
    "        \"additionalProperties\": False,\n",
    "    },\n",
    "}\n",
    "\n",
    "# Pin the model ID and the prompt version. Determinism does NOT come\n",
    "# from a sampling parameter (temperature is removed and returns 400);\n",
    "# runs vary, which is why the gate is a pass RATE over the frozen set.\n",
    "MODEL_ID = \"claude-opus-4-8\"\n",
    "PROMPT_VERSION = \"triage-policy-v7\"\n",
    "\n",
    "async def run_case(case):\n",
    "    start = time.monotonic()\n",
    "    resp = await client.messages.create(\n",
    "        model=MODEL_ID,\n",
    "        max_tokens=1024,\n",
    "        output_config={\"format\": TRIAGE_SCHEMA},\n",
    "        system=[{\"type\": \"text\", \"text\": TRIAGE_POLICY_MANUAL,\n",
    "                 \"cache_control\": {\"type\": \"ephemeral\"}}],\n",
    "        messages=[{\"role\": \"user\", \"content\": case[\"packet\"]}],\n",
    "    )\n",
    "    latency = time.monotonic() - start\n",
    "    text = next(b.text for b in resp.content if b.type == \"text\")\n",
    "    predicted = json.loads(text)[\"queue\"]\n",
    "    return {\"passed\": predicted == case[\"expected_queue\"], \"latency\": latency}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The gate that blocks a merge. Thresholds are set against the 85%\n",
    "# agreement ceiling: 82% on the frozen suite is a few points under the\n",
    "# ceiling, and the p95 guardrail stops an accuracy win that doubles\n",
    "# the tail from shipping unnoticed.\n",
    "GATE = {\"min_pass_rate\": 0.82, \"max_p95_s\": 8.0}\n",
    "\n",
    "async def regression_gate(frozen_cases):\n",
    "    results = await asyncio.gather(*(run_case(c) for c in frozen_cases))\n",
    "    pass_rate = sum(r[\"passed\"] for r in results) / len(results)\n",
    "    p95 = quantiles([r[\"latency\"] for r in results], n=100)[94]\n",
    "    merge_allowed = (pass_rate >= GATE[\"min_pass_rate\"]\n",
    "                     and p95 <= GATE[\"max_p95_s\"])\n",
    "    print({\"prompt_version\": PROMPT_VERSION,\n",
    "           \"cases\": len(results),\n",
    "           \"pass_rate\": round(pass_rate, 3),\n",
    "           \"p95_latency_s\": round(p95, 3),\n",
    "           \"merge_allowed\": merge_allowed})\n",
    "    return merge_allowed\n",
    "\n",
    "# Stated as a sentence for the CI config: the merge is blocked unless\n",
    "# pass rate over the entire frozen suite is at least 0.82 AND p95\n",
    "# latency is at most 8 seconds. No single trial ever decides anything.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Operational note.** When the same frozen suite is re-run at scale offline (for example, re-grading 10,000 historical cases against a candidate prompt overnight), submit it through the Batches API: the 24-hour window is irrelevant to an overnight job and the 50% discount is real money. Batch results return unordered, so join each result to its expected label by `custom_id`, never by position; a positional join silently mis-scores the entire run.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 6: The A/B for a model swap\n",
    "\n",
    "Someone proposes moving the classification call from claude-opus-4-8 to claude-sonnet-5 to cut cost. Write the experiment: change one variable, pre-register the metric and the decision rule before looking at data, and pair the primary with guardrails. A model swap moves quality, cost, and latency at once, so plan the attribution before you start.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Artifact: the experiment specification.**\n",
    "\n",
    "| Element | Specification |\n",
    "| --- | --- |\n",
    "| The one variable | The model of the classification call only: `claude-opus-4-8` to `claude-sonnet-5`. Prompt version, retrieval config, and `output_config` are pinned in both arms |\n",
    "| Traffic | 50/50 random assignment at the claim level so both arms see the same input distribution; two weeks, so novelty effects wash out and weekly claim mix is covered |\n",
    "| Primary metric | Cost-weighted queue accuracy (Part 1), reported per stratum |\n",
    "| Guardrails, pre-registered | False-denial rate up by at most 0.5 points; p95 triage time not up; cost per correctly routed claim down (it is the entire point of the swap) |\n",
    "| Decision rule, written before launch | Ship if the primary is within 1 point of Opus AND every guardrail holds. Any guardrail breach stops the experiment regardless of the primary |\n",
    "| The confound | A model swap moves quality, cost, and latency simultaneously, so no single-metric read can attribute the change. The answer is not to attribute between them but to pin everything else so the whole delta bundle belongs to the swap, then price the bundle: primary plus all guardrails together |\n",
    "\n",
    "**The traps this design closes.** Statistical versus practical significance: with 4,000 claims a day, a 0.2-point accuracy difference will reach significance and still not matter; the decision rule uses thresholds, not p-values alone. Traffic skew: assignment is random per claim, not per adjuster or per hour. And single-metric approval: an experiment with one metric is an experiment designed to be approved, which is why every guardrail carries a stop condition.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The pre-registered decision rule, executable. Written BEFORE the\n",
    "# experiment starts; a dashboard read after the fact always finds a story.\n",
    "AB_SPEC = {\n",
    "    \"variable\": \"classification model: claude-opus-4-8 -> claude-sonnet-5\",\n",
    "    \"held_constant\": [\"prompt version triage-policy-v7\",\n",
    "                      \"retrieval config\", \"output_config\"],\n",
    "    \"primary\": \"cost-weighted queue accuracy\",\n",
    "    \"duration_days\": 14,\n",
    "}\n",
    "\n",
    "def decide(delta):\n",
    "    \"\"\"delta: Sonnet arm minus Opus arm, measured over the full run.\"\"\"\n",
    "    guardrails_hold = (\n",
    "        delta[\"false_denial_pts\"] <= 0.5      # safety guardrail\n",
    "        and delta[\"p95_s\"] <= 0.0             # latency guardrail\n",
    "        and delta[\"cost_per_routed_pct\"] < 0  # the swap must actually save\n",
    "    )\n",
    "    primary_holds = delta[\"accuracy_pts\"] >= -1.0\n",
    "    return \"ship\" if (guardrails_hold and primary_holds) else \"hold\"\n",
    "\n",
    "print(decide({\"accuracy_pts\": -0.4, \"false_denial_pts\": 0.2,\n",
    "              \"p95_s\": -1.1, \"cost_per_routed_pct\": -38.0}))\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}