Evaluation Frameworks & Datasets

1. The Five Metric Families

The objective names five categories, and it names them because architects reach for the wrong one under pressure. Accuracy is the one everyone measures; the other four are the ones that end projects. Each metric answers a question, and each one hides something, which is the part the exam tests.

The costliest of these confusions is the cost one. Cost per request falls while cost per resolved task rises whenever a cheaper or worse model is swapped in: each call is cheaper, but users retry the ones that failed, so the true denominator, resolved tasks, grows slower than the request count. Always define the metric on the unit the business cares about (a resolved ticket, a routed claim), not the unit the API bills you for.

2. Mixed Methodologies: Three Graders

"Mixed methodologies" is the objective's exact phrase, and it is not a suggestion to hedge, it is an architecture. You have three ways to grade an output, they cost wildly different amounts, and they catch different failures. The discipline is to use the cheapest one that can answer the question, and to reserve the expensive one for the question only it can answer.

The architecture that follows from this table is a funnel: programmatic gates run first and catch most regressions (a broken schema needs no judge), the LLM judge runs at scale on what survives, and humans calibrate a sample, both to validate the judge and to adjudicate genuinely ambiguous cases. You do not run all three on everything. You run each where it is cheapest to be right.

3. Designing the Dataset

The eval set is the most valuable artifact in the whole program, because every metric is computed against it. A biased dataset produces confident, precise, wrong numbers. Four sources feed a good one:

Composition matters more than size. Stratify by the segments you care about so a regression in one queue cannot hide behind strength in another. Over-sample failure modes: a dataset that mirrors production is 95% cases you already pass, which means 95% of your eval spend measures nothing. Hold out a set you never look at, because a dataset you tune against stops being a test and becomes training data. And keep a frozen regression suite that never changes, so "did this get worse" has a stable answer.

4. Test Frameworks, Layered Like Software Tests

An eval program is not one big scoreboard; it is a set of tests at different altitudes, exactly as a well-tested codebase has unit, integration, and end-to-end layers. Each layer catches a different class of failure and runs at a different cadence.

Here is a small pytest-style harness that runs a case set concurrently, validates structured output, and reports pass rate and p95 latency, because a change that lifts accuracy while doubling the tail is not an improvement, it is a trade you must see to approve. Note the model ID is pinned and the prompt version is pinned; that, not a sampling parameter, is how you make an eval reproducible.

import anthropic, asyncio, json, time
from statistics import quantiles

client = anthropic.AsyncAnthropic()

TRIAGE_SCHEMA = {
    "type": "json_schema",
    "schema": {
        "type": "object",
        "properties": {
            "queue": {"type": "string",
                      "enum": ["auto_approve", "standard_review",
                               "fraud_review", "request_information"]},
            "cited_clause": {"type": "string"},
        },
        "required": ["queue", "cited_clause"],
        "additionalProperties": False,
    },
}

# Pin the model ID and prompt version. Determinism does NOT come from a
# sampling parameter -- eval runs vary, which is exactly why we report a
# pass RATE over a set, never the result of a single trial.
async def run_case(case):
    start = time.monotonic()
    resp = await client.messages.create(
        model="claude-opus-4-8",
        max_tokens=1024,
        output_config={"format": TRIAGE_SCHEMA},
        system=[{"type": "text", "text": TRIAGE_POLICY_MANUAL,
                 "cache_control": {"type": "ephemeral"}}],
        messages=[{"role": "user", "content": case["packet"]}],
    )
    latency = time.monotonic() - start
    text = next(b.text for b in resp.content if b.type == "text")
    predicted = json.loads(text)["queue"]
    return {"passed": predicted == case["expected_queue"], "latency": latency}


async def evaluate(cases):
    results = await asyncio.gather(*(run_case(c) for c in cases))
    pass_rate = sum(r["passed"] for r in results) / len(results)
    p95 = quantiles([r["latency"] for r in results], n=100)[94]
    return {"pass_rate": pass_rate, "p95_latency_s": round(p95, 3)}

5. LLM-as-Judge, Done Properly

When programmatic checks cannot answer the question ("is this rationale faithful to the cited clause?"), you reach for a judge. Doing it properly means forcing a structured verdict, giving explicit rubric criteria, and randomizing the order of any candidates so position bias cannot leak in. The judge should be at least as capable as the generator for hard rubrics; a cheaper judge is fine only for narrow, well-specified criteria.

import anthropic, random

client = anthropic.Anthropic()

VERDICT_SCHEMA = {
    "type": "json_schema",
    "schema": {
        "type": "object",
        "properties": {
            "verdict": {"type": "string", "enum": ["pass", "fail"]},
            "criterion_scores": {
                "type": "object",
                "properties": {
                    "faithful_to_source": {"type": "integer"},
                    "answers_the_question": {"type": "integer"},
                    "no_fabricated_clause": {"type": "integer"},
                },
                "required": ["faithful_to_source",
                             "answers_the_question",
                             "no_fabricated_clause"],
                "additionalProperties": False,
            },
            "reasoning": {"type": "string"},
        },
        "required": ["verdict", "criterion_scores", "reasoning"],
        "additionalProperties": False,
    },
}

def judge(question, source, candidate_a, candidate_b):
    # Randomize order to defeat position bias. Record the mapping so you
    # can un-shuffle the verdict afterward.
    pair = [("A", candidate_a), ("B", candidate_b)]
    random.shuffle(pair)
    body = "\n\n".join(f"Candidate {label}:\n{text}" for label, text in pair)

    return client.messages.create(
        model="claude-opus-4-8",              # judge >= generator for hard rubrics
        max_tokens=1024,
        thinking={"type": "adaptive"},        # NOT budget_tokens -- that returns 400
        output_config={"effort": "high", "format": VERDICT_SCHEMA},
        system=[{"type": "text", "text": (
            "Score each criterion 0-3 against the SOURCE only. "
            "A fabricated clause is an automatic fail regardless of tone."
        )}],
        messages=[{"role": "user",
                   "content": f"SOURCE:\n{source}\n\nQUESTION:\n{question}\n\n{body}"}],
    )

Two design choices carry the correctness. The structured verdict makes the judge's output machine-comparable and forces it to commit to per-criterion scores rather than a single fuzzy number. And thinking={"type": "adaptive"} lets the judge reason before it rules. Note it is adaptive, not a fixed token budget; the old budget_tokens form is rejected with a 400 on the current models.

The batch harness is a few lines, and the load-bearing detail is the join at the end, where the results are matched back to their expected labels by custom_id, not by the order they arrive:

import anthropic, json
from anthropic.types.message_create_params import MessageCreateParamsNonStreaming
from anthropic.types.messages.batch_create_params import Request

client = anthropic.Anthropic()

# cases: list of {"id": str, "packet": str, "expected_queue": str}
expected = {c["id"]: c["expected_queue"] for c in cases}

batch = client.messages.batches.create(requests=[
    Request(
        custom_id=c["id"],
        params=MessageCreateParamsNonStreaming(
            model="claude-opus-4-8",
            max_tokens=1024,
            output_config={"format": TRIAGE_SCHEMA},
            messages=[{"role": "user", "content": c["packet"]}],
        ),
    )
    for c in cases
])

# ... poll client.messages.batches.retrieve(batch.id) until "ended" ...

passed = total = 0
for result in client.messages.batches.results(batch.id):
    if result.result.type != "succeeded":
        continue
    msg = result.result.message
    text = next(b.text for b in msg.content if b.type == "text")
    predicted = json.loads(text)["queue"]
    # Join by custom_id -- results are UNORDERED. Never index by position.
    passed += predicted == expected[result.custom_id]
    total += 1

print({"pass_rate": passed / total})

6. A/B Testing and Iterative Improvement

Offline eval tells you a change looks better on your dataset. A/B testing tells you it is better in production, but only if you run it as an experiment and not as a hunch. The governing rule: change one variable, pre-register the metric and the decision rule before you look at the data. A dashboard read after the fact will always find a story.

Three traps recur. Novelty effects: a new behaviour looks better simply because it is new, then regresses to the mean. Traffic skew: if the arms do not see the same input distribution, you are comparing populations, not variants. And statistical vs practical significance: with enough traffic, a 0.3% lift is "significant" and still not worth a rollout. Above all, remember that offline eval predicts online performance only as well as your dataset resembles production; the gap between the two is the gap between your eval set and reality.

7. Design Exercise

Objective: build the evaluation program for the claims-triage system from Lab 1: 4,000 claims a day, four queues (auto-approve, standard review, fraud review, request-information), adjusters who agree with each other about 85% of the time. About 60 minutes. Write it as if handing it to the team that will run it.

1. Pick the metrics, one per family. Name the accuracy measure (is it exact-match on the queue, or does a wrong fraud-route cost more than a wrong standard-route, and if so, do you weight it?). Name the latency percentile that satisfies the one-hour SLA. Name the cost unit: per claim or per correctly routed claim. Name the safety and security metrics that a claims system actually has (false denial, PII leakage). Do not proceed until every family has exactly one number.
2. Set the target honestly. Adjusters agree 85% of the time. Write the sentence that states what your accuracy ceiling therefore is, and why a proposal promising 97% should make you suspicious rather than pleased.
3. Design the dataset from the overrides. Every adjuster override from Lab 1's feedback plane is a labelled case. Describe how you stratify it (by queue? by claim value?), which failure modes you over-sample, and what you put in the frozen hold-out you never tune against.
4. Assign each criterion to a grader. "Is the queue label correct": programmatic. "Does the cited clause actually exist in the policy": programmatic. "Is the rationale a faithful reading of that clause": judge. "Is this genuinely ambiguous claim routed defensibly": human. Write the funnel.
5. Define the regression gate. The frozen suite runs in CI. State the pass condition that blocks a merge, and state it as pass rate over the set, plus a p95 guardrail, not as any single trial.
6. Specify the A/B for a model swap. Someone proposes moving the classification call from Opus to Sonnet to cut cost. Write the experiment: the one variable, the primary metric, the guardrail metric, the decision rule, and the confound you must watch (a model swap moves quality, cost, and latency at once; how will you attribute the change?).

Step 2 is the one the exam tests. Step 4 is the one that makes the program affordable to run.

Key Takeaways

• You cannot improve what you cannot measure, and cannot measure what you cannot verify. The eval set is an asset built from production failures and human corrections; design it on day one.
• Five metric families: accuracy, latency, cost, safety, security. Each hides something. Average latency hides the tail (report p95); cost per request hides retries (report cost per resolved task).
• Three graders, used as a funnel: programmatic gates → LLM judge at scale → human calibration on a sample. Use the cheapest grader that can answer the question.
• An uncalibrated judge is a metric you invented. Randomize order, force a rubric, require a structured verdict, and re-measure agreement whenever the rubric or judge changes.
• Inter-annotator agreement caps your achievable accuracy. If humans agree 85% of the time, chasing 99% is overfitting one annotator, not excellence.
• Test frameworks are layered like software tests: unit, integration, end-to-end, regression, canary. Keep a frozen regression suite and a hold-out you never tune against.
• Determinism does not come from sampling parameters (temperature is removed and returns 400). Pin the model ID and prompt version, and report a pass rate over a set, never a single trial.
• A/B testing: change one variable, pre-register the metric and decision rule, and pair every primary metric with a guardrail metric. The Batches API (24h, 50% off, unordered; key by custom_id) is the right tool for a large offline eval.