Diagnosis & Optimization

1. Differential Diagnosis: Symptom → Plane

Lab 1 gave you five planes: ingress, context, reasoning, egress, feedback. Diagnosis is that map read backwards: a symptom is evidence about which plane failed. The skill the exam tests is not knowing every fix; it is refusing to fix the wrong plane. Almost every wrong answer in this domain is a real remedy applied to the plane that wasn't broken.

This is the table to internalize. Read a symptom, name the plane, name the one cheap check that distinguishes it from its neighbors.

Notice how few of these rows point at “the model.” That is the point. In a well-built system the reasoning plane is usually the last suspect, because the other four planes are cheaper to break and cheaper to check.

2. The Guide's Sample 3: The Thing That Changed

This is one of the guide's own sample items, and it is worth teaching in full because it is the differential in miniature:

A RAG system suddenly returns confident but incorrect answers after a document refresh, while latency and model version are unchanged. What is the most likely first place to investigate?

Answer: the retrieval / indexing step is returning irrelevant or stale chunks: a broken re-index, mismatched embeddings, a dimension or model mismatch between the query encoder and the corpus encoder. The reasoning generalizes into a rule you can apply to any incident:

Model unchanged + latency unchanged + data refreshed → the data plane. Two variables were explicitly held constant so you would stop suspecting them. One variable moved. The confident-but-wrong shape is the tell: the model is doing exactly what it should (answering from its context) and the context is poisoned. A confident wrong answer grounded in a bad chunk looks identical to a hallucination and is not one.

3. Hallucination Is a Symptom, Not a Diagnosis

“The model hallucinated” is where lazy incident reviews stop. An architect decomposes it, because the four causes have four different fixes and three of them are not in the model at all.

The architectural move for the second, most common kind is specific: require a citation and reject any output whose cited span does not appear in the retrieved context. That converts a soft “be accurate” into a hard validator the egress plane can enforce. The naive reflex (raise effort and hope, or reach for a sampling knob) is not a fix. (There is no temperature control on current models to turn down; setting it returns a 400.)

4. Prompt Failure vs. Model Mismatch: The Ablation Ladder

Two failures look identical from the outside (a wrong answer) and have opposite fixes. Rewriting the prompt when the model was under-tiered wastes a week; upgrading the model when the prompt was underspecified wastes money forever. You tell them apart with cheap ablations, run in order, each one holding everything constant but one variable.

The disambiguation in row one is the crux. A stronger model passing tells you the current tier was insufficient for this prompt, but a better prompt might have carried the weaker tier. Run the simplify-the-case ablation next: if the weak model handles the easy version, the model can do the task and your prompt was the problem; if it fails even the easy version, the tier was genuinely too low. That is the whole domain in two API calls.

The single most useful ablation is the third: swap retrieved context for a hand-built perfect context. If the case passes on perfect context, you have proven the reasoning plane is fine and localized the bug to retrieval. No prompt engineering required.

import anthropic

client = anthropic.Anthropic()

# The core ablation: same failing case, two sources of context.
# If the hand-built context passes and the retrieved one fails,
# the bug lives in retrieval -- not in the model's reasoning.
def run_case(question, context_text):
    return client.messages.create(
        model="claude-opus-4-8",
        max_tokens=4096,
        thinking={"type": "adaptive"},
        output_config={"effort": "high"},
        system=[{"type": "text", "text": context_text}],
        messages=[{"role": "user", "content": question}],
    )

gold = run_case(FAILING_QUESTION, HAND_BUILT_PERFECT_CONTEXT)
live = run_case(FAILING_QUESTION, retrieve(FAILING_QUESTION))

# gold correct + live wrong  -> retrieval plane (fix the index / embeddings)
# both wrong                 -> reasoning: raise the tier, or the task isn't viable
# both correct               -> the incident isn't reproducible; check ingress / traffic

5. Optimization: The Lever Table

Once the system is correct, you make it cheaper and faster, and this is where architects destroy value by optimizing the wrong metric. Every lever below improves something and costs something. The exam rewards knowing the cost, not just the benefit.

Caching is the standout because it is the rare lever that moves cost and latency in the same direction, which is exactly why it is the answer whenever an item asks for both. But it hides a measurement trap.

The arithmetic that separates architects from tinkerers: optimize cost per resolved task, not cost per request. A cheaper model or a lower effort can cut per-request cost 30% while doubling the retry rate, and two cheap wrong calls plus one expensive right one cost more than one right call at full price. The falling per-request number is the classic false win.

resp = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=1024,
    system=[{"type": "text", "text": STABLE_PREFIX,
             "cache_control": {"type": "ephemeral"}}],
    messages=[{"role": "user", "content": question}],
)

u = resp.usage
OPUS_IN = 5.00 / 1_000_000  # $/token, input

# input_tokens is ONLY the uncached remainder. Add the cache tiers back
# or every cost figure you report is too low.
true_input_cost = (
    u.input_tokens                * OPUS_IN
    + u.cache_read_input_tokens     * OPUS_IN * 0.10   # reads ~0.1x
    + u.cache_creation_input_tokens * OPUS_IN * 1.25   # 5-min writes ~1.25x
)

total_prompt = (u.input_tokens
                + u.cache_read_input_tokens
                + u.cache_creation_input_tokens)
hit_rate = u.cache_read_input_tokens / max(total_prompt, 1)

# A collapsing hit rate is the earliest signal of a poisoned prefix.
assert hit_rate > 0.5, f"cache collapsed to {hit_rate:.0%} -- audit the prefix"

# Emit the request id (public despite the underscore) with every log line.
log.info("call", request_id=resp._request_id,
         hit_rate=round(hit_rate, 3), cost=round(true_input_cost, 5))

6. Monitoring in Production

You cannot diagnose what you did not record. The feedback plane from Lab 1 is what makes every earlier section possible, and it has three distinct instruments, each answering a different question.

Metrics tell you something is wrong; traces tell you where; evals tell you whether the fix actually restored quality. A system with all three green dashboards and no eval set is one silent prompt-deploy away from shipping confidently-wrong answers with perfect uptime.

What to alert on, and one non-obvious leading indicator:

• Error rate and p95 latency — the table stakes.
• Cost per resolved task — not per request, per section 5.
• Guardrail-violation rate — schema rejections, refusals, failed citation checks.
• Cache-hit rate falling — the earliest signal that someone put a timestamp or a per-request ID at the front of the prompt. Quality and correctness are unchanged; cost quietly doubles days before anyone notices the bill.

Emit response._request_id (public despite the leading underscore) with every log line. It is the one identifier that lets Anthropic trace a specific failing call end-to-end, and the first thing support will ask for.

# Budget a prompt BEFORE sending it -- observability starts at design time.
# Never estimate with tiktoken: it is OpenAI's tokenizer and undercounts
# Claude by ~15-20% (much more on code), so it will lie about your window.
count = client.messages.count_tokens(
    model="claude-opus-4-8",
    system=[{"type": "text", "text": STABLE_PREFIX}],
    messages=[{"role": "user", "content": question}],
)

print("prompt tokens:", count.input_tokens)
if count.input_tokens >= 4096:
    print("prefix is long enough to cache on Opus 4.8 (>= 4096 tokens)")
else:
    print("below the Opus 4.8 cache minimum -- cache_control will silently no-op")

7. Design Exercise

Objective: run a real on-call differential end to end. About 45 minutes. Do not propose a fix until step 4.

Scenario. You are paged at 08:00. Overnight: support-answer quality dropped 12% (measured against your eval set); cost per request fell 30%; p95 latency is unchanged. Two things deployed last night: a prompt change and a corpus re-index. Both landed within the same hour. Leadership wants to know what to roll back.

1. List what changed. Two variables moved, one held constant (latency). Write the one sentence that rules out an infrastructure or model-tier cause using the constant.
2. Read the cost drop as evidence, not good news. Quality fell and cost fell together. What single mechanism produces both at once? (Hint: a shorter effective prompt, whether a truncated prefix, a dropped context block, or a cache-miss pattern that changed what actually reaches the model, sends fewer tokens and starves the answer.) Name the two deploys that could each cause it.
3. Design the ablation. You have two suspects and need to isolate one. Write the two runs: the failing eval slice on last night's prompt + yesterday's index, and on yesterday's prompt + last night's index. Which result convicts the re-index? Which convicts the prompt?
4. Name the metric that reveals the culprit. Which single number (retrieved-chunk relevance, average input tokens per call, cache-hit rate, cited-span-exists rate) would have caught this before the eval score moved? State why the cost drop points you at it.
5. Write the rollback decision rule. A sentence of the form: “Roll back X if the ablation shows Y; otherwise roll back Z.” Force yourself to commit to one, not both.
6. Close the feedback loop. What do you add to the deploy pipeline so a quality-down / cost-down pair pages you automatically next time, instead of a human noticing at 08:00?

Step 2 is the one the exam tests: the cost drop is the clue, not the reassurance. Step 3 is the one that gets you the right answer instead of a plausible one.

Key Takeaways

• The thing that changed is the thing that broke. List what moved before forming a theory; the held-constant clauses in a scenario are pointing you at the one variable that didn't stay still.
• Diagnosis is the five planes read backwards: a symptom is evidence about which plane failed. Fixing the wrong plane is the most common wrong answer in the domain.
• Model unchanged + latency unchanged + data refreshed → the data plane. Confident-and-wrong after a corpus change is retrieval until proven otherwise.
• Hallucination is a symptom, not a diagnosis. Decompose it into no-grounding, grounding-ignored, contradictory, or out-of-distribution, and force citations you can validate rather than “raise effort and hope.”
• Separate prompt failure from model mismatch with the ablation ladder: stronger model, simplified case, hand-built context, moved instruction; each holds all but one variable.
• Caching moves cost and latency together; batching trades latency for cost; streaming does not reduce cost; max_tokens is a cap, not a cost control.
• usage.input_tokens is the uncached remainder only; true prompt size adds cache_read_input_tokens and cache_creation_input_tokens. Naive cost math undercounts every cached request.
• Optimize cost per resolved task, not per request. A falling per-request cost while quality drops (or retries rise) is not a win. Alert on a falling cache-hit rate; it is the earliest sign of a poisoned prefix.