Claude Code, Evals & Debugging

Part A: Claude Code (3.1%)

1. The core components

Claude Code is an agentic coding tool that runs in your terminal and drives real edits against your repository. The exam names five building blocks. Know what each one is for; that is what distinguishes them.

The distinction the exam probes: Rules are always-on policy; Skills and Commands are invoked; Agents are dispatched; Memory persists. A one-off "for this task, ignore the linter" is none of these; it is a message in the conversation.

2. The CLAUDE.md hierarchy

CLAUDE.md is where a repository tells Claude how it works. It composes down the directory tree: a project-root CLAUDE.md applies everywhere, and a nested CLAUDE.md deeper in the tree layers additional, more specific instructions on top when Claude works in that subtree. Think of it like scoped configuration: the closer file refines, it does not replace.

The judgment call is what belongs there. Put durable conventions in CLAUDE.md: the test command, the package manager, architectural boundaries, naming rules, "never edit generated files." Do not put one-off task instructions there: "refactor the billing module today" is a prompt, not a convention. A CLAUDE.md that fills up with stale task notes is worse than an empty one, because Claude reads it on every task.

# CLAUDE.md  (project root)

## Commands
- Test:  `pytest -q`
- Lint:  `ruff check .`
- Run:   `python -m app`

## Conventions
- Python primary. Type-hint all public functions.
- Never edit files under `generated/` — they are built by `make proto`.
- API calls go through `app/llm/client.py`; do not call the SDK directly elsewhere.

## Model policy
- Use `claude-opus-4-8` for reasoning tasks, `claude-haiku-4-5` for classification.
- Always log `stop_reason` and `usage`. See app/llm/observability.py.

# ---- app/llm/CLAUDE.md (nested: layers ON TOP of the root file) ----
# In this package specifically:
# - Every new client wrapper needs an eval in tests/evals/ before merge.

3. settings.json — permissions, hooks, environment

Where CLAUDE.md is instructions Claude reads, settings.json is configuration the tool enforces. Three things the exam expects you to recognize:

• Permissions — allow / deny / ask rules for tools and commands. This is your least-privilege surface: pre-approve safe reads, gate anything that writes or runs.
• Hooks — shell commands the harness runs automatically on lifecycle events (e.g. before a tool runs, after a session stops). The harness executes these, not the model, so a hook is how you enforce a behavior deterministically.
• Environment variables — values injected into the session, kept out of prompts and source.

{
  "permissions": {
    "allow": ["Read", "Grep", "Glob", "Bash(pytest:*)", "Bash(ruff:*)"],
    "ask":   ["Bash(git push:*)"],
    "deny":  ["Bash(rm -rf:*)", "Read(./.env)"]
  },
  "hooks": {
    "PostToolUse": [
      { "matcher": "Edit|Write", "hooks": [{ "type": "command", "command": "ruff check ." }] }
    ]
  },
  "env": { "APP_ENV": "ci" }
}

4. Repository initialization and session management

You bootstrap a repo with the /init command, which scans the codebase and drafts a first CLAUDE.md: a starting point to edit, not a finished policy. Sessions are durable work units: you can resume a previous session to keep its full context, or fork one to branch from a known state and explore an alternative without disturbing the original. Fork when you want to try two approaches from the same checkpoint; resume when you are continuing the same thread of work.

5. Headless, streaming, and auto-mode

Interactive use is only half the story. The features that matter for shipping:

• Headless mode (the -p / print flag) — runs a single prompt non-interactively and prints the result. This is how Claude Code runs in CI: no TTY, scriptable, exit-code driven.
• Streaming mode — emits output incrementally (including as structured JSON events) so a wrapping script can react as work happens rather than waiting for the end.
• Auto-mode — lets Claude proceed through steps without pausing for per-action confirmation. Powerful and appropriate only inside tight permissions; never combine broad auto-mode with broad write permissions.

# Headless invocation in a CI job — treat it like any other build step.
- name: Claude review gate
  run: |
    npm install -g @anthropic-ai/[email protected]   # PINNED version, not @latest
    claude -p "Review the staged diff for security regressions. \
      Exit non-zero if any HIGH-severity issue is found." \
      --output-format stream-json
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

6. Claude Code is a build step

The single most important framing for the exam: a Claude Code invocation in CI is a build step, and it deserves the same discipline as any other. Pin the version; @latest in CI is a supply-chain and reproducibility risk. Review the settings.json and CLAUDE.md that ship in the repo the way you review any config that runs in your pipeline. Grant least privilege: the review job above needs to read and run tests, not to push or delete. If you would not let an unreviewed script run with those permissions, do not let Claude Code either.

Part B: Evals, Testing & Debugging (2.6%)

7. Problem-origin isolation: the core skill

This is the objective everything else in Domain 4 serves. When output is wrong, your first decision is not "how do I fix it" but "where does this failure live?" There are exactly two answers, and they look identical from the outside while having opposite fixes:

• The integration layer — your code around the model: request construction, response parsing, tool wiring, control flow.
• The model output — what Claude actually produced given a correctly formed request.

Guess wrong and you waste hours: you tune the prompt when the bug is that you read content[0].text off a thinking block, or you patch parsing when the real problem is the model ignored an instruction. Learn to read the symptom.

The diagnostic that decides it: replay the exact request. Log the full request body and the response._request_id, then send the identical request again.

• If the same request reproduces the failure every time, it is deterministic, almost always the integration layer. Your code does the same wrong thing on every run.
• If the output varies run to run, it is the model. The lever is now prompt, context, or effort, not a code patch.

8. Error type identification and recovery strategy

Recovery starts with one binary: is this error retryable or fatal? Retrying a fatal error is a bug (you hammer the API with a request that can never succeed). Failing a retryable error hard is also a bug (you drop work that would have gone through on the next attempt).

The SDK already does the right thing for the retryable set: it auto-retries connection errors, 408, 409, 429, and ≥ 500 twice with exponential backoff (max_retries defaults to 2; set 0 to disable). Timeouts are retried too, so worst-case wall clock is timeout × (max_retries + 1). Your custom code only needs to add what the SDK does not.

import anthropic

RETRYABLE = (anthropic.RateLimitError,
             anthropic.InternalServerError,
             anthropic.APIConnectionError)

def classify(err: Exception) -> str:
    """Return 'retry' or 'fatal' — the only decision that matters first."""
    if isinstance(err, RETRYABLE):
        return "retry"
    if isinstance(err, (anthropic.BadRequestError,        # 400
                        anthropic.AuthenticationError,    # 401
                        anthropic.PermissionDeniedError,  # 403
                        anthropic.NotFoundError)):         # 404
        return "fatal"
    if isinstance(err, anthropic.APIStatusError):          # catch-all by code
        return "retry" if err.status_code >= 500 else "fatal"
    return "fatal"

try:
    response = client.messages.create(...)
except anthropic.APIError as err:
    if classify(err) == "fatal":
        log.error("fatal, not retrying: %s (request_id=%s)",
                  err, getattr(err, "request_id", None))
        raise
    # else: let the SDK's built-in backoff handle it, or enqueue for later.

9. Trace analysis: log it or you cannot debug it

You cannot analyze a trace you did not record. The floor for every call: model ID, stop_reason, all four usage fields, response._request_id, latency, and every tool call with its arguments and result. Miss stop_reason and you cannot tell after the fact whether an answer was truncated. Miss the usage breakdown and you cannot tell whether a prompt-cache hit silently became a miss and doubled your bill.

import time, json, logging
log = logging.getLogger("llm")

def logged_create(client, **kwargs):
    t0 = time.perf_counter()
    resp = client.messages.create(**kwargs)
    u = resp.usage
    log.info(json.dumps({
        "model":        resp.model,
        "request_id":   resp._request_id,          # public despite the underscore
        "stop_reason":  resp.stop_reason,          # truncation shows up here
        "latency_ms":   round((time.perf_counter() - t0) * 1000),
        "usage": {
            "input":          u.input_tokens,             # uncached remainder only
            "cache_creation": u.cache_creation_input_tokens,
            "cache_read":     u.cache_read_input_tokens,  # cache hit shows up here
            "output":         u.output_tokens,
        },
    }))
    # Total prompt tokens = input + cache_creation + cache_read.
    return resp

10. Evals: a prompt change is a code change

Here is the mindset shift the exam is really testing: editing a prompt is editing code, and code changes need tests. "The new prompt seems better" is a vibe, not evidence. An eval turns it into a number you can compare across prompt versions and across model versions, which is exactly what you need before a model upgrade or a prompt edit ships.

An eval harness has three parts: a fixed set of inputs, a grader (exact match, schema validity, or an LLM judge with a rubric), and a score you record per version. Run it in CI so a regression blocks the merge.

import json, jsonschema

# 1. Fixed inputs with expected properties.
CASES = [
    {"text": "Refund my order #A17, it arrived broken.",
     "expect": {"intent": "refund", "order_id": "A17"}},
    {"text": "What are your hours?",
     "expect": {"intent": "hours"}},
]

SCHEMA = {
    "type": "object",
    "properties": {"intent": {"type": "string"}, "order_id": {"type": "string"}},
    "required": ["intent"], "additionalProperties": False,
}

def grade(output: dict, case: dict) -> float:
    # (a) schema validity — a free, deterministic grader.
    try:
        jsonschema.validate(output, SCHEMA)
    except jsonschema.ValidationError:
        return 0.0
    # (b) exact-match the fields we care about.
    return 1.0 if all(output.get(k) == v for k, v in case["expect"].items()) else 0.5

def run_eval(prompt_version: str) -> float:
    scores = []
    for case in CASES:
        resp = client.messages.create(
            model="claude-haiku-4-5",
            max_tokens=512,
            output_config={"format": {"type": "json_schema", "schema": SCHEMA}},
            system=PROMPTS[prompt_version],
            messages=[{"role": "user", "content": case["text"]}],
        )
        out = json.loads(next(b.text for b in resp.content if b.type == "text"))
        scores.append(grade(out, case))
    return sum(scores) / len(scores)

# Compare versions the way you compare test runs.
print("v1:", run_eval("v1"), " v2:", run_eval("v2"))
assert run_eval("v2") >= run_eval("v1"), "prompt regression — block the merge"

11. Structured output validation is a free grader

Notice what the eval leaned on: schema validity is a deterministic grader that costs you nothing extra. When you request structured output with output_config={"format": {"type": "json_schema", ...}} (or client.messages.parse(..., output_format=Model)), every response either conforms or it does not, a hard pass/fail with no judgment call and no second model in the loop. Make schema validity the first gate in any eval that produces structured data; only reach for an LLM judge when you need to grade something a schema cannot express, like tone or factual accuracy.

Lab Exercise: Diagnose, Then Fix

Objective: practise the isolate-before-you-fix discipline and stand up a minimal eval. About 45 minutes.

1. Instrument first. Wrap your client in the logged_create helper. Confirm every call emits model, stop_reason, all four usage fields, _request_id, and latency.
2. Isolate a deterministic bug. Point your code at a bogus model ID. Observe it fails identically on every run, and note from the symptom table that a reproducible failure is the integration layer. Fix the model ID, not the prompt.
3. Isolate a probabilistic bug. Give a deliberately vague instruction and run it five times. Observe the output varies. Conclude the lever is the prompt, and tighten it; do not touch parsing.
4. Classify errors. Trigger a 404 (bad model) and a 429 (hammer the endpoint or mock it). Run both through classify() and confirm one returns fatal and the other retry. Set max_retries=0 and watch the 429 stop being handled for you.
5. Build the eval. Write three input cases with expected fields and a schema. Grade with schema validity plus exact match. Record a score.
6. Regression-gate it. Change the system prompt, re-run the eval, and compare scores. Wire the assert into a CI step so a lower score fails the build.
7. Claude Code in CI. Add a pinned, headless claude -p review step to a workflow with a least-privilege settings.json. Confirm it exits non-zero on a planted issue.

Steps 2 and 3 are the whole point. If you can reliably tell a reproducible failure from a varying one, you already debug Claude apps better than most.

Key Takeaways

• These two domains are ~5.7% combined, about three questions. Learn them well, then spend your remaining time on Domains 2 and 5.
• Claude Code components: Rules are always-on policy, Skills and Commands are invoked, Agents are dispatched subagents, Agent Memory persists across sessions.
• CLAUDE.md composes down the tree: durable conventions belong there; one-off task instructions do not. settings.json holds permissions, hooks, and env vars.
• A Claude Code run in CI is a build step: pin the version, review the config, grant least privilege. Headless is the -p flag.
• Isolate before you fix. Integration-layer failures reproduce on replay; model-output failures vary. Log the request body and _request_id so you can replay.
• Classify every error as retryable (429, 5xx, connection, 408/409) or fatal (400, 401, 403, 404). The SDK already retries the retryable set twice.
• Always log stop_reason and all four usage fields; without them you cannot diagnose truncation or a cache miss after the fact.
• A prompt change is a code change: gate it with an eval. Schema validity is a free, deterministic grader; make it your first gate.