Model Selection & Optimization

1. LLM Fundamentals: Tokens, Context, and Non-Determinism

Everything in this lab is downstream of one fact: Claude reads and writes tokens, not words or characters. A token is a chunk of text: often a word fragment, sometimes punctuation or whitespace. English prose runs roughly 3–4 characters per token; code, JSON, and non-English text tokenize far less efficiently. You are billed per token, your context window is measured in tokens, and your rate limits are counted in tokens. If your mental model is "words," every cost estimate you make is wrong.

The context window is a hard budget shared by input and output. It is not "how much you can send"; it is the ceiling on input tokens plus generated output tokens combined. On a 1M-token model, a 950K-token prompt leaves room for only ~50K of output no matter what you set max_tokens to. Overrun the window and the request fails; you do not silently get a truncated prompt.

Claude generates text by next-token prediction: given everything so far, it produces a probability distribution over the next token, emits one, appends it, and repeats. This is why streaming works (tokens are produced one at a time) and why longer outputs cost proportionally more time and money.

Prompting techniques you will be asked to name

The exam tests the vocabulary of few-shot prompting. These describe how many worked examples you place in the prompt before the real task:

• Zero-shot — instruction only, no examples. "Classify this review as positive or negative."
• Single-shot (one-shot) — one example demonstrating the input→output format, then the real input.
• Multi-shot (few-shot) — several examples. Best when the task has a specific format, edge cases, or a labeling convention Claude must match exactly.

Multi-shot examples are stable, shared prefix content, which makes them a prime caching target (Section 6). Put them before your varying question, and put the cache breakpoint at the end of the examples.

2. Model Options: Adaptive Thinking, Effort, and Fast Mode

Adaptive thinking is the current mode. Rather than you allocating a fixed thinking budget, Claude decides when and how much to reason. You control the depth (and the total token spend) with the effort parameter, which lives inside output_config, not at the top level. Valid values are low, medium, high, xhigh, and max; the default is high.

response = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=16000,
    thinking={"type": "adaptive", "display": "summarized"},
    output_config={"effort": "high"},   # low | medium | high | xhigh | max  (default: high)
    messages=[{"role": "user", "content": "Refactor this module for testability..."}],
)

Two subtleties the exam rewards. First, on Opus-tier models omitting thinking means no thinking at all; set {"type": "adaptive"} explicitly if you want it. Second, display defaults to "omitted", so reasoning renders as empty thinking blocks unless you ask for "summarized". Effort is not just a quality dial: on agentic work, higher effort often reduces total cost because a better plan up front cuts the number of tool-calling turns.

Fast mode

Fast mode is a research preview on Opus 4.8 and 4.7 only. It runs the same model at up to ~2.5× output tokens per second, at premium pricing. Three things are required on every request: use the beta endpoint (client.beta.messages.create), pass the beta flag fast-mode-2026-02-01, and set speed="fast" as a top-level parameter (not a header, not in extra_body). It is not available with the Batch API.

response = client.beta.messages.create(
    model="claude-opus-4-8", max_tokens=4096,
    speed="fast",
    betas=["fast-mode-2026-02-01"],
    messages=[{"role": "user", "content": "Draft a reply to this ticket."}],
)
# response.usage.speed reports which speed actually ran.

3. Model Selection: Match the Tier to the Task

There are four tiers to choose from. Prices are per million tokens (MTok), input / output.

Learn the reasoning, not the table. The exam does not want you to memorize prices; it wants you to defend a choice. Two failure modes cost real money:

• Defaulting to the top model everywhere wastes both cost and latency. A high-volume stream of short customer replies where speed and price dominate does not need Fable 5; a cheaper, faster tier gives you the same user-visible result for a fraction of the bill.
• Using the cheapest tier for hard reasoning is a false economy. The tokens you "save" come back as rework: wrong answers, extra turns, human correction. Match the tier to the task's actual difficulty.

Effort is a second, orthogonal dial. Model tier sets the capability ceiling; effort sets how hard the model works within it. They combine. A common winning pattern on agentic workloads is a mid tier at high/xhigh effort: higher effort reduces turn count, and fewer turns can cost less overall than a top-tier model run at low effort that flails through many rounds.

4. Breaking Behavior Changes Across Releases

This is its own exam objective, and it is the reason to pin the model ID. A model upgrade is a code change (behavior, defaults, and even the request shape can differ between releases), so it needs an eval run, not a silent swap.

Real, current breakages you must handle when selecting or migrating models:

5. Token Counting and Cost Modeling

Before you can optimize cost, you have to measure tokens accurately. Use the API's own counter, which is model-specific; pass the same model ID you will run inference on.

resp = client.messages.count_tokens(
    model="claude-opus-4-8",
    system=system_prompt,
    messages=[{"role": "user", "content": open("CLAUDE.md").read()}],
)
print(resp.input_tokens)   # accurate, model-specific token count

After a request, response.usage is the source of truth for what you were billed. But reading one field is the classic mistake:

u = response.usage

# The four fields that matter:
u.input_tokens                  # UNCACHED remainder only (full price)
u.cache_creation_input_tokens   # written to cache this request (~1.25x write)
u.cache_read_input_tokens       # served from cache this request (~0.1x)
u.output_tokens                 # generated (full output price)

# Total prompt size is the SUM of the three input fields:
total_prompt = (
    u.input_tokens
    + u.cache_creation_input_tokens
    + u.cache_read_input_tokens
)

# A real per-request cost model (Opus 4.8: $5 in / $25 out per MTok):
IN, OUT = 5.0 / 1e6, 25.0 / 1e6
cost = (
    u.input_tokens                * IN
    + u.cache_creation_input_tokens * IN * 1.25   # 5-min write premium
    + u.cache_read_input_tokens     * IN * 0.10   # cache read
    + u.output_tokens               * OUT
)
print(f"request cost: ${cost:.5f}")

6. Prompt Caching: The Biggest Cost Lever

Caching a large stable prefix can cut cost by up to ~90% on the cached portion. It is also the feature people most often silently misconfigure: there is no error when it fails to cache, just a bill that never goes down. Master this and you master this domain.

How it works

Caching is a prefix match. The cache key is the exact bytes of the rendered prompt up to each cache_control breakpoint. Any byte change anywhere in the prefix invalidates everything after it. Render order is fixed: tools → system → messages. A breakpoint on the last system block therefore caches tools and system together.

response = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=16000,
    system=[
        {"type": "text", "text": SHORT_ROLE},
        {
            "type": "text",
            "text": LARGE_STABLE_CONTEXT,          # e.g. 50KB of frozen docs
            "cache_control": {"type": "ephemeral"},  # optional: "ttl": "1h"
        },
    ],
    messages=[{"role": "user", "content": user_question}],  # volatile: goes AFTER the breakpoint
)

You get a maximum of 4 breakpoints per request. A top-level cache_control={"type": "ephemeral"} on the request auto-places one on the last cacheable block, the simplest option when you do not need fine placement.

The 1-hour TTL keeps entries alive across gaps in bursty traffic, but the doubled write cost means it needs more reads to pay off. If a prefix is used only once, do not cache it; you would pay the write premium for zero reads.

Silent invalidators: the weak / strong pair

The most common cause of a cache that never hits is a byte that changes every request, buried in the prefix. Here is the canonical bug (a timestamp interpolated into the system prompt) and its fix.

from datetime import datetime

# datetime.now() changes the prefix bytes on EVERY request.
# Every request writes a new cache entry; none is ever read.
system = [{
    "type": "text",
    "text": f"You are a support agent. Current time: {datetime.now()}.\n\n{LARGE_STABLE_CONTEXT}",
    "cache_control": {"type": "ephemeral"},
}]
# usage.cache_read_input_tokens will be 0 forever.

# Freeze the system prompt. Put the timestamp in the user turn,
# AFTER the cached prefix, where it invalidates nothing before it.
system = [{
    "type": "text",
    "text": f"You are a support agent.\n\n{LARGE_STABLE_CONTEXT}",
    "cache_control": {"type": "ephemeral"},
}]
messages = [{
    "role": "user",
    "content": f"[time: {datetime.now()}] {user_question}",
}]
# Now cache_read_input_tokens climbs on every repeat request.

Verify hits, then hunt the invalidator

Never assume caching works. Confirm it against usage. If cache_read_input_tokens is 0 across repeated identical-prefix requests, something in the prefix is changing.

def report(resp, label):
    u = resp.usage
    print(f"{label}: write={u.cache_creation_input_tokens} "
          f"read={u.cache_read_input_tokens} uncached={u.input_tokens}")

r1 = client.messages.create(**params); report(r1, "req 1")   # write > 0, read == 0  (cold)
r2 = client.messages.create(**params); report(r2, "req 2")   # read > 0             (warm)

# If req 2 still shows read == 0 with an identical prefix, an invalidator is at work.
# Diff the two rendered prompts byte-for-byte to find the culprit.

Two placement traps

• Shared prefix, varying suffix. When many requests share a large preamble but differ in the final question, put the breakpoint at the end of the shared portion, not the end of the whole prompt. If you cache to the end of a prompt whose tail differs every time, every request writes a distinct entry and nothing is ever read.
• Concurrency. A cache entry becomes readable only after the first response begins streaming. Fire N parallel identical-prefix requests at once and all N pay full price; none can read what the others are still writing. Fan-out pattern: send one request, await the first token, then fire the remaining N−1 so they read the cache the first one just wrote.

7. Batching for Cost

The last cost lever is the Message Batches API (covered in Lab 1): it processes high-volume, latency-tolerant workloads within a 24-hour window at 50% of standard token cost. Caching and batching are complementary: a batch with a shared cached system prompt gets both discounts. Reach for batching whenever the user is not waiting and the work is large. Note that fast mode is not available with the Batch API; the two optimizations point in opposite directions (latency vs. throughput).

8. Lab Exercise: Measure, Break, Model, Sweep

Objective: prove to yourself that caching works, that you can break it, and that you can read the true cost off usage.

1. Measure a cache hit. Build a request with a system prompt of at least 4096 tokens on claude-opus-4-8 and a cache_control breakpoint on the last system block. Send it twice with an identical prefix. Confirm request 1 shows cache_creation_input_tokens > 0, cache_read_input_tokens == 0 and request 2 shows cache_read_input_tokens > 0.
2. Break it deliberately. Interpolate datetime.now() into the system prompt and re-run the pair. Watch cache_read_input_tokens stay at 0. Now move the timestamp into the user turn and confirm the hit returns. You have just reproduced the single most common caching bug.
3. Prove the minimum. Shrink the system prompt to ~3,000 tokens on Opus 4.8. Observe that cache_creation_input_tokens is 0 with no error: it is below the 4096-token minimum and silently does not cache.
4. Build a cost model. Write a function that takes response.usage and returns dollars, using all four fields and the write/read multipliers. Run it on your cached and uncached requests and confirm the cached run is cheaper, and that summing all three input fields recovers the full prompt size.
5. Sweep effort levels. Run the same agentic task at low, high, and xhigh. Record output tokens, turn count, and total cost at each. Confirm for yourself that higher effort can lower total cost by cutting turns.

Step 2 is the one that shows up in production. Do it until the failure is obvious to you at a glance.

Key Takeaways

• Claude bills in tokens, not words. The context window is a hard budget shared by input + output. Outputs are never guaranteed identical, even without sampling params.
• Adaptive thinking is current; effort (low…max, default high) lives in output_config. Higher effort on agentic work often lowers total cost by cutting turns. Fast mode is Opus 4.8/4.7 only, beta, and not on the Batch API.
• Match the tier to the task: top model everywhere wastes money and latency; cheapest tier on hard reasoning is paid back in rework. Tier and effort are orthogonal dials.
• A model upgrade is a code change: pin the model ID and run evals before migrating. budget_tokens, sampling params, and prefill all return 400; display defaults to "omitted"; omitting thinking on Opus-tier means no thinking.
• Count tokens with count_tokens, never tiktoken. input_tokens is the uncached remainder: sum all three input fields for the real prompt size, and build your cost model from usage.
• Prompt caching is the biggest lever: a prefix match, render order tools→system→messages, max 4 breakpoints. Reads ~0.1×, writes 1.25× (5m) / 2× (1h). Below the model's minimum prefix (4096 on Opus 4.8) it silently does not cache. Verify with cache_read_input_tokens; hunt silent invalidators when it stays 0.
• Batching gives 50% off for latency-tolerant volume, and it stacks with caching.