Capstone: End-to-End Architecture Review

1. How to Use This Lab

This is an architecture review, not a tutorial. There is nothing to install and no notebook to run. The spine of the lab is a single realistic system, presented the way a design doc would present it: neutrally, confidently, as if a competent team wrote it. It is not competent. It contains at least one seeded defect in every exam domain.

Do the work in this order:

1. Read the brief and the proposed architecture as a reviewer. Do not scroll ahead to the findings. Open a blank page and write down every defect you can name, tagged with the domain it belongs to. Aim for seven.
2. Force yourself to say the repair out loud. Finding "the retrieval is wrong" is worth little; "swap dense-vector retrieval for an exact clause lookup because the query is a key, not a concept" is the answer the exam rewards.
3. Then read the annotated review. Score yourself: one point per defect found and correctly repaired. Fewer than five, and the linked lab for each miss is your revision list.

The defects are architectural: wrong pattern, wrong retrieval, missing gate, wrong metric, wrong prompt order. None of them is a deprecated API parameter. (A reviewer would of course also reject temperature= or a budget_tokens thinking budget on sight, because both now return HTTP 400 on current models, but that is a syntax error, not an architecture defect, and it is not what this lab is testing.)

2. The Brief

A global bank runs a Regulatory Change Assistant. Analysts track regulatory publications across 14 jurisdictions. For each new publication they must determine which of the bank's internal policies the change affects, draft an impact memo, and route that memo for approval. The bank receives roughly 300 documents a day. Regulators require that every assertion in a memo be traceable to a source: a claim with no citation is a compliance defect on its own.

The team building the assistant has written a design doc. Before you read their architecture, hold their own requirements in front of you, because the requirements are what the architecture will be judged against.

3. The Proposed Architecture

Here is the team's design, in their words. Read it as written; it sounds reasonable. That is the point.

"We built an autonomous agent. On each new publication it decides for itself which tools to call: it searches the policy corpus, reasons about impact, and files the memo. We give it a broad tool set so it can handle anything a regulator throws at us. Retrieval is a vector search over the embedded policy corpus. If the agent finds no affected policy, it files a 'no impact' verdict automatically so analysts only ever see the memos that matter. We measure success by average answer latency and by a rubric score from an LLM judge. We have promised the regulator 99% accuracy on live traffic."

The component table below is how the design doc summarizes it. Every row hides at least one defect.

And here is the core call, roughly as it appears in the codebase. It uses the current API (adaptive thinking, no removed sampling parameters) so nothing here throws a 400. The defect is architectural: look at the order of the system blocks.

import anthropic
from datetime import datetime

client = anthropic.Anthropic()

# The regulatory publication we are analysing today. It changes every run.
incoming_publication = load_todays_publication()

response = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=8000,
    thinking={"type": "adaptive"},
    system=[
        # A fresh timestamp at the very top "so the model knows the date".
        # This byte changes every request, so nothing after it can ever cache.
        {"type": "text",
         "text": f"You are a regulatory analyst. Today is {datetime.now().isoformat()}."},

        # The VARYING document, placed before the static corpus...
        {"type": "text", "text": incoming_publication},

        # ...and the 60,000-token policy corpus AFTER it, with a cache breakpoint
        # that is therefore downstream of two invalidators and never gets reused.
        {"type": "text", "text": POLICY_CORPUS,
         "cache_control": {"type": "ephemeral"}},
    ],
    tools=[
        {"name": "search_policies", "description": "...", "input_schema": {...}},
        {"name": "file_impact_memo", "description": "...", "input_schema": {...}},
        {"name": "delete_policy", "description": "...", "input_schema": {...}},  # never needed
    ],
    messages=[{"role": "user",
               "content": "Decide which policies this change affects, then file the memo."}],
)

4. The Review, Domain by Domain

Seven findings, one per domain. Each has the same shape: the finding, why it fails, the repair, and the lab that covers it in full.

D1 — Solution Design: an agent where a workflow was sufficient

The finding. The design hands control flow to the model: an autonomous agent that decides which tools to call. But the brief's own task description enumerates the steps: retrieve affected policies, draft a memo, route it. The path never changes.

Why it fails. An agent buys non-determinism, higher cost, higher latency, and an audit trail nobody can reconstruct, all to rediscover a sequence the team already wrote down. For a regulated process, the un-reconstructable audit trail is the worst part: you cannot tell a regulator why a given memo was produced.

The repair. A workflow whose classification step is a single augmented-LLM call over the retrieved policy terms, with code owning the sequence. Reserve the agent for the genuinely-unknown-path case, which this is not. See Lab 1 and Lab 2.

D2 — Prompting & Context: the cache can never hit

The finding. Two invalidators sit in front of the corpus. A datetime.now() preamble at the very top changes on every request, and the varying publication is placed before the static 60K-token corpus. The cache breakpoint is downstream of both.

Why it fails. Caching is a prefix match: any byte change before a breakpoint invalidates everything after it. Render order is tools → system → messages, so the stable content must physically come first. As written, the bank re-processes 60K tokens at full price 300 times a day and blows the p95 latency SLA that a warm cache would have protected.

The repair. Put the corpus first with the cache_control breakpoint; move the date into the user turn where it belongs; put the varying publication last, after the breakpoint. See Lab 3.

D3 — Integration: capability bloat, a confused deputy, and the wrong retrieval

The finding. Three integration defects in one row. The agent holds a delete_policy tool it never needs. It authenticates as one service account holding the union of every jurisdiction's permissions. And it retrieves exact clauses like "Article 17(2)" with dense-vector similarity search.

Why it fails. The unused destructive tool is pure blast radius: a tool that can only hurt you. The union-permission service account is the classic confused deputy: a prompt-injected publication could make the assistant act with privileges far beyond the task, across jurisdictions it should never touch. And dense vectors answer "what is similar to this?" when the query is an exact key: "Article 17(2)" is a lookup, and embedding it is architecture theatre that will silently return near-miss clauses.

The repair. Delete delete_policy (least privilege). Scope credentials to the jurisdiction of the document being processed. Replace vector search with an exact clause lookup (a keyed index), keeping semantic search only for the genuinely fuzzy "what else might this touch?" pass. See Lab 4 and Lab 5.

D4 — Evaluation: measuring the wrong things, with an uncalibrated judge

The finding. Success is measured by average latency and by a rubric scored with an LLM judge that was never calibrated against human graders. There is no held-out set.

Why it fails. Average latency hides the tail; the SLA is p95, and an average can look healthy while one memo in twenty times out. An LLM judge that has never been checked against humans is an opinion of unknown correlation with ground truth: you do not know whether a high rubric score means a good memo. And with no held-out set, any prompt change is tuned against the same data it is scored on, so improvements are indistinguishable from overfitting.

The repair. Track p95, not the mean. Calibrate the judge against a human-labelled sample and report the agreement rate before you trust it. Freeze a held-out regression set that no prompt change is allowed to see. See Lab 6 and Lab 7.

D5 — Governance: an auto-filed verdict on the highest-blast-radius branch

The finding. A "no impact" verdict auto-files with no human gate, on the exact branch the brief says produces a regulatory finding when wrong. PII is sent to the model unredacted, and outputs are retained indefinitely.

Why it fails. The brief spends its words on one fact: a wrong "no impact" is a regulatory event. That is a governance constraint, not a UX preference, and the design has automated precisely the decision that most needs a human. Unredacted PII to the model widens the GDPR surface for no benefit; unbounded retention turns every stored memo into a standing liability under data-residency and right-to-erasure rules.

The repair. Put a named human on the "no impact" path (auto-file only positive, low-risk "impact found & routed" cases if at all). Redact PII before the call where it is not needed for the decision. Set a retention policy tied to the residency requirement. See Lab 8.

D6 — Stakeholder & Lifecycle: an accuracy SLA nobody can honour

The finding. The team promised the regulator 99% accuracy on live traffic.

Why it fails. "Accuracy on live traffic" has no denominator you control: you cannot know the true label on a novel publication until an analyst rules on it, so you cannot measure the number you promised in real time, let alone guarantee it. It also anchors the relationship to a figure that a single hard month will breach, converting a capability into a liability. This is a lifecycle-management failure: a commitment made without an evaluation methodology that can support it.

The repair. Commit to what you can evidence: a measured accuracy on a frozen, human-labelled benchmark, a documented human-in-the-loop gate on the high-risk branch, and a monitored p95 latency. Promise the process, not a live-traffic percentage. See Lab 9.

D7 — Developer Productivity & Ops: secrets in CLAUDE.md

The finding. API keys and credentials are pasted into the project CLAUDE.md so the team shares one setup.

Why it fails. CLAUDE.md is committed to the repository and loaded into context on every run; it is documentation, not a vault. Secrets there are exposed to everyone with repo access, leak into logs and traces, and are near-impossible to rotate cleanly. For a bank this is an audit finding by itself.

The repair. Move secrets to environment variables or a managed secret store; reference them from code, never from context. Keep CLAUDE.md for the operational knowledge it is good at: how to run the eval suite, where the runbook lives. See Lab 10.

5. The Repaired Architecture

The repaired core call inverts the two things that mattered most in the code: the prompt order (stable corpus first, behind a cache breakpoint) and the output contract (structured output forcing a citation on every assertion). The date and the varying publication both move into the user turn, after the breakpoint.

import anthropic

client = anthropic.Anthropic()

# Every assertion must carry a source. The schema makes that a hard contract:
# an uncited claim cannot validate, so the regulator's rule is enforced by shape.
IMPACT_SCHEMA = {
    "type": "json_schema",
    "schema": {
        "type": "object",
        "properties": {
            "verdict": {"type": "string", "enum": ["impact", "no_impact"]},
            "affected_policies": {"type": "array", "items": {"type": "string"}},
            "cited_sources": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "clause": {"type": "string"},   # e.g. "Article 17(2)"
                        "quote": {"type": "string"},
                    },
                    "required": ["clause", "quote"],
                    "additionalProperties": False,
                },
            },
        },
        "required": ["verdict", "affected_policies", "cited_sources"],
        "additionalProperties": False,
    },
}

response = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=8000,
    thinking={"type": "adaptive"},
    output_config={"effort": "high", "format": IMPACT_SCHEMA},
    system=[
        # STABLE prefix first: the 60K-token corpus, cached across all 300 docs/day.
        {"type": "text", "text": POLICY_CORPUS,
         "cache_control": {"type": "ephemeral"}},
    ],
    messages=[
        # Everything volatile goes AFTER the breakpoint: the date and today's doc.
        {"role": "user", "content": build_user_turn(today_iso, incoming_publication)},
    ],
)

To prove the fix rather than assume it, read the usage back. If cache_read_input_tokens is zero across repeated runs, a silent invalidator is still sitting in the prefix, and remember that input_tokens reports only the uncached remainder, so the corpus tokens should show up under the cache fields, not here.

u = response.usage
print("written to cache:", u.cache_creation_input_tokens)  # ~1.25x on the cold write
print("served from cache:", u.cache_read_input_tokens)     # ~0.1x on every warm hit
print("uncached remainder:", u.input_tokens)               # today's doc + user turn only

# The corpus should appear under the cache fields, never under input_tokens.
assert u.cache_read_input_tokens > 0, "prefix still has a silent invalidator"

The rest of the repaired design maps cleanly onto the five planes from Lab 1. Every plane now answers its question with the defect closed.

6. The Artifacts You Would Hand Over

A design review is not finished when the code is right; it is finished when the next engineer, the regulator, and the on-call responder can each do their job from what you leave behind. This is the lifecycle deliverable set (Lab 9). Treat it as a checklist: a missing row is a defect the exam will ask about.

7. The Cost Model

Do the arithmetic, because "caching saves money" is not a number and a CFO does not fund adjectives. The figures below are an illustrative model, not a price quote: Claude Opus 4.8 at $5.00 per million input tokens and $25.00 per million output, cache reads at roughly 0.1× input and cold cache writes at 1.25× (5-minute TTL). Assume a 60,000-token policy corpus, a 4,000-token publication, and a 2,000-token memo, at 300 documents/day.

That is the cache hit turning a large daily number into a small one: roughly a 3× reduction, driven almost entirely by the corpus moving from full price to a tenth of it. The single ordering defect in D2 was costing the bank about $76 a day, or ~$28,000 a year, for nothing.

One more distinction the exam likes: cost per request is not cost per resolved task. If 20% of memos need a second pass (a rerun, an analyst correction fed back), you make ~360 model calls to resolve 300 business tasks. At the cached rate:

The CFO cares about the second number, because it is the one tied to a completed unit of work. Report cost per resolved task and you are speaking the pillar (cost per document) the business actually funds.

8. Final Self-Assessment

This is your exit criterion before you sit the exam. For each domain, can you (unprompted, with no scenario pointing at it) do the thing in the right column? If any row is a "no", the linked lab is where you go before booking the test.

9. Where to Go Next

You have now audited a full end-to-end system across every domain the exam weights. Two things remain before you sit CCAR-P. Take the practice exam under time (63 items, 120 minutes) and treat any miss as a pointer back to the lab that owns it. Keep the quick reference open while you review; it is the condensed form of everything the eleven labs built. When the practice exam feels slow because you keep finding the defect before you finish reading the options, you are ready.

Key Takeaways

• Design (D1): fixed, enumerable steps mean a workflow; an autonomous agent on a known path buys non-determinism and an audit trail you cannot reconstruct: fatal in a regulated process.
• Prompting (D2): caching is a prefix match. A datetime.now() preamble or a varying document placed before the static corpus invalidates the cache on every request: stable content first, volatile content after the breakpoint.
• Integration (D3): delete tools you never need, scope credentials to the task (not the union of all permissions), and use an exact lookup (not dense vectors) when the query is a key like "Article 17(2)".
• Evaluation (D4): track p95 not the mean, calibrate an LLM judge against humans before trusting it, and freeze a held-out set so improvement is not overfitting.
• Governance (D5): put the human gate on the branch whose wrong answer is a regulatory finding; redact PII before the call; bound retention to the residency rule.
• Stakeholder (D6): never promise a live-traffic accuracy figure you cannot measure in real time: commit to a gated process and a benchmark you can evidence.
• Dev Productivity (D7): secrets belong in a secret store, never in CLAUDE.md, which is committed and loaded into context every run.
• The through-line: across all seven domains the correct repair subtracts: an agent, a tool, permissions, a promise, a secret. When two options differ by a capability, the one that removes it is usually right.
• The economics: report cost per resolved task, not per request: it is the number tied to a completed unit of work, and the one the business funds.