Stakeholder Communication & Lifecycle Management
Read this number before anything else: Domain 6 is worth nine items: 14% of the exam, more than Models & Prompting. Engineers systematically under-study it because it has no API surface: nothing here has a request body, a status code, or a parameter to memorize. That is exactly why it is on the exam, and exactly why it decides passes. This lab is about the work that happens in rooms, not in code (discovery, decisions, expectations, and the handoff), and it is written by someone who has been in the room when a technically sound project was cancelled anyway.
- Conduct structured discovery and requirement gathering
- Communicate architectural decisions and trade-offs
- Manage stakeholder feedback loops and expectation alignment (including SLAs)
- Document architectures and provide implementation guidance
- Support lifecycle phases (discovery, design, handoff, monitoring, iteration)
1. Structured Discovery
Discovery is the phase where projects are quietly saved or quietly doomed, and it happens before a line of code exists. The mistake that defines a bad discovery is simple: it collects feature requests instead of decisions and constraints. A stakeholder who says "we want it to summarize tickets and also draft replies and also detect sentiment" has handed you a wish list. Your job is to convert wishes into a single named decision, its ground truth, and its blast radius, because those are what you can architect against.
Run discovery as a question framework, grouped by theme. You are not filling in a form; you are hunting for the answer that kills the project early, while it is cheap to kill.
| Theme | The question | What a bad answer tells you |
|---|---|---|
| Decision | What decision or action changes because this system exists? Who acts on the output? | "Lots of people, in lots of ways" means there is no decision, only a demo. |
| Baseline | What is the current process, and how is its performance measured today? | No baseline means no way to prove value at renewal. |
| Ground truth | How do you know today when a human got this wrong? Who arbitrates a dispute? | If nobody can say, you cannot evaluate the system either (Lab 6). |
| Volume & peak | How many per day? What is the peak-hour multiple over the average? | Designing to the average and getting paged at the peak is the classic capacity failure. |
| Latency tolerance | How long can the user wait before they abandon the tool or route around it? | A number nobody will commit to is a number you will breach. |
| Blast radius | What is the cost of one wrong answer reaching production: money, safety, a regulator? | This sets whether you need a human gate (Lab 8), not a UX preference. |
| Regulatory regime | What law governs this data and this decision? Who signs off on compliance? | Discovered late, this rewrites the architecture, not a config value. |
| Data residency | Where may this data physically live and move? Who owns the corpus? | An unowned corpus has no one to keep it current; the system rots. |
| Definition of done | What does success look like at 90 days? What number would make you renew? | If they cannot name it, you are building to a target that will move. |
Two of these questions are worth more than the rest combined, because they surface a doomed project while it is still cheap to walk away:
- "How do you know today when a human got this wrong?" If the honest answer is "we don't, really," then you have no ground truth, no eval set, and no way to defend the system's quality later. That is not a detail to resolve during build; it is a reason to stop.
- "Who loses if this works?" Every automation redistributes something: work, credit, headcount, control. The person who loses is often the person who owns the data you need or the sign-off you require. Find them in discovery, not in month five when they quietly withhold the corpus.
2. Requirements to Architecture Traceability
A requirement you cannot turn into a verifiable acceptance criterion is not a requirement; it is an aspiration, and it should be labelled one out loud so nobody mistakes it for a commitment. The discipline is a table that runs left to right: every business requirement becomes an architectural constraint, and every constraint becomes something you can measure and pass or fail.
| Business requirement | Architectural constraint | Verifiable acceptance criterion |
|---|---|---|
| Denials must be attributable to a named human | Human-in-the-loop gate on the denial branch; no auto-execute | 100% of denials in the audit log carry a named approver ID |
| Claimants should not wait long | End-to-end routing SLO under a fixed bound | p95 time-to-route ≤ 60 minutes measured on live traffic for 30 days |
| Answers must be grounded in policy | Required citation field validated against source text | ≥ 98% of outputs carry a citation that resolves to a real clause |
| The system should feel trustworthy | — no constraint expressible — | Aspiration, not a requirement. Restate it or drop it. |
| Handle current volume with headroom | Capacity sized to peak, not average, with an autoscale ceiling | Load test sustains 2.5× mean throughput with p95 within SLO |
The fourth row is the important one. "Feel trustworthy" cannot be measured, so it cannot be an acceptance criterion, so it is not a requirement. Either it decomposes into something concrete (citation validity, a visible confidence banding, an explanation panel) or you say plainly that it is an aspiration and stop pretending you have committed to it. Undeclared aspirations are how a green dashboard still gets a project cancelled.
3. Communicating Decisions: The ADR
An Architecture Decision Record is the durable memory of why the system is shaped the way it is. Its value is not the decision; anyone can announce a decision. Its value is the two sections that a memo omits: the alternatives you rejected and why, and the negative consequences you accepted. Those sections are what let the next architect (or you, in a year, having forgotten) reopen the question with the same information you had, instead of relitigating it from zero.
Here is a complete, filled-in ADR for a real CCAR-P-flavored decision. Read the alternatives and the negative consequences closely; they are the load-bearing parts.
ADR-014: Human-gated workflow for claims triage, not an autonomous agent
Status: Accepted · Date: 2026-05-12 · Deciders: Lead architect, Head of Claims, Compliance
Context. We route 4,000 claims/day into four queues. Adjusters agree with one another about 85% of the time. Regulation requires that any denial be attributable to a named human. Leadership initially asked for an autonomous agent that "handles claims end to end."
Decision. Build a deterministic workflow whose classification step is a single augmented claude-opus-4-8 call returning a structured verdict with a required cited clause. Auto-approve executes automatically; denial and fraud routes require a named adjuster to confirm before any action.
Alternatives considered.
- Autonomous agent that plans and executes each claim (rejected). The path is identical for every claim (parse, retrieve policy, classify, emit rationale), so model-directed control flow buys non-determinism and an audit trail nobody can reconstruct, for a sequence we already know. It also cannot satisfy the named-human requirement without a gate, at which point it is a workflow wearing a costume.
- Bare augmented LLM call with no workflow around it (rejected). Cannot enforce the denial gate, cannot retrieve policy terms deterministically, and gives us no seam to attach the feedback loop to.
- Rules engine, no model (rejected). The unstructured PDF packets defeat hand-written rules; this is exactly the extraction problem a model is good at. Kept as the fallback path for parser failures.
Consequences — positive. Deterministic, auditable control flow; the named-human requirement is satisfied by construction; adjuster overrides become free labelled data for the eval set (Lab 6); cost is bounded and predictable per claim.
Consequences — negative. Adding a fifth queue or a new branch requires a code change and a redeploy, not a prompt tweak; we have chosen rigidity on purpose. Two model behaviours (classify, cite) are coupled in one call, so a regression in one can mask the other until the eval catches it. The human gate caps throughput on the denial path at adjuster capacity; if denial volume spikes, humans are the bottleneck, by design.
Revisit trigger. Reopen this decision if the adjuster override rate on auto-approved claims falls below 3% for two consecutive quarters (the gate may then be adding latency without adding safety), or if a regulatory change removes the named-human requirement.
4. Communicating Trade-offs to Non-Technical Stakeholders
A stakeholder does not hear your engineering statement. They hear a feeling (usually "it's slow" or "it's broken") and then they act on the feeling. Your job is to replace the feeling with a number, a comparison, and a lever they can pull. Say what is true, say what it means relative to something they understand, and say what it would cost to change it.
| What the engineer says | What the stakeholder hears | What to say instead |
|---|---|---|
| "p95 latency is 4 seconds" | "It's slow" | "19 of 20 users get an answer in under 4 seconds; 1 in 20 waits longer. Halving that tail costs roughly $X/month in a bigger model tier; here is the lever." |
| "The model hallucinates" | "It's broken and I can't trust it" | "It produces an unsupported claim in about 2% of answers. Every one is caught by the citation check before a human sees it, so the number that reaches a user is far lower." |
| "Accuracy is 88%" | "It's wrong 1 in 8 times" | "Your own adjusters agree with each other 85% of the time. We match a human panel; here is the same measurement run on them." |
| "It's non-deterministic" | "It's unreliable / buggy" | "The same input can produce slightly different wording, like two experts writing the same summary. The decision is stable; the prose varies. We measure the decision, not the wording." |
| "We need more eval data" | "It isn't finished / you're stalling" | "To promise a number, we have to measure it on 300 real cases. We have 180. Two more weeks of live traffic gets us there, or we can commit to a wider range now." |
The hardest conversation: non-determinism
The single hardest expectation to align is that this software is not deterministic. People who have used computers their whole lives expect that the same input yields the same output, forever. When it does not, they do not think "stochastic system"; they think "bug," and they lose trust in one motion. You cannot argue them out of this; you have to reframe it. The frame that works: the system is as consistent as a good human expert, and you should measure it the way you would measure one, on the decision it reaches, across many cases, not on whether it says the same sentence twice. Show them two human experts writing the same discharge summary in different words but reaching the same clinical conclusion. That is the mental model to install, and it is worth an exam item.
5. SLAs, SLOs, SLIs, and Error Budgets
These four words are used interchangeably by people who should know better, and the exam tests whether you don't. Learn the distinction crisply, because the objective names it (“including SLAs”) and because promising the wrong one is how architects get sued.
| Term | What it is | Who it faces |
|---|---|---|
| SLI — indicator | A measured number: the actual thing, right now (e.g. observed p95 latency) | Internal (it is just the measurement) |
| SLO — objective | The target you hold yourself to for an SLI (e.g. p95 ≤ 4s, 99% of the time) | Internal (your team's promise to itself) |
| SLA — agreement | A contractual promise to a customer, with a penalty if breached (always looser than the SLO) | External (it has legal and financial teeth) |
| Error budget | The allowed shortfall from 100% (e.g. a 99.5% SLO gives 0.5% to spend) | Internal (the currency of shipping vs. stabilizing) |
Realistic SLIs for a Claude system, and what each one actually protects:
| SLI | Example objective (SLO) | What it protects |
|---|---|---|
| Availability | 99.5% of requests succeed (non-5xx) monthly | The service exists when called |
| p95 end-to-end latency | ≤ 4s at p95 | Users do not abandon or route around it |
| Time-to-first-token | ≤ 1.2s at p95 (streaming) | Perceived responsiveness |
| Groundedness / citation validity | ≥ 98% of answers carry a resolving citation | Trust and auditability |
| Guardrail-violation rate | ≤ 0.1% of outputs trip a safety guardrail | Blast radius (Lab 8) |
| Cost per resolved task | ≤ $0.12 per routed claim | The business case survives renewal |
The trap the exam sets: do not promise an accuracy SLA against ground truth that humans only agree on 85% of the time (callback to Lab 1). Accuracy is an SLO, measured on a frozen eval set you control. It is never an SLA on live traffic, because you cannot control the inputs, you cannot control the disputed labels, and you would be signing a contract to overfit one reviewer's opinion. Contract on things you control (availability, latency, the existence of a citation) and hold accuracy as an internal objective on frozen data.
The error budget is the operational half of the SLO. A 99.5% availability SLO means 0.5% of requests may fail this month before you have broken your promise. That 0.5% is a budget you spend: on shipping risky changes, on load tests in production, on an aggressive new model rollout. When the budget is healthy, you ship; when it is exhausted, you freeze features and stabilize. Who decides how it is spent should be named in advance: typically the service owner, not whoever wants to ship on Friday.
An SLO is best written as a machine-readable document so monitoring can enforce it. This is the one code block in the lab, and note that it is a definition, not a program:
slo:
name: claims-triage-latency
sli: end_to_end_latency_seconds # measured indicator
objective:
percentile: p95
threshold_seconds: 4.0
target: 0.99 # met on 99% of eligible minutes
window: 30d_rolling
error_budget:
allowed_failure_ratio: 0.01 # 1% of the window may breach
on_exhaustion: freeze_feature_launches
owner: service-owner # decides how the budget is spent
note: >
Accuracy is tracked as a SEPARATE objective on a frozen eval set.
It is deliberately NOT an SLA on live traffic (disputed ground truth).6. Feedback Loops and Expectation Alignment
Expectations are set on day one and are almost impossible to move afterward, so the whole game is setting them honestly the first time. The cadence that keeps a project alive:
| Ritual | Cadence | What it prevents |
|---|---|---|
| Demo on real data | As early as possible, once | The six-month walk-back from an expectation a curated demo created |
| Shared SLI dashboard | Always on, one source of truth | Duelling anecdotes; "it felt slow yesterday" versus the p95 |
| Failure-taxonomy review | Standing, weekly or biweekly | Recurring failure modes being rediscovered as surprises |
| Written change log | Every change that touches behaviour | "It used to work" with no record of what moved |
7. Documentation and Implementation Guidance: The Handoff
You will not run this system forever. The handoff package is what determines whether it survives the day you stop paying attention to it. A complete package contains:
| Artifact | What it is | Answers the question |
|---|---|---|
| Architecture diagram | The five planes from Lab 1: ingress, context, reasoning, egress, feedback | What is this, at a glance? |
| ADR log | The decision records, including rejected alternatives and revisit triggers | Why is it shaped this way, and when do we reopen it? |
| Eval suite + how to run it | The frozen dataset, the grader, and the command (Lab 6) | How do we know a change is safe to ship? |
| Runbook | Symptom → first check → escalation, per failure mode | It's 3am and it's broken. What do I do? |
| Cost model + assumptions | Tokens, cache hit rate, model tier, and what each assumes | What will this cost, and what breaks the estimate? |
| Risk register | The identified risks, owners, and mitigations (Lab 8) | What are we knowingly exposed to? |
| On-call rotation | Who is paged, when, and who they escalate to | Who owns it at 3am? |
The runbook is the artifact that decides whether the system survives your departure. A diagram tells the next person what the system is; the runbook tells them what to do when it is on fire and you are unreachable. Model it on Lab 7's differential table: each row is a symptom, the first check that discriminates between causes, and the escalation path when the first check comes back clean.
| Symptom | First check | Escalation |
|---|---|---|
| Latency p95 breached SLO | Is cache hit rate down? Compare token counts vs. baseline | If cache healthy → check upstream model tier / provider status; page service owner |
| Citation-validity SLI dropped | Did the source corpus change or a retrieval index rebuild fail? | If corpus stable → suspect prompt/model regression; freeze deploys, page eval owner |
| Denial queue backing up | Is adjuster staffing at expected level? (The gate is human-capped by design) | If staffed → check classifier over-routing to denial; page claims lead |
8. Lifecycle Phases
The objective names five phases (discovery, design, handoff, monitoring, iteration), and the trap is treating the last two as things that happen after the “real” work. Monitoring and iteration are phases, not afterthoughts. They are what the feedback plane from Lab 1 exists to enable; a system with no monitoring phase is a system you have stopped learning from, and a system with no iteration phase is one whose first version is its last.
| Phase | Artifact produced | Stakeholder engaged | Exit criterion | Failure mode of skipping it |
|---|---|---|---|---|
| Discovery | Decision, ground truth, constraints, blast radius | Business owner, end users, compliance | One named decision with measurable ground truth | You build the wrong thing, beautifully |
| Design | Architecture (five planes), ADR log, SLOs | Lead architect, security, ops | Every requirement traces to an acceptance criterion | Aspirations masquerade as commitments |
| Handoff | The handoff package (§7), runbook, on-call | Operating team, on-call rotation | A stranger can run the eval and follow the runbook | The system dies when you leave |
| Monitoring | Live SLI dashboard, alerting, trace capture | Service owner, on-call | SLIs are observed and alerting fires correctly | Drift is discovered by a customer, not a graph |
| Iteration | Updated eval set, changelog, new ADRs | Product, users, architect | Failures flow back into the eval set and get fixed | The first version is frozen as the final one |
9. Design Exercise
Self-driven lab Lab9_Self_Driven_Lab.ipynbObjective: run the communication and lifecycle work for a system whose architecture you already sketched. You are handed the hospital discharge-summary system from Lab 1's exercise: 900 discharges/day across 11 facilities, PHI under HIPAA, physicians abandon any tool slower than 20 seconds, and every summary is signed by the attending physician. About 60 minutes. Compliance content here is architectural guidance, not legal advice; a real deployment needs a BAA and counsel.
- Run the discovery interview. Write the six questions you would ask the Chief Medical Officer, drawn from the §1 framework. At least one must be a doomed-project question: "How do you know today when a discharge summary was wrong?" or "Who loses if this works?"
- Write one complete ADR for the model-tier decision (which of
claude-opus-4-8,claude-sonnet-5,claude-haiku-4-5, and why). Include at least two rejected alternatives with reasons, one negative consequence you accept, and a revisit trigger. Remember the 20-second latency bound is an architectural constraint on this choice. - Define three SLIs and one SLO, with an error budget for the SLO. At least one SLI must be quality-facing (groundedness). State plainly which number you would put in an SLA and which you would refuse to, and why accuracy against physician edits is the one you refuse.
- Draft the first three rows of the runbook: symptom → first check → escalation. Cover at least a latency breach and a quality-drop symptom.
- Write the demo plan. One sentence on what data you demo on and why it is not curated.
- Write the sentence you would say to a skeptical physician who asks, "What happens when it's wrong?" It must contain a number, a comparison to current practice, and the gate that protects the patient. This single sentence is the entire domain in miniature.
Step 6 is the one that gets you the pilot. Step 1 is the one that tells you whether the pilot should exist.
Check Yourself
Exam-style items. Commit to an answer before expanding.
During discovery for a document-classification system, the business sponsor lists eight features they want and cannot say how anyone currently determines whether a classification was correct. What is the most important thing for the architect to do next?
Correct: B. Discovery collects decisions and constraints, not feature lists, and the missing ingredient here is fatal: with no ground truth you cannot build an eval set, cannot measure quality, and cannot defend the system at renewal. "How do you know today when a human got this wrong?" is the question that surfaces a doomed project while it is still cheap to stop.
A mistakes a wish list for requirements and races past the one unanswered question that determines whether the project is viable at all. C and D are premature architecture: choosing a model or a pattern before you know how success is measured is solving for the wrong variable. The discriminating fact is the sponsor's inability to define correctness.
A vendor's proposed SLA promises "95% accuracy" measured on live production traffic for a triage system whose human experts agree with each other only 85% of the time. Why is this SLA a mistake, and what should replace it?
Correct: B. An SLA is a contractual promise with penalties, so it must cover things you control: availability, latency, the existence of a citation. Accuracy on live traffic fails both tests: you do not control the inputs, and the ground-truth labels are disputed by the very humans who set them. Accuracy is a legitimate objective, but as an SLO measured on a frozen set you control, not an SLA on live data.
A and C both take the 95% number at face value and argue about its size; the number is not the problem, the contract type is. Promising 95% (or 99%) against labels humans agree on 85% of the time is contracting to overfit one reviewer. D invents a nonsensical metric: "p95 accuracy" is not a thing; percentiles describe latency distributions, not correctness.
An architect writes a decision record that states the chosen architecture and lists four reasons it is excellent, with no rejected alternatives and no downsides. Reviewers push back. What is the substantive objection, and what two additions would fix it? (Choose two.)
Correct: A and B. A record with no rejected alternative and no negative consequence is marketing, not an ADR: it documents a conclusion, not a decision. The two sections that make it a decision record are precisely the alternatives-and-why-rejected and the consequences-negative sections, because they are what let a future reader reopen the question with the same information the author had.
C deepens the exact flaw the reviewers named, piling on advocacy where they asked for honesty. D changes the format, not the substance. E throws away the decision itself, which is the one thing an ADR must actually contain; neutrality is not the goal, a defensible and revisitable record is.
Three weeks before handoff, the operating team has the architecture diagram, the ADR log, and the eval suite, but no runbook and no on-call rotation. The lead architect is leaving. Which gap most endangers the system's survival, and why?
Correct: B. Handoff is a lifecycle phase with a specific exit criterion: a stranger can run the eval and follow the runbook. The runbook is the artifact that operates the system under failure (symptom, first check, escalation), and the on-call rotation names who acts. Without them, the first incident after the architect leaves has no owner and no procedure, which is how technically sound systems die.
A is wrong because the eval suite is present and its job (deciding whether a change is safe) is not the job of surviving an incident. C polishes an artifact that already explains the system while ignoring the one that operates it. D is the exact complacency the phase exists to prevent: a diagram tells you what the system is, not what to do when it breaks.
Key Takeaways
- Domain 6 is nine items: 14%, more than Models & Prompting, and it is under-studied precisely because it has no API surface. It is a pass-decider.
- Discovery collects decisions and constraints, not feature lists. The two questions that save doomed projects: "How do you know today when a human got this wrong?" and "Who loses if this works?"
- Every requirement must trace to a verifiable acceptance criterion. One that cannot is an aspiration: label it, don't ship it as a commitment.
- An ADR with no rejected alternative and no negative consequence is marketing. Every ADR needs a revisit trigger: the condition that reopens it.
- Communicate every trade-off with a number, a comparison, and a lever. The hardest alignment problem is non-determinism: reframe it as the consistency of a good human expert.
- Know the four cold: SLI (measured), SLO (internal target), SLA (external contract), error budget (the allowed shortfall). Accuracy is an SLO on a frozen set, never an SLA on live traffic.
- Never demo on curated data: it sets an expectation you will spend two quarters walking back.
- The handoff package explains the system; the runbook operates it. The runbook decides whether the system survives your departure. Monitoring and iteration are phases, not afterthoughts.