Lab 2 · Domain 2 · 21%

Output Evaluation & Validation

This is the largest domain on the exam (more than one question in five), and it is the one that separates people who use Claude from people who can be trusted with what Claude produces. The core skill is uncomfortable: an answer can be fluent, confident, well-formatted, and completely wrong, and nothing on the surface will tell you which. This lab teaches you to read an output the way an editor reads a draft (assuming a mistake is in there somewhere until you have checked) and to decide what to check, how, and when to stop and hand it to a human.

Exam objectives covered

Evaluate Claude-generated outputs for accuracy and completeness
Identify hallucinations, inconsistencies, and biases in responses
Apply fact-checking and validation techniques
Determine when human review or additional verification is required
Edit, adapt, refine, and compare outputs for the intended audience
Organize and curate information and select appropriate output formats (artifacts, inline, structured data)

What you need: a Claude account (claude.ai) and about an hour. This lab is longer than the others on purpose. No installation, no API key, no code.

1. Fluency Is Not Accuracy

Claude writes well. That is precisely the danger. We are trained by a lifetime of reading to treat fluent, well-organized, confident prose as a proxy for a competent, well-informed author, because for humans, most of the time, it is. A person who writes a crisp paragraph citing a specific subsection usually did the reading. Claude breaks that correlation. It can produce the same crisp paragraph whether or not the subsection exists.

So the first habit to build is to separate how an answer sounds from whether it is true. Tone, formatting, and specificity are features of the writing. Accuracy is a property of the world. The exam tests this distinction relentlessly, because it is the root of almost every validation mistake: people approve outputs because they read as authoritative.

Read for claims, not for quality. On a first pass, ignore whether the writing is good. Underline every sentence that asserts a fact about the world: a number, a name, a date, a citation, a cause-and-effect. Those are the only things that can be wrong in a way that matters, and they are what your eye slides past when the prose is smooth.

2. Hallucinations: The Highest-Risk Artifacts

A hallucination is a confident statement that is not grounded in fact: the model producing something plausible in place of something true. Hallucinations are not random noise scattered evenly through an answer. They cluster in predictable places, and the most dangerous ones look the most trustworthy.

The rule to memorize: the more specific and verifiable-looking a detail is, the higher the risk it deserves. Specific-looking details are exactly the artifacts a language model can fabricate convincingly, and they are the ones a reader is least likely to question because specificity reads as evidence of diligence. The highest-risk artifacts:

Citations and references — subsection numbers, statute names, case citations, page numbers, DOIs. A fabricated citation is the classic failure, and it is the one the exam's own sample question is built on.
Statistics and quantities — percentages, dollar figures, dates, counts, growth rates. A number carries an air of precision it may not have earned.
Named entities — people, companies, product names, titles, laws. Plausible-sounding names are easy to generate and hard to spot as invented.
Quotations — anything in quotation marks attributed to a source. Direct quotes carry enormous authority and are frequently paraphrased or fabricated outright.
URLs and links — a well-formed web address that leads nowhere, or somewhere unrelated.

A worked example: the fabricated citation

This mirrors the exam guide's own Sample 1. Suppose an associate asks Claude to summarize a new data-protection regulation, and it returns:

What Claude returned — do not trust it yet

Under Section 14(3)(b) of the Act, data controllers must notify affected individuals within 72 hours of discovering a breach that poses a material risk to their rights. Failure to do so carries an administrative penalty of up to 4% of annual turnover.

Read it as an editor. Every sentence sounds authoritative, and the "72 hours" and "4%" figures feel familiar because they echo real regulations you may have heard of. That familiarity is the trap. The checkable claims here are: (1) the existence and content of Section 14(3)(b) specifically; (2) the 72-hour window; (3) the "material risk" threshold; (4) the 4%-of-turnover penalty. Not one of them is validated by the fact that Claude wrote it confidently. The subsection number is the single highest-risk artifact on the screen, because it is the most specific and the easiest to fabricate.

The correct action is not to reword it, not to ask Claude how sure it is, and certainly not to forward it to the compliance team because it "looks right." It is to open the official text of the regulation and confirm that Section 14(3)(b) exists and says what the summary claims, before it goes anywhere near a decision. That single verification step is the whole point of Domain 2.

Try it in Claude

Deliberately induce a plausible-but-unverifiable detail, then verify it.

Open a new chat and ask something that invites specificity from a domain you can independently check: "Summarize the key notification deadlines in [a real regulation, standard, or policy relevant to your work], and cite the specific section for each."
When the answer comes back, resist the pull of the formatting. List every citation, number, and named threshold as a separate checkable claim.
Now go to the authoritative source: the official published text, not another AI answer and not a random blog. Confirm each claim, one at a time. Note which held up and which did not.
Ask Claude directly: "Which of the specific figures and section numbers in your last answer are you certain are correct, and which should I verify?" Notice that even a candid reply here is still not verification; it is another generated answer. The regulation text is the authority; Claude is not.

3. Self-Reported Confidence Is Not an Accuracy Signal

This is the single most heavily tested idea in the domain, so it gets its own section. Asking Claude "how confident are you?" or "are you sure?" feels like validation. It is not. The confidence rating is generated by the same process that generated the answer, from the same information. It cannot reach outside the conversation to check the world, so it cannot tell you whether the answer matches the world.

A high confidence score can accompany a fabricated citation. A hedge can accompany a correct one. The rating tells you something about the shape of the response, not about its truth. Worse, prompting for confidence often produces a reassuringly high number precisely when you wanted reassurance, which is the opposite of a control.

Weak — feels like validation, validates nothing

Are you confident this is correct? Rate your confidence from 1 to 10.

Strong — surfaces what to check, then you check it

List every factual claim in your answer that a reader would need to verify against an external source (every citation, figure, date, and named entity) as a plain checklist. Do not rate your confidence; just tell me what would need checking.

The strong version is useful not because Claude's self-assessment is trustworthy, but because enumerating the checkable claims is genuinely helpful triage. You still do the checking. The distinction the exam wants: Claude can help you find what to verify; it cannot be the thing that verifies it.

"Ask it to double-check" is not a validation method. Telling Claude to review its own answer in the same conversation can catch an internal contradiction, but it cannot catch a fact that is wrong in a way the conversation gives it no way to know. External claims need an external source. If the answer to "how do I know this is right?" is "because I asked Claude again," you have not validated anything.

4. Inconsistencies and Bias

Accuracy is not only about individual facts. Two other failure classes matter, and both are named in the exam objectives.

Inconsistencies

An internal contradiction is when an output disagrees with itself: a summary that says a project finished under budget in one paragraph and over budget in another, or a table whose totals do not add up to its own rows. These are the one failure class you can often catch without leaving the document, because the evidence is right there. Reconciling numbers against each other, and conclusions against their own stated reasoning, is a fast and high-value first pass.

Bias

Bias in an output can enter from several directions, and the associate is responsible for all of them:

Framing bias — the output presents one side of a genuinely contested question as settled, or adopts a loaded framing from the prompt without flagging it. If you ask "why is our new strategy working?" you will get reasons it is working, not an honest assessment of whether it is.
Representational bias — examples, names, or assumptions that skew toward one group, region, or perspective and quietly treat it as the default.
Inherited bias from your source material — this is the one associates forget. If you hand Claude a skewed dataset, a one-sided memo, or a leading set of survey questions, a faithful summary will faithfully reproduce the skew. The output can be an accurate summary of a biased input. Validation includes asking whether the source was representative, not just whether Claude summarized it correctly.

The defensive move for contested topics is to prompt against the grain: ask for the strongest case against the conclusion, ask what a skeptic would say, ask what the data does not support. An output that can only argue one side has not been evaluated; it has been cheered on.

5. A Practical Validation Workflow

Validation is not "read it again and see if it feels right." It is a repeatable sequence. Run it in order; the early steps make the later ones cheap.

Identify the checkable claims. Separate assertions of fact from framing and phrasing. You cannot fact-check "this is a strong strategy," but you can fact-check "revenue grew 12% in Q2."
Rank by consequence of being wrong. Not every claim earns the same scrutiny. A throwaway aside costs nothing if it is off. A penalty figure headed to the compliance team costs a great deal. Spend your verification effort where an error would hurt most.
Verify against authoritative sources. For the high-consequence claims, go to the primary source: the official text, the original dataset, the system of record. Another AI answer is not a source. A confident restatement is not a source.
Verify quantities independently. Re-derive numbers rather than trusting them. If a total is claimed, add the parts yourself. If a percentage is claimed, check it against the base figure. Numbers are the easiest thing to get wrong and the easiest thing to check.
Check completeness: what was omitted. Accuracy is not only about whether what is present is true; it is about whether what matters is present. A summary that is true but leaves out the one clause that changes the decision is a failure of completeness. Ask: what would a domain expert expect to see here that isn't here?
Confirm scope. Did the output answer the question you asked, about the population you meant, over the period you meant? An analysis of "our customers" that silently excluded a whole region is accurate about the wrong thing.

Claim type in the output	Risk level	Verification action
Citation, statute, subsection, case reference	Highest	Confirm it exists and says what's claimed, in the official primary text
Direct quotation attributed to a source	Highest	Locate the exact wording in the original; treat paraphrase-as-quote as an error
Statistic, dollar figure, date, count	High	Re-derive from the source data; check totals and percentages independently
Named entity (person, law, product, org)	High	Confirm the entity exists and the attributes attached to it are correct
URL or link	Medium–High	Open it; confirm it resolves and points to the claimed content
Causal or comparative claim ("X caused Y", "the largest")	Medium	Check the reasoning against the evidence actually presented; watch for framing bias
General phrasing, tone, structure	Low	Editorial judgment only — not a fact-check target

Consequence, not volume, sets the effort. You will not verify every word of every answer; that would make Claude useless. The skill is proportionality: verify the claims where being wrong is expensive, and let low-stakes phrasing pass. A one-line internal brainstorm and a regulatory summary bound for external counsel deserve very different amounts of your attention, even if they are the same length.

6. When Human Review Is Mandatory

Some outputs must be reviewed by a qualified human before they are used, no matter how good they look. The trigger is not how confident the model seems and not how polished the writing is. The trigger is the cost of being wrong. Confidence is a property of the answer; consequence is a property of the situation, and it is the situation that decides.

Human review is mandatory when an output feeds into any of these:

High-consequence decisions — anything where an error causes real harm, cost, or reputational damage that is hard to undo.
Regulated or compliance-bound work — anything where a mistake creates legal exposure or breaches a rule you are obligated to follow.
Legal, medical, or financial advice or filings — domains where a qualified professional's sign-off is the standard of care, and Claude is a drafting aid, not the authority.
Anything published externally — customer-facing, public, or on-the-record content that carries your organization's name.
Irreversible actions — a decision that cannot be walked back once taken.

Notice what is not on the list: "Claude said it was very confident." A high self-rating never removes a review requirement, and a low one never adds work that the stakes don't justify. Frame every case as: if this is wrong and we act on it, what does it cost? That answer, not the model's tone, tells you whether a human must sign off.

7. Editing, Adapting, and Comparing Outputs

Once an output is accurate, the associate's job shifts to making it fit. The same validated content usually needs to be re-shaped for whoever will read it, and you will often need to choose between two candidate outputs.

Adapting for the audience

One set of facts, three readers, three deliverables. An executive wants the decision and the one number that drives it, in three sentences. A specialist wants the methodology, the caveats, and the edge cases. A customer wants plain language, no jargon, and reassurance about what happens next. Same truth, different altitude. Getting the altitude wrong is its own kind of failure, even when every fact is correct.

Try it in Claude

Re-shape one validated output for three audiences.

Take an answer you have already fact-checked (say, a summary of a project's status or a policy change).
Ask Claude to produce three versions from the same facts: "Rewrite this three ways from the same underlying facts: (a) three sentences for an executive who needs only the decision and the key number; (b) a detailed version for a specialist that keeps the methodology and caveats; (c) plain-language for a customer, no jargon, ending with what happens next."
Check each version against the original for a subtle failure mode: did simplifying introduce an inaccuracy? Compression is where true statements quietly become false ones: a hedge dropped, a "some" turned into "all". Adapting for audience is still a validation task.
Pick the executive version and ask yourself whether it omitted anything a decision-maker actually needs. Completeness and brevity are in tension; you are the one who resolves it.

Comparing two outputs

When you regenerate an answer or try a second prompt, you end up choosing between candidates. Do not choose by vibes: the more fluent one is not automatically the better one, and preferring it is exactly the fluency trap from Section 1. Compare against explicit criteria you name first: which is more accurate, which is more complete, which is better scoped to the audience, which makes fewer unverifiable claims. You can even ask Claude to compare two drafts against those stated criteria, but the criteria come from you, and the final call is a judgment, not a preference.

8. Choosing an Output Format

The exam treats format selection as a real skill, not an afterthought; it is an explicit objective. The same information can be delivered as an artifact, as an inline chat reply, or as structured data like a table or list, and choosing wrong wastes effort or buries the answer.

Artifact — a substantial, self-contained deliverable you will keep, reuse, iterate on, or hand off: a document, a report, a structured piece of content, a polished draft. Artifacts live beside the chat so you can edit them across turns without the content scrolling away. Reach for one when the output is the work product, not just a step toward it.
Inline — the answer sits in the chat itself. Right for short answers, discussion, exploration, quick questions, and anything conversational or throwaway. Promoting a two-line answer into an artifact is overhead; keep it inline.
Structured data — a table, a list, or another organized layout. When the content is comparative or has repeating fields (options against criteria, items with attributes, steps with owners), a table beats prose because it makes gaps and mismatches visible at a glance. Prose can hide an omission; a table with an empty cell cannot.

If the output is…	Best format	Why
A report, brief, or draft you'll revise and reuse	Artifact	Self-contained, editable across turns, survives the scroll
A quick answer, a clarification, or brainstorming	Inline	Conversational; an artifact would be needless overhead
Several options judged on the same criteria	Structured data (table)	Makes comparisons and missing values obvious
A sequence of steps, each with an owner or status	Structured data (list/table)	Repeating fields read faster than a paragraph
A polished document plus a short note about it	Artifact + inline	The deliverable is the artifact; the discussion stays in chat

Format is also a validation aid. Choosing structured data does more than look tidy: a table forces every row to declare the same fields, so a missing driver, an unfilled confidence rating, or an unaccounted-for option becomes visible immediately. Sometimes the fastest way to catch an incomplete answer is to demand it as a table and look for the blank cell.

When to escalate. Everything in this lab is validation done by hand, one output at a time, in your own Claude account, and that is the whole of the Associate's job here. The moment the requirement becomes systematic (automatically checking outputs across thousands of records, building an evaluation pipeline that scores responses at scale, or wiring in programmatic guardrails that flag hallucinations before a human sees them), you have crossed into Claude Architect or Developer territory. Recognizing that line, and handing the work over rather than trying to hand-validate at a volume no human can, is itself the responsible move.

9. Lab Exercise: Run a Full Validation Pass

Objective: take one real, consequential output from your own work and put it through the entire Domain 2 workflow, from spotting the checkable claims to choosing the format you'd actually ship it in.

Generate a real output. Ask Claude for something you would genuinely use and that contains facts: a summary of a document, an analysis with numbers, a policy explanation with references. Save it.
Mark the claims. Go through and underline every citation, figure, date, quote, named entity, and causal claim. These are your fact-check targets; everything else is phrasing.
Rank by consequence. For each marked claim, write one word: how badly would it hurt if this were wrong? Sort so the expensive errors are first.
Verify the top claims against authoritative sources. Take the two or three highest-consequence claims to the primary source and confirm them. Re-derive any number yourself. Record what held and what didn't.
Test completeness and scope. Ask: what did this leave out that a domain expert would expect? Did it answer about the right population and period?
Probe for bias. Ask Claude for the strongest case against its own conclusion. If it can't produce one, or your source material was one-sided, note it.
Make the review call. Decide explicitly: does this output require human sign-off before use? Write down the reason, and confirm the reason is about consequence, not about how confident the answer sounded.
Adapt and choose a format. Re-shape the validated content for its actual audience, and decide whether it ships as an artifact, an inline reply, or a table. Justify the choice in one sentence.

Step 7 is the hinge of the whole domain. If you can articulate why something needs human review in terms of what it would cost to be wrong (and not in terms of the model's confidence), you have internalized the idea the exam tests more than any other.

Check Yourself

Exam-style items. This is the heaviest domain, so there are four. Commit to an answer before expanding.

An associate asks Claude to summarize a new regulation, and Claude produces a confident summary citing a specific subsection number. Before sending the summary to the compliance team, what is the most appropriate action?

A. Send it as-is, since Claude expressed high confidence.
B. Verify the cited subsection against the official regulation text before sharing.
C. Ask Claude to rate its own confidence and send it if the rating is high.
D. Reword the summary to sound more formal, then send it.

Correct: B. A specific-looking citation (a subsection number) is exactly the kind of detail a language model can fabricate convincingly, and it is the highest-risk artifact in the summary. Because the audience is compliance and the cost of a wrong citation is real, the required diligence step is to validate the claim against the authoritative primary source: the official regulation text itself.

A and C both mistake self-reported confidence for an accuracy signal; it is not, because the rating is generated from the same information as the answer and cannot reach outside the conversation to check the world. D changes how the summary sounds, which does nothing about whether it is correct; a more formal tone on a fabricated citation is just a better-dressed error.

An associate has fact-checked a market analysis Claude produced and wants a final assurance it is accurate before presenting to leadership. Which step actually validates the accuracy of the figures?

A. Ask Claude, "Are you confident these numbers are correct?" and proceed if it says yes.
B. Ask Claude to reformat the analysis into a polished artifact so it looks authoritative.
C. Re-derive the key figures from the original source data independently, checking that totals and percentages hold.
D. Regenerate the analysis a second time; if the numbers match the first version, treat them as confirmed.

Correct: C. Validating a quantity means checking it against the world, not against the model. Re-deriving the figures from the primary source data (adding the parts, confirming the percentages against their base) is the only option here that reaches an authority outside the conversation.

A is the confidence trap in its purest form: a self-rating is not evidence of correctness. B improves presentation and, if anything, makes an unvalidated output more dangerous by lending it authority. D is seductive but wrong: two runs from the same model on the same data can reproduce the same error, so agreement between them confirms consistency, not accuracy.

An associate used Claude to draft a customer-facing FAQ that will be published on the company's public help site. Every fact has been checked and is correct. What determines whether a human must review it before it goes live?

A. Nothing further is needed: the facts are verified, so it can publish automatically.
B. Because it is published externally under the company's name, it falls into a category that requires human review regardless of how accurate or confident the draft is.
C. Ask Claude to confirm the tone is appropriate for customers, and publish if it agrees.
D. Review it only if Claude flags low confidence in any section.

Correct: B. The review requirement is set by the consequence of being wrong, not by the model's confidence or even by the accuracy of the individual facts. Externally published, on-the-record content that carries the organization's name is a named high-consequence category: framing, completeness, and reputational risk all need a human's judgment even when every checkable fact holds.

A treats fact-checking as the whole of review, but accuracy of facts doesn't cover tone, framing, or what was omitted for a public audience. C outsources a judgment call back to the model whose output is under review. D makes the model's self-assessed confidence the trigger, the exact inversion of the rule that consequence, not confidence, decides.

An associate needs to present four vendor options to a procurement committee, each compared on price, support level, contract length, and compliance certification. Which output format best fits, and why?

A. A flowing prose summary inline in the chat, describing each vendor in turn.
B. A structured table with the four vendors as rows and the four criteria as columns.
C. Whatever format Claude returns by default, since format is cosmetic.
D. Four separate chats, one per vendor, to keep them from being confused.

Correct: B. Content that compares several options against the same repeating set of criteria is exactly what structured data is for. A table makes mismatches and gaps visible at a glance (a missing compliance certification shows up as an empty cell), and it lets the committee scan across criteria in a way prose cannot support.

A buries the comparison in paragraphs where an omission for one vendor is easy to miss. C is wrong because format is a real objective, not a cosmetic afterthought; the right shape aids both the reader and your own completeness check. D fragments the very comparison the committee needs to make and prevents any side-by-side view.

Key Takeaways

Fluency is not accuracy. Confident, well-formatted, specific prose can be entirely wrong. Read for claims, not for polish.
The most specific-looking details are the highest-risk. Citations, statistics, quotes, named entities, and URLs are where hallucinations hide and where they do the most damage.
Self-reported confidence is not a validation method. Asking Claude how sure it is (or asking it to double-check itself) cannot verify a claim about the world. External claims need external sources.
Validate as a sequence: identify checkable claims, rank by consequence, verify against authoritative sources, re-derive quantities, check completeness, confirm scope.
Watch for inconsistency and bias, including bias inherited from the source material you supplied. An accurate summary of a biased input is still biased.
Consequence, not confidence, decides human review. High-stakes, regulated, legal/medical/financial, externally published, or irreversible: a human signs off regardless of how the answer sounds.
Adapt validated content to the audience without letting compression introduce error, and compare candidates on explicit criteria, not on which reads better.
Choose the format deliberately: artifacts for reusable deliverables, inline for short and conversational, structured data when comparison or repeating fields make gaps visible.