RAG Pipeline Design

1. When NOT to Build RAG

The most expensive RAG mistake is building one you did not need. Every retrieval pipeline you add is an index to keep fresh, an embedding space to keep consistent, a reranker to tune, and a new class of failure (stale or irrelevant chunks) that did not exist before. So the first architectural act is a subtraction test.

Read the last row carefully, because it is the definition, not an afterthought: RAG is for corpora too big to carry and queries too fuzzy to key. If either half is false (the corpus fits, or the query is an exact key), a simpler mechanism wins, and the exam will offer it to you as the correct answer dressed up to look boring.

2. The Pipeline, Stage by Stage

A RAG pipeline is not a black box labelled “vector search.” It is a chain of ten stages, and every one of them has a distinct failure mode. You cannot debug a RAG system (Lab 7) unless you can name which stage broke, so learn the chain as a chain.

The pipeline has a spine: everything upstream of stage 6 decides what can be found, everything downstream decides what gets used. When a RAG system misbehaves, the discipline is to localize the failure to one row of this table before touching the prompt.

3. Chunking Strategies

Chunking is where the most damage is done and the least attention is paid. The governing insight is blunt: chunk boundaries destroy meaning. A table split across two chunks answers no question; half the rows are in one vector and half in another, and neither is retrievable as “the table.” A defined term split from its definition is worse than useless. So the chunker must respect the document’s own structure, not an arbitrary token count.

The senior move on this table is the structural row: chunk on the document’s own structure when it has one. A contract has clauses. A manual has sections. Code has functions. Splitting those on a 512-token grid is an act of vandalism that no reranker can undo. The parent-document pattern is the refinement most candidates miss: you embed and retrieve small, precise units (a sentence, a clause) for accuracy, but feed the model the whole enclosing section so it has enough context to answer.

Structural chunking is also where metadata is born. As you split, you attach the source, the section path, the version, and the access-control tags, because a chunk without metadata cannot be filtered, cited, or secured downstream.

# Structural chunking: split on the document's own structure,
# and attach the metadata every later stage depends on.
# embed() is provider-agnostic pseudocode -- Anthropic ships no
# embedding endpoint; use your chosen third-party embedding model.

def chunk_contract(doc):
    chunks = []
    for clause in doc.iter_clauses():          # respect the real boundary
        # Keep whole tables intact inside their clause -- never split a table.
        text = clause.render(keep_tables_whole=True)
        chunks.append({
            "text": text,
            "vector": embed(text),             # third-party embedding model
            "metadata": {
                "source": doc.id,
                "section": clause.path,        # e.g. "7.2 Limitation of Liability"
                "version": doc.version,        # for staleness / re-index checks
                "tenant_id": doc.tenant_id,    # ACL -- enforced at retrieval
                "effective_date": doc.effective_date,
            },
        })
    return chunks

4. Retrieval Strategies Matched to Data Shape and Query Pattern

This is the exact wording of the objective, so treat the mapping as a table you can reproduce under pressure. The wrong instinct (the one the exam punishes) is to embed everything and run dense vector search for every query. Dense retrieval is one tool. It is the right tool for exactly one query shape.

Here is the strong default in code: run lexical and dense in parallel, fuse the two ranked lists with reciprocal rank fusion, apply the ACL filter before anything reaches the model, then rerank the survivors.

# Hybrid retrieval: lexical (BM25) + dense, fused with reciprocal
# rank fusion, ACL-filtered at retrieval time, then reranked.

def reciprocal_rank_fusion(ranked_lists, k=60):
    scores = {}
    for ranked in ranked_lists:
        for rank, chunk_id in enumerate(ranked):
            scores[chunk_id] = scores.get(chunk_id, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

def retrieve(query, tenant_id, top_k=8):
    lexical = bm25_search(query, limit=50)        # catches rare tokens / codes
    dense = vector_search(embed(query), limit=50) # catches paraphrase / meaning
    fused = reciprocal_rank_fusion([lexical, dense])

    # ACL PREFILTER -- security lives here, not in the prompt.
    allowed = [cid for cid in fused
               if index.metadata(cid)["tenant_id"] == tenant_id]

    # Rerank the ACL-safe survivors; keep only the true top-k.
    return rerank(query, allowed)[:top_k]

5. Indexing: Metadata Is a First-Class Citizen

An index is not a bag of vectors. It is vectors plus the metadata that makes them governable: source, timestamp, version, and access-control / tenant tags. Without that metadata you cannot filter by recency, you cannot cite a source, you cannot detect staleness, and, most importantly, you cannot enforce who is allowed to see what.

Read the retrieval code in section 4 again with this lens: the tenant filter runs before rerank, so a chunk belonging to another tenant is never a candidate for the context, let alone the answer. Multi-tenancy is enforced by the index and the query, not by the prompt.

6. Progressive Discovery vs. Monolithic Context

The objective asks you to evaluate these two strategies, which means knowing the tradeoff, not just the names. Monolithic context retrieves once and stuffs everything relevant into a single window. Progressive discovery (agentic retrieval) gives the model tools and lets it pull what it needs, iteratively, as it reasons.

The decision rule mirrors the workflow-vs-agent ladder from Lab 1: use monolithic context when you can enumerate the relevant slice in advance, and progressive discovery when you cannot. A single-shot Q&A over a retrieved section is monolithic. A research task that discovers which documents matter only after reading the first few is progressive. Note that Skills use progressive disclosure: the model loads a skill’s full instructions only when the task calls for it, which is the same principle applied to instructions rather than data.

7. Attribution and Groundedness

An answer you cannot attribute is an answer you cannot verify. Force the model to cite the chunks it used, and validate that the cited spans actually exist in the retrieved context. Attribution is not a nicety; it is the load-bearing link to the rest of the system: an un-attributable answer means no eval loop (Lab 6) and no defensible audit trail (Lab 8). If you cannot say which chunk produced a claim, you cannot label it right or wrong, and you cannot defend it to a regulator.

The mechanism is structured output: constrain generation to return the answer and the IDs of the chunks it cited, so a downstream check can confirm each cited chunk was actually in the retrieved set.

import anthropic

client = anthropic.Anthropic()

ANSWER_SCHEMA = {
    "type": "json_schema",
    "schema": {
        "type": "object",
        "properties": {
            "answer": {"type": "string"},
            # The model must name the chunks it used. Downstream code
            # rejects any id that was not in the retrieved set.
            "cited_chunk_ids": {
                "type": "array",
                "items": {"type": "string"},
            },
        },
        "required": ["answer", "cited_chunk_ids"],
        "additionalProperties": False,
    },
}

response = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=4096,
    thinking={"type": "adaptive"},
    output_config={"effort": "high", "format": ANSWER_SCHEMA},
    system=[{
        "type": "text",
        "text": "Answer only from the provided context. Every claim must "
                "cite the chunk id it came from. If the context does not "
                "contain the answer, say so.",
        "cache_control": {"type": "ephemeral"},
    }],
    messages=[{"role": "user", "content": build_context(retrieved_chunks, question)}],
)

# Groundedness check: every cited id must exist in what we retrieved.
result = parse(response)
retrieved_ids = {c["id"] for c in retrieved_chunks}
if not set(result["cited_chunk_ids"]).issubset(retrieved_ids):
    raise GroundednessError("model cited a chunk it was never given")

Three things in that call are exam-relevant. The schema is forced through output_config.format, not a deprecated top-level output_format. effort is nested inside output_config alongside format. And thinking is adaptive, never a fixed budget_tokens. The groundedness check that follows is what turns a citation from decoration into a control.

8. Failure Diagnosis

The exam guide publishes a sample item on exactly this, and it is worth stating verbatim. The guide’s own Sample 3: “A RAG system suddenly returns confident but incorrect answers after a document refresh, while latency and model version are unchanged. What is the most likely first place to investigate?”

The correct answer is the retrieval / indexing step: it is returning irrelevant or stale chunks (a broken re-index, or embeddings that no longer match the index). The reasoning is the transferable skill: latency is unchanged, so it is not a serving problem; the model version is unchanged, so it is not the model. The one thing that did change is the data: a document refresh. So the fault is in the data plane, not the reasoning plane.

The corollary is the trap that catches teams during “upgrades.” Swapping in a better embedding model and re-embedding only new documents leaves you with a mixed index: two coordinate systems in one store. Nothing errors. Retrieval just quietly degrades because half the vectors are speaking a different language than the query. Mixing embedding spaces silently returns garbage, and the fix is always the same: re-embed the whole index in one space.

9. Design Exercise

Scenario. A legal-operations team wants Claude to answer questions over a multi-tenant corpus of contracts. The documents are PDFs containing prose, tables (payment schedules, liability caps), and defined terms. Each contract belongs to exactly one tenant, and a user may only see their own tenant’s documents. The queries mix two shapes: exact lookups like “what does clause 7.2 say?” and semantic ones like “which of our contracts cap liability?” Design the retrieval pipeline.

1. Confirm RAG is right. The corpus is large and growing (too big to carry) and half the queries are semantic (too fuzzy to key). RAG qualifies, but note that the other half (“clause 7.2”) is an exact key, so the pipeline must serve both, not force everything through dense search.
2. Parse and chunk structurally. Chunk on clauses, not on a token grid. Keep each table (header and rows) inside one chunk; a split liability schedule answers nothing. Attach source, section (e.g. “7.2”), version, effective_date, and tenant_id to every chunk as you go.
3. Index with metadata as a first-class citizen. Store vectors plus that metadata. The tenant_id and section fields are not optional; they are what make the next two steps possible.
4. Retrieve hybrid, matched to query shape. Route or run both: lexical/BM25 catches “clause 7.2” and defined terms exactly; dense catches “cap liability.” Fuse with reciprocal rank fusion and rerank. Consider parent-document retrieval so a clause hit brings its whole section into context.
5. Enforce the ACL at retrieval, not in the prompt. Apply tenant_id as a prefilter before rerank and before context assembly. A rival tenant’s clause must never become a candidate. Do not, under any circumstances, retrieve across tenants and ask Claude to ignore the wrong ones.
6. Force attribution and check groundedness. Constrain generation to return cited_chunk_ids, and reject any answer citing a chunk that was not retrieved. This gives you the audit trail the legal team needs and the labelled data the eval loop (Lab 6) will consume.

The exercise is a compression of the whole lab: subtraction test, structural chunking, metadata-rich index, hybrid retrieval matched to two query shapes, ACL at retrieval, and forced attribution. If you can defend each of the six steps out loud, you can defend a Domain 3 item.

Key Takeaways

• Reach for RAG last. Exact key → database. Small, stable corpus → cached prefix. Computed answer → tool. RAG is only for corpora too big to carry and queries too fuzzy to key.
• The pipeline is ten stages: ingest, parse, chunk, embed, index, retrieve, rerank, assemble, generate, attribute. Debugging means naming which stage broke.
• Chunk boundaries destroy meaning. A table split across chunks answers nothing. Chunk on the document’s own structure when it has one, and use parent-document retrieval to keep context.
• Match retrieval to query shape: exact key → lookup, rare token → BM25, semantic → dense, both → hybrid + rerank (the strong default). Pure dense on identifiers is the classic wrong answer.
• Metadata (source, timestamp, version, ACL/tenant) is a first-class citizen. ACL filtering happens at retrieval, never in the prompt.
• Monolithic context is simple, cacheable, and degrades with irrelevance; progressive discovery is cheaper on average with higher latency variance and is hard to cache. Enumerable slice → monolithic; unknowable evidence → progressive. Skills use progressive disclosure.
• Force citations and validate that cited spans exist. An un-attributable answer has no eval loop (Lab 6) and no defensible audit (Lab 8).
• When a symptom correlates with a data event (refresh, re-index) and not a model event, investigate the data plane, the guide’s Sample 3. And if you change the embedding model, re-embed the entire index; mixing embedding spaces silently returns garbage.