{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Advanced Decomposition: Durability, Context Management, and Provenance\n", "\n", "*Module 11*\n", "\n", "This module covers the advanced logic needed to keep long-running research sessions accurate as they fill the model's context window: managing and summarizing context, recovering from crashes with state manifests, handling subagent failures intelligently, and preserving provenance so that findings stay attributable to their sources.\n", "\n", "**Lab notebook:** [Module11_Complete.ipynb](lab_files/Module11_Complete.ipynb)\n", "\n", "## 1. Managing Long-Horizon Context\n", "\n", "As sessions fill the model's context window, they can suffer from **context rot**, where the model loses focus on early instructions or misses information in the middle of long inputs, the \"lost in the middle\" effect.\n", "\n", "**Recognize the symptoms before they compound.** A degraded session starts answering from *typical patterns* instead of the specific classes, files, or facts it discovered earlier in the conversation: it invents a plausible-sounding method name instead of the one it read in turn 4, or gives inconsistent answers to the same question late in the session.\n", "\n", "Apply the remedies in this order:\n", "\n", "1. **Trim verbose tool outputs** before they enter history. A 40-field order lookup where only 5 fields matter should be reduced to those 5 fields at the tool boundary. Every irrelevant field you keep out of the transcript delays degradation.\n", "2. **Summarize completed phases** and carry the summary forward in place of the full transcript of that phase. In Claude Code, use the `/compact` command when exploration output has filled the window: it replaces the conversation with a summary and continues from there.\n", "3. **Start a fresh session with a structured summary** when prior tool results are stale or the session has already degraded. Injecting a clean, structured summary of decisions and findings into a new session is more reliable than continuing to push a degraded one forward.\n", "4. **Use scratchpad files.** Have agents write key findings to a file (for example `findings.md`) as they work, and reference that file for subsequent questions. Findings written to disk survive context boundaries, summarization, and even session restarts.\n", "\n", "> **Tip.** **Exam tip:** the ordering matters. Trimming is preventive and cheap; summarization recovers space mid-session; a fresh session with an injected summary is the reset button when the session has already degraded; scratchpad files make all three safer because the facts no longer live only in the transcript.\n", "\n", "### 1a. Position-Aware Input Ordering\n", "\n", "To mitigate the \"lost in the middle\" effect within a single large input, place key findings summaries at the beginning of aggregated inputs, organize detailed results with explicit section headers, and repeat the most critical facts at the end.\n", "\n", "```aggregated input layout\n", "\n", "- Max Budget: $50k\n", "- Current Competitor: ContosoAI\n", "- Decision Date: 2026-09-15\n", "\n", "\n", "## Research Detail: Budget\n", "...\n", "\n", "## Research Detail: Competitor\n", "...\n", "```\n", "\n", "Do not bury critical facts between long raw excerpts. Put the summary first, then the detailed evidence, and restate the key facts at the end when the input is very long.\n", "\n", "## 2. The Case Facts Pattern\n", "\n", "Progressive summarization erodes precise, non-negotiable details: confirmed budget figures, meeting dates, order numbers, and the customer's stated expectations get paraphrased away one summary at a time.\n", "\n", "Extract those critical facts into a `case_facts` block that is **re-included in every prompt, outside any summarized history**. Because it travels with each request rather than living inside the transcript, it is never subject to summarization.\n", "\n", "```case facts block (re-sent with every prompt)\n", "\n", "- Max Budget: $50k\n", "- Renewal Decision Date: 2026-09-15\n", "- Order Number: ORD-33412\n", "- CRM Account ID: acct_7781\n", "- Stated Expectation: refund processed before renewal date\n", "\n", "```\n", "\n", "Rule: `case_facts` is never summarized. However aggressively the conversational narrative is compressed, the hard data arrives verbatim on every turn.\n", "\n", "## 3. Decomposition: Fixed Pipelines vs. Adaptive Plans\n", "\n", "Orchestrators must choose between predictable efficiency and dynamic durability.\n", "\n", "- **Prompt chaining (fixed sequential):** a hard-coded sequence such as research → draft → review. It is predictable and fast but brittle. If research surfaces a deal-breaker, the system may still attempt to draft a useless report.\n", "- **Adaptive decomposition (dynamic):** the coordinator inspects each subagent's output and re-plans the next steps based on findings. If research finds a prospect already uses a competitor, the coordinator skips the standard pitch and spawns a competitor-positioning subagent instead.\n", "\n", "Adaptive systems cost more coordinator turns, but they are the durable pattern for open-ended work.\n", "\n", "## 4. The Subagent Failure-Report Contract\n", "\n", "A coordinator's ability to recover is bounded by the quality of the failure report it receives from a subagent.\n", "\n", "Subagents should return a structured contract instead of generic prose errors.\n", "\n", "*coordinator-level failure report*\n", "```json\n", "{\n", " \"status\": \"failed\",\n", " \"failure_type\": \"TRANSIENT\",\n", " \"partial_results\": [\n", " \"Found pricing page\",\n", " \"Confirmed enterprise plan exists\"\n", " ],\n", " \"recommended_next_step\": \"Retry competitor review after rate limit clears.\"\n", "}\n", "```\n", "\n", "- `failure_type`: an enum such as `TRANSIENT` or `VALIDATION`, allowing the coordinator to branch without re-reading prose.\n", "- `partial_results`: lets the coordinator salvage what worked instead of wasting the entire run.\n", "- `recommended_next_step`: the subagent's local assessment of what it learned about the failure.\n", "\n", "**Outer-loop signal:** `isRetryable` from Module 9 is a tool-level signal. This structured report is the coordinator-level signal for orchestrating complex recoveries.\n", "\n", "### 4a. Coverage Annotations in the Final Synthesis\n", "\n", "Salvaging partial results is only half the contract. The final synthesized report must state **which findings are well-supported and which topic areas have gaps** because a source was unavailable or a subagent failed.\n", "\n", "A report that silently omits a failed subtopic reads as complete when it isn't. The reader has no way to distinguish \"we checked and found nothing\" from \"we never checked.\" Require the coordinator to carry each failure report forward into an explicit coverage annotation, for example: \"Competitor pricing could not be verified; the pricing source timed out. Findings in this section are based on cached data from the prospect's own materials.\"\n", "\n", "## 5. Crash Recovery with State Manifests\n", "\n", "A long-running multi-agent workflow that keeps all of its state in conversation history loses everything when the process crashes. The fix is the same discipline as the scratchpad pattern, made structured: **each agent exports its state to a known location as it works** — findings so far, files analyzed, and open questions.\n", "\n", "*state manifest*\n", "```json\n", "{\n", " \"agent\": \"pricing-researcher\",\n", " \"updated_at\": \"2026-07-06T14:32:00Z\",\n", " \"findings\": [\n", " \"Enterprise tier confirmed at $48k/yr\",\n", " \"Discount requires 2-year commitment\"\n", " ],\n", " \"files_analyzed\": [\"pricing.html\", \"terms.pdf\"],\n", " \"open_questions\": [\"Does the discount apply to renewals?\"]\n", "}\n", "```\n", "\n", "On crash or restart, the coordinator loads each manifest and injects the summaries into fresh agent prompts. The new agents resume from recorded findings and open questions instead of re-running every search and re-reading every file.\n", "\n", "> **Tip.** **Exam tip:** manifests are written continuously *as the agent works*, not at shutdown — a crash by definition gives you no shutdown hook.\n", "\n", "## 6. Provenance: Claim-Source Mappings\n", "\n", "When subagent findings are compressed into summaries without claim-source mappings, **attribution is lost** and the final report can no longer say where any number came from. Require every research subagent to output structured mappings — claim, evidence excerpt, source, and publication date — and require the synthesis step to preserve and merge them rather than flattening findings into prose.\n", "\n", "*claim-source mapping*\n", "```json\n", "{\n", " \"claim\": \"Enterprise plan costs $48k/yr\",\n", " \"evidence_excerpt\": \"Enterprise: $4,000/mo billed annually\",\n", " \"source\": \"https://contoso.example.com/pricing\",\n", " \"source_name\": \"ContosoAI pricing page\",\n", " \"publication_date\": \"2026-05-14\"\n", "}\n", "```\n", "\n", "- **Conflicting statistics from credible sources:** never arbitrarily pick one. Annotate the conflict with *both* values and their attribution, and let the coordinator decide how to reconcile them before synthesis.\n", "- **Temporal data:** require publication or collection dates on every claim. A 2023 figure next to a 2026 figure with dates attached reads as a time series; without dates it reads as a contradiction.\n", "- **Separate established from contested:** the report should distinguish well-established findings (multiple agreeing sources) from contested ones (annotated conflicts), so readers can weigh them appropriately.\n", "- **Render content appropriately:** financial data as tables, news and narrative context as prose, technical findings as structured lists. Format is part of faithful reporting.\n", "\n", "> **Tip.** **Exam tip:** a synthesis that quietly resolves a source conflict has destroyed information. The correct behavior is to surface both values with attribution and dates, and escalate the reconciliation decision to the coordinator.\n", "\n", "## Lab Exercise: Managing a Long-Horizon Research Session\n", "\n", "**Lab notebook:** [Module11_Self_Driven_Lab.ipynb](lab_files/Module11_Self_Driven_Lab.ipynb)\n", "\n", "**Objective:** master tool-output trimming, the case facts pattern, scratchpad continuity, failure-report contracts, and provenance preservation.\n", "\n", "1. **Trim:** intercept a verbose tool result (a 40-field order lookup) and keep only the relevant fields before it enters history. Late in a long session, compare the agent's answers with and without trimming — observe when the untrimmed session starts answering from typical patterns instead of the actual data.\n", "2. **Case facts:** extract amounts, dates, and IDs into a `case_facts` block that is included in every prompt. Verify a detail stated in turn 2 (such as `Max Budget: $50k`) survives verbatim to turn 30, even after the surrounding history has been summarized.\n", "3. **Scratchpad:** have the agent maintain a `findings.md` file while exploring. Kill the session, resume fresh with the scratchpad injected into the new prompt, and verify the agent continues the work without re-running its earlier exploration.\n", "4. **Failure report:** simulate a subagent timeout. Verify the coordinator receives the failure type, the attempted query, and `partial_results` — and that the final report carries a coverage annotation marking the affected topic area as a gap rather than silently omitting it.\n", "5. **Provenance:** run two sources with conflicting statistics through the pipeline. Verify both values survive to the final report with attribution and publication dates, annotated as a conflict rather than arbitrarily reconciled.\n", "\n", "> **Tip.** **Exam tip:** summarization and fresh-session-with-summary preserve narrative continuity; `case_facts` preserves exact facts; scratchpads and manifests preserve state across crashes; failure reports with coverage annotations preserve recovery options; claim-source mappings preserve attribution.\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.0" } }, "nbformat": 4, "nbformat_minor": 5 }