{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "*Module 3 of 12*\n", "\n", "# Tool Design & Tool Choice Strategy\n", "\n", "Once you decide to build an agent (Module 2), the model is the one picking which tool to call. This module gives you the levers that control that decision: disambiguating tool descriptions, the `tool_choice` parameter for forcing prerequisites, strict-mode schema validation, and the agentic loop that ties it all together.\n", "\n", "\n", "## 1. Concepts: Model-Driven Routing\n", "\n", "In an agentic system, the model picks which tool to call, that decision is driven by the tool's `name`, `description`, schema, and the surrounding system prompt. Workflows hard-code the call site; agents read your tool surface and route. Four levers shape that routing:\n", "\n", "- **Tool descriptions are the primary selection mechanism.** Claude reads the `name` and `description` on each tool to choose. If two tools have overlapping purposes (e.g. `analyze_content` vs. `analyze_document`), Claude will misroute, write descriptions that disambiguate *exactly when* each tool is appropriate.\n", "- **Functional overlap must be removed, not merely explained.** If two tools do the same job with different names, rename or split them so each one owns a distinct input shape and outcome.\n", "- **System prompts can override good descriptions.** Keyword-sensitive instructions like \"for any document, call `analyze_document`\" can drag routing away from a better tool description. Audit prompt rules for accidental keyword traps.\n", "- **`tool_choice`** forces or constrains routing on a per-request basis: require *any* tool call, require a *specific* tool, or let the model decide.\n", "- **Strict mode (`strict: True`)** uses grammar-constrained sampling to force tool inputs to match your JSON schema 100%. No trailing commas, no missing required fields, no surprise keys.\n", "\n", "### 1a. Eliminating Functional Overlap\n", "\n", "\"Tool description\" is the most under-engineered part of most agentic systems. A bad description is one that tells Claude what the tool *does*; a good description tells Claude *when to pick it over its siblings*. A better design removes the sibling conflict entirely. Compare:\n", "\n", "*anti-pattern: ambiguous*" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# BAD: descriptions overlap, Claude will guess.\n", "tools = [\n", " {\"name\": \"analyze_content\", \"description\": \"Analyzes content.\"},\n", " {\"name\": \"analyze_document\", \"description\": \"Analyzes a document.\"},\n", "]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*good: disambiguated*" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# GOOD: each description names what the OTHER tool is NOT for.\n", "tools = [\n", " {\n", " \"name\": \"analyze_content\",\n", " \"description\": (\n", " \"Analyzes a single piece of inline text or HTML passed as a string \"\n", " \"(blog post body, email draft, social copy). Use this when the caller \"\n", " \"has the content in-hand. DO NOT use for files on disk or document IDs, \"\n", " \"use analyze_document for those.\"\n", " ),\n", " },\n", " {\n", " \"name\": \"analyze_document\",\n", " \"description\": (\n", " \"Analyzes a stored document referenced by document_id (PDF, Word, \"\n", " \"uploaded file). Use this when the caller has only an identifier or \"\n", " \"a file path. DO NOT use for raw inline text, use analyze_content.\"\n", " ),\n", " },\n", "]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The pattern: state the **input shape**, the **canonical use case**, and an explicit **\"DO NOT use for X, use *sibling_tool* instead\"** clause. That last clause is what eliminates misrouting between similar tools.\n", "\n", "*lab task: rename the overlapping tool*" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# STARTING POINT: both tools sound like generic analysis.\n", "tools = [\n", " {\"name\": \"analyze_content\", \"description\": \"Analyze content and return key points.\"},\n", " {\"name\": \"analyze_document\", \"description\": \"Analyze a document and return key points.\"},\n", "]\n", "\n", "# TARGET DESIGN: split by function and input source.\n", "tools = [\n", " {\n", " \"name\": \"analyze_uploaded_document\",\n", " \"description\": (\n", " \"Extracts claims and entities from an uploaded PDF/DOCX by document_id. \"\n", " \"Use only when the file already exists in document storage. Do not use \"\n", " \"for web search results or raw HTML.\"\n", " ),\n", " },\n", " {\n", " \"name\": \"extract_web_results\",\n", " \"description\": (\n", " \"Extracts title, URL, snippet, publication date, and source credibility \"\n", " \"from web_search results. Use only after web_search/web_fetch. Do not \"\n", " \"use for uploaded files or inline drafts.\"\n", " ),\n", " },\n", "]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1b. Prompt Audit for Keyword-Sensitive Instructions\n", "\n", "Even a clean toolbox can misroute if the system prompt contains broad keyword rules. Review instructions for phrases that bind routing to words in the user request instead of the tool's true preconditions.\n", "\n", "```prompt audit\n", "Risky: \"Whenever the user mentions a document, call analyze_document.\"\n", "Better: \"When the user provides a stored document_id or file path, use analyze_uploaded_document.\n", "When the user asks about search output or fetched web pages, use extract_web_results.\"\n", "```\n", "\n", "### 1c. Controlling Routing with `tool_choice`\n", "\n", "Descriptions guide the model's preference; `tool_choice` overrides it. Three values matter for the exam:\n", "\n", "- **`{\"type\": \"auto\"}`** (default), Claude decides whether to use a tool, which one, and may also reply with plain text.\n", "- **`{\"type\": \"any\"}`**, Claude *must* emit a tool call (no plain-text reply allowed). Use this when you want a structured response and any tool is acceptable.\n", "- **`{\"type\": \"tool\", \"name\": \"verify_identity\"}`**, Claude *must* call the named tool on this turn. Use this to enforce a prerequisite, e.g., identity verification before any account action.\n", "\n", "*forced prerequisite, then any-tool*" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import anthropic\n", "from dotenv import load_dotenv\n", "\n", "load_dotenv() # reads ANTHROPIC_API_KEY from your .env file\n", "\n", "client = anthropic.Anthropic()\n", "\n", "# Turn 1: force identity verification before anything else can happen.\n", "response = client.messages.create(\n", " model=\"claude-opus-4-7\",\n", " max_tokens=2048,\n", " tools=tools,\n", " tool_choice={\"type\": \"tool\", \"name\": \"verify_identity\"},\n", " messages=[{\"role\": \"user\", \"content\": \"Refund my last order.\"}],\n", ")\n", "\n", "# ... append assistant tool_use and the tool_result for verify_identity ...\n", "\n", "# Turn 2: identity is confirmed. Force a tool-based response (no chit-chat).\n", "response = client.messages.create(\n", " model=\"claude-opus-4-7\",\n", " max_tokens=2048,\n", " tools=tools,\n", " tool_choice={\"type\": \"any\"}, # must call SOME tool, model picks which\n", " messages=messages,\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> **Tip.** Architect Tip for the Exam\n", "The exam loves the prerequisite pattern: `tool_choice={\"type\":\"tool\",\"name\":...}` on the first turn forces a specific call (verify identity, fetch customer, look up account); on subsequent turns you switch to `{\"type\":\"any\"}` or `{\"type\":\"auto\"}`. Combine this with Module 9's hooks for defense in depth, `tool_choice` nudges the model, hooks *guarantee* the prerequisite at execution time.\n", "\n", "### 1d. Access Failures vs. Valid Empty Results\n", "\n", "When your code executes a tool, distinguish between two outcomes that look superficially similar but must be reported very differently:\n", "\n", "- **A valid empty result is NOT an error.** If a query ran successfully and simply matched nothing (no orders for that customer, zero rows), return the empty list as a normal `tool_result` with `isError` false. The model should tell the user \"no orders found\", not retry.\n", "- **An access failure IS an error.** A timeout, a permission denial, or a downstream outage means the tool never produced an answer. Return a *structured error* with an `errorCategory` (`transient`, `validation`, `business`, or `permission`) and an `isRetryable` flag so the agent knows whether retrying can help.\n", "\n", "*valid empty result*\n", "```json\n", "{\n", " \"type\": \"tool_result\",\n", " \"tool_use_id\": \"toolu_01A\",\n", " \"is_error\": false,\n", " \"content\": \"{\\\"orders\\\": [], \\\"query_status\\\": \\\"success\\\"}\"\n", "}\n", "```\n", "\n", "*structured access failure*\n", "```json\n", "{\n", " \"type\": \"tool_result\",\n", " \"tool_use_id\": \"toolu_01B\",\n", " \"is_error\": true,\n", " \"content\": \"{\\\"errorCategory\\\": \\\"transient\\\", \\\"isRetryable\\\": true, \\\"message\\\": \\\"Order service timed out after 5s\\\"}\"\n", "}\n", "```\n", "\n", "> **Tip.** **Exam tip:** \"The query returned nothing\" and \"the query could not run\" are different signals. Conflating them either makes the agent retry pointlessly on genuinely empty data or give up on a recoverable outage. The full error taxonomy is covered in Modules 7 and 9.\n", "\n", "## 2. Defining the Toolbox\n", "\n", "Our scenario is a support/CRM agent with two **client-side** custom tools: `lookup_order` reads order history, and `add_lead_to_crm` writes a new prospect. Client-side means that when Claude emits a `tool_use` block for either tool, *your code* is responsible for executing the action and returning the result as a `tool_result`.\n", "\n", "Apply the disambiguation rule from Section 1a: the `add_lead_to_crm` description states the *canonical use case* (\"a validated prospect\") and the *precondition* (\"after industry research is complete\"), so Claude won't fire it before the research step has run. Adding `strict: True` to a tool guarantees, via grammar-constrained sampling, that the tool inputs Claude produces are 100% valid against your JSON schema." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# DEFINE THE TOOLS\n", "# Note: Use 'strict: True' to guarantee schema-valid tool inputs.\n", "\n", "tools = [\n", " {\n", " \"name\": \"lookup_order\",\n", " \"description\": (\n", " \"Looks up a customer's order history by customer_id in the order \"\n", " \"database. Returns a list of orders (possibly empty). Use this for \"\n", " \"any question about existing orders. DO NOT use to create records, \"\n", " \"use add_lead_to_crm for new prospects.\"\n", " ),\n", " \"input_schema\": {\n", " \"type\": \"object\",\n", " \"properties\": {\n", " \"customer_id\": {\"type\": \"string\"},\n", " },\n", " \"required\": [\"customer_id\"],\n", " \"additionalProperties\": False,\n", " },\n", " },\n", " {\n", " \"name\": \"add_lead_to_crm\",\n", " \"description\": \"Adds a validated prospect to the enterprise CRM. Use only after industry research is complete.\",\n", " \"strict\": True,\n", " \"input_schema\": {\n", " \"type\": \"object\",\n", " \"properties\": {\n", " \"company\": {\"type\": \"string\"},\n", " \"contact_email\": {\"type\": \"string\", \"format\": \"email\"},\n", " \"budget_range\": {\"type\": [\"string\", \"null\"]}, # Nullable to prevent hallucinations\n", " },\n", " \"required\": [\"company\", \"contact_email\", \"budget_range\"],\n", " \"additionalProperties\": False,\n", " },\n", " },\n", "]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. The Initial Request & `stop_reason`\n", "\n", "Send the user's support request with the toolbox attached. Don't print just the final text, inspect `stop_reason` and walk the content blocks. On the first turn you should see `tool_use` (Claude asking your code to run `lookup_order`), not `end_turn`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# THE INITIAL REQUEST\n", "messages = [{\n", " \"role\": \"user\",\n", " \"content\": \"What did customer C-1042 order last month, and are any orders still open?\",\n", "}]\n", "\n", "response = client.messages.create(\n", " model=\"claude-opus-4-7\",\n", " max_tokens=8192,\n", " tools=tools,\n", " messages=messages,\n", ")\n", "\n", "# SHOW OUTPUT\n", "print(f\"Stop Reason: {response.stop_reason}\") # Expected: 'tool_use'\n", "for block in response.content:\n", " if block.type == \"tool_use\":\n", " print(f\"Claude requested tool: {block.name}\") # Expected: 'lookup_order'\n", " print(f\"Parameters: {block.input}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Handling the Agentic Loop\n", "\n", "Both tools are **client-side**: the API never executes them for you. Claude emits a `tool_use` block, your code performs the actual lookup or write, and you send the outcome back as a `tool_result` block before Claude can finish its turn.\n", "\n", "- **Turn 1:** Claude emits a `tool_use` block for `lookup_order` with the `customer_id` it extracted from the request.\n", "- **Turn 2:** your application queries the order database and appends a `tool_result` block (keyed by `tool_use_id`) on a user turn, then re-calls the API.\n", "- **Turn 3:** Claude reasons over the returned orders. It may request another tool (continuing the loop) or generate the final `end_turn` response.\n", "\n", "*client-side loop*" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def handle_tool_use(block):\n", " if block.name == \"lookup_order\":\n", " orders = order_db.query(block.input[\"customer_id\"]) # your client-side read\n", " return {\n", " \"type\": \"tool_result\",\n", " \"tool_use_id\": block.id,\n", " \"content\": json.dumps({\"orders\": orders, \"query_status\": \"success\"}),\n", " }\n", " if block.name == \"add_lead_to_crm\":\n", " # crm.write(block.input) # your client-side write\n", " return {\n", " \"type\": \"tool_result\",\n", " \"tool_use_id\": block.id,\n", " \"content\": \"ok: lead written\",\n", " }\n", " raise ValueError(f\"unhandled client-side tool: {block.name}\")\n", "\n", "# Loop until Claude is done: stop_reason drives the control flow.\n", "while response.stop_reason == \"tool_use\":\n", " tool_results = [handle_tool_use(b) for b in response.content if b.type == \"tool_use\"]\n", " messages.append({\"role\": \"assistant\", \"content\": response.content})\n", " messages.append({\"role\": \"user\", \"content\": tool_results})\n", " response = client.messages.create(\n", " model=\"claude-opus-4-7\",\n", " max_tokens=8192,\n", " tools=tools,\n", " messages=messages,\n", " )\n", "\n", "# stop_reason == \"end_turn\": the task is complete." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. Verifying the Final Answer\n", "\n", "Once the loop terminates with `end_turn`, inspect the final text blocks and sanity-check them against the `tool_result` data your code actually returned. If Claude's summary claims three open orders but the lookup returned two, the discrepancy should be caught here, before the answer reaches the user." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# FINAL OUTPUT VERIFICATION\n", "# Compare the final answer to the tool_result data you returned.\n", "assert response.stop_reason == \"end_turn\"\n", "final_text = \"\".join(b.text for b in response.content if b.type == \"text\")\n", "print(\"Final Answer:\", final_text)\n", "\n", "open_orders = [o for o in orders if o[\"status\"] == \"open\"]\n", "if str(len(open_orders)) not in final_text:\n", " print(\"WARNING: answer may not match tool data, review before sending.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> **Tip.** Architectural Summary\n", "\n", "**Safety:** nullable types in the schema (`[\"string\", \"null\"]`) prevent Claude from inventing a budget figure when one isn't in the source data.\n", "**Selection:** tool descriptions, not names alone, drive Claude's routing. Disambiguate any tools whose purposes overlap." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.0" } }, "nbformat": 4, "nbformat_minor": 5 }