Tool use and function calling: a practical implementation guide

Function calling lets a model request a structured action — lookup a customer, submit a ticket, query inventory — in a format your application can parse and route. It is the building block under every agent demo, and it is also where most implementation mistakes live: wrong schema design, unbounded parallel calls, silent retry loops, and observability gaps that turn a broken tool chain into an invisible one.

This guide walks through the practical steps: defining tool schemas that models actually use well, handling parallel calls without blowing through rate limits, recovering from failures without amplifying damage, and adding observability so you know when a tool chain is degrading before it breaks.

TL;DR

Implement function calling in four layers:

Schema design — define tools with clear names, detailed descriptions, and parameter constraints so the model can make reliable choices.
Execution routing — parse the tool call, validate it against current state, execute it, and return the result to the model.
Error and rate-limit handling — distinguish transient failures from permanent ones, cap retries, and use idempotency keys for non-read-only actions.
Observability — log every tool call, response, rejection, and retry so you can debug a chain without reproducing it.

Every provider supports the same basic shape: you define tools, the model picks one and emits structured arguments, your code executes the action, and the result feeds back into the conversation. The differences are in parameter naming, parallel-call support, and schema flexibility. Get the shape right once, and it maps across providers.

Defining tool schemas that models actually use well

The schema is the contract between your application and the model. A good schema makes tool selection obvious. A bad schema produces wrong tool calls that pass every validation check.

Name tools for what they do

Tool names should be action-verb + noun. Models match tool names against the user’s request in natural language terms.

Do this	Not this
`get_customer_by_id`	`customer_id_lookup_v2`
`submit_support_ticket`	`ticket_creation_endpoint`
`check_inventory_availability`	`inv_qty`
`calculate_shipping_cost`	`ship_calc_fn`

Natural, legible names reduce the probability of wrong-tool selection. If two tools have similar purposes, give them distinct, outcome-oriented names — refund_order and cancel_order are clearer than process_order_change.

Write descriptions that do the model’s reasoning for it

The description field is not documentation for developers. It is the model’s decision guide. Include:

what the tool does in plain English;
when to choose this tool over similar ones;
what kind of input is expected (units, formats, allowed values);
what side effects the action has (modifies data, sends a notification, charges money).

Example — weak description:

"Looks up a customer."

Example — useful description:

"Looks up a customer by email or account ID. Use this when the user provides an email or account number and you need their full profile, active orders, or support history. Returns name, email, account tier, and active order IDs. Does not modify any data."

Models are more likely to make correct tool choices when the description pre-filters the decision for them.

Constrain parameters with types and enums

JSON Schema constraints are not optional decorations. They are the model’s only reliable signal about what values are valid. Use:

type — string, number, integer, boolean, array, object
enum — list of allowed string values where choices are bounded
minimum / maximum — for numeric ranges
minLength / maxLength — for string fields
pattern — for format validation (email, phone, postcode)

Example — parameter definition:

{
  "type": "object",
  "properties": {
  "issue_type": {
  "type": "string",
  "enum": ["billing", "technical", "account", "other"],
  "description": "Category of the support issue"
  },
  "priority": {
  "type": "string",
  "enum": ["low", "medium", "high", "urgent"],
  "description": "Severity level. Use 'urgent' only for system-down or data-loss situations."
  },
  "description": {
  "type": "string",
  "minLength": 10,
  "maxLength": 2000,
  "description": "Free-text description of the issue"
  }
  },
  "required": ["issue_type", "priority", "description"]
}

Do not make optional parameters that should be required. Every optional parameter increases the chance the model omits something your downstream system needs.

Handling parallel and sequential tool calls

Tools can be called in sequence (the result of one feeds into the next) or in parallel (independent tools called at the same time). Each pattern has different implementation requirements.

Parallel calls: independent tools run concurrently

OpenAI, Anthropic and Gemini all support the model emitting multiple tool calls in a single response. If the tools have no data dependency on each other, run them concurrently.

Implementation rules for parallel calls:

Cap concurrency. Most providers emit 5–10 parallel calls per response by default. Set a maximum — 5 is a sensible starting point. More than that usually means the model is guessing rather than reasoning.
Collect all results before the next turn. The model needs to see all tool outputs before it can decide what to do next. Do not stream partial results back into the conversation — that causes hallucination over partial state.
Handle partial failures. If three out of five parallel calls succeed and two fail, the model should see the successes and the failures together so it can decide whether to retry, substitute, or ask the user. Log each result independently.

Example parallel flow:

User: "Check stock for items A, B, C, and recommend shipping options for my postcode."

Model emits 4 tool calls:
→ check_inventory("A")
→ check_inventory("B")
→ check_inventory("C")
→ get_shipping_options("SW1A 1AA")

Your code runs all 4 concurrently, collects results, sends them back as a single
tool-result batch. Model picks the next step based on all results.

Sequential calls: chaining dependent tools

When one tool’s output is another tool’s input, the calls must be sequential. The model emits the first tool call, your code runs it, the result goes back, and the model emits the next.

Common sequential patterns:

Lookup then act — get_customer_by_id → submit_refund
Validate then execute — check_budget_available → approve_spend
Search then summarise — search_knowledge_base → format_response

For sequential chains, add a timeout or step limit. A chain that runs more than 8–10 sequential tool calls is usually looping or has lost the thread. Log the full chain so you can inspect the decision path when it degrades.

Error recovery: what to do when a tool call fails

Tool calls fail for three reasons:

Transient failures — network timeout, rate limit hit, downstream service unavailable.
Semantic failures — the model called the right tool with wrong data (wrong ID, stale state, invalid combination).
Rejection failures — your validation or approval gate blocked the call.

Distinguish transient from permanent failures

Return different error signals to the model for each case:

Transient — include a retry_after field or a clear “retryable: true” flag. The model can try again.
Semantic — include the validation failure reason. The model should pick a different tool or ask for clarification, not retry the same call.
Rejection — include the rejection reason and a suggestion for what the model should do instead. Do not let the model re-request the same blocked action.

Example error response structure:

{
  "success": false,
  "error_type": "validation",
  "message": "Customer ID 'abc-123' not found. Available formats: email address or 8-character account ID starting with 'ACC-'.",
  "retryable": false,
  "suggestion": "Ask the user for their account ID or registered email address."
}

Cap retries with idempotency keys

For any tool that creates a side effect (writes data, sends a message, charges money), use an idempotency key. The key should be generated before the first call attempt and included in every retry.

Attempt 1: idempotency_key = "tool-order-123-retry-1"  → timeout
Attempt 2: idempotency_key = "tool-order-123-retry-1"  → success

The downstream service deduplicates based on the key. Set a maximum of 2–3 retries per call. After that, surface the failure to a human instead of looping silently.

Rate limiting tool calls

Rate limits apply at three levels, and you need to handle all of them.

Model-level rate limits

Every provider enforces rate limits on API calls: requests per minute (RPM) and tokens per minute (TPM). Tool calls consume tokens for the tool definitions in the request and the tool results in the response.

Practical limits to set:

Maximum tools-per-request: cap the number of tool definitions sent to the model. 20–30 tool definitions is typically fine for well-named tools. Beyond that, the model’s tool-selection accuracy degrades, and you pay for sending tool descriptions in every request.
Maximum parallel calls per response: 5–8 parallel calls is a reasonable ceiling. More than that usually indicates the model is casting a wide net rather than reasoning.
Total tool chain depth: cap the number of sequential tool calls in a single conversation turn. 15–20 is a high ceiling for legitimate use; anything above that is probably a loop.

Application-level rate limiting

Implement a token-bucket or sliding-window rate limiter per user session. This prevents one agent run from consuming all downstream service capacity.

Rate-limit actions at the tool level, not just the API level:

Read-only tools (lookups, searches): higher limit.
Write tools (submit, update, delete): lower limit, with per-user and per-action tracking.
Admin/privileged tools: explicit approval gate, not just a rate limit.

Downstream service limits

If your tool calls a third-party API (CRM, payment gateway, inventory service), respect that service’s rate limits. A burst of 10 parallel check_inventory calls might work locally but get rate-limited by the downstream service.

Queue tool calls that hit downstream limits with exponential backoff rather than failing them immediately. Return the “queued” status to the model so it can continue with other work.

Observability for tool chains

Tool chains fail in ways that normal application monitoring does not catch. A model might emit 12 tool calls, 3 succeed, 2 fail, and 7 are never executed because the chain terminated early. Without tool-specific observability, you only see the error logs — not the full path.

What to log per tool call

Log every tool invocation with:

Tool name and arguments (the model’s raw request)
Timestamp and latency (milliseconds)
Result or error (full response body)
Conversation ID or trace ID (links calls to the same user interaction)
Retry count (0 for first attempt, 1+ for retries)
Approval gate status (auto-approved, human-approved, rejected)

What to trace per conversation

A tool chain may span multiple model turns. Trace across turns:

Total tool calls made
Total retries
Total failures by type (transient, semantic, rejection, rate limit)
Total cost (input + output tokens per turn, plus downstream API costs)
Chain termination reason (completed successfully, hit step limit, user cancelled, error threshold exceeded)

What to alert on

Set alerts for these signals:

Retry rate > 10% of total tool calls in a 5-minute window
Sequential chain depth > 15 calls
Same error message returned 3+ times on the same tool
Approval gate rejections rising (may indicate the model keeps requesting a blocked action)
Latency > 5 seconds for a single tool execution (ignoring known slow services)

A compact implementation checklist

Before deploying a tool-using LLM feature:

Global applicability

This guide is global. The implementation patterns — schema design, error recovery, rate limiting, observability — apply across providers and jurisdictions. Provider-specific parameter names and capabilities change over time; check the source list below for the dates the current docs were verified.

Methodology

Data checked: 2026-05-29
Sources consulted: OpenAI function calling documentation, Anthropic tool use documentation, Google Gemini function calling documentation, Mistral function calling documentation, Berkeley Function Calling Leaderboard (BFCL V4), JSON Schema specification (draft-07)
Assumptions: This guide assumes the reader is building a tool-using system with a current-generation frontier model (GPT-5, Claude 4, Gemini 2.5 Pro, or equivalent). Older models may have weaker tool-selection accuracy and narrower parallel-call support. Error-handling patterns are general; exact rate-limit values depend on provider and pricing tier.
Limitations: This guide covers implementation mechanics for individual tool chains. It does not cover multi-agent orchestration, MCP server configuration, or complex routing between multiple models. It does not replace a security review of your specific tool-set and approval-gate design. Provider APIs, pricing, and supported schema features change — check the primary docs cited in the source list for your specific provider and date.
Jurisdiction: Global. No jurisdiction-specific regulatory guidance is included.

Trust Stack

Last checked: 2026-05-29
Corrections: Contact us to report errors

Source list

OpenAI function calling — https://developers.openai.com/api/docs/guides/function-calling (accessed 2026-05-29)
Anthropic tool use with Claude — https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview (accessed 2026-05-29)
Google Gemini function calling — https://ai.google.dev/gemini-api/docs/function-calling (accessed 2026-05-29)
Mistral function calling — https://docs.mistral.ai/capabilities/function_calling/ (accessed 2026-05-29)
Berkeley Function Calling Leaderboard V4 — https://gorilla.cs.berkeley.edu/leaderboard.html (accessed 2026-05-29)
JSON Schema specification — https://json-schema.org/specification (accessed 2026-05-29)

Change log

2026-05-29: First draft built from the llm-editor-approved Phase 2 brief with current provider-doc checks, implementation checklist, error-handling patterns, rate-limiting guidance, and observability recommendations.