Function calling lets an LLM request actions from your application, such as retrieving account data, searching a knowledge base, or creating a support ticket. This guide explains how to design tool schemas, validate arguments, manage failures, restrict permissions, and track provider changes so your AI tool use remains predictable as the application evolves.
Overview
LLM function calling, also called tool use, is a structured exchange between a model and your application. The model does not directly execute a function. Instead, it receives a list of available tools, decides whether one may help, and returns a structured request containing a tool name and arguments. Your server validates that request, performs the permitted operation, and sends the result back to the model for a final response.
A typical tool-calling loop has five stages:
- Define a small set of tools and describe their inputs with a schema.
- Send the user request, system instructions, and tool definitions to the model.
- Inspect the model response for a tool call rather than assuming it contains a final answer.
- Validate and authorize the requested operation before execution.
- Return a bounded result to the model, then produce a user-facing response.
The exact request and response fields vary across providers and SDKs. Some APIs describe tools as functions, while others use a broader tools interface. Treat provider documentation and your installed SDK version as the source of truth. Your application should also keep a provider-neutral internal representation so that changing models does not require rewriting business logic.
Function calling is useful in LLM app development, but it is not a security boundary by itself. A schema tells the model what a valid request looks like; it does not grant permission to perform the operation. Authorization, rate limits, audit logging, and confirmation rules must remain in application code. For a deeper security review, see Prompt Injection Prevention: A Security Checklist for LLM Applications.
What to track
1. Tool definitions and schemas
Track every tool as a versioned interface. Record its name, purpose, owner, input fields, required fields, allowed values, maximum lengths, and whether the operation is read-only or mutating. Prefer narrow tools such as lookup_order or create_draft_reply over broad tools such as run_database_query. Narrow interfaces are easier to test and limit the consequences of an incorrect call.
Use schemas that are explicit enough to reject ambiguous input. Define enums for fixed choices, numeric bounds where applicable, and clear descriptions for fields with similar meanings. Avoid asking the model to generate opaque strings when the server can derive them. For example, derive the authenticated user ID from the session instead of exposing a user_id argument that could be manipulated.
2. Validation and authorization
Measure how often tool requests fail validation, are denied by authorization, or require a clarification. These are different failure classes and should not be combined. A malformed date may indicate a schema or prompting problem; a denied account action may be an expected permission check; repeated clarification may indicate that the tool is too broad or the user request lacks necessary information.
Validate arguments on the server with the same rigor used for ordinary API clients. Check types, ranges, ownership, tenant boundaries, and business rules. Then apply authorization independently. A valid request to refund an order is not necessarily an authorized request to refund that order.
3. Execution reliability
Track latency, timeout rate, retry count, downstream status, duplicate execution risk, and the percentage of calls that return usable results. Assign each tool a timeout appropriate to its function. A read-only search may tolerate a short retry, while a payment, deletion, or message-sending operation should not be retried automatically without an idempotency strategy.
Return structured errors to the model rather than leaking raw stack traces or sensitive service responses. A useful error identifies whether the request was invalid, unauthorized, temporarily unavailable, or completed with no result. Keep error payloads short and predictable so the model can decide whether to ask the user, try another permitted tool, or provide a limitation.
4. Provider and SDK behavior
Maintain a small compatibility record for each provider or model family you use. Confirm the current names for tool definitions, tool-call identifiers, argument encoding, parallel calls, forced tool selection, structured outputs, streaming behavior, and usage reporting. These capabilities can differ by model, endpoint, SDK version, or account configuration, so avoid treating a provider-wide label as a guarantee.
| Area | What to compare | Implementation precaution |
|---|---|---|
| Tool declaration | Schema format, required fields, descriptions, and naming rules | Use an internal tool specification and translate it at the adapter layer |
| Tool selection | Automatic selection, forced selection, refusal, and multiple calls | Handle zero, one, and several calls without assuming one fixed response shape |
| Arguments | JSON validity, escaping, partial output, and type preservation | Parse defensively and reject anything that fails schema validation |
| Execution loop | Call identifiers, result correlation, streaming, and turn limits | Store conversation state and correlate every result with the original request |
| Operations | Timeouts, retries, idempotency, and cancellation | Set policy per tool instead of applying one retry rule globally |
For surrounding integration decisions, the AI API Integration Guide covers model selection, cost controls, and production workflow concerns. For tool-heavy applications, also review the patterns in AI Agent Workflow Design.
Cadence and checkpoints
Review tool use on a monthly or quarterly cadence, with additional checks whenever a tool, model, SDK, permission rule, or downstream API changes. A practical review does not need to be a large rewrite. It should compare current behavior with a known test set and inspect the operational data collected since the previous review.
Before adding or changing a tool
- Write the tool's narrow purpose and identify operations it must never perform.
- Define a versioned input schema with examples, limits, and safe defaults.
- Decide whether the action is read-only, reversible, or requires explicit confirmation.
- Specify authorization, timeout, retry, idempotency, and audit requirements.
- Add normal, malformed, ambiguous, unauthorized, and adversarial test cases.
During each periodic review
- Check provider release notes and current API documentation for terminology or behavior changes.
- Run a fixed evaluation set against every supported model and tool combination.
- Compare validation failures, unauthorized requests, latency, timeouts, retries, and user corrections.
- Inspect a sample of tool calls for excessive permissions, sensitive data exposure, and unnecessary chaining.
- Confirm that logs contain enough information to debug calls without storing secrets or unnecessary personal data.
Prompt tests should include tool-selection accuracy, argument correctness, refusal behavior, recovery from tool errors, and final-answer quality. A tool call that is technically valid can still be unsuitable if it uses the wrong account, searches the wrong data source, or reports an unverified result. The Prompt Testing and Evaluation Framework provides a useful structure for building repeatable checks.
How to interpret changes
An increase in tool-call errors is a signal to classify before attempting a fix. If malformed arguments rise after a schema change, improve field descriptions, constraints, or examples and rerun tests. If valid calls are increasingly denied, the issue may be a changed permission model or a mismatch between the tool and the user's actual workflow. If latency rises while call success remains stable, inspect the downstream service and consider parallelizing independent read-only calls where the provider and application support it.
Watch for changes in call frequency as well. A model that calls tools too often may need clearer tool descriptions, stronger instructions about when not to call, or a simpler workflow. A model that rarely calls a necessary tool may be receiving incomplete context or facing an ambiguous schema. Do not optimize for the highest call rate; optimize for correct completion with the fewest safe operations.
Retries deserve separate analysis. Repeated retries on read operations may indicate transient infrastructure problems, but retries on mutating operations can create duplicates. Use idempotency keys where the downstream system supports them, persist execution state, and make duplicate detection part of testing. When an action has meaningful external impact, require a confirmation step or route the request to a controlled workflow rather than allowing an open-ended agent loop.
For retrieval-based tools, separate retrieval quality from tool execution quality. An LLM may correctly call a search tool but receive incomplete or stale documents. The RAG tutorial and RAG application architecture guide address the retrieval side of that problem.
When to revisit
Revisit this implementation guide at least quarterly and immediately after any of the following events:
- You add, remove, rename, or broaden a tool.
- You change a model, provider, endpoint, SDK, schema format, or streaming implementation.
- A downstream API changes its authentication, response format, rate limits, or error behavior.
- Metrics show a meaningful shift in validation failures, timeouts, retries, tool selection, or user corrections.
- A security review identifies prompt injection, excessive permissions, sensitive logging, or missing confirmation controls.
- You introduce a new agent loop, RAG source, automation, or high-impact external action.
Keep a short change log with the date, affected tools, provider or SDK version, test results, and rollback plan. Before deployment, run the same representative cases used in the previous review so that changes are comparable. After deployment, monitor the first batch of production calls more closely than normal and verify that audit records, authorization checks, and failure handling behave as expected.
The most reliable tool-calling systems are deliberately boring: their tools are narrow, their schemas are explicit, their permissions are enforced outside the model, and their behavior is measured over time. Treat each tool as a versioned API rather than a prompt trick, and revisit the contract whenever the model, application, or downstream service changes.
