Structured Outputs for LLMs: JSON Schema Design, Validation, and Error Handling

Overview

LLMs generate text probabilistically. Your application, by contrast, usually expects a defined contract: a string with a known meaning, an array of objects, a limited set of status values, or a field that must be present before the next workflow step can run. Structured outputs connect those two worlds by describing the response shape in a machine-readable schema.

There are three related approaches:

• Prompted JSON: You ask the model to return JSON and parse the result yourself. This is widely portable, but it offers the weakest guarantees.
• JSON mode: The API constrains the response toward valid JSON. This can prevent malformed syntax, but valid JSON may still have the wrong fields, types, or values.
• Schema-enforced structured output: You provide a schema and the API or model integration applies stronger constraints to the generated structure. The exact supported keywords, nesting depth, optional-field behavior, and model availability vary by provider.

None of these approaches proves that the content is true, complete, or safe. Validation checks shape; business rules, authorization, source verification, and human review may still be required. Treat the schema as an interface contract, not as a substitute for application logic.

Checklist by scenario

1. Extracting fields from unstructured text

Start with the smallest useful response. If the task is to identify a support ticket's priority, do not ask for a complete customer profile, explanation, and suggested resolution unless the application needs them.

• Define each field's purpose in one sentence.
• Choose explicit types: string, integer, number, boolean, array, or object.
• Use an enum for controlled values such as low, normal, and urgent.
• Decide how missing information is represented: a nullable field, an empty array, or a separate status.
• Tell the model not to infer a value when the source does not support it.
• Validate the result before storing or routing it.

A compact schema might look like this:

{
  "type": "object",
  "properties": {
    "priority": {
      "type": "string",
      "enum": ["low", "normal", "urgent"]
    },
    "customer_id": {
      "type": ["string", "null"]
    },
    "confidence_note": {
      "type": "string"
    }
  },
  "required": ["priority", "customer_id", "confidence_note"],
  "additionalProperties": false
}

If your selected provider or validator does not support union types such as ["string", "null"], model the value with a supported nullable pattern or use a separate presence indicator. Check the provider's current schema documentation rather than assuming full JSON Schema compatibility.

2. Producing a nested response

Nested objects are useful for invoices, product records, search results, and agent plans, but every additional layer creates more opportunities for ambiguity. Name objects according to their role and keep repeated structures consistent.

• Define the minimum required fields at each level.
• Use arrays when order matters or multiple items are expected.
• Specify whether an empty collection is valid.
• Keep nesting shallow unless the application genuinely needs a hierarchy.
• Test long arrays, empty arrays, missing source data, and mixed-quality input.

For a response that drives a user interface, consider including a stable identifier for each item and a machine-readable status. Avoid making downstream code infer meaning from display text.

3. Selecting an API capability

Use the strongest response control that your provider, model, SDK, and use case support. A practical comparison looks like this:

Major API providers use different names, request formats, supported models, and schema subsets for these features. Before implementation, confirm the current documentation for the provider you are using, then pin an SDK version where appropriate and test the exact production model. Capability labels are not interchangeable: a JSON mode response can be syntactically valid while failing your application's data contract.

4. Connecting output to application code

Separate generation, parsing, validation, and business processing into distinct steps:

1. Send the input and response instructions.
2. Capture the raw response and relevant request metadata for debugging.
3. Parse the response as JSON using a strict parser.
4. Validate it against the schema.
5. Apply business rules, such as allowed ranges or permission checks.
6. Only then write to a database, call a tool, or trigger automation.

In typed applications, map validated data into a dedicated internal type instead of passing an untyped object throughout the codebase. A validator such as a JSON Schema library, a language-native data model, or a runtime type-checking library can provide this boundary. The specific library matters less than making validation mandatory and observable.

5. Retrying a failed generation

Retry only when the failure is recoverable. A malformed response, temporary provider error, or truncated completion may justify a retry. A refusal, invalid input, authorization failure, or repeated schema violation needs a different path.

• Set a small retry limit and use backoff for transient API errors.
• On a validation failure, send a concise error summary rather than blindly repeating the entire conversation.
• Do not include secrets, personal data, or untrusted raw output in a repair prompt unless required and safely handled.
• Record the failure category, model identifier, schema version, and attempt number.
• Provide a fallback such as a review queue or an explicit “unable to extract” result.

For tool calls and agent workflows, validate arguments before execution and apply authorization outside the model. More guidance on that boundary is available in the function calling and tool use implementation guide.

What to double-check

• Required versus optional: A field should be required only when the application cannot proceed without it. Requiring every descriptive field encourages invented filler.
• Null versus empty: Use one consistent convention. A null value can mean “unknown,” while an empty array can mean “known to contain nothing.” Document the distinction.
• Enums: Keep enum values stable and machine-friendly. Add display labels in application code rather than changing the stored values casually.
• Numbers and dates: Decide whether dates use a fixed string format and whether numeric values may be negative, fractional, or outside a defined range. JSON Schema type validation alone may not enforce every business constraint.
• Additional properties: Reject or deliberately preserve unknown keys. Silent acceptance can hide prompt drift and schema changes.
• Truncation: Check the provider response for incomplete generation before parsing. A response can fail because it ended at a token limit rather than because the prompt was unclear.
• Refusals and empty responses: Handle these as explicit outcomes. Do not assume that every request produces a schema-conforming object.
• Prompt injection: If the input includes web pages, documents, emails, or retrieved text, treat that content as untrusted data. Structured output does not make instructions inside retrieved content trustworthy. Review the prompt injection prevention checklist.
• Observability: Track validation failure rates, retry counts, latency, token usage, and representative failure examples without logging sensitive content unnecessarily.

For a RAG system, validate both the answer structure and the citation or source-reference fields. A correctly shaped response can still cite the wrong document or answer beyond the retrieved evidence. The RAG tutorial provides a broader workflow for retrieval reliability.

Common mistakes

Assuming valid JSON means valid data

{"priority":"urgent"} is valid JSON, but it may be unusable if the contract also requires a ticket identifier and a reason. Always run schema and business-rule validation.

Designing a schema from the model's first answer

Start from the consumer's needs, not from an attractive sample response. A schema should describe the stable interface your application needs, while examples should clarify edge cases.

Making every field required

Overly strict schemas turn uncertainty into fabricated values. Give the model a supported way to express missing information and test that behavior explicitly.

Using one repair prompt for every failure

Syntax errors, missing fields, refusal responses, timeouts, and invalid business values have different causes. Classify the failure first, then select a retry, fallback, or review action.

Changing schemas without versioning

Changing an enum, renaming a key, or altering nullability can break stored records and downstream consumers. Use a schema version, migration plan, or compatibility layer when the response is persisted or shared.

Ignoring provider-specific limits

Structured-output support is not identical across APIs. Model availability, supported keywords, nesting, arrays, refusals, streaming, and SDK serialization can change. Treat provider documentation and an integration test as part of the implementation.

When to revisit

Review a structured-output integration whenever its inputs, model, provider, or downstream workflow changes. At minimum, revisit it in these situations:

• Before a seasonal planning or high-volume cycle, when failure handling and capacity matter more.
• When switching models, API providers, SDK versions, or response-control modes.
• When adding a field, enum value, nested object, tool call, or new source type.
• When validation failures, retries, refusals, or parsing errors increase.
• When a RAG index, document format, prompt, or retrieval strategy changes.
• When a schema is used by a new service, database table, user interface, or automation.
• After a security review identifies untrusted input or an unsafe downstream action.

Keep a small regression set containing normal inputs, missing information, conflicting instructions, long inputs, empty results, malformed source text, and adversarial content. Run it whenever the schema or workflow changes. Compare not only whether responses parse, but also whether required fields are accurate, unknown values remain unknown, and downstream actions stay within their intended boundaries. The prompt testing and evaluation framework can help organize this process.

Before shipping, use this final checklist:

1. Write the consumer-first schema and document every field.
2. Choose the strongest structured-output capability available for the exact model and SDK.
3. Test required, optional, nullable, empty, nested, and invalid cases.
4. Parse and validate before any database write, tool call, or automated action.
5. Classify failures and implement bounded retries plus a safe fallback.
6. Log enough metadata to investigate failures without exposing sensitive data.
7. Version the contract and schedule a review after provider or workflow changes.

This workflow turns reliable JSON from a prompting preference into an application boundary: explicit, testable, observable, and safe to update.