Structured JSON output from a model: schema, validation and a bounded retry (Python, pydantic)
A model's JSON can still be wrong. Validate it with pydantic in strict mode, send the errors back a few times, then stop. Tested code, no API key.

Asking a model for JSON is easy. Trusting the answer is the hard part. The text can be cut in the middle because of the token limit, wrapped in a Markdown fence, contain a field you never asked for, use a category that is not in your list, or write "3" where your code expects the number 3. If your program passes that result straight to a database or another service, the error shows up far from where it was born.
This guide builds the part between the model and the rest of your code: a strict pydantic schema, a validation step that says exactly what is wrong, and a retry loop that sends the errors back a limited number of times. The example needs no API key: the tests use a scripted fake model, so they check our code and nothing else. They do not tell you how often a real model gets it wrong; measure that with your own provider and prompts.
Versions checked on 2026-11-17: pydantic 2.14.3 on PyPI (released 2026-11-05), which requires Python 3.10 or newer; the tests ran on Python 3.11 and in CI on 3.12. Only Python is covered; the same idea works in any language with a validation library.
Three levels of "structured"
"Structured output" means three different things, and mixing them causes surprises:
| Level | What you do | What it guarantees |
|---|---|---|
| Ask in the prompt | "Answer with JSON like this: ..." | Nothing. The model usually complies, sometimes not. |
| Constrain the generation | Pass a JSON Schema to a provider feature that limits the tokens the model may produce | The text follows the schema, as far as that provider's feature supports it. Check which keywords it supports and its failure cases in the provider's documentation: I did not test any provider here. |
| Validate in your code | Parse and check the text with a library you control | Your program never receives data that broke your rules. |
The third level is the only one that does not depend on a provider, and it is needed even when you use the second: a constrained answer can still be cut off by the maximum output length, can be refused, and can satisfy the schema while being wrong in meaning. The schema checks shape, not truth. So always validate, and let the schema you send to the provider come from the same code.
One model, two uses
The example classifies a support message. The pydantic model is both the contract for your code and the source of the JSON Schema:
import json
import re
from typing import Callable, Literal
from pydantic import BaseModel, ConfigDict, Field, ValidationError
class Ticket(BaseModel):
"""What we want the model to return for each support message."""
# extra="forbid": a field we did not ask for is an error, not silently dropped.
# strict=True: "3" is not accepted for an int, true/false only for a bool.
model_config = ConfigDict(extra="forbid", strict=True)
category: Literal["billing", "bug", "feature", "other"]
priority: int = Field(ge=1, le=5)
summary: str = Field(min_length=1, max_length=120)
needs_human: bool
Two settings do most of the work:
extra="forbid"turns an unexpected field into an error. The default is'ignore': extra data is dropped silently, so a misspelled optional field would never be noticed.strict=Truestops type coercion. By default pydantic converts when it can: the strict mode page shows'123'becoming the integer123. That is friendly for form input; for model output it hides mistakes. The test below shows the difference on an equivalent input.
Literal[...] gives a closed list of values, and Field(ge=1, le=5) gives a numeric range. Ticket.model_json_schema() returns the JSON Schema for the model (JSON Schema page). The test checks what it produces: the category becomes an enum, the priority gets minimum 1 and maximum 5, and extra="forbid" becomes additionalProperties: false.
Validate the text, not a dictionary
The central call is model_validate_json, which parses and validates in one step (JSON page). Invalid JSON is not a separate exception: it comes back as a ValidationError too, with the error type json_invalid. One except handles both cases.
try:
return model.model_validate_json(strip_fence(raw)), attempt
except ValidationError as e: # also covers invalid JSON (type "json_invalid")
last_error = e
messages.append({"role": "assistant", "content": raw})
messages.append({"role": "user", "content": feedback(e)})
strip_fence removes one Markdown code fence if the model added it. It does nothing else on purpose. Looking for the first { in prose and cutting the last } is how you end up accepting half of one answer glued to half of another. A text that is not exactly JSON is an error, and the retry loop deals with it; the test test_text_around_json_is_not_repaired pins that decision.
_FENCE = re.compile(r"^```(?:json)?\s*(.*?)\s*```$", re.DOTALL)
def strip_fence(text: str) -> str:
"""Models often wrap JSON in a Markdown code fence. Remove exactly one fence, nothing else."""
m = _FENCE.match(text.strip())
return m.group(1) if m else text.strip()
def feedback(error: ValidationError) -> str:
"""Tell the model what was wrong: field path, error type, message. Never the input values."""
lines = []
for e in error.errors(include_input=False, include_url=False):
where = ".".join(str(p) for p in e["loc"]) or "(root)"
lines.append(f"- {where}: {e['msg']} [{e['type']}]")
return "The JSON was not valid. Fix these problems and answer with the corrected JSON only:\n" + "\n".join(lines)
Look at what feedback leaves out. It sends the field path, the message and the error type, but not the value the model produced. errors() accepts include_input=False and include_url=False (see the ValidationError.errors reference in pydantic-core). An invalid value can be long and can contain personal data copied from the user's message. The model already has its own previous answer in the conversation, as the loop below shows.
The bounded retry loop
def extract(call_model: Callable[[list], str], prompt: str, model=Ticket, max_attempts: int = 3):
"""Ask the model, validate the answer, and on failure send the errors back.
call_model(messages) -> str is your own wrapper around any provider.
Returns (validated_object, attempts_used).
"""
messages = [{"role": "user", "content": prompt}]
last_error = None
for attempt in range(1, max_attempts + 1):
raw = call_model(messages)
try:
return model.model_validate_json(strip_fence(raw)), attempt
except ValidationError as e: # also covers invalid JSON (type "json_invalid")
last_error = e
messages.append({"role": "assistant", "content": raw})
messages.append({"role": "user", "content": feedback(e)})
raise StructuredOutputError(max_attempts, last_error)
How it works, step by step:
- The first message is your prompt.
call_modelis any function you write that takes the list of messages and returns the text. Wrap your provider's SDK in it; the example calls no provider. - If validation passes, it returns the object and the number of attempts used. Log that number: it is the cheapest quality metric you can get.
- If it fails, the model's own answer is appended as an
assistantmessage, followed by ausermessage with the list of errors. The model sees what it wrote and what is wrong with it. - After
max_attemptsit raisesStructuredOutputError. There is no endless loop and the cost of one request has a ceiling.
The default is 3 attempts. A model told category: Input should be 'billing', 'bug', 'feature' or 'other' gets a precise instruction to correct. I did not measure how often that works with a real model, so it is a design choice, not a result. Measure your own failure rate before allowing more attempts; each extra attempt is a new billable call (see the token and cost article).
What the tests prove
Fifteen tests, all offline. The most useful one shows why strict mode matters. An equivalent input is accepted by a default (lax) model and rejected by the strict one:
def test_lax_mode_quietly_coerces_strings(self):
t = LaxTicket.model_validate_json('{"category": "bug", "priority": "3", "needs_human": "yes"}')
self.assertEqual((t.priority, t.needs_human), (3, True)) # "3" -> 3 and "yes" -> True
def test_strict_mode_rejects_the_same_input(self):
with self.assertRaises(ValidationError) as cm:
extract.Ticket.model_validate_json(
'{"category": "bug", "priority": "3", "summary": "x", "needs_human": "yes"}')
types = {e["type"] for e in cm.exception.errors()}
self.assertEqual(types, {"int_type", "bool_type"})
Other tests cover an extra field (extra_forbidden), a value outside the enum and the range (literal_error and less_than_equal in one pass, so the model gets all problems at once), and an answer cut at 40 characters, as if max_tokens stopped it (json_invalid).
The retry test checks the conversation itself, not only the result:
def test_errors_are_sent_back_and_second_answer_is_accepted(self):
bad = '{"category": "urgent", "priority": 2, "summary": "Login fails", "needs_human": true}'
model = FakeModel(bad, GOOD)
t, attempts = extract.extract(model, "classify: login fails")
self.assertEqual(attempts, 2)
second_call = model.calls[1]
self.assertEqual([m["role"] for m in second_call], ["user", "assistant", "user"])
self.assertEqual(second_call[1]["content"], bad) # its own bad answer is shown back
self.assertIn("category: Input should be", second_call[2]["content"])
self.assertIn("[literal_error]", second_call[2]["content"])
And one test makes sure the feedback does not echo the invalid input: a "category" containing a card number is rejected, and the second prompt contains only the error text.
Pitfalls I found
- Duplicate keys. If a JSON object repeats a key, pydantic uses the last value, like
json.loads. This is documented and the testtest_duplicate_keys_last_one_winsconfirms it. RFC 8259 says that names within an object should be unique and that the behaviour of software receiving duplicates is unpredictable, so do not rely on it. - A float is not an int in strict mode.
3.0is rejected for anintfield withint_type. If your provider returns numbers like3.0, loosen that field or fix the prompt. - JSON is not Python. The pydantic page notes that JSON has no
datetype, so date strings stay accepted in strict mode when you parse JSON. Test the types you really use. - Valid is not true. An answer such as
"priority": 1, "needs_human": falsefor a message that says the whole service is down passes every rule above. Validation cannot find that. For meaning you need evaluation (a sample of answers checked by people or by a second step), which is a topic of its own. - Do not use partial parsing for the result.
pydantic_core.from_jsoncan parse an unfinished JSON text withallow_partial=True, which is useful to show a streaming answer as it grows. A truncated final answer is still an error: we wantjson_invalidthere, and the default behaviour does that. - User text in the prompt. The text you classify can contain instructions. Validation limits the shape of the output but not what the model is persuaded to write inside valid fields; see prompt injection: practical defenses.
When the retry is not the answer
A retry fixes a formatting slip, not an ambiguous prompt or a schema that is too hard. If many calls need a second attempt, change the prompt or the schema before raising max_attempts: describe each field with Field(description=...), show one example, simplify nested objects, or make a field optional when the information is often missing.
Also keep apart two kinds of failure. A validation error is a problem with the content, so the answer is another model call with feedback. A network error, a 429 or a timeout is a problem with the transport, and has its own rules: see the 429 and Retry-After article and the one on timeouts and cancellation. Keep the two loops separate and let one time budget cover both. If you want to see what a given retry policy does to the number of calls, the retry and backoff simulator shows it, and the HTTP status code reference lists which statuses are worth retrying.
When StructuredOutputError is raised, give the caller a clear outcome: queue the item for a person, return an error, or use a default you chose in advance. Log the field paths and error types, not the user's text, to see which field fails most often.
Run it yourself
The code is in the structured-output folder of the examples repository (MIT licence):
pip install -r requirements.txt # pydantic==2.14.3
python -m unittest -v
The CI run for this article is green. Nothing here calls a real model, so how often the first attempt is valid and what the retry costs are yours to measure.
Summary
- Always validate in your own code, even if the provider can constrain the output to a schema.
- Write the contract once in pydantic:
Literalfor closed lists,Fieldfor ranges,extra="forbid"andstrict=Trueso mistakes are errors and not silent conversions. - Use
model_validate_json: invalid JSON and wrong values come out as the sameValidationError, with a field path and an error type per problem. - Send those errors back, without the input values, for a limited number of attempts, and keep this loop separate from the network retry loop.
- A valid answer is not a correct answer: plan a separate check for meaning.
Sources: pydantic JSON, pydantic strict mode, pydantic JSON Schema, pydantic error handling, pydantic models, RFC 8259, The JSON Data Interchange Format, JSON Schema.
The complete, tested example for this article is in our public repository: adamant-solutions/ai-dev-examples/structured-output