Guide · AI data modeling

Structured data extraction with LLMs: schema in, checked and sourced values out.

Q: How do you do structured data extraction with an LLM?

Write a JSON Schema for the fields you need, with a clear description for each. Send the document and the schema with structured outputs switched on, then validate the values in code, verify each value’s quoted evidence, and send failures or low-confidence values to a person.

Q: What is the difference between JSON mode and structured outputs?

JSON mode guarantees the response is valid JSON. Structured outputs guarantee it also matches your JSON Schema — required keys, types and enums. Neither guarantees the values are correct.

Q: Do structured outputs stop hallucinations?

No. They constrain shape, not truth. A model can return a correctly formatted date that is the wrong date. Evidence fields, validation rules and review catch those.

Q: Which LLMs support structured outputs?

OpenAI, Anthropic and Google all offer schema-constrained output in their APIs, each with its own set of enforced JSON Schema keywords. Check each provider’s documentation for the models and keywords it covers.

Q: How do I get citations for extracted fields?

Add a quote and page number for each field to the schema, then check in code that the quote appears on that page and supports the value. Provider citation features suit prose answers better than schema-bound extraction.

Q: Should I use Pydantic or Zod for LLM extraction?

Both are common: define the record as a typed model, generate the JSON Schema from it, and validate the response against the same model. That keeps the schema, the parser and your code in one definition.

Q: How accurate is LLM data extraction?

It depends on the documents, the schema and the model, so measure it on a labelled golden set of your own documents — per field, and per document with every field right — before deciding what can skip review.

Q: Why does my model return null or invent values?

Usually the schema. A required non-null field pushes the model to invent a value; a vague description makes it pick the wrong one. Allow null where a value may be absent and describe what each field is not.

A model can be forced to return valid JSON. Whether the values in it are right, and where they came from, is a separate job — and the one that decides if anyone can trust the table.

This guide explains the four levels of output control, how constrained decoding guarantees a schema, what the major model providers enforce and what they leave to you, how to design an extraction schema, the five layers of validation, and how to make every value point back to the words it came from.

Updated October 2026 · 7 sources

Read the guide How Botminds does it

Valid by luckValid, checked and sourced

12345

Prompt asks for JSONUsually parses. Sometimes adds prose, drops a brace or renames a key.
JSON modeAlways valid JSON — but any shape. Required keys can still be missing.
Structured outputsConstrained decoding: output matches your JSON Schema, token by token.
Plus validationCode checks formats, ranges, totals and reference data the schema cannot express.
Plus verified evidenceEach value carries a quote and page, checked against the source text.

What is structured data extraction with LLMs?

Structured data extraction is reading named, typed values out of unstructured content — dates, amounts, parties, line items — and returning them as records that match a schema. With LLMs, you give the model a JSON Schema and the document; structured outputs (constrained decoding) guarantee the response fits the schema, and your own validation and citations establish that each value is correct and traceable to its source.

Output control

JSON mode vs structured outputs: valid syntax against a guaranteed schema.

The two are often confused. OpenAI’s guide puts it plainly: both produce valid JSON, but only Structured Outputs ensures schema adherence.

syntax

JSON mode

The model is constrained to emit parseable JSON. It decides the keys and types itself, guided only by the prompt.

Always parses
Keys and types can drift
Required fields can be missing
You validate the shape

schema

Structured outputs

The model is constrained to emit JSON that matches a supplied JSON Schema — required keys, types and enums included.

Matches the schema
No retries for shape errors
Some schema keywords not enforced
You still validate the values

Well-formed JSONWhat is guaranteedWell-formed JSON that fits the schema

A JSON object response formatHow it is requestedA JSON Schema with strict mode

PossibleWrong value, right typePossible

Breaks the shapeRefusal or truncationCan still break the shape — check the stop reason

100%

schema adherence for gpt-4o-2024-08-06 with Structured Outputs on OpenAI’s own complex JSON schema-following eval.

OpenAI, August 2024

<40%

for gpt-4-0613 on the same eval, without Structured Outputs. Both figures are OpenAI’s, measured on its own models.

OpenAI, August 2024

Constrained decoding

How constrained decoding makes structured data extraction hold its shape.

A model writes one token at a time. Constrained decoding turns the schema into a grammar and, at every step, removes the tokens that would break it — so the only outputs possible are valid ones.

Declare the schema
Fields, types, enums and required keys, written as JSON Schema — usually generated from a typed model in code.
JSON Schema
Compile a grammar
The provider turns the schema into a grammar. Anthropic notes extra latency on first use; compiled grammars are cached.
Mask at each token
While generating, tokens the grammar does not allow next are excluded from sampling.
Parse without retries
The response parses and fits the schema — unless the model refused or ran out of output tokens.
stop reason
Check the values
The grammar knows shapes, not facts. A date in the right format can still be the wrong date.
validation

An extraction schema with an evidence array

{
  "type": "object",
  "properties": {
    "invoice_date": {
      "type": ["string", "null"],
      "description": "Issue date, YYYY-MM-DD. Not the due date."
    },
    "total_due": {
      "type": ["number", "null"],
      "description": "Amount payable including tax."
    },
    "currency": { "type": "string", "enum": ["USD", "EUR", "GBP", "OTHER"] },
    "evidence": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "field": { "type": "string" },
          "quote": { "type": "string" },
          "page":  { "type": "integer" }
        },
        "required": ["field", "quote", "page"],
        "additionalProperties": false
      }
    }
  },
  "required": ["invoice_date", "total_due", "currency", "evidence"],
  "additionalProperties": false
}

What the provider docs say to watch

RefusalsOpenAI returns a separate refusal; Anthropic stops with a refusal reason that may not match the schema.
TruncationA response cut off by the output-token setting can be incomplete JSON.
KeywordsGemini ignores length limits on strings; Anthropic rejects some numeric and string constraints unless an SDK moves them into descriptions.
EnumsAnthropic does not guarantee the capitalisation of enum values — compare without case.
SemanticsGoogle warns of “schema-compliant but semantically incorrect outputs”: validate values in your application.

Designing the schema

Designing a schema for structured data extraction.

The schema is both the contract your database enforces and the instruction the model reads. Most extraction errors trace back to a field that was under-described or forced to have a value.

A field-by-field schema checklist

Describe every fieldSay what it is, where it usually appears and what it is not: “issue date, not the due date”. Descriptions are read by the model.
Allow nullIf a value can be absent, make it nullable. A required non-null field invites the model to invent one.
Enumerate categoriesUse enums for document types, currencies and statuses, with an explicit “other”.
Normalise typesDates as ISO 8601 strings, amounts as numbers with a separate currency field, percentages as decimals.
Model tables as arraysLine items, schedules and parties are arrays of objects — one object per row.
Ask for evidenceA quote and page per field turns every value into something a reviewer can check in seconds.
Keep it shallowDeep nesting and very large schemas are harder to follow; Google’s docs note very large schemas may be rejected.

Format can cost accuracy. Tam et al. (2024) found stricter format constraints tended to lower accuracy on reasoning tasks, while helping on classification, and that the order of keys mattered. Where a field needs reasoning — “is this clause a termination right?” — give the model a field to reason in before the answer, or extract in two passes.

Checks after the schema

Validating extracted values: five layers after the schema.

Constrained decoding closes the first layer. The next four are your code and your reference data, and the last is a person; together they catch the errors that matter to the business.

Each layer catches what the one above cannot

Syntax and shapeparses, required keys present, types and enums right

structured outputs

Field rulesdate formats and ranges, positive amounts, check digits such as the IBAN mod-97 check

per field

Cross-field rulesline items sum to the subtotal; subtotal plus tax equals total; due date after issue date

per document

Reference datasupplier exists in the vendor master; PO number is open; currency matches the contract

lookups

Evidence checkthe quoted words exist on the cited page, and the value can be read from them

grounding

Human reviewwhatever fails, or scores low, goes to a person with the page open

human in the loop

Which values may skip the last layer is a measured decision: see straight-through processing and confidence calibration.

Traceable values

Citations for extracted values: where the evidence should come from.

A value without a source cannot be audited. There are three ways to get one, and they suit different jobs.

What are you producing?

A prose answer about documentsUse the provider’s citation feature. Anthropic’s returns the cited text with character ranges for plain text and page ranges for PDFs.

Fields that must match a schemaPut evidence in the schema — a quote and page per field — and verify it in code. Anthropic’s citations cannot be combined with its structured outputs; the request returns an error.

Highlights on the page imageMatch each quote to the words and coordinates from OCR, so a reviewer’s click lands on the exact spot.

Verifying a cited value

The quote existsNormalise whitespace and search the cited page’s text. No match: flag the value.
The value follows from the quoteParse the quote with the field’s own rules and compare with the value.
The page is plausibleAn invoice date cited from the terms and conditions page deserves a second look.

Accuracy

Measuring structured data extraction accuracy.

A single “accuracy” number hides the failures that cost money. Measure at the level the business acts on.

Normalised

Field accuracy, normalised“1,250.00” equals “1250”: compare after normalising formats. The everyday number.most used

Documents fully rightShare of documents with every field right after normalising — what straight-through depends on.STP

Exact

Field exact matchStrict string equality. Useful for identifiers, harsh for amounts and dates.identifiers

Documents exactly rightThe strictest view; good for regulated filings, rarely the working target.strict

Per fieldPer document

Precision and recall per field

Count extracted-and-wrong apart from missed. A nullable field that is always null scores high on precision and zero on recall.

A golden set

Hundreds of labelled documents that mirror production, scored on every prompt or model change. See what is a golden dataset.

Calibrated confidence

A score that means “right nine times in ten” decides which values a person must see.

On Botminds

How Botminds runs structured data extraction from a data model.

On Botminds the schema is not code in a repository. It is the collection’s labels — and each label’s description is the instruction the extraction agent follows.

A collection’s labelsone definition: the agent’s output schema and the database’s columns

Description is the instructionedit the words, change what the AI reads

Five label typestext, number, date, list and record

Record labels for tablesline items as rows, child labels as columns

Required and choicesmissing values flagged; values outside the list reported

Page, words, confidenceevery value keeps its source

Additive changenew fields any time; nothing downstream breaks

An intake agent receives the collection’s fields as its output schema, so one definition drives extraction, review screens, views and the API. Read AI data modeling and the Academy lesson on taxonomy, lifecycle, tags and events.

Questions people ask

Structured data extraction with LLMs: common questions.

The technology behind itAI data modelingFrom documents to typed, searchable records

How do you do structured data extraction with an LLM?

Write a JSON Schema for the fields you need, with a clear description for each. Send the document and the schema with structured outputs switched on, then validate the values in code, verify each value’s quoted evidence, and send failures or low-confidence values to a person.

What is the difference between JSON mode and structured outputs?

JSON mode guarantees the response is valid JSON. Structured outputs guarantee it also matches your JSON Schema — required keys, types and enums. Neither guarantees the values are correct.

Do structured outputs stop hallucinations?

No. They constrain shape, not truth. A model can return a correctly formatted date that is the wrong date. Evidence fields, validation rules and review catch those.

Which LLMs support structured outputs?