AI data modeling

AI data modeling: from pages to typed records.

One model that the AI reads as instructions, the database enforces as a schema, and people read as plain words.

On Botminds the data model runs the system. Labels tell the extraction agent what to find, typed schemas tell the database what to keep, formulas keep derived values current as people edit, and every passage is searchable by meaning and by keyword — across millions of documents and records.

Book a demo Read the deep dive

Unstructured

InvoicePDF · 3 pages

Email“revised payment terms”

Delivery notescanned image

The model

SupplierText

Legal name of the seller, as printed in the header

Invoice dateDate

Date of issue — not the due date

Total dueNumber

Amount payable, tax included

Line itemsRecord

One row per product line

Typed record

supplierNorthwind Supply Ltdp.1

invoice_date2026-09-14p.1

total_due18,420.00p.3

line_items7 rowsp.2–3

each extracted value cites its page

The description is the instruction · the schema is the contract

What is AI data modeling?

AI data modeling is designing the structure of your data so that AI can both fill it and use it: which records exist, which fields each holds, what type and meaning each field has, and how records relate. For AI the model does double duty — it is the schema the database enforces and the instruction the extraction model follows. Done well, it turns unstructured data into structured, typed, searchable records at any volume.

Scales out on KubernetesServices and workers add capacity as work arrives, and give it back when it is done.

Built for millionsMillions of documents, records and agent runs — the indexes and workers grow with them.

Any cloud, any regionAzure, AWS, Google Cloud or on-premises, in the region your data must stay in.

Always onRedundant services, rolling releases and continuous backups, for mission-critical work.

Secure and governedSingle sign-on, encryption, least privilege and a sealed audit trail.

Why it matters

AI data modeling is not the data modeling you know.

Classic data modeling starts from tables that already exist. AI data modeling starts a step earlier — with contracts, emails, statements and scans — and its first job is to say what should be read out of them.

	Classic data modeling	AI data modeling
Starts from	Tables that already exist	Documents, messages and pages
Read by	The database and the analyst	The database, the analyst and the AI that fills it
A field’s definition	A name and a type	A name, a type and a plain-words instruction
A value	Whatever was loaded	Its source page, its words and a confidence score
Change	A migration, planned by engineers	Additive: new fields appear without breaking what was read
Derived values	Computed in a report, later	Recomputed the moment a person corrects an input

Where data lives

Four kinds of collection, and who does the work in each.

Everything on the platform lives in a collection. Its purpose decides what it holds and how data enters; its operating model decides who does the work.

Processing

Holds: Documents and the values read from them
Data enters by: Upload, email, connectors, API
Used for: Work that moves through review stages to a decision

Structured

Holds: Typed rows in a real database table
Data enters by: Forms, file import, sync from SQL, a sheet or an API
Used for: Reference data, registers, the output of extraction

Knowledge

Holds: Documents split into passages, embedded and indexed
Data enters by: Upload and connectors
Used for: What agents search and cite when they answer

Evaluation

Holds: Inputs paired with the answers that are right
Data enters by: Curated by people, or imported
Used for: Golden datasets that test an agent’s changes

Run by

Agentsagents and workflows read, extract, classify, enrich and index

Peoplea work queue: each record is a task for a person, assigned and tracked

Processing → structured

Values read from invoices land as typed rows in a register finance can sort, filter and report on.

Knowledge → agents

Policies and manuals are indexed so an agent cites them while it checks a document. See agentic RAG.

Evaluation → every change

Golden datasets score an agent’s changes against a pinned baseline. See AI agent evaluation.

Taxonomy management

Taxonomy management: the description is the instruction.

A taxonomy is the list of things to read from a document — its labels. A label’s description is the text the extraction agent is given, word for word, so editing a description is how you change what the AI does.

1 · The label

NamePayment terms

TypeList Net 30 · Net 60 · Due on receipt

RequiredYes

Description“The payment terms agreed for this invoice. Use the terms printed on the invoice; if an email in the same file changes them, use the email’s.”

2 · The agent’s request

Every label of the collection becomes one typed field the extraction agent must return:

name what to call it
type what shape the answer takes
description how to find it — read word for word
choices the values to pick from

3 · The valueNet 60

a listed choiceconfidence 0.94

Read fromEmail, page 1“…we agree to move you to net 60 from October…”

types

Five label types

Text, number, date, list and record. Currency and percentage are numbers; a dropdown is a list; a table is a record.

tables

Record labels

Rows — invoice lines, a payment schedule, a financial statement — with child labels as the columns.

rules

Required and choices

A missing required value is flagged; values outside a new list of choices are reported for review.

structure

Nested and grouped

Labels nest and group into categories, so a model of hundreds of fields stays readable.

Field	A weak description	A strong description
Invoice date	“Invoice date.”	“The date the invoice was issued, printed near the invoice number. Not the due date or the delivery date.”
Total due	“Total.”	“The amount payable, tax included, as a number. If a credit note is attached, use the invoice’s own total.”

Schema inference

Typed records and schema inference: drop in a file, get a schema.

Structured collections keep typed rows in real database tables. Design the schema field by field, or upload a spreadsheet and let the platform propose one — reviewed by a person before anything is written.

vendors.xlsx · sampled per column

vendor_id1041 · 1042 · 1187Whole number

credit_limit25000.00 · 7500.5Decimal

activeyes · no · yesYes / no

onboarded14/03/2024 · 2025-01-09Date and time

notesa 340-character comment…Long text

vendor_nameHarbor Freight LinesText

Tried in this order, per column

Whole numbers?Whole number
Numbers with decimals?Decimal
true / false / yes / no?Yes / no
Dates, in any common format?Date and time
Long passages?Long text
Anything elseText

Chosen by a person, never guessed: large whole number, choice list, link to another record.

ReviewCommitStream inKeep in sync

types

Nine field types

Text, long text, whole, large whole and exact decimal numbers, date and time, yes / no, choice, and a link to a record in another collection.

change

Additive schemas

Add fields at any time. Column names are fixed at the first commit, so every report and agent that reads them keeps working.

import

Imports that keep every row

Rows stream in batch after batch. A row that needs fixing is parked with its reason and original values, ready to fix and resend.

sync

Kept in sync

Sync from a SQL database, a spreadsheet in a drive or an API, on the schedule you set.

history

Every change on the record

Old and new value, who made the change, on whose behalf and why — a reason a collection can require.

Relationships

Parent and child collections: invoices and their lines.

Real data nests. An invoice has lines; a loan has collateral; a policy has endorsements. A child collection takes its records from a parent, so each line is a record of its own that still knows where it came from.

Parent
An invoice is read; its header fields are filled and its line table extracted as a record label.
Derive
Each line becomes a record in the child collection, fields mapped from the parent.
Work the child
Lines are filtered, matched to purchase orders and reviewed on their own.
One direction
Records flow parent to child, so the model stays free of loops.

Record labels for the page

When rows matter alongside their document — a balance sheet — review them in place.

Child collections for the work

When rows have a life of their own — matching, approval, reporting — query and work each one.

Links between registers

A link field joins a vendor register and a payments table into one model.

Derivation formulas

Derived values that keep up with the people correcting them.

Totals, ratios and checks depend on other values. When a reviewer corrects an input, everything that depends on it changes at once — and a recompute that started earlier never overwrites the correction.

1A person editsRevenue changes from 4.20m to 4.35m

2Change eventraised in the same save — no polling, no nightly job

3Recompute, in one of two lanes

Formula laneCompiled arithmeticanswers on screen at once

AI laneOne scoped model callfor rules written in words — and checks the formula

4The fenceHas a person edited this value since the recompute started?

yes → the stale result is droppedno → write it

5On screenGross margin and every dependent value repaint in place

Two lanes

Arithmetic runs as a compiled formula. A rule in words — “the covenant is breached if…” — runs as one small, scoped model call. One change event triggers both.

Checked speed

The formula’s answer shows at once while the AI lane checks it. Once they agree, the formula carries the field; on any disagreement the AI lane leads again.

People win

Each recompute carries the time it started. If a person has edited the value since, the older result is set aside.

Unstructured data processing

Knowledge collections: chunk, embed, index — then prove it.

Some data is meant to be consulted, not extracted: policies, manuals, past decisions. Knowledge collections turn documents into passages an agent can search, and a document is marked ready once the index has been checked.

Read
Text comes out of every file, with optical character recognition for scans.
Chunk
Text is split into overlapping passages, with page summaries when you want them.
Embed
Each passage becomes a vector from the collection’s embedding model.
Index
Passages go into a shared or dedicated index, with the expected count recorded first.
Probe
An independent check counts what is really in the index. Then the document is ready.

Meaning and keyword

Vector search finds the passage that answers the question in other words; switch on keyword search per agent for clause numbers, accounts and names.

Inside the read rules

When a person asks, search draws on the collections that person may read. See AI governance.

Clean re-index

Each re-index pass is numbered and older passages are removed, so a retry leaves no duplicates — and a failure says why.

Data at enterprise scale

AI data modeling at enterprise scale: millions of documents and records.

A data model earns its keep at volume. Intake and extraction workers scale out on durable queues, typed records sit in indexed tables, each search index carries its own shard and replica settings, and recompute runs on every runtime replica — on any cloud and in any region.

1 · Arrives

Uploads and emailone file or a whole archive

Connectorsdrives, mailboxes, feeds

SyncSQL, sheets and APIs, on a schedule

API and agentsrecords written by your systems

2 · Queued

Durable queuesabsorb the burst when a million documents land at once — work waits safely, nothing is lost

3 · Worked

Workers scale out on queue depth

Readevery page, scan and table

Typeeach value checked against the schema

Embedpassages turned into vectors

Recomputederived values; rapid edits become one

4 · Kept

Collectionsmillions of typed records in real tables, with indexed columns

Search indexeseach with its own shard and replica settings — a project can keep an index of its own

Every value, traceableits page, its words and its history travel with it

Azure · AWS · Google Cloud · on-premisesany regionrolling releasescontinuous backups

Collections with millions of records

Typed rows live in real tables with indexed columns, so filters, sorts and totals stay quick as registers grow.

Imports that stream

Loads arrive batch after batch and syncs run on schedule; sweeps resume where they stopped, and every bad row is kept.

Indexes set per project

Indexing workers scale out on the queue. Each search index is created with its own shard and replica settings, and a project can keep its knowledge in an index of its own: knowledge bases of millions of files, with no fixed ceiling.

Formulas kept current

Change events are kept durably and drained by every runtime replica; rapid edits to one document collapse into one recompute.

A semantic layer for AI

One semantic layer for people, agents and systems.

A semantic layer gives data business meaning once. On Botminds the labels and fields are that layer: the names reviewers see, the fields agents return, the columns of every view and the parameters of every API call are the same words.

Views

Saved grids — columns, filters, sort and who may see them — plus summary and score views.

Datasheets

Reference tables agents look things up in: SQL and NoSQL databases, spreadsheets, APIs, views and forms.

Agents

An intake agent receives the collection’s fields as its output schema. One definition, many uses.

APIs and pages

Your systems read the same fields through the API; pages show them on cards. See composable UI.

One record — the same names on screen, in agents and in the API

{
  "collection": "Vendor invoices",
  "record": "INV-2207",
  "supplier": {
    "value": "Northwind Supply Ltd",
    "page": 1,
    "confidence": 0.97
  },
  "payment_terms": {
    "value": "Net 60",
    "source": "email"
  },
  "total_due": {
    "value": 18420.00,
    "page": 3
  }
}

Side by side

AI data modeling tools compared.

Teams turning unstructured data into structured data usually reach for one of three things. They solve different parts of the problem.

	BI and warehouse modeling	Prompt-and-parse scripts	Botminds AI data modeling
Reads documents	Tables, after loading	Yes, in code	Yes, from a model people can read
Where instructions live	Not applicable	Prompts in a code base	Each field’s description, edited on screen
Types enforced	Yes	If someone writes the checks	At the label and in the database
Source of each value	Not applicable	Usually lost	The page and words it was read from
Schema change	Migrations	Edit the code, re-run	Additive — new fields any time, nothing downstream breaks
Derived values	In the report	Re-run the script	Recomputed on each edit; people’s edits win
Search for agents	Not its concern	A separate vector store to build	Meaning and keyword search, inside read rules
Volume	Warehouse-scale tables	One script, one machine	Millions of documents and records, scaling out

Where it fits. Botminds feeds your warehouse clean, typed, sourced records — and gives agents a model they can act on. Extraction itself is covered on the Botminds IDP page.

Questions people ask

AI data modeling, answered.

More from the platform: the full architecture, and how security works at Botminds.

Botminds IDP Agentic RAG Human in the loop

What is AI data modeling?

AI data modeling is designing the structure of data so AI can fill it and use it: the records, their fields, each field’s type and meaning, and how records relate. On Botminds each field’s description is also the instruction the extraction agent follows.

How do you turn unstructured data into structured data?

Define the fields you need with plain-words descriptions, point an extraction agent at the documents, and write the values into a typed collection. Each value keeps its page, its words and a confidence score. See Botminds IDP.

What is structured data extraction?

Reading named, typed values out of documents — dates, amounts, parties, tables — rather than summarising them. The output is records a database can store and a system can act on, each traceable to its source.

What is taxonomy management in AI?

Keeping the list of things to read from documents accurate, well described and well organised as needs change. On Botminds labels have types, descriptions, required flags and choices, and a description change changes extraction directly.

What is a semantic layer for AI?

A layer that defines what data means once, so every consumer reads it the same way — agents included. On Botminds the labels and fields serve as that layer across extraction, review screens, views, agent outputs and the API.

Can AI infer a schema from a file?

Yes. Upload a spreadsheet or CSV and the platform samples each column and proposes a type — whole number, decimal, yes/no, date, long text or text. A person reviews and commits the proposal before anything is written.

What happens to rows that don’t match the schema?

They are parked with the reason and their original values, so they can be fixed and resent. Every other row lands, batch after batch.

How much data can an AI data model handle?

Collections hold millions of typed records, knowledge bases hold millions of files, and each search index carries its own shard and replica settings. Intake workers scale on queue depth, on Azure, AWS, Google Cloud or on-premises.

How do derived values stay correct when someone edits a field?

An edit raises a change event and every dependent value is recomputed — by a compiled formula, or a small model call for rules in words. A person’s newer edit always wins over an older recompute.

Does Botminds support vector search and keyword search?

Yes. Knowledge collections are split into passages, embedded and indexed; agents search by meaning, and by keyword once you switch it on. When a person asks, both respect that person’s read rules. See agentic RAG.

Model your data once.

Bring a stack of your documents and the record you wish you had. We’ll build the model with you and fill it on your own data.

Book a demo See the whole platform