Per tool
Pick the tools that change things — send, write, move, pay — and leave reading free.
Human in the loop
Human in the loop, where judgment counts.
People decide what matters, with the evidence open, while millions of values flow straight through.
On Botminds a person can stand between an agent and the actions you choose, two review stages can sign off a decision, and calibrated confidence decides which values need a person at all. Set the threshold yourself, or state the accuracy you need and let the platform derive and certify it from your own audited reviews.
An agent reads the documentEach field gets a confidence score, calibrated as evidence grows
0your threshold, per field1
below the thresholdPeople decide
L1 reviewL2 approval
with the page open at the exact wordsabove the thresholdStraight through
passesaudit sample
a blind audit sample can go to a person, scores hiddenEvery approve, edit and reject is evidence: the curve is refitted, and in target mode the threshold re-derived.
What is human in the loop AI?
Human in the loop (HITL) means people take part in an AI system’s decisions at set points: they approve an action before it runs, review a value before it is used, or sign off a decision before it is final. Done well, people see the work that needs them — and every judgment they make is recorded and makes the system’s confidence more accurate.
Scales out on KubernetesServices and workers add capacity as work arrives, and give it back when it is done.
Built for millionsMillions of documents, records and agent runs — the indexes and workers grow with them.
Any cloud, any regionAzure, AWS, Google Cloud or on-premises, in the region your data must stay in.
Always onRedundant services, rolling releases and continuous backups, for mission-critical work.
Secure and governedSingle sign-on, encryption, least privilege and a sealed audit trail.
Three positions
Human in the loop vs human on the loop vs out of the loop.
Where a person stands relative to the AI decides how much it can safely do. The right answer is the right position for each decision, chosen by risk and by measured accuracy — field by field, and reversible.
In the loop
AI proposesPerson decidesAction
Nothing happens until a person approves, edits or rejects.
FitsIrreversible, regulated or high-value decisions; low confidence.
On Botminds
- Approval on an agent’s tool
- Review and approval stages
- E-signatures
On the loop
AI actsPerson watchesSteps in
The AI acts within limits; a person monitors and intervenes on exceptions.
FitsHigh volume, measured accuracy, a sample checked.
On Botminds
- Straight-through above a threshold
- Blind audit sample
- Drift alarm that switches it off
Out of the loop
AI actsAction
Automated end to end. People design and test it, but are not in each decision.
FitsLow-risk, reversible work: sorting, routing, enrichment.
On Botminds
- Automated stages
- Evaluations when changes publish
- Audit trail of changes
Human in the loop for agents
Human in the loop for AI agents: approve, reject or edit.
An agent that proposes a change and waits for a yes does far more than an agent that reads. You choose which tools and sub-agent hand-offs wait for a person; when the agent reaches one, the run pauses in the conversation and asks.
An approval in a human in the loop agentic workflow
You
Send the payment reminders for invoices more than 30 days overdue.
Agent
Seven invoices qualify. Sending a reminder to each of the seven customers needs your approval. Approve, reject, or edit the list?
You
Edit: take out Kestrel Foods — they paid by wire this morning.
Agent
Done — six reminders sent. Kestrel Foods was left out. Your edit is kept with this conversation.
Three answers
Approve, reject, or edit the arguments before the tool runs. A sub-agent can be sent back with a note.
Plain words
The card says what will happen — in your wording, or one the platform drafts.
On the record
The paused step and its outcome stay with the conversation and the run.
Conversations and volume. Approvals put a person inside a live conversation with an agent. For unattended volume, review stages and work queues put people in the loop — the next sections. See agent harness.
The four-eyes principle
Four-eyes review: AI recommends, people decide in two stages.
The four-eyes principle says an important decision needs two people. Every new processing collection starts with two review stages already built: agents prepare a recommendation, a first reviewer checks it, an approver signs it off.
- 1Intakedocument arrives
- 2AI Processingagents extract and check
- 3AI Recommendationa proposed outcome
- 4L1 Reviewfirst reviewer
- 5L2 Approvalapprover — another person, if required
- ApprovedDeclined
agentspeople8 stages, seeded for every new processing collection
- L1 Review → Needs Infosomething is missing; once it arrives the document goes back through AI Processing
- L2 Approval → L1 Reviewthe approver sends it back with a question
- L1 or L2 → Declinedeither reviewer can decline
- AI Recommendation → Approved / Declinedauto-decide, when you switch it on: high confidence and zero flags
rule
A different person
Require that whoever moves a record out is not whoever moved it in — checked at every door, refusals recorded.
rule
Checks before a move
Required fields filled, no open errors, and the stage’s own check, before a record moves on.
evidence
Signatures
The approver signs with a meaning bound to exactly what they saw. See AI governance.
speed
Auto-decide
Switch it on per collection and a recommendation with high confidence and zero flags stands on its own.
Confidence threshold
A confidence threshold you can trust: calibration.
A score of 0.9 should mean right nine times in ten. Calibration makes it so: it maps each raw score to how often values like it were actually right, field by field, from your reviewers’ own decisions — and it sharpens as evidence grows.
actually rightconfidence the model gave · 0 → 1
actually rightconfidence the model gave · 0 → 1
Shape for illustration — your diagram is drawn from your own reviewed outcomes, field by field.
- Start from the score
A new field uses the model’s raw confidence while reviewers build its evidence.
- Fit a curve
As reviewed outcomes accumulate, the field gets its own curve — smooth while evidence is thin, stepwise once it is rich.
- Prove it
Each curve is tested on held-out folds and promoted when calibration error falls.
- Keep it current
Curves refit daily or on demand over a rolling window; the reliability diagram is there to read for every field.
Straight-through processing
Straight-through processing, decided field by field.
Straight-through processing is the share of work that needs no person. On Botminds you choose the fields, then set each threshold — or state the accuracy you need and let the platform derive the threshold from evidence.
- Choose the fields
Pick the fields that may go straight through. Every other field goes to a person.
- Set the bar
Give a threshold, or a target — “at least 99% right” — and the platform finds the lowest threshold it can certify at 95% confidence.
- Gate the document
A document passes when every chosen field passes; a critical field left empty sends it to review.
- Audit blind
Choose a share of passing documents to send to a person, shown without scores, so the reviewer is not anchored.
- Watch for drift
With an accuracy target set and audits running, if the audited error rate climbs a drift alarm switches straight-through off for the field until a person restores it.
A verdict per field
Each value is marked passed or needs review, with the threshold and calibration it was judged under.
Withdrawn on change
Edit the value, raise the threshold or break a validation rule, and its pass is withdrawn.
Certified from evidence
Values keep flowing to review until audited outcomes certify the target — and the screen says where each field stands.
The reviewer’s screen
The reviewer’s screen: the page open at the value.
Each value keeps the page and the words it was read from, so a click opens the source exactly there.
BorrowerHalvorsen Marine ASpassed
Facility amountUSD 12,500,000passed
Maturity date30 June 2031needs review
Interest margin2.15%needs review
Governing lawEnglish lawpassed
ApproveEditReject
page 37 of 41
“…the Final Maturity Date shall be 30 June 2031, being the date falling…”
the value’s words, highlighted on the page they came fromThe fields that need you
One switch shows the fields still waiting for review, with a count. Passed fields are marked as passed.
Evidence one click away
Click a value and the viewer opens its page with the words highlighted.
Three answers, all counted
Approve, edit or reject — logged with who gave it, and evidence for the next calibration.
Corrections as evidence
Every human in the loop correction is evidence.
Reviewer time is expensive, so every action is read as a verdict on the machine — carefully, so a convenience click is not mistaken for a correction.
| What the reviewer does | Why | Counted as |
|---|---|---|
| Approves the value | A person confirmed it | Right |
| Edits it, same value once normalised | Formatting is not an error | Right |
| Edits it to a different value | The machine’s value was not the answer | Wrong |
| Rejects, deletes or picks another source | The value should not stand | Wrong |
| A machine re-runs the extraction | A re-run is a machine action | Set aside |
| Reviews a blind audit sample | The sample the threshold did not filter | Right or wrong — the unbiased stream |
Human oversight at scale
Human in the loop at scale: hundreds of reviewers, millions of values.
Oversight works at volume when people see the right work first. Calibrated confidence sends the values that need judgment to work queues; a large review team works them in parallel; every decision is recorded, and straight-through grows as confidence is proven.
Millions of valueseach with a calibrated confidence
The gate, per fieldYour threshold — or your accuracy target“at least 99% right” becomes a threshold certified from audited reviews
Straight throughpasses without review · a blind sample can be audited
Work queuesthe values that need a person
A review team of hundreds
L1 reviewers
L2 approvers
least-loaded assignmentweakest firstfour eyes where it countsevery decision recorded
Straight-through grows as confidence is proven
straight throughreviewedshape for illustration · month by month
assignment
Queues for large teams
Each record goes to the least-loaded reviewer in the assigned role, or back to whoever had it at an earlier stage.
review queue
Weakest first
Documents waiting for review are sorted lowest confidence first, or in the order a team chooses.
tasks
People-run queues
A collection can be run by people alone, each record a task; every project has an inbox, and signature requests land there too.
action center
Faults kept apart
Failed documents and stuck work go to an action center with an owner and a next step, apart from content review.
Side by side
Human in the loop options compared.
Three ways teams usually put people in the loop. Each solves part of the problem.
| Interrupts in an agent framework | Classic review workflow tools | Botminds | |
|---|---|---|---|
| Approve an agent’s action | Built by your engineers | Not their concern | Any tool you choose; approve, reject or edit |
| Two-person review | Build it yourself | Yes | Seeded lifecycle; different-person rule per stage |
| Which items reach people | Whatever the code decides | A fixed score cut-off | A per-field threshold on calibrated confidence |
| Is the threshold right? | Unknown | Rarely measured | Set by you — or certified from blind audits and watched by a drift alarm |
| Evidence for the reviewer | Build it yourself | Often the whole document | The page and words behind each value |
| Corrections reused | Rarely | Rarely | Every review is calibration evidence |
| Volume | One agent at a time | Grows with headcount | Straight-through grows with proven confidence |
Start with everything in review. Begin with every field reviewed and the four-eyes lifecycle as it comes, then move fields to straight-through one at a time as the evidence earns it. See Botminds IDP for extraction itself.
Questions people ask
Human in the loop, answered.
More from the platform: the full architecture, and how security works at Botminds.
What is human in the loop in AI?
Human in the loop means people take part in an AI system’s decisions at set points — approving an action, reviewing a value or signing off a decision. On Botminds the points are configured per tool, per stage and per field.
What is the difference between human in the loop and human on the loop?
In the loop, nothing happens until a person decides. On the loop, the AI acts within limits while a person monitors and steps in on exceptions. Botminds does both: approvals and review stages for the first, straight-through with blind audits and drift alarms for the second.
What is a human in the loop agentic workflow?
An agentic workflow where agents do the work but pause for a person at the steps that matter — before sending, paying, moving or deleting — and continue with the person’s answer. On Botminds you choose which tools wait for approve, reject or edit.
What is the four-eyes principle?
A control that requires two people to agree before an important decision takes effect. Botminds seeds it as a lifecycle — L1 review, then L2 approval — and can require a different person at each step, recording any refused attempt.
What is straight-through processing?
The share of items processed with no person involved. On Botminds it is decided per field: a document goes straight through when every chosen field clears its threshold, and a blind sample of those still goes to a person.
How do you set a confidence threshold for human review?
From evidence. Calibrate the scores on reviewed outcomes, then pick the threshold that gives the accuracy you need — or state the accuracy, such as 99%, and let Botminds derive the lowest threshold it can certify at 95% confidence.
What is confidence calibration?
Adjusting a model’s scores so they match how often it is actually right: of values scored 0.9, about nine in ten should be correct. Botminds fits a curve per field from reviewers’ decisions, and it sharpens as evidence grows.
Does human in the loop slow everything down?
Not when it is targeted. Per-field thresholds send uncertain values to people, queues put the weakest first, and the reviewer sees the fields that need them with the source page open — while the rest flows straight through.
How does human review scale to millions of documents?
Calibrated confidence keeps routine values flowing straight through, work queues spread the rest across review teams of hundreds, and every decision becomes evidence that lets more fields go straight through over time.
Is human oversight required by the EU AI Act?
The EU AI Act requires effective human oversight for high-risk AI systems, including the ability to override or stop them. Botminds supplies controls that support it — see AI governance; how it applies depends on your use.
Learn the basics
Confidence calibrationReliability diagrams, ECE, and Platt, isotonic and temperature scaling→Human in the loop vs on the loopIn, on and out of the loop — and the autonomy spectrum between them→Straight-through processing (STP)What STP means in banking, insurance and documents — and how to measure it→All guidesEvery explainer, by topic→
People where they matter most.
Bring a process your team reviews today. We’ll show which fields can go straight through on your own data — and prove it.