Ingestion & connectors

In one line. How documents get into a collection — upload, crawl, feed, or form — and how processed results get out through Export. You’ll be able to. Wire a reusable connection to an external system, schedule a job that pulls documents into a collection, run a one-shot upload, watch a run, and set up an export. Where this lives. Studio ▸ Connector — the Ingestion, Output, and Input Form screens.

Why it matters

A solution is only as good as the documents flowing through it. The Connector section of Studio is the loading dock: it is where you connect to SharePoint, a Google/internal Drive, a mailbox, or a public data feed; schedule pulls; drop in one-off uploads; and, on the way out, configure how finished documents are exported back to your systems. Everything here feeds the collections you built in Collections & schema. Get the model right once and you rarely think about it again — documents just arrive, show a “Processing…” chip, and fill in.

The model: Connector vs Job vs Run

This is the single idea that makes the whole section make sense. From Core concepts:

Term	What it is	Glossary role	Example
Connector	A tested connection — credentials + a target system	identity	“Our Finance SharePoint”
Job	A scoped, scheduled fetch — a connector + a scope + a schedule + a target collection	scope	“The /Invoices library, every 6 hours, into Vendor Invoices”
Run	One execution of a job (or an upload) — the history record	—	“Tuesday 06:00 run: 14 new, 2 duplicates”

You build a connector once and reuse it across many jobs. A job is where you say which documents, how often, and where they land. A run is what you inspect when you want to know what happened.

Under the hood. When a document arrives — by upload, crawl, feed, or form — the backend creates the document row immediately with status Processing… (so it appears in the list at once), stores the raw bytes, and hands off to the orchestrator. The orchestrator runs the collection’s ingestion XFlow (fetch the bytes, assemble the document, write it to Postgres), then extraction agents fill in the schema fields. A single progress row drives the chip you see climbing to 100%. For a Knowledge collection, the intake instead chunks, embeds, and indexes the document for search (standalone image files go through vision page summaries when Image mode is on, else fail with a clear “no readable text” reason) — watch it on the collection’s Index Health tab (Search, indexes & vectorization). The chain stalls at “queued” if the background worker isn’t running — the usual cause of a doc stuck at 0%.

The default Ingestion screen. The modern Ingestion screen this page teaches is Sources v3 — the Connector / Job / Run model. It is simply the Ingestion screen in Studio: the rail entry Studio ▸ Connector ▸ Ingestion is hardwired to the connector/sources-v3 route; there is no flag to flip to reach it. The legacy single-source wizard still exists but is deep-link-only (see below). The separately-scoped ingestionV3 flag, default on, governs the consumer document-list + Data upload button — not this Studio rail entry.

The Ingestion screen (Sources v3)

Studio ▸ Connector ▸ Ingestion. One screen, three tabs. It is also embedded inside a collection’s own Ingestion tab, where one extra Add-Data settings toolbar button appears.

┌─ Ingestion ─────────────────────────────────────────────────────────────────┐
│                        [ Upload ] [ + Connector ] [ + Job ]  (Add-Data       │
│                                                              settings*)      │
│  ┌──────────┬────────────┬────────┐                                          │
│  │  Jobs ▸  │ Connectors │  Runs  │   ← three tabs (Jobs is default)         │
│  └──────────┴────────────┴────────┘                                          │
│                                                                              │
│  Invoices nightly   on Finance SP · cron 0 6 * * * · next 06:00 · active     │
│        [ Run now ] [ Pause ] [ edit ] [ delete ]                             │
│  Contracts drip     on Botminds Drive · every 6h · next 12:00 · active       │
│        [ Run now ] [ Pause ] [ edit ] [ delete ]                             │
│  ──────────────────────────────────────────────────────────────────────     │
│  * Add-Data settings appears only when embedded in a collection's tab.      │
└──────────────────────────────────────────────────────────────────────────────┘

• Toolbar (top-right): Upload (one-shot drop), + Connector (the connection wizard), + Job (the primary action — schedule a pull). When embedded in a collection, an Add-Data settings button also appears.
• Tabs: Jobs (scheduled pulls — the default tab), Connectors (your saved connections), Runs (execution history). The body shows Loading… while fetching.
• Each row carries a kind icon: SharePoint, Email, Drive, Public Feed, or a generic package icon for anything else.

What every control does

Tab	Row shows	Row actions
Jobs	kind icon + job name + “on <connector> · <schedule> · next <date> · active/paused”	Run now, Pause/Resume, edit, delete (confirms; run history is preserved)
Connectors	icon + name + “<kind> · active/paused · test OK/failed/never · used by N jobs”	Reconnect (only when the last test failed — SharePoint only today), Test (live test + latency), Pause/Resume, delete (disabled with a tooltip if jobs still use it — delete the jobs first)
Runs	When · Trigger · Status · funnel counters	row click opens the Run detail dialog

Empty states. Jobs: “No jobs yet · Click + Job to schedule one. (You’ll need a Connector first.)” — Connectors: “No connectors yet”. The order matters: a connector must exist before a job can use it.

The Runs tab and Run detail

The Runs table tracks each execution with a funnel of counters — the eight stages a batch of items passes through:

Enum (enumerated) → Fetch (downloaded) → New → Upd (updated) → Dup (duplicate, skipped) → Disp (dispatched to pipeline) → Merge → Fail.

Status colours: completed is green, partial amber, failed red. Click a row to open the Run detail dialog (1000px): pipeline-summary chips (Enumerate → Fetch → New → Updated → Duplicate → Dispatch → Merge), the status + failed count + last error, then a per-item event log (When · Stage · Item · Status · Message) so you can see exactly which file failed and why.

Tip. Dup climbing on a recurring job is normal and good — it means dedup is working and you are not re-processing files you already have.

The Add-Connector wizard

+ Connector opens a 900px wizard. There are six kinds, and the number of steps varies by kind: SharePoint has 3 steps (kind → auth pattern → form); every other kind has 2 steps (kind → form — the auth-pattern step is skipped). SharePoint’s auth-pattern step offers two tiles: Quick (connect with Microsoft — SMB / pilots) and Enterprise (use your own Entra app — enterprise IT).

Step 1 is a tile grid of the six kinds:

Kind	Tile	Use it for
SharePoint	“Pull docs from a SharePoint site”	Microsoft 365 document libraries
Email (IMAP)	“Pull docs from a mailbox”	Gmail / Outlook / generic IMAP inboxes
Botminds Drive	“Pull docs from this project’s internal drive. No setup needed”	Files already inside the project’s Drive
Public Feed	“Pull from public gov data sources (SEC, CFPB, Federal Register)”	Curated public data feeds
Datasheet	“Re-ingest rows from a datasheet in this project as documents”	Turning Datasheet rows into documents
Azure Blob Storage	“Ingest from an Azure Blob container”	Files in your own Azure storage

The per-kind credential forms and their test rules are enumerated in the Connector catalog — this page teaches the shape; the catalog enumerates every field.

Behaviours to know

• Test-before-save. Email, Public Feed, Drive, Azure Blob, and SharePoint-Enterprise connectors cannot be saved until a live connection test passes (a platform invariant). Editing any field after a green test clears it — you must re-test. (Datasheet is the exception: it’s internal — just pick the datasheet and name it.)
• SharePoint Quick has no Save button — it completes through the Connect with Microsoft OAuth popup. A closed popup reads as “Sign-in cancelled”; a blocked popup shows a warning.
• Edit / Reconnect jumps straight to the form step, pre-filled — but secrets are never pre-filled; re-enter them. The stepper is hidden in edit mode.

The Add-Job wizard

+ Job is the primary action — this is where documents actually start flowing. Three steps (editing an existing job locks the connector, dropping it to two):

1. Pick a connector — choose from the project’s active connectors (radio rows with kind icon + test status + “used by N jobs”). No connectors? The wizard tells you to “Cancel and create a Connector first.” Hidden when editing.
2. Scope & metadata mapping — the Job name (required), then a kind-specific scope (the folder/library/mailbox/feed slice to pull), the Target collection (a real dropdown of your project’s collections — never free text; “Default” is always present), and optional Secondary XFlows (comma-separated pipelines that run after each item is registered). Scope fields per kind:
   • SharePoint — Document library (with Test Drive to resolve a name to a stable id), Folder path, File extensions, Recursive, Max docs/run, plus a “Pull metadata from SharePoint?” toggle that reveals a metadata-mapping grid (map a taxonomy field to a SharePoint column; Discover columns lists the available columns).
   • Email — Folder (default INBOX), Include attachments, Max docs/run.
   • Drive — Folder (read-only; Browse… opens a picker), Extensions filter, Recursive, “Explode structured rows” (Excel/CSV/JSON → one document per row, with an optional Unique ID column), Max docs/run.
   • Public Feed — fields rendered dynamically from the feed’s own shape.
   • Azure Blob — Container, Folder path (blob prefix), Extensions filter, Recursive.
   • Datasheet — Document granularity (one document per row / whole sheet), Max rows/run. New and edited rows are picked up automatically each run.
3. Schedule — choose On-demand only (you press Run now), Recurring (every N Seconds/Minutes/Hours/Days), or a Cron expression (+ timezone). Public feeds also offer preset chips (Hourly / Every 6h / Daily 9am UTC / Weekly Mon 9am / On-demand). A live “Next 5 runs” preview confirms your schedule before you save.

The final button reads Create job (or Save changes when editing).

Scheduling, drip, and dedup

• Scheduling is per-job. A scheduled job fires on its cadence; Run now triggers it immediately regardless of schedule.
• Drip — public feeds (and other recurring jobs) pull only new items each run rather than re-downloading the whole source; the feed’s drip behaviour is described on its tile.
• Dedup — each run reports Dup for items it has seen before, so a recurring job is safe to run often without reprocessing. For exploded structured rows, set a Unique ID column as the business key; leave it blank and the platform dedups on a content hash.

The Upload modal (one-shot)

The Upload toolbar button opens a 720px modal for ad-hoc files — no connector or job needed:

1. Drag-drop or browse multiple files. The list shows each file with its size, a remove control, and Clear all.
2. If multi-file grouping is on, a note explains the files are “grouped into one document set (first file primary).”
3. Choose the Target collection (unless it is pre-filled by context).
4. Submit opens a result panel — complete / partial / failed, “N of M registered, K failed”, the run id, and a per-file breakdown (docId on success, error on failure). Done closes.

Tip. Use Upload to smoke-test a new collection’s schema before you wire a recurring job — drop in five representative files, watch them process, fix the schema, then automate.

Legacy Ingestion (the old path)

Before Sources v3 there was a single-source wizard at route connector/sources — no longer wired to the Ingestion rail entry, reachable by deep link only. It bundled per-source config, an annotator, document-property metadata, and a crawl-job history page. It still works, but all new builds use v3 — the Connector/Job/Run model above is the going-forward shape. Don’t start new work in the legacy wizard.

Export — getting results out

Studio ▸ Connector ▸ Output (route connector/output). Ingestion is the loading dock; Export is the shipping dock — it pushes processed documents (with their extracted data, optionally split or bookmarked) out to Azure Blob storage or the Botminds Drive.

The screen is a standard list + details split: the left rail lists your exports, the right panel shows the selected export’s details — Type, Taxonomy (the fields included, as chips), Storage Type (Azure Blob or Botminds Drive), and Storage Details (account + container, or the drive URL), with edit/delete row actions. Empty state: “Setup an export to see more details” plus + Export.

The Add Export dialog

+ Export opens a dialog whose fields change based on the Export Type you pick — choose the type first, the rest follows. Always present: a Name, the Export Type, and the destination (External Sources: Azure Blob → Storage Name/Key/Container, or Botminds Drive → drive URL).

The notable toggles for a split/bookmark export type:

• Split Input Document, Split on Sequential Values, Split By Start Page — break one input PDF into several output files.
• Grouped Bookmark, First Value As Bookmark, Group By Label, Order Date — build bookmarks/structure in the output.
• Suffix for document name — append a value to output filenames.

And common toggles across types: Export current-job docs only, Add date-wise folders, Add document-wise folders, Export input document (ship the original alongside results), Delete input document from project (clean up after export). Save is disabled while the form is invalid. The full per-type field matrix is in the Connector catalog.

Input Form — capturing structured input directly

Studio ▸ Connector ▸ Input Form (route connector/form_template, visible when enabled for your project). Sometimes you don’t have documents at all — you want a person to type structured input that becomes a record. The Input Form builder defines those form templates, in the familiar list + details split. Empty state: “No Form. Add new form to process” plus + Add Form.

Every template has a Name, Description, and a Type:

• Create Source — the form creates documents/records. You build its Columns (a repeatable list): each column has a Name, a Type (Text / Multiline / Checkbox / Dropdown / Radio), a Label (mapped to a taxonomy field, grouped by taxonomy), and flags Required, Is Title, Is Template. Each column can carry rules (Field + Value conditions that show/validate it). A Use Excel shortcut lets you define columns from a spreadsheet instead. If a column is Is Template, a Templates section appears (Value + an Export Template + a Rendering Template).
• Publish Template — a seed/launch form rather than a record-creator. Its Sub Type is either Connector (pick an ingestion seed page) or Global Variables (pick variables to expose).

Submit with Create (or Update when editing). The full field/rule reference is in the Connector catalog.

Try it yourself

Goal: a recurring Drive-folder job that feeds a collection.

1. Open Studio ▸ Connector ▸ Ingestion — the rail lands you on Sources v3 directly.
2. Click + Connector, pick Botminds Drive. Give it a display name, click Test connection (it reports a sub-folder count), then Save connector. No credentials needed.
3. Click + Job. Step 1: select your new Drive connector.
4. Step 2: name it (e.g. “Contracts intake”), click Browse… and pick a folder, set Extensions to pdf, tick Recursive, and choose your Target collection.
5. Step 3: pick Recurring → every 6 Hours. Check the Next 5 runs preview, then Create job.
6. Back on the Jobs tab, click Run now on your job. Switch to the Runs tab and watch the funnel counters fill; click the run to open Run detail and confirm each file landed.

Faster variant: skip connectors entirely — click Upload, drag in a batch of PDFs, pick the target collection, Submit, and watch the result panel.

Recap

• Connector = identity (a tested connection), Job = scope (a scheduled, targeted pull), Run = history (one execution, with a funnel + event log).
• The modern Ingestion screen is Sources v3 — the default Studio rail entry, with Jobs / Connectors / Runs tabs; the legacy single-source wizard is deep-link-only.
• Build a connector first (six kinds; most require a passing test before save), then a job (connector → scope + mapping → schedule), or just Upload one-shot files.
• Scheduling is per-job; drip pulls only new items; dedup reports Dup and is keyed by a Unique ID column for exploded rows.
• Export (Connector ▸ Output) ships processed results out; Input Form (Connector ▸ Input Form) captures structured input directly.

Where to go next

• Drive & datasheet — the internal Drive a Drive connector reads from, and the SQL datasheet layer.
• Use case: invoice processing — a full ingest-to-export loop, end to end.
• Connector catalog — every connector kind’s fields, the full Export field matrix, and the Input Form column/rule reference.
• Collections & schema — the collections your jobs target.