Automated data lineage is the machine reconstruction of how a data element moves and changes from the system that captures it to the report that shows it, built from metadata, code and logs rather than from interviews and hand-drawn diagrams. For BCBS 239 the bar is set by the ECB, which expects "complete and up-to-date data lineages on data attribute level (starting from data capture and including extraction, transformation and loading)" for in-scope risk indicators and their critical data elements (ECB Guide on RDARR, May 2024, section 3.4). Most automated lineage stops where the data catalog stops, which is usually at the first spreadsheet or manual upload, and that is where risk figures tend to go wrong. Botminds' BCBS 239 Compliance rebuilds lineage from the catalog, the ETL scheduler's job logs, report specifications and file-share access logs, so it finds the workbooks a catalog lists only as a manual upload, and it hands every hop to a person to confirm.
The worked example is a fictional sample bank, Corvane National Bank, and one figure on its September 2026 Board Risk Report: commercial real estate exposure of $14.86 billion.
Six agents along one lineage, and the people who decide
Agent
Lineage agentRebuilds the figure’s path from the catalog, scheduler logs, report specifications and file-share access logs: seven hops to the loan origination system.
Never: confirms a hop; below 0.90 confidence a hop is marked “needs evidence”
Agent
Accuracy agentReconciles each hop to the hop before and splits any difference to the row, tab and cell, with a confidence on each part.
Never: adjusts a figure or a report
Agent
Timeliness agentReplays the last stress drill from scheduler and report-platform logs and times each hop against the target.
Never: re-runs a production job
Agent
Spreadsheet agentFinds end-user computing in the lineage and risk-rates each workbook on materiality, complexity, controls and change history.
Never: edits or locks a workbook
A person decides
Risk Data StewardConfirms or corrects each hop and decides each finding, with a reason code.
Agent
AssessorDrafts each principle’s rating from the findings, citing each finding and its hop, and proposes remediation with an owner and a date.
Never: rates finally, or opens an issue without a steward
Agent
Change agentWatches releases, schema and job changes and report templates; re-traces only the elements a change touches.
Never: changes a rating; pauses and says which system is silent
A person decides
Head of Risk ReportingCountersigns the lineage and decides each rating; the Chief Data Officer approves and internal audit validates.
What is automated data lineage? The ECB's definition of lineage, and what it is for
Lineage, in the ECB's words, "is information about the movement and transformation of data from front (capture) to end". The Guide lists six things it should let a bank do: "understand if data quality controls are sufficient and well placed in the data flow", "identify interconnections between data definitions and taxonomies", check that loaded or transformed fields "are still in line with the reporting requirements and definitions", "support the identification of data points needed for specific ad hoc reporting needs", trace back the source of a data quality incident "in a timely manner", and "allow traceability for (external) validation" (ECB Guide, footnote 23).
Automated lineage is lineage produced by software from evidence the systems already hold. There are three main sources of that evidence: metadata (what a data catalog has scanned and registered), code (the SQL, scripts and job definitions that move and transform data) and logs (what actually ran, read and wrote, and when). Each sees part of the path. Data lineage tools compared sets the methods side by side on one element; this page is about doing it with agents, and about the limits that make agents safe to use.
What does BCBS 239 require for data lineage?
The 2013 principles never use the word lineage. They ask for what lineage proves: integrated "data taxonomies and architecture across the banking group" with metadata, reconciliation of risk data with its sources, documentation of every aggregation process, automated or manual, and an explanation of any manual workaround. The word arrives with supervision. The ECB's 2016 thematic review added "a “data lineage” exercise" on two selected data points, aimed at assessing "credit institutions’ ability to manage the data life cycle, from the original data source through the process and application chain, taking account of the controls applied" (ECB Thematic Review, 2018). The Basel Committee's 2023 progress report found several banks still without a common taxonomy or complete data lineage.
The ECB's Guide is now the most specific text. Lineage is expected at "data attribute level", from capture through extraction, transformation and loading, for the key risk indicators and critical data elements in scope. Data owners are expected to manage the lineage metadata for what they own. The validation function's assessments should cover "all components of the RDARR processes (e.g. IT infrastructure, data lineage and data taxonomy)". And the scope includes supervisory reports, so the lineage of a FINREP or COREP figure is in, not just the board pack.
Can data lineage be automated?
Largely, yes, and the research is clear about where it is hard. Column-level lineage from SQL can be inferred by parsing queries: the authors of LINEAGEX, presented at the IEEE International Conference on Data Engineering in 2025, describe a library that "infers column level lineage from SQL queries" and note that "existing systems introduce overheads by integration with query execution or fail to achieve satisfying accuracy for column lineage" (Zhang, Miao and Wang, 2025). A 2025 preprint, not yet peer reviewed, built a benchmark of "1,700 manually annotated lineages from real-world industrial scripts" across multiple programming languages and found that extraction quality "scales with model size and the sophistication of prompting techniques" (Yin and others, arXiv 2508.07179).
Two things remain hard for any method. A step that leaves no code and no log, such as a figure re-keyed from an email, cannot be traced; it can only be flagged. And automation that is wrong with confidence is worse than none, because a reviewer trusts it. That is why the Basel Committee's 2023 report, which noted banks' hopes that technology could "automate the discovery and visualisation of data flows, and maintain data lineage", also warned that many banks still lacked the quality data any such project needs (BCBS, November 2023). The design answer is a confidence on every inferred hop, a threshold below which nothing is proposed as confirmed, and a person who confirms.
What the lineage of one sample figure looked like once rebuilt
7hops
from the board page to the loan origination system
1
hop the data catalog lists only as a “manual upload”
0.94
confidence of the log join that found it
0.90
threshold below which a hop is marked “needs evidence”
How do you rebuild data lineage from logs?
By joining what systems record about the same file or table at the same time. In the sample bank, the data catalog showed the risk mart loading a "manual upload" and nothing more. The file-share access log and the scheduler log together told the rest: an analyst read the warehouse export of the staging table, wrote a workbook to an upload folder that evening, and the mart's load job read that same file at 02:14 the next morning. Joining the three by file name, time and account gave the missing hop, at 0.94 confidence.
How the missing hop was found: three log lines, joined
The join matters because of what sat in the workbook. Reconciled against the staging table, it added $212.0 million to the board's figure: $187.4 million of participations sold to other banks but counted as held, and $24.6 million of manual overrides with no comment. A catalog-only lineage would have drawn an arrow from staging to the mart and missed both. The Lineage agent proposed the hop; the steward confirmed it with the log lines and both amounts in view.
The log trail beside the reconciliation it made possible
What is attribute-level lineage, and does automation reach it?
Attribute-level lineage traces the individual fields that carry a figure, not just the tables and files. The ECB draws the distinction: "data attributes are often stored as a column in a table and are used in the technical mapping to calculate key risk indicators, whereas data elements impact the specific indicator values" (ECB Guide, footnote 24). A table-level arrow from the loan system to the workbook says the data moved; an attribute-level trace says which property-type code, which sold-percentage column and which commitment amount moved, and what was done to each.
The sample bank shows why the level matters. On October 12, 2026 release 26.4 of the loan origination system renamed one field, prop_type_cd, to property_type_code. A table-level lineage would show no change at all. At attribute level, 14 critical data elements read that field; the Change agent re-traced those 14 and found 13 unchanged and one, office commercial real estate exposure, changed: 112 office loans now fell out of the workbook's mapping tab, understating the figure until the mapping is fixed.
A release re-traces only the elements it touches, and a silent system pauses the re-trace
The six BCBS 239 agents, and what each may never do
An agent in a BCBS 239 process is only as safe as its limits. Each of the six agents in BCBS 239 Compliance has a job it does and actions it is not allowed to take, enforced in the product rather than promised in a policy, and every agent's prompt and model are versioned like a model, with a confidence threshold, an acceptance rate and overrides counted by reason code.
The six agents in BCBS 239 Compliance, their limits, and who decides
| Agent | Does | Never | Who decides |
|---|---|---|---|
| Lineage | Rebuilds each element’s path from catalog, scheduler, report specifications and file-share logs | Confirms a hop; proposes one below 0.90 confidence | Steward confirms; Head of Risk Reporting countersigns |
| Accuracy | Reconciles every hop to the one before; splits differences to row and cell | Adjusts a figure or report; raises differences under materiality | Steward decides the finding |
| Timeliness | Replays the last stress drill from logs; times each hop | Re-runs a production job | Head of Risk Reporting, through the Principle 5 rating |
| Spreadsheet | Finds workbooks in lineage; rates materiality, complexity, controls, change history | Edits or locks a workbook | Data owner attests the controls |
| Assessor | Drafts each principle’s rating and rationale; proposes remediation | Rates finally; opens an issue without a steward | Head of Risk Reporting decides; CDO approves; internal audit validates |
| Change | Watches releases, schema, jobs and templates; re-traces what changed | Changes a rating; marks a silent system unchanged | Steward confirms the re-trace |
Model risk management does not settle whether such agents are "models". The U.S. agencies' revised guidance of April 2026 sets generative and agentic AI aside as too new and too fast-changing to fall within it (Federal Reserve SR 26-2; the passage is quoted in customer risk rating model). Banks still need a control framework for them; treating each agent's prompt and model as versioned, approved and monitored is the conservative choice until supervisors say more.
What to ask before buying automated data lineage for BCBS 239
Ask for a demonstration on your own figure, and ask these questions of what you see. The timeliness principle guide covers the drill replay in depth; data owner vs data steward covers who should confirm what the agents find.
Eight questions for any automated data lineage product, asked on one of your board figures
- Where does it start?
From the figure on the report, or from what the catalog has scanned?
- Which evidence does it read?
Metadata, code, logs, or all three; and which hops each one found.
- Does it see file shares?
Ask it to find a workbook that the catalog lists only as an upload.
- Does it trace to the attribute?
A renamed field should re-open exactly the elements that read it.
- Is every inferred hop scored?
And what happens below the threshold: proposed, or flagged for evidence?
- Who confirms a hop?
A named person with a reason, never the software.
- What does it do when a system is silent?
Pause and say so, or carry the old lineage forward as if nothing changed?
- Can it write anywhere?
It should not need to: read-only access, masked loan-level rows, an audit trail.
Where agent-built lineage fits, and where it does not
It fits where the systems leave evidence: a catalog, a scheduler with job logs, report specifications and file shares with access logs. It does not replace the catalog, which remains the register of definitions and owners; the agents read it. It does not reach a step that leaves no trace, and it says so: such a hop is marked "needs evidence" for a steward to document. And it does not make lineage a one-off project: a figure's path changes with every release, so the value is in re-tracing the elements a change touches, the same day, rather than redrawing everything once a year. The BCBS 239 Compliance page shows the agents working on the sample bank's full set of 312 critical data elements.
Questions people ask
What is automated data lineage?
Software that reconstructs how data moves and changes from its source to a report, using catalog metadata, the code that transforms it and the logs of what actually ran, instead of interviews and diagrams. For BCBS 239 it should reach the attribute level and the spreadsheets in the path.
What is BCBS 239 data lineage?
The traceable path of each key risk indicator and critical data element from capture to report. The ECB expects it to be complete, up to date and at data attribute level, covering extraction, transformation and loading, so that controls, ad hoc requests and incidents can be traced and validated.
Can AI agents do BCBS 239 compliance?
They can do the evidence work: trace lineage, reconcile hops, time drills, rate spreadsheets and draft ratings with citations. They should not confirm lineage, adjust figures or set final ratings; those stay with named people, and the agents' prompts and models should be versioned and approved.
How accurate is automated lineage?
It depends on the evidence and the method. Research on SQL parsing reports high coverage for column lineage, and work on multilingual scripts finds quality improving with model size, but no method can trace a step that leaves no code or log. A confidence score on each inferred hop and human confirmation are the safeguards.
Does automated data lineage replace a data catalog?
No. The catalog holds definitions, owners and the lineage it can scan; agent-built lineage reads it and fills the gaps from scheduler and file-share logs. The two work together: the catalog stays the register, and the traced lineage shows what actually runs.
How often should data lineage be refreshed for BCBS 239?
Whenever something in the path changes, not only at the annual assessment. The ECB expects lineage to be up to date; re-tracing only the elements a release or schema change touches keeps that affordable.