Data lineage tools record how data moves from its sources to the reports that use it, and they come in four kinds, defined by the evidence they read: manual documentation, catalog scanners that harvest metadata, code parsers that read SQL and job definitions, and run-time tools that reconstruct the path from logs of what actually ran. For BCBS 239, the Basel principles for risk data aggregation and reporting, the test that matters is whether a tool can follow one board figure to its source at the level of the individual field, through every system and spreadsheet in between, and keep doing so as systems change. The work is hard because the riskiest hops are often the ones no tool registered: a workbook on a file share, a manual upload, a re-keyed number. Botminds' BCBS 239 Compliance is not a catalog; its agents read the catalog together with scheduler and file-share logs, then reconcile and time each hop, which is what an examiner tests.
The comparison uses one element from a fictional sample bank, Corvane National Bank: commercial real estate exposure of $14.86 billion on page 7 of its September 2026 Board Risk Report, with seven hops from the board page to the loan origination system. No vendor's product is named or rated; methods are.
Catalog lineage against log-reconstructed lineage, on the same board figure
Lineage from the data catalog: Staging table to risk mart to reporting platform: an unbroken arrow.
Lineage rebuilt from logs: Staging table to a workbook on a file share, then to the risk mart: an extra hop.
Lineage from the data catalog: Listed only as a “manual upload” into the mart load job.
Lineage rebuilt from logs: Found by joining the file-share read, the write and the job read by file name, time and account.
Lineage from the data catalog: Not shown: catalog lineage carries structure, not values.
Lineage rebuilt from logs: Each hop reconciled to the one before: $212.0M added at the workbook.
Lineage from the data catalog: Not shown.
Lineage rebuilt from logs: 60 of the 81.6 hours in the June stress drill sat at the workbook.
Lineage from the data catalog: Seen at the next scan, if the scanner covers that system.
Lineage rebuilt from logs: The 14 elements that read the field re-traced; one found changed.
How do data lineage tools work? Four methods, four kinds of evidence
Data lineage tools work by reading one of four kinds of evidence and drawing a graph from it. The method decides what the tool can and cannot see, long before features or price.
Manual documentation means diagrams and spreadsheets drawn from interviews with system owners. It sees whatever people remember, including steps no system logs, and ages from the day it is drawn. Catalog scanning harvests metadata from databases, warehouses, reporting tools and ETL platforms the scanner can connect to, and infers lineage from table and job definitions. It is broad and cheap to refresh, but it sees only registered systems. Code parsing reads SQL, stored procedures and transformation scripts and infers which columns feed which; research systems such as LINEAGEX, demonstrated at the IEEE data engineering conference in 2025, show it can be precise about column lineage inside a database (Zhang, Miao and Wang, ICDE 2025). It cannot see what is not code. Run-time reconstruction reads what actually ran: query logs, scheduler job logs and file access logs. It sees manual uploads and file shares because those leave access records, but it needs those logs to be kept and readable.
Four methods behind data lineage tools, and what each sees
| Method | Evidence read | Sees well | Tends to miss | Upkeep |
|---|---|---|---|---|
| Manual documentation | Interviews, diagrams, spreadsheets | Business meaning; undocumented steps people remember | Anything forgotten; every change after drawing | High: redrawn by hand |
| Catalog scanning | Metadata harvested from connected systems | Breadth across registered databases and reporting tools | Files, shares, manual uploads, unconnected systems | Low: rescanned |
| Code parsing | SQL, procedures, job and script definitions | Column-level flow inside databases and pipelines | Spreadsheets, manual steps, logic outside code | Medium: parsers per language |
| Run-time reconstruction | Query, scheduler and file-access logs | What actually ran, including file shares and uploads | Steps that leave no log; logic inside a file | Medium: needs log retention |
One element, four methods: which hops each would find
Applied to the sample bank's commercial real estate figure, the methods see different parts of the same seven-hop path. The grid is our reading of what each method can see in principle, given where each hop's evidence lives; it is not a test of any product.
What each lineage method can see of one figure’s seven hops
Manual: Sees it
Catalog: Partly
Code: Misses it
Run-time logs: Sees it
Manual: Sees it
Catalog: Sees it
Code: Partly
Run-time logs: Sees it
Manual: Sees it
Catalog: Sees it
Code: Sees it
Run-time logs: Sees it
Manual: Partly
Catalog: Misses it
Code: Misses it
Run-time logs: Sees it
Manual: Sees it
Catalog: Sees it
Code: Sees it
Run-time logs: Sees it
Manual: Sees it
Catalog: Sees it
Code: Partly
Run-time logs: Sees it
Manual: Partly
Catalog: Sees it
Code: Partly
Run-time logs: Sees it
What is the difference between data lineage tools? Granularity, timing and reach
Beyond the method, three properties separate data lineage tools, and each maps to something a supervisor asks. Granularity: table-level lineage says data moved; column or attribute-level lineage says which field moved and how it was transformed. The ECB's RDARR Guide expects lineage at data attribute level for in-scope indicators and their critical data elements (ECB Guide, May 2024), so table-level arrows will not satisfy an inspection. Timing: design-time lineage shows what the code says should happen; run-time lineage shows what did happen on a given night. A failed job, a rerun or a manual fix appears only in the second.
Reach is the third, and the one that most often decides a BCBS 239 outcome: does the tool see outside the databases? A tool that maps every warehouse table perfectly and stops at the file share has mapped everything except the hop where a person intervenes. A fourth difference matters for BCBS 239 specifically. Most lineage tools draw paths without values. A path with no amount at each hop cannot show where a figure went wrong; critical data elements in banking explains why accuracy should be measured hop by hop.
Can data lineage tools see spreadsheets, and can they trace Excel?
Most can register that a spreadsheet exists; few can trace data into it, through its formulas and out again, and fewer still notice a workbook that nobody registered. Spreadsheets matter because regulators keep finding them in the path. The BIS's Financial Stability Institute called Excel spreadsheets "perhaps the most ubiquitous format in financial regulatory reporting", noting they "are limited to around 1 million rows and 16,000 columns" (BIS FSI Insights No 19, 2019). The ECB's thematic review recorded "many reconciliation errors" in regulatory capital owing to "extensive manual intervention", and that "extensive use was made of Excel and other end-user programming, which makes this process rather complex and error-prone" (ECB Thematic Review, 2018).
Tracing a workbook takes two steps. The first is finding it in the lineage: which file was written from which extract and read by which job, which file-access and scheduler logs can show. The second is reading inside it: which tab and cells changed the figure, which values are hard-coded, which were overridden, which formulas depend on what. In the sample bank, the first step found the workbook and the second found the cause: a "Sold %" column left blank on 48 sold participations, and six manual overrides on a tab called Adj. End user computing controls covers what to do once the workbook is found.
Data lineage, or the traceability of data from its origin to its final use, is important for confirming data quality. This remains a challenging component of BCBS 239 for banks.
Basel Committee on Banking Supervision, Newsletter 36, January 2026
Are open source data lineage tools enough?
For lineage inside databases and pipelines, open source parsers and standards can be enough; for BCBS 239 evidence, the licence is the least of it. The Basel Committee's January 2026 newsletter lists what holds banks back: "Legacy systems, distributed data estates and the dynamic nature of data lineage complicate banks’ efforts to confirm end-to-end data traceability", and "Finding appropriate vendor solutions and the resource-intensive nature of identifying and maintaining data lineage can further hinder progress" (BCBS Newsletter 36). Neither problem is solved by a free parser alone.
An open source stack covers the code-parsing method well. What it usually leaves to the bank is the reach (file shares, uploads, reporting platforms), the confirmation workflow (who confirms each hop, with what evidence), the values (reconciling amounts at each hop) and the upkeep as systems change. Those are integration and staffing costs, not licence costs, and they are where a BCBS 239 programme spends most of its lineage budget.
What does a data lineage tool cost, and what drives it?
List prices are rarely published, so compare tools on the cost of the outcome: a confirmed, current lineage for every critical data element. Four things drive it: the number of source systems and connectors, the number of elements in scope and their hops, the number of hops that need a person (spreadsheets, uploads, re-keyed figures), and the hours stewards spend confirming and re-confirming after changes. The sample bank has 312 critical data elements across 41 reports fed by nine source systems, with 27 workbooks somewhere in its lineage.
The Committee's newsletter makes the business case in one line: "demonstrating the business benefits of data lineage, such as cost reduction and improved efficiency, can help secure investment in automated tools". Botminds' own estimate for the sample bank is that interviewing owners and drawing lineage by hand takes about 16 working weeks of effort in each annual cycle, against about 3 weeks of confirming hops when agents rebuild the lineage; that is an estimate on fictional data, not a measured result. Automated data lineage shows how the agents do the rebuilding and where they stop.
What lineage has to show when an examiner tests it
An examiner does not test a lineage tool; an examiner tests a figure. When the ECB ran its own data lineage exercise on two selected data points, the purpose was root causes: it "helped to clarify the risk data management process, contributing to the identification of the root causes of data quality issues across operating units" (ECB Thematic Review, 2018). The same review criticised "The absence of the necessary audit trails", because "key data concerns regarding lineage and selection could not always easily be traced back to the root cause owing to manual processes".
That sets four requirements for any lineage a bank shows a supervisor. It must reach the root cause, which means the hop and, inside a workbook, the cells. It must carry the controls at each hop, so the examiner can see why a defect was not caught. It must show who confirmed it and when, so it is evidence rather than a drawing. And it must be current on the date of the figure being tested, not on the date of the last scan. A tool that meets the first and fails the other three produces a good diagram and a weak answer.
What to ask in a data lineage tool bake-off
Run the comparison on one of your own elements, chosen because it passes through a spreadsheet or an upload. Give every candidate the same element and the same access, and score what comes back.
A one-element bake-off for data lineage tools: eight things to score
- Hops found, against the true path
Agree the true path with the owners first, then count what each tool finds and what it invents.
- The spreadsheet hop
Did it find the workbook, or draw an arrow straight past it?
- Granularity
Table, column or attribute: can it name the field that carries the figure?
- Run-time or design-time
Does it show what ran last night, or what the code says should run?
- Amounts at each hop
Can it show the value at each hop, or only the path?
- A change
Rename a field in a test copy: does it re-open exactly the elements that read it?
- Confirmation
Who confirms each hop, with what evidence, and is it recorded?
- Effort to keep current
Hours per month to keep the element’s lineage confirmed after changes.
Where BCBS 239 Compliance fits among data lineage tools
BCBS 239 Compliance sits on top of a bank's catalog, not in place of it. Its Lineage agent reads the catalog, the ETL scheduler's job logs, report specifications and file-share access logs, and rebuilds each critical figure's path from the board page down; below a 0.90 confidence threshold a hop is marked "needs evidence" rather than proposed. What sets it apart from a lineage tool is what it does with the path: it reconciles the amount at every hop, replays the last stress drill to time each hop, rates the spreadsheets it finds and cites all of it in the principle ratings people decide.
Where it does not fit: a bank that wants enterprise-wide lineage for every table, for purposes beyond risk reporting, needs a catalog and probably a code parser; the agents focus on critical data elements. And no method traces a hop that leaves no evidence. The honest choice is often both: a catalog as the register of definitions and owners, and agent-built, reconciled lineage for the figures a board and a supervisor read.
Lineage with the amount, the owner, the control and the time at every hop
Spreadsheets in the lineage, flagged element by element
Questions people ask
What are the best data lineage tools for BCBS 239?
The ones that reach attribute level, see outside databases to file shares and uploads, show what actually ran, and record who confirmed each hop. In practice that usually means a catalog for definitions and owners, plus run-time lineage with reconciliation for the critical figures a board and supervisor read.
How do data lineage tools work?
They build a graph of data movement from one of four kinds of evidence: interviews and diagrams, metadata harvested from connected systems, parsed code such as SQL and job definitions, or run-time logs of queries, jobs and file access. Each method sees some hops and misses others.
Can lineage tools trace Excel?
Many can register a workbook; fewer can find an unregistered one or trace data through its formulas. Finding it takes file-access and scheduler logs; explaining it takes reading its tabs, cells, hard-coded values and overrides.
What is the difference between data lineage and a data catalog?
A catalog is a register of data assets with definitions, owners and the lineage it can scan. Lineage is the path itself. A catalog's lineage covers registered systems; complete BCBS 239 lineage must also cover the files and manual steps a catalog may not hold.
Does BCBS 239 require a data lineage tool?
No. Neither BCBS 239 nor the ECB Guide requires a tool. The ECB expects complete, current, attribute-level lineage for in-scope indicators and critical data elements; how a bank produces it is its own choice, though by hand it is slow to keep current.
Is column-level lineage the same as attribute-level lineage?
Close, not identical. Column-level lineage traces columns in databases; the ECB's attribute level means the metadata describing each data element, which may live in a database column, a file field or a spreadsheet cell range.