Data Lineage Tools for AI Systems: Proving Which Source Produced an Answer

A support bot tells a customer they have 30 days to return an item. The policy changed to 14 days last quarter. The customer escalates, the team pulls up the answer, and nobody can say which document the bot actually read to produce that number.

That gap is the real job of data lineage tools. They record where a piece of data came from, how it changed, and where it ended up. For a dashboard or a finance report, that trail is usually enough to satisfy anyone who asks. For an AI answer, the trail tends to stop right before the part you need most.

This post covers what data lineage tools actually record, where their graph ends, and what you have to add to prove which source produced an answer. We’ll start with the classic warehouse definition, so a data platform reader gets full value. Then we get to the question a support bot forces on you.

The short version is that one graph is not enough on its own. Warehouse lineage proves a document reached your index. A separate request-level record proves what the model retrieved and whether it used it. You need both, and most teams keep only the first.

What Data Lineage Tools Track

Data lineage is the recorded path a piece of data travels through your systems, from origin through every transformation to its destination. A good lineage graph lets you start at any field and walk backward to its source, or start at a source and see everything downstream that it feeds. That two-way walk is the whole value of the graph.

Table, column and row level lineage

Lineage comes at three grains, and the grain decides what questions you can answer. Table level says one table fed another table. Column level says which field fed which field, so you can trace a single metric back to its inputs. Row level says which record produced which record, and it is the grain that almost no tool serves well.

Row level demand is real and mostly unmet. On r/dataengineering, practitioners ask whether they can trace an individual row through a pipeline, and the common reply is that it gets expensive and is rarely built in. Read that as unserved demand from real engineers, not as a measured statistic about the market.

Lineage compared with provenance

The words lineage and provenance get used loosely, and the difference matters a lot here.

The W3C PROV overview defines provenance as a record of the entities, activities, and people involved in producing something, where the people map to what PROV calls agents. Lineage answers where data went. Provenance answers who or what is responsible for a result.

This distinction runs through the whole topic. A team searching for data lineage tools to support an AI feature usually has a provenance question in mind without naming it. They want to know what produced one specific answer, which is a question about responsibility for a result, not only about the path data took.

Why Standard Data Lineage Tools Stop at the Model Boundary

Your warehouse graph ends at the last table. An AI answer gets produced well past it. A question becomes a vector, chunks are retrieved and ranked, and a subset goes into the prompt where the model writes a sentence. Our RAG architecture guide walks that embed, retrieve, rank, and generate path end to end.

Each of those steps is a place where attribution quietly leaks. Chunking splits a document, so a claim detaches from the file it came from. Reranking drops candidates without recording why. Prompt assembly flattens several documents into one long string, and generation blends all of it into fresh text where the seams disappear.

The consequence is narrow, and worth saying plainly. A lineage tool can prove a document reached your index. It cannot prove that same document produced a given sentence in the output. Those are two different claims, and the space between them is exactly where an audit or an escalation tends to fall apart.

One practitioner framed the deeper problem on Hacker News, in a post titled “You think you have data lineage, but you probably don’t.”

The line worth keeping is that most systems do not preserve lineage, they try to reconstruct it after the fact. That post drew no discussion, so treat it as one practitioner’s framing rather than a settled community view.

That reconstruction is the trap. If nobody stored what the model retrieved at request time, you are left stitching a story together from logs that were never designed to answer the question. The clean fix is to record the evidence during the request, while the retrieval and the output still sit next to each other.

The Two Records That Prove Which Source Produced an Answer

This is the core argument. Proving that a source produced an answer takes two separate records, and most teams keep only one of them. One record captures what the system pulled for a request. The other captures whether the model actually used it. Keep one without the other and the proof does not hold.

Record one, the retrieval record

The retrieval record captures what was retrieved for one request. That means document identifiers, chunk identifiers, retrieval scores, the query text, the top k value, and the data source identifier. It is a per-request artifact, written at the moment of the call. A static lineage graph cannot stand in for it, because the graph is not request specific.

Record two, the usage record

The usage record answers a different question. It says whether the retrieved context actually shaped the output, and to what degree. Retrieval and usage are not the same thing. A document can sit in the prompt, take up tokens, and contribute nothing to the final sentence. Without a usage check, you are guessing about what mattered.

The two records answer two halves of one question. The retrieval record tells you the candidate set that could have produced the answer. The usage record tells you whether that set was read or ignored. A support answer becomes defensible only when you can show both halves for that exact request, side by side.

There’s a ceiling here worth stating, because overclaiming is easy. With both records you can prove which candidate set produced an answer and whether the context was used. You cannot prove that one named document, on its own, produced one specific sentence. That per-sentence claim is stronger than either record supports, so this post does not make it.

That ceiling is the shape of the problem, not a flaw in your setup. Language models blend sources as they write, so a single sentence rarely traces to a single file.

The practical goal is to narrow the answer to a small, recorded candidate set and to show the context was used, which is enough for most audits and support reviews.

How Data Lineage Tools Capture What They Capture

The way a tool captures lineage predicts how it will fail. Every capture method trades off coverage, freshness, and effort in its own way. Once you can name the method behind a graph, you can usually name its blind spot too. That habit is more useful than any vendor comparison chart on this keyword.

OpenLineage is an open platform for the collection and analysis of data lineage. It models the world as datasets, jobs, and runs, with facets that carry extra detail on each. It graduated at LF AI and Data, announced on 20 September 2023, which gives it standing as a shared standard rather than one vendor’s private format.

The fragility is easier to trust from open issues than from adjectives. Across OpenLineage, Marquez, dbt Core, and OpenMetadata, public issues show column level lineage and runtime capture breaking on real pipelines. Maintainers have not dismissed these as edge cases. They remain open threads on the exact features that buyers most want to rely on.

dbt Core documents model level lineage out of the box. Column level lineage is a paid Catalog capability, and dbt now also lists it as a dbt Docs v2 feature available with the dbt Fusion engine and dbt Core 2.0.

State both, because the boundary is moving. That single buying detail is missing from every competing page on this keyword.

No single method wins the table outright; they split into two camps. Warehouse methods cover the warehouse well and stop at its edge. Trace instrumentation covers the request and says little about upstream tables. A full picture usually needs one method from each group, connected at a shared identifier.

Where Retrieval Provenance Lives in a Trace

Once you accept that the evidence lives in the request, the next question is where exactly it sits. The answer is the trace.

OpenTelemetry has GenAI semantic conventions that define a retrieval span, identified in the registry as gen_ai.retrieval.client, where gen_ai.operation.name is retrieval and the span kind is CLIENT. That span is where a retrieval record naturally lands, one span per retrieval step.

The span carries the attributes you would want as evidence. gen_ai.retrieval.documents holds the retrieved documents and is Opt-In. gen_ai.retrieval.query.text holds the query and is Opt-In. gen_ai.data_source.id is Conditionally Required, and gen_ai.retrieval.top_k is Recommended. The two attributes with the real proof are both Opt-In.

That Opt-In status has a practical bite. This area of the convention is still in Development, and default instrumentation will not store the document payload or the query text unless you switch them on. A team that assumes those fields are captured for them will open a trace at audit time and find the evidence fields empty.

The work here has to be deliberate. To prove which documents were retrieved, you enable the document and query attributes and confirm they populate. The names are public, so you can check your own spans today. Getting spans right is its own discipline, and our LLM tracing best practices guide covers hygiene, sampling, and PII.

What Auditors and the EU AI Act Ask For

Regulation is now part of this conversation, and it points in the same direction.

Article 12 of the EU AI Act requires the automatic recording of events, or logs, over the lifetime of a system, at a level that enables traceability. It does not ask for a diagram of your stack. It asks for a durable record of what actually happened.

For certain high-risk systems, listed under Annex III point 1(a), the Act names two specific records to keep. One is the reference database against which input data was checked. The other is the input data for which the search led to a match. Read closely, that pairing is a retrieval record written directly into law.

It’s easy to oversell compliance, so stay factual about it. The Act names logging obligations, not products, and no data lineage tool is a compliance guarantee on its own. What the text gives you is the shape of the evidence a regulator expects, and that shape looks a great deal like the retrieval record described above.

For a team already logging retrieval spans, this reads as reassuring rather than alarming. The record the Act describes is close to the record that good instrumentation produces anyway. For most teams the real gap is a setup detail. The document and query fields were never switched on, so the log exists while the evidence inside it stays thin.

How Do You Choose Data Lineage Tools for an AI Stack?

The buying question usually gets framed as which vendor, and that framing is what trips teams up. A cleaner approach is to choose by layer. Most AI stacks need two layers, a warehouse lineage layer and a request-level tracing layer, because a single product rarely answers both kinds of question well on its own.

The table below sorts the decision by the question you actually need answered. Find your question on the left, read across to the layer that answers it, and check what that layer still leaves open. The gaps in the right column are the whole point.

So warehouse lineage and request-level tracing solve different parts of the problem, and buying one does not deliver the other. A column level graph will not tell you what a model retrieved. A trace will not reconstruct a warehouse join. Plan for both, and connect them at the data source identifier.

If you are early, start with the layer that matches your riskiest question. Teams shipping RAG answers to customers usually feel the request-level gap first. Teams reporting regulated metrics usually feel the warehouse gap first. Sequence the spend to match the risk, and expect to grow into needing both over time.

Capturing Retrieval Provenance with Future AGI

The method in this post needs two things, a per-request retrieval record and a check on whether the context was used.

Future AGI Observability captures each retrieval step as its own span, so the retrieval record has a natural home. Evaluations can then run on a single span rather than on a whole trace.

In Future AGI, a span is one operation inside a trace, a single model call, tool call, retrieval, agent step, guardrail check, or evaluator run. A retriever span captures the query and the documents it returned. In the eval builder, the “Run evaluations on” control targets Spans, Traces, or Sessions, so a retrieval span can be scored on its own.

Future AGI runs both custom evals and built-in ones. Chunk Attribution is a built-in eval that fits the usage question directly, taking context and output, then returning a Result and a Reason. It checks whether the model references the provided context chunks, a pass or fail RAG usage check that catches retrieval succeeding while the model ignores what it pulled.

There’s one boundary to name here. Chunk Attribution tells you the context was used or not used. It is not per-sentence citation to a named document, and this post does not stretch it that far. Instrumentation is handled by traceAI, an open layer built on OpenTelemetry that publishes conventions for LLM calls, tools, and retrieval.

What to Instrument Before Your Next Audit

Come back to the refund window answer from the start. That answer is traceable only if the retrieval record for that one request was kept. If the span stored the document identifiers, the query, and a usage check, the team can show what the bot read. If not, they are reconstructing after the fact and hoping.

The short list to instrument before your next audit runs like this. Keep document and chunk identifiers on the retrieval span. Keep the query text and the top k value. Turn on the document payload attribute deliberately, since it is Opt-In. Then score whether the context was actually used, so retrieval and usage both stay on record.

Keep the ceiling in view while you do it. This setup proves which candidate set produced an answer and whether the context was used. It does not pin one sentence to one file, and it does not need to. That is the right mental model to leave with, and it is enough for most reviews.

Provenance at this level is really an observability habit, captured at the span, one request at a time. Future AGI records it there so the evidence is ready before anyone asks. For the wider practice around production AI, our LLM observability guide maps the rest of the surface.