Skip to main content
GuideServerConfigure and deploy Tabsdata servers on your machine.TutorialsConfigure data integration workflows within a running Tabsdata server.API ReferenceCLI ReferenceRelease Notes
Version: 2.2.1

Lineage

Any number of factors can evolve over the course of a data pipeline's lifecycle, including the shape of processed data, the logic used to move and transform that data, and when specific data was scheduled to be processed. This is called Data Lineage.

Tabsdata's built-in data lineage automatically stores immutable versions of tables and functions as they are modified, preserving a record of how your workflows change over time. This allows users to travel back to historical versions of table data, examine the code that produced them, and see how their workflow logic has evolved.

How Lineage Is Recorded​

All Tabsdata functions declare the Tables they read from and write to. This allows Tabsdata to infer dependencies and execution order, a core mechanic that enables Automatic Orchestration and Execution Plans.

Publisher: output_tables writes a Tabsdata Table.@publisher(source=PostgresSrc(...),output_tables=["orders"],)def clean_orders(orders):cleaned = ...return cleanedTabsdata Tableorders

Because dependencies are declared ahead of time, Tabsdata automatically links each table version to the function and function code that produced it. These versions are then grouped into an execution plan, along with the other tables and functions affected by the execution.

Version 3 of the customers table was written by version 2 of the pub_customers function.Functionpub_customersVersionsVersion 1Oct 12Version 2Oct 24Code@publisher( source=PostgresSrc(queries=[ "SELECT customer_id, name, email, region, " "signup_date::date AS signup_date " "FROM customers", ]), output_tables=["customers"],)def pub_customers(customers): return customersTablecustomersVersionsVersion 1Oct 20Version 2Oct 24Version 3Oct 27SamplenameemailregionAna Ruizana@example.comEMEABen Okaforben@example.comAMERChen Weichen@example.comAPAC5 rowsFunction version 2Created Oct 27, 09:13

Table History​

Each time a Function run successfully returns data, Tabsdata creates a new, immutable table version while preserving all previous versions of the table's data and schema.

Functions can also read previous Table versions. An input written as customers@HEAD^ reads the version before the latest, and customers@HEAD~2..HEAD reads a range of versions ending at the latest.

Uses the latest version of customers as input.

A transformer with input_tables set to customers@HEAD.@transformer(input_tables=["customers@HEAD"],output_tables=["customers_report"],)def report(customers):...

Schema History​

Each Table version stores its own schema, so a Table's version history records changes to its schema as well as its data. Comparing the schema of one Table version with another shows which columns were added, removed, or changed type.

Function History​

Functions have their own version history. Whenever a Function is updated, Tabsdata creates a new Function version, retaining a changelog of all previous versions of the function's code.

Viewing Lineage​

In the Tabsdata User Interface, lineage can be viewed from Tables, Execution Plans, and Functions.

Tables​

Opening a Table shows three tabs: Schema, Lineage, and Sample. The Lineage tab shows a graph of the Execution Plan that wrote the selected table version. This execution plan is unique to the specific table version, and exploring the lineage of other versions can show the evolution of a particular execution plan.

orders_clean
Tabsdata›Collections›sales›orders_clean

Schema

Table schema defines the structure of a table by listing its columns and their data types.

ObjectType
order_idi64
customer_idi64
amountf64
ordered_atdatetime