Architecture Overview
System architecture
Tabsdata is a distributed, horizontally scalable system that runs on Kubernetes and can be deployed across a variety of Kubernetes providers. Tabsdata is organized into two conceptual layers: a resource layer and an execution layer.
The resource layer consists of objects that users create to establish the structure of a Tabsdata workflow. These include objects like Tables and Functions, which determine when a workflow runs, what data flows through it, and what work is applied to that data.
The execution layer consists of objects that Tabsdata creates when it reads and acts on resources from the resource layer. These objects make up the running workflow that performs the work defined in the resource layer. This includes objects like Execution Plans, Transactions, and Function Runs.
Resource layer
Execution layer
Resource Layer
Functions
Functions are Tabsdata's unit of work. A function performs a single discrete step of moving data between external systems and internal Tabsdata Tables. There are three types of Tabsdata Functions:
A publisher reads from an external system and writes into Tabsdata tables.
A transformer reads from Tabsdata tables and writes into new Tabsdata tables.
A subscriber reads from Tabsdata tables and writes out to an external system.
Tables
A unique feature of Tabsdata is that it has its own internal storage for all data produced by functions.
Tabsdata tables persistently store the data produced by functions. Every successful function run that returns data creates a new table version that sits alongside all previous versions of the table.
Schema
Table schema defines the structure of a table by listing its columns and their data types.
| Object | Type |
|---|---|
| first_name | str |
| last_name | str |
| age | i64 |
| nationality | str |
| city | str |
While Tabsdata tables share some similarities to database tables, they also have some key differences. Rather than requiring a predefined schema, each version inside a table carries its own, allowing a table's schema to evolve across versions. This not only makes Tabsdata resilient to schema drift, but also stores a lineage of the table's evolution for observability and debugging.
Table version history is available not only to users for viewing, but also to functions for ingestion. Functions can read historical table versions for use cases that require comparing historical data with incoming data.
Execution Layer
Functions and tables declare how data should be moved, what transformations should be applied, and where data should be stored. Tabsdata's execution layer is what applies these rules and builds the actual runtime workflow that ingests, transforms, and loads your data.
Execution Plans
Traditionally, as more steps are added to a data integration workflow, the more complex it becomes to define how and when each step should execute. This orchestration is often done manually by wiring functions together and explicitly defining the order in which they run through a Directed Acyclic Graph (DAG).
Tabsdata functions do not have this requirement. Because functions read from and write data into Tables rather than directly to other Functions, each Function inherently declares its upstream dependencies (input tables) and downstream dependencies (output tables). Each time a Function is triggered, Tabsdata evaluates these dependencies and automatically builds a directed acyclic graph (DAG), called an Execution Plan, that determines which Functions need to run and in what order.
An Execution Plan automatically executes all functions downstream dependent to the triggering function. This not only removes the need to manually wire DAGs, but it also allows for the handling of race conditions, retries, and ensures atomicity through transactional commits and rollback.
In addition to automatically orchestrating workflows, Execution plans record everything that occurs during execution, capturing metadata such as:
- Execution start and stop time
- Functions executed
- Function code used
- Tables read and written
- Execution status
- Execution logs
Horizontal Scalability
Compute and storage are independently scalable. Each Function runs as an isolated process within the Server's Kubernetes cluster. Storage can be distributed across local file storage or cloud storage providers such as Amazon S3, Azure Blob Storage, and Google Cloud Storage.
Security
Authorization uses a direct permission model, where permissions are assigned directly to users rather than through roles.
Credentials are stored separately from Functions in a secure credential store. This includes both Tabsdata's internal credentials and the credentials used by Connections to authenticate with external systems.
Even when credentials are provided inline through the CLI, Tabsdata automatically extracts and stores them in the credential store.
Deployment
Tabsdata runs on Linux for production and macOS or Windows for development. The underlying Kubernetes can be deployed across several providers
kminus: A lightweight Kubernetes implementation that runs pods as native Python processes instead of Docker containers. The default provider for local development.docker: (coming soon)eks: Amazon Elastic Kubernetes Service (EKS).
AI integration
Tabsdata includes a built-in Model Context Protocol (MCP) Server that allows AI agents to interact with your Tabsdata Server. The MCP Server is backed by a skills library and vector database, providing agents with the context needed to understand and operate on your data workflows. See Connect an AI Agent to get started.
There are two ways to use AI in Tabsdata:
- A built-in agent called Tabby for operational tasks within Tabsdata.
- External coding agents, such as Claude and Codex, connected through the MCP Server.