Architecture Overview
System architecture
Tabsdata is a distributed, horizontally scalable system that runs on Kubernetes and can be deployed across a variety of Kubernetes providers. Resources in Tabsdata are organized into two conceptual layers: a resource layer and an execution layer.
The resource layer contains persistent user-defined resources that define the structure of a Tabsdata workflow. These resources are registered by users through the CLI, UI, or MCP server.
The execution layer contains runtime objects that Tabsdata creates when executing the instructions defined by resources in the resource layer.
Resource layer
Execution layer
Resource Layer
Functions
Functions are Tabsdata's unit of work. A function performs a single discrete step of moving data between external systems and internal Tabsdata Tables. There are three types of Tabsdata Functions:
A publisher reads from an external system and writes into Tabsdata tables. A transformer reads from Tabsdata tables and writes into new Tabsdata tables. A subscriber reads from Tabsdata tables and writes out to an external system.
Connectors and Connections
A Function is responsible for moving data between sources and destinations. Sometimes those sources or destinations are external systems, which require system-specific API logic and credentials to access. Tabsdata abstracts these into two separate resources.
- A Connector contains the logic needed to communicate with a specific type of external system, and translates data to and from tabular DataFrames.
- A Connection is a credential store that allows Functions to authenticate with an external system.
Functions themselves only store the query to the external system and transformation logic. Functions use Connectors and Connections to successfully authenticate and interface with external systems.
Tables
A unique feature of Tabsdata is that it has its own internal storage for all data produced by functions.
Tabsdata tables persistently store the data produced by functions. Every successful function run that returns data creates a new table version that sits alongside all previous versions of the table.
Schema
Table schema defines the structure of a table by listing its columns and their data types.
| Object | Type |
|---|---|
| first_name | str |
| last_name | str |
| age | i64 |
| nationality | str |
| city | str |
While Tabsdata tables share some similarities to database tables, they also have some key differences. Rather than requiring a predefined schema, each version inside a table carries its own, allowing a table's schema to evolve across versions. This not only makes Tabsdata resilient to schema drift, but also stores a lineage of the table's evolution for observability and debugging.
Table version history is available not only to users for viewing, but also to functions for ingestion. Functions can read historical table versions for use cases that require comparing historical data with incoming data.
Execution Layer
Execution Plans
Tabsdata Functions are self-describing. By reading and writing data into tables rather than directly into other functions, functions inherently declare their upstream dependencies (input tables) and downstream dependencies (output tables). Each time a function is triggered, Tabsdata evaluates these dependencies and builds a directed acyclic graph (DAG) called an Execution Plan.
An Execution Plan automatically executes all functions downstream dependent to the triggering function. This not only removes the need to manually wire your DAGs, but it also allows for the handling of race conditions, retries, and ensures atomicity through transactional commits and pristine state.
In addition to automatically orchestrating workflows, Execution plans record everything that occurs during execution, capturing metadata such as:
- Execution start and stop time
- Functions executed
- Function code used
- Tables read and written
- Execution status
- Execution logs
Horizontal Scalability
Compute and storage are independently scalable. Each Function runs as an isolated process within the Server's Kubernetes cluster. Storage can be distributed across local file storage or cloud storage providers such as Amazon S3, Azure Blob Storage, and Google Cloud Storage.
Security
Authorization uses a direct permission model, where permissions are assigned directly to users rather than through roles.
Credentials are stored separately from Functions in a secure credential store. This includes both Tabsdata's internal credentials and the credentials used by Connections to authenticate with external systems.
Even when credentials are provided inline through the CLI, Tabsdata automatically extracts and stores them in the credential store. Credentials are never stored in plaintext.
Deployment
Tabsdata runs on Linux for production and macOS or Windows for development. The underlying Kubernetes can be deployed across several providers
kminus: A lightweight Kubernetes implementation that runs pods as native Python processes instead of Docker containers. The default provider for local development.docker: (coming soon)eks: Amazon Elastic Kubernetes Service (EKS).
AI integration
Tabsdata includes a built-in Model Context Protocol (MCP) Server that allows AI agents to interact with your Tabsdata Server. The MCP Server is backed by a skills library and vector database, providing agents with the context needed to understand and operate on your data workflows. See Connect an AI Agent to get started.
There are two ways to use AI in Tabsdata:
- A built-in agent for operational tasks within Tabsdata.
- External coding agents, such as Claude and Codex, connected through the MCP Server.