Architecture Overview
System architecture
Tabsdata is a distributed, horizontally scalable system that runs on Kubernetes. It decouples the compute, storage, and metadata layers:
Each Tabsdata Function runs as an isolated process on the Tabsdata server's Kubernetes cluster. Built atop Polars Lazyframe Architecture, Lazy evaluation and streaming capabilities allow each worker handle large datasets without being constrained by memory limits.
Tabular data can be stored on local file storage, in a cloud provider, or spread across several providers such as Amazon S3, Azure Blob Storage, Google Cloud Storage, or a network file system.
Scale storage independently from Compute
Projects, Collections, Functions, Tables, Execution Logs, Error History, Code Versioning, and more can be stored locally, in the cloud, or distributed across multiple systems
Architecture Heirarchy
- Tabsdata Server
- Project
- Group
- Collection
- Connection
- Function
- Table
- Version
- Schema
- Data
Tabsdata separates a data integration workflow into logical layers.
Projects organize work
Groups organize workflow stages
Collections contain the logic and credentials required for discrete workflow steps
Functions perform individual ETL steps
Tables preserve the versioned data produced by Functions.
Compute layer
Functions
Functions are Tabsdata's unit of work. A function performs a single discrete step of moving data between external systems or internal Tabsdata Tables. There are three types of Tabsdata Functions:
A publisher reads from an external system and writes into Tabsdata tables. A transformer reads from Tabsdata tables and writes into new Tabsdata tables. A subscriber reads from Tabsdata tables and writes out to an external system.
Reaching external systems
A Function is responsible for moving data between sources and destinations. Sometimes those sources or destinations are external systems, which require system-specific API logic and credentials to access. Tabsdata abstracts these into two separate resources.
- A Connector contains the logic needed to communicate with a specific type of external system, and translates data to and from tabular DataFrames.
- A Connection is a credential store that allows Functions to authenticate with an external system.
Functions themselves only store the query to the external system and transformation logic. Functions use Connectors and Connections to successfully authenticate and interface with external systems.
Storage layer
Tables
A unique feature of Tabsdata is that it has its own internal storage for all data produced by functions.
Tabsdata Tables store the versioned datasets produced by Functions. Every successful Function run that returns data creates a new Table Version, preserving previous Versions rather than overwriting them. By default, Functions use the latest Table Version as input, but they can also be configured to read any previous Version or a range of Versions.
input/persons| Column | Type | Description |
|---|---|---|
first_name | Utf8 | Given name |
last_name | Utf8 | Family name |
age | Int64 | Age in years |
nationality | Utf8 | Nationality |
city | Utf8 | City of residence |
Every Table Version stores its own schema and data. Rather than requiring a predefined schema, Tabsdata infers the schema from the data returned by a Function and stores it alongside each Table Version.
Because every Table Version stores its own schema, Tabsdata workflows are resilient to schema drift. If a source schema changes, Tabsdata infers the updated schema and stores it in a new Table Version rather than failing due to a schema mismatch.
By default, Functions read the latest Table Version, but they can also be configured to read any previous Version or a range of Versions.
Metadata layer
Execution Plans
Tabsdata Functions are self-describing. Each Function declares the Tables it reads from and writes to. When a Function is triggered, Tabsdata evaluates these dependencies to identify downstream Functions and builds a directed acyclic graph (DAG) called an Execution Plan.
An Execution Plan both orchestrates the workflow and records everything that occurred during execution. It captures metadata including:
- Execution start and stop time
- Functions executed
- Function code used
- Tables read and written
- Execution status
- Execution logs
Metadata is stored independently from Table storage.
Security
Authorization uses a direct permission model, where permissions are assigned directly to users rather than through roles.
Credentials are stored separately from Functions in a secure credential store. This includes both Tabsdata's internal credentials and the credentials used by Connections to authenticate with external systems.
Even when credentials are provided inline through the CLI, Tabsdata automatically extracts and stores them in the credential store. Credentials are never stored in plaintext.
Deployment
Tabsdata runs on Linux for production and macOS or Windows for development. The underlying Kubernetes cluster is provided by one of three deployment providers, configured with --provider when using tdkserver CLI commands:
kminus: A lightweight Kubernetes implementation that runs pods as native Python processes instead of Docker containers. The default provider for local development.docker: The Kubernetes cluster included with Docker Desktop.eks: Amazon Elastic Kubernetes Service (EKS).
AI integration
Tabsdata includes a built-in Model Context Protocol (MCP) Server that allows AI agents to interact with your Tabsdata Server. The MCP Server is backed by a skills library and vector database, providing agents with the context needed to understand and operate on your data workflows. See Connect an AI Agent to get started.
There are two ways to use AI in Tabsdata:
- A built-in agent for operational tasks within Tabsdata.
- External coding agents, such as Claude and Codex, connected through the MCP Server.
The MCP Server integrates with Tabsdata's authorization model, ensuring agents can only perform actions that the authenticated user is permitted to perform.