# Introduction > Introduction to Tabsdata, an open-source Python data integration platform: deployment requirements, Python functions with Polars expressions, automatic DAG orchestration, data versioning, MCP integration, and connectors. Source: https://docs.tabsdata.com/guide/introduction Tabsdata is an open-source, distributed, horizontally scalable data integration platform that allows users to turn their Python functions into automated, AI-friendly workflows that move and transform data between a variety of different systems. Built atop [Kubernetes](https://kubernetes.io/) and the [Polars](https://pola.rs/) DataFrame API, Tabsdata can be deployed on almost any machine that runs Python, requires no admin permissions or Docker dependency, and allows users to move and transform tabular data through an expression-based Pythonic interface. Through Tabsdata's MCP integration, users can also use AI to build and analyze their workflows. _Video: Leads data pipeline animation_ ## Key Features - - - - - - - Tabsdata can be deployed on almost any machine and requires only Python as a dependency. See [Architecture Overview](https://docs.tabsdata.com/guide/architecture) for more detail on how these pieces fit together. ## Next steps 1. [Install Tabsdata](https://docs.tabsdata.com/guide/install) -- get the server running locally. 2. [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) -- point your LLM at the MCP server. 3. [Run Your First Data Integration](https://docs.tabsdata.com/guide/run-your-first-data-integration) -- build a working flow end to end. ## LLM-friendly formats The documentation is also published in [llms.txt format](https://llmstxt.org) for AI agents: - [llms.txt](https://docs.tabsdata.com/llms.txt): an index of the guide, tutorial and server administration pages, one line per page. - [llms-full.txt](https://docs.tabsdata.com/llms-full.txt): the full text of every page in that index, in one file of about 1 MB, which can exceed an agent's context window. Any page is available as markdown at its URL with `.md` appended. This page, for example, is at [`/guide/introduction.md`](https://docs.tabsdata.com/guide/introduction.md). ## Community You can join the Tabsdata community on [Github](https://www.github.com/tabsdata/tabsdata/) or [Slack](https://join.slack.com/t/tabsdata-community/shared_invite/zt-322toyigx-ZGFioMV2Gbza4bJDAR7wSQ). ## Contributing We appreciate all contributions, from reporting bugs to implementing new features. You can access the Tabsdata Github repository [here](https://www.github.com/tabsdata/tabsdata/). ## License For license information, see the [Tabsdata website](https://tabsdata.com/license). --- # Install Tabsdata > Installs Tabsdata with pip or uv into a Python 3.12 virtual environment, sets the Anthropic or OpenAI key, runs tdkserver quickstart with the kminus provider, and logs in. Source: https://docs.tabsdata.com/guide/install Deploy a Tabsdata Server on your Machine in under 5 minutes. Tabsdata is published as a python package, installable with `pip` Requirements - Machine with a supported OS - Linux (x86) - macOS (Apple Silicon/ARM64) - Windows (x86) - **Python 3.12.13 or later** - An LLM API key from **Anthropic** or **OpenAI (Required for AI features)** ### Create a Python virtual environment Using a virtual environment provides a layer of isolation and prevents interference from other python environments on your machine. - macOS / Linux - Windows (PowerShell) - conda - uv - pyenv - venv ```bash conda create -n tabsdata python=3.12 -y conda activate tabsdata ``` ```bash uv venv --python 3.12.13 .venv source .venv/bin/activate ``` ```bash pyenv install 3.12.13 pyenv local 3.12.13 python -m venv .venv source .venv/bin/activate ``` ```bash python3 -m venv .venv source .venv/bin/activate ``` - conda - uv - pyenv - venv ```powershell conda create -n tabsdata python=3.12 -y conda activate tabsdata ``` ```powershell uv venv --python 3.12.13 .venv .venv\Scripts\activate ``` ```powershell pyenv install 3.12.13 pyenv local 3.12.13 python -m venv .venv .venv\Scripts\activate ``` ```powershell python -m venv .venv .venv\Scripts\activate ``` ### Install Tabsdata Install `tabsdata-agent` alongside `tabsdata[all]` to enable AI features. - uv - pip ```bash uv pip install "tabsdata[all]" ``` ```bash pip install "tabsdata[all]" ``` ### Set LLM key as environment variable in terminal - macOS / Linux - Windows (PowerShell) - Anthropic - OpenAI ```bash export ANTHROPIC_KEY="sk-ant-..." ``` ```bash export OPENAI_KEY="sk-..." ``` - Anthropic - OpenAI ```powershell $env:ANTHROPIC_KEY = "sk-ant-..." ``` ```powershell $env:OPENAI_KEY = "sk-..." ``` ### Start the Tabsdata server `tdkserver quickstart` initializes and deploys a Tabsdata Instance. The command should take around 30 seconds to fully execute. - Anthropic - OpenAI ```bash tdkserver quickstart --config anthropic-key=env:ANTHROPIC_KEY ``` ```bash tdkserver quickstart --config openai-key=env:OPENAI_KEY ``` Some Notes on Tabsdata Server Providers Tabsdata runs on Kubernetes, and this install uses `kminus`: a lightweight provider that runs pods as native processes in your Python environment instead of Docker containers, so it needs no root or administrator access. See [Deploy a Server on k-minus](https://docs.tabsdata.com/server-admin/kminus) for the full lifecycle. Tabsdata can also be deployed on [AWS EKS](https://docs.tabsdata.com/server-admin/eks-install), [Azure AKS](https://docs.tabsdata.com/server-admin/aks-install), or [Google GKE](https://docs.tabsdata.com/server-admin/gke-install). Installing Third-party Database Drivers If you plan to use the MySQL or PostgreSQL CDC connectors, they need database drivers that Tabsdata cannot redistribute. Please review [Installing Third-Party Database Drivers](https://docs.tabsdata.com/server-admin/third-party-database-drivers) for more information. ### Check Tabsdata server status View all your Tabsdata Instances ```bash tdkserver list ``` Output ``` Instances ┌──────────┬──────────────────────────────────────────────────┬──────────┬─────────┬─────────┬─────────┬─────────────────┬───────────────┬───────────────────────┐ │ Name │ Folder │ Provider │ Version │ State │ Status │ API Server Port │ Database Port │ API Server URL │ ├──────────┼──────────────────────────────────────────────────┼──────────┼─────────┼─────────┼─────────┼─────────────────┼───────────────┼───────────────────────┤ │ tabsdata │ ~/.tabsdata/instances/tabsdata │ kminus │ 2.0.2 │ created │ Running │ 2457 │ 2460 │ http://localhost:2457 │ └──────────┴──────────────────────────────────────────────────┴──────────┴─────────┴─────────┴─────────┴─────────────────┴───────────────┴───────────────────────┘ ``` View the status of the Tabsdata Instance you just deployed. ```bash tdkserver status ``` Output ``` Deployments ┌───────────────┬─────────┬───────┬─────┐ │ Name │ Status │ Ready │ Age │ ├───────────────┼─────────┼───────┼─────┤ │ srv-aiagent │ Running │ 1/1 │ 15s │ │ srv-apiserver │ Running │ 1/1 │ 15s │ │ srv-database │ Running │ 1/1 │ 24s │ └───────────────┴─────────┴───────┴─────┘ Pods ┌────────────────────────────────┬─────────┬───────┬─────┐ │ Name │ Status │ Ready │ Age │ ├────────────────────────────────┼─────────┼───────┼─────┤ │ srv-aiagent-7476c88484-4klx8 │ Running │ 2/2 │ 15s │ │ srv-apiserver-56bbbb559d-8cnxl │ Running │ 1/1 │ 15s │ │ srv-database-fd4f8f85-dsxcj │ Running │ 1/1 │ 24s │ └────────────────────────────────┴─────────┴───────┴─────┘ CronJobs ┌──────────────────┬─────────┬─────────────┬─────────────────┬─────┐ │ Name │ Status │ Expression │ Explanation │ Age │ ├──────────────────┼─────────┼─────────────┼─────────────────┼─────┤ │ srv-aiembeddings │ Running │ */5 * * * * │ Every 5 minutes │ 6s │ │ srv-aimetadata │ Running │ */5 * * * * │ Every 5 minutes │ 6s │ └──────────────────┴─────────┴─────────────┴─────────────────┴─────┘ Jobs ┌────────────────┬──────────┬─────────────┬─────┐ │ Name │ Status │ Completions │ Age │ ├────────────────┼──────────┼─────────────┼─────┤ │ ini-inspector │ Complete │ 1/1 │ 16s │ │ ini-venvloader │ Complete │ 1/1 │ 17s │ └────────────────┴──────────┴─────────────┴─────┘ ``` tip `tdkserver` commands usually require an `--instance` flag that specifies the specific instance you want to interact with. If no `--instance` flag is passed, Tabsdata defaults the instance name to `tabsdata` ### Log in ```bash tdk login --server localhost --user admin --password tabsdata ``` Or access the UI at [http://localhost:2457/login](http://localhost:2457/login) Next: [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent). Point your LLM at the MCP server, then run your first pipeline. --- # Connect to MCP > Installs the tabsdata-agent package and registers the Tabsdata MCP Server with Claude Code or Codex, with per-shell commands for Linux, macOS, and Windows. Source: https://docs.tabsdata.com/guide/connect-agent Connect your coding agent to the [Tabsdata MCP Server](https://docs.tabsdata.com/guide/mcp) to build and run data workflows. Claude Code and Codex are currently supported. Step 1Install the Tabsdata MCP package Install `tabsdata-agent` in the virtual environment containing `tabsdata[all]` from [Install Tabsdata](https://docs.tabsdata.com/guide/install). - uv - pip ```bash uv pip install tabsdata-agent ``` ```bash pip install tabsdata-agent ``` Step 2Connect your agent to MCP Start the agent from a shell whose virtual environment has Tabsdata installed. - Claude Code - Codex ```bash claude mcp add tabsdata -- python -m tabsdatak.ag.mcp - ``` - Linux / macOS - Windows PowerShell - Windows cmd.exe ```bash codex mcp add tabsdata -- "$(python -c 'import sys; print(sys.executable)')" -m tabsdatak.ag.mcp - ``` ```powershell codex mcp add tabsdata -- (python -c "import sys; print(sys.executable)") -m tabsdatak.ag.mcp - ``` ```bat for /f "delims=" %i in ('python -c "import sys; print(sys.executable)"') do codex mcp add tabsdata -- "%i" -m tabsdatak.ag.mcp - ``` Step 3Use the agent You need an active session with the Tabsdata server before starting your agent. Log in, then start it from the same terminal: - Claude Code - Codex ```bash tdk login --server localhost --user admin --password tabsdata claude ``` ```bash tdk login --server localhost --user admin --password tabsdata codex ``` The agent authenticates through your session, so it can only do what you're authorized to do. The Server's AI agent service must be running. See [MCP Server requirements](https://docs.tabsdata.com/guide/mcp#server-requirements). Next: [Run Your First Data Integration](https://docs.tabsdata.com/guide/run-your-first-data-integration) --- # Run Your First Data Integration > Quickstart that has an LLM agent via Tabsdata MCP build and run a workflow: CSV publisher, age-dropping transformer adding full_name, and JSONL subscriber to local files. Source: https://docs.tabsdata.com/guide/run-your-first-data-integration Before you start Finish [Install Tabsdata](https://docs.tabsdata.com/guide/install) and [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) first, and make sure the server is running. This quickstart uses the Tabsdata MCP integration. To deploy and run Tabsdata without an LLM, switch to the developer doc mirror. We will be building a Tabsdata Workflow that: 1. ingests a local `persons.csv` into a Tabsdata table called `persons` 2. builds a second table called `people` that drops the `age` column and adds a `full_name` column 3. writes `people` back to your file system as `people.jsonl` Click any node for what it does Local File Local File Publisher qs\_pub Pending Table persons Pending Transformer qs\_tfr Pending Table people Pending Subscriber qs\_sub Pending Local File Local File Not written yet Step 1Get the example project ```bash tdk examples extract --name quickstart-ai --dir qs-example ``` persons.csv conn-input.yaml conn-output.yaml README.txt Creates `qs-example/`, which contains the source data `persons.csv` and two connection files. Step 2Login to Tabsdata and start your LLM Log in, then start your agent from the same terminal: - Claude Code - Codex ```bash tdk login --server localhost --user admin --password tabsdata claude ``` ```bash tdk login --server localhost --user admin --password tabsdata codex ``` Step 3Prompt your LLM to build a Tabsdata workflow Give the following prompt to your LLM: ```bash Build a tabsdata workflow that 1. ingests `qs-example/input-data/persons.csv` 2. creates a new column for the full name and drops the age column, 3. writes the result as a .jsonl to the `output-data` folder More info can be found under `qs-example/README.md` ``` AI agent session ❯Build a tabsdata workflow that 1. ingests `qs-example/input-data/persons.csv` 2. creates a new column for the full name and drops the age column, 3. writes the result as a .jsonl to the `output-data` folder More info can be found under `qs-example/README.md` Read 1 file, listed 1 directory (ctrl+o to expand) ⏺ persons.csv has first\_name, last\_name and age. Three functions cover this: a publisher for the CSV, a transformer for the column changes, and a subscriber for the .jsonl. Getting the connection and publisher guides. Called tabsdata 3 times (ctrl+o to expand) ⏺ Writing , then a source collection pointed at input-data. Called tabsdata 2 times (ctrl+o to expand) ⏺ Registering the publisher that produces **persons**. ```python @publisher( source=LocalFileSrc(paths=["persons.csv"], format=SrcFileFormat.CSV), output_tables=["persons"], ) def ingest_persons(rows: TableFramesSpec) -> TableFrameSpec: return rows[0] if rows else None ``` Called tabsdata 2 times (ctrl+o to expand) ⏺ Registered and committed. Now the transformer — full\_name from first\_name and last\_name, age dropped. ```python @transformer( input_tables=["persons"], output_tables=["people"], ) def build_people(persons: TableFrameSpec) -> TableFrameSpec: return persons.with_columns( (col("first_name") + " " + col("last_name")).alias("full_name"), ).drop("first_name", "last_name", "age") ``` Called tabsdata 2 times (ctrl+o to expand) ⏺ Registered and committed. Last, the subscriber writing people out as .jsonl. Getting the connection and subscriber guides. Called tabsdata 3 times (ctrl+o to expand) ⏺ Writing , then a destination collection pointed at output-data. Called tabsdata 2 times (ctrl+o to expand) ⏺ Registering the subscriber. ```python @subscriber( tables=["people"], destination=LocalFileDst(path="people.jsonl", format=DstFileFormat.NDJSON), ) def export_people(people: TableFrameSpec) -> None: pass ``` Called tabsdata 3 times (ctrl+o to expand) ⏺ Registered. The publisher's commit triggered the transformer, which triggered the subscriber, all in one execution plan — committed, 0 failures. ⏺ **people.jsonl** is live in `qs-example/output-data/`: ``` ┌──────────────┐ │ full_name │ ├──────────────┤ │ Ada Lovelace │ ├──────────────┤ │ Grace Hopper │ └──────────────┘ ``` Step 4Check Workflow Execution persons.csv people.jsonl conn-input.yaml conn-output.yaml README.txt After your LLM finishes processing your request, you should have a new `people.jsonl` file in the `qs-example/output-data/` folder. Step 5Open the UI Go to [http://localhost:2457](http://localhost:2457) and log in with `admin` / `tabsdata`. You'll see your project, its collections, and the tables the flow just produced. --- # Architecture Overview > Describes Tabsdata's system architecture: the resource layer (Functions as publisher, transformer, subscriber; Tables with versioned schemas) and execution layer (Execution Plans, Transactions, Function Runs), plus deployment providers. Source: https://docs.tabsdata.com/guide/architecture ## System architecture Tabsdata is a distributed, horizontally scalable system that runs on **Kubernetes** and can be deployed across a variety of Kubernetes providers. Tabsdata is organized into two conceptual layers: a **resource layer** and an **execution layer**. The **resource layer** consists of objects that users create to establish the structure of a Tabsdata workflow. These include objects like Tables and Functions, which determine when a workflow runs, what data flows through it, and what work is applied to that data. The **execution layer** consists of objects that Tabsdata creates when it reads and acts on resources from the resource layer. These objects make up the running workflow that performs work on data. This includes objects like Execution Plans, Transactions, and Function Runs. [Server](https://docs.tabsdata.com/guide/server) [Projects](https://docs.tabsdata.com/guide/projects) Resource layer - [Groups](https://docs.tabsdata.com/guide/groups) - [Collections](https://docs.tabsdata.com/guide/collections) - [Functions](https://docs.tabsdata.com/guide/functions) - [Connections](https://docs.tabsdata.com/guide/connections) - [Tables](https://docs.tabsdata.com/guide/tables) Execution layer - [Execution Plan](https://docs.tabsdata.com/guide/execution/execution-plans) - [Transaction](https://docs.tabsdata.com/guide/execution/transactions) - [Function Run](https://docs.tabsdata.com/guide/execution/function-runs) - [Worker](https://docs.tabsdata.com/guide/execution/workers) # Resource Layer ### Functions Functions are Tabsdata's unit of work. A function performs a single discrete step of moving data between external systems and internal Tabsdata Tables. There are three types of Tabsdata Functions: A **publisher** reads from an external system and writes into Tabsdata tables. A **transformer** reads from Tabsdata tables and writes into new Tabsdata tables. A **subscriber** reads from Tabsdata tables and writes out to an external system. Data enters and leaves the server through functions Database postgres Publisher Tabsdata Server Table customers Transformer Table customer\_orders Subscriber Warehouse snowflake ### Tables A unique feature of Tabsdata is that it has its own internal storage for all data produced by functions. Tabsdata tables persistently store the data produced by functions. Every successful function run that returns data creates a new table version that sits alongside all previous versions of the table. persons Tabsdata›Collections›input›persons ### Schema Table schema defines the structure of a table by listing its columns and their data types. | Object | Type | | --- | --- | | first\_name | str | | last\_name | str | | age | i64 | | nationality | str | | city | str | version While Tabsdata tables share some similarities to database tables, they also have some key differences. Rather than requiring a predefined schema, each version inside a table carries its own, allowing a table's schema to evolve across versions. This not only makes Tabsdata resilient to schema drift, but also stores a lineage of the table's evolution for observability and debugging. Table version history is available not only to users for viewing, but also to functions for ingestion. Functions can read historical table versions for use cases that require comparing historical data with incoming data. ## Execution Layer Functions and tables declare how data should be moved. Tabsdata's reads these and builds the actual workflow that ingests, transforms, and loads your data. ### Execution Plans Traditionally, as more steps are added to a data integration workflow, the more complex it is to orchestrate it. Tabsdata does not have this issue. Function are declarative and declare their own dependencies. Any functions that write to tables used by other functions are upstream dependent to them. Any functions that use tables produced by other functions are downstream dependent to them. When a Function is triggered, Tabsdata identifies its downstream dependencies and automatically builds an **Execution Plan**, a DAG (directed acytclic graph) that finds and executes all downstream dependencies. Click any node to trace its dependencies In addition to automatically orchestrating workflows, Execution plans record everything that occurs during execution, capturing metadata such as: - Execution start and stop time - Functions executed - Function code used - Tables read and written - Execution status - Execution logs ## Horizontal Scalability Compute and storage are independently scalable. Each Function runs as an isolated process within the Server's Kubernetes cluster. Storage can be distributed across local file storage or cloud storage providers such as Amazon S3, Azure Blob Storage, and Google Cloud Storage. ## Security Authorization uses a direct permission model, where permissions are assigned directly to users rather than through roles. Credentials are stored separately from Functions in a secure credential store. This includes both Tabsdata's internal credentials and the credentials used by Connections to authenticate with external systems. note Even when credentials are provided inline through the CLI, Tabsdata automatically extracts and stores them in the credential store. ## Deployment Tabsdata runs on **Linux** for production and **macOS** or **Windows** for development. The underlying Kubernetes can be deployed across several providers - **`kminus`**: A lightweight Kubernetes implementation that runs pods as native Python processes instead of Docker containers. The default provider for local development. - **`eks`**: Amazon Elastic Kubernetes Service (EKS). - **`aks`:** Azure Kubernetes Service - **`gke`**: Google Kubernetes Engine ## AI integration Tabsdata includes a built-in **Model Context Protocol (MCP) Server** that allows AI agents to interact with your Tabsdata Server. The MCP Server is backed by a skills library and vector database, providing agents with the context needed to understand and operate on your data workflows. See [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) to get started. There are two ways to use AI in Tabsdata: - A **built-in agent** called Tabby for operational tasks within Tabsdata. - External coding agents, such as **Claude** and **Codex**, connected through the MCP Server. ## Community & contributing [Community Ask questions and talk with other users on Slack. ↗](https://join.slack.com/t/tabsdata-community/shared_invite/zt-322toyigx-ZGFioMV2Gbza4bJDAR7wSQ)[Contribute Report bugs or open a pull request on GitHub. ↗](https://www.github.com/tabsdata/tabsdata/)[License License terms for Tabsdata. ↗](https://tabsdata.com/license) --- # Server > What a Tabsdata server contains, how it runs workflows, and how clients connect to it. Source: https://docs.tabsdata.com/guide/server A **Tabsdata Server** is the top-level Tabsdata process deployed on a machine. It stores and manages Tabsdata resources, executes workflows, stores tabular data, and isolates Tabsdata processes from other processes running on the machine. You can interact with a Server through the Tabsdata UI, CLI, Python API, or MCP integration. ## Resources in a Server The main units of work within Tabsdata are **Functions**, which perform work on data, and **Tables**, which store data produced by Functions. As the the top-level process Tabsdata, Servers store Functions and Tables as well as several organizational layers that nest functions and tables: 1. **Projects** are the top-level workspaces within a Server. Projects isolate their resources and workflows from those in other Projects, allowing multiple teams or workflows to share a single Server without interfering with one another. 2. **Groups** organize resources according to their role in a workflow. For example, a Project might have separate Groups for ingesting source data, cleaning data, and loading processed data. 3. **Collections** define transaction boundaries for their Tables, determining how data is committed or rolled back during an execution. Collections are also credential stores that can provide Functions inside them with the credentials needed to connect to external systems. [Server](https://docs.tabsdata.com/guide/server) [Projects](https://docs.tabsdata.com/guide/projects) [Groups](https://docs.tabsdata.com/guide/groups) [Collections](https://docs.tabsdata.com/guide/collections) [Functions](https://docs.tabsdata.com/guide/functions)[Connections](https://docs.tabsdata.com/guide/connections)[Tables](https://docs.tabsdata.com/guide/tables) ## Connecting to a Server A Tabsdata server uses three dedicated endpoints when running: - A **public endpoint** runs on port `2457` by default. The Tabsdata UI, `tdk` CLI, and Python API use this endpoint to communicate with the Server. - A **private endpoint** runs on port `2458` by default. Tabsdata services use this endpoint for internal communication. - An **AI agent endpoint** runs on port `2459` by default. The Tabsdata Conversational agent, Tabby, uses this endpoint. Using default port configuration, the default local URI for the public endpoint is: `http://localhost:2457` You can connect to a Server using several interfaces: - The **Tabsdata UI** provides a browser-based interface for managing resources, running workflows, and inspecting results. - The **`tdk` CLI** provides command-line access to the Server. - The **Python API** provides programmatic access from Python applications. - The **Tabsdata MCP integration** allows AI agents to interact with the Server through tools. ## Managing a Server The `tdkserver` CLI manages the Server lifecycle, including initializing, creating, starting, and stopping a Server. See [Deploy Tabsdata on k-minus](https://docs.tabsdata.com/server-admin/kminus) for Quickstart and the complete local Server lifecycle. For a managed Kubernetes deployment, choose [AWS EKS](https://docs.tabsdata.com/server-admin/eks-install), [Azure AKS](https://docs.tabsdata.com/server-admin/aks-install), or [Google GKE](https://docs.tabsdata.com/server-admin/gke-install). To connect an AI agent, see [Connect to MCP](https://docs.tabsdata.com/server-admin/connect-agent). --- # Projects > How Projects organize and isolate resources within a Tabsdata Server. Source: https://docs.tabsdata.com/guide/projects A **Project** is the top-level workspace inside a Tabsdata Server. Projects organize related workflows and their resources while isolating them from other Projects on the same Server. This allows multiple teams to share a single Server without interfering with each other's work. ## Working in a Project Most Tabsdata operations are performed within the context of a Project. You can set a default Project for the `tdk` CLI using: ```bash tdk set --proj sales_analytics ``` The selected Project remains the default for subsequent commands. Commands that support the `--proj` option can target a different Project without changing the default. ## Managing Projects Creating a Project establishes an empty workspace within an existing Server. Groups and Collections can then be created to organize the Functions and Tables that make up your workflows. Deleting a Project removes the Project and its resources without affecting other Projects on the Server. See [Create a Project](https://docs.tabsdata.com/tutorials/managing-projects/create), [Update a Project](https://docs.tabsdata.com/tutorials/managing-projects/update), and [Delete a Project](https://docs.tabsdata.com/tutorials/managing-projects/delete) for step-by-step instructions. --- # Groups > How Groups organize Collections by data direction and transformation stage. Source: https://docs.tabsdata.com/guide/groups A **Group** is an organizational layer inside a tabsdata project that organizes workflow steps according to their role in the workflow. By default, all tabsdata projects come with three groups: **sources, default, and destinations** 1. the **sources** group stores everything related to ingesting data from external systems 2. the **default** group stores everything related to transforming data inside tabsdata 3. the **destinations** group stores everything related to loading data into external systems Groups separate workflow resources based on type of data movement Database postgres Tabsdata Server `sources` Publisher load\_users Table users `default` Transformer rank\_users Table top\_users `destinations` Subscriber sync\_users Warehouse snowflake ## System Groups The **sources, default,** and **destinations** groups are special groups that must exist in your workflow. They are dedicated spaces for registering publisher and subscriber functions and cannot be modified, duplicated, or removed from your project. Outside these three system groups, users can create any number of groups to organize their transformer functions. ## Use Cases for Groups Groups are useful for separating workflow steps that happen across tabsdata server boundaries (ingestion, load) from workflow steps that happen inside the tabsdata server (transform). Groups are also useful for separating data transformation workflow steps by the type of data transformation being done. For example, a popular data design pattern is the [Medallion Architecture](https://www.databricks.com/blog/what-is-medallion-architecture) for Data Integration, which breaks data transformation into three layers: - **Bronze:** Layer that ingests raw source data from external systems with the goal of having as close of a representation to data in the external system - **Silver:** Layer that cleans, dedupes, normalizes, and joins related entities - **Gold:** Layer that applies granular, business-specific transformations and creates "consumer-ready" tables. Users looking to implement this design pattern in tabsdata can do so with five groups within Tabsdata: two built-in groups (sources, destinations) and three custom groups (bronze, silver, gold). External systems Group`sources`Publishers Ingest from external systems. Built in Group`bronze`Transformers Raw data, kept as close to the external system as possible. User created Group`silver`Transformers Cleaned, deduped, normalized, and joined. User created Group`gold`Transformers Business-specific transformations, consumer-ready tables. User created Group`destinations`Subscribers Write out to external systems. Built in External systems ## Visualizing Groups The Projects view in the Tabsdata UI shows how data moves between Groups, including the number of Tables, rows, and bytes moving through each stage. ## Managing Groups The `sources`, `default`, and `destinations` Groups are created with every Project. These system Groups cannot be renamed or deleted. Custom Groups can be created, renamed, and deleted within a Project. See [Create a Group](https://docs.tabsdata.com/tutorials/managing-groups/create), [Update a Group](https://docs.tabsdata.com/tutorials/managing-groups/update), and [Delete a Group](https://docs.tabsdata.com/tutorials/managing-groups/delete) for step-by-step instructions. --- # Collections > How Collections contain Functions, Tables, and Connections and define transaction boundaries. Source: https://docs.tabsdata.com/guide/collections A **Collection** is an organizational layer inside of a group that stores tables and functions. Collections help manage credentials for functions and define transaction boundaries for tables. ### Collections for Credential Storage A Function is responsible for moving, and sometimes transforming, data from one place to another. Publishers and Subscribers that read from or write data to external systems also need a way to securely access sensitive credentials for those systems. Instead of hardcoding credentials directly into Function logic, Tabsdata abstracts credential management into a separate resource called a **Connection**. A Connection is defined in a `.yaml` file and bound to a **Collection**. Any Function registered to the same Collection can access those credentials at runtime. This allows users to store their credentials once and then securely provision those credentials to as many functions that need them. [Learn more about connections here](https://docs.tabsdata.com/guide/connections). ### Collections as Transaction Boundaries Triggering a Tabsdata Function creates an Execution Plan that executes the triggering Function along with any functions downstream dependent to it. Because an Execution Plan can span multiple Functions, Tabsdata needs to determine which data should be committed or rolled back if the Execution Plan fails partway through. Collections act as transaction boundaries within an Execution Plan. Tabsdata treats each participating Collection as a separate transaction and commits data to its Tables only when all participating Functions within that Collection complete successfully. If a Function within the transaction fails, Tabsdata rolls back all changes made to Tables within that transaction. ## Managing Collections After creating a Collection, users can register Functions into that collection. If a collection is created inside the **sources** or **destinations** group, it can only store publisher or subscriber functions respectively and requires a **connection** to be bound Deleting a Collection removes the Collection and the resources it contains without affecting other Collections in the Project. See [Create a Collection](https://docs.tabsdata.com/tutorials/managing-collections/create), [Update a Collection](https://docs.tabsdata.com/tutorials/managing-collections/update), and [Delete a Collection](https://docs.tabsdata.com/tutorials/managing-collections/delete) for step-by-step instructions. --- # Connections > How Connections provide Publisher and Subscriber Functions with access to external systems. Source: https://docs.tabsdata.com/guide/connections A **Connection** is a resource that stores the credentials required to access an external system. Publisher and Subscriber Functions need credentials to connect to external systems to read and write data, but they do not store these sensitive credentials directly in their code. Instead, that configuration is defined in a Connection. Connections are `.yaml` files that contain relevant credentials and are attached to Collections, allowing all functions within that collection to access the required credentials during execution. ## Connection Structure Connections are defined using `.yaml` files. The fields in a Connection depend on the external system and Connection type. For example, an S3 Connection stores the bucket, region, and credentials needed to access files in an S3 bucket. conn-s3.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.s3:S3SrcConn spec: bucket: 'acme-ingest' region: 'us-east-1' credentials: kind: awsAccessSecretKeyCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:AwsAccessSecretKey spec: access_key_id: '$secret:AWS_ACCESS_KEY_ID' secret_access_key: '$secret:AWS_SECRET_ACCESS_KEY' ``` - required - required - required - optional ## Connections and Collections Connections are bound to Publisher and Subscriber Collections. A **Publisher Collection** binds a Publisher to an input Connection. The Publisher defines what data to read from the external system and what Tables to create in Tabsdata, while the Connection provides the configuration and credentials needed to access that system during the Publisher's execution. A **Subscriber Collection** binds a Subscriber to an output Connection. The Subscriber defines which Tabsdata Tables to read and what data to write, while the Connection provides the configuration and credentials needed to access the destination system during the Subscriber's execution. ## Credential Security When configuring a connection , a credential can be passed directly as `secret:`, or as a reference to an environment variable, written as `$secret:`. Regardless of the method used, when the connection file is registered into Tabsdata, Tabsdata **dehydrates** the document by extracting any secrets into a separate secret store and storing a reference to the secret in the file. note For security, once credentials are stored in Tabsdata, they can only be accessed by a Tabsdata Function at runtime. --- # Functions > How Tabsdata Functions define, connect, and execute the steps in a data workflow. Source: https://docs.tabsdata.com/guide/functions/ A Tabsdata **Function** is Tabsdata's operational unit of work and defines a single step that reads data from or writes data to a Tabsdata Table. Tabsdata Functions build on the familiar model of a regular Python function, where data comes in through a parameter, the function performs work on that parameter, and the result is returned through the return statement. Tabsdata extends this model to data integration by abstracting data sources and destinations into the simple parameters and return statement of a Python function. Users define their inputs, which can be a SQL query against a database, a file in object storage, or a Tabsdata Table. They then define where they want the output data to be written, whether that is a database table, a file in object storage, or another Tabsdata Table. Tabsdata handles retrieving, converting, and moving the data between those systems. While the data is being passed through the Function, Tabsdata materializes it as a DataFrame object that users can transform through an expression-based API. This allows users to work with tabular data independently of where it is physically stored or how it needs to be moved between systems. ## Function types There are three types of Tabsdata functions: 1. **publishers:** read data from external systems and _publish_ data into Tabsdata Tables 2. **transformers:** read data from Tabsdata Tables and write data into Tabsdata Tables 3. **subscribers:** read data from Tabsdata Tables and _subscribe_ data into Tabsdata Tables Publishers External SystemTabsdata Table Reads an external system and writes Tabsdata Tables. Transformers Tabsdata TableTabsdata Table Reads Tabsdata Tables and writes new Tabsdata Tables. Subscribers Tabsdata TableExternal System Reads Tabsdata Tables and writes an external system. ## Inputs and outputs The decorator maps declared inputs to the Python callable's parameters. It also maps returned values to output Tables or to destination slots in an external system. The number and order of parameters and returned values must match the corresponding declarations. Publishers and Subscribers use a Connector for the system-specific read or write operation. The Collection's [Connection](https://docs.tabsdata.com/guide/connections) supplies the configuration and credentials used by that Connector. Transformers work entirely with Tabsdata Tables and do not use an external Connection. Output Tables belong to the Collection where the Function is registered. Input Table references can identify Tables in other Collections within the same Project, allowing Functions to connect into a larger workflow. ## Triggers A [Trigger](https://docs.tabsdata.com/guide/execution-and-triggers) determines when a Function starts. A Transformer or Subscriber without `trigger_by` runs whenever one of its input Tables commits a new version. A Publisher has no input Tables, so without `trigger_by` it runs only when triggered manually. Set `trigger_by` to a schedule, or to one or more Table references, to control what starts the Function. ## Registration Registration sends the Function definition, source code, and required bundle metadata to the Server. A Function name must be unique within its Collection. Register a decorated callable from a Python file with: ```bash tdk fn register --coll --path :: ``` Pass `--update` when the same command should update an existing Function with that name. Registration records the definition; it does not execute the Function. See [Register a Function](https://docs.tabsdata.com/tutorials/managing-functions/register) for the full procedure. ## Execution A manual Trigger can start any registered Function: ```bash tdk fn trigger --coll --name ``` The Server evaluates the Function's dependencies and creates an Execution Plan. Each Function included in that plan gets a Function Run, and a [Worker](https://docs.tabsdata.com/guide/execution/workers) executes each attempt. A successful run that produces Tables creates new [Table Versions](https://docs.tabsdata.com/guide/tables); the Server keeps the registered Function definition after the Worker exits. Function code does not run in the CLI process that sends the command. The CLI requests the Trigger, while the Server coordinates and records the execution. --- # Publishers > How @publisher reads an external system, maps source slots to function parameters, and writes versioned Tabsdata tables. Source: https://docs.tabsdata.com/guide/functions/publishers A publisher reads data from an external system and publishes it to Tabsdata tables. Each run reads the source, calls the function, and commits the returned tables together. Unlike a stream publisher, a regular publisher does not keep reading. It reads the source only when it runs. The publisher must be registered in a source collection whose connection matches the source. ## In Action Tabsdata server collection · world Database PostgreSQL PostgreSQL · external Reachable Connection conn-postgres.yaml Publisher read\_world Table country //world No versions yet Table city //world No versions yet ## Parameters ```python publisher( source, output_tables, trigger_by=None, on_tables=None, ) ``` | Parameter | Type | | | --- | --- | --- | | `source` | `Src` | required | | `output_tables` | `list[str]` | required | | `trigger_by` | `CronTrigger \| TablesTrigger` | optional | | `on_tables` | `list[OnTablesAction]` | optional | #### `source` The source to read. The source class decides what is read and how many slots are passed to the function. A SQL source has one slot per query. A file source can read several files into one slot. The source does not contain the server address or credentials. Those come from the connection of the collection where the publisher is registered. The page for each connector lists its source classes and settings. #### `output_tables` A non-empty list of local table names, one for each value the function returns. A name must start with a letter, use only letters, numbers, and underscores, and be no longer than 64 characters. The source slot count and output table count do not have to match. A publisher can combine several source slots into one table or derive several tables from one slot. #### `trigger_by` The trigger that starts the publisher automatically: a `CronTrigger` or `TablesTrigger`. Without `trigger_by`, the publisher runs only when triggered manually. See [Execution & Triggers](https://docs.tabsdata.com/guide/execution-and-triggers). #### `on_tables` Actions to run on the output tables after the publisher succeeds, such as [data quality](https://docs.tabsdata.com/guide/data-quality) checks. ## Publisher Code Anatomy read\_world.py ``` 1from tabsdatak.api import publisher, TableFrameSpec2from tabsdatak.conn.postgres import PostgresSrc3 4 1@publisher(2 source=PostgresSrc(3 queries=[8 "SELECT * FROM country",9 "SELECT * FROM city",10 ],11 ),4 output_tables=["country", "city"],13)56def read_world(country: TableFrameSpec, city: TableFrameSpec) -> tuple[TableFrameSpec, TableFrameSpec]:7 return (country, city) ``` - Defines the source and output Tables for the Publisher. - Reads data from the external system. The Collection's Connection provides the address and credentials. - SQL queries that read the country and city data from Postgres. - Tabsdata Tables where the returned data is written. - Receives the query results and returns the data to write to the output Tables. - The `country` and `city` arguments receive the query results in the order listed in `queries`. - The returned data is written to the Tables listed in `output_tables`, in that order. Returning `None` for a Table leaves its current version unchanged. ## The function A publisher function has one parameter per source slot and returns one frame per output table. - **Parameters.** Source slots are passed to the function in source order. Annotate a parameter as `TableFrameSpec` for one frame or `TableFramesSpec` for a list of frames. A slot can be `None` when there is no batch for it. - **Context.** The function can have a final `ctx: TrxCtx` parameter. Use `ctx` to keep values between successful runs. For example, an incremental SQL publisher can save its last processed ID with `ctx.set_attr()` and use it in the next query. - **Return value.** Return values map to `output_tables` in order. One output can be returned on its own; several outputs are returned as a tuple. Returning `None` for one output leaves that table's current version unchanged. - **Slot counts.** The number of source slots does not have to match the number of output tables. The function can combine several inputs into one table or create several tables from one input. Tabsdata checks the slot counts and function signature when the publisher is registered. ## Connections and collections All publishers in a source collection use the same connection. Each publisher has its own source settings, code, and output tables. The connection is created with the collection. Updating a connection changes how every publisher in that collection reaches the external system. ## Lifecycle - **Register.** `tdk fn register` uploads the publisher. Its collection and connection must already exist. - **Run.** A manual, cron, or table trigger starts a transaction. The source is read before the function runs. - **Commit.** A successful run commits the output tables together. Each commit adds a table version. - **Update.** `tdk fn update` replaces the code and decorator settings. - **Delete.** Deleting the function stops future runs but does not delete its output tables or their versions. For step-by-step guides on building a publisher for a specific external system, see [Publishers](https://docs.tabsdata.com/tutorials/publishers/) under How-to Guides. --- # Stream Publishers > How @stream_publisher reads a CDC log or a Kafka topic continuously, and how the batch parameters decide when the function runs. Source: https://docs.tabsdata.com/guide/functions/stream-publishers A stream publisher reads data continuously from a stream source, such as a CDC log or a Kafka topic, and publishes it into Tabsdata tables in batches. A regular publisher runs when it is triggered and reads what is in the source at that moment. A stream publisher has no trigger. Registering it starts a worker that keeps reading the source, collects what it reads, and runs the function once per batch. The `batch_*` parameters of `@stream_publisher` decide when a batch is complete. These sources work with `@stream_publisher`: | Source | Reads | Page | | --- | --- | --- | | `PostgresCdcStreamSrc` | Postgres changes, from the WAL | [Postgres CDC](https://docs.tabsdata.com/guide/connectors/postgres-cdc) | | `MySQLCdcStreamSrc` | MySQL changes, from the binlog | [MySQL CDC](https://docs.tabsdata.com/guide/connectors/mysql-cdc) | | `Db2CdcStreamSrc` | Db2 changes, from the SQL Replication CD tables | [Db2 CDC](https://docs.tabsdata.com/guide/connectors/db2-cdc) | | `KafkaStreamSrc` | Messages from Kafka topics | [Kafka](https://docs.tabsdata.com/guide/connectors/kafka) | All of them use the batch parameters described on this page. ## In Action Tabsdata server collection · events Kafka topic Kafka Kafka · external Reachable Connection conn-kafka.yaml Stream Publisher stream\_page\_views Table page\_views //events No versions yet ## Example stream\_sales\_changes.py ```python from tabsdatak.api import TableFrameSpec, TableFramesSpec, stream_publisher from tabsdatak.conn.postgres import PostgresCdcStreamSrc from tabsdatak.tableframe.functions import concat @stream_publisher( source=PostgresCdcStreamSrc( tables=["public.orders", "public.customers"], replication_slot="tabsdata_sales", start_from="full", ), output_tables=["orders_changes", "customers_changes"], batch_max_secs=300, batch_max_rows=50_000, ) def stream_sales_changes( orders: TableFramesSpec, customers: TableFramesSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return concat(orders), concat(customers) ``` - `source` is the stream source. Its connection comes from the collection the function is registered in, as for any publisher. - `output_tables` are the Tabsdata tables the function writes, in the order the function returns them. - `batch_max_secs` and `batch_max_rows` decide when a batch is handed to the function: here, every 5 minutes, or sooner once 50,000 changes are waiting, but never more often than once a minute. - The function receives one parameter per source slot. Each parameter is a list of frames: the data read for that slot since the previous batch. `concat` combines them into one frame. ## Parameters ```python stream_publisher( source, output_tables, *, batch_min_secs=None, batch_max_secs=None, batch_max_rows=None, batch_max_bytes=None, batch_idle_secs=None, on_tables=None, flush_interval_mins=None, ) ``` | Parameter | Type | | | --- | --- | --- | | `source` | `StreamSrc` | required | | `output_tables` | `list[str]` | required | | `batch_min_secs` | `int` | optional | | `batch_max_secs` | `int` | optional | | `batch_max_rows` | `int` | optional | | `batch_max_bytes` | `int` | optional | | `batch_idle_secs` | `int` | optional | | `on_tables` | `list[OnTablesAction]` | optional | | `flush_interval_mins` | `int` | deprecated | `source` and `output_tables` can be given by position or by name. The other parameters must be given by name. #### `source` The stream source to read: one of the sources in the table above. #### `output_tables` The tables the function writes, as a non-empty list of table names. The number of output tables does not have to match the number of source slots. #### `on_tables` Actions to run on the output tables, such as [data quality](https://docs.tabsdata.com/guide/data-quality) checks, as for a regular publisher. ## Batch parameters A batch is everything the worker has read since the previous batch. Each batch runs the function once, and the function's output tables are committed together: every batch adds one version to each output table. | Parameter | Default | Description | | --- | --- | --- | | `batch_min_secs` | `60` | The shortest time between two batches, in seconds. Never less than 60. | | `batch_max_secs` | none | Seconds after the previous batch at which data that is waiting is handed to the function. | | `batch_max_rows` | none | Rows waiting that make a batch due. | | `batch_max_bytes` | none | Bytes waiting that make a batch due. | | `batch_idle_secs` | none | Seconds after the previous batch at which, if no data is waiting, the function runs with an empty batch. | At least one of `batch_max_secs`, `batch_max_rows` and `batch_max_bytes` is required. ### When a batch is handed to the function The worker checks after every read from the source. Counting from the previous batch (or from when the worker started): - **When data is waiting**, a batch is due once `batch_min_secs` have passed **and** any of these is true: - `batch_max_secs` have passed, - at least `batch_max_rows` rows are waiting, - at least `batch_max_bytes` bytes are waiting. - **When no data is waiting**, nothing happens, unless `batch_idle_secs` is set. Once `batch_idle_secs` have passed, the function runs with an empty batch. As a result: - `batch_min_secs` is a floor for every other setting. If 50,000 rows arrive 10 seconds after a batch, they wait until `batch_min_secs` have passed, and more rows keep joining them in the meantime. - `batch_max_rows` and `batch_max_bytes` make a batch due; they do not cap its size. A batch holds everything that is waiting when it closes, which can be more than the threshold. - Rows and bytes are counted over all the slots of the source together, not per table. For a CDC source, one row is one change. Bytes are approximate: the size of the data files already written plus an estimate of what is still in memory. - `batch_max_secs` counts from the previous batch, not from when the data arrived. After a quiet period longer than `batch_max_secs`, the first data to arrive is handed over right away. Batches are processed one at a time, in order. Tabsdata looks for new batches about every 30 seconds, so a batch can start up to that long after it is due. If the function takes longer than the batch interval, batches queue up and are processed in turn. ### Validation The batch parameters are checked when the function is decorated. A violation raises error `API_29`: - `batch_min_secs`, `batch_max_secs` and `batch_idle_secs` must be whole numbers of seconds, at least `60`. - `batch_max_rows` and `batch_max_bytes` must be whole numbers, at least `1`. - `batch_max_secs` must be at least `batch_min_secs`. - `batch_idle_secs` must be at least `batch_max_secs`, or, without `batch_max_secs`, at least `batch_min_secs`. - At least one of `batch_max_secs`, `batch_max_rows` and `batch_max_bytes` must be set. ### Choosing the parameters | Goal | Parameters | Result | | --- | --- | --- | | Steady, bounded latency | `batch_max_secs=120` | A batch every 2 minutes while data arrives. | | Size first, with a time bound | `batch_max_rows=100_000, batch_max_secs=900` | A batch as soon as 100,000 rows are waiting, and at least every 15 minutes. | | Large batches, few runs | `batch_min_secs=600, batch_max_secs=3600` | At most one batch every 10 minutes, and at least one an hour while data arrives. | | Runs even when the source is quiet | `batch_max_secs=300, batch_idle_secs=3600` | A batch every 5 minutes while data arrives, and an empty run every hour when nothing arrives, for example to keep downstream functions on a schedule. | ### Buffer settings and batch parameters Stream sources also have buffer settings in their `src_cfg`, such as `tabsdata.cdc.buffer.max_rows` or `tabsdata.kafka.buffer.max_rows`. Buffer settings and batch parameters work at different levels: - **Buffer settings** decide when the worker writes what it has read from memory to a data file. They bound the worker's memory. - **Batch parameters** decide when the files written so far, and anything still in memory, are handed to the function as one batch. Most streams only need the batch parameters. The buffer settings of each source are listed on its connector page. ## Stream Publisher Code Anatomy stream\_sales\_changes.py ``` 1from tabsdatak.api import TableFrameSpec, TableFramesSpec, stream_publisher2from tabsdatak.conn.postgres import PostgresCdcStreamSrc3from tabsdatak.tableframe.functions import concat4 5 1@stream_publisher(2 source=PostgresCdcStreamSrc(3 tables=["public.orders", "public.customers"],9 replication_slot="tabsdata_sales",10 start_from="full",11 ),4 output_tables=["orders_changes", "customers_changes"],5 batch_max_secs=300,14 batch_max_rows=50_000,15)67def stream_sales_changes(17 orders: TableFramesSpec,18 customers: TableFramesSpec,19) -> tuple[TableFrameSpec, TableFrameSpec]:8 return concat(orders), concat(customers) ``` - Defines the stream source, output Tables, and the conditions that make a batch due. - Reads changes continuously from Postgres. The Collection's Connection provides the server address and credentials. - Each tracked table is one source slot, in the order listed. - Tabsdata Tables that receive the values returned by the function. - A batch is due after five minutes or when 50,000 rows are waiting, whichever happens first after the minimum interval. - Receives one list of frames per source slot and returns one frame per output Table. - The arguments receive the frames collected for each source slot since the previous batch. - `concat` combines each slot's frames. The returned values map to `output_tables` in order. ## The function A stream publisher function has one parameter per source slot, annotated `TableFramesSpec`, and returns one `TableFrameSpec` per output table. - **Slots.** A CDC source has one slot per entry in its `tables`, in the same order. A Kafka source has a single slot. - **Parameters.** Each parameter receives a list of frames with the data read for that slot since the previous batch. For a CDC source, the list holds at most one frame. For Kafka, it can hold several, possibly with different schemas; combine them with `concat(frames, how="diagonal_relaxed")` if their columns can differ. - **Empty batches.** A slot with no new data receives an empty list. `concat` of an empty list returns `None`. - **Return value.** A tuple with one element per output table, in the order of `output_tables`. With a single output table, the frame can be returned on its own. Returning `None` for a table means no new data: the table gets a version without data, and its previous data stays current. - **No other parameters.** A stream publisher cannot declare a `ctx` parameter or any parameter beyond its slots. `concat` is imported from `tabsdatak.tableframe.functions`. ## Lifecycle - **Register.** `tdk fn register` starts the worker. The function must be registered in a source collection whose connection matches the source; see the source's connector page. - **Disable and enable.** `tdk stream disable` stops the worker; `tdk stream enable` starts it again, from where it stopped. `tdk stream list` shows the stream publishers and their state. - **Update.** `tdk fn update` restarts the worker with the new code and settings. It resumes from where it stopped. - **Delete.** Deleting the function stops the worker. Check the source's connector page for anything left on the source system, such as a Postgres replication slot. - **Failures.** If the worker fails, Tabsdata restarts it, waiting one minute after the first failure and doubling the wait after each further failure, up to 30 minutes. - **Start over.** `tdk fn update --reset-stage` discards the stream's recorded position. The stream starts again as if it were new: from the source's starting position, or with a new full load for a CDC source with `start_from="full"`. On a restart, the stream reads again whatever it had not yet handed to the function. A CDC stream publishes every change once, even across restarts. A Kafka stream is at least once: after a restart, some messages may be published again. ### CDC full load The full load of a CDC source (`start_from="full"`) is not subject to the batch parameters. It is handed to the function as one batch, covering every tracked table, as soon as it has been read. If the load fails or is interrupted, it is discarded and runs again from the start. The batch parameters apply to the changes that follow. ## `flush_interval_mins` (deprecated) `flush_interval_mins` was the only batch setting before 2.2.0. It is deprecated, and is converted to `batch_max_secs = flush_interval_mins * 60`, so it must be at least `1`. - It cannot be combined with any `batch_*` parameter. Doing so raises error `API_30`. - It raises a Python `DeprecationWarning` when the function is decorated, and the worker logs a warning when it starts. Python hides `DeprecationWarning` by default, so the worker log is where the warning shows. To migrate, replace `flush_interval_mins=N` with `batch_max_secs=N * 60`. --- # Transformers > How @transformer reads Tabsdata tables, maps versions to function parameters, and commits transformed output tables. Source: https://docs.tabsdata.com/guide/functions/transformers A transformer reads Tabsdata tables, transforms their data, and writes the result back to Tabsdata. It does not read or write an external system, so it does not use a connection. ## In Action collection · world collection · insights · data product Table country //world Committed Table city //world Committed Transformer find\_big\_cities Table big\_cities //insights No versions yet ## Parameters ```python transformer( input_tables, output_tables, trigger_by=None, on_tables=None, ) ``` | Parameter | Type | | | --- | --- | --- | | `input_tables` | `list[str]` | required | | `output_tables` | `list[str]` | required | | `trigger_by` | `CronTrigger \| TablesTrigger \| AllTablesTrigger` | optional | | `on_tables` | `list[OnTablesAction]` | optional | #### `input_tables` A non-empty list of table references in the form `[collection/]table[@versions]`. Omit the collection for a table in the transformer's collection. Use `@versions` to select a version, a version range, a comma-separated list, or `NEW`. Each entry maps to one function parameter. A single-version reference receives a `TableFrameSpec`; a reference that selects several versions receives a `TableFramesSpec`. #### `output_tables` A non-empty list of local table names, one for each value returned by the function. The output tables belong to the transformer's collection. Each name must start with a letter, contain only letters, numbers, and underscores, and be no longer than 64 characters. #### `trigger_by` The trigger that starts the transformer automatically: a `CronTrigger`, `TablesTrigger`, or `AllTablesTrigger`. Without `trigger_by`, a commit to any input table triggers the transformer. A transformer can also be run manually. The trigger decides when the function runs. The version selector in `input_tables` decides which versions it reads. See [Execution & Triggers](https://docs.tabsdata.com/guide/execution-and-triggers). #### `on_tables` Actions to run on the output tables after the transformer succeeds, such as [data quality](https://docs.tabsdata.com/guide/data-quality) checks. ## Transformer Code Anatomy find\_big\_cities.py ``` 1from tabsdatak.api import transformer, TableFrameSpec2import tabsdatak.api.tableframe as tdf3 4 1@transformer(2 input_tables=["world/country", "world/city"],3 output_tables=["big_cities"],8)45def find_big_cities(country: TableFrameSpec, city: TableFrameSpec) -> tuple[TableFrameSpec]:6 countries = country.rename({"name": "country_name"}).drop(11 ["population", "continent"]12 )13 joined = city.join(countries, left_on="country_code", right_on="code")7 return (joined.filter(tdf.col("population") > 3_000_000),) ``` - Defines the input and output Tables for the Transformer. - Tabsdata Tables to read. Use `collection/table` to reference a Table in another Collection. - Tables where the transformed data is written. These belong to the Transformer's Collection. - Reads the country and city data, joins it, and returns cities with a population above three million. - The arguments receive the data from `input_tables` in the order listed. An argument is `None` if the requested Table version does not exist. - Renames the country name column, removes unused columns, and joins the country and city data. - The returned data is written to the Tables listed in `output_tables`, in that order. Returning `None` for a Table leaves its current version unchanged. ## The function A transformer function has one parameter per input table and returns one frame per output table. - **Parameters.** Inputs are passed to the function in `input_tables` order. Annotate a parameter as `TableFrameSpec` for one frame or `TableFramesSpec` for a list of frames. An input can be `None` when the requested upstream batch is absent. - **Context.** The function can have a final `ctx: TrxCtx` parameter to keep values between successful runs. - **Return value.** Return values map to `output_tables` in order. One output can be returned on its own; several outputs are returned as a tuple. Returning `None` for one output leaves that table's current version unchanged. Tabsdata checks the input count, output count, and function signature when the transformer is registered. ## Collections and table versions A transformer can read tables from its own collection or another collection. Its output tables are always local to the collection where the transformer is registered. Table references are resolved when a run starts. When a trigger starts a larger execution plan, Tabsdata passes the versions produced for that transaction through the plan so downstream transformers read a consistent set of data. ## Lifecycle - **Register.** `tdk fn register` uploads the transformer to its collection. No connection is required. - **Run.** A manual, scheduled, or table trigger starts a transaction and resolves the requested input versions. - **Commit.** A successful run commits the output tables together. Each commit adds a table version. - **Update.** `tdk fn update` replaces the code and decorator settings. - **Delete.** Deleting the function stops future runs but does not delete its output tables or their versions. For step-by-step guidance on building a transformer, see [Transformers](https://docs.tabsdata.com/guide/transformers) under How-to Guides. --- # Subscribers > How @subscriber reads Tabsdata tables, maps function returns to destination slots, and writes to an external system. Source: https://docs.tabsdata.com/guide/functions/subscribers A subscriber reads Tabsdata tables and writes to an external system. The function can filter, rename, or otherwise prepare the data before it is written. A subscriber belongs to a destination collection. That collection's connection provides the address and credentials for the destination. ## In Action Tabsdata server collection · warehouse Table vendors //sales Committed Table items //sales Committed Connection conn-mysql-dest.yaml Subscriber write\_sales Database MySQL MySQL · external Not yet synced ## Parameters ```python subscriber( destination, input_tables, trigger_by=None, ) ``` | Parameter | Type | | | --- | --- | --- | | `destination` | `Dest` | required | | `input_tables` | `list[str]` | required | | `trigger_by` | `CronTrigger \| TablesTrigger \| AllTablesTrigger` | optional | #### `destination` The destination to write to. The destination class decides where the data is written and how many slots the function must return. The destination uses the connection of the collection where the subscriber is registered. The page for each connector lists its destination classes and settings. #### `input_tables` A non-empty list of table references in the form `[collection/]table[@versions]`. Omit the collection for a table in the subscriber's collection. Use `@versions` to select a version, a version range, a comma-separated list, or `NEW`. Each entry maps to one function parameter, in list order. #### `trigger_by` The trigger that starts the subscriber automatically: a `CronTrigger`, `TablesTrigger`, or `AllTablesTrigger`. Without `trigger_by`, a commit to any input table triggers the subscriber. A subscriber can also be run manually. The trigger decides when the function runs. The version selector in `input_tables` decides which data it sends. See [Execution & Triggers](https://docs.tabsdata.com/guide/execution-and-triggers). ## Subscriber Code Anatomy sales-sub.py ``` 1from tabsdatak.api import subscriber, TableFrameSpec2import tabsdatak.api.tableframe as tdf3from tabsdatak.conn.mysql import MySQLDest4 5 1@subscriber(2 destination=MySQLDest(8 tables=["vendors", "items"],9 if_table_exists="replace",10 ),3 input_tables=["sales/vendors", "sales/items"],12)45def write_sales(vendors: TableFrameSpec, items: TableFrameSpec):6 vendors = vendors.filter(tdf.col("active") == 1)7 return (vendors, items) ``` - Defines the input Tables and destination for the Subscriber. - Writes data to the external system. The Collection's Connection provides the address and credentials. - Tabsdata Tables to read. This Subscriber reads `vendors` and `items` from the `sales` Collection. - Receives the input data and returns the data to write to MySQL. - The `vendors` and `items` arguments receive the data from `input_tables` in the order listed. - Filters the vendors to include only active vendors. - The returned data is written to the destination Tables in the order listed in `tables`. Returning `None` for a Table skips the write. ## The function A subscriber function has one parameter per input table and returns one frame per destination slot. - **Parameters.** Inputs are passed to the function in `input_tables` order. Each parameter is a `TableFrameSpec`. An input can be `None` when the requested upstream batch is absent. - **Context.** The function can have a final `ctx: TrxCtx` parameter to keep values between successful runs. - **Return value.** Return values map to the destination slots in destination order. One slot can be returned on its own; several slots are returned as a tuple. Returning `None` for a slot skips that write. - **Slot counts.** The number of input tables does not have to match the number of destination slots. The function can combine several inputs into one write or create several destination tables from one input. Tabsdata checks the slot counts and function signature when the subscriber is registered. ## Connections and collections All subscribers in a destination collection use the same connection. Each subscriber has its own input tables, destination settings, and code. Input tables can come from the subscriber's collection or another collection. Updating the collection's connection changes how every subscriber in that collection reaches the external system. ## Lifecycle - **Register.** `tdk fn register` uploads the subscriber. Its collection and connection must already exist. - **Run.** A manual, scheduled, or table trigger starts a transaction and resolves the requested input versions. - **Write.** The function's returned frames are handed to the destination. The transaction succeeds only after the destination write succeeds. - **Update.** `tdk fn update` replaces the code and decorator settings. - **Delete.** Deleting the function stops future writes. It does not remove data already written to the external system. For step-by-step guides on building a subscriber for a specific external system, see [Subscribers](https://docs.tabsdata.com/tutorials/subscribers/) under How-to Guides. For inserts, updates, and deletes into Snowflake, BigQuery, or Databricks, see [Merging into SQL destinations](https://docs.tabsdata.com/guide/functions/sql-merge). --- # Merging into SQL destinations > How Subscriber Functions apply inserts, updates, and deletes to Snowflake, BigQuery, and Databricks tables. Source: https://docs.tabsdata.com/guide/functions/sql-merge When Subscribing data, you may need to dynamically update, delete, and insert records within a single function. Tabsdata supports **sql merge** writes. Users can specify key columns to match records and an operation column that tells Tabsdata whether to insert, update, or delete each row. The Subscriber returns the changes, and the connector applies them to the destination table. Merge writes are supported by the [Snowflake](https://docs.tabsdata.com/guide/connectors/snowflake), [BigQuery](https://docs.tabsdata.com/guide/connectors/bigquery), and [Databricks](https://docs.tabsdata.com/guide/connectors/databricks) connectors. ## Merge configuration To configure a merge, set `if_table_exists="merge"` on the destination and pass a `merge_cfg` list containing the key and operation columns for each table. For example, this Subscriber reads order changes from Tabsdata and applies them to `ORDERS` in Snowflake: ```python from tabsdatak.api import subscriber, TableFrameSpec from tabsdatak.conn.snowflake import MergeCfg, SnowflakeDest @subscriber( destination=SnowflakeDest( tables=["ORDERS"], if_table_exists="merge", merge_cfg=[MergeCfg(on=["order_id"], op_column="op")], schema_evolution="update", dest_cfg={"snowflake.schema_evolution_engine": "iceberg"}, ), input_tables=["capture/order_changes"], ) def load_orders(changes: TableFrameSpec) -> TableFrameSpec: return changes ``` Here, `on=["order_id"]` matches records by order ID, while `op_column="op"` identifies the column containing the operation. The Function returns the changes from `capture/order_changes` for the connector to write. Because a Subscriber can write to several tables, the `MergeCfg` entries must follow the order of `tables`, and both lists must have the same length. Use a list even for a single table. The `merge_cfg` setting is required for merge writes and cannot be used with other write modes. The `on` setting requires at least one key column, and those columns must exist in both the incoming data and the destination when the merge runs. If several columns form the key, all their values must match. `MergeCfg` is imported from the same connector module as the destination. For Snowflake and BigQuery, merge writes also require an Iceberg schema evolution setting in `dest_cfg`, as described under [Schema evolution](#schema-evolution). ## Row operations After matching records by key, Tabsdata uses the operation column to decide what to write. In the example above, `op` accepts these values: | Marker | Operation | | --- | --- | | `i` | Insert the row if its key does not exist in the destination. | | `u` | Update the row matching its key. | | `d` | Delete the row matching its key. | An insert leaves an existing record unchanged, while an update or delete does nothing if the record is missing. Uppercase `I`, `U`, and `D` are also accepted, but spaces are not removed, so `" d "` is invalid. Tabsdata removes rows with invalid or null operations before processing the batch. Because the operation column is used to control the write, Tabsdata excludes it when creating the destination table or updating its schema. ## Multiple changes to one record A record may change several times within the same batch. Tabsdata combines changes with the same key before writing them, using the first and last valid operations to determine the result. These operations follow the row order in the materialized batch, so the order returned by the Function matters: | First operation | Last operation | Result | | --- | --- | --- | | Insert | Insert or update | Insert using the latest row. | | Insert | Delete | No destination write for this key. | | Update or delete | Insert or update | Update using the latest row. | | Update or delete | Delete | Delete the matching record. | For example, `i → u → d → i` inserts the latest row because the final change adds the record again. With `i → u → d`, nothing is written because the record was created and deleted in the same batch. For a delete, Tabsdata keeps the data from the last insert or update for that key, or the first row if all operations are deletes. This allows delete records to contain just the key columns and operation. The destination therefore receives the combined change, without storing the intermediate changes as history. ## Destination tables and columns If the destination table does not exist, all three connectors can create it from the staged data's schema, excluding the operation column. The database and schema or dataset must already exist, and the Connection must have the required permissions. For existing tables, Snowflake and BigQuery write columns shared by the incoming data and the destination. Updates leave the keys unchanged, while inserts include them. Columns found only in the destination are left unchanged on updates and omitted from inserts, where the destination's defaults and constraints apply. For columns found only in the incoming data, `schema_evolution="strict"` excludes them from the write. Values written to shared columns must still satisfy the destination's types and constraints. Databricks handles column matching through Delta's merge rules. With `strict`, it uses `UPDATE SET *` and `INSERT *`, while `update` enables schema evolution and excludes the operation column. The operation marker should therefore use a separate column from the data being written. ## Schema evolution Incoming data may contain new columns or values too large for the destination's current column types. **Schema evolution** controls whether Tabsdata can change the table to accept that data. Set `schema_evolution` on the destination to choose the behavior. All three destinations default to `"strict"` in Tabsdata 2.1.0: | Setting | Behavior | | --- | --- | | `"strict"` | Does not add or widen data columns on an existing table. A configured watermark column is an exception. | | `"update"` | Allows new data columns and supported type widening. | For Snowflake and BigQuery, the connector uses an Iceberg schema evolution engine to compare the incoming and destination schemas. This engine must be configured for merge writes even when `schema_evolution="strict"`: ```python # SnowflakeDest dest_cfg={"snowflake.schema_evolution_engine": "iceberg"} # BigQueryDest dest_cfg={"bigquery.schema_evolution_engine": "iceberg"} ``` When `schema_evolution="update"`, the engine uses that comparison to apply supported changes through `ALTER TABLE`. The Iceberg setting controls how schemas are compared; it does not convert the destination into an Apache Iceberg table. Databricks uses Delta's `MERGE WITH SCHEMA EVOLUTION` for `schema_evolution="update"`, so it does not require a `schema_evolution_engine` setting. On each merge, Tabsdata also sets the destination table's `delta.enableTypeWidening` property to `true` for `update` or `false` for `strict`. ### Type widening Type widening allows a column to store larger values, such as longer strings or numbers with greater precision. The supported changes depend on the destination: | Destination | Supported widening | | --- | --- | | Snowflake | Wider `VARCHAR` lengths and greater `NUMBER` precision at the same scale. The connector does not widen between different base types. | | BigQuery | Supported numeric promotions, including `INT64 → NUMERIC → BIGNUMERIC → FLOAT64`, and wider type parameters. | | Databricks | Type changes supported by Delta type widening. | Setting `schema_evolution="update"` allows these supported changes. If the incoming data requires a conversion outside these rules, the write can still fail. ## Retrying a write A Subscriber may retry a write after data has already reached the destination. To recognize these retries, Tabsdata can store the Transaction ID in a **watermark column**. If that ID is already in the table, Tabsdata skips the write. Set `watermark_column` on the destination to enable this check: ```python SnowflakeDest( tables=["ORDERS"], if_table_exists="merge", merge_cfg=[MergeCfg(on=["order_id"], op_column="op")], watermark_column="wm", dest_cfg={"snowflake.schema_evolution_engine": "iceberg"}, ) ``` The configured name applies to all tables listed in the destination. If the column is missing, Tabsdata adds it even when `schema_evolution="strict"`. Existing rows have `NULL` in the new column, while inserted and updated rows receive the current Transaction ID. The watermark name must differ from the key columns and operation column, following the destination's rules for uppercase and lowercase names. Watermarks work with `append` and `merge`, but not with `replace`. Retries use the same Transaction ID, while new Transactions have different IDs and write normally. The check depends on a matching ID remaining in the destination. A batch containing only deletes leaves no new watermark value. ## Connector differences The connectors differ in how they stage data, change schemas, and match column names: | Behavior | Snowflake | BigQuery | Databricks | | --- | --- | --- | --- | | Staging | Snowflake stage, then a temporary table | GCS, then a temporary table | Unity Catalog volume, then a temporary staging table | | Schema changes under `update` | Connector applies `ALTER TABLE` using Iceberg schema comparison | Connector applies `ALTER TABLE` using Iceberg schema comparison | Delta merge schema evolution | | Required merge setting | `snowflake.schema_evolution_engine="iceberg"` | `bigquery.schema_evolution_engine="iceberg"` | No engine setting | | Column names | Unquoted names resolve as uppercase; double quotes preserve case | Case-insensitive; backticks quote names | Case-insensitive; backticks quote names | For example, Snowflake resolves an unquoted `order_id` as uppercase, so `on=["order_id"]` matches a stored `ORDER_ID` column. BigQuery and Databricks match column names without considering case, including when those names are quoted. BigQuery also requires destination table names to be enclosed in backticks, as in ``tables=["`orders`"]``. ## Errors Tabsdata checks the configuration before writing and validates the incoming data during execution. The following conditions can cause the merge to fail: | Condition | Result | | --- | --- | | Merge mode without `merge_cfg`, or `merge_cfg` without merge mode | Configuration error. | | `merge_cfg` is not a list, or its length differs from `tables` | Configuration error. | | Empty `on`, or an operation column that matches a key | Configuration error. | | Snowflake or BigQuery merge without the required Iceberg engine setting | Configuration error. | | A watermark with `replace`, or a watermark name that matches a key or operation column | Configuration error. | | Missing operation or key columns in the incoming batch | Runtime error. | | No shared data columns between the batch and destination | Runtime error. | | Several incoming columns match the same configured column name | Runtime error. | For field definitions, see the `MergeCfg` API reference for [Snowflake](https://docs.tabsdata.com/api/category/connectors/destination/mergecfg-snowflake), [BigQuery](https://docs.tabsdata.com/api/category/connectors/destination/mergecfg-bigquery), and [Databricks](https://docs.tabsdata.com/api/category/connectors/destination/mergecfg-databricks). --- # Tables > Covers Tabsdata Tables as persistent, versioned stores of function output: version history, the table reference format (`collection/table@version` with HEAD, INITIAL, and ranges), private `_` tables, and commit-based triggers. Source: https://docs.tabsdata.com/guide/tables Tables are a unique feature of Tabsdata. Unlike other data integration tools that discard intermediate data after execution, Tabsdata stores the data produced by functions in persistent, versioned Tables. Tables make it easier to monitor and debug workflows, inspect data from previous executions, track schema changes over time, and understand how data moves through a Project. Because Tables persist between executions, they can also be used as inputs by other functions and allow Tabsdata to automatically orchestrate workflows based on new data being committed. ## Table Versions Every Table maintains a complete version history. Each successful function execution that writes data to a Table creates a new **version** containing its own data and schema. Previous versions remain available, allowing users and functions to access the state of a Table from a specific execution. Each version also includes metadata about the execution that produced it and the function code used during that execution. persons Tabsdata›Collections›input›persons ### Schema Table schema defines the structure of a table by listing its columns and their data types. | Object | Type | | --- | --- | | first\_name | str | | last\_name | str | | age | i64 | | nationality | str | | city | str | version Because each Table version maintains its own schema, schema changes between executions do not cause previous versions of the Table to become invalid. ## Tables in Action Tables store the data that moves through a Tabsdata workflow. Functions read from and write to Tables depending on their role in the workflow. Publisher functions ingest data from external systems and write it to Tabsdata Tables. Tabsdata server collection · world Database PostgreSQL PostgreSQL · external Reachable Connection conn-postgres.yaml Publisher read\_world Table country //world No versions yet Table city //world No versions yet Transformer functions read from Tabsdata Tables, transform the data, and write the results to new Tables. collection · world collection · insights · data product Table country //world Committed Table city //world Committed Transformer find\_big\_cities Table big\_cities //insights No versions yet Subscriber functions read from Tabsdata Tables and write the data to external systems. Tabsdata server collection · warehouse Table vendors //sales Committed Table items //sales Committed Connection conn-mysql-dest.yaml Subscriber write\_sales Database MySQL MySQL · external Not yet synced ## Referring to a Table Functions refer to their input Tables using the following format: `[collection/]table[@version]` If the Collection is omitted, Tabsdata looks for the Table within the Function's own Collection. For example: ```text persons input/persons input/persons@HEAD ``` The version can be used to select a specific version or set of versions of the Table: - `HEAD` selects the latest version. - `HEAD^` selects the version immediately before `HEAD`. - `HEAD~N` selects a version relative to `HEAD`. - `INITIAL` selects the first version. - A specific version ID selects that version. - Comma-separated version IDs select multiple versions. - `a..b` selects a range of versions. If no version is specified, the Function reads from `HEAD`. Tables can also be referenced across Collections within the same Project. Tables in another Project cannot be referenced. ## Private Tables Tables whose names begin with `_` are **private Tables**. Private Tables can only be read by Functions within their own Collection. This makes them useful for intermediate data that needs to move between Functions in a Collection without being accessible to the rest of the Project. For example: ```text _cleaned_customers ``` A Function in the same Collection can use `_cleaned_customers` as an input, while Functions in other Collections cannot access it. ## Tables and Triggers A new Table commit can also trigger downstream Functions. Transformers and Subscribers are automatically triggered when a new commit is made to one of their input Tables. This allows data to continue moving through a workflow as upstream Functions complete. A commit does not necessarily mean that the data itself changed. When a Publisher writes multiple Tables, it creates a new commit for each Table on every successful execution, even if the rows in a Table are identical to its previous version. Each commit can trigger Functions that use that Table as an input. See [Working with Triggers](https://docs.tabsdata.com/guide/triggers) to control which input Tables trigger a Function. --- # Execution Plans > Explains the execution plan DAG Tabsdata builds when a Function is triggered, how Functions pass data through Tables, determinism, and the tdk plan, trx, fnrun, and worker commands. Source: https://docs.tabsdata.com/guide/execution/execution-plans An **execution plan** is the Directed Acyclic Graph that Tabsdata automatically builds and executes whenever a function is triggered. Tabsdata Functions are self describing, allowing Tabsdata to infer a Function's dependencies and automatically fire it when needed. ## How Tabsdata Automates Orchestration In other systems, DAG nodes need to be manually wired together, with explicit declaration of what data one node passes to the other. In Tabsdata, Functions never directly pass data to other Functions. Instead, Functions write tabular data into Tables and other Functions use those tables as input. When a Table gets new data, all functions that use it as input get triggered. tip Your Functions are also your data flow. Click on a Publisher, Transformer, or Subscriber to see the Execution Plan that is created if that Function is triggered. Click any node to trace its dependencies ## Determinism Execution plans are idempotent and deterministic. When an execution plan is generated, it contains a full snapshot of metadata and configurations used for the plan; such as start and stop timestamps, function code used for the plan, source data ingested, Table data generated, and data written out of Tabsdata ## Execution Plan Architecture An Execution Plan is a DAG of all Functions Tabsdata intends to execute. A plan contains transactions, function runs, and workers. Click into each resource to learn more about what they do. A function is triggered creates [Execution plan](https://docs.tabsdata.com/guide/execution/execution-plans)One per function triggered [Transactions](https://docs.tabsdata.com/guide/execution/transactions)One per collection in the execution plan [Function runs](https://docs.tabsdata.com/guide/execution/function-runs)One per function in the transaction [Workers](https://docs.tabsdata.com/guide/execution/workers)One per attempted code execution (max 3) ## Inspecting a plan Four commands cover the four levels: `tdk plan`, `tdk trx`, `tdk fnrun`, and `tdk worker`. See the [CLI Guide](https://docs.tabsdata.com/cli/tdk/#execution) for what each one accepts. [Execution & Triggers](https://docs.tabsdata.com/guide/execution-and-triggers) covers what fires a plan in the first place, along with the pristine-state guarantee it carries. --- # Transactions > Explains how transactions group Table changes within a Collection so they commit together, preserve pristine state on failure, and make execution plans multi-transactional across Collections. Source: https://docs.tabsdata.com/guide/execution/transactions A **transaction** groups together changes made to Tables within a Collection during an [execution plan](https://docs.tabsdata.com/guide/execution/execution-plans). These changes are committed together, meaning they become available at the same time. Transactions help Tabsdata maintain **pristine state**. If an execution plan fails before a transaction is committed, its output is not published and the affected Tables remain in their previous state. This prevents incomplete or partially processed data from becoming available to users or downstream Functions. See [Execution & Triggers](https://docs.tabsdata.com/guide/execution-and-triggers) for more information about pristine state. ## Collection Transactions In Tabsdata, a Collection acts as a transaction boundary. When an execution plan processes Functions within a Collection, changes made to that Collection's Tables are grouped into a transaction. Once the transaction is successfully committed, all of those changes become available together. This ensures that users and other Functions do not read a Collection while only some of its Tables have been updated. ## Transactions Across Collections A single execution plan can process Functions across multiple Collections. Because each Collection has its own transaction, an execution plan may contain multiple transactions. For example, if an execution plan processes Functions across three Collections, each Collection handles and commits its Table changes through its own transaction. This is why execution plans in Tabsdata are **multi-transactional**. --- # Function Runs > Function Runs cover how each Function execution in an Execution Plan is tracked, including status, logs, Worker assignment, recovery, and replay, with CLI commands linked. Source: https://docs.tabsdata.com/guide/execution/function-runs A **Function Run** represents a single execution of a Function in Tabsdata. Whenever a Function is executed as part of an [Execution Plan](https://docs.tabsdata.com/guide/execution/execution-plans), Tabsdata creates a Function Run to track that individual execution. If an Execution Plan contains multiple Functions, each Function gets its own Function Run. Function Runs allow Tabsdata to track the execution of each Function independently, including its status, logs, inputs, outputs, and the Worker responsible for executing it. ## Function Runs in an Execution Plan An Execution Plan describes the set of Functions that need to run and the order in which they need to run. Function Runs represent the individual executions that take place within that plan. For example, an Execution Plan containing a Publisher, Transformer, and Subscriber would create a separate Function Run for each Function: ```text Execution Plan ├── Publisher Function Run ├── Transformer Function Run └── Subscriber Function Run ``` This makes it possible to see the status of each individual Function rather than only whether the Execution Plan as a whole succeeded or failed. ## Function Runs and Workers Function Runs are executed by [Workers](https://docs.tabsdata.com/guide/execution/workers). When a Function is ready to run, Tabsdata assigns its Function Run to a Worker. The Worker performs the actual processing required by the Function and reports the status of the Function Run back to Tabsdata. This separation allows multiple Function Runs to be processed independently and, when possible, in parallel. In case of failures, function runs can be assigned to new workers and retried. ## Function Runs and Transactions The data produced by a Function Run is written as part of a [Transaction](https://docs.tabsdata.com/guide/execution/transactions). A successful Function Run can produce changes to one or more Tables, but those changes do not become available simply because the Function Run finished. They become available when the Transaction containing those changes is committed. This allows Tabsdata to keep the execution of an individual Function separate from the process of publishing its output. ## Function Run Status Each Function Run has its own status throughout its execution. This makes Function Runs useful for understanding where an Execution Plan is currently running or where a failure occurred. If one Function fails in a plan containing several Functions, you can identify the specific Function Run that failed and inspect its logs without treating the Execution Plan as a single unit. ## Recovery and Replay Because each Function execution is tracked as its own Function Run, Tabsdata can operate on individual runs after they have been created. A failed Function Run can be **recovered**, allowing Tabsdata to retry that execution without starting the entire workflow again. A completed Function Run can also be **replayed**, allowing the Function to execute again using the same inputs. Functions can be replayed even if the source data the function relies on has changed. ## Inspecting Function Runs Function Runs can be inspected through an AI agent connected to Tabsdata, to view their status, logs, and other execution information. AI agent session ❯Show me the logs for function run fr-3f9a2c1e Called tabsdata 1 time (ctrl+o to expand) ⏺ Here's fr-3f9a2c1e's log. ❯What's its status? Called tabsdata 1 time (ctrl+o to expand) ⏺ fr-3f9a2c1e is a Transformer run and finished successfully. See the [CLI Guide](https://docs.tabsdata.com/cli/tdk/#execution) for the complete set of commands for inspecting, recovering, and replaying Function Runs. --- # Workers > Explains what a Tabsdata worker is, how retries work (three attempts by default), and the tdk worker logs, recover, and list commands with their filters. Source: https://docs.tabsdata.com/guide/execution/workers A **worker** is the process that executes the code defined in a Tabsdata Each worker runs as an isolated process on the Tabsdata server's Kubernetes cluster. Performance optimizations such as Polars Streaming and Lazy Evaluation allow workers to process datasets incrementally, allowing them to handle datasets larger than the memory available on the machine's hardware. ## Retries Tabsdata deploys one worker for each execution attempt of the function code. If a worker fails, Tabsdata snapshots the worker's execution log and spins up a new worker to retry the execution. By default, a function run will attempt three executions before stalling. A function is triggered Function runGenerated by the trigger Worker 1Failed Worker 2Failed Worker 3Succeeded Three workers is the default limit. Had the third one failed as well, the function run would stall instead of starting a fourth. ## Logs A worker is where a function's output ends up, so its logs are the first place to look when a function run fails: ```bash $ tdk worker logs --worker WID ``` Pass `--file -` to write the log to stdout. ## Recovery You can retry a single worker without re-running the whole plan: ```bash $ tdk worker recover --worker WID ``` ## Finding a worker `tdk worker list` filters down from any level of the plan, so you can start from whichever identifier you already have: ```bash $ tdk worker list [--status R] [--coll C] [--fn F] [--plan P] [--trx T] [--fnrun FR] ``` See the [CLI Guide](https://docs.tabsdata.com/cli/tdk/#execution) for the full set of commands and the status codes each accepts. --- # Connectors > The connectors built into Tabsdata, and the source and destination classes each one provides. Source: https://docs.tabsdata.com/guide/connectors/ A **connector** is the set of Python classes Tabsdata uses to read from or write to one external system. Every built-in connector is part of the `tabsdatak.conn` package and provides a source class, a destination class, or both, along with the connection types that hold the system's address and credentials. A publisher uses a connector's source class to read from the external system and write into Tabsdata tables. A subscriber uses a connector's destination class to read from Tabsdata tables and write out to the external system. Neither class stores credentials. The address and credentials live on the [connection](https://docs.tabsdata.com/guide/connections) of the collection the function is registered in, so the same connection is shared by every publisher or subscriber in that collection. Kafka and the three CDC connectors provide a stream source class instead of a source class. A [stream publisher](https://docs.tabsdata.com/guide/functions/stream-publishers) uses a stream source to consume the external system continuously, and runs each time a batch of messages or changes rolls over instead of on a manual, cron, or table trigger. ## Built-in connectors Each connector's page covers its connection fields and the options of its classes, and links to a full walkthrough in Tutorials. | Connector | Publisher | Stream publisher | Subscriber | | --- | --- | --- | --- | | [Amazon S3](https://docs.tabsdata.com/guide/connectors/amazon-s3) | `S3Src` | | `S3Dest` | | [Apache Druid](https://docs.tabsdata.com/guide/connectors/druid) | | | `DruidDest` | | [Azure Blob Storage](https://docs.tabsdata.com/guide/connectors/azure) | `AzureBlobSrc` | | `AzureBlobDest` | | [BigQuery](https://docs.tabsdata.com/guide/connectors/bigquery) | | | `BigQueryDest` | | [ClickHouse](https://docs.tabsdata.com/guide/connectors/clickhouse) | | | `ClickHouseDest` | | [Databricks](https://docs.tabsdata.com/guide/connectors/databricks) | `DatabricksSrc` | | `DatabricksDest` | | [Db2](https://docs.tabsdata.com/guide/connectors/db2) | `Db2Src` | | `Db2Dest` | | [Db2 CDC](https://docs.tabsdata.com/guide/connectors/db2-cdc) | | `Db2CdcStreamSrc` | | | [Google Cloud Storage](https://docs.tabsdata.com/guide/connectors/gcp) | `GCSSrc` | | `GCSDest` | | [Kafka](https://docs.tabsdata.com/guide/connectors/kafka) | | `KafkaStreamSrc` | | | [Local File System](https://docs.tabsdata.com/guide/connectors/local-file-system) | `LocalFileSrc` | | `LocalFileDest` | | [MariaDB](https://docs.tabsdata.com/guide/connectors/mariadb) | `MySQLSrc` | | `MySQLDest` | | [MongoDB](https://docs.tabsdata.com/guide/connectors/mongodb) | | | `MongoDBDest` | | [MSSQL](https://docs.tabsdata.com/guide/connectors/mssql) | `MSSQLSrc` | | `MSSQLDest` | | [MySQL](https://docs.tabsdata.com/guide/connectors/mysql) | `MySQLSrc` | | `MySQLDest` | | [MySQL CDC](https://docs.tabsdata.com/guide/connectors/mysql-cdc) | | `MySQLCdcStreamSrc` | | | [Oracle](https://docs.tabsdata.com/guide/connectors/oracle) | `OracleSrc` | | `OracleDest` | | [Postgres](https://docs.tabsdata.com/guide/connectors/postgres) | `PostgresSrc` | | `PostgresDest` | | [Postgres CDC](https://docs.tabsdata.com/guide/connectors/postgres-cdc) | | `PostgresCdcStreamSrc` | | | [Salesforce](https://docs.tabsdata.com/guide/connectors/salesforce) | `SalesforceSrc`, `SalesforceReportSrc` | | | | [Snowflake](https://docs.tabsdata.com/guide/connectors/snowflake) | | | `SnowflakeDest` | | [StarRocks](https://docs.tabsdata.com/guide/connectors/starrocks) | | | `StarRocksDest` | note The MariaDB, MySQL, Oracle, and Postgres connectors need a database driver that Tabsdata does not redistribute. See [Installing Third-Party Database Drivers](https://docs.tabsdata.com/server-admin/third-party-database-drivers) to add the driver to your Tabsdata server. ## Other systems To connect a system that has no built-in connector, write a custom connector. See [Custom Connectors](https://docs.tabsdata.com/guide/plugins). --- # Amazon S3 > Reference for the Tabsdata Amazon S3 connector. Source: https://docs.tabsdata.com/guide/connectors/amazon-s3 The Amazon S3 connector lets Tabsdata read files from and write files to Amazon S3. The `S3Src` connector can be used by a publisher function to read data from Amazon S3 into a Tabsdata table. The `S3Dest` connector can be used by a subscriber function to write data from a Tabsdata table into Amazon S3. ## Supported formats | Format | Publish | Subscribe | | --- | --- | --- | | `AUTO` | Yes | Yes | | `CSV` | Yes | Yes | | `JSON` | Yes | Yes | | `AVRO` | Yes | Yes | | `PARQUET` | Yes | Yes | | `LOG` | Yes | No | ## Publisher The `S3Src` connector can be used by a publisher function to read data from Amazon S3 into a Tabsdata table. [See the full publisher walkthrough](https://docs.tabsdata.com/tutorials/publishers/amazon-s3) The publisher is registered in a collection created with a `AwsAccessSecretKey` connection. The connection, and how to create the collection, are described in [Publisher connection](#publisher-connection), at the end of this page. ### Prerequisites - The bucket exists and is reachable from the Tabsdata server. - The AWS principal can list the bucket and read every object matched by `paths`. - The object format and compression match `format` and `format_cfg`. ### Publisher Configure `S3Src` as the `source` of a publisher function to select the files that Tabsdata reads from S3. publish\_orders.py ```python from tabsdatak.api import publisher from tabsdatak.conn.s3 import S3Src, SrcFileFormat @publisher( source=S3Src( paths=[ "incoming/orders.parquet", ], ), output_tables=["orders"], ) def publish_orders(orders): return orders ``` - required - optional - optional - optional - optional ### Function Config Parameters #### `paths` Defines the files that the publisher reads from S3. Each path is relative to the `base_path` configured in `S3SrcConn`. Source paths: - cannot begin or end with `/` - cannot contain empty path segments - can contain a `*` glob in the final path segment For example: ```python S3Src( paths=[ "orders/2026/*.parquet", "customers/customers.csv", ] ) ``` Each element in `paths` represents one source slot and maps positionally to an argument in the publisher function. When a path uses a glob to match multiple files, the matching files are provided through the corresponding publisher input. #### `format` Sets the format used to read the source files. Supported formats are `AUTO`, `CSV`, `JSON`, `AVRO`, `PARQUET`, and `LOG`. The default is `AUTO`, which determines the format from the file extension. #### `format_cfg` Sets format-specific reader options. Configuration is keyed by `SrcFileFormat`. #### `initial_last_modified` Sets the initial cutoff for incremental ingestion. On the first run, Tabsdata imports files modified at or after the specified timestamp. The cutoff then advances so subsequent runs only import newer files. #### `src_cfg` Sets additional source configuration. The following options are supported: | Option | Description | | --- | --- | | `tabsdata.file.chunk_size` | Controls the size of chunks used when reading files. | | `tabsdata.src_metadata.drop` | When `True`, excludes the `@td.file.path` metadata column. | By default, rows read from S3 include a `@td.file.path` metadata column containing the `s3://` URI of the file the row was read from. ## Subscriber The `S3Dest` connector can be used by a subscriber function to write data from a Tabsdata table into Amazon S3. [See the full subscriber walkthrough](https://docs.tabsdata.com/tutorials/subscribers/amazon-s3) The subscriber is registered in a collection created with a `AwsAccessSecretKey` connection. The connection, and how to create the collection, are described in [Subscriber connection](#subscriber-connection), at the end of this page. ### Prerequisites - The bucket exists and is reachable from the Tabsdata server. - The AWS principal can write objects under the configured `base_path`. - The destination path is relative and does not contain a glob. ### Subscriber Configure `S3Dest` as the `destination` of a subscriber function to define where Tabsdata writes files in S3. write\_orders.py ```python from tabsdatak.api import subscriber from tabsdatak.conn.s3 import S3Dest, DestFileFormat @subscriber( destination=S3Dest( paths=[ "exports/orders.parquet", ], ), input_tables=["orders"], ) def write_orders(orders): return orders ``` - required - optional - optional - optional ### Function Config Parameters #### `paths` Defines where the subscriber writes files in S3. Each path is relative to the `base_path` configured in `S3DestConn`. Destination paths: - cannot begin or end with `/` - cannot contain empty path segments - cannot contain globs Each element in `paths` represents one destination slot and maps positionally to a value returned by the subscriber function. Destination paths can contain variables that Tabsdata substitutes when the subscriber runs: | Variable | Value | | --- | --- | | `${EXEC_PLAN_ID}` | Current execution plan ID. | | `${TRX_ID}` | Current transaction ID. | | `${EXEC_PLAN_TS}` | Timestamp when the execution plan started. | For example: ```python S3Dest( paths=[ "exports/${EXEC_PLAN_ID}/orders.parquet", ] ) ``` Dynamic paths can be used to create a new S3 object for each execution instead of overwriting an existing object. #### `format` Sets the format used to write the destination file. Supported formats are `AUTO`, `CSV`, `JSON`, `AVRO`, and `PARQUET`. The default is `AUTO`, which determines the format from the destination file extension. #### `format_cfg` Sets format-specific writer options. Configuration is keyed by `DestFileFormat`. #### `dest_cfg` Sets additional destination configuration. The `tabsdata.file.chunk_size` option controls the size of chunks used when writing files. ## File behavior Each entry in a publisher's `paths` is one source slot. A literal path reads one object; a final-segment `*` glob can read several. The slots map by position to the publisher function arguments and then to `output_tables`. `initial_last_modified` sets the cutoff for the first run. After a successful run, Tabsdata advances the cutoff so the same unchanged objects are not published again. Each entry in a subscriber's `paths` is one destination slot and maps by position to a returned value. Use `${EXEC_PLAN_ID}`, `${TRX_ID}`, or `${EXEC_PLAN_TS}` in a path when every run needs a new object. A fixed path writes to the same location again. ## Operational notes - Without `trigger_by`, a publisher runs only when triggered manually, while a subscriber runs when any input table commits. Set `trigger_by` to change when the function runs. - Tabsdata validates the connection document when the collection is created, but it does not contact Amazon S3 then. A bad path, credential, or permission fails when the function runs. - Globs are allowed only for publisher source paths, and only in the final path segment. Subscriber destination paths cannot contain globs. - All configured paths are relative to the connection's `base_path`. ## Publisher connection Amazon S3 publishers use `S3SrcConn` to define the S3 bucket and credentials used by the publisher. conn-s3-in.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.s3:S3SrcConn spec: bucket: 'acme-ingest' region: 'us-east-1' credentials: kind: awsAccessSecretKeyCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:AwsAccessSecretKey spec: access_key_id: 'secret:AKIAIOSFODNN7EXAMPLE' secret_access_key: 'secret:wJalrXUtnFEMI0EXAMPLEKEY' ``` - required - required - required - optional - optional ### Create the collection A publisher is registered in a source collection, and a source collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type s3-file-in --file conn-s3-in.yaml ``` Edit the template file with the Amazon S3 connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-s3-in.yaml ``` ```bash tdk collection create --name s3_in --conn-file conn-s3-in.yaml ``` Then register the publisher in the collection: ```bash tdk fn register --coll s3_in --path publish_orders.py::publish_orders ``` - The collection goes in the `sources` group, because the connection is a source connection. `--group` can be left out; if it is given, it must be `sources`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `S3Src` for a collection created with `S3SrcConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name s3_in --conn-file conn-s3-in.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. ### Connection Config Parameters #### `bucket` The name of the S3 bucket that contains the files to publish. #### `region` The AWS region where the S3 bucket is located, such as `us-east-1`. #### `credentials` The AWS credentials used to access the bucket. `AwsAccessSecretKey` requires an `access_key_id` and `secret_access_key`. #### `base_path` Sets the base path within the S3 bucket. All paths configured in `S3Src` are relative to this path. `base_path` starts with `/`, cannot end with `/`, and cannot contain empty path segments. The default is `/`. For example, with: ```yaml base_path: /hr ``` the source path: ```text employees/2026.csv ``` refers to: ```text /hr/employees/2026.csv ``` within the configured S3 bucket. #### `conn_cfg` Optional connector configuration overrides. ## Subscriber connection Amazon S3 subscribers use `S3DestConn` to define the S3 bucket and credentials used by the subscriber. conn-s3-out.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.s3:S3DestConn spec: bucket: 'acme-exports' region: 'us-east-1' credentials: kind: awsAccessSecretKeyCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:AwsAccessSecretKey spec: access_key_id: 'secret:AKIAIOSFODNN7EXAMPLE' secret_access_key: 'secret:wJalrXUtnFEMI0EXAMPLEKEY' ``` - required - required - required - optional - optional ### Create the collection A subscriber is registered in a destination collection, and a destination collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type s3-file-out --file conn-s3-out.yaml ``` Edit the template file with the Amazon S3 connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-s3-out.yaml ``` ```bash tdk collection create --name s3_out --conn-file conn-s3-out.yaml ``` Then register the subscriber in the collection: ```bash tdk fn register --coll s3_out --path write_orders.py::write_orders ``` - The collection goes in the `destinations` group, because the connection is a destination connection. `--group` can be left out; if it is given, it must be `destinations`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `S3Dest` for a collection created with `S3DestConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name s3_out --conn-file conn-s3-out.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. ### Connection Config Parameters #### `bucket` The name of the S3 bucket where the subscriber writes files. #### `region` The AWS region where the S3 bucket is located, such as `us-east-1`. #### `credentials` The AWS credentials used to access the bucket. `AwsAccessSecretKey` requires an `access_key_id` and `secret_access_key`. #### `base_path` Sets the base path within the S3 bucket. All paths configured in `S3Dest` are relative to this path. `base_path` starts with `/`, cannot end with `/`, and cannot contain empty path segments. The default is `/`. #### `conn_cfg` Optional connector configuration overrides. --- # Apache Druid > Reference for the Tabsdata Apache Druid destination connector, cloud staging, and subscriber configuration. Source: https://docs.tabsdata.com/guide/connectors/druid The Apache Druid connector lets Tabsdata subscribers write data from Tabsdata tables into Druid datasources. The connector stages the data as Parquet files in cloud storage, then submits a Druid ingestion task that loads it from there. [See the full subscriber walkthrough](https://docs.tabsdata.com/tutorials/subscribers/druid) The subscriber is registered in a collection created with a `DruidDestConn` connection. The connection, and how to create the collection, are described in [Connection](#connection), at the end of this page. ### Prerequisites - The Druid Overlord API is reachable from the Tabsdata server. - The Overlord credentials can submit and inspect ingestion tasks. - Tabsdata can write to the S3 or Azure staging location and Druid can read the staged Parquet files. ### Subscriber Configure `DruidDest` as the `destination` of a subscriber function to define which datasources Tabsdata writes in Druid. write\_events.py ```python from tabsdatak.api import subscriber, TableFrameSpec from tabsdatak.conn.druid import DruidDest @subscriber( destination=DruidDest( tables=[ "events", ], ), input_tables=["events"], ) def write_events(events: TableFrameSpec) -> TableFrameSpec: return events ``` - required - optional - optional - optional - optional - optional - optional - optional - optional ### Function Config Parameters #### `tables` Defines the Druid datasources the subscriber writes to. Each name is 1 to 128 characters, does not start with `.` or `_`, and contains no `/`, backslash or NUL. Each element in `tables` represents one destination slot and maps positionally to a value returned by the subscriber function. #### `if_table_exists` `"append"` (the default) adds rows to the datasource. `"replace"` overwrites it. #### `time_column` The column Druid uses as its primary `__time` field. Druid requires a time column on every datasource; when the rows have none, leave `time_column` unset and the ingest time is used instead. #### `segment_granularity` The time span of each Druid segment: `ALL` (the default), `NONE`, `SECOND`, `MINUTE`, `FIFTEEN_MINUTE`, `THIRTY_MINUTE`, `HOUR`, `SIX_HOUR`, `DAY`, `WEEK`, `MONTH`, `QUARTER` or `YEAR`. #### `max_concurrent_subtasks`, `max_rows_per_segment`, `max_rows_in_memory` and `http_timeout_sec` Tuning for the ingestion task: its concurrency (default `1`), the row cap per segment (default `5000000`), the in-memory row cap per task (default `1000000`), and the HTTP request timeout in seconds (default `300`). #### `dest_cfg` Sets additional destination configuration. The accepted keys are `druid.logging_level`, `druid.poll_interval_seconds`, `druid.retry_attempts` and `druid.retry_backoff_seconds`. ## Write behavior Each entry in `tables` is one destination slot and maps by position to a value returned by the subscriber function. Returning `None` for a slot skips that write. `if_table_exists="append"` adds data to the datasource. `"replace"` overwrites it. `time_column` selects the row timestamp used as Druid `__time`; without it, the connector uses the ingest time. ## Operational notes - The connection is to the Overlord API, not a SQL endpoint. - A write finishes only after the submitted ingestion task finishes. `http_timeout_sec`, polling, retry, and backoff settings control how Tabsdata waits for it. - `segment_granularity` and the row limits change the ingestion task, not the Tabsdata input table. - Tabsdata does not contact Druid or the staging service when the collection is created. Those checks happen when the subscriber runs. ## Connection Druid subscribers use `DruidDestConn` to define the Overlord, the credentials, and the cloud storage the data is staged in. conn-druid.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.druid:DruidDestConn spec: overlord_url: 'http://druid-overlord.internal:8081' credentials: kind: userPasswordCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:UserPassword spec: user: 'secret:tabsdata_writer' password: 'secret:example-password' staging: kind: druidS3Location apiVersion: '1.0' type: tabsdatak.conn.druid:S3Location spec: bucket: 'acme-staging' region: 'us-east-1' credentials: kind: awsAccessSecretKeyCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:AwsAccessSecretKey spec: access_key_id: 'secret:example-access-key' secret_access_key: 'secret:example-secret-key' base_path: '/druid' ``` - required - required - required - optional ### Create the collection A subscriber is registered in a destination collection, and a destination collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type druid-bulk-out --file conn-druid.yaml ``` Edit the template file with the Druid connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-druid.yaml ``` ```bash tdk collection create --name druid_out --conn-file conn-druid.yaml ``` Then register the subscriber in the collection: ```bash tdk fn register --coll druid_out --path write_events.py::write_events ``` - The collection goes in the `destinations` group, because the connection is a destination connection. `--group` can be left out; if it is given, it must be `destinations`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `DruidDest` for a collection created with `DruidDestConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name druid_out --conn-file conn-druid.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. ### Connection Config Parameters #### `overlord_url` The `http://` or `https://` URL of the Druid Overlord, for example `http://druid.example:8081`. Accepts `secret:` too. #### `credentials` A `UserPassword` used for HTTP basic authentication with the Overlord. Cloud storage credentials are configured separately under `staging`. #### `staging` An `S3Location` or an `AzBlobLocation` from `tabsdatak.conn.druid`, where the Parquet files are staged before ingestion. Tabsdata must be able to upload the staged files, and Druid must be able to read them. #### `conn_cfg` Optional connector configuration. --- # Azure Blob Storage > Reference for the Tabsdata Azure Blob Storage connector. Source: https://docs.tabsdata.com/guide/connectors/azure The Azure Blob Storage connector lets Tabsdata read files from and write files to Azure Blob Storage. The `AzureBlobSrc` connector can be used by a publisher function to read data from Azure Blob Storage into a Tabsdata table. The `AzureBlobDest` connector can be used by a subscriber function to write data from a Tabsdata table into Azure Blob Storage. ## Supported formats | Format | Publish | Subscribe | | --- | --- | --- | | `AUTO` | Yes | Yes | | `CSV` | Yes | Yes | | `JSON` | Yes | Yes | | `AVRO` | Yes | Yes | | `PARQUET` | Yes | Yes | | `LOG` | Yes | No | ## Publisher The `AzureBlobSrc` connector can be used by a publisher function to read data from Azure Blob Storage into a Tabsdata table. [See the full publisher walkthrough](https://docs.tabsdata.com/tutorials/publishers/azure) The publisher is registered in a collection created with a `AzureBlobSrcConn` connection. The connection, and how to create the collection, are described in [Publisher connection](#publisher-connection), at the end of this page. ### Prerequisites - The container exists and is reachable from the Tabsdata server. - The storage account can list and read every blob matched by `paths`. - The blob format and compression match `format` and `format_cfg`. ### Publisher Configure `AzureBlobSrc` as the `source` of a publisher function to select the files that Tabsdata reads from Azure Blob Storage. publish\_orders.py ```python from tabsdatak.api import publisher from tabsdatak.conn.azureblob import AzureBlobSrc, SrcFileFormat @publisher( source=AzureBlobSrc( paths=[ "incoming/orders.parquet", ], ), output_tables=["orders"], ) def publish_orders(orders): return orders ``` - required - optional - optional - optional - optional ### Function Config Parameters #### `paths` Defines the files that the publisher reads from Azure Blob Storage. Each path is relative to the `base_path` configured in `AzureBlobSrcConn`. Source paths: - cannot begin or end with `/` - cannot contain empty path segments - can contain a `*` glob in the final path segment For example: ```python AzureBlobSrc( paths=[ "orders/2026/*.parquet", "customers/customers.csv", ] ) ``` Each element in `paths` represents one source slot and maps positionally to an argument in the publisher function. When a path uses a glob to match multiple files, the matching files are provided through the corresponding publisher input. #### `format` Sets the format used to read the source files. Supported formats are `AUTO`, `CSV`, `JSON`, `AVRO`, `PARQUET`, and `LOG`. The default is `AUTO`, which determines the format from the file extension. #### `format_cfg` Sets format-specific reader options. Configuration is keyed by `SrcFileFormat`. #### `initial_last_modified` Sets the initial cutoff for incremental ingestion. On the first run, Tabsdata imports files modified at or after the specified timestamp. The cutoff then advances so subsequent runs only import newer files. #### `src_cfg` Sets additional source configuration. The following options are supported: | Option | Description | | --- | --- | | `tabsdata.file.chunk_size` | Controls the size of chunks used when reading files. | | `tabsdata.src_metadata.drop` | When `True`, excludes the `@td.file.path` metadata column. | By default, rows read from Azure Blob Storage include a `@td.file.path` metadata column containing the source file's URI. ## Subscriber The `AzureBlobDest` connector can be used by a subscriber function to write data from a Tabsdata table into Azure Blob Storage. [See the full subscriber walkthrough](https://docs.tabsdata.com/tutorials/subscribers/azure) The subscriber is registered in a collection created with a `AzureBlobDestConn` connection. The connection, and how to create the collection, are described in [Subscriber connection](#subscriber-connection), at the end of this page. ### Prerequisites - The container exists and is reachable from the Tabsdata server. - The storage account can create and write blobs under the configured `base_path`. - The destination path is relative and does not contain a glob. ### Subscriber Configure `AzureBlobDest` as the `destination` of a subscriber function to define where Tabsdata writes files in Azure Blob Storage. write\_orders.py ```python from tabsdatak.api import subscriber from tabsdatak.conn.azureblob import AzureBlobDest, DestFileFormat @subscriber( destination=AzureBlobDest( paths=[ "exports/orders.parquet", ], ), input_tables=["orders"], ) def write_orders(orders): return orders ``` - required - optional - optional - optional ### Function Config Parameters #### `paths` Defines where the subscriber writes files in Azure Blob Storage. Each path is relative to the `base_path` configured in `AzureBlobDestConn`. Destination paths: - cannot begin or end with `/` - cannot contain empty path segments - cannot contain globs Each element in `paths` represents one destination slot and maps positionally to a value returned by the subscriber function. Destination paths can contain variables that Tabsdata substitutes when the subscriber runs: | Variable | Value | | --- | --- | | `${EXEC_PLAN_ID}` | Current execution plan ID. | | `${TRX_ID}` | Current transaction ID. | | `${EXEC_PLAN_TS}` | Timestamp when the execution plan started. | For example: ```python AzureBlobDest( paths=[ "exports/${EXEC_PLAN_ID}/orders.parquet", ] ) ``` Dynamic paths can be used to create a new object for each execution instead of overwriting an existing object. #### `format` Sets the format used to write the destination file. Supported formats are `AUTO`, `CSV`, `JSON`, `AVRO`, and `PARQUET`. The default is `AUTO`, which determines the format from the destination file extension. #### `format_cfg` Sets format-specific writer options. Configuration is keyed by `DestFileFormat`. #### `dest_cfg` Sets additional destination configuration. The `tabsdata.file.chunk_size` option controls the size of chunks used when writing files. ## File behavior Each entry in a publisher's `paths` is one source slot. A literal path reads one blob; a final-segment `*` glob can read several. The slots map by position to the publisher function arguments and then to `output_tables`. `initial_last_modified` sets the cutoff for the first run. After a successful run, Tabsdata advances the cutoff so the same unchanged blobs are not published again. Each entry in a subscriber's `paths` is one destination slot and maps by position to a returned value. Use `${EXEC_PLAN_ID}`, `${TRX_ID}`, or `${EXEC_PLAN_TS}` in a path when every run needs a new blob. A fixed path writes to the same location again. ## Operational notes - Without `trigger_by`, a publisher runs only when triggered manually, while a subscriber runs when any input table commits. Set `trigger_by` to change when the function runs. - Tabsdata validates the connection document when the collection is created, but it does not contact Azure Blob Storage then. A bad path, credential, or permission fails when the function runs. - Globs are allowed only for publisher source paths, and only in the final path segment. Subscriber destination paths cannot contain globs. - All configured paths are relative to the connection's `base_path`. ## Publisher connection Azure Blob Storage publishers use `AzureBlobSrcConn` to define the container and credentials used by the publisher. conn-azure-in.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.azureblob:AzureBlobSrcConn spec: container: 'acme-ingest' credentials: kind: azureAccountKeyCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:AzureAccountKey spec: account_name: 'secret:mystorageaccount' account_key: 'secret:8Zs0RExAmPlEkEy0000==' ``` - required - required - optional - optional ### Create the collection A publisher is registered in a source collection, and a source collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type az-file-in --file conn-azure-in.yaml ``` Edit the template file with the Azure Blob Storage connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-azure-in.yaml ``` ```bash tdk collection create --name azure_in --conn-file conn-azure-in.yaml ``` Then register the publisher in the collection: ```bash tdk fn register --coll azure_in --path publish_orders.py::publish_orders ``` - The collection goes in the `sources` group, because the connection is a source connection. `--group` can be left out; if it is given, it must be `sources`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `AzureBlobSrc` for a collection created with `AzureBlobSrcConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name azure_in --conn-file conn-azure-in.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. ### Connection Config Parameters #### `container` The name of the Azure Blob container that contains the files to publish. #### `credentials` The Azure storage-account credentials used to access the container. `AzureAccountKey` requires an `account_name` and `account_key`. #### `base_path` Sets the base path within the container. All paths configured in `AzureBlobSrc` are relative to this path. `base_path` starts with `/`, cannot end with `/`, and cannot contain empty path segments. The default is `/`. #### `conn_cfg` Optional connector configuration overrides. ## Subscriber connection Azure Blob Storage subscribers use `AzureBlobDestConn` to define the container and credentials used by the subscriber. conn-azure-out.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.azureblob:AzureBlobDestConn spec: container: 'acme-exports' credentials: kind: azureAccountKeyCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:AzureAccountKey spec: account_name: 'secret:mystorageaccount' account_key: 'secret:8Zs0RExAmPlEkEy0000==' ``` - required - required - optional - optional ### Create the collection A subscriber is registered in a destination collection, and a destination collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type az-file-out --file conn-azure-out.yaml ``` Edit the template file with the Azure Blob Storage connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-azure-out.yaml ``` ```bash tdk collection create --name azure_out --conn-file conn-azure-out.yaml ``` Then register the subscriber in the collection: ```bash tdk fn register --coll azure_out --path write_orders.py::write_orders ``` - The collection goes in the `destinations` group, because the connection is a destination connection. `--group` can be left out; if it is given, it must be `destinations`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `AzureBlobDest` for a collection created with `AzureBlobDestConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name azure_out --conn-file conn-azure-out.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. ### Connection Config Parameters #### `container` The name of the Azure Blob container where the subscriber writes files. #### `credentials` The Azure storage-account credentials used to access the container. `AzureAccountKey` requires an `account_name` and `account_key`. #### `base_path` Sets the base path within the container. All paths configured in `AzureBlobDest` are relative to this path. `base_path` starts with `/`, cannot end with `/`, and cannot contain empty path segments. The default is `/`. #### `conn_cfg` Optional connector configuration overrides. --- # BigQuery > Reference for the Tabsdata BigQuery connector. Source: https://docs.tabsdata.com/guide/connectors/bigquery The BigQuery connector lets Tabsdata write tables into BigQuery. BigQuery is destination-only: there is no publisher-side connector for reading from BigQuery. The `BigQueryDest` connector can be used by a subscriber function to write data from a Tabsdata table into BigQuery. [See the full subscriber walkthrough](https://docs.tabsdata.com/tutorials/subscribers/bigquery) The subscriber is registered in a collection created with a `BigQueryDestConn` connection. The connection, and how to create the collection, are described in [Connection](#connection), at the end of this page. ### Prerequisites - The BigQuery project and dataset exist, and the service account can create and load tables in them. - The GCS staging bucket exists. The service account can write staged Parquet files there and BigQuery can load them. - The project and dataset are set either in each table name or as connection defaults. ### Subscriber For merge writes, schema evolution, and retry watermarks, see [Merging into SQL destinations](https://docs.tabsdata.com/guide/functions/sql-merge). Configure `BigQueryDest` as the `destination` of a subscriber function to define which tables Tabsdata writes in BigQuery. write\_orders.py ```python from tabsdatak.api import subscriber from tabsdatak.conn.bigquery import BigQueryDest @subscriber( destination=BigQueryDest( tables=[ "orders", ], ), input_tables=["orders"], ) def write_orders(orders): return orders ``` - required - optional - optional - optional ### Function Config Parameters #### `tables` Defines the destination tables the subscriber writes to in BigQuery. Each element in `tables` represents one destination slot and maps positionally to a value returned by the subscriber function. Each name is `[[project.]dataset.]table`; a bare `table` or `dataset.table` is completed from the connection's `project`/`dataset` defaults. BigQuery wraps the whole name in one backtick pair, e.g. `` `my-project.ds.t` ``. #### `if_table_exists` `"merge"` applies inserts, updates, and deletes by key and requires a `merge_cfg` list. See [Merge configuration](https://docs.tabsdata.com/guide/functions/sql-merge#merge-configuration) for a complete example. Controls what happens when the destination table already exists. `"append"` (the default) adds rows. `"replace"` drops and recreates the table. #### `schema_evolution` Controls how new columns in the returned frame are handled. `"update"` allows new columns. `"strict"` is the default and leaves existing data columns unchanged. See [Merging into SQL destinations](https://docs.tabsdata.com/guide/functions/sql-merge#schema-evolution) for merge-specific behavior. #### `dest_cfg` Sets additional destination configuration. Supported keys include `bigquery.logging_level` and `bigquery.schema_evolution_engine`. Merge mode requires `bigquery.schema_evolution_engine="iceberg"`. ## Write behavior Each entry in `tables` is one destination slot and maps by position to a value returned by the subscriber function. Returning `None` for a slot skips that write. `if_table_exists="append"` adds rows. `"replace"` drops and recreates the table. `"merge"` applies inserts, updates, and deletes by key and needs `merge_cfg`. With `schema_evolution="strict"`, new columns are not added to an existing table. `"update"` allows new columns during append-style writes. ## Operational notes - Each write is staged as Parquet in GCS before the BigQuery load job runs. Both the staging write and the BigQuery load must succeed. - Bare table names use the connection's `project` and `dataset`. Fully qualified names override those defaults. - Tabsdata does not contact GCS or BigQuery when the collection is created. Connection and permission errors appear when the subscriber runs. ## Connection BigQuery subscribers use `BigQueryDestConn` to define the GCS staging location and credentials used by the subscriber. The connector stages each write as parquet in GCS, then runs a load job against BigQuery referencing that object. conn-bigquery.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.bigquery:BigQueryDestConn spec: staging: bucket: 'acme-stage' credentials: kind: gcpJsonCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:GCPJson spec: json: 'secret:{"type": "service_account", ...}' base_path: '/bigquery' credentials: kind: gcpJsonCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:GCPJson spec: json: 'secret:{"type": "service_account", ...}' ``` - required - required - optional - optional - optional - optional ### Create the collection A subscriber is registered in a destination collection, and a destination collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type bigquery-bulk-out --file conn-bigquery.yaml ``` Edit the template file with the BigQuery connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-bigquery.yaml ``` ```bash tdk collection create --name bigquery_out --conn-file conn-bigquery.yaml ``` Then register the subscriber in the collection: ```bash tdk fn register --coll bigquery_out --path write_orders.py::write_orders ``` - The collection goes in the `destinations` group, because the connection is a destination connection. `--group` can be left out; if it is given, it must be `destinations`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `BigQueryDest` for a collection created with `BigQueryDestConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name bigquery_out --conn-file conn-bigquery.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. ### Connection Config Parameters #### `staging` The `GCSLocation` used as the parquet staging bucket for load jobs: a `bucket`, `credentials` (GCP service-account JSON), and `base_path` (defaults to `/`, same rules as other connectors' `base_path`: starts with `/`, cannot end with `/`, cannot contain empty path segments). #### `credentials` The GCP service-account JSON credentials (`GCPJson`) used to run the BigQuery load job, separate from the credentials on `staging` used to upload to GCS. #### `project` The default GCP project id. Required when `dataset` is set. #### `dataset` The default BigQuery dataset id. Requires `project` to be set. #### `entity_whitelist` A YAML list of regular expressions, not a comma-separated string. It is checked to be a list of valid regular expressions, but it has no effect yet. Leave it out. #### `conn_cfg` Optional connector configuration overrides. --- # ClickHouse > Reference for the Tabsdata ClickHouse destination connector, cloud staging, and subscriber configuration. Source: https://docs.tabsdata.com/guide/connectors/clickhouse The ClickHouse connector lets Tabsdata subscribers write data from Tabsdata tables into ClickHouse. The connector stages the data as Parquet files in cloud storage, then loads it into ClickHouse from there. [See the full subscriber walkthrough](https://docs.tabsdata.com/tutorials/subscribers/clickhouse) The subscriber is registered in a collection created with a `ClickHouseDestConn` connection. The connection, and how to create the collection, are described in [Connection](#connection), at the end of this page. ### Prerequisites - The ClickHouse HTTP or HTTPS endpoint is reachable from the Tabsdata server. - The ClickHouse user can create and write destination tables, and can drop them when replace mode is used. - Tabsdata can write to the staging location and ClickHouse can read the staged Parquet files. ### Subscriber Configure `ClickHouseDest` as the `destination` of a subscriber function to define which tables Tabsdata writes in ClickHouse. write\_orders.py ```python from tabsdatak.api import subscriber, TableFrameSpec from tabsdatak.conn.clickhouse import ClickHouseDest @subscriber( destination=ClickHouseDest( tables=[ "orders", ], ), input_tables=["orders"], ) def write_orders(orders: TableFrameSpec) -> TableFrameSpec: return orders ``` - required - optional - optional - optional ### Function Config Parameters #### `tables` Defines the destination tables the subscriber writes to in ClickHouse, as a non-empty list. Each name is `table` or `database.table`; quote a part with backticks if it has special characters. Each element in `tables` represents one destination slot and maps positionally to a value returned by the subscriber function. #### `if_table_exists` `"append"` (the default) adds rows to the existing table. `"replace"` overwrites the table: the first run creates it, and later runs drop and recreate it. #### `schema_evolution` `"strict"` (the default) rejects a write whose schema differs from the table's. `"iceberg"` applies Iceberg-style schema changes, so an appended frame with a new column adds the column to the table. #### `dest_cfg` Sets additional destination configuration. The only accepted key is `clickhouse.logging_level`. ## Write behavior Each entry in `tables` is one destination slot and maps by position to a value returned by the subscriber function. Returning `None` for a slot skips that write. `if_table_exists="append"` adds rows. `"replace"` drops and recreates the table. With `schema_evolution="strict"`, a schema mismatch fails; `"iceberg"` applies supported schema changes. ## Operational notes - The connector stages Parquet files before ClickHouse reads them. A run needs both staging access and ClickHouse access. - GCS staging needs service-account JSON for uploads and HMAC credentials for ClickHouse's S3-compatible reads. - Tabsdata does not contact ClickHouse or the staging service when the collection is created. Those checks happen when the subscriber runs. ## Connection ClickHouse subscribers use `ClickHouseDestConn` to define the server, the credentials, and the cloud storage the data is staged in. conn-clickhouse.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.clickhouse:ClickHouseDestConn spec: host: 'clickhouse.internal' credentials: kind: userPasswordCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:UserPassword spec: user: 'secret:tabsdata_writer' password: 'secret:example-password' staging: kind: clickHouseS3Location apiVersion: '1.0' type: tabsdatak.conn.clickhouse:S3Location spec: bucket: 'acme-staging' region: 'us-east-1' credentials: kind: awsAccessSecretKeyCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:AwsAccessSecretKey spec: access_key_id: 'secret:example-access-key' secret_access_key: 'secret:example-secret-key' base_path: '/clickhouse' ``` - required - optional - optional - optional - required - required - optional - optional ### Create the collection A subscriber is registered in a destination collection, and a destination collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type clickhouse-bulk-out --file conn-clickhouse.yaml ``` Edit the template file with the ClickHouse connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-clickhouse.yaml ``` ```bash tdk collection create --name clickhouse_out --conn-file conn-clickhouse.yaml ``` Then register the subscriber in the collection: ```bash tdk fn register --coll clickhouse_out --path write_orders.py::write_orders ``` - The collection goes in the `destinations` group, because the connection is a destination connection. `--group` can be left out; if it is given, it must be `destinations`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `ClickHouseDest` for a collection created with `ClickHouseDestConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name clickhouse_out --conn-file conn-clickhouse.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. ### Connection Config Parameters #### `host`, `port`, `database` and `secure` `host` identifies the ClickHouse server. `port` is the HTTP(S) port, from 1 to 65535, and defaults to `"8123"`. `database` defaults to `"default"` and supplies the database for destination table names without one. `secure` is `"true"` for HTTPS and `"false"` (the default) for HTTP. #### `credentials` A `UserPassword` containing the ClickHouse username and password. Cloud storage credentials are configured separately under `staging`. #### `staging` Use one of the location classes from `tabsdatak.conn.clickhouse`: | Location | Required fields | Credentials | | --- | --- | --- | | `S3Location` | `bucket`, `region`, `credentials` | `AwsAccessSecretKey` | | `AzBlobLocation` | `container`, `credentials` | `AzureAccountKey` | | `GCSLocation` | `bucket`, `credentials`, `hmac_credentials` | `GCPJson` for uploads and `GCPHmac` for ClickHouse reads | Tabsdata must be able to upload the staged files, and ClickHouse must be able to read them. For GCS, both credential types are required: uploads use the native GCS API, while ClickHouse reads through the S3-compatible endpoint. #### `entity_whitelist` A YAML list of regular expressions, not a comma-separated string. It is checked to be a list of valid regular expressions, but it has no effect yet. Leave it out. #### `conn_cfg` Optional connector configuration. --- # Databricks > Reference for the Tabsdata Databricks connector. Source: https://docs.tabsdata.com/guide/connectors/databricks The Databricks connector lets Tabsdata run queries against Databricks and write tables into Databricks. The `DatabricksSrc` connector can be used by a publisher function to read data from Databricks into a Tabsdata table. The `DatabricksDest` connector can be used by a subscriber function to write data from a Tabsdata table into Databricks. ## Publisher The `DatabricksSrc` connector can be used by a publisher function to read data from Databricks into a Tabsdata table. [See the full publisher walkthrough](https://docs.tabsdata.com/tutorials/publishers/databricks) The publisher is registered in a collection created with a `DatabricksSrcConn` connection. The connection, and how to create the collection, are described in [Publisher connection](#publisher-connection), at the end of this page. ### Prerequisites - The Databricks workspace and SQL warehouse are reachable from the Tabsdata server. - The personal access token can use the SQL warehouse and read every table referenced by `queries`. - The configured Unity Catalog volume exists and the token can read and write its staging files. ### Publisher Configure `DatabricksSrc` as the `source` of a publisher function to select the queries that Tabsdata runs against Databricks. publish\_orders.py ```python from tabsdatak.api import publisher from tabsdatak.conn.databricks import DatabricksSrc @publisher( source=DatabricksSrc( queries=[ "SELECT * FROM main.default.orders", ], ), output_tables=["orders"], ) def publish_orders(orders): return orders ``` - required - optional - optional ### Function Config Parameters #### `queries` Defines the SQL queries the publisher runs against Databricks. Each element in `queries` represents one source slot and maps positionally to an argument in the publisher function. #### `initial_values` The first-run values for named parameters in `queries`, written `:name`. On later runs each parameter takes the value the publisher stored with `ctx.set_attr(name, value)`, so a query can read only the rows added since the last run. Parameters bind values such as IDs and timestamps, not identifiers. Catalog, schema and table names have to be written into the query, or set with `catalog` and `schema` on the connection. For example: ```python from tabsdatak.api import TableFrameSpec, TrxCtx, publisher from tabsdatak.conn.databricks import DatabricksSrc @publisher( source=DatabricksSrc( queries=["SELECT * FROM main.default.orders WHERE id > :last_id ORDER BY id"], initial_values={"last_id": 0}, ), output_tables=["orders"], ) def publish_orders(orders: TableFrameSpec, ctx: TrxCtx) -> TableFrameSpec: row = orders.max_for("id") if row is not None and row["id"] is not None: ctx.set_attr("last_id", int(row["id"])) return orders ``` #### `src_cfg` Sets additional source configuration. ## Subscriber The `DatabricksDest` connector can be used by a subscriber function to write data from a Tabsdata table into Databricks. [See the full subscriber walkthrough](https://docs.tabsdata.com/tutorials/subscribers/databricks) The subscriber is registered in a collection created with a `DatabricksDestConn` connection. The connection, and how to create the collection, are described in [Subscriber connection](#subscriber-connection), at the end of this page. ### Prerequisites - The Databricks workspace and SQL warehouse are reachable from the Tabsdata server. - The personal access token can use the SQL warehouse and create or write the destination tables. - The configured Unity Catalog volume exists and the token can read and write its staging files. ### Subscriber For merge writes, schema evolution, and retry watermarks, see [Merging into SQL destinations](https://docs.tabsdata.com/guide/functions/sql-merge). Configure `DatabricksDest` as the `destination` of a subscriber function to define which tables Tabsdata writes in Databricks. write\_orders.py ```python from tabsdatak.api import subscriber from tabsdatak.conn.databricks import DatabricksDest @subscriber( destination=DatabricksDest( tables=[ "orders", ], ), input_tables=["orders"], ) def write_orders(orders): return orders ``` - required - optional - optional - optional ### Function Config Parameters #### `tables` Defines the destination tables the subscriber writes to in Databricks. Each element in `tables` represents one destination slot and maps positionally to a value returned by the subscriber function. Destination table names can be fully qualified or completed from the connection's `catalog`/`schema` defaults. #### `if_table_exists` `"merge"` applies inserts, updates, and deletes by key and requires a `merge_cfg` list. See [Merge configuration](https://docs.tabsdata.com/guide/functions/sql-merge#merge-configuration) for a complete example. Controls what happens when the destination table already exists. `"append"` (the default) adds rows. `"replace"` overwrites the table. #### `schema_evolution` Controls how new columns in the returned frame are handled. `"update"` forwards `mergeSchema=true` to `COPY INTO` so new columns are added. `"strict"` is the default and leaves existing data columns unchanged. See [Merging into SQL destinations](https://docs.tabsdata.com/guide/functions/sql-merge#schema-evolution) for merge-specific behavior. #### `dest_cfg` Sets additional destination configuration. ## Query results Each entry in `queries` produces one source slot. The slots map by position to the publisher function arguments and then to `output_tables`. For incremental reads, put named parameters such as `:last_id` in a query, set their first values with `initial_values`, and save the next values with `ctx.set_attr()` after a successful run. ## Write behavior Each entry in `tables` is one destination slot and maps by position to a returned value. Returning `None` for a slot skips that write. `if_table_exists="append"` adds rows, `"replace"` overwrites the table, and `"merge"` applies inserts, updates, and deletes by key. With `schema_evolution="update"`, append-style writes can add columns; `"strict"` does not change existing data columns. ## Operational notes - Publisher queries and subscriber writes use the configured SQL warehouse. The Unity Catalog volume is the staging area used to move data. - One- and two-part table names use the connection's `catalog` and `schema` defaults. Fully qualified names override them. - Tabsdata does not contact the workspace when the collection is created. Workspace, token, warehouse, volume, and permission errors appear when the function runs. ## Publisher connection Databricks publishers use `DatabricksSrcConn` to define the workspace, token, and warehouse used by the publisher. conn-databricks-in.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.databricks:DatabricksSrcConn spec: host_url: 'secret:https://dbc-00000000.cloud.databricks.com' token: 'secret:dapi0example0token0000' volume: 'main.default.tabsdata_stage' ``` - required - required - required - optional - optional - optional - optional ### Create the collection A publisher is registered in a source collection, and a source collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type databricks-bulk-in --file conn-databricks-in.yaml ``` Edit the template file with the Databricks connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-databricks-in.yaml ``` ```bash tdk collection create --name databricks_in --conn-file conn-databricks-in.yaml ``` Then register the publisher in the collection: ```bash tdk fn register --coll databricks_in --path publish_orders.py::publish_orders ``` - The collection goes in the `sources` group, because the connection is a source connection. `--group` can be left out; if it is given, it must be `sources`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `DatabricksSrc` for a collection created with `DatabricksSrcConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name databricks_in --conn-file conn-databricks-in.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. ### Connection Config Parameters #### `host_url` The Databricks workspace URL, stored as a secret. #### `token` The Databricks personal access token used for workspace access. #### `volume` The managed Databricks volume the connector uses to stage files during reads and writes. #### `warehouse` The SQL warehouse name. Use this or `warehouse_id`, not both. #### `warehouse_id` An alternative to `warehouse`. The two fields are mutually exclusive. #### `catalog` The default catalog for partially qualified table names. #### `schema` The default schema. Requires `catalog` to be set. ## Subscriber connection Databricks subscribers use `DatabricksDestConn` to define the workspace, token, and warehouse used by the subscriber. conn-databricks-out.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.databricks:DatabricksDestConn spec: host_url: 'secret:https://dbc-00000000.cloud.databricks.com' token: 'secret:dapi0example0token0000' volume: 'main.default.tabsdata_stage' ``` - required - required - required - optional - optional - optional - optional ### Create the collection A subscriber is registered in a destination collection, and a destination collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type databricks-bulk-out --file conn-databricks-out.yaml ``` Edit the template file with the Databricks connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-databricks-out.yaml ``` ```bash tdk collection create --name databricks_out --conn-file conn-databricks-out.yaml ``` Then register the subscriber in the collection: ```bash tdk fn register --coll databricks_out --path write_orders.py::write_orders ``` - The collection goes in the `destinations` group, because the connection is a destination connection. `--group` can be left out; if it is given, it must be `destinations`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `DatabricksDest` for a collection created with `DatabricksDestConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name databricks_out --conn-file conn-databricks-out.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. ### Connection Config Parameters #### `host_url` The Databricks workspace URL, stored as a secret. #### `token` The Databricks personal access token used for writes. #### `volume` The managed volume used for staged files during `COPY INTO`. #### `warehouse` The SQL warehouse name. Use this or `warehouse_id`, not both. #### `warehouse_id` An alternative to `warehouse`. The two fields are mutually exclusive. #### `catalog` The default catalog for partially qualified table names. #### `schema` The default schema. Requires `catalog` to be set. --- # Db2 > Reference for the Tabsdata Db2 connector. Source: https://docs.tabsdata.com/guide/connectors/db2 The Db2 connector lets Tabsdata run queries against Db2 and write tables into Db2. The `Db2Src` connector can be used by a publisher function to read data from Db2 into a Tabsdata table. The `Db2Dest` connector can be used by a subscriber function to write data from a Tabsdata table into Db2. ## Publisher The `Db2Src` connector can be used by a publisher function to read data from Db2 into a Tabsdata table. [See the full publisher walkthrough](https://docs.tabsdata.com/tutorials/publishers/db2) The publisher is registered in a collection created with a `Db2SrcConn` connection. The connection, and how to create the collection, are described in [Publisher connection](#publisher-connection), at the end of this page. ### Prerequisites - The Db2 server is reachable from the Tabsdata server. - The connection user can run every query in `queries` and read every referenced table or view. ### Publisher Configure `Db2Src` as the `source` of a publisher function to select the queries that Tabsdata runs against Db2. publish\_orders.py ```python from tabsdatak.api import publisher from tabsdatak.conn.db2 import Db2Src @publisher( source=Db2Src( queries=[ "SELECT * FROM orders", ], ), output_tables=["orders"], ) def publish_orders(orders): return orders ``` - required - optional - optional ### Function Config Parameters #### `queries` Defines the SQL queries the publisher runs against Db2. Each element in `queries` represents one source slot and maps positionally to an argument in the publisher function. For example: ```python Db2Src( queries=[ "SELECT * FROM country", "SELECT * FROM city", ], ) ``` #### `initial_values` The first-run values for named parameters in `queries`, written `:name`. On later runs each parameter takes the value the publisher stored with `ctx.set_attr(name, value)`, so a query can read only the rows added since the last run. Db2 folds unquoted identifiers to uppercase, so the column read back is `ID`. Parameters bind values such as IDs and timestamps, not identifiers. Schema and table names have to be written into the query. For example: ```python from tabsdatak.api import TableFrameSpec, TrxCtx, publisher from tabsdatak.conn.db2 import Db2Src @publisher( source=Db2Src( queries=["SELECT * FROM orders WHERE id > :last_id ORDER BY id"], initial_values={"last_id": 0}, ), output_tables=["orders"], ) def publish_orders(orders: TableFrameSpec, ctx: TrxCtx) -> TableFrameSpec: row = orders.max_for("ID") if row is not None and row["ID"] is not None: ctx.set_attr("last_id", int(row["ID"])) return orders ``` #### `src_cfg` Sets additional source configuration. ## Subscriber The `Db2Dest` connector can be used by a subscriber function to write data from a Tabsdata table into Db2. The subscriber is registered in a collection created with a `Db2DestConn` connection. The connection, and how to create the collection, are described in [Subscriber connection](#subscriber-connection), at the end of this page. ### Prerequisites - The Db2 server is reachable from the Tabsdata server. - The connection user can create missing destination tables and insert rows. With `if_table_exists="replace"`, it can also truncate existing tables. ### Subscriber Configure `Db2Dest` as the `destination` of a subscriber function to define which tables Tabsdata writes in Db2. write\_orders.py ```python from tabsdatak.api import subscriber from tabsdatak.conn.db2 import Db2Dest @subscriber( destination=Db2Dest( tables=[ "orders", ], ), input_tables=["orders"], ) def write_orders(orders): return orders ``` - required - optional - optional ### Function Config Parameters #### `tables` Defines the destination tables the subscriber writes to in Db2. Each element in `tables` represents one destination slot and maps positionally to a value returned by the subscriber function. For example: ```python Db2Dest( tables=[ "vendors", "items", ], ) ``` #### `if_table_exists` Controls what happens when the destination table already has rows. `"append"` (the default) keeps inserting into the existing rows. `"replace"` truncates the table before inserting. Neither mode changes the destination table's schema, so the returned frame's columns need to match the table Db2 already has. #### `dest_cfg` Sets additional destination configuration. ## Query results Each entry in `queries` produces one source slot. The slots map by position to the publisher function arguments and then to `output_tables`. Keep those three lists in the same order. Queries run when the publisher is triggered. They are not a continuous change feed. To read incrementally, put named parameters such as `:last_id` in the query, set their first values with `initial_values`, and save the next values with `ctx.set_attr()` after each successful run. ## Write behavior Each entry in `tables` is one destination slot and maps by position to a value returned by the subscriber function. Returning `None` for a slot skips that write. `if_table_exists="append"` keeps the rows already in the destination and inserts the returned rows. `"replace"` truncates the table before inserting. Neither mode changes an existing table's schema, so column additions, removals, and type changes need a database migration before the subscriber runs. ## Operational notes - Without `trigger_by`, a publisher runs only when triggered manually, while a subscriber runs when any input table commits. Set `trigger_by` to change when the function runs. - Tabsdata checks the connection document when the collection is created, but it does not connect to Db2 then. A bad host, credential, query, or permission fails when the function runs. - Every function in the collection uses the same connection. Use another collection when functions need another database or another set of credentials. ## Publisher connection Db2 publishers use `Db2SrcConn` to define the database connection and credentials used by the publisher. conn-db2-in.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.db2:Db2SrcConn spec: uri: 'db2+ibm_db://db.internal:50000/world' credentials: kind: userPasswordCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:UserPassword spec: user: 'secret:tabsdata_reader' password: 'secret:example-password' ``` - required - required - optional - optional ### Create the collection A publisher is registered in a source collection, and a source collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type db2-sql-in --file conn-db2-in.yaml ``` Edit the template file with the Db2 connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-db2-in.yaml ``` ```bash tdk collection create --name db2_in --conn-file conn-db2-in.yaml ``` Then register the publisher in the collection: ```bash tdk fn register --coll db2_in --path publish_orders.py::publish_orders ``` - The collection goes in the `sources` group, because the connection is a source connection. `--group` can be left out; if it is given, it must be `sources`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `Db2Src` for a collection created with `Db2SrcConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name db2_in --conn-file conn-db2-in.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. ### Connection Config Parameters #### `uri` The SQLAlchemy-style connection URI: `db2+ibm_db://host:port/database`. Written as a plain value here, but accepts `secret:` too. #### `credentials` The database user and password, spliced into the URI at connect time. `UserPassword` requires a `user` and `password`. #### `entity_whitelist` A YAML list of regular expressions, not a comma-separated string. It is checked to be a list of valid regular expressions, but it has no effect yet. Leave it out. #### `conn_cfg` Driver-side options forwarded to the connection as SQLAlchemy `connect_args`. ## Subscriber connection Db2 subscribers use `Db2DestConn` to define the database connection and credentials used by the subscriber. conn-db2-out.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.db2:Db2DestConn spec: uri: 'db2+ibm_db://db.internal:50000/warehouse' credentials: kind: userPasswordCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:UserPassword spec: user: 'secret:tabsdata_reader' password: 'secret:example-password' ``` - required - required - optional - optional ### Create the collection A subscriber is registered in a destination collection, and a destination collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type db2-sql-out --file conn-db2-out.yaml ``` Edit the template file with the Db2 connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-db2-out.yaml ``` ```bash tdk collection create --name db2_out --conn-file conn-db2-out.yaml ``` Then register the subscriber in the collection: ```bash tdk fn register --coll db2_out --path write_orders.py::write_orders ``` - The collection goes in the `destinations` group, because the connection is a destination connection. `--group` can be left out; if it is given, it must be `destinations`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `Db2Dest` for a collection created with `Db2DestConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name db2_out --conn-file conn-db2-out.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. ### Connection Config Parameters #### `uri` The SQLAlchemy-style connection URI: `db2+ibm_db://host:port/database`. Written as a plain value here, but accepts `secret:` too. #### `credentials` The database user and password, spliced into the URI at connect time. `UserPassword` requires a `user` and `password`. #### `entity_whitelist` A YAML list of regular expressions, not a comma-separated string. It is checked to be a list of valid regular expressions, but it has no effect yet. Leave it out. #### `conn_cfg` Driver-side options forwarded to the connection as SQLAlchemy `connect_args`. --- # Db2 CDC > Reference for the Tabsdata Db2 CDC stream source: SQL Replication prerequisites, the full load, change rows, and the connection. Source: https://docs.tabsdata.com/guide/connectors/db2-cdc The Db2 CDC connector lets Tabsdata capture the changes made to Db2 tables (inserts, updates and deletes) and publish them into Tabsdata tables. With `start_from="full"`, the stream starts with a full load: it takes a consistent snapshot of the tracked tables and then switches to change capture from the exact point of that snapshot, with no gaps and no duplicates. The full load and the change capture are one stream publisher, so there is nothing to reconcile between them. The `Db2CdcStreamSrc` connector can be used by a [stream publisher](https://docs.tabsdata.com/guide/functions/stream-publishers) function to read the change-data (CD) tables that Db2 SQL Replication fills, and write every captured change into a Tabsdata table. Unlike `Db2Src`, which runs queries on a trigger, a CDC source runs continuously: registering the stream publisher starts a long-lived worker that keeps reading the CD tables until the function is disabled or deleted. ## Publisher The `Db2CdcStreamSrc` connector can be used by a stream publisher function to capture changes from Db2 into Tabsdata tables. [See the full publisher walkthrough](https://docs.tabsdata.com/tutorials/publishers/db2-cdc) The stream publisher is registered in a collection created with a `Db2CdcStreamSrcConn` connection. The connection, and how to create the collection, are described in [Connection](#connection), at the end of this page. ### Prerequisites Db2 CDC builds on Db2 SQL Replication: the ASN Capture program (`asncap`) reads the Db2 log and copies each change of a registered table into that table's CD table. Tabsdata reads the CD tables. - SQL Replication is set up in the source database: the `ASN` capture control tables exist, and `asncap` is running. - Each tracked source table is registered for capture (a row in `ASN.IBMSNAP_REGISTER`), with an active registration and its own CD table. Every column is registered with its before-image; a registration without before-images is refused. - The connecting user can `SELECT` the CD tables and `ASN.IBMSNAP_REGISTER`, `ASN.IBMSNAP_PRUNCNTL` and `ASN.IBMSNAP_RESTART`, and `ASN.IBMSNAP_UOW` when starting from `"tail"` or a timestamp. - For `start_from="full"`, the connecting user can also `SELECT` every tracked source table and `EXECUTE` the `SYSPROC.MON_GET_TRANSACTION_LOG` table function (for example through the `SYSMON` authority), and no tracked table is registered to capture updates as a delete plus an insert (`CHG_UPD_TO_DEL_INS = 'Y'`). ### Publisher Configure `Db2CdcStreamSrc` as the `source` of a `@stream_publisher` function to select the CD tables whose changes Tabsdata captures. stream\_sales\_changes.py ```python from tabsdatak.api import TableFrameSpec, TableFramesSpec, stream_publisher from tabsdatak.conn.db2 import Db2CdcStreamSrc from tabsdatak.tableframe.functions import concat @stream_publisher( source=Db2CdcStreamSrc( tables=[ "ASN.CD_ORDERS", "ASN.CD_CUSTOMERS", ], start_from="full", ), batch_max_secs=120, output_tables=["orders_changes", "customers_changes"], ) def stream_sales_changes( orders: TableFramesSpec, customers: TableFramesSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return concat(orders), concat(customers) ``` - required - required - optional - optional - optional `batch_max_secs=120` hands the captured changes to the function at least every two minutes. The `batch_*` parameters decide when a batch of changes is handed to the function, and are the same for every stream source. See [Stream Publishers](https://docs.tabsdata.com/guide/functions/stream-publishers#batch-parameters) for all of them. A stream publisher differs from a regular publisher in three ways: - It has no trigger. Registering it starts the stream worker, which runs until the function is disabled or deleted. - Each element of `tables` is one function parameter, and each parameter receives a list of frames: the changes captured for that table since the previous batch. Fold them with `concat`, which also returns `None` when the batch carried nothing for that table. - The function must not declare a `ctx` parameter. ### Function Config Parameters #### `tables` Defines the CD tables to read, as qualified `cd_schema.cd_table` names. These are the change-data tables, not the source tables. The connector finds each CD table's source table through `ASN.IBMSNAP_REGISTER`. Db2 folds unquoted names to uppercase, so write them in uppercase. Each element in `tables` represents one source slot and maps positionally to an argument in the publisher function and to an entry in `output_tables`. For example: ```python Db2CdcStreamSrc( tables=[ "ASN.CD_ORDERS", "ASN.CD_CUSTOMERS", ], start_from="tail", ) ``` #### `start_from` Where the stream starts the first time it runs. Later runs resume from the position the stream has recorded, which it keeps for each CD table separately. - `"full"`: a full load of every tracked source table, then the changes captured after it. See [Full load](#full-load). - `"tail"`: only changes captured after the stream starts. - `"head"`: every change still in each CD table. - `CommitSeqPosition(seq="...")`: one commit sequence for every table, as the hex form of the 16-byte `IBMSNAP_COMMITSEQ` value. - `TableCommitSeqPosition(seqs={"ASN.CD_ORDERS": "...", ...})`: a commit sequence per CD table, for example after re-snapshotting some tables. Every CD table in `tables` must be listed; a table left out fails the stream. - `TimestampPosition(ts=...)`: from the first commit at or after a point in time, looked up in `ASN.IBMSNAP_UOW`. A naive datetime is read in the Db2 server's time zone. #### Full load With `start_from="full"`, the stream first emits every row of each tracked source table and then captures the changes from the exact position of that snapshot. The snapshot is consistent across all the tracked tables, and no change is missed or repeated between the snapshot and the change capture. - Each table's snapshot is preceded by one `LOAD` row (`@td.cdc.meta.op = "l"`), and its rows arrive as inserts. A `LOAD` row tells the consumer to empty its copy of the table before applying the rows that follow. - The full load reads the source tables rather than the CD tables, and locks nothing. The changes committed while the tables are read are taken from the CD tables and reconciled with the rows read, so that every change is published once. - `asncap` must be running, since the load waits for Capture to reach the end of the load before handing over to the change stream. To resynchronize a destination later, keep `start_from="full"` and reset the function's stage: ```bash tdk fn update --reset-stage ``` The stream then starts again with a new full load and its `LOAD` rows. Resetting the stage with any other `start_from` restarts at that position and sends no `LOAD` row. #### `columns_format` How each change row lays out the table's column values: `"columns"` (the default), `"struct"` or `"map"`. See [Change rows](#change-rows). #### `flatten_columns` When `True` (the default), the new column values are written as ordinary top-level columns, so each change row reads like a row of the source table. `columns_format` then only shapes the old values. See [Change rows](#change-rows). #### `src_cfg` Sets additional source configuration. Only these keys are accepted: | Key | Default | Description | | --- | --- | --- | | `tabsdata.cdc.buffer.max_rows` | `10000` | Rows buffered in the worker before they are written out to a file. | | `tabsdata.cdc.buffer.max_bytes` | none | Bytes buffered before they are written out. Measuring bytes costs time on every change; prefer bounding by rows. | | `tabsdata.cdc.buffer.max_secs` | `60` | Longest time a change waits in the buffer. | | `tabsdata.cdc.poll.max_secs` | `1` | Longest time one read of the CD tables waits for changes. | | `tabsdata.cdc.poll.idle_secs` | `1` | Pause after a read that found no changes. | | `tabsdata.cdc.fetch_size` | `100000` | Rows fetched per round trip. | | `tabsdata.cdc.chunk_size` | `100000` | Rows per chunk while writing the full load. | | `tabsdata.cdc.initial_load.max_window_changes` | `100000` | Most changes that may be committed to the tracked tables while the full load reads them. Above it, the load fails; run it again at a quieter time or raise the limit. | | `db2.cdc.logging_level` | `40` | Logging verbosity of the connector, as a Python logging level number (`10` debug, `20` info, `30` warning, `40` error). | The buffer settings control when captured changes are written to files inside the worker. When those files are handed to the function is controlled by the [batch parameters](https://docs.tabsdata.com/guide/functions/stream-publishers#batch-parameters) of `@stream_publisher`. ## Change rows Every row the stream emits is one change. Besides the column values, each row carries change metadata: | Column | Type | Description | | --- | --- | --- | | `@td.cdc.meta.op` | `str` | The operation: `"i"` insert, `"u"` update, `"d"` delete, `"l"` load (start of a full load). | | `@td.cdc.meta.tx` | `str` | The commit sequence of the transaction, in hex. Full load rows carry `initial-load`. | | `@td.cdc.meta.sq` | `int` | The change's sequence (`IBMSNAP_INTENTSEQ`). | | `@td.cdc.meta.ts` | `datetime` (UTC) | When the transaction committed. Full load rows carry the time the snapshot was taken. | | `@td.cdc.meta.seg` | `decimal` | The transaction's commit sequence (`IBMSNAP_COMMITSEQ`). Full load rows carry `-1`. | | `@td.cdc.meta.ord` | `decimal` | The change's sequence (`IBMSNAP_INTENTSEQ`). Ordering by `seg` and then `ord` gives the order the changes were committed in. | | `@td.cdc.meta.fmt` | `str` | The `columns_format` the row was produced with. | | `@td.cdc.meta.flat` | `bool` | The `flatten_columns` setting the row was produced with. | | `@td.cdc.meta.sch` | `str` | A hash of the source table's schema when the change was captured. It changes when the table's columns change. | The column values are laid out according to `flatten_columns` and `columns_format`: | | New values | Old values | | --- | --- | --- | | `flatten_columns=True` (default) | One top-level column per source column, with the source column's name. | Depends on `columns_format`. | | `columns_format="columns"` (default) | `@td.cdc.data.col.new.` per column, when not flattened. | `@td.cdc.data.col.old.` per column. | | `columns_format="struct"` | `@td.cdc.data.row.new`, one struct, when not flattened. | `@td.cdc.data.row.old`, one struct. | | `columns_format="map"` | `@td.cdc.data.map.new`, a JSON object as a string, when not flattened. | `@td.cdc.data.map.old`, a JSON object as a string. | What each operation carries: | Operation | New values | Old values | | --- | --- | --- | | Insert (`i`) | The inserted row. | Empty. | | Update (`u`) | The row after the update. | The row before the update. | | Delete (`d`) | The deleted row. | Empty. | | Load (`l`) | Empty. | Empty. | Empty values are null columns with `"columns"`, a struct whose fields are all null with `"struct"`, and a JSON object whose values are all null with `"map"`. A `LOAD` row tells the consumer to empty its copy of the table before applying the rows that follow. SQL Replication does not copy LOB or XML values into the CD tables, so those columns always arrive as null. Use `"columns"` with `flatten_columns=True` for changes that land in a table and get queried. Use `"struct"` or `"map"` when the source table's schema may change while the stream runs, to carry the values opaquely instead of changing the output table's columns. ## Operational notes - The stream reads only what `asncap` has captured. If Capture stops, the stream sees no new changes until it is running again. - If a tracked table is deregistered, its registration is deactivated, Capture cold-starts, or pruning removes CD rows the stream has not read yet, the stream fails with an error explaining what happened. The fix is to restore the registration if needed and run a new full load: reset the stage with `start_from="full"`. - A CD table added to `tables` after the stream started has no recorded position, so the stream fails until a new full load runs. ## Connection Db2 CDC stream publishers use `Db2CdcStreamSrcConn` to define the database connection and credentials used by the stream. conn-db2-cdc.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.db2:Db2CdcStreamSrcConn spec: uri: 'db2://db.internal:50000/SALES' credentials: kind: userPasswordCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:UserPassword spec: user: 'secret:tabsdata_cdc' password: 'secret:example-password' ``` - required - required - optional - optional ### Create the collection A stream publisher is registered in a source collection, and a source collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type db2-cdc --file conn-db2-cdc.yaml ``` Edit the template file with the Db2 connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-db2-cdc.yaml ``` ```bash tdk collection create --name sales_cdc --conn-file conn-db2-cdc.yaml ``` Then register the stream publisher in the collection: ```bash tdk fn register --coll sales_cdc --path stream_sales_changes.py::stream_sales_changes ``` - The collection goes in the `sources` group, because the connection is a source connection. `--group` can be left out; if it is given, it must be `sources`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `Db2CdcStreamSrc` for a collection created with `Db2CdcStreamSrcConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name sales_cdc --conn-file conn-db2-cdc.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. - A batch `Db2Src` publisher cannot share this collection, because it uses a different connection (`db2-sql-in`). ### Connection Config Parameters #### `uri` The SQLAlchemy-style connection URI: `db2://host:port/database` (or `ibm_db_sa://`). Written as a plain value here, but accepts `secret:` too. #### `credentials` The database user and password, spliced into the URI at connect time. `UserPassword` requires a `user` and `password`. #### `entity_whitelist` A YAML list of regular expressions, not a comma-separated string. It is checked to be a list of valid regular expressions, but it has no effect yet: it does not limit which tables a stream can track. Leave it out. #### `conn_cfg` ibm\_db connection keywords, forwarded as SQLAlchemy `connect_args` to every connection the stream opens, including the full load's. --- # Google Cloud Storage > Reference for the Tabsdata Google Cloud Storage connector. Source: https://docs.tabsdata.com/guide/connectors/gcp The Google Cloud Storage connector lets Tabsdata read files from and write files to Google Cloud Storage. The `GCSSrc` connector can be used by a publisher function to read data from Google Cloud Storage into a Tabsdata table. The `GCSDest` connector can be used by a subscriber function to write data from a Tabsdata table into Google Cloud Storage. ## Supported formats | Format | Publish | Subscribe | | --- | --- | --- | | `AUTO` | Yes | Yes | | `CSV` | Yes | Yes | | `JSON` | Yes | Yes | | `AVRO` | Yes | Yes | | `PARQUET` | Yes | Yes | | `LOG` | Yes | No | ## Publisher The `GCSSrc` connector can be used by a publisher function to read data from Google Cloud Storage into a Tabsdata table. [See the full publisher walkthrough](https://docs.tabsdata.com/tutorials/publishers/gcp) The publisher is registered in a collection created with a `GCSSrcConn` connection. The connection, and how to create the collection, are described in [Publisher connection](#publisher-connection), at the end of this page. ### Prerequisites - The bucket exists and is reachable from the Tabsdata server. - The service account can list the bucket and read every object matched by `paths`. - The object format and compression match `format` and `format_cfg`. ### Publisher Configure `GCSSrc` as the `source` of a publisher function to select the files that Tabsdata reads from Google Cloud Storage. publish\_orders.py ```python from tabsdatak.api import publisher from tabsdatak.conn.gcs import GCSSrc, SrcFileFormat @publisher( source=GCSSrc( paths=[ "incoming/orders.parquet", ], ), output_tables=["orders"], ) def publish_orders(orders): return orders ``` - required - optional - optional - optional - optional ### Function Config Parameters #### `paths` Defines the files that the publisher reads from Google Cloud Storage. Each path is relative to the `base_path` configured in `GCSSrcConn`. Source paths: - cannot begin or end with `/` - cannot contain empty path segments - can contain a `*` glob in the final path segment For example: ```python GCSSrc( paths=[ "orders/2026/*.parquet", "customers/customers.csv", ] ) ``` Each element in `paths` represents one source slot and maps positionally to an argument in the publisher function. When a path uses a glob to match multiple files, the matching files are provided through the corresponding publisher input. #### `format` Sets the format used to read the source files. Supported formats are `AUTO`, `CSV`, `JSON`, `AVRO`, `PARQUET`, and `LOG`. The default is `AUTO`, which determines the format from the file extension. #### `format_cfg` Sets format-specific reader options. Configuration is keyed by `SrcFileFormat`. #### `initial_last_modified` Sets the initial cutoff for incremental ingestion. On the first run, Tabsdata imports files modified at or after the specified timestamp. The cutoff then advances so subsequent runs only import newer files. #### `src_cfg` Sets additional source configuration. The following options are supported: | Option | Description | | --- | --- | | `tabsdata.file.chunk_size` | Controls the size of chunks used when reading files. | | `tabsdata.src_metadata.drop` | When `True`, excludes the `@td.file.path` metadata column. | By default, rows read from Google Cloud Storage include a `@td.file.path` metadata column containing the source file's URI. ## Subscriber The `GCSDest` connector can be used by a subscriber function to write data from a Tabsdata table into Google Cloud Storage. [See the full subscriber walkthrough](https://docs.tabsdata.com/tutorials/subscribers/gcp) The subscriber is registered in a collection created with a `GCSDestConn` connection. The connection, and how to create the collection, are described in [Subscriber connection](#subscriber-connection), at the end of this page. ### Prerequisites - The bucket exists and is reachable from the Tabsdata server. - The service account can create and write objects under the configured `base_path`. - The destination path is relative and does not contain a glob. ### Subscriber Configure `GCSDest` as the `destination` of a subscriber function to define where Tabsdata writes files in Google Cloud Storage. write\_orders.py ```python from tabsdatak.api import subscriber from tabsdatak.conn.gcs import GCSDest, DestFileFormat @subscriber( destination=GCSDest( paths=[ "exports/orders.parquet", ], ), input_tables=["orders"], ) def write_orders(orders): return orders ``` - required - optional - optional - optional ### Function Config Parameters #### `paths` Defines where the subscriber writes files in Google Cloud Storage. Each path is relative to the `base_path` configured in `GCSDestConn`. Destination paths: - cannot begin or end with `/` - cannot contain empty path segments - cannot contain globs Each element in `paths` represents one destination slot and maps positionally to a value returned by the subscriber function. Destination paths can contain variables that Tabsdata substitutes when the subscriber runs: | Variable | Value | | --- | --- | | `${EXEC_PLAN_ID}` | Current execution plan ID. | | `${TRX_ID}` | Current transaction ID. | | `${EXEC_PLAN_TS}` | Timestamp when the execution plan started. | For example: ```python GCSDest( paths=[ "exports/${EXEC_PLAN_ID}/orders.parquet", ] ) ``` Dynamic paths can be used to create a new object for each execution instead of overwriting an existing object. #### `format` Sets the format used to write the destination file. Supported formats are `AUTO`, `CSV`, `JSON`, `AVRO`, and `PARQUET`. The default is `AUTO`, which determines the format from the destination file extension. #### `format_cfg` Sets format-specific writer options. Configuration is keyed by `DestFileFormat`. #### `dest_cfg` Sets additional destination configuration. The `tabsdata.file.chunk_size` option controls the size of chunks used when writing files. ## File behavior Each entry in a publisher's `paths` is one source slot. A literal path reads one object; a final-segment `*` glob can read several. The slots map by position to the publisher function arguments and then to `output_tables`. `initial_last_modified` sets the cutoff for the first run. After a successful run, Tabsdata advances the cutoff so the same unchanged objects are not published again. Each entry in a subscriber's `paths` is one destination slot and maps by position to a returned value. Use `${EXEC_PLAN_ID}`, `${TRX_ID}`, or `${EXEC_PLAN_TS}` in a path when every run needs a new object. A fixed path writes to the same location again. ## Operational notes - Without `trigger_by`, a publisher runs only when triggered manually, while a subscriber runs when any input table commits. Set `trigger_by` to change when the function runs. - Tabsdata validates the connection document when the collection is created, but it does not contact Google Cloud Storage then. A bad path, credential, or permission fails when the function runs. - Globs are allowed only for publisher source paths, and only in the final path segment. Subscriber destination paths cannot contain globs. - All configured paths are relative to the connection's `base_path`. ## Publisher connection Google Cloud Storage publishers use `GCSSrcConn` to define the bucket and credentials used by the publisher. conn-gcs-in.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.gcs:GCSSrcConn spec: bucket: 'acme-ingest' credentials: kind: gcpJsonCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:GCPJson spec: json: 'secret:{"type":"service_account",...}' ``` - required - required - optional - optional - optional ### Create the collection A publisher is registered in a source collection, and a source collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type gcs-file-in --file conn-gcs-in.yaml ``` Edit the template file with the Google Cloud Storage connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-gcs-in.yaml ``` ```bash tdk collection create --name gcs_in --conn-file conn-gcs-in.yaml ``` Then register the publisher in the collection: ```bash tdk fn register --coll gcs_in --path publish_orders.py::publish_orders ``` - The collection goes in the `sources` group, because the connection is a source connection. `--group` can be left out; if it is given, it must be `sources`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `GCSSrc` for a collection created with `GCSSrcConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name gcs_in --conn-file conn-gcs-in.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. ### Connection Config Parameters #### `bucket` The name of the GCS bucket that contains the files to publish. #### `credentials` The service-account JSON credential used to access the bucket. `GCPJson` requires a `json` key holding the full service-account key document. #### `project` Optional GCP project id associated with the bucket. #### `base_path` Sets the base path within the bucket. All paths configured in `GCSSrc` are relative to this path. `base_path` starts with `/`, cannot end with `/`, and cannot contain empty path segments. The default is `/`. #### `conn_cfg` Optional connector configuration overrides. ## Subscriber connection Google Cloud Storage subscribers use `GCSDestConn` to define the bucket and credentials used by the subscriber. conn-gcs-out.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.gcs:GCSDestConn spec: bucket: 'acme-exports' credentials: kind: gcpJsonCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:GCPJson spec: json: 'secret:{"type":"service_account",...}' ``` - required - required - optional - optional - optional ### Create the collection A subscriber is registered in a destination collection, and a destination collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type gcs-file-out --file conn-gcs-out.yaml ``` Edit the template file with the Google Cloud Storage connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-gcs-out.yaml ``` ```bash tdk collection create --name gcs_out --conn-file conn-gcs-out.yaml ``` Then register the subscriber in the collection: ```bash tdk fn register --coll gcs_out --path write_orders.py::write_orders ``` - The collection goes in the `destinations` group, because the connection is a destination connection. `--group` can be left out; if it is given, it must be `destinations`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `GCSDest` for a collection created with `GCSDestConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name gcs_out --conn-file conn-gcs-out.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. ### Connection Config Parameters #### `bucket` The name of the GCS bucket where the subscriber writes files. #### `credentials` The service-account JSON credential used to access the bucket. `GCPJson` requires a `json` key holding the full service-account key document. #### `project` Optional GCP project id associated with the bucket. #### `base_path` Sets the base path within the bucket. All paths configured in `GCSDest` are relative to this path. `base_path` starts with `/`, cannot end with `/`, and cannot contain empty path segments. The default is `/`. #### `conn_cfg` Optional connector configuration overrides. --- # Kafka > Reference for the Tabsdata Kafka stream source: message formats, schema registries, dead letter topics, consumer groups, and the connection. Source: https://docs.tabsdata.com/guide/connectors/kafka The Kafka connector lets Tabsdata read messages from Kafka topics as they arrive, and publish them into Tabsdata tables. The `KafkaStreamSrc` connector can be used by a [stream publisher](https://docs.tabsdata.com/guide/functions/stream-publishers) function to consume a topic, decode its messages from JSON, Avro or Protobuf, with or without a schema registry, and write them into a Tabsdata table. Unlike a regular publisher, which runs on a trigger, a Kafka source runs continuously: registering the stream publisher starts a long-lived worker that keeps consuming the topic until the function is disabled or deleted. ## Publisher The `KafkaStreamSrc` connector can be used by a stream publisher function to publish Kafka messages into Tabsdata tables. [See the full publisher walkthrough](https://docs.tabsdata.com/tutorials/publishers/kafka) The stream publisher is registered in a collection created with a `KafkaStreamSrcConn` connection. The connection, and how to create the collection, are described in [Connection](#connection), at the end of this page. ### Prerequisites - The brokers are reachable from the Tabsdata server. - The connecting principal can `Describe` and `Read` the topic (every matching topic, for a regular expression), `Read` the consumer group, and read the cluster metadata. - With a `dlq_topic`, the principal can also `Describe` and `Write` the dead letter topic, and the topic exists. The connector does not create it. - With a Confluent schema registry, the registry user can read schemas. - With an AWS Glue schema registry, the AWS credentials allow `glue:GetSchemaVersion` and `glue:GetSchemaByDefinition`, and also `glue:RegisterSchemaVersion` and `glue:CreateSchema` unless the schema given in the publisher is already registered. See [`schema`](#schema). ### Publisher Configure `KafkaStreamSrc` as the `source` of a `@stream_publisher` function to select the topic Tabsdata consumes. read\_orders.py ```python from tabsdatak.api import TableFrameSpec, TableFramesSpec, stream_publisher from tabsdatak.conn.kafka import KafkaStreamSrc from tabsdatak.tableframe.functions import concat ORDER_SCHEMA = """ { "type": "object", "properties": { "order_id": {"type": "integer"}, "customer": {"type": "string"}, "amount": {"type": "number"} }, "required": ["order_id", "customer", "amount"] } """ @stream_publisher( source=KafkaStreamSrc( topic="orders", data_format="json", schema=ORDER_SCHEMA, ), batch_max_secs=120, output_tables=["orders"], ) def read_orders(orders: TableFramesSpec) -> TableFrameSpec: return concat(orders, how="diagonal_relaxed") ``` - required - required - required - optional - optional `batch_max_secs=120` hands the consumed messages to the function at least every two minutes. The `batch_*` parameters decide when a batch of messages is handed to the function, and are the same for every stream source. See [Stream Publishers](https://docs.tabsdata.com/guide/functions/stream-publishers#batch-parameters) for all of them. A stream publisher differs from a regular publisher in three ways: - It has no trigger. Registering it starts the stream worker, which runs until the function is disabled or deleted. - A Kafka source has a single slot, so the function has a single parameter. It receives a list of frames: the messages consumed since the previous batch. Fold them with `concat`, which also returns `None` when the batch carried no messages. - The function must not declare a `ctx` parameter. The table names come from `output_tables`, not from the topic. ### Function Config Parameters #### `topic` The topic to consume. A name starting with `^` is a regular expression, and the stream consumes every topic that matches it, all into the same slot. The `@td.kafka.topic` column of each row tells which topic the message came from. #### `data_format` How the message values are encoded: - `"json"`: a JSON object per message. - `"avro"`: Avro. - `Protobuf(message_name=".")`: Protobuf. `message_name` is the full name of the message type, including the `.proto` package: `com.example.Order`, not `Order`. ```python from tabsdatak.conn.kafka import KafkaStreamSrc, Protobuf KafkaStreamSrc( topic="orders", data_format=Protobuf(message_name="com.example.Order"), schema=ORDER_PROTO, ) ``` #### `schema` The schema of the messages, as a string. What it is used for depends on the format and on the connection's schema registry: | Format | No schema registry | Confluent | AWS Glue | | --- | --- | --- | --- | | `"json"` | A JSON Schema. Every message is validated against it. An empty string skips validation. | A JSON Schema the messages are validated against. An empty string validates against the registry's schema. | Used to find the schema in the registry. Messages are validated against the registry's schema. | | `"avro"` | The Avro schema the messages were written with. | The schema to read the messages as (Avro reader schema). An empty string reads them as written. | Used to find the schema in the registry. Messages are read with the registry's schema. | | `Protobuf` | The `.proto` definition. | The `.proto` definition. | The `.proto` definition. | - A `.proto` definition is compiled on its own: it cannot `import` other files, including the Google well-known types such as `google/protobuf/timestamp.proto`. - With AWS Glue, the stream looks the schema up in the registry each time it starts. If the registry does not have it, the stream registers it as a new version of `schema_name`, or creates `schema_name` if it does not exist. - The schema is not checked when the function is defined. A malformed schema fails the stream when it starts. - The schema does not set the column types of the output table. They come from the decoded values; see [Message rows](#message-rows). #### `dlq_topic` A dead letter topic. A message that cannot be decoded, does not match the schema, or does not fit the column types of the batch is written to this topic, and the stream carries on. Without a `dlq_topic`, such a message stops the stream. Tabsdata restarts the stream after a while, but it stops again on the same message, so set a `dlq_topic` for any topic that may carry malformed messages. Each dead letter is a JSON object with the original value (`original_value`), the error (`error`), and the message's `topic`, `partition`, `offset` and `timestamp`. A message that was decoded but did not fit the column types carries the decoded message in `record` instead of `original_value`. The message key is kept for decoding errors. Headers are not copied. #### `src_cfg` Sets additional source configuration. Only these keys are accepted: | Key | Default | Description | | --- | --- | --- | | `tabsdata.kafka.buffer.max_rows` | `10000` | Messages buffered in the worker before they are written out to a file. | | `tabsdata.kafka.buffer.max_bytes` | none | Bytes buffered before they are written out, measured on the decoded messages. | | `tabsdata.kafka.buffer.max_secs` | `60` | Longest time a message waits in the buffer. | | `tabsdata.kafka.poll.max_secs` | `2` | Longest time one read of the topic takes. | | `tabsdata.kafka.poll.idle_secs` | `0` | Pause after a read that found no messages. | | `tabsdata.kafka.consume.max_messages` | `10000` | Most messages fetched from the brokers at once. | | `kafka.connect_timeout_secs` | `16` | Seconds to wait for the brokers when the stream starts, and for the dead letter topic to acknowledge writes. | | `kafka.logging_level` | `40` | Logging verbosity of the connector, as a Python logging level number (`10` debug, `20` info, `30` warning, `40` error). | The buffer settings control when consumed messages are written to files inside the worker. When those files are handed to the function is controlled by the [batch parameters](https://docs.tabsdata.com/guide/functions/stream-publishers#batch-parameters) of `@stream_publisher`. ## Message rows Each message becomes one row: the fields of the decoded message value, as columns sorted by name, plus one metadata column: | Column | Type | Description | | --- | --- | --- | | `@td.kafka.topic` | `str` | The topic the message came from. | The message key, headers, timestamp, partition and offset are not part of the row. Column types come from the decoded values: - **JSON:** integers are `int64`, numbers are `int64` or `float64`, strings are strings (a date in a string stays a string), objects are structs and arrays are lists. - **Avro:** Avro types map to their Arrow counterparts, including logical types: `timestamp-millis` becomes a UTC timestamp and `date` a date. - **Protobuf:** messages are converted the way Protobuf converts them to JSON: `int64` and `uint64` fields become strings, `bytes` become base64 strings, enums become their names, and fields left at their default value are included. A message with a `null` value, such as a compaction tombstone, cannot be decoded and goes to the `dlq_topic`. The function can receive several frames in one batch. A new frame starts when the messages change shape: without a schema registry, when a message has a field the frame does not; with a registry, when the schema version changes. Combine them with `concat(frames, how="diagonal_relaxed")`, which fills the missing columns with nulls. Within a frame, the first messages fix the column types. A later message whose values do not fit them goes to the `dlq_topic`. ## Operational notes ### Where the stream starts There is no start position to configure. The stream resumes from the offsets committed to the consumer group; a group with none starts from the earliest message the topic retains. ### Delivery A Kafka stream is at least once. The stream commits the offsets of a batch once the batch is handed over. If the stream stops between handing over a batch and committing it, the messages of that batch are published again after the restart, so remove duplicates downstream if they matter. Disabling the function stops the stream. Messages read but not yet handed over are read again when it is enabled. ### One consumer group per collection Every stream publisher in a collection consumes with the `group_id` of the collection's connection, so Kafka treats them as members of one consumer group: - Two stream publishers that read the same topic split its partitions between them, and each publishes only part of the messages. - Stream publishers that read different topics each get all the partitions of their own topic, and their committed offsets do not mix. They still take part in each other's rebalances whenever one of them starts, stops or restarts. With the connector's default `partition.assignment.strategy` (`cooperative-sticky`), the others keep their partitions. With an eager strategy set in `conn_cfg` (`range` or `roundrobin`), every rebalance takes all partitions from every member, and each stream publisher hands over what it has read as an early batch (see [Rebalances](#rebalances)). - The same applies to collections whose connections use the same `group_id`. Give each stream publisher that reads a topic its own collection, with its own `group_id`. ### Replaying a topic `tdk fn update --reset-stage` does not rewind a Kafka stream, because its position is the consumer group's. To read a topic again, update the collection's connection with a new `group_id`, which makes every stream publisher in the collection start over, or reset the group's offsets for that topic with the Kafka tools while the function is disabled. ### Rebalances When the brokers move partitions away from the stream, everything read from them so far is handed over at once, even before `batch_min_secs` has passed. ### Retention Messages the brokers delete before the stream reads them are skipped. If the brokers expire the group's offsets while the stream is stopped, the stream starts again from the earliest message retained. ### Failures If the brokers cannot be reached when the stream starts, or a schema registry cannot be reached while decoding, the stream stops without committing, and Tabsdata restarts it after a while. Nothing is sent to the `dlq_topic` for these errors. The stream warns in its log when the time between two reads of the topic gets close to the consumer's `max.poll.interval.ms`. Raise it in `conn_cfg` if batches take long to hand over. ## Connection Kafka stream publishers use `KafkaStreamSrcConn` to define the brokers, the consumer group and the credentials used by the stream. conn-kafka.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.kafka:KafkaStreamSrcConn spec: servers: 'broker1.internal:9092,broker2.internal:9092' group_id: 'tabsdata_orders' ``` - required - required - optional - optional - optional ### Create the collection A stream publisher is registered in a source collection, and a source collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type kafka-topic-in --file conn-kafka.yaml ``` Edit the template file with the Kafka connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-kafka.yaml ``` ```bash tdk collection create --name orders_kafka --conn-file conn-kafka.yaml ``` Then register the stream publisher in the collection: ```bash tdk fn register --coll orders_kafka --path read_orders.py::read_orders ``` - The collection goes in the `sources` group, because the connection is a source connection. `--group` can be left out; if it is given, it must be `sources`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `KafkaStreamSrc` for a collection created with `KafkaStreamSrcConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name orders_kafka --conn-file conn-kafka.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. - Every stream publisher in the collection consumes with the connection's `group_id`. Two stream publishers of one collection that read the same topic would split its partitions between them, each publishing only part of the messages. Give each stream publisher that reads a topic its own collection, with its own `group_id`. See [One consumer group per collection](#one-consumer-group-per-collection). ### Connection Config Parameters #### `servers` The bootstrap servers, as a comma-separated list of `host:port`. Written as a plain value here, but accepts `secret:` too. #### `group_id` The Kafka consumer group the stream consumes with. The stream commits its progress to this group, and resumes from it after a restart. A `group_id` with no committed offsets starts from the earliest message the topic retains. Written as a plain value here, but accepts `secret:` too. The consumer group belongs to the connection, not to the function: every stream publisher registered in the collection consumes with this same `group_id`. See [One consumer group per collection](#one-consumer-group-per-collection). #### `credentials` A user and password for SASL authentication. With `credentials`, the connector uses `SCRAM-SHA-256` over `SASL_PLAINTEXT`, which does not encrypt the connection, and ignores any `security.protocol` or `sasl.mechanism` in `conn_cfg`. For TLS, another SASL mechanism, or mutual TLS, leave `credentials` out and set the client properties in `conn_cfg` instead. #### `schema_registry` The schema registry the messages are encoded against. Leave it out for messages encoded without a registry. It is a mapping with the fields of one of two registry kinds, recognized by their fields. A Confluent schema registry takes a required `url`, optional `credentials` (`UserPassword`), and optional `configs` with extra Confluent schema registry client properties: ```yaml schema_registry: url: 'https://registry.internal:8081' credentials: kind: userPasswordCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:UserPassword spec: user: 'secret:registry_user' password: 'secret:example-password' ``` An AWS Glue schema registry takes a required `region_name`, `credentials` (`AwsAccessSecretKey`), `registry_name` and `schema_name`, and optional `configs` with extra keyword arguments for the AWS Glue client, such as `endpoint_url`: ```yaml schema_registry: region_name: 'us-east-1' registry_name: 'orders-registry' schema_name: 'orders' credentials: kind: awsAccessSecretKeyCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:AwsAccessSecretKey spec: access_key_id: 'secret:AKIAEXAMPLE' secret_access_key: 'secret:example-secret-key' ``` #### `conn_cfg` Kafka client properties, as [librdkafka](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md) names them, for example `security.protocol`, `ssl.ca.location`, `sasl.mechanism` or `sasl.username`. They are passed as they are to the consumer, and to the producer that writes to the `dlq_topic`. ```yaml conn_cfg: security.protocol: 'SASL_SSL' sasl.mechanism: 'SCRAM-SHA-512' sasl.username: 'tabsdata' sasl.password: 'example-password' ssl.ca.location: '/etc/tabsdata/kafka-ca.pem' ``` - `bootstrap.servers`, `group.id`, `enable.auto.commit` (`false`) and `auto.offset.reset` (`earliest`) are always set by the connector, and cannot be changed here. - `partition.assignment.strategy` (`cooperative-sticky`) can be changed here. - Values in `conn_cfg` are not secrets: they are stored and shown as written, passwords included. Use `credentials` for a password when SCRAM-SHA-256 is enough. - File paths, such as `ssl.ca.location`, must exist where Tabsdata runs the stream. --- # Local File System > Reference for the Tabsdata Local File System connector. Source: https://docs.tabsdata.com/guide/connectors/local-file-system The Local File System connector lets Tabsdata read files from and write files to the server's own filesystem. The `LocalFileSrc` connector can be used by a publisher function to read data from local files into a Tabsdata table. The `LocalFileDest` connector can be used by a subscriber function to write data from a Tabsdata table into local files. ## Supported formats | Format | Publish | Subscribe | | --- | --- | --- | | `AUTO` | Yes | Yes | | `CSV` | Yes | Yes | | `JSON` | Yes | Yes | | `AVRO` | Yes | Yes | | `PARQUET` | Yes | Yes | | `LOG` | Yes | No | ## Publisher The `LocalFileSrc` connector can be used by a publisher function to read data from local files into a Tabsdata table. [See the full publisher walkthrough](https://docs.tabsdata.com/tutorials/publishers/local-file-system) The publisher is registered in a collection created with a `LocalFileSrcConn` connection. The connection, and how to create the collection, are described in [Publisher connection](#publisher-connection), at the end of this page. ### Prerequisites - The configured `base_path` exists on the Tabsdata server and is readable by the function worker. - Every source path stays under `base_path`; parent-directory segments are refused. - The file format and compression match `format` and `format_cfg`. ### Publisher Configure `LocalFileSrc` as the `source` of a publisher function to select the files that Tabsdata reads from disk. publish\_orders.py ```python from tabsdatak.api import publisher from tabsdatak.conn.localfile import LocalFileSrc, SrcFileFormat @publisher( source=LocalFileSrc( paths=[ "incoming/orders.parquet", ], ), output_tables=["orders"], ) def publish_orders(orders): return orders ``` - required - optional - optional - optional - optional ### Function Config Parameters #### `paths` Defines the files that the publisher reads from disk. Each path is relative to the `base_path` configured in `LocalFileSrcConn`. Source paths: - cannot begin or end with `/` - cannot contain empty path segments - can contain a `*` glob in the final path segment For example: ```python LocalFileSrc( paths=[ "orders/2026/*.parquet", "customers/customers.csv", ] ) ``` Each element in `paths` represents one source slot and maps positionally to an argument in the publisher function. When a path uses a glob to match multiple files, the matching files are provided through the corresponding publisher input. #### `format` Sets the format used to read the source files. Supported formats are `AUTO`, `CSV`, `JSON`, `AVRO`, `PARQUET`, and `LOG`. The default is `AUTO`, which determines the format from the file extension. #### `format_cfg` Sets format-specific reader options. Configuration is keyed by `SrcFileFormat`. #### `initial_last_modified` Sets the initial cutoff for incremental ingestion. On the first run, Tabsdata imports files modified at or after the specified timestamp. The cutoff then advances so subsequent runs only import newer files. #### `src_cfg` Sets additional source configuration. The following options are supported: | Option | Description | | --- | --- | | `tabsdata.file.chunk_size` | Controls the size of chunks used when reading files. | | `tabsdata.src_metadata.drop` | When `True`, excludes the `@td.file.path` metadata column. | By default, rows read from disk include a `@td.file.path` metadata column containing the source file's path. ## Subscriber The `LocalFileDest` connector can be used by a subscriber function to write data from a Tabsdata table into local files. [See the full subscriber walkthrough](https://docs.tabsdata.com/tutorials/subscribers/local-file-system) The subscriber is registered in a collection created with a `LocalFileDestConn` connection. The connection, and how to create the collection, are described in [Subscriber connection](#subscriber-connection), at the end of this page. ### Prerequisites - The configured `base_path` exists on the Tabsdata server and is writable by the function worker. - Every destination path stays under `base_path`; parent-directory segments and globs are refused. - Enough disk space is available for the files written by the run. ### Subscriber Configure `LocalFileDest` as the `destination` of a subscriber function to define where Tabsdata writes files on disk. write\_orders.py ```python from tabsdatak.api import subscriber from tabsdatak.conn.localfile import LocalFileDest, DestFileFormat @subscriber( destination=LocalFileDest( paths=[ "exports/orders.parquet", ], ), input_tables=["orders"], ) def write_orders(orders): return orders ``` - required - optional - optional - optional ### Function Config Parameters #### `paths` Defines where the subscriber writes files on disk. Each path is relative to the `base_path` configured in `LocalFileDestConn`. Destination paths: - cannot begin or end with `/` - cannot contain empty path segments - cannot contain globs Each element in `paths` represents one destination slot and maps positionally to a value returned by the subscriber function. Destination paths can contain variables that Tabsdata substitutes when the subscriber runs: | Variable | Value | | --- | --- | | `${EXEC_PLAN_ID}` | Current execution plan ID. | | `${TRX_ID}` | Current transaction ID. | | `${EXEC_PLAN_TS}` | Timestamp when the execution plan started. | For example: ```python LocalFileDest( paths=[ "exports/${EXEC_PLAN_ID}/orders.parquet", ] ) ``` Dynamic paths can be used to create a new file for each execution instead of overwriting an existing one. #### `format` Sets the format used to write the destination file. Supported formats are `AUTO`, `CSV`, `JSON`, `AVRO`, and `PARQUET`. The default is `AUTO`, which determines the format from the destination file extension. #### `format_cfg` Sets format-specific writer options. Configuration is keyed by `DestFileFormat`. #### `dest_cfg` Sets additional destination configuration. The `tabsdata.file.chunk_size` option controls the size of chunks used when writing files. ## File behavior Each entry in a publisher's `paths` is one source slot. A literal path reads one file; a final-segment `*` glob can read several. The slots map by position to the publisher function arguments and then to `output_tables`. `initial_last_modified` sets the cutoff for the first run. After a successful run, Tabsdata advances the cutoff so the same unchanged files are not published again. Each entry in a subscriber's `paths` is one destination slot and maps by position to a returned value. Use `${EXEC_PLAN_ID}`, `${TRX_ID}`, or `${EXEC_PLAN_TS}` in a path when every run needs a new file. A fixed path writes to the same location again. ## Operational notes - Without `trigger_by`, a publisher runs only when triggered manually, while a subscriber runs when any input table commits. Set `trigger_by` to change when the function runs. - Tabsdata validates the connection document when the collection is created, but it does not contact local filesystem then. A bad path, credential, or permission fails when the function runs. - Globs are allowed only for publisher source paths, and only in the final path segment. Subscriber destination paths cannot contain globs. - All configured paths are relative to the connection's `base_path`. ## Publisher connection Local File System publishers use `LocalFileSrcConn` to define the root filesystem path used by the publisher. Unlike the cloud storage connectors, there is no bucket and no credentials: the connector reads directly off the server's own disk. conn-local-in.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.localfile:LocalFileSrcConn spec: base_path: '/data/hr' ``` - required - optional ### Create the collection A publisher is registered in a source collection, and a source collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type local-file-in --file conn-local-in.yaml ``` Edit the template file with the Local File System connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-local-in.yaml ``` ```bash tdk collection create --name local_in --conn-file conn-local-in.yaml ``` Then register the publisher in the collection: ```bash tdk fn register --coll local_in --path publish_orders.py::publish_orders ``` - The collection goes in the `sources` group, because the connection is a source connection. `--group` can be left out; if it is given, it must be `sources`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `LocalFileSrc` for a collection created with `LocalFileSrcConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name local_in --conn-file conn-local-in.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. ### Connection Config Parameters #### `base_path` The absolute filesystem path that acts as the root for all paths the publisher reads. Unlike the cloud storage connectors, `base_path` is required here: there is no bucket to fall back on, so the server needs a starting point on disk. #### `conn_cfg` Optional connector configuration overrides. ## Subscriber connection Local File System subscribers use `LocalFileDestConn` to define the root filesystem path used by the subscriber. Unlike the cloud storage connectors, there is no bucket and no credentials: the connector writes directly onto the server's own disk. conn-local-out.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.localfile:LocalFileDestConn spec: base_path: '/data/exports' ``` - required - optional ### Create the collection A subscriber is registered in a destination collection, and a destination collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type local-file-out --file conn-local-out.yaml ``` Edit the template file with the Local File System connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-local-out.yaml ``` ```bash tdk collection create --name local_out --conn-file conn-local-out.yaml ``` Then register the subscriber in the collection: ```bash tdk fn register --coll local_out --path write_orders.py::write_orders ``` - The collection goes in the `destinations` group, because the connection is a destination connection. `--group` can be left out; if it is given, it must be `destinations`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `LocalFileDest` for a collection created with `LocalFileDestConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name local_out --conn-file conn-local-out.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. ### Connection Config Parameters #### `base_path` The absolute filesystem path that acts as the root for all paths the subscriber writes. Unlike the cloud storage connectors, `base_path` is required here: there is no bucket to fall back on, so the server needs a starting point on disk. #### `conn_cfg` Optional connector configuration overrides. --- # MariaDB > Reference for the Tabsdata MariaDB connector. Source: https://docs.tabsdata.com/guide/connectors/mariadb The MariaDB connector lets Tabsdata run queries against MariaDB and write tables into MariaDB. The `MySQLSrc` connector can be used by a publisher function to read data from MariaDB into a Tabsdata table. The `MySQLDest` connector can be used by a subscriber function to write data from a Tabsdata table into MariaDB. warning The MariaDB connector needs the `mysql-connector-python` driver, which Tabsdata does not redistribute. See [Installing Third-Party Database Drivers](https://docs.tabsdata.com/server-admin/third-party-database-drivers) to add it to your Tabsdata server. ## Publisher The `MySQLSrc` connector can be used by a publisher function to read data from MariaDB into a Tabsdata table. [See the full publisher walkthrough](https://docs.tabsdata.com/tutorials/publishers/mariadb) The publisher is registered in a collection created with a `MySQLSrcConn` connection. The connection, and how to create the collection, are described in [Publisher connection](#publisher-connection), at the end of this page. ### Prerequisites - The MariaDB server is reachable from the Tabsdata server. - The connection user can run every query in `queries` and read every referenced table or view. ### Publisher Configure `MySQLSrc` as the `source` of a publisher function to select the queries that Tabsdata runs against MariaDB. publish\_orders.py ```python from tabsdatak.api import publisher from tabsdatak.conn.mysql import MySQLSrc @publisher( source=MySQLSrc( queries=[ "SELECT * FROM orders", ], ), output_tables=["orders"], ) def publish_orders(orders): return orders ``` - required - optional - optional ### Function Config Parameters #### `queries` Defines the SQL queries the publisher runs against MariaDB. Each element in `queries` represents one source slot and maps positionally to an argument in the publisher function. For example: ```python MySQLSrc( queries=[ "SELECT * FROM country", "SELECT * FROM city", ], ) ``` #### `initial_values` The first-run values for named parameters in `queries`, written `:name`. On later runs each parameter takes the value the publisher stored with `ctx.set_attr(name, value)`, so a query can read only the rows added since the last run. Parameters bind values such as IDs and timestamps, not identifiers. Schema and table names have to be written into the query. For example: ```python from tabsdatak.api import TableFrameSpec, TrxCtx, publisher from tabsdatak.conn.mysql import MySQLSrc @publisher( source=MySQLSrc( queries=["SELECT * FROM orders WHERE id > :last_id ORDER BY id"], initial_values={"last_id": 0}, ), output_tables=["orders"], ) def publish_orders(orders: TableFrameSpec, ctx: TrxCtx) -> TableFrameSpec: row = orders.max_for("id") if row is not None and row["id"] is not None: ctx.set_attr("last_id", int(row["id"])) return orders ``` #### `src_cfg` Sets additional source configuration. ## Subscriber The `MySQLDest` connector can be used by a subscriber function to write data from a Tabsdata table into MariaDB. [See the full subscriber walkthrough](https://docs.tabsdata.com/tutorials/subscribers/mariadb) The subscriber is registered in a collection created with a `MySQLDestConn` connection. The connection, and how to create the collection, are described in [Subscriber connection](#subscriber-connection), at the end of this page. ### Prerequisites - The MariaDB server is reachable from the Tabsdata server. - The connection user can create missing destination tables and insert rows. With `if_table_exists="replace"`, it can also truncate existing tables. ### Subscriber Configure `MySQLDest` as the `destination` of a subscriber function to define which tables Tabsdata writes in MariaDB. write\_orders.py ```python from tabsdatak.api import subscriber from tabsdatak.conn.mysql import MySQLDest @subscriber( destination=MySQLDest( tables=[ "orders", ], ), input_tables=["orders"], ) def write_orders(orders): return orders ``` - required - optional - optional ### Function Config Parameters #### `tables` Defines the destination tables the subscriber writes to in MariaDB. Each element in `tables` represents one destination slot and maps positionally to a value returned by the subscriber function. For example: ```python MySQLDest( tables=[ "vendors", "items", ], ) ``` #### `if_table_exists` Controls what happens when the destination table already has rows. `"append"` (the default) keeps inserting into the existing rows. `"replace"` truncates the table before inserting. Neither mode changes the destination table's schema, so the returned frame's columns need to match the table MariaDB already has. #### `dest_cfg` Sets additional destination configuration. ## Query results Each entry in `queries` produces one source slot. The slots map by position to the publisher function arguments and then to `output_tables`. Keep those three lists in the same order. Queries run when the publisher is triggered. They are not a continuous change feed. To read incrementally, put named parameters such as `:last_id` in the query, set their first values with `initial_values`, and save the next values with `ctx.set_attr()` after each successful run. ## Write behavior Each entry in `tables` is one destination slot and maps by position to a value returned by the subscriber function. Returning `None` for a slot skips that write. `if_table_exists="append"` keeps the rows already in the destination and inserts the returned rows. `"replace"` truncates the table before inserting. Neither mode changes an existing table's schema, so column additions, removals, and type changes need a database migration before the subscriber runs. ## Operational notes - Without `trigger_by`, a publisher runs only when triggered manually, while a subscriber runs when any input table commits. Set `trigger_by` to change when the function runs. - Tabsdata checks the connection document when the collection is created, but it does not connect to MariaDB then. A bad host, credential, query, or permission fails when the function runs. - Every function in the collection uses the same connection. Use another collection when functions need another database or another set of credentials. ## Publisher connection MariaDB publishers use `MySQLSrcConn` to define the database connection and credentials used by the publisher. conn-mariadb-in.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.mysql:MySQLSrcConn spec: uri: 'mariadb://db.internal:3306/world' credentials: kind: userPasswordCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:UserPassword spec: user: 'secret:tabsdata_reader' password: 'secret:example-password' ``` - required - required - optional - optional ### Create the collection A publisher is registered in a source collection, and a source collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type mysql-sql-in --file conn-mariadb-in.yaml ``` Edit the template file with the MariaDB connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-mariadb-in.yaml ``` ```bash tdk collection create --name mariadb_in --conn-file conn-mariadb-in.yaml ``` Then register the publisher in the collection: ```bash tdk fn register --coll mariadb_in --path publish_orders.py::publish_orders ``` - The collection goes in the `sources` group, because the connection is a source connection. `--group` can be left out; if it is given, it must be `sources`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `MySQLSrc` for a collection created with `MySQLSrcConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name mariadb_in --conn-file conn-mariadb-in.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. MariaDB uses the MySQL connection types; the `mariadb://` URI scheme is accepted in `uri`. ### Connection Config Parameters #### `uri` The SQLAlchemy-style connection URI: `mariadb://host:port/database`. Written as a plain value here, but accepts `secret:` too. #### `credentials` The database user and password, spliced into the URI at connect time. `UserPassword` requires a `user` and `password`. #### `entity_whitelist` A YAML list of regular expressions, not a comma-separated string. It is checked to be a list of valid regular expressions, but it has no effect yet. Leave it out. #### `conn_cfg` Driver-side options forwarded to the connection as SQLAlchemy `connect_args`. ## Subscriber connection MariaDB subscribers use `MySQLDestConn` to define the database connection and credentials used by the subscriber. conn-mariadb-out.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.mysql:MySQLDestConn spec: uri: 'mariadb://db.internal:3306/warehouse' credentials: kind: userPasswordCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:UserPassword spec: user: 'secret:tabsdata_reader' password: 'secret:example-password' ``` - required - required - optional - optional ### Create the collection A subscriber is registered in a destination collection, and a destination collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type mysql-sql-out --file conn-mariadb-out.yaml ``` Edit the template file with the MariaDB connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-mariadb-out.yaml ``` ```bash tdk collection create --name mariadb_out --conn-file conn-mariadb-out.yaml ``` Then register the subscriber in the collection: ```bash tdk fn register --coll mariadb_out --path write_orders.py::write_orders ``` - The collection goes in the `destinations` group, because the connection is a destination connection. `--group` can be left out; if it is given, it must be `destinations`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `MySQLDest` for a collection created with `MySQLDestConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name mariadb_out --conn-file conn-mariadb-out.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. MariaDB uses the MySQL connection types; the `mariadb://` URI scheme is accepted in `uri`. ### Connection Config Parameters #### `uri` The SQLAlchemy-style connection URI: `mariadb://host:port/database`. Written as a plain value here, but accepts `secret:` too. #### `credentials` The database user and password, spliced into the URI at connect time. `UserPassword` requires a `user` and `password`. #### `entity_whitelist` A YAML list of regular expressions, not a comma-separated string. It is checked to be a list of valid regular expressions, but it has no effect yet. Leave it out. #### `conn_cfg` Driver-side options forwarded to the connection as SQLAlchemy `connect_args`. --- # MongoDB > Reference for the Tabsdata MongoDB destination connector and subscriber configuration. Source: https://docs.tabsdata.com/guide/connectors/mongodb The MongoDB connector lets Tabsdata subscribers write data from Tabsdata tables into MongoDB collections, one document per row. [See the full subscriber walkthrough](https://docs.tabsdata.com/tutorials/subscribers/mongodb) The subscriber is registered in a collection created with a `MongoDBDestConn` connection. The connection, and how to create the collection, are described in [Connection](#connection), at the end of this page. ### Prerequisites - The MongoDB deployment is reachable from the Tabsdata server. - The MongoDB user can create and write the target collections, and can drop them when replace mode is used. - `use_trxs=True` requires a replica set or sharded cluster that supports transactions. ### Subscriber Configure `MongoDBDest` as the `destination` of a subscriber function to define which MongoDB collections Tabsdata writes. write\_orders.py ```python from tabsdatak.api import subscriber, TableFrameSpec from tabsdatak.conn.mongodb import MongoDBDest @subscriber( destination=MongoDBDest( collections_with_ids=[("sales_db.orders", "order_id")], ), input_tables=["orders"], ) def write_orders(orders: TableFrameSpec) -> TableFrameSpec: return orders ``` - required - optional - optional - optional - optional - optional - optional - optional - optional ### Function Config Parameters #### `collections_with_ids` A list of `(collection, id_field)` pairs. `collection` is the target collection, written `.` or as a bare `` in the database named in the connection URI. `id_field` is the field used as each document's `_id`; `None` lets MongoDB generate the ids. Each pair in `collections_with_ids` represents one destination slot and maps positionally to a value returned by the subscriber function. #### `if_collection_exists` `"append"` (the default) leaves the existing documents in place. `"replace"` drops the collection before writing. #### `update_existing` When `True` (the default), a document whose id already exists is updated (an upsert). When `False`, documents are inserted and an existing id fails the batch. Ignored when `id_field` is `None`, since generated ids never collide. #### `use_trxs` and `docs_per_trx` `docs_per_trx` sets the documents per write batch, `1000` by default. With `use_trxs=True`, each batch is written in a MongoDB transaction, which requires a replica set. #### `maintain_order` When `True`, documents are inserted in input order. Slower, but the server-side order is deterministic. #### `fail_on_duplicate_key` When `True` (the default), a duplicate-key error fails the write. When `False`, it is logged and the batch continues. #### `log_intermediate_files` When `True`, each intermediate JSONL file is logged before upload, for debugging large writes. #### `dest_cfg` Sets additional destination configuration. The only accepted key is `mongodb.logging_level`. `pymongo.*` options go in the connection's `conn_cfg`. ## Write behavior Each `(collection, id_field)` entry in `collections_with_ids` is one destination slot. A named `id_field` becomes the document `_id`; `None` lets MongoDB generate it. `if_collection_exists="append"` keeps existing documents. `"replace"` drops the collection first. With `update_existing=True`, matching ids are upserted; with it disabled, the connector inserts and an id collision fails the batch. ## Operational notes - `docs_per_trx` is both the write-batch size and, when transactions are enabled, the number of documents per transaction. - `maintain_order=True` keeps input order but gives up some write throughput. - `fail_on_duplicate_key=False` logs duplicate-key errors and continues the batch. Use it only when skipping those documents is acceptable. - Tabsdata does not connect to MongoDB when the collection is created. URI, credential, topology, and permission errors appear when the subscriber runs. ## Connection MongoDB subscribers use `MongoDBDestConn` to define the server and the credentials used by the subscriber. conn-mongodb.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.mongodb:MongoDBDestConn spec: uri: 'mongodb://mongo.internal:27017/sales_db' credentials: kind: userPasswordCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:UserPassword spec: user: 'secret:tabsdata_writer' password: 'secret:example-password' ``` - required - required - optional ### Create the collection A subscriber is registered in a destination collection, and a destination collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type mongodb-out --file conn-mongodb.yaml ``` Edit the template file with the MongoDB connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-mongodb.yaml ``` ```bash tdk collection create --name mongodb_out --conn-file conn-mongodb.yaml ``` Then register the subscriber in the collection: ```bash tdk fn register --coll mongodb_out --path write_orders.py::write_orders ``` - The collection goes in the `destinations` group, because the connection is a destination connection. `--group` can be left out; if it is given, it must be `destinations`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `MongoDBDest` for a collection created with `MongoDBDestConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name mongodb_out --conn-file conn-mongodb.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. ### Connection Config Parameters #### `uri` The MongoDB connection URI without credentials, starting with `mongodb://` or `mongodb+srv://`, for example `mongodb://host:27017/sales_db`. The database in the URI is the one bare collection names are written to. Accepts `secret:` too. #### `credentials` A `UserPassword` with the MongoDB user and password, spliced into the URI when the subscriber writes. #### `conn_cfg` Options for the MongoDB client. `pymongo.*` keys are forwarded to `pymongo.MongoClient`. --- # MSSQL > Reference for the Tabsdata MSSQL connector. Source: https://docs.tabsdata.com/guide/connectors/mssql The MSSQL connector lets Tabsdata run queries against MSSQL and write tables into MSSQL. The `MSSQLSrc` connector can be used by a publisher function to read data from MSSQL into a Tabsdata table. The `MSSQLDest` connector can be used by a subscriber function to write data from a Tabsdata table into MSSQL. ## Publisher The `MSSQLSrc` connector can be used by a publisher function to read data from MSSQL into a Tabsdata table. [See the full publisher walkthrough](https://docs.tabsdata.com/tutorials/publishers/mssql) The publisher is registered in a collection created with a `MSSQLSrcConn` connection. The connection, and how to create the collection, are described in [Publisher connection](#publisher-connection), at the end of this page. ### Prerequisites - The MSSQL server is reachable from the Tabsdata server. - The connection user can run every query in `queries` and read every referenced table or view. ### Publisher Configure `MSSQLSrc` as the `source` of a publisher function to select the queries that Tabsdata runs against MSSQL. publish\_orders.py ```python from tabsdatak.api import publisher from tabsdatak.conn.mssql import MSSQLSrc @publisher( source=MSSQLSrc( queries=[ "SELECT * FROM orders", ], ), output_tables=["orders"], ) def publish_orders(orders): return orders ``` - required - optional - optional ### Function Config Parameters #### `queries` Defines the SQL queries the publisher runs against MSSQL. Each element in `queries` represents one source slot and maps positionally to an argument in the publisher function. For example: ```python MSSQLSrc( queries=[ "SELECT * FROM country", "SELECT * FROM city", ], ) ``` #### `initial_values` The first-run values for named parameters in `queries`, written `:name`. On later runs each parameter takes the value the publisher stored with `ctx.set_attr(name, value)`, so a query can read only the rows added since the last run. Parameters bind values such as IDs and timestamps, not identifiers. Schema and table names have to be written into the query. For example: ```python from tabsdatak.api import TableFrameSpec, TrxCtx, publisher from tabsdatak.conn.mssql import MSSQLSrc @publisher( source=MSSQLSrc( queries=["SELECT * FROM orders WHERE id > :last_id ORDER BY id"], initial_values={"last_id": 0}, ), output_tables=["orders"], ) def publish_orders(orders: TableFrameSpec, ctx: TrxCtx) -> TableFrameSpec: row = orders.max_for("id") if row is not None and row["id"] is not None: ctx.set_attr("last_id", int(row["id"])) return orders ``` #### `src_cfg` Sets additional source configuration. ## Subscriber The `MSSQLDest` connector can be used by a subscriber function to write data from a Tabsdata table into MSSQL. [See the full subscriber walkthrough](https://docs.tabsdata.com/tutorials/subscribers/mssql) The subscriber is registered in a collection created with a `MSSQLDestConn` connection. The connection, and how to create the collection, are described in [Subscriber connection](#subscriber-connection), at the end of this page. ### Prerequisites - The MSSQL server is reachable from the Tabsdata server. - The connection user can create missing destination tables and insert rows. With `if_table_exists="replace"`, it can also truncate existing tables. ### Subscriber Configure `MSSQLDest` as the `destination` of a subscriber function to define which tables Tabsdata writes in MSSQL. write\_orders.py ```python from tabsdatak.api import subscriber from tabsdatak.conn.mssql import MSSQLDest @subscriber( destination=MSSQLDest( tables=[ "orders", ], ), input_tables=["orders"], ) def write_orders(orders): return orders ``` - required - optional - optional ### Function Config Parameters #### `tables` Defines the destination tables the subscriber writes to in MSSQL. Each element in `tables` represents one destination slot and maps positionally to a value returned by the subscriber function. For example: ```python MSSQLDest( tables=[ "vendors", "items", ], ) ``` #### `if_table_exists` Controls what happens when the destination table already has rows. `"append"` (the default) keeps inserting into the existing rows. `"replace"` truncates the table before inserting. Neither mode changes the destination table's schema, so the returned frame's columns need to match the table MSSQL already has. #### `dest_cfg` Sets additional destination configuration. ## Query results Each entry in `queries` produces one source slot. The slots map by position to the publisher function arguments and then to `output_tables`. Keep those three lists in the same order. Queries run when the publisher is triggered. They are not a continuous change feed. To read incrementally, put named parameters such as `:last_id` in the query, set their first values with `initial_values`, and save the next values with `ctx.set_attr()` after each successful run. ## Write behavior Each entry in `tables` is one destination slot and maps by position to a value returned by the subscriber function. Returning `None` for a slot skips that write. `if_table_exists="append"` keeps the rows already in the destination and inserts the returned rows. `"replace"` truncates the table before inserting. Neither mode changes an existing table's schema, so column additions, removals, and type changes need a database migration before the subscriber runs. ## Operational notes - Without `trigger_by`, a publisher runs only when triggered manually, while a subscriber runs when any input table commits. Set `trigger_by` to change when the function runs. - Tabsdata checks the connection document when the collection is created, but it does not connect to MSSQL then. A bad host, credential, query, or permission fails when the function runs. - Every function in the collection uses the same connection. Use another collection when functions need another database or another set of credentials. ## Publisher connection MSSQL publishers use `MSSQLSrcConn` to define the database connection and credentials used by the publisher. conn-mssql-in.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.mssql:MSSQLSrcConn spec: uri: 'mssql+pyodbc://db.internal:1433/world' credentials: kind: userPasswordCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:UserPassword spec: user: 'secret:tabsdata_reader' password: 'secret:example-password' ``` - required - required - optional - optional ### Create the collection A publisher is registered in a source collection, and a source collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type mssql-sql-in --file conn-mssql-in.yaml ``` Edit the template file with the MSSQL connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-mssql-in.yaml ``` ```bash tdk collection create --name mssql_in --conn-file conn-mssql-in.yaml ``` Then register the publisher in the collection: ```bash tdk fn register --coll mssql_in --path publish_orders.py::publish_orders ``` - The collection goes in the `sources` group, because the connection is a source connection. `--group` can be left out; if it is given, it must be `sources`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `MSSQLSrc` for a collection created with `MSSQLSrcConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name mssql_in --conn-file conn-mssql-in.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. ### Connection Config Parameters #### `uri` The SQLAlchemy-style connection URI: `mssql+pyodbc://host:port/database`. Written as a plain value here, but accepts `secret:` too. #### `credentials` The database user and password, spliced into the URI at connect time. `UserPassword` requires a `user` and `password`. #### `entity_whitelist` A YAML list of regular expressions, not a comma-separated string. It is checked to be a list of valid regular expressions, but it has no effect yet. Leave it out. #### `conn_cfg` Driver-side options forwarded to the connection as SQLAlchemy `connect_args`. ## Subscriber connection MSSQL subscribers use `MSSQLDestConn` to define the database connection and credentials used by the subscriber. conn-mssql-out.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.mssql:MSSQLDestConn spec: uri: 'mssql+pyodbc://db.internal:1433/warehouse' credentials: kind: userPasswordCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:UserPassword spec: user: 'secret:tabsdata_reader' password: 'secret:example-password' ``` - required - required - optional - optional ### Create the collection A subscriber is registered in a destination collection, and a destination collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type mssql-sql-out --file conn-mssql-out.yaml ``` Edit the template file with the MSSQL connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-mssql-out.yaml ``` ```bash tdk collection create --name mssql_out --conn-file conn-mssql-out.yaml ``` Then register the subscriber in the collection: ```bash tdk fn register --coll mssql_out --path write_orders.py::write_orders ``` - The collection goes in the `destinations` group, because the connection is a destination connection. `--group` can be left out; if it is given, it must be `destinations`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `MSSQLDest` for a collection created with `MSSQLDestConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name mssql_out --conn-file conn-mssql-out.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. ### Connection Config Parameters #### `uri` The SQLAlchemy-style connection URI: `mssql+pyodbc://host:port/database`. Written as a plain value here, but accepts `secret:` too. #### `credentials` The database user and password, spliced into the URI at connect time. `UserPassword` requires a `user` and `password`. #### `entity_whitelist` A YAML list of regular expressions, not a comma-separated string. It is checked to be a list of valid regular expressions, but it has no effect yet. Leave it out. #### `conn_cfg` Driver-side options forwarded to the connection as SQLAlchemy `connect_args`. --- # MySQL > Reference for the Tabsdata MySQL connector. Source: https://docs.tabsdata.com/guide/connectors/mysql The MySQL connector lets Tabsdata run queries against MySQL and write tables into MySQL. The `MySQLSrc` connector can be used by a publisher function to read data from MySQL into a Tabsdata table. The `MySQLDest` connector can be used by a subscriber function to write data from a Tabsdata table into MySQL. warning The MySQL connector needs the `mysql-connector-python` driver, which Tabsdata does not redistribute. See [Installing Third-Party Database Drivers](https://docs.tabsdata.com/server-admin/third-party-database-drivers) to add it to your Tabsdata server. ## Publisher The `MySQLSrc` connector can be used by a publisher function to read data from MySQL into a Tabsdata table. [See the full publisher walkthrough](https://docs.tabsdata.com/tutorials/publishers/mysql) The publisher is registered in a collection created with a `MySQLSrcConn` connection. The connection, and how to create the collection, are described in [Publisher connection](#publisher-connection), at the end of this page. ### Prerequisites - The MySQL server is reachable from the Tabsdata server. - The connection user can run every query in `queries` and read every referenced table or view. ### Publisher Configure `MySQLSrc` as the `source` of a publisher function to select the queries that Tabsdata runs against MySQL. publish\_orders.py ```python from tabsdatak.api import publisher from tabsdatak.conn.mysql import MySQLSrc @publisher( source=MySQLSrc( queries=[ "SELECT * FROM orders", ], ), output_tables=["orders"], ) def publish_orders(orders): return orders ``` - required - optional - optional ### Function Config Parameters #### `queries` Defines the SQL queries the publisher runs against MySQL. Each element in `queries` represents one source slot and maps positionally to an argument in the publisher function. For example: ```python MySQLSrc( queries=[ "SELECT * FROM country", "SELECT * FROM city", ], ) ``` #### `initial_values` The first-run values for named parameters in `queries`, written `:name`. On later runs each parameter takes the value the publisher stored with `ctx.set_attr(name, value)`, so a query can read only the rows added since the last run. Parameters bind values such as IDs and timestamps, not identifiers. Schema and table names have to be written into the query. For example: ```python from tabsdatak.api import TableFrameSpec, TrxCtx, publisher from tabsdatak.conn.mysql import MySQLSrc @publisher( source=MySQLSrc( queries=["SELECT * FROM orders WHERE id > :last_id ORDER BY id"], initial_values={"last_id": 0}, ), output_tables=["orders"], ) def publish_orders(orders: TableFrameSpec, ctx: TrxCtx) -> TableFrameSpec: row = orders.max_for("id") if row is not None and row["id"] is not None: ctx.set_attr("last_id", int(row["id"])) return orders ``` #### `src_cfg` Sets additional source configuration. ## Subscriber The `MySQLDest` connector can be used by a subscriber function to write data from a Tabsdata table into MySQL. [See the full subscriber walkthrough](https://docs.tabsdata.com/tutorials/subscribers/mysql) The subscriber is registered in a collection created with a `MySQLDestConn` connection. The connection, and how to create the collection, are described in [Subscriber connection](#subscriber-connection), at the end of this page. ### Prerequisites - The MySQL server is reachable from the Tabsdata server. - The connection user can create missing destination tables and insert rows. With `if_table_exists="replace"`, it can also truncate existing tables. ### Subscriber Configure `MySQLDest` as the `destination` of a subscriber function to define which tables Tabsdata writes in MySQL. write\_orders.py ```python from tabsdatak.api import subscriber from tabsdatak.conn.mysql import MySQLDest @subscriber( destination=MySQLDest( tables=[ "orders", ], ), input_tables=["orders"], ) def write_orders(orders): return orders ``` - required - optional - optional ### Function Config Parameters #### `tables` Defines the destination tables the subscriber writes to in MySQL. Each element in `tables` represents one destination slot and maps positionally to a value returned by the subscriber function. For example: ```python MySQLDest( tables=[ "vendors", "items", ], ) ``` #### `if_table_exists` Controls what happens when the destination table already has rows. `"append"` (the default) keeps inserting into the existing rows. `"replace"` truncates the table before inserting. Neither mode changes the destination table's schema, so the returned frame's columns need to match the table MySQL already has. #### `dest_cfg` Sets additional destination configuration. ## Query results Each entry in `queries` produces one source slot. The slots map by position to the publisher function arguments and then to `output_tables`. Keep those three lists in the same order. Queries run when the publisher is triggered. They are not a continuous change feed. To read incrementally, put named parameters such as `:last_id` in the query, set their first values with `initial_values`, and save the next values with `ctx.set_attr()` after each successful run. ## Write behavior Each entry in `tables` is one destination slot and maps by position to a value returned by the subscriber function. Returning `None` for a slot skips that write. `if_table_exists="append"` keeps the rows already in the destination and inserts the returned rows. `"replace"` truncates the table before inserting. Neither mode changes an existing table's schema, so column additions, removals, and type changes need a database migration before the subscriber runs. ## Operational notes - Without `trigger_by`, a publisher runs only when triggered manually, while a subscriber runs when any input table commits. Set `trigger_by` to change when the function runs. - Tabsdata checks the connection document when the collection is created, but it does not connect to MySQL then. A bad host, credential, query, or permission fails when the function runs. - Every function in the collection uses the same connection. Use another collection when functions need another database or another set of credentials. ## Publisher connection MySQL publishers use `MySQLSrcConn` to define the database connection and credentials used by the publisher. conn-mysql-in.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.mysql:MySQLSrcConn spec: uri: 'mysql://db.internal:3306/world' credentials: kind: userPasswordCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:UserPassword spec: user: 'secret:tabsdata_reader' password: 'secret:example-password' ``` - required - required - optional - optional ### Create the collection A publisher is registered in a source collection, and a source collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type mysql-sql-in --file conn-mysql-in.yaml ``` Edit the template file with the MySQL connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-mysql-in.yaml ``` ```bash tdk collection create --name mysql_in --conn-file conn-mysql-in.yaml ``` Then register the publisher in the collection: ```bash tdk fn register --coll mysql_in --path publish_orders.py::publish_orders ``` - The collection goes in the `sources` group, because the connection is a source connection. `--group` can be left out; if it is given, it must be `sources`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `MySQLSrc` for a collection created with `MySQLSrcConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name mysql_in --conn-file conn-mysql-in.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. ### Connection Config Parameters #### `uri` The SQLAlchemy-style connection URI: `mysql://host:port/database`. Written as a plain value here, but accepts `secret:` too. #### `credentials` The database user and password, spliced into the URI at connect time. `UserPassword` requires a `user` and `password`. #### `entity_whitelist` A YAML list of regular expressions, not a comma-separated string. It is checked to be a list of valid regular expressions, but it has no effect yet. Leave it out. #### `conn_cfg` Driver-side options forwarded to the connection as SQLAlchemy `connect_args`. ## Subscriber connection MySQL subscribers use `MySQLDestConn` to define the database connection and credentials used by the subscriber. conn-mysql-out.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.mysql:MySQLDestConn spec: uri: 'mysql://db.internal:3306/warehouse' credentials: kind: userPasswordCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:UserPassword spec: user: 'secret:tabsdata_reader' password: 'secret:example-password' ``` - required - required - optional - optional ### Create the collection A subscriber is registered in a destination collection, and a destination collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type mysql-sql-out --file conn-mysql-out.yaml ``` Edit the template file with the MySQL connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-mysql-out.yaml ``` ```bash tdk collection create --name mysql_out --conn-file conn-mysql-out.yaml ``` Then register the subscriber in the collection: ```bash tdk fn register --coll mysql_out --path write_orders.py::write_orders ``` - The collection goes in the `destinations` group, because the connection is a destination connection. `--group` can be left out; if it is given, it must be `destinations`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `MySQLDest` for a collection created with `MySQLDestConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name mysql_out --conn-file conn-mysql-out.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. ### Connection Config Parameters #### `uri` The SQLAlchemy-style connection URI: `mysql://host:port/database`. Written as a plain value here, but accepts `secret:` too. #### `credentials` The database user and password, spliced into the URI at connect time. `UserPassword` requires a `user` and `password`. #### `entity_whitelist` A YAML list of regular expressions, not a comma-separated string. It is checked to be a list of valid regular expressions, but it has no effect yet. Leave it out. #### `conn_cfg` Driver-side options forwarded to the connection as SQLAlchemy `connect_args`. --- # MySQL CDC > Reference for the Tabsdata MySQL CDC stream source: prerequisites, the full load, change rows, and the connection. Source: https://docs.tabsdata.com/guide/connectors/mysql-cdc The MySQL CDC connector lets Tabsdata capture the changes made to MySQL tables (inserts, updates, deletes and truncates) as they are committed, and publish them into Tabsdata tables. With `start_from="full"`, the stream starts with a full load: it takes a consistent snapshot of the tracked tables and then switches to change capture from the exact point of that snapshot, with no gaps and no duplicates. The full load and the change capture are one stream publisher, so there is nothing to reconcile between them. The `MySQLCdcStreamSrc` connector can be used by a [stream publisher](https://docs.tabsdata.com/guide/functions/stream-publishers) function to follow the MySQL binary log (binlog) as a replica would and write every captured change into a Tabsdata table. Unlike `MySQLSrc`, which runs queries on a trigger, a CDC source runs continuously: registering the stream publisher starts a long-lived worker that keeps reading the binlog until the function is disabled or deleted. ## Publisher The `MySQLCdcStreamSrc` connector can be used by a stream publisher function to capture changes from MySQL into Tabsdata tables. [See the full publisher walkthrough](https://docs.tabsdata.com/tutorials/publishers/mysql-cdc) The stream publisher is registered in a collection created with a `MySQLCdcStreamSrcConn` connection. The connection, and how to create the collection, are described in [Connection](#connection), at the end of this page. ### Prerequisites - The server's binlog is set up for row-based capture: `log_bin = ON`, `binlog_format = ROW`, `binlog_row_image = FULL` and `gtid_mode = ON`. The stream checks these at start and refuses a server that does not have them. - The connecting user holds the replication privileges a binlog reader needs (`REPLICATION SLAVE` and `REPLICATION CLIENT`). - When streaming from a replica, the replica runs with `log_replica_updates = ON`, and with `replica_preserve_commit_order = ON` if it applies transactions in parallel. - For `start_from="full"`, the connecting user also holds `SELECT` on every tracked table, and `RELOAD` where the server grants it, for the global read lock. The tracked tables are InnoDB. Managed services such as Amazon RDS and Aurora refuse the global read lock, and the full load still runs there with the same transactional consistency. ### Publisher Configure `MySQLCdcStreamSrc` as the `source` of a `@stream_publisher` function to select the tables whose changes Tabsdata captures. stream\_sales\_changes.py ```python from tabsdatak.api import TableFrameSpec, TableFramesSpec, stream_publisher from tabsdatak.conn.mysql import MySQLCdcStreamSrc from tabsdatak.tableframe.functions import concat @stream_publisher( source=MySQLCdcStreamSrc( tables=[ "sales.orders", "sales.customers", ], start_from="full", ), batch_max_secs=120, output_tables=["orders_changes", "customers_changes"], ) def stream_sales_changes( orders: TableFramesSpec, customers: TableFramesSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return concat(orders), concat(customers) ``` - required - required - optional - optional - optional `batch_max_secs=120` hands the captured changes to the function at least every two minutes. The `batch_*` parameters decide when a batch of changes is handed to the function, and are the same for every stream source. See [Stream Publishers](https://docs.tabsdata.com/guide/functions/stream-publishers#batch-parameters) for all of them. A stream publisher differs from a regular publisher in three ways: - It has no trigger. Registering it starts the stream worker, which runs until the function is disabled or deleted. - Each element of `tables` is one function parameter, and each parameter receives a list of frames: the changes captured for that table since the previous batch. Fold them with `concat`, which also returns `None` when the batch carried nothing for that table. - The function must not declare a `ctx` parameter. ### Function Config Parameters #### `tables` Defines the MySQL tables whose changes are captured, as qualified `database.table` names. Each element in `tables` represents one source slot and maps positionally to an argument in the publisher function and to an entry in `output_tables`. For example: ```python MySQLCdcStreamSrc( tables=[ "sales.orders", "sales.customers", ], start_from="tail", ) ``` #### `start_from` Where the stream starts the first time it runs. Later runs resume from the position the stream has recorded. - `"full"`: a full load of every tracked table, then the changes committed after it. See [Full load](#full-load). - `"tail"`: only changes committed after the stream starts. - `"head"`: replay from the oldest change still in binlog retention. - `GtidPosition(gtid=":")`: after a global transaction id. The given GTIDs count as already read, and the stream starts with the next transaction. A range such as `":1-"` is accepted too. - `BinlogPosition(file="mysql-bin.000001", pos=4)`: from a binlog file and byte offset, as `SHOW BINARY LOG STATUS` or `SHOW BINLOG EVENTS` report them. Position `4` is the start of a file. - `TimestampPosition(ts=...)`: from the first binlog event at or after a point in time. A naive datetime is read as UTC. A time before the oldest retained binlog is refused. #### Full load With `start_from="full"`, the stream first emits every row of each tracked table and then captures the changes from the exact binlog position of that snapshot. The snapshot is consistent across all the tracked tables, and no change is missed or repeated between the snapshot and the change capture. - Each table's snapshot is preceded by one `LOAD` row (`@td.cdc.meta.op = "l"`), and its rows arrive as inserts. A `LOAD` row tells the consumer to empty its copy of the table before applying the rows that follow. - The tables are read in one consistent-snapshot transaction. To pin that snapshot to a binlog position, the stream briefly takes the global read lock. The lock is held only while the position is read, not while the tables are scanned. On servers that refuse the lock, such as Amazon RDS and Aurora, the snapshot is just as consistent. To resynchronize a destination later, keep `start_from="full"` and reset the function's stage: ```bash tdk fn update --reset-stage ``` The stream then starts again with a new full load and its `LOAD` rows. Resetting the stage with any other `start_from` restarts at that position and sends no `LOAD` row. #### `columns_format` How each change row lays out the table's column values: `"columns"` (the default), `"struct"` or `"map"`. See [Change rows](#change-rows). #### `flatten_columns` When `True` (the default), the new column values are written as ordinary top-level columns, so each change row reads like a row of the source table. `columns_format` then only shapes the old values. See [Change rows](#change-rows). #### `src_cfg` Sets additional source configuration. Only these keys are accepted: | Key | Default | Description | | --- | --- | --- | | `tabsdata.cdc.buffer.max_rows` | `10000` | Rows buffered in the worker before they are written out to a file. | | `tabsdata.cdc.buffer.max_bytes` | none | Bytes buffered before they are written out. Measuring bytes costs time on every change; prefer bounding by rows. | | `tabsdata.cdc.buffer.max_secs` | `60` | Longest time a change waits in the buffer. | | `tabsdata.cdc.poll.max_secs` | `1` | Longest time one read of the binlog waits for changes. | | `tabsdata.cdc.poll.idle_secs` | `1` | Pause after a read that found no changes. | | `tabsdata.cdc.max_large_text_length` | `1024` | Longest value, in characters, accepted from an unbounded text column. A longer value fails the stream. | | `tabsdata.cdc.fetch_size` | `100000` | Rows fetched per round trip while reading the full load. | | `tabsdata.cdc.chunk_size` | `100000` | Rows per chunk while writing the full load. | | `mysql.cdc.logging_level` | `40` | Logging verbosity of the connector, as a Python logging level number (`10` debug, `20` info, `30` warning, `40` error). | The buffer settings control when captured changes are written to files inside the worker. When those files are handed to the function is controlled by the [batch parameters](https://docs.tabsdata.com/guide/functions/stream-publishers#batch-parameters) of `@stream_publisher`. ## Change rows Every row the stream emits is one change. Besides the column values, each row carries change metadata: | Column | Type | Description | | --- | --- | --- | | `@td.cdc.meta.op` | `str` | The operation: `"i"` insert, `"u"` update, `"d"` delete, `"t"` truncate, `"l"` load (start of a full load). | | `@td.cdc.meta.tx` | `str` | The transaction's GTID. Full load rows carry `initial-load`. | | `@td.cdc.meta.sq` | `int` | The change's sequence within its transaction. | | `@td.cdc.meta.ts` | `datetime` (UTC) | When the transaction committed. Full load rows carry the time the snapshot was taken. | | `@td.cdc.meta.seg` | `decimal` | The transaction's number in its GTID. Full load rows carry `-1`. | | `@td.cdc.meta.ord` | `decimal` | The position of the change within the transaction. Ordering by `seg` and then `ord` gives the order the changes were committed in. | | `@td.cdc.meta.fmt` | `str` | The `columns_format` the row was produced with. | | `@td.cdc.meta.flat` | `bool` | The `flatten_columns` setting the row was produced with. | | `@td.cdc.meta.sch` | `str` | A hash of the source table's schema when the change was captured. It changes when the table's columns change. | The column values are laid out according to `flatten_columns` and `columns_format`: | | New values | Old values | | --- | --- | --- | | `flatten_columns=True` (default) | One top-level column per source column, with the source column's name. | Depends on `columns_format`. | | `columns_format="columns"` (default) | `@td.cdc.data.col.new.` per column, when not flattened. | `@td.cdc.data.col.old.` per column. | | `columns_format="struct"` | `@td.cdc.data.row.new`, one struct, when not flattened. | `@td.cdc.data.row.old`, one struct. | | `columns_format="map"` | `@td.cdc.data.map.new`, a JSON object as a string, when not flattened. | `@td.cdc.data.map.old`, a JSON object as a string. | What each operation carries: | Operation | New values | Old values | | --- | --- | --- | | Insert (`i`) | The inserted row. | Empty. | | Update (`u`) | The row after the update. | The complete row before the update. | | Delete (`d`) | The deleted row. | Empty. | | Truncate (`t`), Load (`l`) | Empty. | Empty. | Empty values are null columns with `"columns"`, a struct whose fields are all null with `"struct"`, and a JSON object whose values are all null with `"map"`. A `LOAD` row tells the consumer to empty its copy of the table before applying the rows that follow. A `TRUNCATE` row reports that the source table was truncated. `BLOB` columns are not captured: they always arrive as null. Use `"columns"` with `flatten_columns=True` for changes that land in a table and get queried. Use `"struct"` or `"map"` when the source table's schema may change while the stream runs, to carry the values opaquely instead of changing the output table's columns. ## Operational notes - MySQL has no replication slot, so a stopped stream holds nothing on the server. It resumes from its recorded GTID, provided the server still retains the binlogs from that point. If they were purged, the stream fails and needs a new full load: reset the stage with `start_from="full"`. - A stream follows one source server. Switching it to another server (for example after a failover) is not supported: the new server's changes would be dropped as already published. Start a new stream instead. ## Connection MySQL CDC stream publishers use `MySQLCdcStreamSrcConn` to define the database connection and credentials used by the stream. conn-mysql-cdc.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.mysql:MySQLCdcStreamSrcConn spec: uri: 'mysql://db.internal:3306/sales' credentials: kind: userPasswordCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:UserPassword spec: user: 'secret:tabsdata_cdc' password: 'secret:example-password' ``` - required - required - optional - optional - optional ### Create the collection A stream publisher is registered in a source collection, and a source collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type mysql-cdc --file conn-mysql-cdc.yaml ``` Edit the template file with the MySQL connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-mysql-cdc.yaml ``` ```bash tdk collection create --name sales_cdc --conn-file conn-mysql-cdc.yaml ``` Then register the stream publisher in the collection: ```bash tdk fn register --coll sales_cdc --path stream_sales_changes.py::stream_sales_changes ``` - The collection goes in the `sources` group, because the connection is a source connection. `--group` can be left out; if it is given, it must be `sources`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `MySQLCdcStreamSrc` for a collection created with `MySQLCdcStreamSrcConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name sales_cdc --conn-file conn-mysql-cdc.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. - A batch `MySQLSrc` publisher cannot share this collection, because it uses a different connection (`mysql-sql-in`). ### Connection Config Parameters #### `uri` The SQLAlchemy-style connection URI: `mysql://host:port/database`. Written as a plain value here, but accepts `secret:` too. #### `credentials` The database user and password, spliced into the URI at connect time. `UserPassword` requires a `user` and `password`. #### `server_id` The replica id the stream registers with when it reads the binlog. Defaults to `512`. MySQL admits one replica per `server_id`: a second stream connecting with the same id disconnects the first. Give every stream publisher reading the same MySQL server its own connection with a distinct `server_id`, and keep them clear of the ids your real replicas use. #### `entity_whitelist` A YAML list of regular expressions, not a comma-separated string. It is checked to be a list of valid regular expressions, but it has no effect yet: it does not limit which tables a stream can track. Leave it out. #### `conn_cfg` pymysql connection keywords such as `ssl_ca`, `ssl_cert`, `ssl_key`, `ssl_disabled`, `charset` or `connect_timeout`. They apply to every connection the stream opens: catalog reads, the binlog reader, the full load and the server checks. --- # Oracle > Reference for the Tabsdata Oracle connector. Source: https://docs.tabsdata.com/guide/connectors/oracle The Oracle connector lets Tabsdata run queries against Oracle and write tables into Oracle. The `OracleSrc` connector can be used by a publisher function to read data from Oracle into a Tabsdata table. The `OracleDest` connector can be used by a subscriber function to write data from a Tabsdata table into Oracle. warning The Oracle connector needs the `oracledb` driver, which Tabsdata does not redistribute. See [Installing Third-Party Database Drivers](https://docs.tabsdata.com/server-admin/third-party-database-drivers) to add it to your Tabsdata server. ## Publisher The `OracleSrc` connector can be used by a publisher function to read data from Oracle into a Tabsdata table. [See the full publisher walkthrough](https://docs.tabsdata.com/tutorials/publishers/oracle) The publisher is registered in a collection created with a `OracleSrcConn` connection. The connection, and how to create the collection, are described in [Publisher connection](#publisher-connection), at the end of this page. ### Prerequisites - The Oracle server is reachable from the Tabsdata server. - The connection user can run every query in `queries` and read every referenced table or view. ### Publisher Configure `OracleSrc` as the `source` of a publisher function to select the queries that Tabsdata runs against Oracle. publish\_orders.py ```python from tabsdatak.api import publisher from tabsdatak.conn.oracle import OracleSrc @publisher( source=OracleSrc( queries=[ "SELECT * FROM orders", ], ), output_tables=["orders"], ) def publish_orders(orders): return orders ``` - required - optional - optional ### Function Config Parameters #### `queries` Defines the SQL queries the publisher runs against Oracle. Each element in `queries` represents one source slot and maps positionally to an argument in the publisher function. For example: ```python OracleSrc( queries=[ "SELECT * FROM country", "SELECT * FROM city", ], ) ``` #### `initial_values` The first-run values for named parameters in `queries`, written `:name`. On later runs each parameter takes the value the publisher stored with `ctx.set_attr(name, value)`, so a query can read only the rows added since the last run. Parameters bind values such as IDs and timestamps, not identifiers. Schema and table names have to be written into the query. For example: ```python from tabsdatak.api import TableFrameSpec, TrxCtx, publisher from tabsdatak.conn.oracle import OracleSrc @publisher( source=OracleSrc( queries=["SELECT * FROM orders WHERE id > :last_id ORDER BY id"], initial_values={"last_id": 0}, ), output_tables=["orders"], ) def publish_orders(orders: TableFrameSpec, ctx: TrxCtx) -> TableFrameSpec: row = orders.max_for("id") if row is not None and row["id"] is not None: ctx.set_attr("last_id", int(row["id"])) return orders ``` #### `src_cfg` Sets additional source configuration. ## Subscriber The `OracleDest` connector can be used by a subscriber function to write data from a Tabsdata table into Oracle. [See the full subscriber walkthrough](https://docs.tabsdata.com/tutorials/subscribers/oracle) The subscriber is registered in a collection created with a `OracleDestConn` connection. The connection, and how to create the collection, are described in [Subscriber connection](#subscriber-connection), at the end of this page. ### Prerequisites - The Oracle server is reachable from the Tabsdata server. - The connection user can create missing destination tables and insert rows. With `if_table_exists="replace"`, it can also truncate existing tables. ### Subscriber Configure `OracleDest` as the `destination` of a subscriber function to define which tables Tabsdata writes in Oracle. write\_orders.py ```python from tabsdatak.api import subscriber from tabsdatak.conn.oracle import OracleDest @subscriber( destination=OracleDest( tables=[ "orders", ], ), input_tables=["orders"], ) def write_orders(orders): return orders ``` - required - optional - optional ### Function Config Parameters #### `tables` Defines the destination tables the subscriber writes to in Oracle. Each element in `tables` represents one destination slot and maps positionally to a value returned by the subscriber function. For example: ```python OracleDest( tables=[ "vendors", "items", ], ) ``` #### `if_table_exists` Controls what happens when the destination table already has rows. `"append"` (the default) keeps inserting into the existing rows. `"replace"` truncates the table before inserting. Neither mode changes the destination table's schema, so the returned frame's columns need to match the table Oracle already has. #### `dest_cfg` Sets additional destination configuration. ## Query results Each entry in `queries` produces one source slot. The slots map by position to the publisher function arguments and then to `output_tables`. Keep those three lists in the same order. Queries run when the publisher is triggered. They are not a continuous change feed. To read incrementally, put named parameters such as `:last_id` in the query, set their first values with `initial_values`, and save the next values with `ctx.set_attr()` after each successful run. ## Write behavior Each entry in `tables` is one destination slot and maps by position to a value returned by the subscriber function. Returning `None` for a slot skips that write. `if_table_exists="append"` keeps the rows already in the destination and inserts the returned rows. `"replace"` truncates the table before inserting. Neither mode changes an existing table's schema, so column additions, removals, and type changes need a database migration before the subscriber runs. ## Operational notes - Without `trigger_by`, a publisher runs only when triggered manually, while a subscriber runs when any input table commits. Set `trigger_by` to change when the function runs. - Tabsdata checks the connection document when the collection is created, but it does not connect to Oracle then. A bad host, credential, query, or permission fails when the function runs. - Every function in the collection uses the same connection. Use another collection when functions need another database or another set of credentials. ## Publisher connection Oracle publishers use `OracleSrcConn` to define the database connection and credentials used by the publisher. conn-oracle-in.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.oracle:OracleSrcConn spec: uri: 'oracle://db.internal:1521/world' credentials: kind: userPasswordCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:UserPassword spec: user: 'secret:tabsdata_reader' password: 'secret:example-password' ``` - required - required - optional - optional ### Create the collection A publisher is registered in a source collection, and a source collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type oracle-sql-in --file conn-oracle-in.yaml ``` Edit the template file with the Oracle connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-oracle-in.yaml ``` ```bash tdk collection create --name oracle_in --conn-file conn-oracle-in.yaml ``` Then register the publisher in the collection: ```bash tdk fn register --coll oracle_in --path publish_orders.py::publish_orders ``` - The collection goes in the `sources` group, because the connection is a source connection. `--group` can be left out; if it is given, it must be `sources`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `OracleSrc` for a collection created with `OracleSrcConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name oracle_in --conn-file conn-oracle-in.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. ### Connection Config Parameters #### `uri` The SQLAlchemy-style connection URI: `oracle://host:port/database`. Written as a plain value here, but accepts `secret:` too. #### `credentials` The database user and password, spliced into the URI at connect time. `UserPassword` requires a `user` and `password`. #### `entity_whitelist` A YAML list of regular expressions, not a comma-separated string. It is checked to be a list of valid regular expressions, but it has no effect yet. Leave it out. #### `conn_cfg` Driver-side options forwarded to the connection as SQLAlchemy `connect_args`. ## Subscriber connection Oracle subscribers use `OracleDestConn` to define the database connection and credentials used by the subscriber. conn-oracle-out.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.oracle:OracleDestConn spec: uri: 'oracle://db.internal:1521/warehouse' credentials: kind: userPasswordCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:UserPassword spec: user: 'secret:tabsdata_reader' password: 'secret:example-password' ``` - required - required - optional - optional ### Create the collection A subscriber is registered in a destination collection, and a destination collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type oracle-sql-out --file conn-oracle-out.yaml ``` Edit the template file with the Oracle connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-oracle-out.yaml ``` ```bash tdk collection create --name oracle_out --conn-file conn-oracle-out.yaml ``` Then register the subscriber in the collection: ```bash tdk fn register --coll oracle_out --path write_orders.py::write_orders ``` - The collection goes in the `destinations` group, because the connection is a destination connection. `--group` can be left out; if it is given, it must be `destinations`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `OracleDest` for a collection created with `OracleDestConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name oracle_out --conn-file conn-oracle-out.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. ### Connection Config Parameters #### `uri` The SQLAlchemy-style connection URI: `oracle://host:port/database`. Written as a plain value here, but accepts `secret:` too. #### `credentials` The database user and password, spliced into the URI at connect time. `UserPassword` requires a `user` and `password`. #### `entity_whitelist` A YAML list of regular expressions, not a comma-separated string. It is checked to be a list of valid regular expressions, but it has no effect yet. Leave it out. #### `conn_cfg` Driver-side options forwarded to the connection as SQLAlchemy `connect_args`. --- # Postgres > Reference for the Tabsdata Postgres connector. Source: https://docs.tabsdata.com/guide/connectors/postgres The Postgres connector lets Tabsdata run queries against Postgres and write tables into Postgres. The `PostgresSrc` connector can be used by a publisher function to read data from Postgres into a Tabsdata table. The `PostgresDest` connector can be used by a subscriber function to write data from a Tabsdata table into Postgres. warning The PostgreSQL connector needs the `psycopg2-binary` driver, which Tabsdata does not redistribute. See [Installing Third-Party Database Drivers](https://docs.tabsdata.com/server-admin/third-party-database-drivers) to add it to your Tabsdata server. ## Publisher The `PostgresSrc` connector can be used by a publisher function to read data from Postgres into a Tabsdata table. [See the full publisher walkthrough](https://docs.tabsdata.com/tutorials/publishers/postgres) The publisher is registered in a collection created with a `PostgresSrcConn` connection. The connection, and how to create the collection, are described in [Publisher connection](#publisher-connection), at the end of this page. ### Prerequisites - The Postgres server is reachable from the Tabsdata server. - The connection user can run every query in `queries` and read every referenced table or view. ### Publisher Configure `PostgresSrc` as the `source` of a publisher function to select the queries that Tabsdata runs against Postgres. publish\_orders.py ```python from tabsdatak.api import publisher from tabsdatak.conn.postgres import PostgresSrc @publisher( source=PostgresSrc( queries=[ "SELECT * FROM orders", ], ), output_tables=["orders"], ) def publish_orders(orders): return orders ``` - required - optional - optional ### Function Config Parameters #### `queries` Defines the SQL queries the publisher runs against Postgres. Each element in `queries` represents one source slot and maps positionally to an argument in the publisher function. For example: ```python PostgresSrc( queries=[ "SELECT * FROM country", "SELECT * FROM city", ], ) ``` #### `initial_values` The first-run values for named parameters in `queries`, written `:name`. On later runs each parameter takes the value the publisher stored with `ctx.set_attr(name, value)`, so a query can read only the rows added since the last run. Parameters bind values such as IDs and timestamps, not identifiers. Schema and table names have to be written into the query. For example: ```python from tabsdatak.api import TableFrameSpec, TrxCtx, publisher from tabsdatak.conn.postgres import PostgresSrc @publisher( source=PostgresSrc( queries=["SELECT * FROM orders WHERE id > :last_id ORDER BY id"], initial_values={"last_id": 0}, ), output_tables=["orders"], ) def publish_orders(orders: TableFrameSpec, ctx: TrxCtx) -> TableFrameSpec: row = orders.max_for("id") if row is not None and row["id"] is not None: ctx.set_attr("last_id", int(row["id"])) return orders ``` #### `src_cfg` Sets additional source configuration. ## Subscriber The `PostgresDest` connector can be used by a subscriber function to write data from a Tabsdata table into Postgres. [See the full subscriber walkthrough](https://docs.tabsdata.com/tutorials/subscribers/postgres) The subscriber is registered in a collection created with a `PostgresDestConn` connection. The connection, and how to create the collection, are described in [Subscriber connection](#subscriber-connection), at the end of this page. ### Prerequisites - The Postgres server is reachable from the Tabsdata server. - The connection user can create missing destination tables and insert rows. With `if_table_exists="replace"`, it can also truncate existing tables. ### Subscriber Configure `PostgresDest` as the `destination` of a subscriber function to define which tables Tabsdata writes in Postgres. write\_orders.py ```python from tabsdatak.api import subscriber from tabsdatak.conn.postgres import PostgresDest @subscriber( destination=PostgresDest( tables=[ "orders", ], ), input_tables=["orders"], ) def write_orders(orders): return orders ``` - required - optional - optional ### Function Config Parameters #### `tables` Defines the destination tables the subscriber writes to in Postgres. Each element in `tables` represents one destination slot and maps positionally to a value returned by the subscriber function. For example: ```python PostgresDest( tables=[ "vendors", "items", ], ) ``` #### `if_table_exists` Controls what happens when the destination table already has rows. `"append"` (the default) keeps inserting into the existing rows. `"replace"` truncates the table before inserting. Neither mode changes the destination table's schema, so the returned frame's columns need to match the table Postgres already has. #### `dest_cfg` Sets additional destination configuration. ## Query results Each entry in `queries` produces one source slot. The slots map by position to the publisher function arguments and then to `output_tables`. Keep those three lists in the same order. Queries run when the publisher is triggered. They are not a continuous change feed. To read incrementally, put named parameters such as `:last_id` in the query, set their first values with `initial_values`, and save the next values with `ctx.set_attr()` after each successful run. ## Write behavior Each entry in `tables` is one destination slot and maps by position to a value returned by the subscriber function. Returning `None` for a slot skips that write. `if_table_exists="append"` keeps the rows already in the destination and inserts the returned rows. `"replace"` truncates the table before inserting. Neither mode changes an existing table's schema, so column additions, removals, and type changes need a database migration before the subscriber runs. ## Operational notes - Without `trigger_by`, a publisher runs only when triggered manually, while a subscriber runs when any input table commits. Set `trigger_by` to change when the function runs. - Tabsdata checks the connection document when the collection is created, but it does not connect to Postgres then. A bad host, credential, query, or permission fails when the function runs. - Every function in the collection uses the same connection. Use another collection when functions need another database or another set of credentials. ## Publisher connection Postgres publishers use `PostgresSrcConn` to define the database connection and credentials used by the publisher. conn-postgres-in.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.postgres:PostgresSrcConn spec: uri: 'postgres://db.internal:5432/world' credentials: kind: userPasswordCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:UserPassword spec: user: 'secret:tabsdata_reader' password: 'secret:example-password' ``` - required - required - optional - optional ### Create the collection A publisher is registered in a source collection, and a source collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type postgres-sql-in --file conn-postgres-in.yaml ``` Edit the template file with the Postgres connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-postgres-in.yaml ``` ```bash tdk collection create --name postgres_in --conn-file conn-postgres-in.yaml ``` Then register the publisher in the collection: ```bash tdk fn register --coll postgres_in --path publish_orders.py::publish_orders ``` - The collection goes in the `sources` group, because the connection is a source connection. `--group` can be left out; if it is given, it must be `sources`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `PostgresSrc` for a collection created with `PostgresSrcConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name postgres_in --conn-file conn-postgres-in.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. ### Connection Config Parameters #### `uri` The SQLAlchemy-style connection URI: `postgres://host:port/database`. Written as a plain value here, but accepts `secret:` too. #### `credentials` The database user and password, spliced into the URI at connect time. `UserPassword` requires a `user` and `password`. #### `entity_whitelist` A YAML list of regular expressions, not a comma-separated string. It is checked to be a list of valid regular expressions, but it has no effect yet. Leave it out. #### `conn_cfg` Driver-side options forwarded to the connection as SQLAlchemy `connect_args`. ## Subscriber connection Postgres subscribers use `PostgresDestConn` to define the database connection and credentials used by the subscriber. conn-postgres-out.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.postgres:PostgresDestConn spec: uri: 'postgres://db.internal:5432/warehouse' credentials: kind: userPasswordCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:UserPassword spec: user: 'secret:tabsdata_reader' password: 'secret:example-password' ``` - required - required - optional - optional ### Create the collection A subscriber is registered in a destination collection, and a destination collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type postgres-sql-out --file conn-postgres-out.yaml ``` Edit the template file with the Postgres connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-postgres-out.yaml ``` ```bash tdk collection create --name postgres_out --conn-file conn-postgres-out.yaml ``` Then register the subscriber in the collection: ```bash tdk fn register --coll postgres_out --path write_orders.py::write_orders ``` - The collection goes in the `destinations` group, because the connection is a destination connection. `--group` can be left out; if it is given, it must be `destinations`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `PostgresDest` for a collection created with `PostgresDestConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name postgres_out --conn-file conn-postgres-out.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. ### Connection Config Parameters #### `uri` The SQLAlchemy-style connection URI: `postgres://host:port/database`. Written as a plain value here, but accepts `secret:` too. #### `credentials` The database user and password, spliced into the URI at connect time. `UserPassword` requires a `user` and `password`. #### `entity_whitelist` A YAML list of regular expressions, not a comma-separated string. It is checked to be a list of valid regular expressions, but it has no effect yet. Leave it out. #### `conn_cfg` Driver-side options forwarded to the connection as SQLAlchemy `connect_args`. --- # Postgres CDC > Reference for the Tabsdata Postgres CDC stream source: prerequisites, the full load, change rows, and the connection. Source: https://docs.tabsdata.com/guide/connectors/postgres-cdc The Postgres CDC connector lets Tabsdata capture the changes made to Postgres tables (inserts, updates, deletes and truncates) as they are committed, and publish them into Tabsdata tables. With `start_from="full"`, the stream starts with a full load: it takes a consistent snapshot of the tracked tables and then switches to change capture from the exact point of that snapshot, with no gaps and no duplicates. The full load and the change capture are one stream publisher, so there is nothing to reconcile between them. The `PostgresCdcStreamSrc` connector can be used by a [stream publisher](https://docs.tabsdata.com/guide/functions/stream-publishers) function to follow the Postgres write-ahead log (WAL) through a logical replication slot and write every captured change into a Tabsdata table. Unlike `PostgresSrc`, which runs queries on a trigger, a CDC source runs continuously: registering the stream publisher starts a long-lived worker that keeps reading the WAL until the function is disabled or deleted. ## Publisher The `PostgresCdcStreamSrc` connector can be used by a stream publisher function to capture changes from Postgres into Tabsdata tables. [See the full publisher walkthrough](https://docs.tabsdata.com/tutorials/publishers/postgres-cdc) The stream publisher is registered in a collection created with a `PostgresCdcStreamSrcConn` connection. The connection, and how to create the collection, are described in [Connection](#connection), at the end of this page. ### Prerequisites - The server runs with `wal_level = logical`. - The [wal2json](https://github.com/eulerto/wal2json) logical decoding output plugin, version 2.0 or later, is installed on the server. - The connecting role has the `REPLICATION` attribute. - Every tracked table has a replica identity: a primary key that is not `DEFERRABLE`, or `REPLICA IDENTITY FULL`, or `REPLICA IDENTITY USING INDEX` on a unique index. A table without one is refused, because Postgres would drop its updates and deletes from the WAL. - A tracked table with large text columns needs `REPLICA IDENTITY FULL`, or the `postgres.cdc.unchanged_large_text_as_null` setting described under [`src_cfg`](#src_cfg). - For `start_from="full"`, the connecting role can also `SELECT` every tracked table. ### Publisher Configure `PostgresCdcStreamSrc` as the `source` of a `@stream_publisher` function to select the tables whose changes Tabsdata captures. stream\_sales\_changes.py ```python from tabsdatak.api import TableFrameSpec, TableFramesSpec, stream_publisher from tabsdatak.conn.postgres import PostgresCdcStreamSrc from tabsdatak.tableframe.functions import concat @stream_publisher( source=PostgresCdcStreamSrc( tables=[ "public.orders", "public.customers", ], replication_slot="tabsdata_sales", start_from="full", ), batch_max_secs=120, output_tables=["orders_changes", "customers_changes"], ) def stream_sales_changes( orders: TableFramesSpec, customers: TableFramesSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return concat(orders), concat(customers) ``` - required - required - required - optional - optional - optional - optional - optional `batch_max_secs=120` hands the captured changes to the function at least every two minutes. The `batch_*` parameters decide when a batch of changes is handed to the function, and are the same for every stream source. See [Stream Publishers](https://docs.tabsdata.com/guide/functions/stream-publishers#batch-parameters) for all of them. A stream publisher differs from a regular publisher in three ways: - It has no trigger. Registering it starts the stream worker, which runs until the function is disabled or deleted. - Each element of `tables` is one function parameter, and each parameter receives a list of frames: the changes captured for that table since the previous batch. Fold them with `concat`, which also returns `None` when the batch carried nothing for that table. - The function must not declare a `ctx` parameter. ### Function Config Parameters #### `tables` Defines the Postgres tables whose changes are captured, as qualified `schema.table` names. Each element in `tables` represents one source slot and maps positionally to an argument in the publisher function and to an entry in `output_tables`. For example: ```python PostgresCdcStreamSrc( tables=[ "public.orders", "public.customers", ], replication_slot="tabsdata_sales", start_from="tail", ) ``` #### `replication_slot` The logical replication slot the stream reads through. Postgres keeps the WAL the slot has not yet consumed, which is what lets the stream resume where it left off after a restart. Each stream publisher needs its own slot. Tabsdata never drops the slot, not even when the function is deleted. A slot that is no longer read makes the server keep WAL indefinitely, so drop it with `pg_drop_replication_slot('')` once its stream is gone. #### `start_from` Where the stream starts the first time it runs. Later runs resume from the position the stream has recorded. - `"full"`: a full load of every tracked table, then the changes committed after it. See [Full load](#full-load). - `"tail"`: only changes committed after the stream starts. - `"head"`: replay from the oldest WAL the slot retains. - `LsnPosition(lsn=...)`: an explicit WAL position, given as the 64-bit integer that `pg_current_wal_lsn() - '0/0'::pg_lsn` returns. #### Full load With `start_from="full"`, the stream first emits every row of each tracked table and then captures the changes from the exact position of that snapshot. The snapshot is consistent across all the tracked tables, and no change is missed or repeated between the snapshot and the change capture. - Each table's snapshot is preceded by one `LOAD` row (`@td.cdc.meta.op = "l"`), and its rows arrive as inserts. A `LOAD` row tells the consumer to empty its copy of the table before applying the rows that follow. - No table is locked. The snapshot is the one Postgres exports when a replication slot is created: the stream's own slot when the stream creates it, or otherwise a temporary slot that is created for the load and dropped once the load is read. To resynchronize a destination later, keep `start_from="full"` and reset the function's stage: ```bash tdk fn update --reset-stage ``` The stream then starts again with a new full load and its `LOAD` rows. Resetting the stage with any other `start_from` restarts at that position and sends no `LOAD` row. #### `replication_slot_behavior` `"reuse"` (the default) requires the slot to exist already, for example when a DBA provisions it. `"create"` creates the slot, and does nothing if it already exists. #### `publication_name` A Postgres publication that scopes what the slot decodes (wal2json's `publication_names` option). #### `columns_format` How each change row lays out the table's column values: `"columns"` (the default), `"struct"` or `"map"`. See [Change rows](#change-rows). #### `flatten_columns` When `True` (the default), the new column values are written as ordinary top-level columns, so each change row reads like a row of the source table. `columns_format` then only shapes the old values. See [Change rows](#change-rows). #### `src_cfg` Sets additional source configuration. Only these keys are accepted: | Key | Default | Description | | --- | --- | --- | | `tabsdata.cdc.buffer.max_rows` | `10000` | Rows buffered in the worker before they are written out to a file. | | `tabsdata.cdc.buffer.max_bytes` | none | Bytes buffered before they are written out. Measuring bytes costs time on every change; prefer bounding by rows. | | `tabsdata.cdc.buffer.max_secs` | `60` | Longest time a change waits in the buffer. | | `tabsdata.cdc.poll.max_secs` | `1` | Longest time one read of the WAL waits for changes. | | `tabsdata.cdc.poll.idle_secs` | `1` | Pause after a read that found no changes. | | `tabsdata.cdc.max_large_text_length` | `1024` | Longest value, in characters, accepted from an unbounded text column. A longer value fails the stream. | | `tabsdata.cdc.fetch_size` | `100000` | Rows fetched per round trip while reading the full load. | | `tabsdata.cdc.chunk_size` | `100000` | Rows per chunk while writing the full load. | | `postgres.cdc.wal2json_format_version` | `2` | wal2json output format, `1` or `2`. Truncates are only captured with `2`. | | `postgres.cdc.unchanged_large_text_as_null` | unset | Set to `true` to accept that, on a table without `REPLICA IDENTITY FULL`, an update that does not touch a large text column delivers that column as null. When unset, such a table refuses to start the stream. | | `postgres.cdc.accept_gap` | unset | Lets a stopped stream resume past one reported gap between its position and the slot's. Give the value exactly as the error message prints it; it matches no other gap. | | `postgres.cdc.logging_level` | `40` | Logging verbosity of the connector, as a Python logging level number (`10` debug, `20` info, `30` warning, `40` error). | The buffer settings control when captured changes are written to files inside the worker. When those files are handed to the function is controlled by the [batch parameters](https://docs.tabsdata.com/guide/functions/stream-publishers#batch-parameters) of `@stream_publisher`. ## Change rows Every row the stream emits is one change. Besides the column values, each row carries change metadata: | Column | Type | Description | | --- | --- | --- | | `@td.cdc.meta.op` | `str` | The operation: `"i"` insert, `"u"` update, `"d"` delete, `"t"` truncate, `"l"` load (start of a full load). | | `@td.cdc.meta.tx` | `str` | The source transaction id. Full load rows carry `initial-load`. | | `@td.cdc.meta.sq` | `int` | The change's sequence within its transaction. | | `@td.cdc.meta.ts` | `datetime` (UTC) | When the transaction committed. Full load rows carry the time the snapshot was taken. | | `@td.cdc.meta.seg` | `decimal` | The position of the transaction's commit in the WAL. Full load rows carry `-1`. | | `@td.cdc.meta.ord` | `decimal` | The position of the change within the transaction. Ordering by `seg` and then `ord` gives the order the changes were committed in. | | `@td.cdc.meta.fmt` | `str` | The `columns_format` the row was produced with. | | `@td.cdc.meta.flat` | `bool` | The `flatten_columns` setting the row was produced with. | | `@td.cdc.meta.sch` | `str` | A hash of the source table's schema when the change was captured. It changes when the table's columns change. | The column values are laid out according to `flatten_columns` and `columns_format`: | | New values | Old values | | --- | --- | --- | | `flatten_columns=True` (default) | One top-level column per source column, with the source column's name. | Depends on `columns_format`. | | `columns_format="columns"` (default) | `@td.cdc.data.col.new.` per column, when not flattened. | `@td.cdc.data.col.old.` per column. | | `columns_format="struct"` | `@td.cdc.data.row.new`, one struct, when not flattened. | `@td.cdc.data.row.old`, one struct. | | `columns_format="map"` | `@td.cdc.data.map.new`, a JSON object as a string, when not flattened. | `@td.cdc.data.map.old`, a JSON object as a string. | What each operation carries: | Operation | New values | Old values | | --- | --- | --- | | Insert (`i`) | The inserted row. | Empty. | | Update (`u`) | The row after the update. | The row before the update. Postgres logs the complete previous row only for tables with `REPLICA IDENTITY FULL`; otherwise only the replica identity columns are present. | | Delete (`d`) | The deleted row. Without `REPLICA IDENTITY FULL`, only the replica identity columns are present. | Empty. | | Truncate (`t`), Load (`l`) | Empty. | Empty. | Empty values are null columns with `"columns"`, a struct whose fields are all null with `"struct"`, and a JSON object whose values are all null with `"map"`. A `LOAD` row tells the consumer to empty its copy of the table before applying the rows that follow. A `TRUNCATE` row reports that the source table was truncated. `bytea` and `oid` columns are not captured: they always arrive as null. Use `"columns"` with `flatten_columns=True` for changes that land in a table and get queried. Use `"struct"` or `"map"` when the source table's schema may change while the stream runs, to carry the values opaquely instead of changing the output table's columns. ## Operational notes - Stopping a stream does not lose changes: the slot keeps the WAL, and the stream resumes from its recorded position. The server also keeps that WAL until the stream consumes it, or until the slot is dropped. - If the slot is dropped or invalidated, the server is restored from a backup, or the stream's position no longer matches the server, the stream fails with an error explaining what happened. In most of those cases the fix is a new full load: reset the stage with `start_from="full"`. - A slot can only be read by one connection at a time. A stream whose slot is still held by another connection fails after a timeout. ## Connection Postgres CDC stream publishers use `PostgresCdcStreamSrcConn` to define the database connection and credentials used by the stream. conn-postgres-cdc.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.postgres:PostgresCdcStreamSrcConn spec: uri: 'postgres://db.internal:5432/sales' credentials: kind: userPasswordCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:UserPassword spec: user: 'secret:tabsdata_cdc' password: 'secret:example-password' ``` - required - required - optional - optional ### Create the collection A stream publisher is registered in a source collection, and a source collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type postgres-cdc --file conn-postgres-cdc.yaml ``` Edit the template file with the Postgres connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-postgres-cdc.yaml ``` ```bash tdk collection create --name sales_cdc --conn-file conn-postgres-cdc.yaml ``` Then register the stream publisher in the collection: ```bash tdk fn register --coll sales_cdc --path stream_sales_changes.py::stream_sales_changes ``` - The collection goes in the `sources` group, because the connection is a source connection. `--group` can be left out; if it is given, it must be `sources`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `PostgresCdcStreamSrc` for a collection created with `PostgresCdcStreamSrcConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name sales_cdc --conn-file conn-postgres-cdc.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. - A batch `PostgresSrc` publisher cannot share this collection, because it uses a different connection (`postgres-sql-in`). ### Connection Config Parameters #### `uri` The SQLAlchemy-style connection URI: `postgres://host:port/database` (or `postgresql://`). Written as a plain value here, but accepts `secret:` too. #### `credentials` The database user and password, spliced into the URI at connect time. `UserPassword` requires a `user` and `password`. The role must have the `REPLICATION` attribute. #### `entity_whitelist` A YAML list of regular expressions, not a comma-separated string. It is checked to be a list of valid regular expressions, but it has no effect yet: it does not limit which tables a stream can track. Leave it out. #### `conn_cfg` psycopg2 connection keywords such as `sslmode`, `sslrootcert`, `connect_timeout` or `application_name`. They apply to every connection the stream opens: catalog reads, the full load and the replication connection. The replication slot is configured on the publisher, not on the connection, so one connection can serve several stream publishers as long as each uses its own slot. --- # Salesforce > Reference for the Tabsdata Salesforce connectors: SOQL queries, reports, and the shared connection. Source: https://docs.tabsdata.com/guide/connectors/salesforce The Salesforce connectors let Tabsdata read data from Salesforce into Tabsdata tables. The `SalesforceSrc` connector can be used by a publisher function to run a SOQL query against Salesforce and write the result into a Tabsdata table. The `SalesforceReportSrc` connector can be used by a publisher function to read Salesforce reports into Tabsdata tables. Both connectors use the same connection class, `SalesforceSrcConn`, so one collection can hold both kinds of publisher. ## Publisher The `SalesforceSrc` connector can be used by a publisher function to read the result of a SOQL query into a Tabsdata table. [See the full publisher walkthrough](https://docs.tabsdata.com/tutorials/publishers/salesforce) The publisher is registered in a collection created with a `SalesforceSrcConn` connection. The connection, and how to create the collection, are described in [Connection](#connection), at the end of this page. ### Prerequisites - API access is enabled for the Salesforce user. - The username, password, and security token are valid for the instance. - The user can read every object and field referenced by the SOQL query. ### Publisher Configure `SalesforceSrc` as the `source` of a publisher function to select the SOQL query that Tabsdata runs against Salesforce. publish\_accounts.py ```python from tabsdatak.api import publisher, TableFrameSpec from tabsdatak.conn.salesforce import SalesforceSrc @publisher( source=SalesforceSrc( queries=["SELECT Id, Name FROM Account"], ), output_tables=["accounts"], ) def publish_accounts(accounts: TableFrameSpec) -> TableFrameSpec: return accounts ``` - required - optional - optional - optional - optional ### Function Config Parameters #### `queries` A list holding one SOQL query. The list currently takes exactly one entry, and its result maps to the one argument of the publisher function and the one entry in `output_tables`. #### `include_deleted` When `True`, the query also reads deleted rows, through the `queryAll` endpoint. Defaults to `False`. #### `initial_last_modified` A cutoff substituted for any `$lastModified` token in the query, for example `WHERE SystemModStamp > $lastModified`, so a run reads only the records changed after it. #### `api_version` Overrides the Salesforce API version, written `.`, for example `"60.0"`. #### `src_cfg` Sets additional source configuration. The accepted keys are `salesforce.logging_level` and `tabsdata.src_metadata.drop`. ## Report publisher The `SalesforceReportSrc` connector can be used by a publisher function to read Salesforce reports into Tabsdata tables. [See the full report walkthrough](https://docs.tabsdata.com/tutorials/publishers/salesforce-reports) The publisher is registered in a collection created with a `SalesforceSrcConn` connection. The connection, and how to create the collection, are described in [Connection](#connection), at the end of this page. ### Prerequisites - API access is enabled for the Salesforce user. - The user can open every report in `reports` and read the fields used by its filters. - Incremental reads need the last-modified column to be present in the report. ### Report publisher Configure `SalesforceReportSrc` as the `source` of a publisher function to select the reports that Tabsdata reads. publish\_customers.py ```python from tabsdatak.api import publisher, TableFrameSpec from tabsdatak.conn.salesforce import SalesforceReportSrc @publisher( source=SalesforceReportSrc( reports=[ "00O000000000001", ], ), output_tables=["customers"], ) def publish_customers(customers: TableFrameSpec) -> TableFrameSpec: return customers ``` - required - optional - optional - optional - optional - optional - optional - optional ### Function Config Parameters #### `reports` The reports to read, as report ids (15 or 18 alphanumeric characters) or report names, as a non-empty list. Each element in `reports` represents one source slot and maps positionally to an argument in the publisher function and to an entry in `output_tables`. #### `column_name_strategy` `"columnName"` (the default) names the columns after the report's API column names. `"label"` uses the display labels. #### `find_report_by` How an entry of `reports` is looked up. Unset, an entry that looks like an id is looked up as an id and anything else as a name. `"id"` or `"name"` forces one lookup, for an entry that could be either. #### `filter` A list of `(column, operator, value)` triples applied on the Salesforce server before the report is downloaded. `operator` is one of `equals`, `notEqual`, `lessThan`, `greaterThan`, `lessOrEqual`, `greaterOrEqual`, `contains`, `notContain`, `startsWith`, `includes` or `excludes`. #### `filter_logic` A boolean expression over the `filter` entries, numbered from 1, for example `"(1 AND 2) OR 3"`. #### `last_modified_column` and `initial_last_modified` An incremental read: only the rows whose `last_modified_column` is after `initial_last_modified` are read. Each of the two requires the other. #### `src_cfg` Sets additional source configuration. The accepted keys are `salesforce.logging_level` and `tabsdata.src_metadata.drop`. ## Result rows `SalesforceSrc.queries` currently takes one SOQL query, which produces the publisher's one source slot. The selected Salesforce fields become columns in that slot. For incremental SOQL reads, include `SystemModStamp` in the selected fields and compare it with `$lastModified`. The first run uses `initial_last_modified`; later runs use the highest `SystemModStamp` already seen. Set `include_deleted=True` when deleted records also need to be returned. Each entry in `SalesforceReportSrc.reports` produces one source slot. Report ids or names, publisher arguments, and `output_tables` map by position. Report filters are applied by Salesforce before the rows are downloaded. ## Operational notes - SOQL and report publishers use the same `SalesforceSrcConn`, so both can be registered in one source collection. - A report incremental read needs both `last_modified_column` and `initial_last_modified`. Without them, the full report is read on each run. - Returning `None` from the publisher means there is no new table version to publish. - Tabsdata does not contact Salesforce when the collection is created. Login, object, report, field, and API-version errors appear when the function runs. ## Connection Salesforce publishers use `SalesforceSrcConn` to define the Salesforce login used by both connectors. conn-salesforce.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.salesforce:SalesforceSrcConn spec: credentials: kind: salesforceTokenCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:SalesforceToken spec: username: 'secret:integration@acme.com' password: 'secret:example-password' security_token: 'secret:exampleSecurityToken00' ``` - required - optional - optional - optional - optional - optional ### Create the collection A publisher is registered in a source collection, and a source collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type salesforce-sosql-in --file conn-salesforce.yaml ``` Edit the template file with the Salesforce connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-salesforce.yaml ``` ```bash tdk collection create --name salesforce_in --conn-file conn-salesforce.yaml ``` Then register the publisher in the collection: ```bash tdk fn register --coll salesforce_in --path publish_accounts.py::publish_accounts ``` - The collection goes in the `sources` group, because the connection is a source connection. `--group` can be left out; if it is given, it must be `sources`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `SalesforceSrc` or `SalesforceReportSrc` for a collection created with `SalesforceSrcConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name salesforce_in --conn-file conn-salesforce.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. - The template type can be `salesforce-sosql-in` or `salesforce-report-in`. Both write the same `SalesforceSrcConn`, so a collection created from either holds both kinds of publisher. ### Connection Config Parameters #### `credentials` A `SalesforceToken` with the `username`, `password` and `security_token` of the Salesforce user. #### `instance_url` An explicit instance URL. It must be an `https://` URL on a Salesforce domain (`salesforce.com`, `force.com`, `cloudforce.com` or `salesforce-setup.com`). Accepts `secret:` too. #### `category` and `objects` Hints for entity discovery. `category` is a category name. `objects` is a non-empty list of Salesforce object API names, each 1 to 80 characters, starting with a letter, optionally ending in `__c` or `__r`. Only one of the two can be set. #### `entity_whitelist` A YAML list of regular expressions, not a comma-separated string. It is checked to be a list of valid regular expressions, but it has no effect yet. Leave it out. #### `conn_cfg` Optional connector configuration. --- # Snowflake > Reference for the Tabsdata Snowflake connector. Source: https://docs.tabsdata.com/guide/connectors/snowflake The Snowflake connector lets Tabsdata write tables into Snowflake. Snowflake is destination-only: there is no publisher-side connector for reading from Snowflake. The `SnowflakeDest` connector can be used by a subscriber function to write data from a Tabsdata table into Snowflake. [See the full subscriber walkthrough](https://docs.tabsdata.com/tutorials/subscribers/snowflake) The subscriber is registered in a collection created with a `SnowflakeDestConn` connection. The connection, and how to create the collection, are described in [Connection](#connection), at the end of this page. ### Prerequisites - The Snowflake account is reachable from the Tabsdata server. - The configured role can use the warehouse, database, and schema, and can create or write the destination tables. - If `stage` is set, the configured role can use that permanent stage. If it is left out, the connector creates a session-scoped temporary stage. ### Subscriber For merge writes, schema evolution, and retry watermarks, see [Merging into SQL destinations](https://docs.tabsdata.com/guide/functions/sql-merge). Configure `SnowflakeDest` as the `destination` of a subscriber function to define which tables Tabsdata writes in Snowflake. write\_orders.py ```python from tabsdatak.api import subscriber from tabsdatak.conn.snowflake import SnowflakeDest @subscriber( destination=SnowflakeDest( tables=[ "orders", ], ), input_tables=["orders"], ) def write_orders(orders): return orders ``` - required - optional - optional ### Function Config Parameters #### `tables` Defines the destination tables the subscriber writes to in Snowflake. Each element in `tables` represents one destination slot and maps positionally to a value returned by the subscriber function. Each name is a 1-, 2-, or 3-part name (`table`, `schema.table`, or `database.schema.table`); double-quote a part for a case-sensitive or special-character name. #### `if_table_exists` `"merge"` applies inserts, updates, and deletes by key and requires a `merge_cfg` list. See [Merge configuration](https://docs.tabsdata.com/guide/functions/sql-merge#merge-configuration) for a complete example. Controls what happens when the destination table already exists. `"append"` (the default) adds rows. `"replace"` issues `TRUNCATE` before `COPY INTO`, so the existing schema is preserved. #### `dest_cfg` Sets additional destination configuration. Supported keys include `snowflake.logging_level` and `snowflake.schema_evolution_engine`. Merge mode requires `snowflake.schema_evolution_engine="iceberg"`. ## Write behavior Each entry in `tables` is one destination slot and maps by position to a value returned by the subscriber function. Returning `None` for a slot skips that write. `if_table_exists="append"` adds rows. `"replace"` truncates before `COPY INTO`, so the existing table definition stays in place. `"merge"` applies inserts, updates, and deletes by key and needs `merge_cfg`. ## Operational notes - A one-part table name uses the connection's database and schema. Two- and three-part names override one or both defaults. - Replace mode preserves the existing schema. Change the table separately before returning columns that it does not accept. - Tabsdata does not connect to Snowflake when the collection is created. Account, role, warehouse, stage, and permission errors appear when the subscriber runs. ## Connection Snowflake subscribers use `SnowflakeDestConn` to define the account, credentials, and default namespace used by the subscriber. conn-snowflake.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.snowflake:SnowflakeDestConn spec: account: 'ABCDEFG-XY12345' credentials: kind: userPasswordCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:UserPassword spec: user: 'secret:tabsdata_svc' password: 'secret:CorrectHorseBatteryStaple' ``` - required - required - optional - optional - optional - optional - optional - optional ### Create the collection A subscriber is registered in a destination collection, and a destination collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type snowflake-bulk-out --file conn-snowflake.yaml ``` Edit the template file with the Snowflake connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-snowflake.yaml ``` ```bash tdk collection create --name snowflake_out --conn-file conn-snowflake.yaml ``` Then register the subscriber in the collection: ```bash tdk fn register --coll snowflake_out --path write_orders.py::write_orders ``` - The collection goes in the `destinations` group, because the connection is a destination connection. `--group` can be left out; if it is given, it must be `destinations`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `SnowflakeDest` for a collection created with `SnowflakeDestConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name snowflake_out --conn-file conn-snowflake.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. ### Connection Config Parameters #### `account` The Snowflake account identifier or locator, such as `ABCDEFG-XY12345`. #### `credentials` The Snowflake credentials used to authenticate. `UserPassword` requires a `user` and `password`; a personal access token can be passed as the password. #### `role` An optional role the connection runs `USE` for before writing. #### `database` The default database. Required when `schema` is set. #### `schema` The default schema. Requires `database` to be set. #### `warehouse` An optional warehouse the connection runs `USE WAREHOUSE` for. `USE WAREHOUSE` is only issued when this is set. #### `stage` An optional permanent stage used for `PUT` and `COPY INTO`. When unset, the plugin creates a session-scoped temporary stage instead. #### `conn_cfg` Optional connector-config overrides, forwarded to `snowflake-connector-python`. --- # StarRocks > Reference for the Tabsdata StarRocks destination connector, cloud staging, and subscriber configuration. Source: https://docs.tabsdata.com/guide/connectors/starrocks The StarRocks connector lets Tabsdata Subscribers write data from Tabsdata Tables into StarRocks. The connector stages the data as Parquet files in cloud storage, then loads it into StarRocks through its native `FILES()` table function. Destination tables in StarRocks must already exist as the connector does not create them automatically. [See the full subscriber walkthrough](https://docs.tabsdata.com/tutorials/subscribers/starrocks) The subscriber is registered in a collection created with a `StarRocksDestConn` connection. The connection, and how to create the collection, are described in [Connection](#connection), at the end of this page. ### Prerequisites - The StarRocks frontend is reachable from the Tabsdata server. - Every destination table already exists. The connector cannot infer its key model or distribution strategy. - Tabsdata can write to the staging location and StarRocks can read the staged Parquet files. - The StarRocks user can insert into the destination tables and truncate them when replace mode is used. ### Subscriber Configure `StarRocksDest` as the `destination` of a Subscriber Function. Each returned frame maps to one entry in `tables`, in order. write\_orders.py ```python from tabsdatak.api import subscriber from tabsdatak.conn.starrocks import StarRocksDest @subscriber( destination=StarRocksDest( tables=[ "orders", ], ), input_tables=["sales/orders"], ) def write_orders(orders): return orders ``` - required - optional - optional - optional ### Function Config Parameters #### `tables` Names of existing StarRocks tables, one per returned frame. Names can be `table` or `database.table`; an unqualified name uses the Connection's `database`. Backtick-quote individual name parts containing special characters. Create the destination tables before running the Subscriber. StarRocks table creation requires a physical layout, including a key definition and distribution strategy, that the connector cannot infer from a frame. #### `if_table_exists` `"append"` (the default) inserts rows into the existing table. `"replace"` truncates the existing table and then inserts the returned rows. Neither mode creates a missing table. #### `schema_evolution` `"strict"` (the default) rejects schema changes. `"iceberg"` applies Iceberg-style schema changes to the existing destination table; it does not change the destination into an Iceberg table. #### `dest_cfg` Optional destination configuration. The supported key is `starrocks.logging_level`. ## Write behavior Each entry in `tables` is one destination slot and maps by position to a value returned by the Subscriber Function. Returning `None` for a slot skips that write. `if_table_exists="append"` inserts rows into the existing table. `"replace"` truncates it first. Neither mode creates a missing table. `schema_evolution="strict"` rejects schema changes. `"iceberg"` applies supported Iceberg-style changes to the StarRocks table; it does not turn the table into an Iceberg table. ## Operational notes - Writes are staged as Parquet, then loaded through StarRocks `FILES()`. - S3 and Azure staging use one credential set. GCS needs service-account JSON for the upload and HMAC credentials for StarRocks's S3-compatible read. - Tabsdata does not contact StarRocks or the staging service when the collection is created. Those checks happen when the subscriber runs. ## Connection `StarRocksDestConn` defines the StarRocks frontend address, database credentials, and cloud staging location. Register the Connection on the Collection where the Subscriber runs. conn-starrocks.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdatak.conn.starrocks:StarRocksDestConn spec: host: 'starrocks-fe.internal' credentials: kind: userPasswordCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:UserPassword spec: user: 'secret:tabsdata_writer' password: 'secret:example-password' staging: kind: starRocksS3Location apiVersion: '1.0' type: tabsdatak.conn.starrocks:S3Location spec: bucket: 'acme-staging' region: 'us-east-1' credentials: kind: awsAccessSecretKeyCredentials apiVersion: '1.0' type: tabsdatak.conn.common.types:AwsAccessSecretKey spec: access_key_id: 'secret:example-access-key' secret_access_key: 'secret:example-secret-key' base_path: '/starrocks' ``` - required - optional - optional - required - required - optional - optional ### Create the collection A subscriber is registered in a destination collection, and a destination collection is created with its connection. Every function registered in the collection uses that connection. Generate a connection template, which lists all the connection fields: ```bash tdk connection template --type starrocks-bulk-out --file conn-starrocks.yaml ``` Edit the template file with the StarRocks connection information, check it, and create the collection with it: ```bash tdk connection validate --file conn-starrocks.yaml ``` ```bash tdk collection create --name starrocks_out --conn-file conn-starrocks.yaml ``` Then register the subscriber in the collection: ```bash tdk fn register --coll starrocks_out --path write_orders.py::write_orders ``` - The collection goes in the `destinations` group, because the connection is a destination connection. `--group` can be left out; if it is given, it must be `destinations`. - `secret:` gives a secret value in the file itself. `$secret:` reads it from an environment variable when the collection is created, so the value never appears in the file. Either way, Tabsdata stores the secret separately, and the stored connection only shows a placeholder. - Functions in the collection must use the connector of its connection: `StarRocksDest` for a collection created with `StarRocksDestConn`. Transformers go in a collection of their own. - To change the connection later, for example to rotate a password, run `tdk collection update --name starrocks_out --conn-file conn-starrocks.yaml`. The connector cannot change; to use another one, create another collection. - Creating a source or destination collection requires permission to manage the project. ### Connection Config Parameters #### `host`, `port`, and `database` `host` identifies the StarRocks frontend. `port` is the MySQL-protocol port, from 1 to 65535, and defaults to `"9030"`. `database` defaults to `"default"` and supplies the database for unqualified destination table names. These fields accept strings or secret references. #### `credentials` A `UserPassword` containing the StarRocks username and password. Cloud storage credentials are configured separately under `staging`. #### `staging` Use one of the location classes from `tabsdatak.conn.starrocks`: | Location | Required fields | Credentials | | --- | --- | --- | | `S3Location` | `bucket`, `region`, `credentials` | `AwsAccessSecretKey` | | `AzBlobLocation` | `container`, `credentials` | `AzureAccountKey` | | `GCSLocation` | `bucket`, `credentials`, `hmac_credentials` | `GCPJson` for uploads and `GCPHmac` for StarRocks reads | Tabsdata must be able to upload the staged files, and StarRocks must be able to read them. For GCS, both credential types are required: uploads use the native GCS API, while StarRocks reads through the S3-compatible endpoint. Each location accepts `base_path`, which defaults to `/`, and an optional `cfg` dictionary for storage-client configuration. A non-root `base_path` starts with `/`, has no trailing `/`, and contains no empty path segments. #### `entity_whitelist` A YAML list of regular expressions, not a comma-separated string. It is checked to be a list of valid regular expressions, but it has no effect yet. Leave it out. #### `conn_cfg` Optional connector configuration overrides. --- # Grok for Log Files > Grok method of TableFrame parses string columns into typed columns using Grok patterns and a ColumnDefinition schema, with predefined patterns, repeated captures, and null handling for non-matching rows. Source: https://docs.tabsdata.com/guide/grok-for-log-files **Grok** parses lines of text, such as the lines of a log file, into typed columns. The `grok` method of a TableFrame matches a Grok pattern against a string column and appends one column for each capture listed in a schema. Grok is usually applied in the Publisher that reads the log file, so the Table it writes is already structured. Because `grok` is a TableFrame method, it can also be used in a Transformer on a Table that holds raw text. For a full walkthrough that reads a log file and publishes the parsed rows, see [Publish data from log files with Grok](https://docs.tabsdata.com/tutorials/publishers/grok-for-log-files). ## Parsing a column `grok` takes three arguments: - The name of the string column to parse, or an expression that resolves to one - A Grok pattern whose named captures become the new columns - A schema that maps each capture name to a `ColumnDefinition` ```python from tabsdatak.tableframe.grok import ColumnDefinition from tabsdatak.tableframe.datatypes import String, Int32, Int64 PATTERN = ( r"%{IPV4:client_ip} " r"%{USER:ident} " r"%{USER:auth} " r"\[%{HTTPDATE:timestamp}\] " r'"%{WORD:method} ' r'%{URIPATHPARAM:request} ' r'HTTP/%{NUMBER:http_version}" ' r"%{INT:response_code} %{INT:bytes}" ) SCHEMA = { "client_ip": ColumnDefinition("ip_address", String), "method": ColumnDefinition("http_method", String), "response_code": ColumnDefinition("status_code", Int32), "bytes": ColumnDefinition("response_bytes", Int64), "timestamp": ColumnDefinition("request_time", String), } parsed = lines.grok("column_1", PATTERN, SCHEMA) ``` lines | column\_1 | | --- | | 203.0.113.10 - - \[10/Oct/2026:13:55:01 +0000\] "GET /index.html HTTP/1.1" 200 512 | | not a log line | 2 rows parsed | column\_1str | ip\_addressstr | http\_methodstr | status\_codei32 | response\_bytesi64 | request\_timestr | | --- | --- | --- | --- | --- | --- | | 203.0.113.10 - - \[10/Oct/2026:13:55:01 +0000\] "GET /index.html HTTP/1.1" 200 512 | 203.0.113.10 | GET | 200 | 512 | 10/Oct/2026:13:55:01 +0000 | | not a log line | null | null | null | null | null | 2 rows The result keeps every existing column and follows these rules: - One column is added for each entry in the schema, in schema order. - Captures that are not in the schema are not added. The pattern above captures `ident`, `auth`, `request`, and `http_version`, but none of them become columns. - The captured text is cast to the `dtype` in the capture's `ColumnDefinition`. - A row whose text does not match the pattern gets `null` in every new column, so a malformed line never fails the Function. ## Patterns A capture is written as `%{PATTERN:capture}`, where `PATTERN` is a predefined pattern name and `capture` is the name the schema refers to. Text between captures must match literally, so characters that have a meaning in regular expressions, such as `[` and `]`, are escaped with a backslash. The full capture syntax is `%{name:alias:extract:definition}`: - **name**: a predefined pattern name, such as `IPV4`, `WORD`, or `INT` - **alias**: the capture name - **extract**: reserved for future use - **definition**: a regular expression, required when no predefined `name` is given Tabsdata includes 320 predefined patterns from the Elasticsearch Grok pattern library. Commonly used patterns for log files include: | Pattern | Matches | | --- | --- | | `IPV4` | An IPv4 address | | `WORD` | A single word | | `INT` | An integer | | `NUMBER` | An integer or decimal number | | `TIMESTAMP_ISO8601` | An ISO 8601 timestamp | | `HTTPDATE` | A timestamp in web server access log format | | `LOGLEVEL` | A log level, such as `INFO` or `ERROR` | | `UUID` | A UUID | | `URI` | A URI | | `DATA` | Any text, matching as little as possible | | `GREEDYDATA` | Any text, matching as much as possible | | `COMBINEDAPACHELOG` | A full Apache combined log line | | `SYSLOGBASE` | The header of a syslog line | ## Schema The schema is a dictionary keyed by capture name. Each value is a `ColumnDefinition` from `tabsdatak.tableframe.grok` with two fields: - **name**: the name of the new column. `None` keeps the capture name, and any other value renames the column. - **dtype**: the data type the captured text is cast to, from `tabsdatak.tableframe.datatypes`. Defaults to `String`. ```python from tabsdatak.tableframe.grok import ColumnDefinition from tabsdatak.tableframe.datatypes import String PATTERN = r"%{TIMESTAMP_ISO8601:ts} %{LOGLEVEL:level} %{WORD:service}: %{GREEDYDATA:message}" SCHEMA = { "ts": ColumnDefinition(None, String), "level": ColumnDefinition("level", String), "service": ColumnDefinition("service", String), "message": ColumnDefinition("message", String), } ``` ## Repeated captures When a pattern uses the same capture name more than once, each repeat is numbered: the first keeps the name, and later ones get a `[1]`, `[2]`, … suffix. The schema refers to repeats by those numbered names. ```python from tabsdatak.tableframe.grok import ColumnDefinition from tabsdatak.tableframe.datatypes import String, Int64 PATTERN = r"%{WORD:city}-%{INT:year} to %{WORD:city}-%{INT:year}" SCHEMA = { "city": ColumnDefinition("from_city", String), "city[1]": ColumnDefinition("to_city", String), "year[1]": ColumnDefinition("to_year", Int64), } ``` The `grok_fields` helper lists the capture names a pattern produces, which is useful for building the schema of a long pattern: ```python from tabsdata.expansions.tableframe.features.grok.engine import grok_fields grok_fields(r"%{WORD:city}-%{INT:year} to %{WORD:city}-%{INT:year}") # ['city', 'city[1]', 'year', 'year[1]'] ``` --- # Custom Connectors > Walkthrough for building a custom Tabsdata connector as a Python package: SQLite source and destination, pyproject entry points, error codes, config classes, plugin logic, and installation. Source: https://docs.tabsdata.com/guide/plugins A **custom connector** allows Tabsdata to connect to external systems that are not officially supported by the current suite of Tabsdata connectors. Custom Connectors are python packages that can be coded by hand or by AI agents connected through Tabsdata's MCP Server. ## File structure This guide builds a SQLite connector as an example. Create a directory named `sqlite_connector` with the following files. The Python package lives in `src/tabsdata_sqlite`. pyproject.toml error.py \_\_init\_\_.py \_plugin.py `pyproject.toml` defines the package, dependencies, and entry points Tabsdata uses to discover the connector. `error.py` defines errors for invalid configuration and failed reads or writes. `__init__.py` defines the connection and function configuration classes users import. `_plugin.py` implements the read and write logic and registers the connector with Tabsdata. tip Download the connector template using the button at the top right of the file tree, or copy the files below. The highlighted lines are the main parts you would change when adapting this example to another system. ### pyproject.toml `pyproject.toml` defines the package metadata, Python dependencies, and entry points Tabsdata uses to load the connector. When adapting this file: 1. Set the package `name`, `version`, and `description`. 2. Add any Python packages the connector needs to `dependencies`. 3. Define one entry point for each source or destination the package provides. pyproject.toml ```toml [build-system] requires = ["setuptools>=69"] build-backend = "setuptools.build_meta" [project] name = "tabsdata-conn-sqlite" version = "0.1.0" description = "Tabsdata source and destination connector for SQLite databases." requires-python = ">=3.12" dependencies = [ "tabsdata>=2.1,<2.2", ] [tool.setuptools.packages.find] where = ["src"] [project.entry-points."tabsdatak.connectors"] src_sqlite = "tabsdata_sqlite._plugin:SQLITE_SRC" dest_sqlite = "tabsdata_sqlite._plugin:SQLITE_DEST" ``` ### error.py `error.py` defines the different errors that can occur during connector usage. `SQLITE_1` through `SQLITE_4` are assigned to the configuration classes in `__init__.py` and report invalid connection or function settings. - `SQLITE_1` when the source connection document fails validation (invalid credentials) - `SQLITE_1` when the destination connection document fails validation (invalid credentials) - `SQLITE_3` when the source connector fails validation (missing/invalid arguments provided) - `SQLITE_4` when the destination connector fails validation (missing/invalid arguments provided) You can define additional error codes in `error.py` and raise them in `_plugin.py` to handle failures during execution. - `SQLITE_5` when a query fails. - `SQLITE_6` when the number of returned tables does not match the configured destination tables. - `SQLITE_7` when a write fails. Message placeholders such as `{table}` are filled when `.exception()` is called. error.py ```python from tabsdatak.conn.common.error import ConnException, ConnInitException from tabsdatak.error import ErrorCode, ErrorDef class SqliteRuntimeException(ConnException): """A SQLite read or write failed.""" class SqliteErrorCode(ErrorCode): SQLITE_1 = ErrorDef(ConnInitException, "SqliteSrcConn validation failed") SQLITE_2 = ErrorDef(ConnInitException, "SqliteDestConn validation failed") SQLITE_3 = ErrorDef(ConnInitException, "SqliteSrc validation failed") SQLITE_4 = ErrorDef(ConnInitException, "SqliteDest validation failed") SQLITE_5 = ErrorDef(SqliteRuntimeException, "SQLite query {index} failed") SQLITE_6 = ErrorDef( SqliteRuntimeException, "SQLite received {slots} table(s) for {tables} target table(s)", ) SQLITE_7 = ErrorDef(SqliteRuntimeException, "SQLite write to {table} failed") ``` ### `__init__.py` `__init__.py` defines the classes users import from `tabsdata_sqlite` and what arguments these classes can accept. - `SqliteSrcConn`: Connection Document for Source Connectors - `SqliteSrc`: Source Connector - `SqliteDestConn`: Connection for Destination Connectors - `SqliteDest`: Destination Connector Within each class, define the configuration fields and their types. Pass the appropriate validation error code as the first argument to the `@_api.dataclass` decorator. ```text @_api.dataclass(SqliteErrorCode.SQLITE_1, kw_only=True) class SqliteSrcConn(Conn): """Connection to the SQLite database a publisher reads.""" path: StrOrSecretSpec ``` tip The `SqlLiteSrcConn` above is configured to accept a single `path` argument as either a string or a secret. If this condition is not met, it throws the `SQLite_1` error code. \_\_init\_\_.py ```python from typing import Annotated, Literal from pydantic import StringConstraints import tabsdatak._api as _api from tabsdatak.api import Dest, Secret, Src, StrOrSecretSpec from tabsdatak.spi import Conn from tabsdata_sqlite.error import SqliteErrorCode TableName = Annotated[str, StringConstraints(pattern=r"^[A-Za-z_][A-Za-z0-9_]*$")] def resolve(spec: StrOrSecretSpec) -> str: return spec.value() if isinstance(spec, Secret) else spec @_api.dataclass(SqliteErrorCode.SQLITE_1, kw_only=True) class SqliteSrcConn(Conn): """Connection to the SQLite database a publisher reads.""" path: StrOrSecretSpec @_api.dataclass(SqliteErrorCode.SQLITE_2, kw_only=True) class SqliteDestConn(Conn): """Connection to the SQLite database a subscriber writes.""" path: StrOrSecretSpec @_api.dataclass(SqliteErrorCode.SQLITE_3, kw_only=True) class SqliteSrc(Src): """Queries a publisher runs against SQLite.""" queries: list[str] @_api.dataclass(SqliteErrorCode.SQLITE_4, kw_only=True) class SqliteDest(Dest): """SQLite tables that receive a subscriber's returned data.""" tables: list[TableName] if_table_exists: Literal["append", "replace"] = "replace" ``` Under this configuration, - the source connection defines **where** Tabsdata connects - the source connector defines **what** query to read from the database - the destination connection defines **where** Tabsdata connects - the destination connector defines **what** Tabsdata tables are written into SQLite and whether to append or replace existing tables. During execution, Tabsdata passes both objects to the connector plugin. \_\_init\_\_.py · Connection ``` class SqliteSrcConn(Conn): path: StrOrSecretSpec ``` → conn-sqlite-in.yaml ``` kind: connectionDef apiVersion: '1.0' type: tabsdata_sqlite:SqliteSrcConn spec: path: str:/data/shop.db ``` \_\_init\_\_.py · Source ``` class SqliteSrc(Src): queries: list[str] ``` → pub\_shop.py ``` source=SqliteSrc( queries=[ "SELECT * FROM customers", "SELECT * FROM orders", ], ) ``` ### \_plugin.py `_plugin.py` contains the connector's runtime logic that uses the inputs defined in `__init__.py` `SqliteSrcPlugin.read_in`: 1. Runs each query from the Publisher configuration. 2. Writes each query result to a Parquet file in `ctx.work_dir`. 3. Returns those files in the same order as `src.queries`. If a query fails, the plugin raises `SQLITE_5` with the query index and original exception. `SqliteDestPlugin.write_out`: 1. Checks that the number of returned table slots matches `dest.tables`. 2. Matches each returned file to the destination table in the same position. 3. Skips any `None` file reference. 4. Reads each remaining Parquet file. 5. Replaces or appends to the destination table based on `dest.if_table_exists`. \_plugin.py ```python import sqlite3 from pathlib import Path import polars as pl from tabsdatak.spi import ( DestContext, DestDef, DestPlugin, SrcDef, SrcPlugin, SrcPluginCtx, TableFileInput, TableFileSpec, TableMode, ) from tabsdata_sqlite import ( SqliteDest, SqliteDestConn, SqliteSrc, SqliteSrcConn, resolve, ) from tabsdata_sqlite.error import SqliteErrorCode def connect(conn: SqliteSrcConn | SqliteDestConn) -> sqlite3.Connection: return sqlite3.connect(resolve(conn.path)) class SqliteSrcPlugin(SrcPlugin[SqliteSrcConn, SqliteSrc]): def read_in( self, ctx: SrcPluginCtx, conn: SqliteSrcConn, src: SqliteSrc, ) -> list[list[TableFileInput]]: con = connect(conn) try: slots = [] for index, query in enumerate(src.queries): try: frame = pl.read_database(query, con) except Exception as e: raise SqliteErrorCode.SQLITE_5.exception(cause=e, index=index) parquet = Path(ctx.work_dir) / f"{index}.parquet" frame.write_parquet(parquet) slots.append([TableFileInput(file=parquet)]) return slots finally: con.close() class SqliteDestPlugin(DestPlugin[SqliteDestConn, SqliteDest]): def write_out( self, ctx: DestContext, conn: SqliteDestConn, dest: SqliteDest, tables: list[TableFileSpec], ) -> None: if len(tables) != len(dest.tables): raise SqliteErrorCode.SQLITE_6.exception( slots=len(tables), tables=len(dest.tables) ) con = connect(conn) try: with con: con.execute("BEGIN") for table, parquet in zip(dest.tables, tables): if parquet is None: continue try: frame = pl.read_parquet(parquet) columns = ", ".join( '"' + c.replace('"', '""') + '"' for c in frame.columns ) if dest.if_table_exists == "replace": con.execute(f'DROP TABLE IF EXISTS "{table}"') con.execute( f'CREATE TABLE IF NOT EXISTS "{table}" ({columns})' ) marks = ", ".join("?" for _ in frame.columns) con.executemany( f'INSERT INTO "{table}" ({columns}) VALUES ({marks})', frame.iter_rows(), ) except Exception as e: raise SqliteErrorCode.SQLITE_7.exception(cause=e, table=table) finally: con.close() SQLITE_SRC = SrcDef( conn=SqliteSrcConn, type_="sqlite-sql-in", system="SQLite", src_version="v1", src=SqliteSrc, table_mode=TableMode.SINGLE, cardinality=lambda src: len(src.queries), plugin=SqliteSrcPlugin, explorer=None, icon=None, ) SQLITE_DEST = DestDef( conn=SqliteDestConn, type_="sqlite-sql-out", system="SQLite", dest_version="v1", dest=SqliteDest, cardinality=lambda dest: len(dest.tables), plugin=SqliteDestPlugin, explorer=None, icon=None, ) ``` ## Installing a connector The connector package must be installed in two places: 1. Wherever `tdk` runs locally, so Tabsdata can validate connections and register Functions. 2. In the server's Function environment, so the connector is available when those Functions execute. - Local - Git - PyPI Step 1Install the local package where tdk runs From the directory containing `sqlite_connector`, run: ```bash pip install ./sqlite_connector ``` Step 2Add the local package to the server's Function environment requirements.txt ```text tabsdata-conn-sqlite @ file:///absolute/path/to/sqlite_connector ``` Step 3Update the server's Function environment ```bash tdkserver venv update \ --name fn \ --requirements requirements.txt ``` danger This command overwrites Tabsdata's existing list of package requirements. Include every complementary package the environment still needs, not just the new one being added. Step 1Install the Git package where tdk runs Replace `` and `` with the repository containing the connector: ```bash pip install "git+https://github.com//.git#subdirectory=sqlite_connector" ``` Step 2Add the Git package to the server's Function environment requirements.txt ```text tabsdata-conn-sqlite @ git+https://github.com//.git#subdirectory=sqlite_connector ``` Step 3Update the server's Function environment After `tdkserver quickstart` has created the instance, update its `fn` environment to include the package. ```bash tdkserver venv update \ --instance \ --name fn \ --requirements requirements.txt ``` danger This command overwrites Tabsdata's existing list of package requirements. Include every complementary package the environment still needs, not just the new one being added. Step 1Install the published package where tdk runs After publishing the connector to PyPI, install its package: ```bash pip install tabsdata-conn-sqlite==0.1.0 ``` Step 2Add the published package to the server's Function environment requirements.txt ```text tabsdata-conn-sqlite==0.1.0 ``` Step 3Update the server's Function environment After `tdkserver quickstart` has created the instance, update its `fn` environment to include the package. ```bash tdkserver venv update \ --instance \ --name fn \ --requirements requirements.txt ``` danger This command overwrites Tabsdata's existing list of package requirements. Include every complementary package the environment still needs, not just the new one being added. ## Using the connector Step 4Generate the connection documents This works as long as `SQLITE_SRC` and `SQLITE_DEST` are registered through the `tabsdatak.connectors` entry points in `pyproject.toml`. tip Run `tdk connection types` to confirm that `tdk` can discover both custom connector types. Generate the appropriate connection template: - Source - Destination ```bash tdk connection template --type sqlite-sql-in --file conn-sqlite-in.yaml ``` ```bash tdk connection template --type sqlite-sql-out --file conn-sqlite-out.yaml ``` Step 5Fill out the connection documents The `spec` field contains the arguments defined in `__init__.py`. A completed document will look like this: - Source - Destination conn-sqlite-in.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdata_sqlite:SqliteSrcConn spec: path: str:/data/shop.db ``` conn-sqlite-out.yaml ```yaml kind: connectionDef apiVersion: '1.0' type: tabsdata_sqlite:SqliteDestConn spec: path: str:/data/warehouse.db ``` Step 6Attach the connections to Collections - New Collection - Existing Collection Create a new collection and attach the filled-out connection document. - Source - Destination ```bash tdk collection create --name shop --group sources --conn-file conn-sqlite-in.yaml ``` ```bash tdk collection create --name warehouse --group destinations --conn-file conn-sqlite-out.yaml ``` Update an existing collection with the filled-out connection document. - Source - Destination ```bash tdk collection update --name shop --conn-file conn-sqlite-in.yaml ``` ```bash tdk collection update --name warehouse --conn-file conn-sqlite-out.yaml ``` ## Use the connectors in a Function - Publisher - Subscriber pub\_shop.py ```python from tabsdatak.api import publisher, TableFrameSpec from tabsdata_sqlite import SqliteSrc @publisher( source=SqliteSrc(queries=["SELECT * FROM customers", "SELECT * FROM orders"]), output_tables=["customers", "orders"], ) def pub_shop( customers: TableFrameSpec, orders: TableFrameSpec ) -> tuple[TableFrameSpec, TableFrameSpec]: return customers, orders ``` ### Register into Tabsdata ```bash tdk fn register --coll shop --path pub_shop.py::pub_shop ``` sub\_warehouse.py ```python from tabsdatak.api import subscriber, TableFrameSpec from tabsdata_sqlite import SqliteDest @subscriber( destination=SqliteDest(tables=["customers", "orders"], if_table_exists="replace"), input_tables=["shop/customers", "shop/orders"], ) def sub_warehouse( customers: TableFrameSpec, orders: TableFrameSpec ) -> tuple[TableFrameSpec, TableFrameSpec]: return customers, orders ``` ### Register into Tabsdata ```bash tdk fn register --coll warehouse --path sub_warehouse.py::sub_warehouse ``` --- # Automatic Orchestration > How Tabsdata determines which functions to run and in what order based on the tables they read and write. Source: https://docs.tabsdata.com/guide/automatic-orchestration In Tabsdata, **Automatic Orchestration** handles the order in which Functions run based on the Tables they read and write. Instead of requiring users to manually define a DAG, Tabsdata uses the declared dependencies between Tables and Functions to determine what needs to run. When a Function is triggered, Tabsdata automatically identifies any downstream Functions that depend on its output and executes them in the appropriate order. _Video: A publisher is triggered from the CLI, Tabsdata builds an execution plan by following the tables each function declares, and each table that gets new data starts the function that reads it, until the plan commits new versions of customers and customers_clean._ ## Traditional Orchestration Traditionally, orchestrating a data pipeline involves manually defining a Directed Acyclic Graph (DAG) that specifies how each step depends on the others and the order in which they should execute. As a pipeline grows or changes, the DAG has to be updated to account for any new or modified dependencies. This also means that adding a new step to a pipeline requires updating its orchestration logic, even if the step's inputs and outputs are already known. ## How Tabsdata Orchestrates Functions In Tabsdata, Functions do not pass data directly to one another. Instead, Functions write their output to Tabsdata Tables, which can then be read by other Functions. Each Function declares the Tables it reads from and writes to as part of its decorator: - **Publishers** declare the Tables they write to using `output_tables`. - **Transformers** declare the Tables they read from using `input_tables` and the Tables they write to using `output_tables`. - **Subscribers** declare the Tables they read from using `input_tables`. Because these dependencies are declared ahead of time, Tabsdata already knows which Functions depend on each other. When a Function writes a new version of a Table, Tabsdata can identify every downstream Function that reads from it. For example, registering a new Transformer that reads from the `customers` Table automatically adds it to the workflow. The next time `customers` receives a new version, Tabsdata knows to execute the Transformer without requiring any changes to an existing DAG. ## Tables as Triggers In Tabsdata, new Table versions are what trigger downstream Functions to run. By default, Transformers and Subscribers run whenever one of their input Tables receives a new version, unless a different trigger is specified using `trigger_by`. Publishers do not have input Tables, so they can be triggered manually, through a cron schedule, or by a Table specified in `trigger_by`. Stream Publishers are triggered whenever a new batch of data is ready from their stream source. See [Execution and Triggers](https://docs.tabsdata.com/guide/execution-and-triggers) for more information about the available trigger types. ## Execution Plans Whenever a Function is triggered, Tabsdata uses the declared Table dependencies to build an [Execution Plan](https://docs.tabsdata.com/guide/execution/execution-plans) containing all the downstream Functions that need to run. The Execution Plan determines the order in which Functions execute, including Functions across different Collections, so that each Function runs after the Tables it depends on have been updated. As the plan executes, each Function reads the data written by the upstream Functions in that same execution. However, Tabsdata does not commit any of the new Table versions until every Function in the Execution Plan has completed successfully. If any Function fails, none of the new Table versions are committed. This prevents partially completed workflows from leaving Tables in inconsistent states. Tabsdata also records which Table versions each Function read and wrote during the execution, allowing users to trace how data was produced and how it moved through the workflow. See [Lineage](https://docs.tabsdata.com/guide/lineage) for more information. --- # Execution & Triggers > Covers the four Tabsdata function trigger types (manual via `tdk fn trigger` or the UI, CRON schedule, new-data on input table versions, and stream workers such as SQL CDC and Kafka), plus automatic execution plans and streaming publishers. Source: https://docs.tabsdata.com/guide/execution-and-triggers ## Triggers A **trigger** decides _when_ a Function runs. There are four kinds of triggers: Manually When you run it When a function is executed through the `tdk fn trigger` CLI Command or through the `trigger` button in the Tabsdata UI On a schedule At set times When a function is executed on a CRON schedule On new data When an input Table receives new data Execute the Function whenever an input Table receives a new Version. On a stream batch When a batch of streamed data is due Execute the Function each time a stream worker, such as SQL CDC or Kafka, hands over a batch ## Automatic orchestration No matter which trigger fires a function, Tabsdata doesn't run just that one function. It follows the data dependencies and builds an [execution plan](https://docs.tabsdata.com/guide/execution/execution-plans): the graph of every function that must run as a result, in the correct order, across collections. There is no orchestration for you to wire up. ## Pristine state Every table in an execution plan stays in sync, because new data is only committed if every function in the plan succeeds. A partial failure commits nothing. ## Streaming publishers A normal publisher pulls a batch of data each time it runs. A **streaming publisher** stays running continuously, creating new table versions as data arrives: for change-data-capture feeds, log tails, and message queues like [Kafka](https://docs.tabsdata.com/tutorials/publishers/kafka). --- # Manual Triggers > Covers triggering a Tabsdata Function by hand with `tdk fn trigger` from the CLI or the UI, the default for Publishers without `trigger_by`, and how the resulting Execution Plan runs downstream Functions. Source: https://docs.tabsdata.com/guide/triggers/manual A manual trigger runs a Function on demand, from the CLI or the UI. Nothing about the Function's definition needs to change to allow it: every Function can be triggered by hand, including one that already has a cron or Table trigger configured. _Video: A Function triggered from the Tabsdata CLI_ ## From the CLI ```bash tdk fn trigger --coll --name ``` `` is the name the Function was registered under, not the name of its file. ## The default for Publishers A Publisher reads no Tabsdata Tables, so there is nothing upstream that could fire it. With no `trigger_by` on the decorator, a Publisher runs only when triggered manually. Transformers and Subscribers behave differently, because they do have input Tables. Left unconfigured, they run on a commit to any of them. See [Table Commit Triggers](https://docs.tabsdata.com/guide/triggers/table-commit). ## What a manual trigger actually starts Triggering one Function does not run only that Function. Tabsdata builds an [Execution Plan](https://docs.tabsdata.com/guide/execution/execution-plans) covering every downstream Function that has to run as a result, and nothing commits unless all of them succeed. For the UI walkthrough and the full `trigger_by` reference, see [Working with Triggers](https://docs.tabsdata.com/guide/triggers). --- # Cron Triggers > Cron triggers in Tabsdata: crontab mask syntax with UTC schedules, optional start and end window, and enabling or disabling Functions with tdk fn enable and disable. Wait, that's over-long and has a colon-style fragment. Corrected: Cron Triggers: five-field Unix crontab masks run in UTC, optional start and end windows, and enabling or disabling Functions with tdk fn enable/disable. Source: https://docs.tabsdata.com/guide/triggers/cron A scheduled trigger runs a Function at fixed times, whether or not any upstream data has changed. Publishers, Transformers, and Subscribers can all use one. _Video: A cron schedule triggering a Function run_ Configure it by passing a `CronTrigger` to `trigger_by`. ```python from tabsdatak.api import CronTrigger, publisher, TableFrameSpec from tabsdatak.conn.mysql import MySQLSrc @publisher( trigger_by=CronTrigger("0 * * * *"), source=MySQLSrc(queries=["SELECT * FROM sales"]), output_tables=["sales"], ) def read_sales(sales: TableFrameSpec) -> TableFrameSpec: return sales ``` ## The mask The first argument is a standard Unix crontab mask with five fields and minute precision: ```text Minute Hour Day Month Day-of-Week ``` | Mask | Runs | | --- | --- | | `0,30 * * * *` | every 30 minutes, on the hour and at minute 30 | | `0 0 * * *` | daily at 00:00 | | `0 22 * * FRI` | Fridays at 22:00 | Non-standard shorthands such as `@hourly` and `@daily` are **not** supported. note All crontab schedules are interpreted in **UTC**. ## Limiting the window `CronTrigger` also takes optional `start` and `end` parameters, each either a timezone-aware Python `datetime` or an ISO 8601 string in UTC (`2026-01-01T12:00:00Z`). With them set, the mask only fires between those two times. With them omitted, it fires indefinitely from the moment the Function is registered. ## Enabled and disabled Functions A cron schedule only runs while its Function is enabled. Registration defaults to enabled, and `tdk fn register` takes `--enabled` / `--disabled` explicitly. An already-registered Function is toggled with `tdk fn enable` and `tdk fn disable`. A disabled Function ignores its mask entirely. It does not queue missed runs and fire them on re-enable. ## Where credentials come from The example above names a MySQL query but carries no host and no password. Those live on the Collection's Connection. See [Connections](https://docs.tabsdata.com/guide/connections). --- # Table Commit Triggers > Configures Table commit triggers via the `trigger_by` decorator option: default, empty-list, and named-Table behavior, cross-Collection `collection/table` references, and returning None to skip commits. Source: https://docs.tabsdata.com/guide/triggers/table-commit A Table trigger runs a Function whenever one of the Tables it watches receives a new Version. This is what makes a pipeline cascade: a Publisher commits, the Transformers reading that Table run, they commit in turn, and the chain continues to the Subscribers at the end. _Video: A new Table Version triggering a Function run_ Configure it with the `trigger_by` list on the decorator. ## The three configurations **Omitted.** Transformers and Subscribers fire on a commit to any of their input Tables. Publishers have no input Tables, so they get no automatic trigger at all and run manually or on a schedule. **An empty list.** `trigger_by=[]` disables automatic triggering. Use it when a Transformer or Subscriber should read its input Tables but not wake up when they change. **A non-empty list.** The Function watches exactly the Tables named and nothing else. This is how you trigger on a subset of the input Tables, or on a Table the Function does not read at all. ```python @transformer( trigger_by=["sales", "inventory/stock"], input_tables=["sales"], output_tables=["sales_by_rep"], ) def summarize(sales: TableFrameSpec) -> TableFrameSpec: return sales ``` ## Referencing another Collection's Tables Write a Table outside the Function's own Collection as `collection/table`, as `inventory/stock` does above. Unqualified names resolve inside the Function's own Collection. Cross-Collection references work only within a single Project. A reference that crosses Projects surfaces as `Dependency table '' does not exist`, because the row is not visible rather than missing. A Table cannot trigger the Function that writes it. ## A commit is not the same as a change Every run that returns data commits a new Version, whether or not the rows differ from the previous one. A Publisher writing three Tables commits all three on every run, so a Function watching any of them fires each time, even for the Tables whose contents were identical. Returning `None` from a slot is the exception: it commits nothing and leaves the current Version in place, so downstream Functions watching that Table do not fire. --- # Stream Triggers > Stream trigger page: when a @stream_publisher batch is due (batch_min_secs, batch_max_secs, batch_max_rows, batch_max_bytes, batch_idle_secs) and the Kafka, MySQL CDC, Postgres CDC, and Db2 CDC stream sources. Source: https://docs.tabsdata.com/guide/triggers/stream A stream trigger runs a Function each time a batch of streamed data is due. It is the only trigger a `@stream_publisher` has: that decorator takes no `trigger_by` at all. _Video: A Kafka stream triggering a Function run when a batch is due_ ```python from tabsdatak.api import stream_publisher, TableFramesSpec, TableFrameSpec from tabsdatak.conn.kafka import KafkaStreamSrc from tabsdatak.tableframe.functions import concat @stream_publisher( source=KafkaStreamSrc( topic="users", data_format="avro", schema=AVRO_SCHEMA, ), batch_max_secs=300, batch_max_rows=500, output_tables=["users"], ) def ingest(users: TableFramesSpec) -> TableFrameSpec: return concat(users) ``` ## How a batch becomes a run A stream worker reads the source continuously. What it reads is buffered in memory and written out to data files as the buffer fills. When a batch is due, the files written so far, and anything still in memory, are staged and handed to the Function as one batch. The Function runs once per batch and commits a new Version of each of its output Tables. The worker is long-lived and independent of the Function. It keeps reading while the previous batch is being processed, so a slow Function does not drop records. Batches are processed one at a time, in order, and if the Function takes longer than the batch interval, batches queue up and are processed in turn. ## When a batch is due The `batch_*` parameters of `@stream_publisher` decide when a batch is due: - `batch_min_secs` is the shortest time between two batches, and never less than 60 seconds. - Once `batch_min_secs` have passed, a batch is due when `batch_max_secs` have passed, or when at least `batch_max_rows` rows or `batch_max_bytes` bytes are waiting. At least one of the three is required. - With no data waiting, nothing runs, unless `batch_idle_secs` is set. Once `batch_idle_secs` have passed, the Function runs with an empty batch. Tabsdata looks for new batches about every 30 seconds, so a batch can start up to that long after it is due. See [Batch parameters](https://docs.tabsdata.com/guide/functions/stream-publishers#batch-parameters) for the full rules and examples. The buffer settings in a source's `src_cfg`, such as `tabsdata.kafka.buffer.max_rows`, only decide when the worker writes what it has read to a file. They do not make a batch due. ## Available stream sources | Source | System | Output Tables | | --- | --- | --- | | `KafkaStreamSrc` | Apache Kafka | the Tables named in `output_tables` | | `MySQLCdcStreamSrc` | MySQL | one per tracked Table | | `PostgresCdcStreamSrc` | PostgreSQL | one per tracked Table | | `Db2CdcStreamSrc` | IBM Db2 | one per tracked Table | All four are registered as stream sources, which `tdk connection types` shows in the **Mode** column as `Stream Source`. Like ordinary sources they belong to a `sources` Collection, and the address and credentials live on that Collection's Connection rather than on the source object. The CDC sources track a list of qualified Table names and resume from a stored position. Postgres additionally requires a logical replication slot, since the WAL it reads is retained on the server until the consumer confirms it. See [Postgres CDC](https://docs.tabsdata.com/guide/connectors/postgres-cdc), [MySQL CDC](https://docs.tabsdata.com/guide/connectors/mysql-cdc), [Db2 CDC](https://docs.tabsdata.com/guide/connectors/db2-cdc), and [Kafka](https://docs.tabsdata.com/guide/connectors/kafka). ## Constraints on the Function A `@stream_publisher` differs from the other decorators in three ways: - It takes one `TableFramesSpec` parameter per source slot, always in array mode, because a batch is a list of frames rather than a single frame. - It returns one `TableFrame` per output Table, the same as a Publisher. - It must **not** declare a `ctx` parameter. It runs inside the runtime's stream loop rather than per-transaction, so there is no `TrxCtx` at the call boundary and declaring one fails registration. See [Stream Publishers](https://docs.tabsdata.com/guide/functions/stream-publishers) for the decorator's parameters and lifecycle. --- # Secrets Management > Explains how Tabsdata stores credentials in Connections (dehydrated into a secret store), how to register them with tdk connection template and tdk collection create/update, and the security tradeoffs of passing secrets to an LLM via MCP, environment variables, or files. Source: https://docs.tabsdata.com/guide/secrets-management/ Credentials used by Publisher and Subscriber Functions are stored in Connections. Connections are credential stores that are indirectly attached to a function and enable the function access to specific credentials. To register a credential into Tabsdata, the user must generate a `.yaml` connection document template through the `tdk connection template` cli command and fill in the required credentials. Once the document is completed, it must be attached to a collection through the `tdk collection create` or `tdk collection update` cli command. During connection registration, Tabsdata **dehydrates** the Connection document by extracting its credentials into a separate secret store and replacing them in the stored Connection with references to those secrets. Once stored, credentials are only made available to a Function when it runs. In fact, once credentials are in Tabsdata, there is no way to read them out of the system. This is by design, ensuring your secrets are always secure. When creating workflows through MCP, it is important to remember that you must provide your credentials to your LLM in order for it to pass those credentials into Tabsdata. It is important to understand the security implications of doing so. There are several ways to pass secrets to an LLM for creating connections. ## Create the Connection Yourself The most secure option is to create the Connection outside of the agent. You can ask the agent to generate a Connection template, then add the credentials yourself, and register the Connection with Tabsdata using the `tdk collection create` or `tdk connection update`\` CLI command. The agent can then create or register Functions against the Collection without ever accessing the credentials required. This keeps the credentials out of the agent's context and session history. ## Provide Credentials Through Environment Variables Connection templates can reference credentials using `$secret:`. When working through an agent, the environment variable must be available to the process running the LLM and the Tabsdata server. This means the variable must be set in the shell environment before both processes are started. This avoids placing the credential directly in the agent prompt, but still gives the agent process access to it. Environment variables may also be visible to other processes running in the same environment. For these reasons, environment variables are better suited for development and testing than for long-lived production credentials. Adding or updating secrets also requires restarting both the agent and the Tabsdata server with the updated shell environment. warning Storing credentials in environment variables does not prevent the agent from accessing their values. However, it allows the agent to pass the environment variable name as a literal to the Tabsdata server, which can then resolve the value server-side. This reduces the likelihood of the secret being captured in the agent's transcript. ## Provide Credentials Directly to the Agent You can also provide credentials directly to the agent and have it create the Connection for you. This is the simplest option, but the credential passes through the agent's context and may be stored in its session history. Credentials provided this way may be at increased risk for exposure. ## Provide Credentials Through a File In scenarios where the agent is requesting multiple points of data, or in sessions where providing credentials directly inline requires pasting the same content multiple times, you can pass all relevant data into a `.txt` file and pass the path to the agent during your session warning Putting credentials in a file and asking an agent to read that file does not prevent the agent from accessing them. Once the agent reads the file, the credentials have passed through the agent just as if they were provided directly. --- # Service Tokens > How a service token lets CI jobs, schedulers and scripts call Tabsdata as its owner without a password, and which of the owner's permissions it carries. Source: https://docs.tabsdata.com/guide/service-tokens A **service token** is a credential owned by a user that automation such as CI jobs, schedulers and scripts uses to call Tabsdata as that user, without a password. A service token acts with a narrowed subset of its owner's permissions: project development work only. ## How a service token works Each user account holds at most two service tokens, in slots `1` and `2`. With two slots, you can create a new token while the old one is still in use. A token starts with `tdst_` and is displayed only once, when it is created. The server stores only a SHA-256 digest of the token and can never show it again. Every token is created with a lifetime in days. The deployment's `service_tokens.max_days` setting is the ceiling (90 in the shipped configurations). A token that never expires (`-1`) works only where `max_days` is `-1`. If `max_days` is not set, service tokens are disabled on that deployment. A token stops working when any of the following happens: - the token expires - the owner or an administrator revokes the token - the owner's account is disabled - the owner's account is deleted Changing the owner's password does not affect existing tokens. A session started with a token is never renewed, and lasts until the token expires or is revoked. Clients send the token in the `Authorization: Bearer ` header. ### Rotating a token To rotate a token without downtime: 1. Create a token in the free slot. 2. Move your consumers to the new token. 3. Wait until **Last used** for the old slot stops changing. **Last used** updates about every ten minutes. 4. Revoke the old slot. ## Managing tokens in the UI Open **Account → Service tokens**. The page shows **Slot 1** and **Slot 2**. A filled slot shows the token's prefix (`tdst_v1_…`), its comment, and when the token was created, when it expires and when it was last used. 1. To create a token, click **Create token** in an empty slot. Choose **Expires after a number of days** (90 by default) or **Never expires**, add an optional comment of up to 200 characters, then click **Create token**. 2. Click **Copy** in the **Token created in slot N** panel. The token is shown only on this page. After you click **Dismiss** or leave the page, you can't get the token back. If you lose it, revoke the slot and create a new token. 3. To revoke a token, click **Revoke** in a filled slot, then **Revoke token**. Every client using that token stops working immediately. ## Managing tokens with the tdk CLI To manage tokens, log in with your username and password first. ```bash tdk auth service-token create --slot 1 --expires-in-days 90 --comment "nightly CI" tdk auth service-token list tdk auth service-token revoke --slot 1 ``` - `create` prints the token once. `--expires-in-days` has no default: pass a positive number, or `-1` for a token that never expires. - `create` is refused if the slot already has a token. Revoke that slot first. - `list` shows each slot's hint, comment, creation time, expiry and **Last used** (updated about every ten minutes). To run commands as a token, export the token and log in. With no value, `--token` reads the `TDK_TOKEN` environment variable, so the token never appears in shell history: ```bash export TDK_TOKEN=tdst_v1_... tdk login --server https://tabsdata.example.com --token ``` `--token` cannot be combined with `--user` or `--password`. A token login is never renewed. When the token expires or is revoked, log in again with a new token. See [tdk auth](https://docs.tabsdata.com/cli/tdk/auth/) for every flag of the `service-token` commands. ## Permissions a token carries A token carries two of its owner's project permissions, `view_project` and `develop_project`, in the projects where the owner holds them. A token never has more permissions than its owner. The server reads the owner's current grants every time the token is used. If an administrator removes a project grant from the owner, the token loses that grant immediately. Higher roles count. If the owner has `manage_project` on a project, the token gets `develop_project` there. A system admin's token can therefore do development work in every project. | Area | What the token can do | | --- | --- | | Projects | List and read the owner's projects; read their groups, metrics, snapshots and in-flight work | | Collections | Create, update, delete and read collections; read a collection's connection | | Functions | Register, update, patch, enable, delete and trigger functions; read history, executions and bundles | | Tables | Read schema, samples, data versions and lineage; download data; delete tables | | Runs | List plans, transactions and function runs; cancel and recover them; replay function runs | | Workers | List workers and read or tail their logs | | Identity | `GET /auth/info` (who am I) and read its own token record | ## Permissions a token does not carry Every other operation is refused with `403 This operation cannot be performed with a service token`, even when the owner holds the permission. A leaked token therefore can't grant access, create accounts or create another token. | Not carried | What this blocks for a token | | --- | --- | | `manage_access` (system) | Managing users, permission grants and other users' tokens | | `manage_roles` (system) | Creating, editing and assigning roles | | `create_projects` (system) | Creating projects | | `manage_project` (project) | Updating or deleting a project; creating, changing or deleting its groups | | Token management | Creating, listing or revoking tokens, including its own. A token can't create its own replacement | | Account self-service | Changing the owner's password; refreshing or extending its own session | An administrator with `manage_access` can list and revoke any user's tokens through the REST API (`/users/{user}/service-tokens`), using a password session. --- # User Interface > Tabsdata web UI pages for the Overview, Projects, Collections, Functions, Tables, Executions, Worker logs, Triggers, Search, Tabby, Security, System, and Account, plus the local URL and demo. Source: https://docs.tabsdata.com/guide/user-interface The Tabsdata **User Interface** (UI) is a web interface for inspecting and managing the resources within a Tabsdata Server. When running Tabsdata locally, the UI is available at `http://localhost:2457`, where `2457` is the Server's default public port. Sign in using the same username and password used with the `tdk` CLI. The UI demo provides a read-only Tabsdata Server with three Projects and existing execution history. Actions that create, modify, trigger, or delete resources are disabled. [Explore the UI](https://docs.tabsdata.com/ui-demo/) ## Overview The **Overview** page summarizes activity across every Project in the Server. For each Project, the page shows: - Tables committed - Rows committed - Running Executions - Warnings - Errors Activity can be viewed across six time ranges. ## Projects The **Projects** page shows activity for the selected Project, including data that is currently in flight and data that has already been committed. The page also includes a graph of committed data over time. From the UI, you can: - Create Projects - Share Projects - Switch the currently selected Project Most pages in the UI are scoped to the selected Project. Use the Project selector in the side panel to switch between Projects. ## Collections The **Collections** page lists the Collections within the selected Project. Opening a Collection shows its: - Functions - Tables - Connection configuration Sensitive credentials stored in a Connection are hidden from the UI. From the UI, you can also: - Create, edit, and delete Collections - Grant a Collection access to Tables in another Collection - Remove previously granted access ## Functions Opening a **Function** shows the code registered with Tabsdata and its version history. Function versions are retained so you can inspect previous versions, including the version used for a particular Function Run. From the UI, you can: - Trigger a Function - Delete a Function Editing a Function's code is performed using `tdk` rather than through the UI. ## Tables Opening a **Table** shows its schema, sample rows, and version history. Each Table version includes the Execution that wrote it, allowing changes in the Table to be traced back to the Execution that produced them. The Table page also includes **Run SQL query**, which lets you edit and run a SQL query against the Table's sample data. From the UI, you can also delete a Table. ## Executions The **Executions** page shows the Execution Plans created within the selected Project. Execution Plans are separated into: - Active - Issues - Finished Opening an Execution Plan shows a graph of its execution flow and provides access to the Transactions, Function Runs, and Workers created as part of the Execution. Transactions, Function Runs, and Workers can each be filtered by status. ### Worker logs Opening a Worker provides access to the logs produced while it was running. Worker logs can be: - Viewed in the UI - Viewed in their raw form - Downloaded When a Function Run fails, its Worker logs contain the output and errors produced during execution. ### Triggers The Executions area also shows the triggers configured for the Project, including: - Cron Triggers - Stream Sources This lets you inspect both what causes Functions to run and the Execution Plans created when they run. ## Search The **Search** page searches for resources across the Tabsdata Server. Use Search to locate a resource without navigating through its Project and Collection. ## Tabby **Tabby** is the built-in Tabsdata agent for interacting with and inspecting a Tabsdata Server. See [Tabby](https://docs.tabsdata.com/guide/tabby) for setup, commands, and an example chat. Tabby requires an Anthropic or OpenAI API key to be configured on the Server. From Tabby, you can: - Start a new chat - Continue existing chats - Stop a response while it is being generated - Search previous messages - Export a chat as Markdown - Use page-specific suggested prompts - Run a Server health check Suggested prompts depend on the page you are viewing. For example, a Collection page may include prompts for listing Functions or creating a Publisher. The **Run health check** action generates an AI-written report about the current health of the Server. ## Security The **Security** page manages users, roles, and permissions for the Tabsdata Server. From the UI, you can: - Create, edit, and delete users - Create and delete roles - Add users to roles - Remove users from roles - Grant permissions - Remove permissions ## System The **System** page shows the status and runtime information for the Tabsdata Server. ## Account The **Account** page shows information about the currently signed-in user. From the UI, you can also: - Change your password - Log out - Switch between light and dark themes - Pin resources to the side panel - Clear pinned resources The Server's SSL certificate can also be downloaded from the login page. --- # Tabby > The built-in operational agent, opened from the UI. It answers questions about the server's resources, monitors dataflows, traces failures to root cause and suggests fixes, and, when the server allows it, creates and changes resources on request. It runs in the server's AI agent service (port 2459) and needs an Anthropic or OpenAI key, set with `tdkserver quickstart --config anthropic-key=env:ANTHROPIC_API_KEY`. Source: https://docs.tabsdata.com/guide/tabby **Tabby** is the built-in Tabsdata agent, opened from the **Tabby** page of the [User Interface](https://docs.tabsdata.com/guide/user-interface). Tabby answers questions about the resources on a Tabsdata Server and, when the Server allows it, creates and changes them on request. Tabby Did any execution plans fail in customer\_management? No. The 10 most recent execution plans in customer\_management all finished, and each one committed 5 function runs with no errors. The three latest: | Execution plan | Triggered by | Status | Duration | | --- | --- | --- | --- | | ep20260924-14 | salesforce/ingest\_opportunities | Finished | 11s | | ep20260924-13 | salesforce/ingest\_leads | Finished | 12s | | ep20260924-12 | salesforce/ingest\_customers | Finished | 12s | Each plan runs the Salesforce ingest for one entity, then its bronze, silver, and gold transformers and the Snowflake load. anthropic claude-sonnet-4-6 Ask anything about your data flows… (⌘K) ↑ The UI demo includes the Tabby page. Messages cannot be sent from the demo. [Open Tabby in the UI demo](https://docs.tabsdata.com/ui-demo/tabby) ## Requirements Tabby runs in the Server's AI agent service, on port `2459` by default, and requires an Anthropic or OpenAI API key. When no key is configured, the Server starts without the AI agent service and Tabby is unavailable. The key is set when the instance is created, with `--config anthropic-key=…` or `--config openai-key=…`: ```bash tdkserver quickstart --config anthropic-key=env:ANTHROPIC_API_KEY ``` To add or change the key on an existing instance, download the instance's config, set `api_key` under `ai_providers.anthropic` or `ai_providers.openai` in `aiagent.yaml`, and apply the change: ```bash tdkserver config get --with-secrets tdkserver config put --apply ``` note The [MCP Server](https://docs.tabsdata.com/guide/mcp) and the conversational agent both run within the same AI Agent process. Although only the conversational agent requires a valid API key, an API key must still be provided during server initialization. Otherwise, the AI Agent process will not start on port `2459`. To use the MCP Server, ensure that an API key is passed to the server during initialization. If the conversational agent is not being used, this can be a placeholder or dummy key. ## Starting a chat Open **Tabby** from the side navigation, or press **⌘K** from any page to ask Tabby about the page you are viewing. A new chat offers four starting prompts: - Set up a data flow - Connect to an external system - Check executions health - Show me what Tabsdata can do Other pages suggest prompts for what they show. A Collection page, for example, suggests listing its Functions, listing its Tables, and creating a Publisher. ## Commands Commands are typed into the message box. | Command | Description | | --- | --- | | `/help` | Show available commands | | `/new` | Start a new conversation | | `/clear` | Clear the current conversation | | `/export` | Export the conversation as Markdown | | `/models` | List available providers and models | | `/provider ` | Switch AI provider | | `/model ` | Switch to a specific model, by full or partial name | | `/health` | Show agent health and status | | `/pop [name]` | Open resources from the conversation in new tabs, or one resource by name | The provider and model can also be switched from the selector above the message box. ## Managing chats Chats are stored on the Server and can be continued later. From a chat, you can: - Search the chat's messages - Export the chat as Markdown - Stop a response while it is being generated - Retry a response that failed ## Building custom connectors Tabby can write a custom connector for a system that Tabsdata does not support natively. Tabby's reference includes the connector interface in `tabsdatak.spi`, and every connector installed on the Server, including custom ones, is part of what Tabby knows. Installing the connector remains a manual step. See [Custom Connectors](https://docs.tabsdata.com/guide/plugins). ## Changes to resources Whether Tabby can create, change, or delete resources is set by `allow_writes` in the Server's `aiagent.yaml` config. --- # MCP > The MCP page covers the `python -m tabsdatak.ag.mcp -` proxy command, its use of `tdk login` sessions, the `/ai-agent/proxy/mcp` endpoint on port 2459, and the API key and connection troubleshooting steps. Source: https://docs.tabsdata.com/guide/mcp The **Model Context Protocol (MCP) Server** lets an external AI agent work with your Tabsdata Server. An agent can inspect resources, write and register Functions, query Tables, and investigate failed executions. Claude Code and Codex connect to it through the `tabsdata-agent` package. The MCP Server and [Tabby](https://docs.tabsdata.com/guide/tabby) share the same tools. Tabby is the chat in the Tabsdata UI; MCP makes those tools available to the agent you use in your terminal or editor. ## Working with Tabsdata The MCP Server gives the agent access to: - **Resources**: Projects, Groups, Collections, Connections, Functions, and Tables. - **Data**: Table schemas, data versions, and queries against published data. - **Executions**: Execution Plans, Transactions, Function Runs, and logs for diagnosing failures. - **Reference**: SDK documentation, connector information, and code examples for writing Functions. - **Skills**: instructions for tasks such as creating a data flow or fixing an execution. The agent searches for the tools and skills it needs as it works. For example, you can ask it to list the Tables in a Collection, inspect a failed Function Run, or build a Publisher from a database table. ## Connection Your agent starts a local MCP proxy with: ```bash python -m tabsdatak.ag.mcp - ``` The trailing `-` selects standard input and output as the transport between the agent and the proxy. This command belongs in the agent's MCP configuration. See [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) for the installation commands. The proxy reads the Server address and session saved by `tdk login`, then forwards requests to the Server's `/ai-agent/proxy/mcp` endpoint over HTTP. The agent uses your Tabsdata session and has your permissions. The proxy reads the session again on each request. If your login expires or you switch Servers, run `tdk login` again. The next request uses the new session without restarting the proxy. ## Server requirements The MCP Server runs in the same AI agent service as Tabby, on port `2459` by default. That service must be running before an external agent can use its tools. An Anthropic or OpenAI API key must be set when the Server is created for the AI agent service to start. MCP itself does not need a valid LLM key, so a placeholder is sufficient when only using an external agent. Tabby requires a valid key. See [Tabby requirements](https://docs.tabsdata.com/guide/tabby#requirements) for setting or changing it. ## Connection problems If the agent cannot reach Tabsdata, check that the Server and its AI agent service are running. If the session is no longer valid, log in again and retry the request. If Python cannot find `tabsdatak.ag.mcp`, install `tabsdata-agent` in the environment used by the agent's MCP command. --- # Data Quality > Data quality checks in Tabsdata: DataQuality classifiers (IsNotNull, IsBetween, IsIn, Matches, ScaleCategorizer) and operators (Filter, Select, Summary, Fail, Enrich) passed via on_tables to publishers and transformers. Source: https://docs.tabsdata.com/guide/data-quality **Data quality** checks validate the rows of a Table as a Function writes it. A check is a `DataQuality` object passed to a Publisher or Transformer through the `on_tables` argument of its decorator. After the Function succeeds, Tabsdata runs the check against the named output Table, where it can label rows, remove or copy rows that fail, write a summary, or fail the Transaction. Data quality checks are available on `@publisher`, `@stream_publisher`, and `@transformer`. `@subscriber` does not take `on_tables`. ```python from tabsdatak.api import transformer, TableFrameSpec from tabsdatak.dataquality import ( AnyFailed, DataQuality, Filter, IsBetween, IsIn, IsNotNull, Summary, ) # transformer that dedupes the raw orders, with a data quality check that moves # rows failing any rule to orders_rejected and writes a summary of the verdicts @transformer( input_tables=["raw_orders"], output_tables=["orders"], on_tables=[ DataQuality( table="orders", classifiers=[ IsNotNull(["order_id"]), IsBetween(["amount"], min_val=0, max_val=10000), IsIn(["status"], values={"open", "shipped", "cancelled"}), ], operators=[ Filter(AnyFailed(), to_table="orders_rejected", include_quality_columns="criteria"), Summary(), ], ), ], ) def clean_orders(raw_orders: TableFrameSpec) -> TableFrameSpec: return raw_orders.unique() ``` A `DataQuality` check has three parts: - **table**: the output Table the check runs on - **classifiers**: the rules that label each row, at least one - **operators**: what to do with the labels, at least one ## Classifiers A classifier evaluates one or more columns and writes a verdict for each row into a new verdict column. Classifiers fall into two kinds: **boolean classifiers** decide whether each value passes a rule, and **categorizers** assign each value to a bin. ### Boolean classifiers | Checks | Classifiers | | --- | --- | | Null and NaN | `IsNull`, `IsNotNull`, `IsNan`, `IsNotNan`, `IsNullOrNan`, `IsNotNullNorNan` | | Sign of a number | `IsPositive`, `IsNegative`, `IsZero`, `IsNotZero`, `IsPositiveOrZero`, `IsNegativeOrZero` | | Booleans | `IsTrue`, `IsFalse` | | Ranges | `IsBetween`, `IsNotBetween` | | Sets of values | `IsIn`, `IsNotIn` | | Text | `Matches`, `DoesNotMatch`, `HasLength` | ```python from tabsdatak.dataquality import HasLength, IsBetween, IsIn, IsNotNull, Matches IsNotNull(["order_id", "customer_id"]) IsBetween(["amount"], min_val=0, max_val=10000, closed_on="both") IsIn(["status"], values={"open", "shipped", "cancelled"}) Matches(["email"], pattern=r"^[^@]+@[^@]+$") HasLength(["country_code"], min_len=2, max_len=2) ``` Classifiers share these settings: - **column\_names**: the columns to classify. Each entry is a column name, or a `(column, verdict_column)` tuple that names the verdict column. When omitted, the classifier applies to every user column. - **on\_missing\_column**: `"ignore"` (the default) skips a named column that is not in the Table, and `"fail"` fails the check. - **on\_wrong\_type**: `"ignore"` (the default) skips a column whose data type the classifier does not support, and `"fail"` fails the check. - **on\_wrong\_value**: `"ignore"` (the default) gives a null verdict for a value that cannot be evaluated, and `"fail"` fails the check. - **tags**: labels that scope operators to this classifier. - **prefix**: text added to the start of every verdict column name. Defaults to `@td.dq.`, and `""` adds no prefix. ### Categorizers `ScaleCategorizer` assigns each numeric value to a bin of a scale. The verdict column holds the bin number, or a reserved value for a null, a NaN, or a value below or above the scale. | Scale | Bin edges | | --- | --- | | `LinearScale` | Equal-width bins | | `LogarithmicScale` | Spaced logarithmically | | `ExponentialScale` | Spaced exponentially | | `MonomialScale` | Bin widths that grow as a power law | | `IdentityScale` | One bin per integer | ```python from tabsdatak.dataquality import InBins, LinearScale, ScaleCategorizer, Select # splits amount into five equal-width bins between 0 and 10000 ScaleCategorizer([("amount", "amount_bin")], scale=LinearScale(0.0, 10000.0, bins=5), prefix="") # copies orders in the top bin, or above the scale, to large_orders Select(InBins(column_name="amount_bin", bins={5, "overflow"}), to_table="large_orders") ``` Bins are numbered from 1. A scale created with `use_bin_zero=True` gives its minimum value a bin 0 of its own. ## Criteria Criteria select the rows that the `Filter`, `Select`, and `Fail` operators act on. | Criteria | Selects rows where | | --- | --- | | `AllOk()` | Every boolean classifier passes | | `AnyFailed()` | At least one boolean classifier fails | | `InBins(column_name, bins)` | A categorizer's bin is one of `bins` | | `NotInBins(column_name, bins)` | A categorizer's bin is not one of `bins` | `AllOk` and `AnyFailed` treat a null or NaN verdict as a failure, unless `none_is_ok=True` or `nan_is_ok=True` is set. `InBins` and `NotInBins` accept bin numbers from 0 to 100 and the special bins `"none"`, `"nan"`, `"underflow"`, and `"overflow"`. ## Operators | Operator | Effect | | --- | --- | | `Enrich()` | Adds each classifier's verdict column to the Table, or to `to_table`. No rows are removed. | | `Summary()` | Writes a summary with one row per classifier, holding its verdict counts or bin counts, to a Table named `_dq_summary`, or to `table`. | | `Filter(criteria)` | Removes the selected rows. They are discarded, or written to `to_table`. | | `Select(criteria, to_table)` | Copies the selected rows to `to_table`, leaving the Table unchanged. | | `Fail(criteria, threshold)` | Fails the Transaction when the number of selected rows reaches the threshold. | `Filter` and `Select` take `include_quality_columns` to choose which verdict columns are carried into `to_table`: `"none"` (the default), `"criteria"`, or `"all"`. The checked Table never carries them. ### Failing a Transaction `Fail` takes a threshold of rows, as a count or as a percentage of the Table: ```python from tabsdatak.dataquality import AnyFailed, Fail, PercentThreshold, RowCountThreshold # fails as soon as one row fails a rule Fail(criteria=AnyFailed(), threshold=RowCountThreshold(1)) # fails when a twentieth of the table fails a rule Fail(criteria=AnyFailed(), threshold=PercentThreshold(5)) ``` A failed Transaction commits none of its Tables. ## Tags Tags scope an operator to some of a check's classifiers. An operator with tags applies only to the classifiers that carry a matching tag, and an operator without tags applies to all of them. ```python from tabsdatak.dataquality import ( AnyFailed, DataQuality, Fail, Filter, IsNotNull, Matches, RowCountThreshold, ) DataQuality( table="customers", classifiers=[ IsNotNull(["customer_id"], tags=["critical"]), Matches(["email"], pattern=r"^[^@]+@[^@]+$", tags=["cleanup"]), ], operators=[ # fails the transaction on a missing key Fail(criteria=AnyFailed(tags=["critical"]), threshold=RowCountThreshold(1)), # moves rows with a malformed email aside without failing Filter(AnyFailed(tags=["cleanup"]), to_table="customers_bad_email"), ], ) ``` --- # Data Quality Trends > How Summary(trends=Trends()) keeps a history of data quality checks in a trend Table, and how the Tabsdata UI charts it. Source: https://docs.tabsdata.com/guide/data-quality-trends A `Summary` records how a Table's data quality looked in one Transaction. **Trends** keep that history: every time the check runs, its summary is folded into a trend Table that holds the recent runs one by one, plus weekly and monthly totals that reach further back. Reading the trend Table shows whether a rule fails more or less often over time, without keeping every version of the summary. Trends are off by default. Turn them on by passing `Trends()` to the `trends` argument of `Summary`: ```python from tabsdatak.api import transformer, TableFrameSpec from tabsdatak.dataquality import ( DataQuality, IsBetween, IsIn, IsNotNull, Summary, Trends, ) @transformer( input_tables=["raw_orders"], output_tables=["orders"], on_tables=[ DataQuality( table="orders", classifiers=[ IsNotNull(["order_id"]), IsBetween(["amount"], min_val=0, max_val=10000), IsIn(["status"], values={"open", "shipped", "cancelled"}), ], operators=[Summary(trends=Trends())], ), ], ) def clean_orders(raw_orders: TableFrameSpec) -> TableFrameSpec: return raw_orders.unique() ``` Each run of `clean_orders` writes `orders_dq_summary` as before, and Tabsdata then folds that summary into `orders_dq_summary_trends`. The classifiers run while the data is written, in the same execution of the Function. Tabsdata does not read the Table again afterwards to compute the summary or the trend, so there is no separate quality job and no second pass over the data. The summary and trend Tables are Tabsdata Tables, versioned like any other. Every execution creates a new version of both, so the summary and the trend of any past execution can be read as they were at that point. See [Data Quality](https://docs.tabsdata.com/guide/data-quality) for the classifiers and operators a check is built from. ## Data quality in the Tabsdata UI The Tabsdata UI recognizes data quality Tables and shows them without any setup. A Table that has data quality checks is marked with a **Q** next to its name, in the Table list and in the execution plan graph. [![Data quality summary of the leads table in the execution plan graph](https://docs.tabsdata.com/assets/images/dq-summary-d7d7d79eb3b60900f76a1d6648dc1d21.jpg)](https://docs.tabsdata.com/assets/images/dq-summary-d7d7d79eb3b60900f76a1d6648dc1d21.jpg) _The data quality summary of the `leads` Table for one execution, opened from the execution plan graph: 100 rows checked, 10 checks, grouped by column. Each check shows its pass rate and how many rows failed or were null, and the lowest pass rate is called out at the top._ [![Pass rate by classifier for the leads table, run by run](https://docs.tabsdata.com/assets/images/dq-trends-5c9d6994b2a361d85f29a0616b39eeed.jpg)](https://docs.tabsdata.com/assets/images/dq-trends-5c9d6994b2a361d85f29a0616b39eeed.jpg) _The **Data Quality Trends** tab of the `leads` Table, with its trend chart expanded: the pass rate of every classifier, execution by execution. Hovering over an execution shows all the pass rates at that point._ A Table's data quality shows in two places: - **The Table's page.** A Table with a summary has a **Data Quality Summaries** tab, and a Table with a trend has a **Data Quality Trends** tab. Each tab lists the Table's summary or trend Tables; select one to chart it. - **The execution plan graph.** Right-click a Table and open **Data Quality** > **Data Quality Summary** or **Data Quality Trend** to see the summary or trend as it was written by that execution. The trend chart, titled **Pass rate by classifier**, plots `pass_pct` for every classifier: - **Runs**, **Weeks**, and **Months** switch between the `run`, `week`, and `month` rows of the trend Table. Buckets are in UTC, and weeks start on Monday. - The legend button shows or hides individual classifiers, and the expand button opens a larger chart. - On the **Data Quality Trends** tab, a trend Table can be pinned, so that its chart stays on the board while other trend Tables are selected. ## Trends `Trends` configures the trend Table: its name and how long each tier keeps its rows. ```python from tabsdatak.dataquality import Summary, Trends Summary(trends=Trends()) Summary(trends=Trends(table="orders_quality_history", days=14, weeks=26, months=36)) Summary(trends=Trends(days=7, weeks=13, months=0)) ``` The first keeps the default retention, the second names the Table and keeps longer history, and the third keeps no month tier. ### Parameters #### `table` The Table the trend is written to, as a valid table name. Defaults to `None`, which writes to `_dq_summary_trends`, where `` is the Table the `DataQuality` check runs on. The default name does not follow `Summary(table=...)`: a summary written to `orders_quality` still gets its trend in `orders_dq_summary_trends` unless `table` is set. A name given here is used as is, with no suffix added. The trend Table is created in the same collection as the Function. #### `days` How many days a run is kept as its own row, as a non-negative integer. Defaults to `7`. A run counts towards its week and month from its first execution, so expiring its run row does not remove it from the coarser tiers. `0` keeps no run rows. #### `weeks` How many weeks a week row is kept, counted from the Monday that starts it, as a non-negative integer. Defaults to `12`, which keeps the current week and the 11 before it. `0` keeps no week rows. #### `months` How many months a month row is kept, counted from the 1st of the month, as a non-negative integer. Defaults to `24`, which keeps the current month and the 23 before it. `0` keeps no month rows. At least one of `days`, `weeks`, and `months` must be greater than zero. To keep no trend, leave `trends` unset. ## How the trend Table is organized ### Tiers The trend Table holds four kinds of rows, told apart by the `grain` column: | `grain` | One row per | `bucket` holds | | --- | --- | --- | | `run` | classifier and run | The time of the run | | `week` | classifier and week | Monday 00:00 UTC of the week | | `month` | classifier and month | The 1st of the month, 00:00 UTC | | `folded` | Transaction already counted | The time of the run | The tiers overlap. A run is added to its week and its month as soon as it happens, not when its run row expires, so the current week row already includes the runs that are still listed one by one, and a month row includes every run of that month. Each tier expires on its own: a week that falls out of the `weeks` window is still counted in its month row. Weeks and months are bucketed independently, so a week that crosses into a new month is one week row while its runs count towards two month rows. Adding week rows together does not give a month row. ### Retention Retention is measured against the newest run folded into the Table, not against the wall clock. With the defaults, after a run on a given day the Table holds: - a run row for each run in the 7 days before it - a week row for that week and each of the 11 weeks before it - a month row for that month and each of the 23 months before it Rows only expire when the trend is updated, so a Table whose check stops running keeps its last rows as they were. ### Folded rows and recovered Transactions Each `folded` row records one Transaction whose results the week and month rows already include. It carries the Transaction ID and the time of the run, and no counts. When a summary arrives for a Transaction that has a `folded` row, for example because the Transaction was recovered, it is not counted a second time. Summaries that arrive out of order are still counted once each. A `folded` row is kept for as long as the week or month tier could still be holding that run. When both `weeks` and `months` are `0`, no `folded` rows are kept. ## Trend Table schema | Column | Type | Description | | --- | --- | --- | | `grain` | string | `run`, `week`, `month`, or `folded`. | | `bucket` | datetime (UTC) | The run time for `run` and `folded` rows, the Monday that starts the week for `week` rows, and the 1st of the month for `month` rows. | | `classifier` | string | The verdict column the counts belong to, as in the summary, for example `@td.dq.order_id_is_not_null` or the name given in a `(column, verdict_column)` tuple with its prefix. Null on `folded` rows. | | `runs` | int64 | `1` on a `run` row; the number of runs in the bucket on `week` and `month` rows. Null on `folded` rows. | | `passed` | int64 | Rows that passed the classifier. | | `failed` | int64 | Rows that failed the classifier. | | `nulls` | int64 | Rows with a null verdict. | | `nans` | int64 | Rows with a NaN verdict. | | `pass_pct` | float64 | `passed * 100 / (passed + failed)`. Null and NaN verdicts are left out. Null when `passed + failed` is `0`, and on `folded` rows. | | `txn` | string | The Transaction ID on `run` and `folded` rows. Null on `week` and `month` rows. | | `folded_through` | datetime (UTC) | The time of the newest run folded so far. Every row of a version holds the same value. | On `week` and `month` rows, `passed`, `failed`, `nulls`, and `nans` are summed over the runs in the bucket, so `pass_pct` is the share of passing rows across the whole bucket, not an average of the runs' percentages. Like every Tabsdata Table, the trend Table also carries the `$td.` system columns. Only boolean classifiers are tracked. `ScaleCategorizer` results appear in the summary but not in the trend Table, and a summary with only categorizers adds no rows to its trend Table. Tags on `Summary` apply as usual: the trend tracks the classifiers the summary includes. ## How the trend Table is maintained Registering a Function that declares `Summary(trends=Trends(...))` also registers a second Function, the trend Function. The trend Function: - is named after the trend Table with `_trend` appended, for example `orders_dq_summary_trends_trend` - reads the latest summary and the latest version of the trend Table, and writes a new version of the trend Table - runs in the same execution as the parent Function, after the summary is written - is registered even in a collection that does not otherwise accept transformers Each run writes a new version of the whole trend Table: the previous version is read back, the new summary is merged in, and the rows that fell out of their tier's window are dropped. Earlier versions of the trend Table stay available like any other Table's. The trend Table is linked to the Table the check runs on, not to the summary, and its data versions report that Table as their parent. ## Lifecycle The trend Function belongs to the Function that declares the `Summary`, its parent, and follows it: - **Updating** the parent Function with the same declaration keeps the same trend Function. Changing `days`, `weeks`, or `months` keeps its name. - **Renaming** the trend Table, through `table`, replaces the trend Function with one named after the new Table. - **Adding** `trends` to an existing `Summary` registers a trend Function on the next update. - **Removing** `trends`, or the whole `DataQuality` check, deletes the trend Function. The trend Table is kept, frozen, with its history. - **Deleting** the parent Function deletes the trend Function, and leaves their Tables frozen. Registering the parent Function again with `tdk fn register --reuse-tables` takes the frozen Tables back, history included. - **Disabling** or **enabling** the parent Function does the same to the trend Function. - **Moving** the check to another Table of the same Function keeps the summary and trend Tables and links them to the new Table. - **No longer producing** the Table the check runs on freezes that Table, its summary, and its trend Table. Deleting the frozen Table also deletes the summary and trend Tables derived from it. The trend Function cannot be managed directly. Updating, patching, deleting, enabling, disabling, or triggering it by name is refused with: ```text Function '' is a child function managed by its parent; it cannot be updated, patched, deleted, enabled or triggered directly ``` Make the change on the parent Function instead. ## Examples ### Tracking a source report run by run A sales operations team publishes the Salesforce _Lead Lifetime_ report once a week and wants to know which data entry problems are getting better. The publisher checks every run and keeps a trend: ```python from tabsdatak.api import TableFrameSpec, publisher from tabsdatak.conn.salesforce import SalesforceReportSrc from tabsdatak.dataquality import ( DataQuality, HasLength, IsBetween, IsIn, IsNotIn, IsNotNull, IsPositive, Matches, Summary, Trends, ) @publisher( source=SalesforceReportSrc(reports=["Lead Lifetime Report"], find_report_by="name"), output_tables=["leads"], on_tables=[ DataQuality( table="leads", classifiers=[ IsNotNull([("EMAIL", "email_present")]), Matches([("EMAIL", "email_valid")], pattern=r"^[^@]+@[^@]+\.[A-Za-z]{2,}$"), IsNotNull([("LEAD_SOURCE", "lead_source_present")]), IsNotIn([("COMPANY", "company_provided")], values={"[not provided]", "N/A", "Unknown"}), IsIn([("RATING", "rating_valid")], values={"Hot", "Warm", "Cold"}), IsBetween([("EMPLOYEES", "employees_in_range")], min_val=1, max_val=10_000), HasLength([("LAST_NAME", "last_name_valid")], min_len=2, max_len=80), IsPositive([("ANNUAL_REVENUE", "revenue_positive")]), ], operators=[Summary(trends=Trends(days=35))], ) ], ) def leads_report(leads: TableFrameSpec) -> TableFrameSpec: return leads ``` The report runs once a week, so with the default `days=7` only the latest run would keep a row of its own. `days=35` keeps the last five runs individually. The trend lands in `leads_dq_summary_trends`, with one classifier per verdict column: `@td.dq.email_present`, `@td.dq.email_valid`, and so on. After a few weeks, the trend separates the problems that are being fixed, such as missing emails once the web form makes the field required, from those that hold steady and those that come and go with list imports. For example, the month row for August 2026 might read: | grain | bucket | classifier | runs | passed | failed | pass\_pct | | --- | --- | --- | --- | --- | --- | --- | | month | 2026-08-01 00:00:00 UTC | @td.dq.email\_present | 4 | 362 | 38 | 90.5 | ### Inspecting the trend Table Use `tdk table data` with `--sql` to look at one tier: ```bash tdk table data --coll sales --name leads_dq_summary_trends --sql "SELECT bucket, classifier, runs, failed, pass_pct FROM leads_dq_summary_trends WHERE grain = 'week' ORDER BY classifier, bucket" --print ``` `folded_through` shows which run the trend has caught up to. The trend Function commits in the same Transaction as the summary, so a committed summary version always has its trend version next to it. ### Reading trends downstream A transformer can take the trend Table as an input like any other Table, for example to publish a weekly quality scorecard for a dashboard: ```python from tabsdatak.api import TableFrameSpec, transformer from tabsdatak.tableframe.functions import col @transformer( input_tables=["sales/leads_dq_summary_trends"], output_tables=["leads_weekly_quality"], ) def leads_weekly_quality(trends: TableFrameSpec) -> TableFrameSpec: return trends.filter(col("grain") == "week").select( "bucket", "classifier", "runs", "passed", "failed", "pass_pct" ) ``` Each new version of the trend Table triggers the transformer, so the scorecard is refreshed after every run of the report. ### Keeping only recent history A pipeline that runs every few minutes produces many run rows. Keeping a day of runs and a quarter of weeks, with no month tier, bounds the Table's size: ```python from tabsdatak.dataquality import Summary, Trends Summary(trends=Trends(days=1, weeks=13, months=0)) ``` ## Limitations and validation - Trends are only maintained for boolean classifiers. Categorizer results stay in the summary Table. - Retention values are whole numbers of days, weeks, or months, `0` or greater. A negative value, a float, a string, or a boolean is refused with error `DQC_32`, as is an invalid `table` name. - Setting `days`, `weeks`, and `months` all to `0` is refused with error `DQC_33`. Leave `trends` unset to keep no trend. - Every Table a check produces must have a unique name, and that includes trend Tables. Two `Summary` operators with trends on the same check need different `table` names for their trends, or registration fails with error `DQPE_4`. - Week and month buckets are always in UTC; weeks start on Monday and months on the 1st. There is no option to bucket in another time zone. - A tier that was disabled and is later enabled starts from the runs that follow; it is not filled in from older run rows. - The trend Function is managed through the Function that declares the `Summary`. It cannot be triggered, updated, or deleted on its own. --- # Lineage > How Tabsdata records where each table version came from and how to view that lineage in the Tabsdata UI. Source: https://docs.tabsdata.com/guide/lineage Any number of factors can evolve over the course of a data pipeline's lifecycle, including the shape of processed data, the logic used to move and transform that data, and when specific data was scheduled to be processed. This is called **Data Lineage**. Tabsdata's built-in data lineage automatically stores immutable versions of tables and functions as they are modified, preserving a record of how your workflows change over time. This allows users to travel back to historical versions of table data, examine the code that produced them, and see how their workflow logic has evolved. _Video: In the Tabsdata UI, the trigger time of the newest execution plan is highlighted and the plan is opened from the Executions list, the pub_customers function on its graph is right-clicked to show its code, and the customers table is right-clicked and Go to sample opens the version the plan wrote._ ## How Lineage Is Recorded All Tabsdata functions declare the Tables they read from and write to. This allows Tabsdata to infer dependencies and execution order, a core mechanic that enables Automatic Orchestration and Execution Plans. Because dependencies are declared ahead of time, Tabsdata automatically links each table version to the function and function code that produced it. These versions are then grouped into an execution plan, along with the other tables and functions affected by the execution. ## Table History Each time a Function run successfully returns data, Tabsdata creates a new, immutable table version while preserving all previous versions of the table's data and schema. _Video: In the Tabsdata UI, two older versions of the customers table are selected in turn from the Collections panel, and the Sample tab shows the email and region values that changed between them._ Functions can also read previous Table versions. An input written as `customers@HEAD^` reads the version before the latest, and `customers@HEAD~2..HEAD` reads a range of versions ending at the latest. Uses the latest version of customers as input. ## Schema History Each Table version stores its own schema, so a Table's version history records changes to its schema as well as its data. Comparing the schema of one Table version with another shows which columns were added, removed, or changed type. _Video: In the Tabsdata UI, the customers table version before a schema change is opened, then the table version marked New data with schema changes, and the Schema tab shows region added, signup_date changed from str to date, and fax removed._ ## Function History Functions have their own version history. Whenever a Function is updated, Tabsdata creates a new Function version, retaining a changelog of all previous versions of the function's code. _Video: In the Tabsdata UI, the two older versions of the pub_customers function are opened in turn from the Collections panel, and the Snippet tab shows the lines of its query that changed between versions._ ## Viewing Lineage In the Tabsdata [User Interface](https://docs.tabsdata.com/guide/user-interface), lineage can be viewed from Tables, Execution Plans, and Functions. ### Tables Opening a Table shows three tabs: **Schema**, **Lineage**, and **Sample**. The Lineage tab shows a graph of the Execution Plan that wrote the selected table version. This execution plan is unique to the specific table version, and exploring the lineage of other versions can show the evolution of a particular execution plan. orders\_clean Tabsdata›Collections›sales›orders\_clean ### Schema Table schema defines the structure of a table by listing its columns and their data types. | Object | Type | | --- | --- | | order\_id | i64 | | customer\_id | i64 | | amount | f64 | | ordered\_at | datetime | --- # Living Semantic Catalog > How Tabsdata builds a catalog of what every connected source and destination holds, keeps it current as those systems change, and how agents use it. Source: https://docs.tabsdata.com/guide/living-semantic-catalog The **living semantic catalog** is the record the Tabsdata Server keeps of what every connected source and destination holds and how that data can be used. Each source and destination collection has an entry in it: the entities its connection exposes, their fields, and an AI-generated classification of the data. Tabsdata builds the catalog on its own. Soon after a source or destination collection is created, the server connects to the external system, fetches its metadata, and classifies what it found. The catalog is living because it follows the external systems: when their metadata changes, the connection is classified again, so nobody curates the catalog by hand and it reflects what the systems contain as of the last refresh. Agents connected to Tabsdata read the catalog when a user asks them to publish from or subscribe to an external system, and use it to tell the user what data is there and how to use it. _Video: A source collection named crm is created from the CLI with a PostgreSQL connection document, the server probes that connection, reads five tables into the living semantic catalog with a schema fingerprint, and the classifier labels the data as confidential finance data. The next metadata poll finds a new refunds table, the fingerprint changes, and the connection is classified again in full._ ## How the Catalog Is Built Creating a source or destination collection with its connection starts the process. Nothing else is needed from the user. 1. Soon after the collection is created, the server runs a probe against the external system with the collection's connection. The probe first checks that the system can be reached with the credentials the connection holds. 2. The probe then reads the system's own metadata: the entities it holds, such as tables, reports, files, objects, or topics, and the fields of each entity. Tabsdata stores the entities with their names and fields, and a fingerprint of the whole schema. Data is not copied into the catalog. 3. The server's classifier sends the entity list and the collection's description to the configured AI model, together with a few sample rows when the server allows sampling (`catalog.maintenance_samples`). The model's answer is recorded as the connection's [classification](#classification). 4. The entities and their classification become the connection's entry in the catalog, searchable by keyword and by meaning. The probe repeats on a schedule, and the classification is redone whenever the metadata the probe finds has changed. See [Keeping It Current](#keeping-it-current). ## What Is Fetched From Each System The probe reads the metadata the external system already keeps about itself, through the same connector that later moves the data. Each entity is stored with its fully qualified name, its kind, and its fields, plus details that depend on the system. | System | Entities | Recorded for each entity | | --- | --- | --- | | SQL databases: Postgres, MySQL, MariaDB, SQL Server, Oracle, Db2 | tables and views, by database and schema | columns with type, primary key, and column comments | | Databricks | tables, by Unity Catalog catalog and schema | columns with type and comments | | Salesforce | reports, by folder, with their report id; SOQL objects | report columns; object fields and whether they are required | | Object stores and files: Amazon S3, Azure Blob Storage, Google Cloud Storage, local files | buckets and prefixes | the file patterns found under each prefix, their format, and the columns read from them | | MongoDB | collections, by database | collection type (collection, view, time series), indexes, and validators | | Kafka | topics | partition count | ## Classification To classify a connection, the server gives the configured AI model a compact description of the whole connection and asks for a structured judgement of the data. The model receives every entity with its fields, grouped the way the system groups them (database and schema, folder, bucket, namespace), the collection's description, and any sample rows the classifier read. The result is one classification per connection: | Field | What it says | | --- | --- | | Label | a short name for the data, such as "Customer data" or "Market signals" | | Domain | the business domain: CRM, finance, marketing, engineering, operations, product, HR, supply chain, security, infrastructure, IoT, analytics, ecommerce | | Contains | what the data holds, one item per subject | | Use cases | what this data is typically used for | | Sensitivity | public, internal, confidential, or restricted | | Freshness | real-time, hourly, daily, weekly, monthly, or static | | Typical sources | where this kind of data usually comes from | | Search terms | the synonyms and business words people would search for it by | Names, comments, and sample values read from the external system are passed to the model as data, never as instructions, and sample values are never copied into the classification. Each classification records the fingerprint of the schema it was made from. Comparing that fingerprint with the current one tells Tabsdata when the classification is out of date. ## Keeping It Current Tabsdata re-reads each connection's metadata on a schedule and whenever a user or an agent asks for a refresh. The schedule is set by `catalog.refresh_interval` in the server configuration, which is 24 hours as shipped. On every read, the probe compares the fingerprint of the metadata it found with the stored fingerprint. If the fingerprints match, the catalog keeps its entities and its classification. If they differ, because of a new table, an altered table, a new report, or a new file pattern, the stored entities are replaced. Because the stored classification was made for the old schema, the connection is then classified again in full. Tabsdata never attempts a partial update: the model sees the whole connection every time, so the label, the subjects, and the search terms stay consistent with everything the connection now holds. A classification can also be requested by hand, for example after a collection's description was improved, and the connection is classified again on the next sweep. An existing classification can be discarded, and the next sweep then redoes it. ## How Agents Use the Catalog When a user asks an agent connected to Tabsdata to publish from or subscribe to an external system, the agent reads the catalog before it writes any code. To find data, the agent searches the catalog across every connection the user can see. A question like "Where do we keep invoices?" becomes a search by keyword and by meaning over collection names and descriptions, entity names, fields, and the classification's search terms, and the agent answers from the hits instead of guessing names. To describe one source or destination collection, the agent lists the collection's entities and their fields, and says what the classification found the data to be and how sensitive and how fresh it is. To propose a function, the agent works from the entities in the catalog. For a source, the agent proposes a publisher over the entities that matter, with the real column names and types. For a destination, the agent proposes a subscriber for the tables to deliver, written to the entities the destination exposes. See [MCP](https://docs.tabsdata.com/guide/mcp) to connect an agent to Tabsdata. ## Working With the Catalog in tdk The catalog is read and maintained with `tdk`. Every command takes `-P ` and names a source or destination collection, because each connection belongs to exactly one collection. | Task | Command | | --- | --- | | Show the connection's catalog status: last check, last refresh, entity count, and whether the entity list was truncated | `tdk connection get -n ` | | List the entities and, with `--fields`, their fields | `tdk connection entities -n [--fields] [--filter fqn:lk:]` | | Check that the external system can be reached | `tdk connection check -n ` | | Re-read the metadata now | `tdk connection refresh -n ` | | Show the classification and whether the schema changed since it was made | `tdk ai catalog show -n ` | | Classify the connection again on the next sweep | `tdk ai catalog classify -n ` | | Discard the classification so the next sweep redoes it | `tdk ai catalog unclassify -n ` | | Search every connection you can see, by keyword and by meaning | `tdk ai search connections --query ""` | Viewing a connection's catalog status and its classification requires permission to view the project. Listing its entities, checking it, refreshing it, reading sample rows, and requesting a classification require permission to develop in the project. Discarding a classification requires permission to manage the project. --- # Contact Us > Lists the Tabsdata contact channels: the company website, the community Slack workspace invite, and the GitHub repository. Source: https://docs.tabsdata.com/guide/operations/contact-support You can connect with us on: [Website](https://www.tabsdata.com/) [Slack](https://join.slack.com/t/tabsdata-community/shared_invite/zt-322toyigx-ZGFioMV2Gbza4bJDAR7wSQ) [Github](https://www.github.com/tabsdata/tabsdata/) --- # Publish data from Amazon S3 > Publish data from Amazon S3 covers the S3 source connection setup, the S3Src publisher with paths and globs, and format handling for mixed or extensionless objects. Source: https://docs.tabsdata.com/tutorials/publishers/amazon-s3 Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) ## Ingest objects from a bucket warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you publish trips.parquet out of amazon s3 Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ The S3 publisher guide and connection template are loaded. Four values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - bucket - region - AWS keys ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ Both keys found. Reading back what is not secret: bucket my-bucket, region us-east-1, collection s3\_src. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **s3\_src** created from it in group `sources` with `base_path` `/data`. Connection check passed. Called tabsdata 2 times (ctrl+o to expand) ⏺ The publisher, registered in `s3_src`: ```python @publisher( source=S3Src(paths=["trips.parquet"]), output_tables=["trips"], ) def read_trips(trips: TableFramesSpec) -> TableFrameSpec: return trips[0] ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ **s3\_src/trips** is live, holds 1,000,000 rows across 21 columns: ``` ┌──────────────────┬──────────────┐ │ field │ value │ ├──────────────────┼──────────────┤ │ collection │ s3_src │ ├──────────────────┼──────────────┤ │ table │ trips │ ├──────────────────┼──────────────┤ │ rows │ 1,000,000 │ ├──────────────────┼──────────────┤ │ columns │ 21 │ ├──────────────────┼──────────────┤ │ function version │ fv20260813-1 │ ├──────────────────┼──────────────┤ │ execution │ ep20260813-1 │ └──────────────────┴──────────────┘ ``` AI agent session ❯Can you publish departments.csv, employees.parquet and salaries.csv out of amazon s3, each into its own table Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ The S3 publisher guide and connection template are loaded. Four values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - bucket - region - AWS keys ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ Both keys found. Reading back what is not secret: bucket my-bucket, region us-east-1, collection s3\_src. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **s3\_src** created from it in group `sources` with `base_path` `/data`. Connection check passed. Called tabsdata 2 times (ctrl+o to expand) ⏺ The publisher, registered in `s3_src`: ```python @publisher( source=S3Src(paths=["departments.csv", "employees.parquet", "salaries.csv"]), output_tables=["departments", "employees", "salaries"], ) def read_hr( departments: TableFramesSpec, employees: TableFramesSpec, salaries: TableFramesSpec, ) -> tuple[TableFrameSpec, TableFrameSpec, TableFrameSpec]: return (departments[0], employees[0], salaries[0]) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ Three tables are live in **s3\_src**, one per path: ``` ┌─────────────┬───────────────────┬───────┬─────────┐ │ table │ from │ rows │ columns │ ├─────────────┼───────────────────┼───────┼─────────┤ │ departments │ departments.csv │ 14 │ 3 │ ├─────────────┼───────────────────┼───────┼─────────┤ │ employees │ employees.parquet │ 1,204 │ 11 │ ├─────────────┼───────────────────┼───────┼─────────┤ │ salaries │ salaries.csv │ 1,204 │ 5 │ └─────────────┴───────────────────┴───────┴─────────┘ ``` AI agent session ❯Can you publish every monthly trips file out of amazon s3 into one table Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ The S3 publisher guide and connection template are loaded. Four values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - bucket - region - AWS keys ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ Both keys found. Reading back what is not secret: bucket my-bucket, region us-east-1, collection s3\_src. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **s3\_src** created from it in group `sources` with `base_path` `/data`. Connection check passed. `trips_*.parquet` matches 3 objects under `/data`: - trips\_2026\_01.parquet - trips\_2026\_02.parquet - trips\_2026\_03.parquet Called tabsdata 2 times (ctrl+o to expand) ⏺ The publisher, registered in `s3_src`: ```python @publisher( source=S3Src(paths=["trips_*.parquet"]), output_tables=["trips"], ) def read_trips(trips: TableFramesSpec) -> TableFrameSpec: return concat(trips) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ **s3\_src/trips** is live, one table holding the 3 files the glob matched: ``` ┌───────────────────────┬───────────┐ │ matched file │ rows │ ├───────────────────────┼───────────┤ │ trips_2026_01.parquet │ 412,900 │ ├───────────────────────┼───────────┤ │ trips_2026_02.parquet │ 388,140 │ ├───────────────────────┼───────────┤ │ trips_2026_03.parquet │ 441,022 │ ├───────────────────────┼───────────┤ │ s3_src/trips │ 1,242,062 │ └───────────────────────┴───────────┘ ``` note Files of different formats in one `paths` list need no extra configuration. `AUTO` reads each file's extension separately, so a `.csv` and a `.parquet` in the same publisher both land correctly. The case that does need work is objects whose extensions do not say what they are. `format` is one override for the whole source rather than one per path, so setting it reads every path as that format. Files that genuinely differ and cannot be recognised by name have to be split across publishers, one per format. --- # Publish data from Azure Blob Storage > Publishes CSV blobs from an Azure Blob Storage container into Tabsdata tables using AzureBlobSrc, covering paths and globs, base_path, file_info="uri", and initial_last_modified. Source: https://docs.tabsdata.com/tutorials/publishers/azure Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) ## Ingest blobs from a container Each entry in `paths` fills one input slot and produces one output table. An exact blob name fills its slot with a single frame; a glob fills it with one frame per matched blob, which the function folds down to one. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you publish departments.csv and all monthly employee files out of azure blob storage Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ The Azure Blob publisher guide and connection template are loaded. Four values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - container - account name and account key - blob names or patterns ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ Account key found. Reading back what is not secret: container acme-ingest, account acmeingeststorage, collection hr\_src, base path /hr. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **hr\_src** created from it in group `sources` with `base_path` `/hr`. Connection check passed. `employees_*.csv` matches 3 blobs under `/hr`: - employees\_2026\_01.csv - employees\_2026\_02.csv - employees\_2026\_03.csv Called tabsdata 2 times (ctrl+o to expand) ⏺ The publisher, registered in `hr_src`. The exact name returns its one frame, and the glob folds its three with `concat()`: ```python @publisher( source=AzureBlobSrc( paths=["departments.csv", "employees_*.csv"], file_info="uri", ), output_tables=["departments", "employees"], ) def read_hr( departments: TableFramesSpec, employees: TableFramesSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return (departments[0], concat(employees)) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ Two tables are live in **hr\_src**. `file_info="uri"` added a column naming the blob each row came from, which is how the three months stay distinguishable after the concat: ``` ┌─────────────┬──────────────────────────────────┬───────┬─────────┐ │ table │ from │ rows │ columns │ ├─────────────┼──────────────────────────────────┼───────┼─────────┤ │ departments │ departments.csv │ 14 │ 4 │ ├─────────────┼──────────────────────────────────┼───────┼─────────┤ │ employees │ 3 blobs matching employees_*.csv │ 3,612 │ 12 │ └─────────────┴──────────────────────────────────┴───────┴─────────┘ ``` Paths resolve against the connection's `base_path`, so the publisher names blobs and never the container. `initial_last_modified` limits a run to blobs modified after a timestamp, and the cutoff advances on every later run. --- # Publish data from Databricks > Publishes Databricks SQL query results to Tabsdata tables using the DatabricksSrc connector, with {placeholder} substitution via initial_values, staging volumes, and warehouse or warehouse_id options. Source: https://docs.tabsdata.com/tutorials/publishers/databricks Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) ## Ingest query results from Databricks Each entry in `queries` produces one output table. A query can carry `{placeholder}` names that `initial_values` fills in, which is how one publisher reads the same tables out of a different catalog or schema without the SQL changing. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you publish the country and city tables out of databricks Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ The Databricks publisher guide and connection template are loaded. Five values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - workspace host URL and access token - staging volume - warehouse - tables to read ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ Token found, and the host URL is a secret field too, so both are held out of the stored document. Reading back what is not secret: volume main.default.tabsdata\_stage, warehouse analytics\_wh, catalog main, schema default, collection world\_src. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **world\_src** created from it in group `sources`, reading through the analytics\_wh warehouse and staging in main.default.tabsdata\_stage. Connection check passed. Called tabsdata 2 times (ctrl+o to expand) ⏺ The publisher, registered in `world_src`. Catalog and schema are placeholders rather than literals, so the same code runs against another catalog by changing one dict: ```python @publisher( source=DatabricksSrc( queries=[ "SELECT * FROM {catalog}.{schema}.country", "SELECT * FROM {catalog}.{schema}.city", ], initial_values={"catalog": "main", "schema": "default"}, ), output_tables=["country", "city"], ) def read_world( country: TableFramesSpec, city: TableFramesSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return (country[0], city[0]) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. The read went through the warehouse and landed in the staging volume before Tabsdata picked it up. ⏺ Two tables are live in **world\_src**, one per query, in the order the queries were listed: ``` ┌─────────┬──────────────────────┬───────┬─────────┐ │ table │ from │ rows │ columns │ ├─────────┼──────────────────────┼───────┼─────────┤ │ country │ main.default.country │ 239 │ 15 │ ├─────────┼──────────────────────┼───────┼─────────┤ │ city │ main.default.city │ 4,079 │ 5 │ └─────────┴──────────────────────┴───────┴─────────┘ ``` `warehouse` and `warehouse_id` name the same thing two ways and are mutually exclusive, so the connection carries one or the other. The staging volume has to be a managed Databricks volume the token can write to, since the connector unloads through it rather than streaming rows. --- # Publish data from Db2 > Publishes query results from IBM Db2 into Tabsdata tables through the Db2Src source and db2+ibm_db connection URI, one output table per query in `queries`. Source: https://docs.tabsdata.com/tutorials/publishers/db2 Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) ## Ingest query results from Db2 Each entry in `queries` produces one output table. The URI uses the `db2+ibm_db` scheme, which is the driver the connector reads through. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you publish the COUNTRY and CITY tables out of db2 Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ The Db2 publisher guide and connection template are loaded. Four values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - connection URI - user and password - tables to read ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ Password found. Reading back what is not secret: uri db2+ibm\_db://db.internal:50000/world, collection world\_src, tables COUNTRY and CITY. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **world\_src** created from it in group `sources`. Connection check passed. Called tabsdata 2 times (ctrl+o to expand) ⏺ The publisher, registered in `world_src`: ```python @publisher( source=Db2Src( queries=[ "SELECT * FROM COUNTRY", "SELECT * FROM CITY", ], ), output_tables=["country", "city"], ) def read_world( country: TableFramesSpec, city: TableFramesSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return (country[0], city[0]) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ Two tables are live in **world\_src**, one per query, in the order the queries were listed: ``` ┌─────────┬───────────────────────┬───────┬─────────┐ │ table │ from │ rows │ columns │ ├─────────┼───────────────────────┼───────┼─────────┤ │ country │ SELECT * FROM COUNTRY │ 239 │ 15 │ ├─────────┼───────────────────────┼───────┼─────────┤ │ city │ SELECT * FROM CITY │ 4,079 │ 5 │ └─────────┴───────────────────────┴───────┴─────────┘ ``` For change data capture rather than a full read on every run, see [Publish data from Db2 CDC](https://docs.tabsdata.com/tutorials/publishers/db2-cdc), which uses a stream publisher instead. --- # Publish data from Db2 CDC > Publishes Db2 change data capture into Tabsdata with @stream_publisher and Db2CdcStreamSrc, covering tables, start_from options (tail, full, head, explicit position), batch_max_secs, and output_tables. Source: https://docs.tabsdata.com/tutorials/publishers/db2-cdc Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) ## Follow the change log instead of re-reading tables A CDC source uses `@stream_publisher` rather than `@publisher`. A long-lived worker follows the Db2 change log, and each time a batch is due it hands the changes captured since the previous batch to the publisher, which commits them. One output table per tracked table, so `tables` and `output_tables` line up position by position. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you publish changes to the ORDERS and CUSTOMERS tables out of db2 Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ That is change data capture rather than a query read, so this is a Db2 CDC stream publisher. Its guide and connection template are loaded. Five values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - connection URI - user and password - tables to track, fully qualified - where to start reading, and how often to hand over a batch ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ Password found. Reading back what is not secret: uri db2+ibm\_db://db.internal:50000/sales, tracking SALES.ORDERS and SALES.CUSTOMERS, starting from tail, collection sales\_cdc. Called tabsdata 4 times (ctrl+o to expand) ⏺ Collection **sales\_cdc** created in group `sources`, with a `Db2CdcStreamSrcConn` to db.internal:50000/sales. Connection check passed, and both tables have change capture enabled. Called tabsdata 2 times (ctrl+o to expand) ⏺ The stream publisher, registered in `sales_cdc`. `start_from="tail"` means the worker begins at the current log position and never backfills, so the first version holds only changes committed after registration: ```python @stream_publisher( source=Db2CdcStreamSrc( tables=["SALES.ORDERS", "SALES.CUSTOMERS"], start_from="tail", ), batch_max_secs=120, output_tables=["orders_changes", "customers_changes"], ) def read_sales_changes( orders: TableFramesSpec, customers: TableFramesSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return (concat(orders), concat(customers)) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Stream worker started. First batch committed within 2 minutes, 0 failures. ⏺ Two tables are live in **sales\_cdc**, and both gain a version per batch: ``` ┌───────────────────┬─────────────────┬───────────────────┐ │ table │ tracking │ rows this version │ ├───────────────────┼─────────────────┼───────────────────┤ │ orders_changes │ SALES.ORDERS │ 1,842 │ ├───────────────────┼─────────────────┼───────────────────┤ │ customers_changes │ SALES.CUSTOMERS │ 96 │ └───────────────────┴─────────────────┴───────────────────┘ ``` `start_from` also accepts `"full"` for a full load of the tracked tables followed by their changes, `"head"` to begin at the oldest retained log, or an explicit position. Rows carry the change operation, so a downstream transformer can apply inserts, updates and deletes rather than treating every row as new. A stream publisher takes no `trigger_by` and must not declare a `ctx` parameter. See [Db2 CDC](https://docs.tabsdata.com/guide/connectors/db2-cdc) for every option of the source. --- # Publish data from Google Cloud Storage > Publishes objects from a Google Cloud Storage bucket into Tabsdata tables using a GCS source connection with a service account JSON key, `paths` globs, and `initial_last_modified`. Source: https://docs.tabsdata.com/tutorials/publishers/gcp Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) ## Ingest objects from a bucket Each entry in `paths` fills one input slot and produces one output table. Credentials are a service account JSON key rather than an id and secret pair, so the whole key goes in as one value. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you publish departments.csv and the monthly employees files out of google cloud storage Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ The GCS publisher guide and connection template are loaded. Four values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - bucket and project - service account JSON key - object names or patterns ❯everything you need is in Read 2 files, called tabsdata 2 times (ctrl+o to expand) ⏺ Service account key read from ~/acme-sa.json and held as one secret value. Reading back what is not secret: bucket acme-ingest, project acme-analytics, collection hr\_src, base path /hr. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **hr\_src** created from it in group `sources` with `base_path` `/hr`. Connection check passed. `employees_*.csv` matches 3 objects under `/hr`: - employees\_2026\_01.csv - employees\_2026\_02.csv - employees\_2026\_03.csv Called tabsdata 2 times (ctrl+o to expand) ⏺ The publisher, registered in `hr_src`. The exact name returns its one frame, and the glob folds its three with `concat()`: ```python @publisher( source=GCSSrc( paths=["departments.csv", "employees_*.csv"], initial_last_modified="2026-01-01T00:00:00Z", ), output_tables=["departments", "employees"], ) def read_hr( departments: TableFramesSpec, employees: TableFramesSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return (departments[0], concat(employees)) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ Two tables are live in **hr\_src**. `initial_last_modified` was set to the start of the year, so this first run took only objects modified since then: ``` ┌─────────────┬────────────────────────────────────┬───────┬─────────┐ │ table │ from │ rows │ columns │ ├─────────────┼────────────────────────────────────┼───────┼─────────┤ │ departments │ departments.csv │ 14 │ 3 │ ├─────────────┼────────────────────────────────────┼───────┼─────────┤ │ employees │ 3 objects matching employees_*.csv │ 3,612 │ 11 │ └─────────────┴────────────────────────────────────┴───────┴─────────┘ ``` The cutoff advances on its own after that first run, so each later trigger reads only what arrived since the previous one. Paths resolve against the connection's `base_path`, so the publisher names objects and never the bucket. --- # Publish data from log files with Grok > Parses single-column log text into typed TableFrame columns with the grok function, using Elasticsearch-style patterns, named captures, and a schema, with local file and object storage publishers. Source: https://docs.tabsdata.com/tutorials/publishers/grok-for-log-files Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) ## Ingest log files as typed columns The grok function is a powerful log parsing tool for TableFrames that extracts structured data from log files and similar text formats. Originally developed for log analysis, it transforms a single text column into multiple typed columns by applying grok patterns with named captures. Grok is based on the Elasticsearch implementation and follows the same pattern syntax. For comprehensive pattern documentation, see the official [Elasticsearch Grok documentation](https://www.elastic.co/docs/explore-analyze/scripting/grok) and the [predefined pattern library](https://github.com/elastic/elasticsearch/tree/main/libs/grok/src/main/resources/patterns). Grok uses regular expressions with predefined patterns to parse text. Each pattern includes named captures that become columns in your result. The function takes three inputs: - Column Expression: Any expression that selects one column containing text to parse - Pattern: A grok pattern string with named captures - Schema: A mapping that defines output column names and data types for each capture While not technically a full-fledged connector like S3, Postgres, or Databricks, the grok log parser can be used with Publishers for Local File Systems and Object Storage in order to parse and convert single column logs into structured, tabular data. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you publish access.log into a table using grok log parsing Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ That needs a file publisher plus a grok pattern to split the lines. Three values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - log directory and file - log format, or a sample line ❯everything you need is in Read 2 files, called tabsdata 2 times (ctrl+o to expand) ⏺ Read the first line of access.log to confirm the format before writing a pattern against it: ```text 192.168.1.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 ``` ⏺ Combined access log, as stated. No credentials here, so nothing is secret in this connection: a local file source carries only `base_path`. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **log\_src** created from it in group `sources` with `base_path` `/var/log/acme`. Connection check passed. Called tabsdata 3 times (ctrl+o to expand) ⏺ Checked the pattern with `grok_fields` before writing the schema, which returned 9 captures. Five of them are worth keeping and the rest are matched and discarded, so the schema maps only those five: ```python PATTERN = ( r"%{IPV4:client_ip} " r"%{USER:ident} " r"%{USER:auth} " r"\[%{HTTPDATE:timestamp}\] " r'"%{WORD:method} ' r'%{URIPATHPARAM:request} ' r'HTTP/%{NUMBER:http_version}" ' r"%{INT:response_code} %{INT:bytes}" ) SCHEMA = { "client_ip": tdf.Column("ip_address", td.String), "method": tdf.Column("http_method", td.String), "response_code": tdf.Column("status_code", td.Int32), "bytes": tdf.Column("response_bytes", td.Int64), "timestamp": tdf.Column("request_time", td.String), } @publisher( source=LocalFileSrc( paths=["access.log"], format=SrcFileFormat.CSV, format_cfg={ SrcFileFormat.CSV: {"has_header": False, "separator": "\x00"}, }, ), output_tables=["access_logs"], ) def read_access_logs(lines: TableFramesSpec) -> tuple[TableFrameSpec]: if lines is None: return (None,) return (lines[0].grok("column_1", PATTERN, SCHEMA),) ``` ⏺ The CSV options turn the header off and set a separator that cannot occur in the text, so each line reaches `grok` whole rather than being split on its commas first. Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ **log\_src/access\_logs** is live, 5 typed columns out of what was one text column. 12 rows did not match the pattern and carry null across all five: ``` ┌────────────────┬────────┬───────────────┐ │ column │ type │ from capture │ ├────────────────┼────────┼───────────────┤ │ ip_address │ String │ client_ip │ ├────────────────┼────────┼───────────────┤ │ http_method │ String │ method │ ├────────────────┼────────┼───────────────┤ │ status_code │ Int32 │ response_code │ ├────────────────┼────────┼───────────────┤ │ response_bytes │ Int64 │ bytes │ ├────────────────┼────────┼───────────────┤ │ request_time │ String │ timestamp │ └────────────────┴────────┴───────────────┘ ``` A row the pattern fails to match is kept with null in every grok column, so nothing is dropped and the run does not fail. A block of nulls across the parsed columns means the pattern stopped matching rather than that the source went empty. Reading logs out of a bucket rather than a local directory changes only the connection, to `s3-file-in` or `az-file-in`. The parsing lives in the publisher body either way. The pattern syntax follows the Elasticsearch implementation. See the [Elasticsearch Grok documentation](https://www.elastic.co/docs/explore-analyze/scripting/grok) for the reference and the [predefined pattern library](https://github.com/elastic/elasticsearch/tree/main/libs/grok/src/main/resources/patterns) for the names available to `%{NAME:alias}`. --- # Publish data from a Kafka topic > Publishes a Kafka topic into a Tabsdata table with the @stream_publisher decorator and KafkaStreamSrc, covering schema, dlq_topic, and batch_max_secs/batch_max_bytes settings. Source: https://docs.tabsdata.com/tutorials/publishers/kafka Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) ## Ingest a topic continuously A Kafka source lets a stream publisher ingest a Kafka topic into a Tabsdata table as messages arrive, handing them to the function in batches. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you publish the page\_views kafka topic into a table Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ The Kafka publisher guide and connection template are loaded. This one is a stream publisher, so it also needs to know when to hand over a batch. Six values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - bootstrap servers and SASL credentials - consumer group - topic - message format and schema - when to hand over a batch ❯everything you need is in Read 2 files, called tabsdata 2 times (ctrl+o to expand) ⏺ SASL password found, and the Avro schema read from ~/page\_views.avsc. Reading back what is not secret: broker.internal:9092, group tabsdata\_page\_views, topic page\_views, format avro, dead letter topic page\_views\_dlq, collection events\_src. Called tabsdata 4 times (ctrl+o to expand) ⏺ Collection **events\_src** created in group `sources`, with a `KafkaStreamSrcConn` to broker.internal:9092. The topic exists with 6 partitions. Called tabsdata 2 times (ctrl+o to expand) ⏺ The stream publisher, registered in `events_src`. A batch is due after 5 minutes, or sooner once 128 MB are waiting, but never sooner than a minute after the previous one: ```python @stream_publisher( source=KafkaStreamSrc( topic="page_views", data_format="avro", schema=PAGE_VIEWS_SCHEMA, dlq_topic="page_views_dlq", ), batch_max_secs=300, batch_max_bytes=128 * 1024 * 1024, output_tables=["page_views"], ) def read_page_views(page_views: TableFramesSpec) -> TableFrameSpec: return concat(page_views, how="diagonal_relaxed") ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Stream worker started. First batch committed after 128 MB, 0 failures. ⏺ **events\_src/page\_views** is live and keeps gaining versions, one per batch: ``` ┌───────────────────┬───────────────┐ │ field │ value │ ├───────────────────┼───────────────┤ │ collection │ events_src │ ├───────────────────┼───────────────┤ │ table │ page_views │ ├───────────────────┼───────────────┤ │ rows this version │ 1,904,338 │ ├───────────────────┼───────────────┤ │ columns │ 14 │ ├───────────────────┼───────────────┤ │ batch due on │ size (128 MB) │ ├───────────────────┼───────────────┤ │ offsets committed │ yes │ └───────────────────┴───────────────┘ ``` `schema` is required, and messages that fail to deserialize go to `dlq_topic` while the stream carries on. Without a `dlq_topic`, such a message stops the stream. A stream publisher takes no `trigger_by` at all, and it must not declare a `ctx` parameter, because it runs inside the stream loop rather than once per transaction. See [Kafka](https://docs.tabsdata.com/guide/connectors/kafka) for every option of the source and the connection, and [Batch parameters](https://docs.tabsdata.com/guide/functions/stream-publishers#batch-parameters) for when a batch is due. --- # Publish data from local files > Shows how to publish a local CSV file as a Tabsdata table by defining a LocalFileSrcConn connection with base_path, a publisher with LocalFileSrc source, and output_tables, and verifying the result. Source: https://docs.tabsdata.com/tutorials/publishers/local-file-system AI agent session ❯Can you make a table 'persons' that ingests ~/data/people/people.csv Listed 1 directory, called tabsdata (ctrl+o to expand) ⏺ CSV has 3 columns (first\_name, last\_name, age) and 2 data rows. In Tabsdata a table is the output of a publisher function, so I need to register one. Getting the publisher types. Called tabsdata (ctrl+o to expand) ⏺ Needs a LocalFile connection first. `LocalFileSrc` paths are relative to a connection's `base_path`. Getting both guides. Called tabsdata 2 times (ctrl+o to expand) ⏺ Writing the connection to , then creating a source collection from it, pointed at the data directory. Called tabsdata 2 times (ctrl+o to expand) ⏺ Registering the publisher that produces **persons**. ```python @publisher( source=LocalFileSrc(paths=["people.csv"], format=SrcFileFormat.CSV), output_tables=["persons"], ) def ingest_persons(rows: TableFramesSpec) -> TableFrameSpec: return rows[0] if rows else None ``` Called tabsdata 2 times (ctrl+o to expand) ⏺ Registered and committed. Verifying the rows landed. Called tabsdata 2 times (ctrl+o to expand) ⏺ **persons** is live with both rows: ``` ┌────────────┬───────────┬─────┐ │ first_name │ last_name │ age │ ├────────────┼───────────┼─────┤ │ Ada │ Lovelace │ 36 │ ├────────────┼───────────┼─────┤ │ Grace │ Hopper │ 45 │ └────────────┴───────────┴─────┘ ``` ⏺ What got built, in order: 1. Source collection `local_files` in group `sources`, carrying a `LocalFileSrcConn` with `base_path` set to the data directory. Connection check passed. 2. Publisher `ingest_persons`, output table `persons`. 3. One execution, committed, 0 failures. ⏺ Two things worth knowing: - A table cannot exist on its own in Tabsdata, it is the output of a function. So `persons` came with a publisher attached, not as a standalone object. - `persons` landed in `local_files` rather than in a collection you already had, because a connection-backed publisher is registered in its own source collection. Getting the data into another collection is a transformer reading `local_files/persons`. --- # Publish data from MariaDB > Publishes MariaDB query results as Tabsdata tables through the MySQL connector, covering the MySQLSrcConn connection, MySQLSrc source, and the mysql-connector-python driver requirement. Source: https://docs.tabsdata.com/tutorials/publishers/mariadb Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) warning The MariaDB connector needs the `mysql-connector-python` driver, which Tabsdata does not redistribute. See [Installing Third-Party Database Drivers](https://docs.tabsdata.com/server-admin/third-party-database-drivers) to add it to your Tabsdata server. ## Ingest query results from MariaDB MariaDB speaks the MySQL wire protocol, so the MySQL connector reads it: the connection type is `MySQLSrcConn` and the source class is `MySQLSrc`. Each entry in `queries` produces one output table. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you publish the orders and customers tables out of mariadb Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ MariaDB is read through the MySQL connector, and its guide and connection template are loaded. Four values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - connection URI - user and password - tables to read ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ Password found. Reading back what is not secret: uri mysql://maria.internal:3306/sales, collection sales\_src, tables orders and customers. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , with the MySQL connector type and the MariaDB URI, and collection **sales\_src** created from it in group `sources`. Connection check passed. Called tabsdata 2 times (ctrl+o to expand) ⏺ The publisher, registered in `sales_src`: ```python @publisher( source=MySQLSrc( queries=[ "SELECT * FROM orders", "SELECT * FROM customers", ], ), output_tables=["orders", "customers"], ) def read_sales( orders: TableFramesSpec, customers: TableFramesSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return (orders[0], customers[0]) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ Two tables are live in **sales\_src**, one per query, in the order the queries were listed: ``` ┌───────────┬─────────────────────────┬────────┬─────────┐ │ table │ from │ rows │ columns │ ├───────────┼─────────────────────────┼────────┼─────────┤ │ orders │ SELECT * FROM orders │ 48,112 │ 9 │ ├───────────┼─────────────────────────┼────────┼─────────┤ │ customers │ SELECT * FROM customers │ 3,204 │ 7 │ └───────────┴─────────────────────────┴────────┴─────────┘ ``` The `mysql://` scheme is correct for MariaDB. Nothing in the connection or the publisher names MariaDB specifically, so a server migrated between the two needs no change here. --- # Publish data from SQL Server > Publishes SQL Server query results to Tabsdata using the MSSQLSrc source and MSSQLSrcConn connection, covering the ODBC driver URI, credentials, and queries. Source: https://docs.tabsdata.com/tutorials/publishers/mssql Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) ## Ingest query results from SQL Server Each entry in `queries` produces one output table. The URI carries the ODBC driver name, which is the part most often wrong on a first attempt. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you publish the country and city tables out of sql server Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ The SQL Server publisher guide and connection template are loaded. Four values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - connection URI, including the ODBC driver - user and password - tables to read ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ Password found. Reading back what is not secret: db.internal:1433/warehouse over ODBC Driver 18, collection warehouse\_src, tables dbo.country and dbo.city. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **warehouse\_src** created from it in group `sources`. Connection check passed. Called tabsdata 2 times (ctrl+o to expand) ⏺ The publisher, registered in `warehouse_src`: ```python @publisher( source=MSSQLSrc( queries=[ "SELECT * FROM dbo.country", "SELECT * FROM dbo.city", ], ), output_tables=["country", "city"], ) def read_warehouse( country: TableFramesSpec, city: TableFramesSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return (country[0], city[0]) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ Two tables are live in **warehouse\_src**. The schema prefix stays in the query and the output table names drop it: ``` ┌─────────┬───────────────────────────┬───────┬─────────┐ │ table │ from │ rows │ columns │ ├─────────┼───────────────────────────┼───────┼─────────┤ │ country │ SELECT * FROM dbo.country │ 239 │ 15 │ ├─────────┼───────────────────────────┼───────┼─────────┤ │ city │ SELECT * FROM dbo.city │ 4,079 │ 5 │ └─────────┴───────────────────────────┴───────┴─────────┘ ``` Output table names are yours to choose and need not match the source table. `dbo.country` becomes `country` here because the schema prefix carries no meaning once the rows are in Tabsdata. --- # Publish data from MySQL > Publishes query results from a MySQL database into Tabsdata tables using MySQLSrc, with a MySQLSrcConn connection and the mysql-connector-python driver requirement. Source: https://docs.tabsdata.com/tutorials/publishers/mysql Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) warning The MySQL connector needs the `mysql-connector-python` driver, which Tabsdata does not redistribute. See [Installing Third-Party Database Drivers](https://docs.tabsdata.com/server-admin/third-party-database-drivers) to add it to your Tabsdata server. ## Ingest query results from MySQL Each entry in `queries` produces one output table, so the query list and the table list line up position by position. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you publish the country, city and country\_language tables out of mysql Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ The MySQL publisher guide and connection template are loaded. Four values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - connection URI - user and password - tables to read ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ Password found. Reading back what is not secret: uri mysql://db.internal:3306/world, collection world\_src, three tables. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **world\_src** created from it in group `sources`. Connection check passed. Called tabsdata 2 times (ctrl+o to expand) ⏺ The publisher, registered in `world_src`: ```python @publisher( source=MySQLSrc( queries=[ "SELECT * FROM country", "SELECT * FROM city", "SELECT * FROM country_language", ], ), output_tables=["country", "city", "country_language"], ) def read_world( country: TableFramesSpec, city: TableFramesSpec, country_language: TableFramesSpec, ) -> tuple[TableFrameSpec, TableFrameSpec, TableFrameSpec]: return (country[0], city[0], country_language[0]) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ Three tables are live in **world\_src**, one per query, in the order the queries were listed: ``` ┌──────────────────┬────────────────────────────────┬───────┬─────────┐ │ table │ from │ rows │ columns │ ├──────────────────┼────────────────────────────────┼───────┼─────────┤ │ country │ SELECT * FROM country │ 239 │ 15 │ ├──────────────────┼────────────────────────────────┼───────┼─────────┤ │ city │ SELECT * FROM city │ 4,079 │ 5 │ ├──────────────────┼────────────────────────────────┼───────┼─────────┤ │ country_language │ SELECT * FROM country_language │ 984 │ 4 │ └──────────────────┴────────────────────────────────┴───────┴─────────┘ ``` The connection carries the URI and the credentials, so the publisher holds SQL and nothing about where the database is. Registering the same code into a second source collection reads the same queries against a different server. --- # Publish data from MySQL CDC > Publishes MySQL binlog changes to Tabsdata tables with @stream_publisher and MySQLCdcStreamSrc, covering start_from options, batch_max_secs, output_tables, and the mysql-connector-python driver requirement. Source: https://docs.tabsdata.com/tutorials/publishers/mysql-cdc Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) warning The MySQL CDC connector needs the `mysql-connector-python` driver, which Tabsdata does not redistribute. See [Installing Third-Party Database Drivers](https://docs.tabsdata.com/server-admin/third-party-database-drivers) to add it to your Tabsdata server. ## Follow the binlog instead of re-reading tables A CDC source uses `@stream_publisher` rather than `@publisher`. A long-lived worker follows the MySQL binlog, and each time a batch is due it hands the changes captured since the previous batch to the publisher, which commits them. One output table per tracked table, so `tables` and `output_tables` line up position by position. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you stream CDC changes to the orders and customers tables out of mysql Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ That is a MySQL CDC stream publisher. Its guide and connection template are loaded. Five values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - connection URI - user and password - tables to track, fully qualified - where to start reading, and how often to hand over a batch ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ Password found. Reading back what is not secret: uri mysql://db.internal:3306/sales, tracking sales.orders and sales.customers, starting from tail, collection sales\_cdc. Called tabsdata 4 times (ctrl+o to expand) ⏺ Collection **sales\_cdc** created in group `sources`, with a `MySQLCdcStreamSrcConn` to db.internal:3306. Connection check passed, and the user has the replication grants the binlog reader needs. Called tabsdata 2 times (ctrl+o to expand) ⏺ The stream publisher, registered in `sales_cdc`. `start_from="tail"` means the worker begins at the current binlog position and never backfills, so the first version holds only changes committed after registration: ```python @stream_publisher( source=MySQLCdcStreamSrc( tables=["sales.orders", "sales.customers"], start_from="tail", ), batch_max_secs=120, output_tables=["orders_changes", "customers_changes"], ) def read_sales_changes( orders: TableFramesSpec, customers: TableFramesSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return (concat(orders), concat(customers)) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Stream worker started. First batch committed within 2 minutes, 0 failures. ⏺ Two tables are live in **sales\_cdc**, and both gain a version per batch: ``` ┌───────────────────┬─────────────────┬───────────────────┐ │ table │ tracking │ rows this version │ ├───────────────────┼─────────────────┼───────────────────┤ │ orders_changes │ sales.orders │ 1,842 │ ├───────────────────┼─────────────────┼───────────────────┤ │ customers_changes │ sales.customers │ 96 │ └───────────────────┴─────────────────┴───────────────────┘ ``` `start_from` also accepts `"full"` for a full load of the tracked tables followed by their changes, `"head"` to begin at the oldest retained binlog, or an explicit position. Rows carry the change operation, so a downstream transformer can apply inserts, updates and deletes rather than treating every row as new. A stream publisher takes no `trigger_by` and must not declare a `ctx` parameter. See [MySQL CDC](https://docs.tabsdata.com/guide/connectors/mysql-cdc) for every option of the source. --- # Publish data from Oracle > Publishes query results from Oracle databases into Tabsdata tables using the OracleSrc connector, covering the `oracledb` driver install, the connection YAML, and the publisher `queries` and `output_tables` arguments. Source: https://docs.tabsdata.com/tutorials/publishers/oracle Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) warning The Oracle connector needs the `oracledb` driver, which Tabsdata does not redistribute. See [Installing Third-Party Database Drivers](https://docs.tabsdata.com/server-admin/third-party-database-drivers) to add it to your Tabsdata server. ## Ingest query results from Oracle Each entry in `queries` produces one output table. Oracle identifiers are upper case unless they were created quoted, so the queries carry upper case names while the output tables use whatever you name them. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you publish the COUNTRY and CITY tables out of oracle Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ The Oracle publisher guide and connection template are loaded. Four values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - connection URI - user and password - tables to read ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ Password found. Reading back what is not secret: uri oracle://db.internal:1521/world, collection world\_src, tables COUNTRY and CITY. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **world\_src** created from it in group `sources`. Connection check passed. Called tabsdata 2 times (ctrl+o to expand) ⏺ The publisher, registered in `world_src`: ```python @publisher( source=OracleSrc( queries=[ "SELECT * FROM COUNTRY", "SELECT * FROM CITY", ], ), output_tables=["country", "city"], ) def read_world( country: TableFramesSpec, city: TableFramesSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return (country[0], city[0]) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ Two tables are live in **world\_src**, named in lower case even though the queries are upper case: ``` ┌─────────┬───────────────────────┬───────┬─────────┐ │ table │ from │ rows │ columns │ ├─────────┼───────────────────────┼───────┼─────────┤ │ country │ SELECT * FROM COUNTRY │ 239 │ 15 │ ├─────────┼───────────────────────┼───────┼─────────┤ │ city │ SELECT * FROM CITY │ 4,079 │ 5 │ └─────────┴───────────────────────┴───────┴─────────┘ ``` Column names arrive as Oracle reports them, which is upper case for unquoted identifiers. A transformer downstream can rename them if the rest of your pipeline expects lower case. --- # Publish data from Postgres > Publishes query results from a PostgreSQL database into Tabsdata with the PostgresSrc source, PostgresSrcConn connection, and psycopg2-binary driver requirement. Source: https://docs.tabsdata.com/tutorials/publishers/postgres Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) warning The PostgreSQL connector needs the `psycopg2-binary` driver, which Tabsdata does not redistribute. See [Installing Third-Party Database Drivers](https://docs.tabsdata.com/server-admin/third-party-database-drivers) to add it to your Tabsdata server. ## Ingest query results from Postgres Each entry in `queries` produces one output table, so the query list and the table list line up position by position. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you publish the country and city tables out of postgres Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ The Postgres publisher guide and connection template are loaded. Four values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - connection URI - user and password - tables to read ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ Password found. Reading back what is not secret: uri postgres://db.internal:5432/world, collection world\_src, tables country and city. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **world\_src** created from it in group `sources`. Connection check passed. Called tabsdata 2 times (ctrl+o to expand) ⏺ The publisher, registered in `world_src`: ```python @publisher( source=PostgresSrc( queries=[ "SELECT * FROM country", "SELECT * FROM city", ], ), output_tables=["country", "city"], ) def read_world( country: TableFramesSpec, city: TableFramesSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return (country[0], city[0]) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ Two tables are live in **world\_src**, one per query, in the order the queries were listed: ``` ┌─────────┬───────────────────────┬───────┬─────────┐ │ table │ from │ rows │ columns │ ├─────────┼───────────────────────┼───────┼─────────┤ │ country │ SELECT * FROM country │ 239 │ 15 │ ├─────────┼───────────────────────┼───────┼─────────┤ │ city │ SELECT * FROM city │ 4,079 │ 5 │ └─────────┴───────────────────────┴───────┴─────────┘ ``` The connection carries the URI and the credentials, so the publisher holds SQL and nothing about where the database is. Registering the same code into a second source collection reads the same queries against a different server. --- # Publish data from Postgres CDC > Postgres CDC stream publisher setup with @stream_publisher, PostgresCdcStreamSrc, replication slot options, start_from values, and slot cleanup. Source: https://docs.tabsdata.com/tutorials/publishers/postgres-cdc Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) warning The PostgreSQL CDC connector needs the `psycopg2-binary` driver, which Tabsdata does not redistribute. See [Installing Third-Party Database Drivers](https://docs.tabsdata.com/server-admin/third-party-database-drivers) to add it to your Tabsdata server. ## Follow the WAL instead of re-reading tables A CDC source uses `@stream_publisher` rather than `@publisher`. A long-lived worker follows the write-ahead log through a logical replication slot, and each time a batch is due it hands the changes captured since the previous batch to the publisher, which commits them. One output table per tracked table, so `tables` and `output_tables` line up position by position. Postgres is the one CDC source that needs a replication slot named up front, because the slot is what holds the WAL position between runs. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you stream CDC changes to the orders and customers tables out of postgres Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ That is a Postgres CDC stream publisher. Its guide and connection template are loaded. Six values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - connection URI - user and password - tables to track, fully qualified - replication slot name - where to start reading, and how often to hand over a batch ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ Password found. Reading back what is not secret: uri postgres://db.internal:5432/sales, tracking public.orders and public.customers, slot tabsdata\_sales, starting from tail, collection sales\_cdc. Called tabsdata 4 times (ctrl+o to expand) ⏺ Collection **sales\_cdc** created in group `sources`, with a `PostgresCdcStreamSrcConn` to db.internal:5432/sales. Connection check passed, `wal_level` is `logical`, and the slot tabsdata\_sales already exists so the default `"reuse"` behavior applies. Called tabsdata 2 times (ctrl+o to expand) ⏺ The stream publisher, registered in `sales_cdc`. `flatten_columns` defaults to true, so each source column arrives as its own column rather than nested inside a struct: ```python @stream_publisher( source=PostgresCdcStreamSrc( tables=["public.orders", "public.customers"], replication_slot="tabsdata_sales", replication_slot_behavior="reuse", start_from="tail", ), batch_max_secs=120, output_tables=["orders_changes", "customers_changes"], ) def read_sales_changes( orders: TableFramesSpec, customers: TableFramesSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return (concat(orders), concat(customers)) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Stream worker started. First batch committed within 2 minutes, 0 failures. ⏺ Two tables are live in **sales\_cdc**, and both gain a version per batch: ``` ┌───────────────────┬──────────────────┬───────────────────┐ │ table │ tracking │ rows this version │ ├───────────────────┼──────────────────┼───────────────────┤ │ orders_changes │ public.orders │ 1,842 │ ├───────────────────┼──────────────────┼───────────────────┤ │ customers_changes │ public.customers │ 96 │ └───────────────────┴──────────────────┴───────────────────┘ ``` Set `replication_slot_behavior` to `"create"` to have the connector make the slot instead of expecting it. `start_from` also accepts `"full"` for a full load of the tracked tables followed by their changes, `"head"` for the oldest retained WAL, or an explicit LSN. Tabsdata never drops the slot, not even when the function is deleted, and a slot that is no longer read makes the server keep WAL indefinitely. Drop it with `pg_drop_replication_slot('')` once its stream is gone. See [Postgres CDC](https://docs.tabsdata.com/guide/connectors/postgres-cdc) for every option of the source. --- # Publish data from Salesforce Reports > Publishes Salesforce report data into Tabsdata using SalesforceReportSrc, covering report name or id lookup with find_report_by, column_name_strategy, filter and filter_logic grouping, and last_modified_column for incremental reads. Source: https://docs.tabsdata.com/tutorials/publishers/salesforce-reports Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) ## Ingest a Salesforce report A report publisher names a report that already exists in Salesforce rather than a query you write. The report's own definition decides which columns come back, so the publisher adds filters instead of selecting columns. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you publish the contacts report out of salesforce, US and France rows only Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ The Salesforce Reports publisher guide and connection template are loaded. Four values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - username, password and security token - report name or id - filters, and how they combine ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ All three credentials found. Reading back what is not secret: report contacts, column names from columnName, two country filters combined as 1 or 2, collection sf\_reports. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **sf\_reports** created from it in group `sources`. Connection check passed, and the report resolved by name. Called tabsdata 2 times (ctrl+o to expand) ⏺ The publisher, registered in `sf_reports`. `column_name_strategy="columnName"` means the filters have to use the report's `columnName` values, so `COUNTRY` rather than the label a reader sees in the UI: ```python @publisher( source=SalesforceReportSrc( report=["contacts"], column_name_strategy="columnName", filter=[ ("COUNTRY", "equals", "USA"), ("COUNTRY", "equals", "France"), ], filter_logic="1 or 2", ), output_tables=["contacts"], ) def read_contacts(contacts: TableFramesSpec) -> tuple[TableFrameSpec]: return (contacts[0],) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. The filters ran on the Salesforce side, so only matching rows crossed the wire. ⏺ **sf\_reports/contacts** is live, holding the report's own columns named from `columnName`: ``` ┌─────────────────┬────────────┐ │ field │ value │ ├─────────────────┼────────────┤ │ collection │ sf_reports │ ├─────────────────┼────────────┤ │ table │ contacts │ ├─────────────────┼────────────┤ │ report │ contacts │ ├─────────────────┼────────────┤ │ rows │ 2,341 │ ├─────────────────┼────────────┤ │ columns │ 9 │ ├─────────────────┼────────────┤ │ filters applied │ 1 or 2 │ └─────────────────┴────────────┘ ``` `filter_logic` refers to filters by position, so `1` is the first filter in the list and `2` the second. `"1 or (2 and 3)"` groups three of them. Without it, the filters carry no stated relationship. A report referenced by name uses what Salesforce calls the report unique name, also shown as the developer name, rather than the label in the UI. `find_report_by` states whether a value is an id or a name when the inference would guess wrong. For an incremental read, `last_modified_column` names the column the comparison runs against, and that column has to be part of the report definition. --- # Publish data from a Salesforce SOQL query > Publishes Salesforce SOQL query results into Tabsdata tables with the SalesforceSrc source, covering queries, initial_last_modified, the $lastModified incremental filter, include_deleted, and the tabsdata[salesforce] install. Source: https://docs.tabsdata.com/tutorials/publishers/salesforce Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) ## Ingest SOQL query results Each entry in `queries` produces one output table, so the query list and the table list line up position by position. A query written against `SystemModStamp` reads only what changed since the previous run. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you publish contacts and accounts out of salesforce, only what has changed since the start of the year Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ The Salesforce publisher guide and connection template are loaded. Four values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - username, password and security token - objects to query - timestamp to start from ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ All three credentials found. Note that Salesforce wants the username rather than the email on the account; reader@acme.example is being used as given. Reading back what is not secret: objects Contact and Account, starting from 2026-01-01, collection sf\_src. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **sf\_src** created from it in group `sources`. Connection check passed, and the instance URL was inferred from the credentials. Called tabsdata 2 times (ctrl+o to expand) ⏺ The publisher, registered in `sf_src`. Both queries select `SystemModStamp` and compare it against `$lastModified`, which is what makes the read incremental rather than a full pull every run: ```python @publisher( source=SalesforceSrc( queries=[ "SELECT Id, AccountId, FirstName, LastName, Email, SystemModStamp " "FROM Contact WHERE SystemModStamp > $lastModified", "SELECT Name, Type, SystemModStamp " "FROM Account WHERE SystemModStamp > $lastModified", ], initial_last_modified="2026-01-01T00:00:00.000+0000", ), output_tables=["contacts", "accounts"], ) def read_sales( contacts: TableFramesSpec, accounts: TableFramesSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return (contacts[0], accounts[0]) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ Two tables are live in **sf\_src**, holding rows modified since 2026-01-01 rather than the whole object: ``` ┌──────────┬─────────┬───────┬─────────┐ │ table │ from │ rows │ columns │ ├──────────┼─────────┼───────┼─────────┤ │ contacts │ Contact │ 8,402 │ 6 │ ├──────────┼─────────┼───────┼─────────┤ │ accounts │ Account │ 1,190 │ 3 │ └──────────┴─────────┴───────┴─────────┘ ``` On the next run `$lastModified` takes the highest `SystemModStamp` already read rather than the `initial_last_modified` value, so each trigger picks up where the last one stopped. Deleted rows are invisible to that comparison, since a deleted row stops appearing instead of getting a newer timestamp. Ask for `include_deleted` when the pipeline needs to know that a row went away. The connector ships separately from the core package, so `pip install 'tabsdata[salesforce]'` has to have run in the environment the server uses. --- # Create a TableFrame > Shows how to create a TableFrame programmatically from a Python dict, Pandas DataFrame, or Polars LazyFrame/DataFrame via `from_dict`, `from_pandas`, and `from_polars`, and how to convert back with `to_dict`, `to_pandas`, `to_polars_df`, and `to_polars_lf`. Source: https://docs.tabsdata.com/tutorials/transformations/create-a-tableframe While TableFrames are typically received as input in publishers and transformers, you can also create them programmatically for testing or data generation purposes. ## From a Dictionary Use `TableFrame.from_dict()` to create a TableFrame from a Python dictionary: transform.py ```python from tabsdatak.tableframe import TableFrame data = { "name": ["Alice", "Bob", "Charlie"], "age": [25, 30, 35], "city": ["New York", "London", "Paris"] } tf = TableFrame.from_dict(data) ``` ## From Pandas Convert a Pandas DataFrame to a TableFrame: transform.py ```python import pandas as pd from tabsdatak.tableframe import TableFrame df = pd.DataFrame({ "product": ["Widget", "Gadget"], "price": [19.99, 29.99] }) tf = TableFrame.from_pandas(df) ``` ## From Polars Convert from Polars LazyFrame or DataFrame: transform.py ```python import polars as pl from tabsdatak.tableframe import TableFrame # From Polars LazyFrame lf = pl.LazyFrame({"x": [1, 2, 3], "y": [4, 5, 6]}) tf = TableFrame.from_polars(lf) # From Polars DataFrame pdf = pl.DataFrame({"x": [1, 2, 3], "y": [4, 5, 6]}) tf = TableFrame.from_polars(pdf) ``` ## Converting TableFrames Export TableFrames to other formats: transform.py ```python # To dictionary data_dict = tf.to_dict() # To Pandas DataFrame pandas_df = tf.to_pandas() # To Polars DataFrame polars_df = tf.to_polars_df() # To Polars LazyFrame polars_lf = tf.to_polars_lf() ``` See [TableFrame API Guide](https://docs.tabsdata.com/guide/tables) for the whole API in one place. --- # Select and Modify Columns > Covers Tabsdata column projection in transformers: select() with names, data-type and name-pattern selectors, position selectors, exclude(), drop(), with_columns(), lit(), and rename(). Source: https://docs.tabsdata.com/tutorials/transformations/select-columns Projection operations allow you to select, transform, and reshape columns in your data. Use the Tabsdata MCP Integration with claude code or openai codex to easily create new tables ## Select columns by name AI agent session ❯make a new table that takes the persons table and selects only the first\_name and last\_name columns ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — select() takes plain column-name strings. Called tabsdata 5 times (ctrl+o to expand) ⏺ The transformer, registered as fv20260812-2 in subscriptions: ```python @transformer( input_tables=["local_files/persons"], output_tables=["person_names"], ) def select_person_names(persons: TableFrameSpec) -> TableFrameSpec: return persons.select("first_name", "last_name") ``` ⏺ Execution ep20260812-2 — Committed, 0 failures. ⏺ subscriptions/person\_names is live — 2 columns, age dropped: ``` ┌────────────┬───────────┐ │ first_name │ last_name │ ├────────────┼───────────┤ │ Ada │ Lovelace │ ├────────────┼───────────┤ │ Grace │ Hopper │ └────────────┴───────────┘ ``` Transformed tables persons | first\_name | last\_name | age | | --- | --- | --- | | Ada | Lovelace | 36 | | Grace | Hopper | 45 | 2 rows person\_names | first\_name | last\_name | | --- | --- | | Ada | Lovelace | | Grace | Hopper | 2 rows ## Select columns by data type AI agent session ❯make a new table that takes the records table and selects only the columns with a numeric type ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — you named a type rather than columns, so this is select() with the cs.numeric() selector, which resolves against whatever columns the table holds at run time. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["records"], output_tables=["numeric_only"], ) def select_numeric(records: TableFrameSpec) -> TableFrameSpec: return records.select(cs.numeric()) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/numeric\_only is live — 2 of 4 columns. name is a String and created is a Date, so both dropped out: ``` ┌────┬───────┐ │ id │ price │ ├────┼───────┤ │ 1 │ 9.99 │ ├────┼───────┤ │ 2 │ 4.50 │ └────┴───────┘ ``` Transformed tables records | idInt64 | priceFloat64 | nameString | createdDate | | --- | --- | --- | --- | | 1 | 9.99 | widget | 2024-03-15 | | 2 | 4.50 | gadget | 2024-11-01 | 2 rows numeric\_only | idInt64 | priceFloat64 | | --- | --- | | 1 | 9.99 | | 2 | 4.50 | 2 rows AI agent session ❯make a new table that takes the records table and selects only the columns with a text type ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — select() with cs.string(), which matches the String dtype rather than anything that looks like text. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["records"], output_tables=["strings_only"], ) def select_strings(records: TableFrameSpec) -> TableFrameSpec: return records.select(cs.string()) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/strings\_only is live — 1 of 4 columns: ``` ┌────────┐ │ name │ ├────────┤ │ widget │ ├────────┤ │ gadget │ └────────┘ ``` Transformed tables records | idInt64 | priceFloat64 | nameString | createdDate | | --- | --- | --- | --- | | 1 | 9.99 | widget | 2024-03-15 | | 2 | 4.50 | gadget | 2024-11-01 | 2 rows strings\_only | nameString | | --- | | widget | | gadget | 2 rows AI agent session ❯make a new table that takes the records table and selects only the columns with a date or time type ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — select() with cs.temporal(), which covers Date, Datetime and Time rather than Date alone. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["records"], output_tables=["dates_only"], ) def select_temporal(records: TableFrameSpec) -> TableFrameSpec: return records.select(cs.temporal()) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/dates\_only is live — 1 of 4 columns: ``` ┌────────────┐ │ created │ ├────────────┤ │ 2024-03-15 │ ├────────────┤ │ 2024-11-01 │ └────────────┘ ``` Transformed tables records | idInt64 | priceFloat64 | nameString | createdDate | | --- | --- | --- | --- | | 1 | 9.99 | widget | 2024-03-15 | | 2 | 4.50 | gadget | 2024-11-01 | 2 rows dates\_only | createdDate | | --- | | 2024-03-15 | | 2024-11-01 | 2 rows AI agent session ❯make a new table that takes the records table and selects only the columns typed Int64 or Float64 ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — select() with cs.by\_dtype(), which takes the dtypes by name. Narrower than cs.numeric(): an Int32 column is numeric but would not match this. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["records"], output_tables=["ints_and_floats"], ) def select_by_dtype(records: TableFrameSpec) -> TableFrameSpec: return records.select(cs.by_dtype(Int64, Float64)) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/ints\_and\_floats is live — 2 of 4 columns: ``` ┌────┬───────┐ │ id │ price │ ├────┼───────┤ │ 1 │ 9.99 │ ├────┼───────┤ │ 2 │ 4.50 │ └────┴───────┘ ``` Transformed tables records | idInt64 | priceFloat64 | nameString | createdDate | | --- | --- | --- | --- | | 1 | 9.99 | widget | 2024-03-15 | | 2 | 4.50 | gadget | 2024-11-01 | 2 rows ints\_and\_floats | idInt64 | priceFloat64 | | --- | --- | | 1 | 9.99 | | 2 | 4.50 | 2 rows ## Select columns by name pattern AI agent session ❯make a new table that takes the orders table and selects only the columns whose name starts with "customer" ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — select() with cs.starts\_with("customer\_"). Matching the prefix rather than listing the two columns survives a third one arriving upstream. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["orders"], output_tables=["customer_cols"], ) def select_customer_cols(orders: TableFrameSpec) -> TableFrameSpec: return orders.select(cs.starts_with("customer_")) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/customer\_cols is live — 2 of 5 columns: ``` ┌─────────────┬───────────────┐ │ customer_id │ customer_name │ ├─────────────┼───────────────┤ │ c1 │ Ada │ ├─────────────┼───────────────┤ │ c2 │ Grace │ └─────────────┴───────────────┘ ``` Transformed tables orders | customer\_id | customer\_name | price | amount\_1 | note\_id | | --- | --- | --- | --- | --- | | c1 | Ada | 10 | 5 | n1 | | c2 | Grace | 20 | 7 | n2 | 2 rows customer\_cols | customer\_id | customer\_name | | --- | --- | | c1 | Ada | | c2 | Grace | 2 rows AI agent session ❯make a new table that takes the orders table and selects only the columns whose name ends with "\_id" ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — select() with cs.ends\_with("\_id"), so where the columns sit in the table does not matter. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["orders"], output_tables=["id_cols"], ) def select_id_cols(orders: TableFrameSpec) -> TableFrameSpec: return orders.select(cs.ends_with("_id")) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/id\_cols is live — customer\_id and note\_id, which are not adjacent in the source: ``` ┌─────────────┬─────────┐ │ customer_id │ note_id │ ├─────────────┼─────────┤ │ c1 │ n1 │ ├─────────────┼─────────┤ │ c2 │ n2 │ └─────────────┴─────────┘ ``` Transformed tables orders | customer\_id | customer\_name | price | amount\_1 | note\_id | | --- | --- | --- | --- | --- | | c1 | Ada | 10 | 5 | n1 | | c2 | Grace | 20 | 7 | n2 | 2 rows id\_cols | customer\_id | note\_id | | --- | --- | | c1 | n1 | | c2 | n2 | 2 rows AI agent session ❯make a new table that takes the orders table and selects only the columns whose name contains "price" ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — select() with cs.contains("price"), which matches the substring anywhere in the name, so list\_price would come through too. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["orders"], output_tables=["price_cols"], ) def select_price_cols(orders: TableFrameSpec) -> TableFrameSpec: return orders.select(cs.contains("price")) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/price\_cols is live — 1 of 5 columns matched: ``` ┌───────┐ │ price │ ├───────┤ │ 10 │ ├───────┤ │ 20 │ └───────┘ ``` Transformed tables orders | customer\_id | customer\_name | price | amount\_1 | note\_id | | --- | --- | --- | --- | --- | | c1 | Ada | 10 | 5 | n1 | | c2 | Grace | 20 | 7 | n2 | 2 rows price\_cols | price | | --- | | 10 | | 20 | 2 rows AI agent session ❯make a new table that takes the orders table and selects only the columns whose name matches "amount\_" followed by digits ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — a prefix match would also take a column like amount\_note, so this needs select() with cs.matches() and an anchored pattern. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["orders"], output_tables=["amount_cols"], ) def select_amount_cols(orders: TableFrameSpec) -> TableFrameSpec: return orders.select(cs.matches(r"^amount_\d+$")) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/amount\_cols is live — 1 of 5 columns matched: ``` ┌──────────┐ │ amount_1 │ ├──────────┤ │ 5 │ ├──────────┤ │ 7 │ └──────────┘ ``` Transformed tables orders | customer\_id | customer\_name | price | amount\_1 | note\_id | | --- | --- | --- | --- | --- | | c1 | Ada | 10 | 5 | n1 | | c2 | Grace | 20 | 7 | n2 | 2 rows amount\_cols | amount\_1 | | --- | | 5 | | 7 | 2 rows ## Select columns by position AI agent session ❯make a new table that takes the orders table and selects only the first three columns ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — position rather than name or type, so select() with cs.first(3), counting from the left. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["orders"], output_tables=["first_three"], ) def select_first_three(orders: TableFrameSpec) -> TableFrameSpec: return orders.select(cs.first(3)) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/first\_three is live — the leading 3 of 5 columns. A publisher that starts emitting a new column in the middle would change what this picks, without erroring: ``` ┌─────────────┬───────────────┬───────┐ │ customer_id │ customer_name │ price │ ├─────────────┼───────────────┼───────┤ │ c1 │ Ada │ 10 │ ├─────────────┼───────────────┼───────┤ │ c2 │ Grace │ 20 │ └─────────────┴───────────────┴───────┘ ``` Transformed tables orders | customer\_id | customer\_name | price | amount\_1 | note\_id | | --- | --- | --- | --- | --- | | c1 | Ada | 10 | 5 | n1 | | c2 | Grace | 20 | 7 | n2 | 2 rows first\_three | customer\_id | customer\_name | price | | --- | --- | --- | | c1 | Ada | 10 | | c2 | Grace | 20 | 2 rows AI agent session ❯make a new table that takes the orders table and selects only the last two columns ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — select() with cs.last(2), counting from the right. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["orders"], output_tables=["last_two"], ) def select_last_two(orders: TableFrameSpec) -> TableFrameSpec: return orders.select(cs.last(2)) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/last\_two is live — the trailing 2 of 5 columns: ``` ┌──────────┬─────────┐ │ amount_1 │ note_id │ ├──────────┼─────────┤ │ 5 │ n1 │ ├──────────┼─────────┤ │ 7 │ n2 │ └──────────┴─────────┘ ``` Transformed tables orders | customer\_id | customer\_name | price | amount\_1 | note\_id | | --- | --- | --- | --- | --- | | c1 | Ada | 10 | 5 | n1 | | c2 | Grace | 20 | 7 | n2 | 2 rows last\_two | amount\_1 | note\_id | | --- | --- | | 5 | n1 | | 7 | n2 | 2 rows AI agent session ❯make a new table that takes the orders table and selects only the columns at positions 0, 2 and 4 ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — select() with cs.by\_index(), which takes positions one at a time and is 0-based. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["orders"], output_tables=["by_position"], ) def select_by_index(orders: TableFrameSpec) -> TableFrameSpec: return orders.select(cs.by_index(0, 2, 4)) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/by\_position is live — the first, third and fifth columns: ``` ┌─────────────┬───────┬─────────┐ │ customer_id │ price │ note_id │ ├─────────────┼───────┼─────────┤ │ c1 │ 10 │ n1 │ ├─────────────┼───────┼─────────┤ │ c2 │ 20 │ n2 │ └─────────────┴───────┴─────────┘ ``` Transformed tables orders | customer\_id | customer\_name | price | amount\_1 | note\_id | | --- | --- | --- | --- | --- | | c1 | Ada | 10 | 5 | n1 | | c2 | Grace | 20 | 7 | n2 | 2 rows by\_position | customer\_id | price | note\_id | | --- | --- | --- | | c1 | 10 | n1 | | c2 | 20 | n2 | 2 rows ## Select everything except some columns AI agent session ❯make a new table that takes the records table and selects every column except internal\_id and metadata ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — select() with cs.exclude(), which keeps every column you did not name. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["records"], output_tables=["public_cols"], ) def select_public_cols(records: TableFrameSpec) -> TableFrameSpec: return records.select(cs.exclude("internal_id", "metadata")) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/public\_cols is live — id left standing: ``` ┌────┐ │ id │ ├────┤ │ 1 │ ├────┤ │ 2 │ └────┘ ``` Transformed tables records | id | internal\_id | metadata | | --- | --- | --- | | 1 | x1 | seed | | 2 | x2 | import | 2 rows public\_cols | id | | --- | | 1 | | 2 | 2 rows ## Drop columns AI agent session ❯make a new table that takes the records table and drops the col1, col2 and col3 columns ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — drop() names what goes rather than what stays, which is the shorter way round here. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["records"], output_tables=["records_clean"], ) def drop_spare_cols(records: TableFrameSpec) -> TableFrameSpec: return records.drop("col1", "col2", "col3") ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/records\_clean is live — id left standing: ``` ┌────┐ │ id │ ├────┤ │ 1 │ ├────┤ │ 2 │ └────┘ ``` Transformed tables records | id | col1 | col2 | col3 | | --- | --- | --- | --- | | 1 | a | b | c | | 2 | d | e | f | 2 rows records\_clean | id | | --- | | 1 | | 2 | 2 rows AI agent session ❯make a new table that takes the readings table and drops every column whose name ends in \_raw ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — selectors work inside drop() as well as select(), so cs.ends\_with("\_raw") clears the family without naming each column. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["readings"], output_tables=["readings_clean"], ) def drop_raw_cols(readings: TableFrameSpec) -> TableFrameSpec: return readings.drop(cs.ends_with("_raw")) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/readings\_clean is live — score\_raw dropped, score and label kept: ``` ┌───────┬───────┐ │ score │ label │ ├───────┼───────┤ │ 9 │ high │ ├───────┼───────┤ │ 3 │ low │ └───────┴───────┘ ``` Transformed tables readings | scoreInt64 | score\_rawFloat64 | labelString | | --- | --- | --- | | 9 | 9.4 | high | | 3 | 2.6 | low | 2 rows readings\_clean | scoreInt64 | labelString | | --- | --- | | 9 | high | | 3 | low | 2 rows ## Add or replace columns AI agent session ❯make a new table that takes the orders table and adds a total column, uppercases the name column, and stamps every row with a currency column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — with\_columns() rather than select(), since it keeps every column you do not name and select() would keep only the listed ones. Called tabsdata 4 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["orders"], output_tables=["orders_priced"], ) def add_totals(orders: TableFrameSpec) -> TableFrameSpec: return orders.with_columns( (col("price") * col("quantity")).alias("total"), col("name").str.to_uppercase().alias("name"), lit("USD").alias("currency"), ) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/orders\_priced is live — price and quantity came through untouched, name was replaced in place because the alias reuses its name, and total and currency are new: ``` ┌────────┬───────┬──────────┬───────┬──────────┐ │ name │ price │ quantity │ total │ currency │ ├────────┼───────┼──────────┼───────┼──────────┤ │ WIDGET │ 10 │ 3 │ 30 │ USD │ ├────────┼───────┼──────────┼───────┼──────────┤ │ GADGET │ 4 │ 2 │ 8 │ USD │ └────────┴───────┴──────────┴───────┴──────────┘ ``` Transformed tables orders | name | price | quantity | | --- | --- | --- | | widget | 10 | 3 | | gadget | 4 | 2 | 2 rows orders\_priced | name | price | quantity | total | currency | | --- | --- | --- | --- | --- | | WIDGET | 10 | 3 | 30 | USD | | GADGET | 4 | 2 | 8 | USD | 2 rows ## Add columns with literal values AI agent session ❯make a new table that takes the records table and stamps every row with a version, status, is\_valid and pi column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — four constants rather than expressions over existing columns, so with\_columns() with lit(), which fills a column with the same value for every row. Called tabsdata 4 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["records"], output_tables=["records_stamped"], ) def add_literals(records: TableFrameSpec) -> TableFrameSpec: return records.with_columns( lit(1).alias("version"), lit("active").alias("status"), lit(True).alias("is_valid"), lit(3.14159).alias("pi"), ) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/records\_stamped is live — 4 new columns beside the id that was already there: ``` ┌────┬─────────┬────────┬──────────┬─────────┐ │ id │ version │ status │ is_valid │ pi │ ├────┼─────────┼────────┼──────────┼─────────┤ │ 1 │ 1 │ active │ true │ 3.14159 │ ├────┼─────────┼────────┼──────────┼─────────┤ │ 2 │ 1 │ active │ true │ 3.14159 │ └────┴─────────┴────────┴──────────┴─────────┘ ``` Transformed tables records | id | | --- | | 1 | | 2 | 2 rows records\_stamped | id | version | status | is\_valid | pi | | --- | --- | --- | --- | --- | | 1 | 1 | active | true | 3.14159 | | 2 | 1 | active | true | 3.14159 | 2 rows ## Keep only derived columns AI agent session ❯make a new table that takes the orders table and keeps only a full\_name column and a price\_with\_tax column, both derived from other columns ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — "nothing else" makes this select() rather than with\_columns(). The expressions read columns that will not survive into the output, which is fine, they are read before the projection happens. Called tabsdata 4 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["orders"], output_tables=["order_summary"], ) def select_derived(orders: TableFrameSpec) -> TableFrameSpec: return orders.select( (col("first_name") + " " + col("last_name")).alias("full_name"), (col("price") * 1.1).alias("price_with_tax"), ) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/order\_summary is live — first\_name, last\_name and price are gone, and the columns come out in the order the expressions were listed: ``` ┌──────────────┬────────────────┐ │ full_name │ price_with_tax │ ├──────────────┼────────────────┤ │ Ada Lovelace │ 11 │ ├──────────────┼────────────────┤ │ Grace Hopper │ 22 │ └──────────────┴────────────────┘ ``` Transformed tables orders | first\_name | last\_name | price | | --- | --- | --- | | Ada | Lovelace | 10 | | Grace | Hopper | 20 | 2 rows order\_summary | full\_name | price\_with\_tax | | --- | --- | | Ada Lovelace | 11 | | Grace Hopper | 22 | 2 rows ## Rename columns AI agent session ❯make a new table that takes the records table and renames the old\_name column to new\_name and the customer\_id column to cust\_id ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — names only, no values touched, so rename() with a dict of old name to new name. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["records"], output_tables=["records_renamed"], ) def rename_cols(records: TableFrameSpec) -> TableFrameSpec: return records.rename({ "old_name": "new_name", "customer_id": "cust_id", }) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/records\_renamed is live — same data under new headers. This version carries the new names and every earlier version keeps the old ones, so anything downstream reading old\_name breaks on its next run: ``` ┌──────────┬─────────┐ │ new_name │ cust_id │ ├──────────┼─────────┤ │ Ada │ c1 │ ├──────────┼─────────┤ │ Grace │ c2 │ └──────────┴─────────┘ ``` Transformed tables records | old\_name | customer\_id | | --- | --- | | Ada | c1 | | Grace | c2 | 2 rows records\_renamed | new\_name | cust\_id | | --- | --- | | Ada | c1 | | Grace | c2 | 2 rows --- # Filter Rows > Filter rows in a Tabsdata transformer with filter() using col() comparisons, &, |, ~, xor, is_in, is_between, is_null, is_not_nan, and unique() with subset and keep. Source: https://docs.tabsdata.com/tutorials/transformations/filter-rows ## Filter rows by a comparison AI agent session ❯make a new table that takes the persons table and keeps only the rows where the age column is greater than 18 ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — filter() takes a boolean expression, so this is col("age") > 18. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["persons"], output_tables=["adults"], ) def filter_adults(persons: TableFrameSpec) -> TableFrameSpec: return persons.filter(col("age") > 18) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/adults is live — Bo is 17, so 2 of 3 rows kept: ``` ┌──────┬─────┐ │ name │ age │ ├──────┼─────┤ │ Ada │ 25 │ ├──────┼─────┤ │ Cy │ 30 │ └──────┴─────┘ ``` Transformed tables persons | name | age | | --- | --- | | Ada | 25 | | Bo | 17 | | Cy | 30 | 3 rows adults | name | age | | --- | --- | | Ada | 25 | | Cy | 30 | 2 rows AI agent session ❯make a new table that takes the accounts table and keeps only the rows where the status column equals "active" ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — filter() with ==. A single = would be an assignment rather than a comparison and would not parse here. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["accounts"], output_tables=["active_accounts"], ) def filter_active(accounts: TableFrameSpec) -> TableFrameSpec: return accounts.filter(col("status") == "active") ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/active\_accounts is live — Bo is archived, so 2 of 3 rows kept: ``` ┌──────┬────────┐ │ name │ status │ ├──────┼────────┤ │ Ada │ active │ ├──────┼────────┤ │ Cy │ active │ └──────┴────────┘ ``` Transformed tables accounts | name | status | | --- | --- | | Ada | active | | Bo | archived | | Cy | active | 3 rows active\_accounts | name | status | | --- | --- | | Ada | active | | Cy | active | 2 rows AI agent session ❯make a new table that takes the items table and keeps only the rows where the category column does not equal "archived" ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — filter() with !=, which keeps every row that is not archived rather than only the ones that are live today. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["items"], output_tables=["live_items"], ) def filter_live(items: TableFrameSpec) -> TableFrameSpec: return items.filter(col("category") != "archived") ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/live\_items is live — relic dropped, 2 of 3 rows kept: ``` ┌────────┬──────────┐ │ name │ category │ ├────────┼──────────┤ │ widget │ live │ ├────────┼──────────┤ │ gadget │ live │ └────────┴──────────┘ ``` Transformed tables items | name | category | | --- | --- | | widget | live | | relic | archived | | gadget | live | 3 rows live\_items | name | category | | --- | --- | | widget | live | | gadget | live | 2 rows ## Combine conditions AI agent session ❯make a new table that takes the persons table and keeps only the rows where the age column is greater than 18 and less than 65 ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — & joins the two conditions, and each side needs its own parentheses because & binds tighter than >. (col("age") > 18).and\_(...) is the method form if you would rather chain. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["persons"], output_tables=["working_age"], ) def filter_working_age(persons: TableFrameSpec) -> TableFrameSpec: return persons.filter((col("age") > 18) & (col("age") < 65)) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/working\_age is live — Bo is 17 and Cy is 70, so 1 of 3 rows kept: ``` ┌──────┬─────┐ │ name │ age │ ├──────┼─────┤ │ Ada │ 25 │ └──────┴─────┘ ``` Transformed tables persons | name | age | | --- | --- | | Ada | 25 | | Bo | 17 | | Cy | 70 | 3 rows working\_age | name | age | | --- | --- | | Ada | 25 | 1 row AI agent session ❯make a new table that takes the accounts table and keeps only the rows where the status column is "active" or "pending" ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — | for either condition, with the same parenthesis rule as &. .or\_() is the method form. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["accounts"], output_tables=["open_accounts"], ) def filter_open(accounts: TableFrameSpec) -> TableFrameSpec: return accounts.filter((col("status") == "active") | (col("status") == "pending")) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/open\_accounts is live — Bo is closed, so 2 of 3 rows kept: ``` ┌──────┬─────────┐ │ name │ status │ ├──────┼─────────┤ │ Ada │ active │ ├──────┼─────────┤ │ Cy │ pending │ └──────┴─────────┘ ``` Transformed tables accounts | name | status | | --- | --- | | Ada | active | | Bo | closed | | Cy | pending | 3 rows open\_accounts | name | status | | --- | --- | | Ada | active | | Cy | pending | 2 rows AI agent session ❯make a new table that takes the records table and keeps only the rows where the is\_deleted column is false ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — is\_deleted is already Boolean, so this inverts it with ~ rather than comparing it to false. .not\_() is the method form. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["records"], output_tables=["live_records"], ) def filter_live_records(records: TableFrameSpec) -> TableFrameSpec: return records.filter(~(col("is_deleted"))) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/live\_records is live — row 2 dropped, 2 of 3 rows kept: ``` ┌────┬────────────┐ │ id │ is_deleted │ ├────┼────────────┤ │ 1 │ false │ ├────┼────────────┤ │ 3 │ false │ └────┴────────────┘ ``` Transformed tables records | idInt64 | is\_deletedBoolean | | --- | --- | | 1 | false | | 2 | true | | 3 | false | 3 rows live\_records | idInt64 | is\_deletedBoolean | | --- | --- | | 1 | false | | 3 | false | 2 rows AI agent session ❯make a new table that takes the records table and keeps only the rows where exactly one of the flag\_a and flag\_b columns is true ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — "exactly one" is exclusive or, which is a method rather than an operator: col("flag\_a").xor(col("flag\_b")). Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["records"], output_tables=["one_flag_only"], ) def filter_one_flag(records: TableFrameSpec) -> TableFrameSpec: return records.filter(col("flag_a").xor(col("flag_b"))) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/one\_flag\_only is live — row 2 has both flags and row 3 has neither, so 1 of 3 rows kept: ``` ┌────┬────────┬────────┐ │ id │ flag_a │ flag_b │ ├────┼────────┼────────┤ │ 1 │ true │ false │ └────┴────────┴────────┘ ``` Transformed tables records | idInt64 | flag\_aBoolean | flag\_bBoolean | | --- | --- | --- | | 1 | true | false | | 2 | true | true | | 3 | false | false | 3 rows one\_flag\_only | idInt64 | flag\_aBoolean | flag\_bBoolean | | --- | --- | --- | | 1 | true | false | 1 row ## Filter by range or membership AI agent session ❯make a new table that takes the items table and keeps only the rows where the price column is between 10 and 100 ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — is\_between() rather than two comparisons joined by &, and it includes both ends of the range. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["items"], output_tables=["mid_price"], ) def filter_mid_price(items: TableFrameSpec) -> TableFrameSpec: return items.filter(col("price").is_between(10.0, 100.0)) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/mid\_price is live — widget is 9.5 and anvil is 250, so 1 of 3 rows kept: ``` ┌────────┬───────┐ │ item │ price │ ├────────┼───────┤ │ gadget │ 42 │ └────────┴───────┘ ``` Transformed tables items | item | price | | --- | --- | | widget | 9.5 | | gadget | 42 | | anvil | 250 | 3 rows mid\_price | item | price | | --- | --- | | gadget | 42 | 1 row AI agent session ❯make a new table that takes the offices table and keeps only the rows where the country column is either USA, Canada or Mexico ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — is\_in() with the list, which stays readable where three == conditions chained with | would not. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["offices"], output_tables=["north_america"], ) def filter_north_america(offices: TableFrameSpec) -> TableFrameSpec: return offices.filter(col("country").is_in(["USA", "Canada", "Mexico"])) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/north\_america is live — Lyon dropped, 2 of 3 rows kept. Mexico matched nothing, which is not an error: ``` ┌──────────┬─────────┐ │ city │ country │ ├──────────┼─────────┤ │ Austin │ USA │ ├──────────┼─────────┤ │ Montreal │ Canada │ └──────────┴─────────┘ ``` Transformed tables offices | city | country | | --- | --- | | Austin | USA | | Lyon | France | | Montreal | Canada | 3 rows north\_america | city | country | | --- | --- | | Austin | USA | | Montreal | Canada | 2 rows AI agent session ❯make a new table that takes the contacts table and keeps only the rows where the email column is not null ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — is\_not\_null() rather than != null. Comparing against null does not evaluate the way a value comparison does, so the check has to be its own method. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["contacts"], output_tables=["with_email"], ) def filter_with_email(contacts: TableFrameSpec) -> TableFrameSpec: return contacts.filter(col("email").is_not_null()) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/with\_email is live — Bo has no email, so 2 of 3 rows kept: ``` ┌──────┬───────────┐ │ name │ email │ ├──────┼───────────┤ │ Ada │ ada@x.com │ ├──────┼───────────┤ │ Cy │ cy@x.com │ └──────┴───────────┘ ``` Transformed tables contacts | name | email | | --- | --- | | Ada | ada@x.com | | Bo | null | | Cy | cy@x.com | 3 rows with\_email | name | email | | --- | --- | | Ada | ada@x.com | | Cy | cy@x.com | 2 rows AI agent session ❯make a new table that takes the records table and keeps only the rows where the optional\_field column is null ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — is\_null(), the inverse of is\_not\_null(), which is the one to reach for when the missing rows are the ones you want to look at. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["records"], output_tables=["missing_field"], ) def filter_missing_field(records: TableFrameSpec) -> TableFrameSpec: return records.filter(col("optional_field").is_null()) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/missing\_field is live — row 1 has a value, so 2 of 3 rows kept: ``` ┌────┬────────────────┐ │ id │ optional_field │ ├────┼────────────────┤ │ 2 │ null │ ├────┼────────────────┤ │ 3 │ null │ └────┴────────────────┘ ``` Transformed tables records | id | optional\_field | | --- | --- | | 1 | set | | 2 | null | | 3 | null | 3 rows missing\_field | id | optional\_field | | --- | --- | | 2 | null | | 3 | null | 2 rows AI agent session ❯make a new table that takes the readings table and keeps only the rows where the ratio column is not NaN ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — is\_not\_nan() rather than is\_not\_null(). NaN is a float value rather than a missing one, so a null check would let it through. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["readings"], output_tables=["valid_ratios"], ) def filter_valid_ratios(readings: TableFrameSpec) -> TableFrameSpec: return readings.filter(col("ratio").is_not_nan()) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/valid\_ratios is live — row 2 is NaN, so 2 of 3 rows kept: ``` ┌────┬───────┐ │ id │ ratio │ ├────┼───────┤ │ 1 │ 0.5 │ ├────┼───────┤ │ 3 │ 1.25 │ └────┴───────┘ ``` Transformed tables readings | id | ratio | | --- | --- | | 1 | 0.5 | | 2 | NaN | | 3 | 1.25 | 3 rows valid\_ratios | id | ratio | | --- | --- | | 1 | 0.5 | | 3 | 1.25 | 2 rows ## Keep unique rows AI agent session ❯make a new table that takes the visits table and removes the rows that are duplicated across every column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — unique() with no arguments compares every column, so a row collapses only when it matches another one completely. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["visits"], output_tables=["visits_deduped"], ) def dedupe_visits(visits: TableFrameSpec) -> TableFrameSpec: return visits.unique() ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/visits\_deduped is live — ada/mon appeared twice, so 3 of 4 rows kept: ``` ┌──────┬─────┐ │ user │ day │ ├──────┼─────┤ │ ada │ mon │ ├──────┼─────┤ │ bo │ mon │ ├──────┼─────┤ │ cy │ tue │ └──────┴─────┘ ``` Transformed tables visits | user | day | | --- | --- | | ada | mon | | bo | mon | | ada | mon | | cy | tue | 4 rows visits\_deduped | user | day | | --- | --- | | ada | mon | | bo | mon | | cy | tue | 3 rows AI agent session ❯make a new table that takes the visits table and keeps one row per customer\_id and date pair ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — unique() with subset, which decides what counts as a duplicate on those two columns alone. spend can differ between duplicates and one of them still goes. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["visits"], output_tables=["one_per_day"], ) def one_visit_per_day(visits: TableFrameSpec) -> TableFrameSpec: return visits.unique(subset=["customer_id", "date"]) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/one\_per\_day is live — c1/mon appeared twice with different spend, so 3 of 4 rows kept: ``` ┌─────────────┬──────┬───────┐ │ customer_id │ date │ spend │ ├─────────────┼──────┼───────┤ │ c1 │ mon │ 10 │ ├─────────────┼──────┼───────┤ │ c2 │ tue │ 7 │ ├─────────────┼──────┼───────┤ │ c1 │ wed │ 5 │ └─────────────┴──────┴───────┘ ``` Transformed tables visits | customer\_id | date | spend | | --- | --- | --- | | c1 | mon | 10 | | c1 | mon | 12 | | c2 | tue | 7 | | c1 | wed | 5 | 4 rows one\_per\_day | customer\_id | date | spend | | --- | --- | --- | | c1 | mon | 10 | | c2 | tue | 7 | | c1 | wed | 5 | 3 rows AI agent session ❯make a new table that takes the visits table and keeps one row per value of the customer\_id column, the first of each ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — unique() with subset and keep="first", which takes the earliest row in each group in the order the data arrives. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["visits"], output_tables=["first_visit"], ) def first_visit_per_customer(visits: TableFrameSpec) -> TableFrameSpec: return visits.unique(subset=["customer_id"], keep="first") ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/first\_visit is live — one row each for c1 and c2, so 2 of 4 rows kept: ``` ┌─────────────┬──────┬───────┐ │ customer_id │ date │ spend │ ├─────────────┼──────┼───────┤ │ c1 │ mon │ 10 │ ├─────────────┼──────┼───────┤ │ c2 │ tue │ 7 │ └─────────────┴──────┴───────┘ ``` Transformed tables visits | customer\_id | date | spend | | --- | --- | --- | | c1 | mon | 10 | | c1 | mon | 12 | | c2 | tue | 7 | | c1 | wed | 5 | 4 rows first\_visit | customer\_id | date | spend | | --- | --- | --- | | c1 | mon | 10 | | c2 | tue | 7 | 2 rows AI agent session ❯make a new table that takes the visits table and keeps one row per value of the customer\_id column, the last of each ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — the same subset with keep="last", which takes the final row in each group instead of the first. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["visits"], output_tables=["latest_visit"], ) def latest_visit_per_customer(visits: TableFrameSpec) -> TableFrameSpec: return visits.unique(subset=["customer_id"], keep="last") ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/latest\_visit is live — c1 is now the wed row rather than the mon one: ``` ┌─────────────┬──────┬───────┐ │ customer_id │ date │ spend │ ├─────────────┼──────┼───────┤ │ c1 │ wed │ 5 │ ├─────────────┼──────┼───────┤ │ c2 │ tue │ 7 │ └─────────────┴──────┴───────┘ ``` Transformed tables visits | customer\_id | date | spend | | --- | --- | --- | | c1 | mon | 10 | | c1 | mon | 12 | | c2 | tue | 7 | | c1 | wed | 5 | 4 rows latest\_visit | customer\_id | date | spend | | --- | --- | --- | | c1 | wed | 5 | | c2 | tue | 7 | 2 rows --- # Transform Strings > Covers Tabsdata string transformations via the str accessor: changing case, matching and extracting with contains, starts_with, ends_with and regex, replacing, trimming, padding, slicing, measuring lengths and finding positions, and parsing log lines with grok. Source: https://docs.tabsdata.com/tutorials/transformations/strings ## Change case AI agent session ❯make a new table that takes the persons table and uppercases the name column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.to_uppercase()` on the column. Aliasing it back to name replaces the column instead of adding one. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["persons"], output_tables=["persons_upper"], ) def upper_name(persons: TableFrameSpec) -> TableFrameSpec: expr = col("name").str.to_uppercase() return persons.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/persons\_upper is live — name uppercased in place: ``` ┌──────────────┬─────┐ │ name │ age │ ├──────────────┼─────┤ │ ADA LOVELACE │ 36 │ ├──────────────┼─────┤ │ GRACE HOPPER │ 45 │ └──────────────┴─────┘ ``` Transformed tables persons | name | age | | --- | --- | | ada Lovelace | 36 | | Grace HOPPER | 45 | 2 rows persons\_upper | name | age | | --- | --- | | ADA LOVELACE | 36 | | GRACE HOPPER | 45 | 2 rows AI agent session ❯make a new table that takes the persons table and lowercases the name column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.to_lowercase()`, which is the usual first step before comparing strings that users typed. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["persons"], output_tables=["persons_lower"], ) def lower_name(persons: TableFrameSpec) -> TableFrameSpec: expr = col("name").str.to_lowercase() return persons.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/persons\_lower is live — name lowercased in place: ``` ┌──────────────┬─────┐ │ name │ age │ ├──────────────┼─────┤ │ ada lovelace │ 36 │ ├──────────────┼─────┤ │ grace hopper │ 45 │ └──────────────┴─────┘ ``` Transformed tables persons | name | age | | --- | --- | | ada Lovelace | 36 | | Grace HOPPER | 45 | 2 rows persons\_lower | name | age | | --- | --- | | ada lovelace | 36 | | grace hopper | 45 | 2 rows AI agent session ❯make a new table that takes the persons table and puts the name column in title case ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.to_titlecase()` capitalises the first letter of each word and lowercases the rest. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["persons"], output_tables=["persons_titled"], ) def title_name(persons: TableFrameSpec) -> TableFrameSpec: expr = col("name").str.to_titlecase() return persons.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/persons\_titled is live — name in title case: ``` ┌──────────────┬─────┐ │ name │ age │ ├──────────────┼─────┤ │ Ada Lovelace │ 36 │ ├──────────────┼─────┤ │ Grace Hopper │ 45 │ └──────────────┴─────┘ ``` Transformed tables persons | name | age | | --- | --- | | ada Lovelace | 36 | | Grace HOPPER | 45 | 2 rows persons\_titled | name | age | | --- | --- | | Ada Lovelace | 36 | | Grace Hopper | 45 | 2 rows ## Test a string against a pattern AI agent session ❯make a new table that takes the contacts table and flags the rows whose email column ends in @example.com ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.ends_with()` returns a Boolean per row, so this adds a flag column rather than filtering. Use it inside `filter()` if you want the rows dropped. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["contacts"], output_tables=["contacts_flagged"], ) def flag_internal(contacts: TableFrameSpec) -> TableFrameSpec: expr = col("email").str.ends_with("@example.com") return contacts.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/contacts\_flagged is live — a Boolean column, true where the suffix matched: ``` ┌───────┐ │ email │ ├───────┤ │ True │ ├───────┤ │ False │ └───────┘ ``` Transformed tables contacts | email | | --- | | ada@example.com | | bo@other.org | 2 rows contacts\_flagged | email | | --- | | True | | False | 2 rows AI agent session ❯make a new table that takes the contacts table and flags the rows whose phone column starts with +1 ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.starts_with()`, the prefix counterpart. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["contacts"], output_tables=["contacts_flagged"], ) def flag_us(contacts: TableFrameSpec) -> TableFrameSpec: expr = col("phone").str.starts_with("+1") return contacts.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/contacts\_flagged is live — true where the prefix matched: ``` ┌───────┐ │ phone │ ├───────┤ │ True │ ├───────┤ │ False │ └───────┘ ``` Transformed tables contacts | phone | | --- | | +15550101 | | +445550101 | 2 rows contacts\_flagged | phone | | --- | | True | | False | 2 rows AI agent session ❯make a new table that takes the tickets table and flags the rows whose description column mentions urgent ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.contains()` with a plain string, which matches the substring anywhere in the value. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["tickets"], output_tables=["tickets_flagged"], ) def flag_urgent(tickets: TableFrameSpec) -> TableFrameSpec: expr = col("description").str.contains("urgent") return tickets.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/tickets\_flagged is live — true where the substring appears: ``` ┌─────────────┐ │ description │ ├─────────────┤ │ True │ ├─────────────┤ │ False │ └─────────────┘ ``` Transformed tables tickets | description | | --- | | urgent disk failure | | routine check | 2 rows tickets\_flagged | description | | --- | | True | | False | 2 rows AI agent session ❯make a new table that takes the lines table and flags the rows whose text column mentions error, warning or critical ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.contains_any()` takes the list and matches if any one of them appears, which beats three contains calls joined by |. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["lines"], output_tables=["lines_flagged"], ) def flag_severity(lines: TableFrameSpec) -> TableFrameSpec: expr = col("text").str.contains_any(["error", "warning", "critical"]) return lines.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/lines\_flagged is live — true where any of the three words appears: ``` ┌───────┐ │ text │ ├───────┤ │ True │ ├───────┤ │ False │ └───────┘ ``` Transformed tables lines | text | | --- | | disk error on sda | | all clear | 2 rows lines\_flagged | text | | --- | | True | | False | 2 rows AI agent session ❯make a new table that takes the parts table and flags the rows whose code column is three letters, a dash and four digits ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.contains()` also takes a regex, so an anchored pattern checks the whole shape rather than looking for a substring. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["parts"], output_tables=["parts_flagged"], ) def flag_code_shape(parts: TableFrameSpec) -> TableFrameSpec: expr = col("code").str.contains(r"^[A-Z]{3}-\d{4}$") return parts.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/parts\_flagged is live — true only where the whole value matches the pattern: ``` ┌───────┐ │ code │ ├───────┤ │ True │ ├───────┤ │ False │ └───────┘ ``` Transformed tables parts | code | | --- | | ABC-1234 | | abc-12 | 2 rows parts\_flagged | code | | --- | | True | | False | 2 rows ## Extract and replace AI agent session ❯make a new table that takes the contacts table and pulls the mail domain out of the email column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.extract()` with a capture group, and `group_index=1` picks that group rather than the whole match. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["contacts"], output_tables=["contacts_domain"], ) def extract_domain(contacts: TableFrameSpec) -> TableFrameSpec: expr = col("email").str.extract(r"@(\w+)\.", group_index=1) return contacts.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/contacts\_domain is live — the captured group, not the whole match: ``` ┌─────────┐ │ email │ ├─────────┤ │ example │ ├─────────┤ │ other │ └─────────┘ ``` Transformed tables contacts | email | | --- | | ada@example.com | | bo@other.org | 2 rows contacts\_domain | email | | --- | | example | | other | 2 rows AI agent session ❯make a new table that takes the notes table and replaces the first "old" in the text column with "new" ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.replace()` changes the first occurrence only. `replace_all` is the one that changes every occurrence. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["notes"], output_tables=["notes_replaced"], ) def replace_first(notes: TableFrameSpec) -> TableFrameSpec: expr = col("text").str.replace("old", "new") return notes.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/notes\_replaced is live — only the first occurrence in each value changed: ``` ┌───────────────────────┐ │ text │ ├───────────────────────┤ │ new habits die hard │ ├───────────────────────┤ │ no change here │ └───────────────────────┘ ``` Transformed tables notes | text | | --- | | old habits die hard | | no change here | 2 rows notes\_replaced | text | | --- | | new habits die hard | | no change here | 2 rows AI agent session ❯make a new table that takes the notes table and collapses every run of whitespace in the text column to a single space ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.replace_all()` with a regex, which is the normal way to normalise whitespace. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["notes"], output_tables=["notes_squeezed"], ) def squeeze_spaces(notes: TableFrameSpec) -> TableFrameSpec: expr = col("text").str.replace_all(r"\s+", " ") # Replace all whitespace return notes.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/notes\_squeezed is live — every whitespace run collapsed: ``` ┌─────────────────────┐ │ text │ ├─────────────────────┤ │ old habits die hard │ ├─────────────────────┤ │ no change here │ └─────────────────────┘ ``` Transformed tables notes | text | | --- | | old habits die hard | | no change here | 2 rows notes\_squeezed | text | | --- | | old habits die hard | | no change here | 2 rows AI agent session ❯make a new table that takes the notes table and replaces a, b and c with x, y and z in the text column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.replace_many()` takes two lists, matched up by position, and makes one pass rather than one per pair. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["notes"], output_tables=["notes_replaced"], ) def replace_many(notes: TableFrameSpec) -> TableFrameSpec: expr = col("text").str.replace_many(["a", "b", "c"], ["x", "y", "z"]) return notes.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/notes\_replaced is live — all three substitutions applied in one pass: ``` ┌──────────┐ │ text │ ├──────────┤ │ xyz zxy │ ├──────────┤ │ no mxtzh │ └──────────┘ ``` Transformed tables notes | text | | --- | | abc cab | | no match | 2 rows notes\_replaced | text | | --- | | xyz zxy | | no mxtzh | 2 rows ## Trim, pad and slice AI agent session ❯make a new table that takes the people table and trims the whitespace off both ends of the name column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.strip_chars()` with no argument trims whitespace from both ends and leaves the inside alone. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["people"], output_tables=["people_trimmed"], ) def trim_name(people: TableFrameSpec) -> TableFrameSpec: expr = col("name").str.strip_chars() # Both sides return people.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/people\_trimmed is live — padding gone from both ends: ``` ┌───────┐ │ name │ ├───────┤ │ Ada │ ├───────┤ │ Grace │ └───────┘ ``` Transformed tables people | name | | --- | | Ada | | Grace | 2 rows people\_trimmed | name | | --- | | Ada | | Grace | 2 rows AI agent session ❯make a new table that takes the people table and trims the whitespace off the start of the name column only ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.strip_chars_start()` touches the left side only. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["people"], output_tables=["people_trimmed"], ) def trim_start(people: TableFrameSpec) -> TableFrameSpec: expr = col("name").str.strip_chars_start() # Left side return people.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/people\_trimmed is live — leading whitespace gone, trailing kept: ``` ┌─────────┐ │ name │ ├─────────┤ │ Ada │ ├─────────┤ │ Grace │ └─────────┘ ``` Transformed tables people | name | | --- | | Ada | | Grace | 2 rows people\_trimmed | name | | --- | | Ada | | Grace | 2 rows AI agent session ❯make a new table that takes the people table and trims the whitespace off the end of the name column only ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.strip_chars_end()` touches the right side only. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["people"], output_tables=["people_trimmed"], ) def trim_end(people: TableFrameSpec) -> TableFrameSpec: expr = col("name").str.strip_chars_end() # Right side return people.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/people\_trimmed is live — trailing whitespace gone, leading kept: ``` ┌─────────┐ │ name │ ├─────────┤ │ Ada │ ├─────────┤ │ Grace │ └─────────┘ ``` Transformed tables people | name | | --- | | Ada | | Grace | 2 rows people\_trimmed | name | | --- | | Ada | | Grace | 2 rows AI agent session ❯make a new table that takes the routes table and removes the /api/ prefix from the path column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.strip_prefix()` removes that exact prefix and leaves a value that does not start with it untouched. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["routes"], output_tables=["routes_trimmed"], ) def drop_api_prefix(routes: TableFrameSpec) -> TableFrameSpec: expr = col("path").str.strip_prefix("/api/") return routes.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/routes\_trimmed is live — the prefix removed where present: ``` ┌─────────┐ │ path │ ├─────────┤ │ users │ ├─────────┤ │ /health │ └─────────┘ ``` Transformed tables routes | path | | --- | | /api/users | | /health | 2 rows routes\_trimmed | path | | --- | | users | | /health | 2 rows AI agent session ❯make a new table that takes the files table and removes the .txt extension from the filename column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.strip_suffix()`, the same idea at the end of the value. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["files"], output_tables=["files_trimmed"], ) def drop_extension(files: TableFrameSpec) -> TableFrameSpec: expr = col("filename").str.strip_suffix(".txt") return files.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/files\_trimmed is live — the extension removed: ``` ┌───────────┐ │ filename │ ├───────────┤ │ notes │ ├───────────┤ │ image.png │ └───────────┘ ``` Transformed tables files | filename | | --- | | notes.txt | | image.png | 2 rows files\_trimmed | filename | | --- | | notes | | image.png | 2 rows AI agent session ❯make a new table that takes the orders table and pads the id column out to 10 characters with leading zeros ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.pad_start(10, "0")` pads on the left up to that width and leaves anything already longer alone. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["orders"], output_tables=["orders_padded"], ) def pad_id(orders: TableFrameSpec) -> TableFrameSpec: expr = col("id").str.pad_start(10, "0") # Left pad with zeros return orders.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/orders\_padded is live — ids padded to width 10: ``` ┌────────────┐ │ id │ ├────────────┤ │ 0000000042 │ ├────────────┤ │ 0000001337 │ └────────────┘ ``` Transformed tables orders | id | | --- | | 42 | | 1337 | 2 rows orders\_padded | id | | --- | | 0000000042 | | 0000001337 | 2 rows AI agent session ❯make a new table that takes the parts table and pads the code column out to 20 characters with trailing spaces ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.pad_end()` pads on the right, which is what fixed-width output wants. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["parts"], output_tables=["parts_padded"], ) def pad_code(parts: TableFrameSpec) -> TableFrameSpec: expr = col("code").str.pad_end(20, " ") # Right pad with spaces return parts.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/parts\_padded is live — codes padded to width 20: ``` ┌──────────────────────┐ │ code │ ├──────────────────────┤ │ ABC │ ├──────────────────────┤ │ DEFGH │ └──────────────────────┘ ``` Transformed tables parts | code | | --- | | ABC | | DEFGH | 2 rows parts\_padded | code | | --- | | ABC | | DEFGH | 2 rows AI agent session ❯make a new table that takes the invoices table and zero-fills the number column to 5 characters ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.zfill()` is pad\_start with zeros, except it keeps a leading sign in front of the padding. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["invoices"], output_tables=["invoices_filled"], ) def zfill_number(invoices: TableFrameSpec) -> TableFrameSpec: expr = col("number").str.zfill(5) # Zero-fill to length return invoices.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/invoices\_filled is live — numbers zero-filled to width 5: ``` ┌────────┐ │ number │ ├────────┤ │ 00042 │ ├────────┤ │ 01337 │ └────────┘ ``` Transformed tables invoices | number | | --- | | 42 | | 1337 | 2 rows invoices\_filled | number | | --- | | 00042 | | 01337 | 2 rows AI agent session ❯make a new table that takes the notes table and keeps only the first 10 characters of the text column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.head(10)` takes from the front and returns the whole value when it is shorter. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["notes"], output_tables=["notes_head"], ) def head_text(notes: TableFrameSpec) -> TableFrameSpec: expr = col("text").str.head(10) # First 10 characters return notes.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/notes\_head is live — first 10 characters of each value: ``` ┌────────────┐ │ text │ ├────────────┤ │ abcdefghij │ ├────────────┤ │ short │ └────────────┘ ``` Transformed tables notes | text | | --- | | abcdefghijklmnop | | short | 2 rows notes\_head | text | | --- | | abcdefghij | | short | 2 rows AI agent session ❯make a new table that takes the notes table and keeps only the last 5 characters of the text column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.tail(5)` takes from the end. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["notes"], output_tables=["notes_tail"], ) def tail_text(notes: TableFrameSpec) -> TableFrameSpec: expr = col("text").str.tail(5) # Last 5 characters return notes.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/notes\_tail is live — last 5 characters of each value: ``` ┌───────┐ │ text │ ├───────┤ │ lmnop │ ├───────┤ │ short │ └───────┘ ``` Transformed tables notes | text | | --- | | abcdefghijklmnop | | short | 2 rows notes\_tail | text | | --- | | lmnop | | short | 2 rows AI agent session ❯make a new table that takes the notes table and takes 10 characters of the text column starting at position 5 ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.slice()` takes an offset and a length, and the offset is 0-based. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["notes"], output_tables=["notes_sliced"], ) def slice_text(notes: TableFrameSpec) -> TableFrameSpec: expr = col("text").str.slice(offset=5, length=10) # Substring return notes.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/notes\_sliced is live — 10 characters from offset 5: ``` ┌────────────┐ │ text │ ├────────────┤ │ fghijklmno │ ├────────────┤ │ │ └────────────┘ ``` Transformed tables notes | text | | --- | | abcdefghijklmnop | | short | 2 rows notes\_sliced | text | | --- | | fghijklmno | | | 2 rows AI agent session ❯make a new table that takes the notes table and reverses the text column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.reverse()` reverses by character, not by byte, so multi-byte characters survive. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["notes"], output_tables=["notes_reversed"], ) def reverse_text(notes: TableFrameSpec) -> TableFrameSpec: expr = col("text").str.reverse() return notes.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/notes\_reversed is live — each value reversed: ``` ┌──────────────────┐ │ text │ ├──────────────────┤ │ ponmlkjihgfedcba │ ├──────────────────┤ │ trohs │ └──────────────────┘ ``` Transformed tables notes | text | | --- | | abcdefghijklmnop | | short | 2 rows notes\_reversed | text | | --- | | ponmlkjihgfedcba | | trohs | 2 rows ## Measure and locate AI agent session ❯make a new table that takes the tickets table and measures the character length of the description column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.len_chars()` counts characters, which is what a person means by length. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["tickets"], output_tables=["tickets_lengths"], ) def measure_chars(tickets: TableFrameSpec) -> TableFrameSpec: expr = col("description").str.len_chars() # Character count return tickets.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/tickets\_lengths is live — the character count per row: ``` ┌─────────────┐ │ description │ ├─────────────┤ │ 12 │ ├─────────────┤ │ 2 │ └─────────────┘ ``` Transformed tables tickets | description | | --- | | disk failure | | ok | 2 rows tickets\_lengths | description | | --- | | 12 | | 2 | 2 rows AI agent session ❯make a new table that takes the payloads table and measures the byte length of the data column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.len_bytes()` counts bytes, so a multi-byte character counts more than once. This is the one for storage limits. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["payloads"], output_tables=["payloads_lengths"], ) def measure_bytes(payloads: TableFrameSpec) -> TableFrameSpec: expr = col("data").str.len_bytes() # Byte count return payloads.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/payloads\_lengths is live — the byte count per row, which differs from the character count on non-ASCII values: ``` ┌──────┐ │ data │ ├──────┤ │ 5 │ ├──────┤ │ 2 │ └──────┘ ``` Transformed tables payloads | data | | --- | | café | | ok | 2 rows payloads\_lengths | data | | --- | | 5 | | 2 | 2 rows AI agent session ❯make a new table that takes the lines table and finds the position of the first @ in the text column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.find()` returns the index of the first occurrence, 0-based, and null when there is no match. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["lines"], output_tables=["lines_positions"], ) def find_at(lines: TableFrameSpec) -> TableFrameSpec: expr = col("text").str.find("@") # First occurrence index return lines.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/lines\_positions is live — the index of the first match, null where absent: ``` ┌──────┐ │ text │ ├──────┤ │ 3 │ ├──────┤ │ None │ └──────┘ ``` Transformed tables lines | text | | --- | | ada@example.com | | no at sign | 2 rows lines\_positions | text | | --- | | 3 | | None | 2 rows AI agent session ❯make a new table that takes the lines table and counts how many times error appears in the log column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.count_matches()` with a word-boundary regex, so it counts the word rather than the substring inside a longer word. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["lines"], output_tables=["lines_counts"], ) def count_errors(lines: TableFrameSpec) -> TableFrameSpec: expr = col("log").str.count_matches(r"\berror\b") # Count pattern matches return lines.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/lines\_counts is live — the match count per row: ``` ┌─────┐ │ log │ ├─────┤ │ 2 │ ├─────┤ │ 0 │ └─────┘ ``` Transformed tables lines | log | | --- | | error then error again | | all clear | 2 rows lines\_counts | log | | --- | | 2 | | 0 | 2 rows ## Parse a log line AI agent session ❯make a new table that takes the access table and parses the log\_line column into named columns ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `grok()` on the frame rather than an expression on one column: a grok pattern produces several columns at once, so it is a frame-level operation. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["access"], output_tables=["access_parsed"], ) def parse_access(access: TableFrameSpec) -> TableFrameSpec: return access.grok( "log_line", r"%{IP:client_ip} - - \[%{HTTPDATE:timestamp}\] \"%{WORD:method} %{URIPATHPARAM:path}\"" ) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/access\_parsed is live — the log line broken into the named fields: ``` ┌──────────┬───────────┬───────────┬────────┬──────┐ │ log_line │ client_ip │ timestamp │ method │ path │ ├──────────┼───────────┼───────────┼────────┼──────┤ │ │ │ │ │ │ └──────────┴───────────┴───────────┴────────┴──────┘ ``` Transformed tables access | log\_line | | --- | | 10.0.0.1 - - \[12/Mar/2024:09:15:00 +0000\] "GET /users" | 1 row access\_parsed | log\_line | client\_ip | timestamp | method | path | | --- | --- | --- | --- | --- | | 10.0.0.1 - - \[12/Mar/2024:09:15:00 +0000\] "GET /users" | 10.0.0.1 | 12/Mar/2024:09:15:00 +0000 | GET | /users | 1 row --- # Transform Numbers > Arithmetic, rounding, powers, logs, trigonometry, and row-to-row differences for Tabsdata table columns, with the transformer code for each operator and method. Source: https://docs.tabsdata.com/tutorials/transformations/numbers ## Arithmetic AI agent session ❯make a new table that takes the orders table and adds 10 to the price column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — the `+` operator works directly on a column expression, so no method call is needed. `add()` is the method form and does the same thing. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["orders"], output_tables=["orders_bumped"], ) def bump_price(orders: TableFrameSpec) -> TableFrameSpec: expr = col("price") + 10 return orders.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/orders\_bumped is live — 10 added to every price: ``` ┌───────┐ │ price │ ├───────┤ │ 110 │ ├───────┤ │ 50 │ └───────┘ ``` Transformed tables orders | price | | --- | | 100 | | 40 | 2 rows orders\_bumped | price | | --- | | 110 | | 50 | 2 rows AI agent session ❯make a new table that takes the orders table and subtracts the discount column from the price column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `-` between two columns subtracts row by row. `sub()` is the method form. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["orders"], output_tables=["orders_net"], ) def net_price(orders: TableFrameSpec) -> TableFrameSpec: expr = col("price") - col("discount") return orders.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/orders\_net is live — discount taken off each price: ``` ┌───────┬──────────┐ │ price │ discount │ ├───────┼──────────┤ │ 85 │ 15 │ ├───────┼──────────┤ │ 35 │ 5 │ └───────┴──────────┘ ``` Transformed tables orders | price | discount | | --- | --- | | 100 | 15 | | 40 | 5 | 2 rows orders\_net | price | discount | | --- | --- | | 85 | 15 | | 35 | 5 | 2 rows AI agent session ❯make a new table that takes the orders table and multiplies the quantity column by the unit\_price column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `*` between two columns multiplies row by row. `mul()` is the method form. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["orders"], output_tables=["orders_totalled"], ) def line_total(orders: TableFrameSpec) -> TableFrameSpec: expr = col("quantity") * col("unit_price") return orders.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/orders\_totalled is live — quantity times unit price per row: ``` ┌──────────┬────────────┐ │ quantity │ unit_price │ ├──────────┼────────────┤ │ 30 │ 10 │ ├──────────┼────────────┤ │ 9 │ 4.5 │ └──────────┴────────────┘ ``` Transformed tables orders | quantity | unit\_price | | --- | --- | | 3 | 10 | | 2 | 4.5 | 2 rows orders\_totalled | quantity | unit\_price | | --- | --- | | 30 | 10 | | 9 | 4.5 | 2 rows AI agent session ❯make a new table that takes the orders table and divides the total column by the count column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `/` divides row by row and yields a float. `truediv()` is the method form. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["orders"], output_tables=["orders_per_unit"], ) def per_unit(orders: TableFrameSpec) -> TableFrameSpec: expr = col("total") / col("count") return orders.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/orders\_per\_unit is live — total divided by count per row: ``` ┌───────┬───────┐ │ total │ count │ ├───────┼───────┤ │ 30 │ 3 │ ├───────┼───────┤ │ 12.5 │ 4 │ └───────┴───────┘ ``` Transformed tables orders | total | count | | --- | --- | | 90 | 3 | | 50 | 4 | 2 rows orders\_per\_unit | total | count | | --- | --- | | 30 | 3 | | 12.5 | 4 | 2 rows AI agent session ❯make a new table that takes the pairs table and adds the b column to the a column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `add()` is the method form of `+`, and it reads better when you are chaining several operations off one expression. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["pairs"], output_tables=["pairs_sum"], ) def add_columns(pairs: TableFrameSpec) -> TableFrameSpec: expr = col("a").add(col("b")) return pairs.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/pairs\_sum is live — a plus b per row: ``` ┌────┬───┐ │ a │ b │ ├────┼───┤ │ 9 │ 2 │ ├────┼───┤ │ 13 │ 4 │ └────┴───┘ ``` Transformed tables pairs | a | b | | --- | --- | | 7 | 2 | | 9 | 4 | 2 rows pairs\_sum | a | b | | --- | --- | | 9 | 2 | | 13 | 4 | 2 rows AI agent session ❯make a new table that takes the pairs table and subtracts the b column from the a column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `sub()`, the method form of `-`. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["pairs"], output_tables=["pairs_difference"], ) def sub_columns(pairs: TableFrameSpec) -> TableFrameSpec: expr = col("a").sub(col("b")) return pairs.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/pairs\_difference is live — a minus b per row: ``` ┌───┬───┐ │ a │ b │ ├───┼───┤ │ 5 │ 2 │ ├───┼───┤ │ 5 │ 4 │ └───┴───┘ ``` Transformed tables pairs | a | b | | --- | --- | | 7 | 2 | | 9 | 4 | 2 rows pairs\_difference | a | b | | --- | --- | | 5 | 2 | | 5 | 4 | 2 rows AI agent session ❯make a new table that takes the pairs table and multiplies the a column by the b column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `mul()`, the method form of `*`. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["pairs"], output_tables=["pairs_product"], ) def mul_columns(pairs: TableFrameSpec) -> TableFrameSpec: expr = col("a").mul(col("b")) return pairs.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/pairs\_product is live — a times b per row: ``` ┌────┬───┐ │ a │ b │ ├────┼───┤ │ 14 │ 2 │ ├────┼───┤ │ 36 │ 4 │ └────┴───┘ ``` Transformed tables pairs | a | b | | --- | --- | | 7 | 2 | | 9 | 4 | 2 rows pairs\_product | a | b | | --- | --- | | 14 | 2 | | 36 | 4 | 2 rows AI agent session ❯make a new table that takes the pairs table and divides the a column by the b column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `truediv()` is the method form of `/`, and it always produces a float rather than truncating. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["pairs"], output_tables=["pairs_quotient"], ) def div_columns(pairs: TableFrameSpec) -> TableFrameSpec: expr = col("a").truediv(col("b")) return pairs.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/pairs\_quotient is live — a divided by b per row: ``` ┌──────┬───┐ │ a │ b │ ├──────┼───┤ │ 3.5 │ 2 │ ├──────┼───┤ │ 2.25 │ 4 │ └──────┴───┘ ``` Transformed tables pairs | a | b | | --- | --- | | 7 | 2 | | 9 | 4 | 2 rows pairs\_quotient | a | b | | --- | --- | | 3.5 | 2 | | 2.25 | 4 | 2 rows AI agent session ❯make a new table that takes the pairs table and takes the remainder of the a column divided by the b column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `mod()` gives the remainder, which is the operation to reach for when you want "every nth row" or a bucket index. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["pairs"], output_tables=["pairs_remainder"], ) def mod_columns(pairs: TableFrameSpec) -> TableFrameSpec: expr = col("a").mod(col("b")) # Modulo return pairs.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/pairs\_remainder is live — the remainder per row: ``` ┌───┬───┐ │ a │ b │ ├───┼───┤ │ 1 │ 2 │ ├───┼───┤ │ 1 │ 4 │ └───┴───┘ ``` Transformed tables pairs | a | b | | --- | --- | | 7 | 2 | | 9 | 4 | 2 rows pairs\_remainder | a | b | | --- | --- | | 1 | 2 | | 1 | 4 | 2 rows ## Round and clip AI agent session ❯make a new table that takes the items table and rounds the price column to 2 decimal places ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `round(2)` rounds to that many decimal places. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["items"], output_tables=["items_rounded"], ) def round_price(items: TableFrameSpec) -> TableFrameSpec: expr = col("price").round(2) # Round to 2 decimal places return items.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/items\_rounded is live — price to 2 decimal places: ``` ┌────────┬───────┐ │ item │ price │ ├────────┼───────┤ │ widget │ 19.99 │ ├────────┼───────┤ │ gadget │ 4.2 │ └────────┴───────┘ ``` Transformed tables items | item | price | | --- | --- | | widget | 19.9876 | | gadget | 4.201 | 2 rows items\_rounded | item | price | | --- | --- | | widget | 19.99 | | gadget | 4.2 | 2 rows AI agent session ❯make a new table that takes the readings table and rounds the value column to 3 significant figures ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `round_sig_figs(3)` counts significant digits rather than decimal places, so the number of decimals varies with magnitude. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["readings"], output_tables=["readings_rounded"], ) def round_value(readings: TableFrameSpec) -> TableFrameSpec: expr = col("value").round_sig_figs(3) # Round to 3 significant figures return readings.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/readings\_rounded is live — 3 significant figures, so the decimal count differs per row: ``` ┌──────────┐ │ value │ ├──────────┤ │ 1230 │ ├──────────┤ │ 0.000988 │ └──────────┘ ``` Transformed tables readings | value | | --- | | 1234.5678 | | 0.00098765 | 2 rows readings\_rounded | value | | --- | | 1230 | | 0.000988 | 2 rows AI agent session ❯make a new table that takes the orders table and rounds the amount column down to a whole number ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `floor()` always goes down, which for a negative number means away from zero. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["orders"], output_tables=["orders_floored"], ) def floor_amount(orders: TableFrameSpec) -> TableFrameSpec: expr = col("amount").floor() # Round down return orders.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/orders\_floored is live — each amount rounded down: ``` ┌────────┐ │ amount │ ├────────┤ │ 19 │ ├────────┤ │ -4 │ └────────┘ ``` Transformed tables orders | amount | | --- | | 19.7 | | \-3.2 | 2 rows orders\_floored | amount | | --- | | 19 | | \-4 | 2 rows AI agent session ❯make a new table that takes the orders table and rounds the amount column up to a whole number ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `ceil()` always goes up. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["orders"], output_tables=["orders_ceiled"], ) def ceil_amount(orders: TableFrameSpec) -> TableFrameSpec: expr = col("amount").ceil() # Round up return orders.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/orders\_ceiled is live — each amount rounded up: ``` ┌────────┐ │ amount │ ├────────┤ │ 20 │ ├────────┤ │ -3 │ └────────┘ ``` Transformed tables orders | amount | | --- | | 19.7 | | \-3.2 | 2 rows orders\_ceiled | amount | | --- | | 20 | | \-3 | 2 rows AI agent session ❯make a new table that takes the results table and clamps the score column between 0 and 100 ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `clip()` pulls anything outside the bounds to the nearest bound rather than dropping the row. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["results"], output_tables=["results_clipped"], ) def clip_score(results: TableFrameSpec) -> TableFrameSpec: expr = col("score").clip(lower_bound=0, upper_bound=100) return results.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/results\_clipped is live — values outside 0 to 100 pulled to the bound: ``` ┌───────┐ │ score │ ├───────┤ │ 100 │ ├───────┤ │ 0 │ ├───────┤ │ 73 │ └───────┘ ``` Transformed tables results | score | | --- | | 142 | | \-8 | | 73 | 3 rows results\_clipped | score | | --- | | 100 | | 0 | | 73 | 3 rows ## Powers, roots and logs AI agent session ❯make a new table that takes the readings table and takes the absolute value of the change column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `abs()` drops the sign. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["readings"], output_tables=["readings_absolute"], ) def abs_change(readings: TableFrameSpec) -> TableFrameSpec: expr = col("change").abs() return readings.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/readings\_absolute is live — the sign dropped: ``` ┌────────┐ │ change │ ├────────┤ │ 4.5 │ ├────────┤ │ 2 │ └────────┘ ``` Transformed tables readings | change | | --- | | \-4.5 | | 2 | 2 rows readings\_absolute | change | | --- | | 4.5 | | 2 | 2 rows AI agent session ❯make a new table that takes the readings table and takes the sign of the value column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `sign()` returns -1, 0 or 1 rather than the value itself. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["readings"], output_tables=["readings_signs"], ) def sign_value(readings: TableFrameSpec) -> TableFrameSpec: expr = col("value").sign() # Returns -1, 0, or 1 return readings.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/readings\_signs is live — -1, 0 or 1 per row: ``` ┌───────┐ │ value │ ├───────┤ │ -1 │ ├───────┤ │ 0 │ ├───────┤ │ 1 │ └───────┘ ``` Transformed tables readings | value | | --- | | \-4 | | 0 | | 7 | 3 rows readings\_signs | value | | --- | | \-1 | | 0 | | 1 | 3 rows AI agent session ❯make a new table that takes the readings table and negates the value column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `neg()` flips the sign of every value. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["readings"], output_tables=["readings_negated"], ) def neg_value(readings: TableFrameSpec) -> TableFrameSpec: expr = col("value").neg() # Negate return readings.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/readings\_negated is live — every sign flipped: ``` ┌───────┐ │ value │ ├───────┤ │ 4 │ ├───────┤ │ -7 │ └───────┘ ``` Transformed tables readings | value | | --- | | \-4 | | 7 | 2 rows readings\_negated | value | | --- | | 4 | | \-7 | 2 rows AI agent session ❯make a new table that takes the values table and squares the x column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `pow(2)` raises the column to that power. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["values"], output_tables=["values_squared"], ) def square_x(values: TableFrameSpec) -> TableFrameSpec: expr = col("x").pow(2) # Square return values.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/values\_squared is live — each value squared: ``` ┌─────┐ │ x │ ├─────┤ │ 64 │ ├─────┤ │ 729 │ └─────┘ ``` Transformed tables values | x | | --- | | 8 | | 27 | 2 rows values\_squared | x | | --- | | 64 | | 729 | 2 rows AI agent session ❯make a new table that takes the values table and takes the square root of the x column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `sqrt()`. A negative input gives NaN rather than an error, so a `filter` or a `clip` first is worth considering. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["values"], output_tables=["values_rooted"], ) def sqrt_x(values: TableFrameSpec) -> TableFrameSpec: expr = col("x").sqrt() # Square root return values.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/values\_rooted is live — the square root per row: ``` ┌────────┐ │ x │ ├────────┤ │ 3 │ ├────────┤ │ 1.4142 │ └────────┘ ``` Transformed tables values | x | | --- | | 9 | | 2 | 2 rows values\_rooted | x | | --- | | 3 | | 1.4142 | 2 rows AI agent session ❯make a new table that takes the values table and takes the cube root of the x column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `cbrt()`, which unlike sqrt is defined for negative inputs. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["values"], output_tables=["values_rooted"], ) def cbrt_x(values: TableFrameSpec) -> TableFrameSpec: expr = col("x").cbrt() # Cube root return values.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/values\_rooted is live — the cube root per row: ``` ┌───┐ │ x │ ├───┤ │ 3 │ ├───┤ │ 2 │ └───┘ ``` Transformed tables values | x | | --- | | 27 | | 8 | 2 rows values\_rooted | x | | --- | | 3 | | 2 | 2 rows AI agent session ❯make a new table that takes the values table and takes the natural log of the x column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `log()` with no argument is the natural log, base e. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["values"], output_tables=["values_logged"], ) def log_x(values: TableFrameSpec) -> TableFrameSpec: expr = col("x").log() # Natural logarithm return values.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/values\_logged is live — the natural log per row: ``` ┌────────┐ │ x │ ├────────┤ │ 0 │ ├────────┤ │ 2.3026 │ └────────┘ ``` Transformed tables values | x | | --- | | 1 | | 10 | 2 rows values\_logged | x | | --- | | 0 | | 2.3026 | 2 rows AI agent session ❯make a new table that takes the values table and takes the base-10 log of the x column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `log10()`, which is the one you want for orders of magnitude. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["values"], output_tables=["values_logged"], ) def log10_x(values: TableFrameSpec) -> TableFrameSpec: expr = col("x").log10() # Base-10 logarithm return values.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/values\_logged is live — the base-10 log per row: ``` ┌───┐ │ x │ ├───┤ │ 0 │ ├───┤ │ 1 │ └───┘ ``` Transformed tables values | x | | --- | | 1 | | 10 | 2 rows values\_logged | x | | --- | | 0 | | 1 | 2 rows AI agent session ❯make a new table that takes the values table and takes the log of 1 plus the x column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `log1p()` rather than log(1 + x). It keeps precision when x is very small, where the addition would lose it. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["values"], output_tables=["values_logged"], ) def log1p_x(values: TableFrameSpec) -> TableFrameSpec: expr = col("x").log1p() # log(1 + x) return values.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/values\_logged is live — log(1 + x) per row: ``` ┌────────┐ │ x │ ├────────┤ │ 0 │ ├────────┤ │ 0.6931 │ └────────┘ ``` Transformed tables values | x | | --- | | 0 | | 1 | 2 rows values\_logged | x | | --- | | 0 | | 0.6931 | 2 rows AI agent session ❯make a new table that takes the values table and raises e to the power of the x column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `exp()`, the inverse of `log()`. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["values"], output_tables=["values_exponentiated"], ) def exp_x(values: TableFrameSpec) -> TableFrameSpec: expr = col("x").exp() # e^x return values.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/values\_exponentiated is live — e raised to each value: ``` ┌────────┐ │ x │ ├────────┤ │ 1 │ ├────────┤ │ 2.7183 │ └────────┘ ``` Transformed tables values | x | | --- | | 0 | | 1 | 2 rows values\_exponentiated | x | | --- | | 1 | | 2.7183 | 2 rows ## Trigonometry AI agent session ❯make a new table that takes the readings table and takes the sine of the angle column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `sin()`. The input is read in radians, so use `radians()` first if the column holds degrees. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["readings"], output_tables=["readings_sine"], ) def sin_angle(readings: TableFrameSpec) -> TableFrameSpec: expr = col("angle").sin() return readings.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/readings\_sine is live — the sine of each angle: ``` ┌────────┐ │ angle │ ├────────┤ │ 0 │ ├────────┤ │ 0.8415 │ └────────┘ ``` Transformed tables readings | angle | | --- | | 0 | | 1 | 2 rows readings\_sine | angle | | --- | | 0 | | 0.8415 | 2 rows AI agent session ❯make a new table that takes the readings table and takes the cosine of the angle column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `cos()`, also in radians. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["readings"], output_tables=["readings_cosine"], ) def cos_angle(readings: TableFrameSpec) -> TableFrameSpec: expr = col("angle").cos() return readings.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/readings\_cosine is live — the cosine of each angle: ``` ┌────────┐ │ angle │ ├────────┤ │ 1 │ ├────────┤ │ 0.5403 │ └────────┘ ``` Transformed tables readings | angle | | --- | | 0 | | 1 | 2 rows readings\_cosine | angle | | --- | | 1 | | 0.5403 | 2 rows AI agent session ❯make a new table that takes the readings table and takes the tangent of the angle column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `tan()`, in radians. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["readings"], output_tables=["readings_tangent"], ) def tan_angle(readings: TableFrameSpec) -> TableFrameSpec: expr = col("angle").tan() return readings.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/readings\_tangent is live — the tangent of each angle: ``` ┌────────┐ │ angle │ ├────────┤ │ 0 │ ├────────┤ │ 1.5574 │ └────────┘ ``` Transformed tables readings | angle | | --- | | 0 | | 1 | 2 rows readings\_tangent | angle | | --- | | 0 | | 1.5574 | 2 rows AI agent session ❯make a new table that takes the readings table and takes the cotangent of the angle column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `cot()`, the reciprocal of tan. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["readings"], output_tables=["readings_cotangent"], ) def cot_angle(readings: TableFrameSpec) -> TableFrameSpec: expr = col("angle").cot() return readings.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/readings\_cotangent is live — the cotangent of each angle: ``` ┌─────────┐ │ angle │ ├─────────┤ │ 0.6421 │ ├─────────┤ │ -0.4577 │ └─────────┘ ``` Transformed tables readings | angle | | --- | | 1 | | 2 | 2 rows readings\_cotangent | angle | | --- | | 0.6421 | | \-0.4577 | 2 rows AI agent session ❯make a new table that takes the readings table and takes the arcsine of the x column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `arcsin()` returns radians, and its input has to be between -1 and 1. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["readings"], output_tables=["readings_arcsine"], ) def arcsin_x(readings: TableFrameSpec) -> TableFrameSpec: expr = col("x").arcsin() return readings.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/readings\_arcsine is live — the arcsine in radians: ``` ┌────────┐ │ x │ ├────────┤ │ 0 │ ├────────┤ │ 0.5236 │ └────────┘ ``` Transformed tables readings | x | | --- | | 0 | | 0.5 | 2 rows readings\_arcsine | x | | --- | | 0 | | 0.5236 | 2 rows AI agent session ❯make a new table that takes the readings table and takes the arccosine of the x column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `arccos()`, same domain limit as arcsin. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["readings"], output_tables=["readings_arccosine"], ) def arccos_x(readings: TableFrameSpec) -> TableFrameSpec: expr = col("x").arccos() return readings.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/readings\_arccosine is live — the arccosine in radians: ``` ┌────────┐ │ x │ ├────────┤ │ 1.5708 │ ├────────┤ │ 1.0472 │ └────────┘ ``` Transformed tables readings | x | | --- | | 0 | | 0.5 | 2 rows readings\_arccosine | x | | --- | | 1.5708 | | 1.0472 | 2 rows AI agent session ❯make a new table that takes the readings table and takes the arctangent of the x column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `arctan()`, which accepts any real input. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["readings"], output_tables=["readings_arctangent"], ) def arctan_x(readings: TableFrameSpec) -> TableFrameSpec: expr = col("x").arctan() return readings.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/readings\_arctangent is live — the arctangent in radians: ``` ┌────────┐ │ x │ ├────────┤ │ 0 │ ├────────┤ │ 0.4636 │ └────────┘ ``` Transformed tables readings | x | | --- | | 0 | | 0.5 | 2 rows readings\_arctangent | x | | --- | | 0 | | 0.4636 | 2 rows AI agent session ❯make a new table that takes the readings table and takes the hyperbolic sine of the x column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `sinh()`, the hyperbolic form rather than the circular one. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["readings"], output_tables=["readings_hyperbolic"], ) def sinh_x(readings: TableFrameSpec) -> TableFrameSpec: expr = col("x").sinh() return readings.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/readings\_hyperbolic is live — the hyperbolic sine: ``` ┌────────┐ │ x │ ├────────┤ │ 0 │ ├────────┤ │ 0.5211 │ └────────┘ ``` Transformed tables readings | x | | --- | | 0 | | 0.5 | 2 rows readings\_hyperbolic | x | | --- | | 0 | | 0.5211 | 2 rows AI agent session ❯make a new table that takes the readings table and takes the hyperbolic cosine of the x column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `cosh()`. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["readings"], output_tables=["readings_hyperbolic"], ) def cosh_x(readings: TableFrameSpec) -> TableFrameSpec: expr = col("x").cosh() return readings.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/readings\_hyperbolic is live — the hyperbolic cosine: ``` ┌────────┐ │ x │ ├────────┤ │ 1 │ ├────────┤ │ 1.1276 │ └────────┘ ``` Transformed tables readings | x | | --- | | 0 | | 0.5 | 2 rows readings\_hyperbolic | x | | --- | | 1 | | 1.1276 | 2 rows AI agent session ❯make a new table that takes the readings table and takes the hyperbolic tangent of the x column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `tanh()`, which flattens any input into the range -1 to 1. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["readings"], output_tables=["readings_hyperbolic"], ) def tanh_x(readings: TableFrameSpec) -> TableFrameSpec: expr = col("x").tanh() return readings.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/readings\_hyperbolic is live — the hyperbolic tangent: ``` ┌────────┐ │ x │ ├────────┤ │ 0 │ ├────────┤ │ 0.4621 │ └────────┘ ``` Transformed tables readings | x | | --- | | 0 | | 0.5 | 2 rows readings\_hyperbolic | x | | --- | | 0 | | 0.4621 | 2 rows AI agent session ❯make a new table that takes the readings table and converts the degrees column to radians ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `radians()`. Every trig function here takes radians, so this is usually the step before them. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["readings"], output_tables=["readings_radians"], ) def to_radians(readings: TableFrameSpec) -> TableFrameSpec: expr = col("degrees").radians() # Degrees to radians return readings.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/readings\_radians is live — degrees converted to radians: ``` ┌─────────┐ │ degrees │ ├─────────┤ │ 3.1416 │ ├─────────┤ │ 1.5708 │ └─────────┘ ``` Transformed tables readings | degrees | | --- | | 180 | | 90 | 2 rows readings\_radians | degrees | | --- | | 3.1416 | | 1.5708 | 2 rows AI agent session ❯make a new table that takes the readings table and converts the radians column to degrees ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `degrees()`, the conversion back for output. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["readings"], output_tables=["readings_degrees"], ) def to_degrees(readings: TableFrameSpec) -> TableFrameSpec: expr = col("radians").degrees() # Radians to degrees return readings.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/readings\_degrees is live — radians converted to degrees: ``` ┌─────────┐ │ radians │ ├─────────┤ │ 180 │ ├─────────┤ │ 90.0002 │ └─────────┘ ``` Transformed tables readings | radians | | --- | | 3.14159 | | 1.5708 | 2 rows readings\_degrees | radians | | --- | | 180 | | 90.0002 | 2 rows ## Compare with earlier rows AI agent session ❯make a new table that takes the readings table and takes the change in the value column from the previous row ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `diff()` compares each row with the one before it, so the first row has no predecessor and comes out null. It reads the table in its current order. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["readings"], output_tables=["readings_delta"], ) def row_delta(readings: TableFrameSpec) -> TableFrameSpec: expr = col("value").diff() # Difference from previous row return readings.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/readings\_delta is live — the change from the previous row, null on the first: ``` ┌───────┐ │ value │ ├───────┤ │ null │ ├───────┤ │ 3 │ ├───────┤ │ -4 │ └───────┘ ``` Transformed tables readings | value | | --- | | 10 | | 13 | | 9 | 3 rows readings\_delta | value | | --- | | null | | 3 | | \-4 | 3 rows AI agent session ❯make a new table that takes the readings table and takes the change in the value column from 2 rows back ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `diff(n=2)` looks two rows back, so the first two rows come out null. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["readings"], output_tables=["readings_delta_2"], ) def row_delta_2(readings: TableFrameSpec) -> TableFrameSpec: expr = col("value").diff(n=2) # Difference from 2 rows ago return readings.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/readings\_delta\_2 is live — the change from 2 rows back: ``` ┌───────┐ │ value │ ├───────┤ │ null │ ├───────┤ │ null │ ├───────┤ │ -1 │ └───────┘ ``` Transformed tables readings | value | | --- | | 10 | | 13 | | 9 | 3 rows readings\_delta\_2 | value | | --- | | null | | null | | \-1 | 3 rows --- # Transform Dates and Times > Extracts date and time components, shifts dates, truncates timestamps, converts durations to totals, handles time zones, and formats dates with dt accessors in Tabsdata. Source: https://docs.tabsdata.com/tutorials/transformations/dates-and-times ## Pull a component out AI agent session ❯make a new table that takes the events table and extracts the year from the date column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.year()` returns the year as an integer. Every component accessor lives under `dt`. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_year"], ) def get_year(events: TableFrameSpec) -> TableFrameSpec: expr = col("date").dt.year() return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_year is live — the year as an integer: ``` ┌──────┐ │ date │ ├──────┤ │ 2024 │ ├──────┤ │ 2024 │ └──────┘ ``` Transformed tables events | date | | --- | | 2024-03-15 | | 2024-11-01 | 2 rows events\_year | date | | --- | | 2024 | | 2024 | 2 rows AI agent session ❯make a new table that takes the events table and extracts the month number from the date column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.month()` returns 1 to 12, not a month name. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_month"], ) def get_month(events: TableFrameSpec) -> TableFrameSpec: expr = col("date").dt.month() return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_month is live — the month number: ``` ┌──────┐ │ date │ ├──────┤ │ 3 │ ├──────┤ │ 11 │ └──────┘ ``` Transformed tables events | date | | --- | | 2024-03-15 | | 2024-11-01 | 2 rows events\_month | date | | --- | | 3 | | 11 | 2 rows AI agent session ❯make a new table that takes the events table and extracts the day of the month from the date column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.day()` returns the day within the month, 1 to 31. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_day"], ) def get_day(events: TableFrameSpec) -> TableFrameSpec: expr = col("date").dt.day() return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_day is live — the day of the month: ``` ┌──────┐ │ date │ ├──────┤ │ 15 │ ├──────┤ │ 1 │ └──────┘ ``` Transformed tables events | date | | --- | | 2024-03-15 | | 2024-11-01 | 2 rows events\_day | date | | --- | | 15 | | 1 | 2 rows AI agent session ❯make a new table that takes the events table and extracts the calendar quarter from the date column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.quarter()` returns 1 to 4 on calendar quarters, not on a fiscal year. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_quarter"], ) def get_quarter(events: TableFrameSpec) -> TableFrameSpec: expr = col("date").dt.quarter() return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_quarter is live — the calendar quarter: ``` ┌──────┐ │ date │ ├──────┤ │ 1 │ ├──────┤ │ 4 │ └──────┘ ``` Transformed tables events | date | | --- | | 2024-03-15 | | 2024-11-01 | 2 rows events\_quarter | date | | --- | | 1 | | 4 | 2 rows AI agent session ❯make a new table that takes the events table and extracts the ISO week number from the date column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.week()` is the ISO week number, which is why early January can land in week 52 of the previous year. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_week"], ) def get_week(events: TableFrameSpec) -> TableFrameSpec: expr = col("date").dt.week() return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_week is live — the ISO week number: ``` ┌──────┐ │ date │ ├──────┤ │ 11 │ ├──────┤ │ 44 │ └──────┘ ``` Transformed tables events | date | | --- | | 2024-03-15 | | 2024-11-01 | 2 rows events\_week | date | | --- | | 11 | | 44 | 2 rows AI agent session ❯make a new table that takes the events table and extracts the weekday from the date column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.weekday()` is 0-based from Monday, so Monday is 0 and Sunday is 6. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_weekday"], ) def get_weekday(events: TableFrameSpec) -> TableFrameSpec: expr = col("date").dt.weekday() # 0 = Monday return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_weekday is live — the weekday, Monday being 0: ``` ┌──────┐ │ date │ ├──────┤ │ 4 │ ├──────┤ │ 4 │ └──────┘ ``` Transformed tables events | date | | --- | | 2024-03-15 | | 2024-11-01 | 2 rows events\_weekday | date | | --- | | 4 | | 4 | 2 rows AI agent session ❯make a new table that takes the events table and extracts the day of the year from the date column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.ordinal_day()` counts 1 to 366 from the start of the year. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_day_of_year"], ) def get_ordinal_day(events: TableFrameSpec) -> TableFrameSpec: expr = col("date").dt.ordinal_day() # Day of year (1-366) return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_day\_of\_year is live — the day of the year: ``` ┌──────┐ │ date │ ├──────┤ │ 75 │ ├──────┤ │ 306 │ └──────┘ ``` Transformed tables events | date | | --- | | 2024-03-15 | | 2024-11-01 | 2 rows events\_day\_of\_year | date | | --- | | 75 | | 306 | 2 rows AI agent session ❯make a new table that takes the events table and flags whether the date column falls in a leap year ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.is_leap_year()` returns a Boolean rather than a number. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_leap"], ) def flag_leap_year(events: TableFrameSpec) -> TableFrameSpec: expr = col("date").dt.is_leap_year() return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_leap is live — true where the year is a leap year: ``` ┌──────┐ │ date │ ├──────┤ │ true │ ├──────┤ │ true │ └──────┘ ``` Transformed tables events | date | | --- | | 2024-03-15 | | 2024-11-01 | 2 rows events\_leap | date | | --- | | true | | true | 2 rows AI agent session ❯make a new table that takes the events table and extracts the hour from the timestamp column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.hour()` on a Datetime, returning 0 to 23. A Date column has no hour to give. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_hour"], ) def get_hour(events: TableFrameSpec) -> TableFrameSpec: expr = col("timestamp").dt.hour() return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_hour is live — the hour, 0 to 23: ``` ┌───────────┐ │ timestamp │ ├───────────┤ │ 9 │ ├───────────┤ │ 23 │ └───────────┘ ``` Transformed tables events | timestamp | | --- | | 2024-03-15 09:42:17 | | 2024-11-01 23:05:00 | 2 rows events\_hour | timestamp | | --- | | 9 | | 23 | 2 rows AI agent session ❯make a new table that takes the events table and extracts the minute from the timestamp column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.minute()`, 0 to 59. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_minute"], ) def get_minute(events: TableFrameSpec) -> TableFrameSpec: expr = col("timestamp").dt.minute() return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_minute is live — the minute: ``` ┌───────────┐ │ timestamp │ ├───────────┤ │ 42 │ ├───────────┤ │ 5 │ └───────────┘ ``` Transformed tables events | timestamp | | --- | | 2024-03-15 09:42:17 | | 2024-11-01 23:05:00 | 2 rows events\_minute | timestamp | | --- | | 42 | | 5 | 2 rows AI agent session ❯make a new table that takes the events table and extracts the second from the timestamp column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.second()`, 0 to 59, with no fractional part. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_second"], ) def get_second(events: TableFrameSpec) -> TableFrameSpec: expr = col("timestamp").dt.second() return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_second is live — the whole second: ``` ┌───────────┐ │ timestamp │ ├───────────┤ │ 17 │ ├───────────┤ │ 0 │ └───────────┘ ``` Transformed tables events | timestamp | | --- | | 2024-03-15 09:42:17 | | 2024-11-01 23:05:00 | 2 rows events\_second | timestamp | | --- | | 17 | | 0 | 2 rows AI agent session ❯make a new table that takes the events table and extracts the millisecond from the timestamp column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.millisecond()`, the sub-second part in thousandths. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_millisecond"], ) def get_millisecond(events: TableFrameSpec) -> TableFrameSpec: expr = col("timestamp").dt.millisecond() return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_millisecond is live — the millisecond component: ``` ┌───────────┐ │ timestamp │ ├───────────┤ │ 123 │ ├───────────┤ │ 0 │ └───────────┘ ``` Transformed tables events | timestamp | | --- | | 2024-03-15 09:42:17.123456 | | 2024-11-01 23:05:00.000900 | 2 rows events\_millisecond | timestamp | | --- | | 123 | | 0 | 2 rows AI agent session ❯make a new table that takes the events table and extracts the microsecond from the timestamp column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.microsecond()`, the sub-second part in millionths. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_microsecond"], ) def get_microsecond(events: TableFrameSpec) -> TableFrameSpec: expr = col("timestamp").dt.microsecond() return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_microsecond is live — the microsecond component: ``` ┌───────────┐ │ timestamp │ ├───────────┤ │ 123456 │ ├───────────┤ │ 900 │ └───────────┘ ``` Transformed tables events | timestamp | | --- | | 2024-03-15 09:42:17.123456 | | 2024-11-01 23:05:00.000900 | 2 rows events\_microsecond | timestamp | | --- | | 123456 | | 900 | 2 rows AI agent session ❯make a new table that takes the events table and extracts the nanosecond from the timestamp column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.nanosecond()`, the finest component available. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_nanosecond"], ) def get_nanosecond(events: TableFrameSpec) -> TableFrameSpec: expr = col("timestamp").dt.nanosecond() return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_nanosecond is live — the nanosecond component: ``` ┌───────────┐ │ timestamp │ ├───────────┤ │ 123456000 │ ├───────────┤ │ 900000 │ └───────────┘ ``` Transformed tables events | timestamp | | --- | | 2024-03-15 09:42:17.123456 | | 2024-11-01 23:05:00.000900 | 2 rows events\_nanosecond | timestamp | | --- | | 123456000 | | 900000 | 2 rows AI agent session ❯make a new table that takes the events table and extracts the ISO year from the date column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.iso_year()` pairs with `dt.week()`, and it differs from the calendar year for a few days at each year boundary. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_iso_year"], ) def get_iso_year(events: TableFrameSpec) -> TableFrameSpec: expr = col("date").dt.iso_year() return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_iso\_year is live — the ISO year, which can differ from the calendar year: ``` ┌──────┐ │ date │ ├──────┤ │ 2024 │ ├──────┤ │ 2024 │ └──────┘ ``` Transformed tables events | date | | --- | | 2024-03-15 | | 2024-11-01 | 2 rows events\_iso\_year | date | | --- | | 2024 | | 2024 | 2 rows ## Shift a date AI agent session ❯make a new table that takes the events table and moves the date column a week later ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.offset_by("1w")` takes a duration string rather than a number, so the unit is explicit in the code. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_plus_week"], ) def shift_week(events: TableFrameSpec) -> TableFrameSpec: expr = col("date").dt.offset_by("1w") # Add 1 week return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_plus\_week is live — every date a week later: ``` ┌────────────┐ │ date │ ├────────────┤ │ 2024-03-22 │ ├────────────┤ │ 2024-11-08 │ └────────────┘ ``` Transformed tables events | date | | --- | | 2024-03-15 | | 2024-11-01 | 2 rows events\_plus\_week | date | | --- | | 2024-03-22 | | 2024-11-08 | 2 rows AI agent session ❯make a new table that takes the events table and moves the date column 3 days earlier ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — a negative duration string, `"-3d"`, shifts backwards. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_minus_3d"], ) def shift_back(events: TableFrameSpec) -> TableFrameSpec: expr = col("date").dt.offset_by("-3d") # Subtract 3 days return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_minus\_3d is live — every date 3 days earlier: ``` ┌────────────┐ │ date │ ├────────────┤ │ 2024-03-12 │ ├────────────┤ │ 2024-10-29 │ └────────────┘ ``` Transformed tables events | date | | --- | | 2024-03-15 | | 2024-11-01 | 2 rows events\_minus\_3d | date | | --- | | 2024-03-12 | | 2024-10-29 | 2 rows AI agent session ❯make a new table that takes the events table and moves the date column 2 months later ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `"2mo"` for months, since `"m"` means minutes. Month arithmetic clamps rather than overflowing, so the 31st plus a month lands on the last day of the shorter month. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_plus_2mo"], ) def shift_months(events: TableFrameSpec) -> TableFrameSpec: expr = col("date").dt.offset_by("2mo") # Add 2 months return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_plus\_2mo is live — every date 2 months later: ``` ┌────────────┐ │ date │ ├────────────┤ │ 2024-05-15 │ ├────────────┤ │ 2025-01-01 │ └────────────┘ ``` Transformed tables events | date | | --- | | 2024-03-15 | | 2024-11-01 | 2 rows events\_plus\_2mo | date | | --- | | 2024-05-15 | | 2025-01-01 | 2 rows AI agent session ❯make a new table that takes the events table and moves the date column 5 business days later ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.add_business_days(5)` skips weekends rather than adding 5 calendar days. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_plus_5bd"], ) def shift_business(events: TableFrameSpec) -> TableFrameSpec: expr = col("date").dt.add_business_days(5) return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_plus\_5bd is live — 5 business days later, weekends skipped: ``` ┌────────────┐ │ date │ ├────────────┤ │ 2024-03-22 │ ├────────────┤ │ 2024-11-08 │ └────────────┘ ``` Transformed tables events | date | | --- | | 2024-03-15 | | 2024-11-01 | 2 rows events\_plus\_5bd | date | | --- | | 2024-03-22 | | 2024-11-08 | 2 rows AI agent session ❯make a new table that takes the events table and snaps the date column to the first of its month ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.month_start()`, which is the usual key for a monthly rollup. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_month_start"], ) def snap_month_start(events: TableFrameSpec) -> TableFrameSpec: expr = col("date").dt.month_start() # First day of month return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_month\_start is live — each date snapped to the first of its month: ``` ┌────────────┐ │ date │ ├────────────┤ │ 2024-03-01 │ ├────────────┤ │ 2024-11-01 │ └────────────┘ ``` Transformed tables events | date | | --- | | 2024-03-15 | | 2024-11-01 | 2 rows events\_month\_start | date | | --- | | 2024-03-01 | | 2024-11-01 | 2 rows AI agent session ❯make a new table that takes the events table and snaps the date column to the last day of its month ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.month_end()`, which lands on 28, 29, 30 or 31 depending on the month. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_month_end"], ) def snap_month_end(events: TableFrameSpec) -> TableFrameSpec: expr = col("date").dt.month_end() # Last day of month return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_month\_end is live — each date snapped to its month end: ``` ┌────────────┐ │ date │ ├────────────┤ │ 2024-03-31 │ ├────────────┤ │ 2024-11-30 │ └────────────┘ ``` Transformed tables events | date | | --- | | 2024-03-15 | | 2024-11-01 | 2 rows events\_month\_end | date | | --- | | 2024-03-31 | | 2024-11-30 | 2 rows ## Round a timestamp down AI agent session ❯make a new table that takes the events table and truncates the timestamp column to the hour ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.truncate("1h")` zeroes everything finer than the hour. It rounds down always, unlike a round. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_by_hour"], ) def truncate_hour(events: TableFrameSpec) -> TableFrameSpec: expr = col("timestamp").dt.truncate("1h") # Truncate to hour return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_by\_hour is live — minutes and seconds zeroed: ``` ┌─────────────────────┐ │ timestamp │ ├─────────────────────┤ │ 2024-03-15 09:00:00 │ ├─────────────────────┤ │ 2024-11-01 23:00:00 │ └─────────────────────┘ ``` Transformed tables events | timestamp | | --- | | 2024-03-15 09:42:17 | | 2024-11-01 23:05:00 | 2 rows events\_by\_hour | timestamp | | --- | | 2024-03-15 09:00:00 | | 2024-11-01 23:00:00 | 2 rows AI agent session ❯make a new table that takes the events table and truncates the timestamp column to the day ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.truncate("1d")` keeps the Datetime type with a zeroed time, rather than converting to a Date. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_by_day"], ) def truncate_day(events: TableFrameSpec) -> TableFrameSpec: expr = col("timestamp").dt.truncate("1d") # Truncate to day return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_by\_day is live — the time zeroed, the type unchanged: ``` ┌─────────────────────┐ │ timestamp │ ├─────────────────────┤ │ 2024-03-15 00:00:00 │ ├─────────────────────┤ │ 2024-11-01 00:00:00 │ └─────────────────────┘ ``` Transformed tables events | timestamp | | --- | | 2024-03-15 09:42:17 | | 2024-11-01 23:05:00 | 2 rows events\_by\_day | timestamp | | --- | | 2024-03-15 00:00:00 | | 2024-11-01 00:00:00 | 2 rows AI agent session ❯make a new table that takes the events table and truncates the timestamp column to the month ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.truncate("1mo")` snaps to the first instant of the month. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_by_month"], ) def truncate_month(events: TableFrameSpec) -> TableFrameSpec: expr = col("timestamp").dt.truncate("1mo") # Truncate to month return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_by\_month is live — snapped to the start of the month: ``` ┌─────────────────────┐ │ timestamp │ ├─────────────────────┤ │ 2024-03-01 00:00:00 │ ├─────────────────────┤ │ 2024-11-01 00:00:00 │ └─────────────────────┘ ``` Transformed tables events | timestamp | | --- | | 2024-03-15 09:42:17 | | 2024-11-01 23:05:00 | 2 rows events\_by\_month | timestamp | | --- | | 2024-03-01 00:00:00 | | 2024-11-01 00:00:00 | 2 rows AI agent session ❯make a new table that takes the events table and truncates the timestamp column to the week ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.truncate("1w")` snaps back to the Monday of that week. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_by_week"], ) def truncate_week(events: TableFrameSpec) -> TableFrameSpec: expr = col("timestamp").dt.truncate("1w") # Truncate to week return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_by\_week is live — snapped back to Monday: ``` ┌─────────────────────┐ │ timestamp │ ├─────────────────────┤ │ 2024-03-11 00:00:00 │ ├─────────────────────┤ │ 2024-10-28 00:00:00 │ └─────────────────────┘ ``` Transformed tables events | timestamp | | --- | | 2024-03-15 09:42:17 | | 2024-11-01 23:05:00 | 2 rows events\_by\_week | timestamp | | --- | | 2024-03-11 00:00:00 | | 2024-10-28 00:00:00 | 2 rows ## Express a duration as a number AI agent session ❯make a new table that takes the spans table and expresses the duration column in days ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.total_days()` converts the whole duration to days rather than reading a days field, so 36 hours is 1. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["spans"], output_tables=["spans_days"], ) def duration_days(spans: TableFrameSpec) -> TableFrameSpec: expr = col("duration").dt.total_days() return spans.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/spans\_days is live — each duration in whole days: ``` ┌──────────┐ │ duration │ ├──────────┤ │ 2 │ ├──────────┤ │ 0 │ └──────────┘ ``` Transformed tables spans | duration | | --- | | 2d 03:00:00 | | 0d 05:30:00 | 2 rows spans\_days | duration | | --- | | 2 | | 0 | 2 rows AI agent session ❯make a new table that takes the spans table and expresses the duration column in hours ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.total_hours()`, the whole duration in hours. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["spans"], output_tables=["spans_hours"], ) def duration_hours(spans: TableFrameSpec) -> TableFrameSpec: expr = col("duration").dt.total_hours() return spans.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/spans\_hours is live — each duration in whole hours: ``` ┌──────────┐ │ duration │ ├──────────┤ │ 51 │ ├──────────┤ │ 5 │ └──────────┘ ``` Transformed tables spans | duration | | --- | | 2d 03:00:00 | | 0d 05:30:00 | 2 rows spans\_hours | duration | | --- | | 51 | | 5 | 2 rows AI agent session ❯make a new table that takes the spans table and expresses the duration column in minutes ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.total_minutes()`. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["spans"], output_tables=["spans_minutes"], ) def duration_minutes(spans: TableFrameSpec) -> TableFrameSpec: expr = col("duration").dt.total_minutes() return spans.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/spans\_minutes is live — each duration in whole minutes: ``` ┌──────────┐ │ duration │ ├──────────┤ │ 3060 │ ├──────────┤ │ 330 │ └──────────┘ ``` Transformed tables spans | duration | | --- | | 2d 03:00:00 | | 0d 05:30:00 | 2 rows spans\_minutes | duration | | --- | | 3060 | | 330 | 2 rows AI agent session ❯make a new table that takes the spans table and expresses the duration column in seconds ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.total_seconds()`, the most common one for a rate calculation. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["spans"], output_tables=["spans_seconds"], ) def duration_seconds(spans: TableFrameSpec) -> TableFrameSpec: expr = col("duration").dt.total_seconds() return spans.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/spans\_seconds is live — each duration in whole seconds: ``` ┌──────────┐ │ duration │ ├──────────┤ │ 183600 │ ├──────────┤ │ 19800 │ └──────────┘ ``` Transformed tables spans | duration | | --- | | 2d 03:00:00 | | 0d 05:30:00 | 2 rows spans\_seconds | duration | | --- | | 183600 | | 19800 | 2 rows AI agent session ❯make a new table that takes the spans table and expresses the duration column in milliseconds ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.total_milliseconds()`. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["spans"], output_tables=["spans_milliseconds"], ) def duration_ms(spans: TableFrameSpec) -> TableFrameSpec: expr = col("duration").dt.total_milliseconds() return spans.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/spans\_milliseconds is live — each duration in milliseconds: ``` ┌───────────┐ │ duration │ ├───────────┤ │ 183600000 │ ├───────────┤ │ 19800000 │ └───────────┘ ``` Transformed tables spans | duration | | --- | | 2d 03:00:00 | | 0d 05:30:00 | 2 rows spans\_milliseconds | duration | | --- | | 183600000 | | 19800000 | 2 rows AI agent session ❯make a new table that takes the spans table and expresses the duration column in microseconds ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.total_microseconds()`, the finest of the total accessors. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["spans"], output_tables=["spans_microseconds"], ) def duration_us(spans: TableFrameSpec) -> TableFrameSpec: expr = col("duration").dt.total_microseconds() return spans.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/spans\_microseconds is live — each duration in microseconds: ``` ┌──────────────┐ │ duration │ ├──────────────┤ │ 183600000000 │ ├──────────────┤ │ 19800000000 │ └──────────────┘ ``` Transformed tables spans | duration | | --- | | 2d 03:00:00 | | 0d 05:30:00 | 2 rows spans\_microseconds | duration | | --- | | 183600000000 | | 19800000000 | 2 rows ## Work with time zones AI agent session ❯make a new table that takes the events table and shows the timestamp column in America/New\_York ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.convert_time_zone()` keeps the same instant and changes how it reads, so the wall-clock digits move. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_ny"], ) def to_new_york(events: TableFrameSpec) -> TableFrameSpec: expr = col("timestamp").dt.convert_time_zone("America/New_York") return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_ny is live — the same instants read in New York time: ``` ┌─────────────────────────┐ │ timestamp │ ├─────────────────────────┤ │ 2024-03-15 10:00:00 EDT │ ├─────────────────────────┤ │ 2024-11-01 10:00:00 EDT │ └─────────────────────────┘ ``` Transformed tables events | timestamp | | --- | | 2024-03-15 14:00:00 UTC | | 2024-11-01 14:00:00 UTC | 2 rows events\_ny | timestamp | | --- | | 2024-03-15 10:00:00 EDT | | 2024-11-01 10:00:00 EDT | 2 rows AI agent session ❯make a new table that takes the events table and stamps the timestamp column as UTC without moving the instant ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.replace_time_zone()` is the opposite of convert: the digits stay and the instant moves. Reach for it when a column was stored without a zone and you know what it was. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_utc"], ) def stamp_utc(events: TableFrameSpec) -> TableFrameSpec: expr = col("timestamp").dt.replace_time_zone("UTC") return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_utc is live — the digits unchanged, the zone now UTC: ``` ┌─────────────────────────┐ │ timestamp │ ├─────────────────────────┤ │ 2024-03-15 14:00:00 UTC │ ├─────────────────────────┤ │ 2024-11-01 14:00:00 UTC │ └─────────────────────────┘ ``` Transformed tables events | timestamp | | --- | | 2024-03-15 14:00:00 | | 2024-11-01 14:00:00 | 2 rows events\_utc | timestamp | | --- | | 2024-03-15 14:00:00 UTC | | 2024-11-01 14:00:00 UTC | 2 rows AI agent session ❯make a new table that takes the events table and reads the base UTC offset of the timestamp column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.base_utc_offset()` gives the zone offset before any daylight saving is added. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_base_offset"], ) def base_offset(events: TableFrameSpec) -> TableFrameSpec: expr = col("timestamp").dt.base_utc_offset() return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_base\_offset is live — the standard offset from UTC: ``` ┌───────────┐ │ timestamp │ ├───────────┤ │ -5h │ ├───────────┤ │ -5h │ └───────────┘ ``` Transformed tables events | timestamp | | --- | | 2024-03-15 10:00:00 EDT | | 2024-01-15 10:00:00 EST | 2 rows events\_base\_offset | timestamp | | --- | | \-5h | | \-5h | 2 rows AI agent session ❯make a new table that takes the events table and reads the daylight saving offset of the timestamp column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.dst_offset()` gives only the daylight saving part, so it is zero outside summer time. Add it to the base offset for the total. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_dst_offset"], ) def dst_offset(events: TableFrameSpec) -> TableFrameSpec: expr = col("timestamp").dt.dst_offset() return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_dst\_offset is live — the daylight saving part alone: ``` ┌───────────┐ │ timestamp │ ├───────────┤ │ 1h │ ├───────────┤ │ 0h │ └───────────┘ ``` Transformed tables events | timestamp | | --- | | 2024-03-15 10:00:00 EDT | | 2024-01-15 10:00:00 EST | 2 rows events\_dst\_offset | timestamp | | --- | | 1h | | 0h | 2 rows ## Format and convert AI agent session ❯make a new table that takes the events table and formats the date column as text in YYYY-MM-DD ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.to_string()` with a format string. This turns the column into text, so no further date operations will work on it. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_formatted"], ) def format_date(events: TableFrameSpec) -> TableFrameSpec: expr = col("date").dt.to_string("%Y-%m-%d") return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_formatted is live — the date rendered as text: ``` ┌────────────┐ │ date │ ├────────────┤ │ 2024-03-15 │ ├────────────┤ │ 2024-11-01 │ └────────────┘ ``` Transformed tables events | dateDate | | --- | | 2024-03-15 | | 2024-11-01 | 2 rows events\_formatted | dateString | | --- | | 2024-03-15 | | 2024-11-01 | 2 rows AI agent session ❯make a new table that takes the events table and formats the timestamp column as text with the time included ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.strftime()` takes the same format codes and is the name to reach for out of habit; both produce text. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_formatted"], ) def format_timestamp(events: TableFrameSpec) -> TableFrameSpec: expr = col("timestamp").dt.strftime("%Y-%m-%d %H:%M:%S") return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_formatted is live — the timestamp rendered as text: ``` ┌─────────────────────┐ │ timestamp │ ├─────────────────────┤ │ 2024-03-15 09:42:17 │ ├─────────────────────┤ │ 2024-11-01 23:05:00 │ └─────────────────────┘ ``` Transformed tables events | timestamp | | --- | | 2024-03-15 09:42:17 | | 2024-11-01 23:05:00 | 2 rows events\_formatted | timestamp | | --- | | 2024-03-15 09:42:17 | | 2024-11-01 23:05:00 | 2 rows AI agent session ❯make a new table that takes the events table and converts the timestamp column to seconds since the epoch ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.epoch(time_unit="s")` counts from 1970-01-01, and the unit is explicit so you are not guessing at the scale. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_epoch"], ) def to_epoch(events: TableFrameSpec) -> TableFrameSpec: expr = col("timestamp").dt.epoch(time_unit="s") # Seconds since epoch return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_epoch is live — seconds since 1970-01-01: ``` ┌────────────┐ │ timestamp │ ├────────────┤ │ 1710495737 │ ├────────────┤ │ 1730502300 │ └────────────┘ ``` Transformed tables events | timestamp | | --- | | 2024-03-15 09:42:17 | | 2024-11-01 23:05:00 | 2 rows events\_epoch | timestamp | | --- | | 1710495737 | | 1730502300 | 2 rows AI agent session ❯make a new table that takes the events table and converts the timestamp column to microseconds since the epoch ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `dt.timestamp()` defaults to microseconds, which is the difference from `epoch()` where you name the unit. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_micros"], ) def to_micros(events: TableFrameSpec) -> TableFrameSpec: expr = col("timestamp").dt.timestamp() # Microseconds since epoch return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_micros is live — microseconds since 1970-01-01: ``` ┌──────────────────┐ │ timestamp │ ├──────────────────┤ │ 1710495737000000 │ ├──────────────────┤ │ 1730502300000000 │ └──────────────────┘ ``` Transformed tables events | timestamp | | --- | | 2024-03-15 09:42:17 | | 2024-11-01 23:05:00 | 2 rows events\_micros | timestamp | | --- | | 1710495737000000 | | 1730502300000000 | 2 rows --- # Aggregate and Group > Aggregation expressions (count, len, sum, mean, median, min, max, first, last, n_unique), group_by with agg, whole-group aggregates, and rank methods in Tabsdata transformers. Source: https://docs.tabsdata.com/tutorials/transformations/aggregate-and-group ## Aggregate a whole column AI agent session ❯make a new table that takes the events table and counts the non-null values in the id column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `count()` on the column, which counts the non-null values and ignores the nulls. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_id_count"], ) def count_ids(events: TableFrameSpec) -> TableFrameSpec: return events.select(col("id").count()) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_id\_count is live — one row, one column: the non-null ids: ``` ┌────┐ │ id │ ├────┤ │ 2 │ └────┘ ``` Transformed tables events | id | | --- | | 1 | | null | | 3 | 3 rows events\_id\_count | id | | --- | | 2 | 1 row AI agent session ❯make a new table that takes the events table and counts every row, including the ones with a null id ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `len()` rather than `count()`. len counts rows, so nulls are included; count skips them. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_row_count"], ) def count_rows(events: TableFrameSpec) -> TableFrameSpec: return events.select(col("id").len()) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_row\_count is live — one row holding the row count, nulls included: ``` ┌────┐ │ id │ ├────┤ │ 3 │ └────┘ ``` Transformed tables events | id | | --- | | 1 | | null | | 3 | 3 rows events\_row\_count | id | | --- | | 3 | 1 row AI agent session ❯make a new table that takes the orders table and totals the amount column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `sum()` on the column collapses every row into one. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["orders"], output_tables=["orders_total"], ) def total_amount(orders: TableFrameSpec) -> TableFrameSpec: return orders.select(col("amount").sum()) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/orders\_total is live — the whole column added up: ``` ┌────────┐ │ amount │ ├────────┤ │ 60 │ └────────┘ ``` Transformed tables orders | amount | | --- | | 10 | | 20 | | 30 | 3 rows orders\_total | amount | | --- | | 60 | 1 row AI agent session ❯make a new table that takes the orders table and averages the amount column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `mean()`, which divides by the count of non-null values rather than by the row count. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["orders"], output_tables=["orders_average"], ) def average_amount(orders: TableFrameSpec) -> TableFrameSpec: return orders.select(col("amount").mean()) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/orders\_average is live — the arithmetic mean of the column: ``` ┌────────┐ │ amount │ ├────────┤ │ 20 │ └────────┘ ``` Transformed tables orders | amount | | --- | | 10 | | 20 | | 30 | 3 rows orders\_average | amount | | --- | | 20 | 1 row AI agent session ❯make a new table that takes the orders table and takes the median of the amount column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `median()`, which is the middle value and so is not dragged by an outlier the way mean is. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["orders"], output_tables=["orders_median"], ) def median_amount(orders: TableFrameSpec) -> TableFrameSpec: return orders.select(col("amount").median()) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/orders\_median is live — the middle value of the column: ``` ┌────────┐ │ amount │ ├────────┤ │ 20 │ └────────┘ ``` Transformed tables orders | amount | | --- | | 10 | | 20 | | 30 | 3 rows orders\_median | amount | | --- | | 20 | 1 row AI agent session ❯make a new table that takes the events table and finds the earliest value in the date column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `min()` works on dates as well as numbers, since both order. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_earliest"], ) def earliest_date(events: TableFrameSpec) -> TableFrameSpec: return events.select(col("date").min()) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_earliest is live — the smallest value, which for a Date is the earliest: ``` ┌────────────┐ │ date │ ├────────────┤ │ 2024-01-02 │ └────────────┘ ``` Transformed tables events | date | | --- | | 2024-03-15 | | 2024-01-02 | 2 rows events\_earliest | date | | --- | | 2024-01-02 | 1 row AI agent session ❯make a new table that takes the readings table and finds the largest value in the value column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `max()` on the column. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["readings"], output_tables=["readings_highest"], ) def highest_value(readings: TableFrameSpec) -> TableFrameSpec: return readings.select(col("value").max()) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/readings\_highest is live — the largest value in the column: ``` ┌───────┐ │ value │ ├───────┤ │ 7 │ └───────┘ ``` Transformed tables readings | value | | --- | | 7 | | 3 | 2 rows readings\_highest | value | | --- | | 7 | 1 row AI agent session ❯make a new table that takes the events table and takes the value of the timestamp column from the first row ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `first()` takes the value in the first row as the data arrives, which is not the same as the earliest unless the table is sorted. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_first"], ) def first_timestamp(events: TableFrameSpec) -> TableFrameSpec: return events.select(col("timestamp").first()) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_first is live — the value from the first row, in arrival order: ``` ┌───────────┐ │ timestamp │ ├───────────┤ │ 09:00 │ └───────────┘ ``` Transformed tables events | timestamp | | --- | | 09:00 | | 17:30 | 2 rows events\_first | timestamp | | --- | | 09:00 | 1 row AI agent session ❯make a new table that takes the events table and takes the value of the timestamp column from the last row ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `last()`, the same positional idea from the other end. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_last"], ) def last_timestamp(events: TableFrameSpec) -> TableFrameSpec: return events.select(col("timestamp").last()) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_last is live — the value from the final row: ``` ┌───────────┐ │ timestamp │ ├───────────┤ │ 17:30 │ └───────────┘ ``` Transformed tables events | timestamp | | --- | | 09:00 | | 17:30 | 2 rows events\_last | timestamp | | --- | | 17:30 | 1 row AI agent session ❯make a new table that takes the items table and counts the distinct values in the category column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `n_unique()` counts distinct values rather than rows, so a category appearing twice counts once. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["items"], output_tables=["items_category_count"], ) def count_categories(items: TableFrameSpec) -> TableFrameSpec: return items.select(col("category").n_unique()) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/items\_category\_count is live — the number of distinct values: ``` ┌──────────┐ │ category │ ├──────────┤ │ 2 │ └──────────┘ ``` Transformed tables items | category | | --- | | tools | | toys | | tools | 3 rows items\_category\_count | category | | --- | | 2 | 1 row ## Group rows and aggregate AI agent session ❯make a new table that takes the sales table and totals the sales column and averages the quantity column per category ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `group_by("category").agg(...)`. Each expression in agg becomes one output column, and alias names it. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["sales"], output_tables=["sales_by_category"], ) def sales_by_category(sales: TableFrameSpec) -> TableFrameSpec: return sales.group_by("category").agg( col("sales").sum().alias("total_sales"), col("quantity").mean().alias("avg_quantity") ) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/sales\_by\_category is live — one row per category, with the two aggregates you asked for: ``` ┌──────────┬─────────────┬──────────────┐ │ category │ total_sales │ avg_quantity │ ├──────────┼─────────────┼──────────────┤ │ tools │ 160 │ 5 │ ├──────────┼─────────────┼──────────────┤ │ toys │ 40 │ 2 │ └──────────┴─────────────┴──────────────┘ ``` Transformed tables sales | category | sales | quantity | | --- | --- | --- | | tools | 100 | 4 | | toys | 40 | 2 | | tools | 60 | 6 | 3 rows sales\_by\_category | category | total\_sales | avg\_quantity | | --- | --- | --- | | tools | 160 | 5 | | toys | 40 | 2 | 2 rows AI agent session ❯make a new table that takes the orders table and totals the revenue column and counts the order\_id column per region and product type ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `group_by()` takes several columns, and the group is the combination of their values. Only combinations that occur in the data get a row. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["orders"], output_tables=["orders_by_region_type"], ) def revenue_by_region(orders: TableFrameSpec) -> TableFrameSpec: return orders.group_by("region", "product_type").agg( col("revenue").sum().alias("total_revenue"), col("order_id").count().alias("order_count") ) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/orders\_by\_region\_type is live — one row per region and product type pair that exists: ``` ┌────────┬──────────────┬───────────────┬─────────────┐ │ region │ product_type │ total_revenue │ order_count │ ├────────┼──────────────┼───────────────┼─────────────┤ │ east │ tool │ 150 │ 2 │ ├────────┼──────────────┼───────────────┼─────────────┤ │ west │ toy │ 70 │ 1 │ └────────┴──────────────┴───────────────┴─────────────┘ ``` Transformed tables orders | region | product\_type | revenue | order\_id | | --- | --- | --- | --- | | east | tool | 100 | o1 | | east | tool | 50 | o2 | | west | toy | 70 | o3 | 3 rows orders\_by\_region\_type | region | product\_type | total\_revenue | order\_count | | --- | --- | --- | --- | | east | tool | 150 | 2 | | west | toy | 70 | 1 | 2 rows AI agent session ❯make a new table that takes the payments table and totals the amount column per year and month of the timestamp column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — there is no month column to group on, so `group_by()` takes expressions that derive year and month from the timestamp and groups on the results. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["payments"], output_tables=["payments_by_month"], ) def monthly_totals(payments: TableFrameSpec) -> TableFrameSpec: return payments.group_by( col("timestamp").dt.year().alias("year"), col("timestamp").dt.month().alias("month") ).agg( col("amount").sum().alias("monthly_total") ) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/payments\_by\_month is live — one row per year and month, from a table that had neither column: ``` ┌──────┬───────┬───────────────┐ │ year │ month │ monthly_total │ ├──────┼───────┼───────────────┤ │ 2024 │ 1 │ 50 │ ├──────┼───────┼───────────────┤ │ 2024 │ 2 │ 50 │ └──────┴───────┴───────────────┘ ``` Transformed tables payments | timestamp | amount | | --- | --- | | 2024-01-15 10:00:00 | 30 | | 2024-01-28 12:00:00 | 20 | | 2024-02-02 09:00:00 | 50 | 3 rows payments\_by\_month | year | month | monthly\_total | | --- | --- | --- | | 2024 | 1 | 50 | | 2024 | 2 | 50 | 2 rows ## Aggregate every column in a group AI agent session ❯make a new table that takes the sales table and counts the rows in each category ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — the shorthand form. `group_by("category").count()` applies to the frame rather than to a named expression, so you get a count without writing an agg. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["sales"], output_tables=["sales_counts"], ) def count_by_category(sales: TableFrameSpec) -> TableFrameSpec: return sales.group_by("category").count() ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/sales\_counts is live — one row per category with its row count: ``` ┌──────────┬───────┐ │ category │ sales │ ├──────────┼───────┤ │ tools │ 2 │ ├──────────┼───────┤ │ toys │ 1 │ └──────────┴───────┘ ``` Transformed tables sales | category | sales | | --- | --- | | tools | 100 | | toys | 40 | | tools | 60 | 3 rows sales\_counts | category | sales | | --- | --- | | tools | 2 | | toys | 1 | 2 rows AI agent session ❯make a new table that takes the sales table and sums every column other than category, per category ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `group_by("category").sum()` sums every non-key column at once, which is the difference from agg: you name no columns and get all of them. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["sales"], output_tables=["sales_sums"], ) def sum_by_category(sales: TableFrameSpec) -> TableFrameSpec: return sales.group_by("category").sum() ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/sales\_sums is live — every non-key column summed inside each category: ``` ┌──────────┬───────┐ │ category │ sales │ ├──────────┼───────┤ │ tools │ 160 │ ├──────────┼───────┤ │ toys │ 40 │ └──────────┴───────┘ ``` Transformed tables sales | category | sales | | --- | --- | | tools | 100 | | toys | 40 | | tools | 60 | 3 rows sales\_sums | category | sales | | --- | --- | | tools | 160 | | toys | 40 | 2 rows AI agent session ❯make a new table that takes the sales table and averages every column other than category, per category ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `mean()` on the group, applied to every non-key column. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["sales"], output_tables=["sales_means"], ) def mean_by_category(sales: TableFrameSpec) -> TableFrameSpec: return sales.group_by("category").mean() ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/sales\_means is live — every non-key column averaged inside each category: ``` ┌──────────┬───────┐ │ category │ sales │ ├──────────┼───────┤ │ tools │ 80 │ ├──────────┼───────┤ │ toys │ 40 │ └──────────┴───────┘ ``` Transformed tables sales | category | sales | | --- | --- | | tools | 100 | | toys | 40 | | tools | 60 | 3 rows sales\_means | category | sales | | --- | --- | | tools | 80 | | toys | 40 | 2 rows AI agent session ❯make a new table that takes the sales table and takes the smallest value of every column other than category, per category ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `min()` on the group. Each column is reduced on its own, so a row of the output need not match any row of the input. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["sales"], output_tables=["sales_mins"], ) def min_by_category(sales: TableFrameSpec) -> TableFrameSpec: return sales.group_by("category").min() ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/sales\_mins is live — the smallest value of each non-key column per category: ``` ┌──────────┬───────┐ │ category │ sales │ ├──────────┼───────┤ │ tools │ 60 │ ├──────────┼───────┤ │ toys │ 40 │ └──────────┴───────┘ ``` Transformed tables sales | category | sales | | --- | --- | | tools | 100 | | toys | 40 | | tools | 60 | 3 rows sales\_mins | category | sales | | --- | --- | | tools | 60 | | toys | 40 | 2 rows AI agent session ❯make a new table that takes the sales table and takes the largest value of every column other than category, per category ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `max()` on the group, column by column. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["sales"], output_tables=["sales_maxes"], ) def max_by_category(sales: TableFrameSpec) -> TableFrameSpec: return sales.group_by("category").max() ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/sales\_maxes is live — the largest value of each non-key column per category: ``` ┌──────────┬───────┐ │ category │ sales │ ├──────────┼───────┤ │ tools │ 100 │ ├──────────┼───────┤ │ toys │ 40 │ └──────────┴───────┘ ``` Transformed tables sales | category | sales | | --- | --- | | tools | 100 | | toys | 40 | | tools | 60 | 3 rows sales\_maxes | category | sales | | --- | --- | | tools | 100 | | toys | 40 | 2 rows ## Rank values AI agent session ❯make a new table that takes the results table and ranks the score column, with tied scores sharing the average rank ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `rank()` returns a value per row rather than collapsing the table, so it goes through `with_columns`. method="average" is the default and gives tied rows the mean of the ranks they span. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["results"], output_tables=["results_ranked"], ) def rank_average(results: TableFrameSpec) -> TableFrameSpec: return results.with_columns(col("score").rank(method="average").alias("avg_rank")) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/results\_ranked is live — a rank column beside the scores. Ada and Cy tie on 7 and share 2.5: ``` ┌────────┬───────┬──────────┐ │ player │ score │ avg_rank │ ├────────┼───────┼──────────┤ │ Ada │ 7 │ 2.5 │ ├────────┼───────┼──────────┤ │ Bo │ 5 │ 1 │ ├────────┼───────┼──────────┤ │ Cy │ 7 │ 2.5 │ └────────┴───────┴──────────┘ ``` Transformed tables results | player | score | | --- | --- | | Ada | 7 | | Bo | 5 | | Cy | 7 | 3 rows results\_ranked | player | score | avg\_rank | | --- | --- | --- | | Ada | 7 | 2.5 | | Bo | 5 | 1 | | Cy | 7 | 2.5 | 3 rows AI agent session ❯make a new table that takes the results table and ranks the score column so tied values take the lower rank ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — method="min" gives every tied row the lowest rank in the tie, which is what competition ranking means. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["results"], output_tables=["results_ranked"], ) def rank_min(results: TableFrameSpec) -> TableFrameSpec: return results.with_columns(col("score").rank(method="min").alias("min_rank")) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/results\_ranked is live — the tie takes 2, and 3 is skipped: ``` ┌────────┬───────┬──────────┐ │ player │ score │ min_rank │ ├────────┼───────┼──────────┤ │ Ada │ 7 │ 2 │ ├────────┼───────┼──────────┤ │ Bo │ 5 │ 1 │ ├────────┼───────┼──────────┤ │ Cy │ 7 │ 2 │ └────────┴───────┴──────────┘ ``` Transformed tables results | player | score | | --- | --- | | Ada | 7 | | Bo | 5 | | Cy | 7 | 3 rows results\_ranked | player | score | min\_rank | | --- | --- | --- | | Ada | 7 | 2 | | Bo | 5 | 1 | | Cy | 7 | 2 | 3 rows AI agent session ❯make a new table that takes the results table and ranks the score column so tied values take the higher rank ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — method="max", the same tie handled from the other end. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["results"], output_tables=["results_ranked"], ) def rank_max(results: TableFrameSpec) -> TableFrameSpec: return results.with_columns(col("score").rank(method="max").alias("max_rank")) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/results\_ranked is live — the tie takes 3, and 2 is skipped: ``` ┌────────┬───────┬──────────┐ │ player │ score │ max_rank │ ├────────┼───────┼──────────┤ │ Ada │ 7 │ 3 │ ├────────┼───────┼──────────┤ │ Bo │ 5 │ 1 │ ├────────┼───────┼──────────┤ │ Cy │ 7 │ 3 │ └────────┴───────┴──────────┘ ``` Transformed tables results | player | score | | --- | --- | | Ada | 7 | | Bo | 5 | | Cy | 7 | 3 rows results\_ranked | player | score | max\_rank | | --- | --- | --- | | Ada | 7 | 3 | | Bo | 5 | 1 | | Cy | 7 | 3 | 3 rows AI agent session ❯make a new table that takes the results table and ranks the score column with no gaps in the numbers ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — method="dense" is like min except the next distinct value takes the following integer rather than leaving a gap. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["results"], output_tables=["results_ranked"], ) def rank_dense(results: TableFrameSpec) -> TableFrameSpec: return results.with_columns(col("score").rank(method="dense").alias("dense_rank")) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/results\_ranked is live — the tie takes 2 and nothing is skipped: ``` ┌────────┬───────┬────────────┐ │ player │ score │ dense_rank │ ├────────┼───────┼────────────┤ │ Ada │ 7 │ 2 │ ├────────┼───────┼────────────┤ │ Bo │ 5 │ 1 │ ├────────┼───────┼────────────┤ │ Cy │ 7 │ 2 │ └────────┴───────┴────────────┘ ``` Transformed tables results | player | score | | --- | --- | | Ada | 7 | | Bo | 5 | | Cy | 7 | 3 rows results\_ranked | player | score | dense\_rank | | --- | --- | --- | | Ada | 7 | 2 | | Bo | 5 | 1 | | Cy | 7 | 2 | 3 rows AI agent session ❯make a new table that takes the results table and ranks the score column so every row gets a distinct number ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — method="ordinal" gives every row its own rank, breaking ties by the order the rows arrive in. method="random" breaks them without regard to order. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["results"], output_tables=["results_ranked"], ) def rank_ordinal(results: TableFrameSpec) -> TableFrameSpec: return results.with_columns(col("score").rank(method="ordinal").alias("rank")) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/results\_ranked is live — every row has a distinct rank, the tie split by arrival order: ``` ┌────────┬───────┬──────┐ │ player │ score │ rank │ ├────────┼───────┼──────┤ │ Ada │ 7 │ 2 │ ├────────┼───────┼──────┤ │ Bo │ 5 │ 1 │ ├────────┼───────┼──────┤ │ Cy │ 7 │ 3 │ └────────┴───────┴──────┘ ``` Transformed tables results | player | score | | --- | --- | | Ada | 7 | | Bo | 5 | | Cy | 7 | 3 rows results\_ranked | player | score | rank | | --- | --- | --- | | Ada | 7 | 2 | | Bo | 5 | 1 | | Cy | 7 | 3 | 3 rows --- # Join Tables > Joins two Tabsdata tables with `join()` on one or more key columns, covering the inner, left, right, full, semi and anti join types, `left_on`/`right_on`, and `suffix` for duplicate names. Source: https://docs.tabsdata.com/tutorials/transformations/join-tables Every session on this page joins the same two tables. `orders` has three rows and `customers` has three, and they overlap on two `customer_id` values: `o3` points at `c9`, which no customer has, and `c3` has no order. What each join does with those two odd rows is the whole difference between them. ```text orders customers order_id customer_id amount customer_id name o1 c1 10 c1 Ada o2 c2 20 c2 Grace o3 c9 5 c3 Cy ``` ## Join two tables AI agent session ❯make a new table that takes the orders and customers tables and joins them on the customer\_id column, dropping the orders with no customer ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — two input tables, so the transformer takes a slot each, and `join()` on customer\_id. Dropping the unmatched orders is the default, how="inner", so it can be left off. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["orders", "customers"], output_tables=["order_names"], ) def join_orders(orders: TableFrameSpec, customers: TableFrameSpec) -> TableFrameSpec: return orders.join(customers, on="customer_id") ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/order\_names is live — 2 of 3 orders. o3 points at c9 and dropped out, and c3 never appears because it has no order: ``` ┌──────────┬─────────────┬────────┬───────┐ │ order_id │ customer_id │ amount │ name │ ├──────────┼─────────────┼────────┼───────┤ │ o1 │ c1 │ 10 │ Ada │ ├──────────┼─────────────┼────────┼───────┤ │ o2 │ c2 │ 20 │ Grace │ └──────────┴─────────────┴────────┴───────┘ ``` Transformed tables orders | order\_id | customer\_id | amount | | --- | --- | --- | | o1 | c1 | 10 | | o2 | c2 | 20 | | o3 | c9 | 5 | 3 rows customers | customer\_id | name | | --- | --- | | c1 | Ada | | c2 | Grace | | c3 | Cy | 3 rows order\_names | order\_id | customer\_id | amount | name | | --- | --- | --- | --- | | o1 | c1 | 10 | Ada | | o2 | c2 | 20 | Grace | 2 rows AI agent session ❯make a new table that takes the orders and customers tables and joins them on the customer\_id column, keeping every order ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — "every order" makes this how="left". Orders with no matching customer stay, with null in the columns that came from customers. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["orders", "customers"], output_tables=["order_names"], ) def join_orders(orders: TableFrameSpec, customers: TableFrameSpec) -> TableFrameSpec: return orders.join(customers, on="customer_id", how="left") ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/order\_names is live — all 3 orders, o3 with a null name: ``` ┌──────────┬─────────────┬────────┬───────┐ │ order_id │ customer_id │ amount │ name │ ├──────────┼─────────────┼────────┼───────┤ │ o1 │ c1 │ 10 │ Ada │ ├──────────┼─────────────┼────────┼───────┤ │ o2 │ c2 │ 20 │ Grace │ ├──────────┼─────────────┼────────┼───────┤ │ o3 │ c9 │ 5 │ null │ └──────────┴─────────────┴────────┴───────┘ ``` Transformed tables orders | order\_id | customer\_id | amount | | --- | --- | --- | | o1 | c1 | 10 | | o2 | c2 | 20 | | o3 | c9 | 5 | 3 rows customers | customer\_id | name | | --- | --- | | c1 | Ada | | c2 | Grace | | c3 | Cy | 3 rows order\_names | order\_id | customer\_id | amount | name | | --- | --- | --- | --- | | o1 | c1 | 10 | Ada | | o2 | c2 | 20 | Grace | | o3 | c9 | 5 | null | 3 rows AI agent session ❯make a new table that takes the orders and customers tables and joins them on the customer\_id column, keeping every customer ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — how="right" keeps every row of the table passed to join() rather than the one it is called on, so the customer with no order survives instead of the order with no customer. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["orders", "customers"], output_tables=["customer_orders"], ) def join_customers(orders: TableFrameSpec, customers: TableFrameSpec) -> TableFrameSpec: return orders.join(customers, on="customer_id", how="right") ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/customer\_orders is live — all 3 customers. c3 has no order, so the order columns are null, and o3 is gone: ``` ┌──────────┬─────────────┬────────┬───────┐ │ order_id │ customer_id │ amount │ name │ ├──────────┼─────────────┼────────┼───────┤ │ o1 │ c1 │ 10 │ Ada │ ├──────────┼─────────────┼────────┼───────┤ │ o2 │ c2 │ 20 │ Grace │ ├──────────┼─────────────┼────────┼───────┤ │ null │ c3 │ null │ Cy │ └──────────┴─────────────┴────────┴───────┘ ``` Transformed tables orders | order\_id | customer\_id | amount | | --- | --- | --- | | o1 | c1 | 10 | | o2 | c2 | 20 | | o3 | c9 | 5 | 3 rows customers | customer\_id | name | | --- | --- | | c1 | Ada | | c2 | Grace | | c3 | Cy | 3 rows customer\_orders | order\_id | customer\_id | amount | name | | --- | --- | --- | --- | | o1 | c1 | 10 | Ada | | o2 | c2 | 20 | Grace | | null | c3 | null | Cy | 3 rows AI agent session ❯make a new table that takes the orders and customers tables and joins them on the customer\_id column, keeping every row from both ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — how="full" keeps unmatched rows from both sides, so this is the only join here that loses nothing. It is spelled "full" rather than "outer". Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["orders", "customers"], output_tables=["orders_customers"], ) def join_all(orders: TableFrameSpec, customers: TableFrameSpec) -> TableFrameSpec: return orders.join(customers, on="customer_id", how="full") ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/orders\_customers is live — 4 rows, one per matched pair plus o3 and c3 each carrying nulls on the side they have no row for: ``` ┌──────────┬─────────────┬────────┬───────┐ │ order_id │ customer_id │ amount │ name │ ├──────────┼─────────────┼────────┼───────┤ │ o1 │ c1 │ 10 │ Ada │ ├──────────┼─────────────┼────────┼───────┤ │ o2 │ c2 │ 20 │ Grace │ ├──────────┼─────────────┼────────┼───────┤ │ o3 │ c9 │ 5 │ null │ ├──────────┼─────────────┼────────┼───────┤ │ null │ c3 │ null │ Cy │ └──────────┴─────────────┴────────┴───────┘ ``` Transformed tables orders | order\_id | customer\_id | amount | | --- | --- | --- | | o1 | c1 | 10 | | o2 | c2 | 20 | | o3 | c9 | 5 | 3 rows customers | customer\_id | name | | --- | --- | | c1 | Ada | | c2 | Grace | | c3 | Cy | 3 rows orders\_customers | order\_id | customer\_id | amount | name | | --- | --- | --- | --- | | o1 | c1 | 10 | Ada | | o2 | c2 | 20 | Grace | | o3 | c9 | 5 | null | | null | c3 | null | Cy | 4 rows AI agent session ❯make a new table that takes the orders and customers tables and keeps the orders whose customer\_id column is in the customers table, adding no customer columns ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — how="semi" filters rather than widens: it keeps the left rows that have a match and adds no columns from the right table. That is what "without adding customer columns" asks for. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["orders", "customers"], output_tables=["known_orders"], ) def keep_known_orders(orders: TableFrameSpec, customers: TableFrameSpec) -> TableFrameSpec: return orders.join(customers, on="customer_id", how="semi") ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/known\_orders is live — same three columns as orders, 2 of 3 rows: ``` ┌──────────┬─────────────┬────────┐ │ order_id │ customer_id │ amount │ ├──────────┼─────────────┼────────┤ │ o1 │ c1 │ 10 │ ├──────────┼─────────────┼────────┤ │ o2 │ c2 │ 20 │ └──────────┴─────────────┴────────┘ ``` Transformed tables orders | order\_id | customer\_id | amount | | --- | --- | --- | | o1 | c1 | 10 | | o2 | c2 | 20 | | o3 | c9 | 5 | 3 rows customers | customer\_id | name | | --- | --- | | c1 | Ada | | c2 | Grace | | c3 | Cy | 3 rows known\_orders | order\_id | customer\_id | amount | | --- | --- | --- | | o1 | c1 | 10 | | o2 | c2 | 20 | 2 rows AI agent session ❯make a new table that takes the orders and customers tables and keeps the orders whose customer\_id column is not in the customers table ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — how="anti" is the inverse of semi, keeping the left rows with no match and adding no columns. This is the shape to reach for when you are hunting orphans. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["orders", "customers"], output_tables=["orphan_orders"], ) def find_orphan_orders(orders: TableFrameSpec, customers: TableFrameSpec) -> TableFrameSpec: return orders.join(customers, on="customer_id", how="anti") ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/orphan\_orders is live — o3 alone, the order pointing at c9: ``` ┌──────────┬─────────────┬────────┐ │ order_id │ customer_id │ amount │ ├──────────┼─────────────┼────────┤ │ o3 │ c9 │ 5 │ └──────────┴─────────────┴────────┘ ``` Transformed tables orders | order\_id | customer\_id | amount | | --- | --- | --- | | o1 | c1 | 10 | | o2 | c2 | 20 | | o3 | c9 | 5 | 3 rows customers | customer\_id | name | | --- | --- | | c1 | Ada | | c2 | Grace | | c3 | Cy | 3 rows orphan\_orders | order\_id | customer\_id | amount | | --- | --- | --- | | o3 | c9 | 5 | 1 row ## Join on more than one column AI agent session ❯make a new table that takes the visits and targets tables and joins them on both the customer\_id and date columns ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `on` takes a list when the key is more than one column, and a row matches only when every column in the list agrees. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["visits", "targets"], output_tables=["visits_vs_target"], ) def join_on_day(visits: TableFrameSpec, targets: TableFrameSpec) -> TableFrameSpec: return visits.join(targets, on=["customer_id", "date"]) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/visits\_vs\_target is live — the c1/tue visit had no target row for that day, so it dropped: ``` ┌─────────────┬──────┬───────┬────────┐ │ customer_id │ date │ spend │ target │ ├─────────────┼──────┼───────┼────────┤ │ c1 │ mon │ 10 │ 12 │ ├─────────────┼──────┼───────┼────────┤ │ c2 │ mon │ 7 │ 5 │ └─────────────┴──────┴───────┴────────┘ ``` Transformed tables visits | customer\_id | date | spend | | --- | --- | --- | | c1 | mon | 10 | | c2 | mon | 7 | | c1 | tue | 4 | 3 rows targets | customer\_id | date | target | | --- | --- | --- | | c1 | mon | 12 | | c2 | mon | 5 | 2 rows visits\_vs\_target | customer\_id | date | spend | target | | --- | --- | --- | --- | | c1 | mon | 10 | 12 | | c2 | mon | 7 | 5 | 2 rows AI agent session ❯make a new table that takes the visits and targets tables and joins them where the key columns are called cust\_id and visit\_date on one side ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — the names differ between the two tables, so `on` will not do it. `left_on` and `right_on` take the two lists instead, and they have to be the same length and in the same order. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["visits", "targets"], output_tables=["visits_vs_target"], ) def join_on_day(visits: TableFrameSpec, targets: TableFrameSpec) -> TableFrameSpec: return visits.join( targets, left_on=["cust_id", "visit_date"], right_on=["customer_id", "date"], ) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/visits\_vs\_target is live — matched on the same two values under their own names: ``` ┌─────────┬────────────┬───────┬────────┐ │ cust_id │ visit_date │ spend │ target │ ├─────────┼────────────┼───────┼────────┤ │ c1 │ mon │ 10 │ 12 │ ├─────────┼────────────┼───────┼────────┤ │ c2 │ mon │ 7 │ 5 │ └─────────┴────────────┴───────┴────────┘ ``` Transformed tables visits | cust\_id | visit\_date | spend | | --- | --- | --- | | c1 | mon | 10 | | c2 | mon | 7 | | c1 | tue | 4 | 3 rows targets | customer\_id | date | target | | --- | --- | --- | | c1 | mon | 12 | | c2 | mon | 5 | 2 rows visits\_vs\_target | cust\_id | visit\_date | spend | target | | --- | --- | --- | --- | | c1 | mon | 10 | 12 | | c2 | mon | 7 | 5 | 2 rows ## Handle duplicate column names AI agent session ❯make a new table that takes the orders and returns tables and joins them on the id column, renaming the amount column that both of them carry ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — both sides carry amount, and only the join key is merged into one column. `suffix` renames the right-hand copy of every other collision. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["orders", "returns"], output_tables=["orders_returns"], ) def join_returns(orders: TableFrameSpec, returns: TableFrameSpec) -> TableFrameSpec: return orders.join(returns, on="id", suffix="_return") ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/orders\_returns is live — id appears once, and the returns table's amount came through as amount\_return: ``` ┌────┬────────┬───────────────┐ │ id │ amount │ amount_return │ ├────┼────────┼───────────────┤ │ o1 │ 10 │ 4 │ ├────┼────────┼───────────────┤ │ o2 │ 20 │ 20 │ └────┴────────┴───────────────┘ ``` Transformed tables orders | id | amount | | --- | --- | | o1 | 10 | | o2 | 20 | 2 rows returns | id | amount | | --- | --- | | o1 | 4 | | o2 | 20 | 2 rows orders\_returns | id | amount | amount\_return | | --- | --- | --- | | o1 | 10 | 4 | | o2 | 20 | 20 | 2 rows --- # Concatenate Tables > Concatenate tables with `concat()` and its `how` modes (vertical, vertical_relaxed, diagonal, diagonal_relaxed), with transformer examples and the None-handling rules. Source: https://docs.tabsdata.com/tutorials/transformations/concatenate-tables ## Stack tables into one `how` decides what `concat()` does when the inputs do not line up. AI agent session ❯make a new table that takes the sales\_east and sales\_west tables and stacks their rows into one table ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `concat()` appends rows, and the default how="vertical" is right here because both inputs carry the same three columns with the same types. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["sales_east", "sales_west"], output_tables=["sales_all"], ) def combine_sales(east: TableFrameSpec, west: TableFrameSpec) -> TableFrameSpec: return concat([east, west]) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/sales\_all is live — 2 rows from sales\_east then 2 from sales\_west, in that order, columns unchanged: ``` ┌────────┬───────┬────────┐ │ region │ month │ amount │ ├────────┼───────┼────────┤ │ east │ jan │ 10 │ ├────────┼───────┼────────┤ │ east │ feb │ 20 │ ├────────┼───────┼────────┤ │ west │ jan │ 30 │ ├────────┼───────┼────────┤ │ west │ feb │ 40 │ └────────┴───────┴────────┘ ``` Transformed tables sales\_east | region | month | amount | | --- | --- | --- | | east | jan | 10 | | east | feb | 20 | 2 rows sales\_west | region | month | amount | | --- | --- | --- | | west | jan | 30 | | west | feb | 40 | 2 rows sales\_all | region | month | amount | | --- | --- | --- | | east | jan | 10 | | east | feb | 20 | | west | jan | 30 | | west | feb | 40 | 4 rows AI agent session ❯make a new table that takes the sales\_east and sales\_west tables and stacks their rows into one table, casting a column that is typed differently in each ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — the columns match and the types do not, so how="vertical\_relaxed" casts amount to a common supertype instead of refusing. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["sales_east", "sales_west"], output_tables=["sales_all"], ) def combine_sales(east: TableFrameSpec, west: TableFrameSpec) -> TableFrameSpec: return concat([east, west], how="vertical_relaxed") ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/sales\_all is live — amount came out Float64, so the Int64 value from sales\_east was cast: ``` ┌────────┬────────┐ │ region │ amount │ ├────────┼────────┤ │ east │ 10.0 │ ├────────┼────────┤ │ west │ 30.5 │ └────────┴────────┘ ``` Transformed tables sales\_east | regionString | amountInt64 | | --- | --- | | east | 10 | 1 row sales\_west | regionString | amountFloat64 | | --- | --- | | west | 30.5 | 1 row sales\_all | regionString | amountFloat64 | | --- | --- | | east | 10.0 | | west | 30.5 | 2 rows AI agent session ❯make a new table that takes the sales\_east and sales\_west tables and stacks their rows into one table, unioning the columns only one of them has ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — the columns differ, which how="vertical" rejects. how="diagonal" takes the union of the columns and fills the gaps with null. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["sales_east", "sales_west"], output_tables=["sales_all"], ) def combine_sales(east: TableFrameSpec, west: TableFrameSpec) -> TableFrameSpec: return concat([east, west], how="diagonal") ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/sales\_all is live — 4 columns, the union of both inputs. The east row has no discount and the west row has no amount: ``` ┌────────┬───────┬────────┬──────────┐ │ region │ month │ amount │ discount │ ├────────┼───────┼────────┼──────────┤ │ east │ jan │ 10 │ null │ ├────────┼───────┼────────┼──────────┤ │ west │ jan │ null │ 5 │ └────────┴───────┴────────┴──────────┘ ``` Transformed tables sales\_east | region | month | amount | | --- | --- | --- | | east | jan | 10 | 1 row sales\_west | region | month | discount | | --- | --- | --- | | west | jan | 5 | 1 row sales\_all | region | month | amount | discount | | --- | --- | --- | --- | | east | jan | 10 | null | | west | jan | null | 5 | 2 rows AI agent session ❯make a new table that takes the sales\_east and sales\_west tables and stacks their rows into one table, reconciling both the columns and the types ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — the inputs differ in both their columns and their types, so this needs how="diagonal\_relaxed", which unions the columns and casts to a supertype in one pass. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["sales_east", "sales_west"], output_tables=["sales_all"], ) def combine_sales(east: TableFrameSpec, west: TableFrameSpec) -> TableFrameSpec: return concat([east, west], how="diagonal_relaxed") ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/sales\_all is live — the union of both column sets, each column keeping the type it needs: ``` ┌────────┬────────┬──────────┐ │ region │ amount │ discount │ ├────────┼────────┼──────────┤ │ east │ 10 │ null │ ├────────┼────────┼──────────┤ │ west │ null │ 5.5 │ └────────┴────────┴──────────┘ ``` Transformed tables sales\_east | regionString | amountInt64 | | --- | --- | | east | 10 | 1 row sales\_west | regionString | discountFloat64 | | --- | --- | | west | 5.5 | 1 row sales\_all | regionString | amountInt64 | discountFloat64 | | --- | --- | --- | | east | 10 | null | | west | null | 5.5 | 2 rows note `None` entries in the list are skipped. A list that is `None`, empty, or empty once the `None` entries are dropped makes `concat()` return `None`, which leaves the output table on its current version instead of committing a new one. --- # Clean and Sort Data > Covers handling null and NaN values, sorting rows with sort() and its descending and nulls_last options, and changing column types with cast(), str.to_date/to_datetime/to_time/to_integer, and dt.to_string in Tabsdata transformers. Source: https://docs.tabsdata.com/tutorials/transformations/clean-data ## Handle null values AI agent session ❯make a new table that takes the contacts table and drops the rows that have a null in any column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `drop_nulls()` with no arguments looks at every column, so one null anywhere takes the row. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["contacts"], output_tables=["contacts_complete"], ) def drop_incomplete(contacts: TableFrameSpec) -> TableFrameSpec: return contacts.drop_nulls() ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/contacts\_complete is live — the row with a null email went, 2 of 3 rows kept: ``` ┌──────┬───────────┐ │ name │ email │ ├──────┼───────────┤ │ Ada │ ada@x.com │ ├──────┼───────────┤ │ Cy │ cy@x.com │ └──────┴───────────┘ ``` Transformed tables contacts | name | email | | --- | --- | | Ada | ada@x.com | | Bo | null | | Cy | cy@x.com | 3 rows contacts\_complete | name | email | | --- | --- | | Ada | ada@x.com | | Cy | cy@x.com | 2 rows AI agent session ❯make a new table that takes the contacts table and drops the rows with a null in the email or phone column, ignoring nulls elsewhere ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `drop_nulls(subset=[...])` narrows the check to those columns, so a null in any other column no longer costs you the row. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["contacts"], output_tables=["contacts_reachable"], ) def drop_unreachable(contacts: TableFrameSpec) -> TableFrameSpec: return contacts.drop_nulls(subset=["email", "phone"]) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/contacts\_reachable is live — only email and phone were checked: ``` ┌──────┬───────────┬──────────┐ │ name │ email │ phone │ ├──────┼───────────┼──────────┤ │ Ada │ ada@x.com │ 555-0101 │ └──────┴───────────┴──────────┘ ``` Transformed tables contacts | name | email | phone | | --- | --- | --- | | Ada | ada@x.com | 555-0101 | | Bo | null | 555-0102 | | Cy | cy@x.com | null | 3 rows contacts\_reachable | name | email | phone | | --- | --- | --- | | Ada | ada@x.com | 555-0101 | 1 row AI agent session ❯make a new table that takes the stock table and replaces every null in every column with 0 ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `fill_null(0)` applies one value to every column, which only makes sense when they are all numeric. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["stock"], output_tables=["stock_filled"], ) def fill_zeros(stock: TableFrameSpec) -> TableFrameSpec: return stock.fill_null(0) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/stock\_filled is live — every null became 0: ``` ┌────────┬──────────┐ │ item │ quantity │ ├────────┼──────────┤ │ widget │ 4 │ ├────────┼──────────┤ │ gadget │ 0 │ └────────┴──────────┘ ``` Transformed tables stock | item | quantity | | --- | --- | | widget | 4 | | gadget | null | 2 rows stock\_filled | item | quantity | | --- | --- | | widget | 4 | | gadget | 0 | 2 rows AI agent session ❯make a new table that takes the accounts table and replaces the nulls in the status column with "unknown", leaving the other columns alone ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — a per-column fill, so it goes through an expression and `with_columns` rather than the frame-wide `fill_null`. Other columns keep their nulls. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["accounts"], output_tables=["accounts_status_filled"], ) def fill_status(accounts: TableFrameSpec) -> TableFrameSpec: expr = col("status").fill_null("unknown") return accounts.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/accounts\_status\_filled is live — status filled, and the nulls in other columns left as they were: ``` ┌──────┬─────────┐ │ name │ status │ ├──────┼─────────┤ │ Ada │ active │ ├──────┼─────────┤ │ Bo │ unknown │ └──────┴─────────┘ ``` Transformed tables accounts | name | status | | --- | --- | | Ada | active | | Bo | null | 2 rows accounts\_status\_filled | name | status | | --- | --- | | Ada | active | | Bo | unknown | 2 rows AI agent session ❯make a new table that takes the orders table and fills the nulls with a different value in each column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `fill_null()` takes a dict when each column needs its own value, which beats one `with_columns` expression per column. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["orders"], output_tables=["orders_filled"], ) def fill_defaults(orders: TableFrameSpec) -> TableFrameSpec: return orders.fill_null({ "quantity": 0, "status": "pending", "price": 0.0 }) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/orders\_filled is live — each column took its own default: ``` ┌──────────┬─────────┬───────┐ │ quantity │ status │ price │ ├──────────┼─────────┼───────┤ │ 4 │ pending │ 9.99 │ ├──────────┼─────────┼───────┤ │ 0 │ shipped │ 0.0 │ └──────────┴─────────┴───────┘ ``` Transformed tables orders | quantity | status | price | | --- | --- | --- | | 4 | null | 9.99 | | null | shipped | null | 2 rows orders\_filled | quantity | status | price | | --- | --- | --- | | 4 | pending | 9.99 | | 0 | shipped | 0.0 | 2 rows ## Handle NaN values AI agent session ❯make a new table that takes the readings table and drops the rows that have a NaN in any column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `drop_nans()` rather than `drop_nulls()`. NaN is a float value, not a missing one, so the null check would leave it in place. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["readings"], output_tables=["readings_valid"], ) def drop_nan_rows(readings: TableFrameSpec) -> TableFrameSpec: return readings.drop_nans() ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/readings\_valid is live — the NaN row went: ``` ┌────┬───────┬────────────┐ │ id │ ratio │ percentage │ ├────┼───────┼────────────┤ │ 1 │ 0.5 │ 90 │ └────┴───────┴────────────┘ ``` Transformed tables readings | id | ratio | percentage | | --- | --- | --- | | 1 | 0.5 | 90 | | 2 | NaN | 80 | | 3 | 0.9 | NaN | 3 rows readings\_valid | id | ratio | percentage | | --- | --- | --- | | 1 | 0.5 | 90 | 1 row AI agent session ❯make a new table that takes the readings table and drops the rows with a NaN in the ratio or percentage column, ignoring NaN elsewhere ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `drop_nans(subset=[...])`, the same narrowing that `drop_nulls` takes. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["readings"], output_tables=["readings_metrics_valid"], ) def drop_nan_metrics(readings: TableFrameSpec) -> TableFrameSpec: return readings.drop_nans(subset=["ratio", "percentage"]) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/readings\_metrics\_valid is live — only ratio and percentage were checked: ``` ┌────┬───────┬────────────┐ │ id │ ratio │ percentage │ ├────┼───────┼────────────┤ │ 1 │ 0.5 │ 90 │ └────┴───────┴────────────┘ ``` Transformed tables readings | id | ratio | percentage | | --- | --- | --- | | 1 | 0.5 | 90 | | 2 | NaN | 80 | | 3 | 0.9 | NaN | 3 rows readings\_metrics\_valid | id | ratio | percentage | | --- | --- | --- | | 1 | 0.5 | 90 | 1 row AI agent session ❯make a new table that takes the readings table and replaces every NaN in every column with 0.0 ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `fill_nan(0.0)` across the frame. The value is a float because NaN only occurs in float columns. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["readings"], output_tables=["readings_nan_filled"], ) def fill_nan_zeros(readings: TableFrameSpec) -> TableFrameSpec: return readings.fill_nan(0.0) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/readings\_nan\_filled is live — every NaN became 0.0: ``` ┌────┬───────┬────────────┐ │ id │ ratio │ percentage │ ├────┼───────┼────────────┤ │ 1 │ 0.5 │ 90 │ ├────┼───────┼────────────┤ │ 2 │ 0.0 │ 80 │ ├────┼───────┼────────────┤ │ 3 │ 0.9 │ 0.0 │ └────┴───────┴────────────┘ ``` Transformed tables readings | id | ratio | percentage | | --- | --- | --- | | 1 | 0.5 | 90 | | 2 | NaN | 80 | | 3 | 0.9 | NaN | 3 rows readings\_nan\_filled | id | ratio | percentage | | --- | --- | --- | | 1 | 0.5 | 90 | | 2 | 0.0 | 80 | | 3 | 0.9 | 0.0 | 3 rows AI agent session ❯make a new table that takes the readings table and replaces the NaN in the rate column with 0.0, leaving the other columns alone ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — per-column again, so an expression through `with_columns`. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["readings"], output_tables=["readings_rate_filled"], ) def fill_rate(readings: TableFrameSpec) -> TableFrameSpec: expr = col("rate").fill_nan(0.0) return readings.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/readings\_rate\_filled is live — rate filled, NaN elsewhere untouched: ``` ┌────┬──────┐ │ id │ rate │ ├────┼──────┤ │ 1 │ 0.5 │ ├────┼──────┤ │ 2 │ 0.0 │ └────┴──────┘ ``` Transformed tables readings | id | rate | | --- | --- | | 1 | 0.5 | | 2 | NaN | 2 rows readings\_rate\_filled | id | rate | | --- | --- | | 1 | 0.5 | | 2 | 0.0 | 2 rows ## Sort rows AI agent session ❯make a new table that takes the events table and sorts the rows by the date column, earliest first ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `sort("date")`, which is ascending by default. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_sorted"], ) def sort_by_date(events: TableFrameSpec) -> TableFrameSpec: return events.sort("date") ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_sorted is live — same rows, earliest date first: ``` ┌──────────┬────────────┬───────┐ │ category │ date │ value │ ├──────────┼────────────┼───────┤ │ a │ 2024-01-15 │ 9 │ ├──────────┼────────────┼───────┤ │ b │ 2024-03-01 │ 2 │ ├──────────┼────────────┼───────┤ │ a │ 2024-05-20 │ 5 │ └──────────┴────────────┴───────┘ ``` Transformed tables events | category | date | value | | --- | --- | --- | | b | 2024-03-01 | 2 | | a | 2024-01-15 | 9 | | a | 2024-05-20 | 5 | 3 rows events\_sorted | category | date | value | | --- | --- | --- | | a | 2024-01-15 | 9 | | b | 2024-03-01 | 2 | | a | 2024-05-20 | 5 | 3 rows AI agent session ❯make a new table that takes the events table and sorts the rows by the date column, latest first ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `descending=True` flips it. The rows are the same rows; only their order changed. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_sorted_desc"], ) def sort_by_date_desc(events: TableFrameSpec) -> TableFrameSpec: return events.sort("date", descending=True) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_sorted\_desc is live — latest date first: ``` ┌──────────┬────────────┬───────┐ │ category │ date │ value │ ├──────────┼────────────┼───────┤ │ a │ 2024-05-20 │ 5 │ ├──────────┼────────────┼───────┤ │ b │ 2024-03-01 │ 2 │ ├──────────┼────────────┼───────┤ │ a │ 2024-01-15 │ 9 │ └──────────┴────────────┴───────┘ ``` Transformed tables events | category | date | value | | --- | --- | --- | | b | 2024-03-01 | 2 | | a | 2024-01-15 | 9 | | a | 2024-05-20 | 5 | 3 rows events\_sorted\_desc | category | date | value | | --- | --- | --- | | a | 2024-05-20 | 5 | | b | 2024-03-01 | 2 | | a | 2024-01-15 | 9 | 3 rows AI agent session ❯make a new table that takes the events table and sorts the rows by the category column ascending then the date column descending ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `sort()` takes a list of columns, and `descending` takes a list the same length so each column gets its own direction. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_sorted_grouped"], ) def sort_by_category(events: TableFrameSpec) -> TableFrameSpec: return events.sort(["category", "date"], descending=[False, True]) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_sorted\_grouped is live — grouped by category, newest first inside each: ``` ┌──────────┬────────────┬───────┐ │ category │ date │ value │ ├──────────┼────────────┼───────┤ │ a │ 2024-05-20 │ 5 │ ├──────────┼────────────┼───────┤ │ a │ 2024-01-15 │ 9 │ ├──────────┼────────────┼───────┤ │ b │ 2024-03-01 │ 2 │ └──────────┴────────────┴───────┘ ``` Transformed tables events | category | date | value | | --- | --- | --- | | b | 2024-03-01 | 2 | | a | 2024-01-15 | 9 | | a | 2024-05-20 | 5 | 3 rows events\_sorted\_grouped | category | date | value | | --- | --- | --- | | a | 2024-05-20 | 5 | | a | 2024-01-15 | 9 | | b | 2024-03-01 | 2 | 3 rows AI agent session ❯make a new table that takes the readings table and sorts the rows by the value column with the nulls at the bottom ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `nulls_last=True`. Without it the nulls sort first, which is rarely what a reader of the table wants. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["readings"], output_tables=["readings_sorted"], ) def sort_nulls_last(readings: TableFrameSpec) -> TableFrameSpec: return readings.sort("value", nulls_last=True) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/readings\_sorted is live — nulls pushed to the bottom: ``` ┌────┬───────┐ │ id │ value │ ├────┼───────┤ │ 3 │ 3 │ ├────┼───────┤ │ 1 │ 7 │ ├────┼───────┤ │ 2 │ null │ └────┴───────┘ ``` Transformed tables readings | id | value | | --- | --- | | 1 | 7 | | 2 | null | | 3 | 3 | 3 rows readings\_sorted | id | value | | --- | --- | | 3 | 3 | | 1 | 7 | | 2 | null | 3 rows ## Change a column type AI agent session ❯make a new table that takes the records table and casts the id column from text to an integer ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `cast(Int64)` on the column through `with_columns`. The values read the same, the type does not. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["records"], output_tables=["records_typed"], ) def cast_id(records: TableFrameSpec) -> TableFrameSpec: expr = col("id").cast(Int64) return records.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/records\_typed is live — id is Int64 now: ``` ┌────┐ │ id │ ├────┤ │ 1 │ ├────┤ │ 2 │ └────┘ ``` Transformed tables records | idString | | --- | | 1 | | 2 | 2 rows records\_typed | idInt64 | | --- | | 1 | | 2 | 2 rows AI agent session ❯make a new table that takes the items table and casts the price column to a float ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `cast(Float64)`. Same values, wider type, which starts to matter once you divide with it. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["items"], output_tables=["items_typed"], ) def cast_price(items: TableFrameSpec) -> TableFrameSpec: expr = col("price").cast(Float64) return items.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/items\_typed is live — price is Float64 now: ``` ┌───────┐ │ price │ ├───────┤ │ 19.0 │ ├───────┤ │ 4.0 │ └───────┘ ``` Transformed tables items | priceInt64 | | --- | | 19 | | 4 | 2 rows items\_typed | priceFloat64 | | --- | | 19.0 | | 4.0 | 2 rows AI agent session ❯make a new table that takes the accounts table and casts the is\_active column to true and false rather than 1 and 0 ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `cast(Boolean)`, which reads 1 as true and 0 as false. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["accounts"], output_tables=["accounts_typed"], ) def cast_active(accounts: TableFrameSpec) -> TableFrameSpec: expr = col("is_active").cast(Boolean) return accounts.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/accounts\_typed is live — is\_active is Boolean now: ``` ┌───────────┐ │ is_active │ ├───────────┤ │ true │ ├───────────┤ │ false │ └───────────┘ ``` Transformed tables accounts | is\_activeInt64 | | --- | | 1 | | 0 | 2 rows accounts\_typed | is\_activeBoolean | | --- | | true | | false | 2 rows AI agent session ❯make a new table that takes the orders table and casts the id column to text and the amount column to a float ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `cast()` on the frame takes a dict of column to type, shorter than one expression per column. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["orders"], output_tables=["orders_typed"], ) def cast_columns(orders: TableFrameSpec) -> TableFrameSpec: return orders.cast({"id": String, "amount": Float64}) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/orders\_typed is live — both columns retyped in one pass: ``` ┌────┬────────┐ │ id │ amount │ ├────┼────────┤ │ 1 │ 19.0 │ ├────┼────────┤ │ 2 │ 4.0 │ └────┴────────┘ ``` Transformed tables orders | idInt64 | amountInt64 | | --- | --- | | 1 | 19 | | 2 | 4 | 2 rows orders\_typed | idString | amountFloat64 | | --- | --- | | 1 | 19.0 | | 2 | 4.0 | 2 rows AI agent session ❯make a new table that takes the events table and parses the date\_str column into a real date ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.to_date()` with the format string. A cast will not do it: the text has to be parsed, not reinterpreted. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_dated"], ) def parse_date(events: TableFrameSpec) -> TableFrameSpec: expr = col("date_str").str.to_date("%Y-%m-%d") return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_dated is live — date\_str is a Date now: ``` ┌────────────┐ │ date_str │ ├────────────┤ │ 2024-03-15 │ ├────────────┤ │ 2024-11-01 │ └────────────┘ ``` Transformed tables events | date\_strString | | --- | | 2024-03-15 | | 2024-11-01 | 2 rows events\_dated | date\_strDate | | --- | | 2024-03-15 | | 2024-11-01 | 2 rows AI agent session ❯make a new table that takes the events table and parses the ts\_str column into a datetime ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.to_datetime()` with a format that carries the time as well as the date. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_stamped"], ) def parse_datetime(events: TableFrameSpec) -> TableFrameSpec: expr = col("ts_str").str.to_datetime("%Y-%m-%d %H:%M:%S") return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_stamped is live — ts\_str is a Datetime now: ``` ┌─────────────────────┐ │ ts_str │ ├─────────────────────┤ │ 2024-03-15 09:42:17 │ ├─────────────────────┤ │ 2024-11-01 23:05:00 │ └─────────────────────┘ ``` Transformed tables events | ts\_strString | | --- | | 2024-03-15 09:42:17 | | 2024-11-01 23:05:00 | 2 rows events\_stamped | ts\_strDatetime | | --- | | 2024-03-15 09:42:17 | | 2024-11-01 23:05:00 | 2 rows AI agent session ❯make a new table that takes the events table and parses the time\_str column into a time of day ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.to_time()`, which keeps the time and attaches no date to it. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_timed"], ) def parse_time(events: TableFrameSpec) -> TableFrameSpec: expr = col("time_str").str.to_time("%H:%M:%S") return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_timed is live — time\_str is a Time now: ``` ┌──────────┐ │ time_str │ ├──────────┤ │ 09:42:17 │ ├──────────┤ │ 23:05:00 │ └──────────┘ ``` Transformed tables events | time\_strString | | --- | | 09:42:17 | | 23:05:00 | 2 rows events\_timed | time\_strTime | | --- | | 09:42:17 | | 23:05:00 | 2 rows AI agent session ❯make a new table that takes the records table and parses the num\_str column into an integer ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `str.to_integer()` parses the digits. `cast(Int64)` reinterprets, this parses. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["records"], output_tables=["records_parsed"], ) def parse_integer(records: TableFrameSpec) -> TableFrameSpec: expr = col("num_str").str.to_integer() return records.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/records\_parsed is live — num\_str is Int64 now: ``` ┌─────────┐ │ num_str │ ├─────────┤ │ 42 │ ├─────────┤ │ 1337 │ └─────────┘ ``` Transformed tables records | num\_strString | | --- | | 42 | | 1337 | 2 rows records\_parsed | num\_strInt64 | | --- | | 42 | | 1337 | 2 rows AI agent session ❯make a new table that takes the events table and formats the date column as text in YYYY-MM-DD ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — the other direction, so `dt.to_string()` with the format you want out. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["events"], output_tables=["events_formatted"], ) def format_date(events: TableFrameSpec) -> TableFrameSpec: expr = col("date").dt.to_string("%Y-%m-%d") return events.with_columns(expr) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/events\_formatted is live — date is text now: ``` ┌────────────┐ │ date │ ├────────────┤ │ 2024-03-15 │ ├────────────┤ │ 2024-11-01 │ └────────────┘ ``` Transformed tables events | dateDate | | --- | | 2024-03-15 | | 2024-11-01 | 2 rows events\_formatted | dateString | | --- | | 2024-03-15 | | 2024-11-01 | 2 rows --- # Inspect a TableFrame > Inspect a TableFrame's schema, dtypes, column checks, emptiness, and query plan with tf.dtypes, has_cols, assert_has_cols, has_same_schema, is_empty, empty, clear, explain, and show_graph. Source: https://docs.tabsdata.com/tutorials/transformations/inspect-a-tableframe Access metadata about your TableFrame. ## Schema Information transform.py ```python # Get column data types dtypes = tf.dtypes # Check if columns exist tf.has_cols("column_name") tf.has_cols(["col1", "col2"]) # Assert columns exist (raises error if not) tf.assert_has_cols(["required_col1", "required_col2"]) # Compare schemas tf.has_same_schema(other_tf) ``` ## Empty Checks transform.py ```python # Check if TableFrame is empty tf.is_empty() # Create an empty TableFrame with same schema empty_tf = tf.empty() # Clear all rows cleared_tf = tf.clear() ``` ## Query Explanation For debugging and optimization: transform.py ```python # Get query plan explanation plan = tf.explain() print(plan) # Visualize query graph tf.show_graph() ``` See [TableFrame API Guide](https://docs.tabsdata.com/guide/tables) for the whole API in one place. --- # Reshape Data > Covers reshaping Tabsdata tables with transformers: pivoting rows to columns with extract_as_columns(), unpivoting with extract_as_rows(), and expanding structs with unnest(). Source: https://docs.tabsdata.com/tutorials/transformations/reshape-data ## Turn row values into columns AI agent session ❯make a new table that takes the metrics table and gives one row per host and turns each value of the metric\_name column into its own column ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — the metric names are values in a column rather than columns, so this is `extract_as_columns()`. It takes the column holding the names and the column holding the numbers, and everything not named becomes the key that rows collapse on. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["metrics"], output_tables=["metrics_wide"], ) def widen_metrics(metrics: TableFrameSpec) -> TableFrameSpec: return metrics.extract_as_columns( column="metric_name", value_column="metric_value", ) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/metrics\_wide is live — 4 rows became 2, one per host, with cpu and mem as columns: ``` ┌──────┬─────┬─────┐ │ host │ cpu │ mem │ ├──────┼─────┼─────┤ │ web1 │ 80 │ 60 │ ├──────┼─────┼─────┤ │ web2 │ 35 │ 45 │ └──────┴─────┴─────┘ ``` Transformed tables metrics | host | metric\_name | metric\_value | | --- | --- | --- | | web1 | cpu | 80 | | web1 | mem | 60 | | web2 | cpu | 35 | | web2 | mem | 45 | 4 rows metrics\_wide | host | cpu | mem | | --- | --- | --- | | web1 | 80 | 60 | | web2 | 35 | 45 | 2 rows ## Turn columns into rows AI agent session ❯make a new table that takes the sales table and folds the three month columns into one row each, per region ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `extract_as_rows()`, the inverse of extract\_as\_columns. It takes the columns to fold in, plus names for the two columns it produces: one holding the old column name, one holding its value. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["sales"], output_tables=["sales_long"], ) def lengthen_sales(sales: TableFrameSpec) -> TableFrameSpec: return sales.extract_as_rows( columns=["jan_sales", "feb_sales", "mar_sales"], variable_name="month", value_name="sales", ) ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/sales\_long is live — 1 row became 3, and month carries the old column names verbatim: ``` ┌────────┬───────────┬───────┐ │ region │ month │ sales │ ├────────┼───────────┼───────┤ │ east │ jan_sales │ 10 │ ├────────┼───────────┼───────┤ │ east │ feb_sales │ 20 │ ├────────┼───────────┼───────┤ │ east │ mar_sales │ 30 │ └────────┴───────────┴───────┘ ``` Transformed tables sales | region | jan\_sales | feb\_sales | mar\_sales | | --- | --- | --- | --- | | east | 10 | 20 | 30 | 1 row sales\_long | region | month | sales | | --- | --- | --- | | east | jan\_sales | 10 | | east | feb\_sales | 20 | | east | mar\_sales | 30 | 3 rows ## Expand a struct column AI agent session ❯make a new table that takes the people table and expands the fields inside the address struct column into their own columns ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — `unnest("address")` replaces the struct column with one column per field, taking the field names as the new column names. The row count does not change. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in subscriptions: ```python @transformer( input_tables=["people"], output_tables=["people_flat"], ) def flatten_address(people: TableFrameSpec) -> TableFrameSpec: return people.unnest("address") ``` ⏺ Execution committed, 0 failures. ⏺ subscriptions/people\_flat is live — address is gone, replaced by city and zip: ``` ┌───────┬───────────┬───────┐ │ name │ city │ zip │ ├───────┼───────────┼───────┤ │ Ada │ Austin │ 78701 │ ├───────┼───────────┼───────┤ │ Grace │ Arlington │ 22201 │ └───────┴───────────┴───────┘ ``` Transformed tables people | name | address | | --- | --- | | Ada | {city: Austin, zip: 78701} | | Grace | {city: Arlington, zip: 22201} | 2 rows people\_flat | name | city | zip | | --- | --- | --- | | Ada | Austin | 78701 | | Grace | Arlington | 22201 | 2 rows --- # Subscribe data to Amazon S3 > Writes Tabsdata tables to Amazon S3 as Parquet objects using an S3Dest subscriber, covering the paths, input_tables, connection YAML, and trigger_by options. Source: https://docs.tabsdata.com/tutorials/subscribers/amazon-s3 Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) ## Write tables out to Amazon S3 One entry in `paths` per input table, in order, so each table lands as its own Parquet object. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you write the hr departments and employees tables out to s3 as parquet Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ That is a subscriber, so it goes in a destination collection rather than a source one. The guide and connection template are loaded. Three values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - destination address and credentials - target objects ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ Credentials found. Reading back what is not secret: bucket acme-exports, region us-east-1, base path /hr, collection exports. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **exports** created from it in group `destinations`. Connection check passed. Called tabsdata 2 times (ctrl+o to expand) ⏺ The subscriber, registered in `exports`: ```python from tabsdatak.api import subscriber, TableFrameSpec from tabsdatak.conn.s3 import S3Dest @subscriber( destination=S3Dest( paths=["departments_${EXEC_PLAN_TS}.parquet", "employees_${EXEC_PLAN_TS}.parquet"], ), input_tables=["hr/departments", "hr/employees"], ) def write_files( departments: TableFrameSpec, employees: TableFrameSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return (departments, employees) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ Two objects written under `/hr`, one per input table: ``` ┌─────────────────────┬────────────────┬───────┬────────┐ │ object │ from table │ rows │ bytes │ ├─────────────────────┼────────────────┼───────┼────────┤ │ departments.parquet │ hr/departments │ 14 │ 2.1 KB │ ├─────────────────────┼────────────────┼───────┼────────┤ │ employees.parquet │ hr/employees │ 1,204 │ 86 KB │ └─────────────────────┴────────────────┴───────┴────────┘ ``` Input tables are written as `collection/table`, since a subscriber reads from collections other than its own. A subscriber declares no `output_tables`: it returns one frame per destination slot, and returning `None` for a slot writes nothing that run. Without `trigger_by`, the subscriber runs whenever any of its input tables gets a new commit, so an export stays current without a schedule. --- # Subscribe data to Apache Druid > Writes Tabsdata tables out to Apache Druid with the DruidDest subscriber, covering the DruidDestConn connection, S3 staging, if_table_exists, and time_column settings. Source: https://docs.tabsdata.com/tutorials/subscribers/druid Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) ## Write tables out to Apache Druid One entry in `tables` per input table, in order. `if_table_exists="replace"` rebuilds each destination table from the frame, so a changed schema does not fail the run. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you write the sales vendors and items tables out to druid Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ That is a subscriber, so it goes in a destination collection rather than a source one. The guide and connection template are loaded. Three values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - destination address and credentials - target tables ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ Both credential sets found, the database one and the staging one. Reading back what is not secret: coordinator druid.internal:8081, staging in acme-staging, collection analytics. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **analytics** created from it in group `destinations`. Connection check passed. Called tabsdata 2 times (ctrl+o to expand) ⏺ The subscriber, registered in `analytics`: ```python from tabsdatak.api import subscriber, TableFrameSpec from tabsdatak.conn.druid import DruidDest @subscriber( destination=DruidDest( tables=["vendors", "items"], if_table_exists="replace", time_column="created_at", segment_granularity="DAY", ), input_tables=["sales/vendors", "sales/items"], ) def write_sales( vendors: TableFrameSpec, items: TableFrameSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return (vendors, items) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. The frames were staged as Parquet in acme-staging, then loaded. ⏺ Two tables written in **analytics**, one per input table: ``` ┌─────────┬───────────────┬───────┐ │ table │ from table │ rows │ ├─────────┼───────────────┼───────┤ │ vendors │ sales/vendors │ 312 │ ├─────────┼───────────────┼───────┤ │ items │ sales/items │ 8,940 │ └─────────┴───────────────┴───────┘ ``` Input tables are written as `collection/table`, since a subscriber reads from collections other than its own. A subscriber declares no `output_tables`: it returns one frame per destination slot, and returning `None` for a slot writes nothing that run. Without `trigger_by`, the subscriber runs whenever any of its input tables gets a new commit, so an export stays current without a schedule. The connector ships separately from the core package, so `pip install 'tabsdata[druid]'` has to have run in the environment the server uses. --- # Register tables in AWS Glue > Subscriber setup for writing Tabsdata tables to Amazon S3 as Parquet files and registering them in AWS Glue Data Catalog as Iceberg tables, using td.S3Destination and td.AWSGlue. Source: https://docs.tabsdata.com/tutorials/catalogs/aws-glue You can use a built-in Tabsdata subscriber to write tables from the Tabsdata to Amazon S3 as parquet files and add the corresponding metadata to AWS Glue Data Catalog. This also enables Iceberg tables in AWS. ## Example (Subscriber - AWS Glue - Iceberg tables) Here is an example subscriber named `write_employees`. It reads departments table and multiple employees tables from Tabsdata. Subsequently, it writes the tables to the output HR folder. This subscriber executes automatically as soon as a new commit occurs on any of its input tables. ```python import tabsdata as td s3_credentials = td.S3AccessKeyCredentials( aws_access_key_id=td.HashiCorpSecret("path-to-secret","S3_ACCESS_KEY"), aws_secret_access_key=td.HashiCorpSecret("path-to-secret","S3_SECRET_KEY"), ) catalog_definition = [ { "name": "default", "type": "glue", "client.region": "us-east-1", } ] @td.subscriber( tables=["departments", "employees_1", "employees_2"], destination=td.S3Destination( [ "s3://opt/hr/departments.parquet", "s3://opt/hr/employees_*.parquet", ], s3_credentials, region = "us-east-1", catalog = td.AWSGlue( definition=catalog_definition, tables=[ "iceberg_namespace.table", "iceberg_namespace_2.table", "iceberg_namespace_3.table", ], credentials = s3_credentials, .... ), ), ) def write_employees(tf1: td.TableFrame, tf2: td.TableFrame, tf3: td.TableFrame): return tf1, tf2, tf3 ``` Where: `S3_ACCESS_KEY` is the value of your Amazon S3 access key. `S3_SECRET_KEY` is the value of your Amazon S3 secret key. note - After defining the function, you need to register it with a Tabsdata collection. For more information, see here. ## Setup (Subscriber - AWS Glue - Iceberg tables) The following code uses placeholder values for defining a subscriber to write tables from the Tabsdata server to Amazon S3 as parquet files and and add the corresponding metadata to AWS Glue Data Catalog. ```python import tabsdata as td s3_credentials = td.S3AccessKeyCredentials( aws_access_key_id=td.HashiCorpSecret("path-to-secret","S3_ACCESS_KEY"), aws_secret_access_key=td.HashiCorpSecret("path-to-secret","S3_SECRET_KEY"), ) @td.subscriber( tables=["", ""], destination=td.S3Destination( ["s3://", "s3://"], credentials = s3_credentials, region = "", catalog = td.AWSGlue( definition = "", tables = ["",""], auto_create_at = ["",""], if_table_exists = "", partitioned_table = "", schema_strategy = "", ), ), trigger_by=["", ""]), def (: td.TableFrame, : td.TableFrame): return , ``` Following properties are defined in the setup code above: ### tables ``, ``… are the names of the Tabsdata tables to be written to the external system. ### destination #### `` ``, ``… are the full system directory paths to the parquet files to write. Please note files must be Parquet files. You can specify as many file paths as needed. You can define the destination files in the following ways: **File Path** To write by file path where the file extension is included as part of the file path, define the destination as follows: ```python destination=td.S3Destination(["s3://","s3://"], s3_credentials), ``` ``… have the extensions of the file included in the file name as part of the path. **File Format** To write files by file format where format is declared separately and not in the file name, define the destination as follows: ```python destination=td.S3Destination([ "s3://", "s3://", ], s3_credentials, format="parquet"), ``` `""`, `""`… don’t have the extension in the file name. The extension to all files is mentioned separately in `format` property. #### credentials A subscriber needs credentials to write files from Amazon S3. Here the value is defined using a variable `s3_credentials`. The variable is an object of class `S3AccessKeyCredentials` with following values. `S3_ACCESS_KEY` is the value of your Amazon S3 access key. `S3_SECRET_KEY` is the value of your Amazon S3 secret key. `path-to-secret` is the name of the default key value store in HashiCorp where the credential values are required to be stored. You can use different ways to store the credentials which are highlighted here in the documentation. #### region `` is where your S3 bucket is located. #### catalog This property is used to define the properties of your catalog. ##### definition This property defines the catalog. It is of dictionary format, and matches the [pyiceberg catalog definition](https://py.iceberg.apache.org/configuration/#catalogs). The value is simply passed through. ##### tables These are the names of the Iceberg tables we want to add the parquet files to. They need to have the same number as the parquet files being created in S3. ##### auto\_create\_at If you don’t specify `auto_create_at` the table must exist in the Iceberg catalog. If you want the publisher to create the table if it does not exists, in the `auto_create_at` you have to provide an S3 URI where the entry point of Iceberg metadata of the table will be created. Typically you would put the base path where you write the parquet files. ##### if\_table\_exists This is an optional property to define the strategy to follow when the table already exists. `replace` will create a new table in Iceberg, overwriting the existing one, and `append` will append to the existing data in the table. Defaults to `append`. ##### partitioned\_table This option indicates that the Iceberg table definition corresponds to a partitioned table. If the Iceberg table is a partitioned table, the subscriber function should return an array of TableFrames with each of those TableFrames containing data for one partition only. It is the responsibility of the user publisher creator to partition the data accordingly before returning it. Partition tables can only be used with already existing (and partitioned) Iceberg tables. `auto_create_at` is not supported, and when using partition tables the `if_table_exists` strategy must be `append`. ##### schema\_strategy Schema strategy indicates if the schema in iceberg will be modified (augmented) if necessary to be compatible with the parquet files being uploaded. if using `strict`, it will fail if the schemas differ. #### Important Points to note: - Existing tables in AWS Glue can have their schema updated (augmented) to match the written parquet files. - Tables can be automatically created in AWS Glue using Tabsdata table schema. - The parquet files are registered with AWS Glue as external files for a table. - Future versions will add support for replacing partitions in Iceberg partitioned tables. ### None as an input and output A subscriber may receive and return a `None` value instead of TableFrames. When a subscriber receives a `None` value instead of a TableFrame it means that the requested table dependency version does not exist. When a subscriber returns a `None` value instead of a TableFrame it means there is no new data to write to the external system. This helps in avoiding the creation of multiple copies of the same data. ### \[Optional\] trigger\_by ``, ``… are the names of the tables in the Tabsdata server. A new commit to any of these tables triggers the subscriber. All listed trigger tables must exist in the server before registering the subscriber. Defining trigger tables is optional. If you don’t define the `trigger_by` property, the subscriber will be triggered by any of its input tables. If you define the `trigger_by` property, then only those tables listed in the property can automatically trigger the subscriber. For more information, see Working with Triggers. ### `` `` is the name for the subscriber that you are configuring. ### `` `` governs the processing performed by the subscriber. You can specify function logic to be a simple write or to perform additional processing as needed. For more information about the function logic that you can include, see Working with Tables. ``, ``… are the names for the variables that temporarily store source data for processing. ``, ``… are the output from the function that are written to the external system. --- # Subscribe data to Azure Blob Storage > Writes Tabsdata tables to Azure Blob Storage as Parquet files using an @subscriber with AzureBlobDest, covering connection YAML, account key credentials, paths, and trigger_by. Source: https://docs.tabsdata.com/tutorials/subscribers/azure Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) ## Write tables out to Azure Blob Storage One entry in `paths` per input table, in order, so each table lands as its own Parquet object. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you write the hr departments and employees tables out to azure blob storage as parquet Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ That is a subscriber, so it goes in a destination collection rather than a source one. The guide and connection template are loaded. Three values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - destination address and credentials - target objects ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ Credentials found. Reading back what is not secret: container acme-exports, account acmeexportstorage, base path /hr, collection exports. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **exports** created from it in group `destinations`. Connection check passed. Called tabsdata 2 times (ctrl+o to expand) ⏺ The subscriber, registered in `exports`: ```python from tabsdatak.api import subscriber, TableFrameSpec from tabsdatak.conn.azureblob import AzureBlobDest @subscriber( destination=AzureBlobDest( paths=["departments_${EXEC_PLAN_TS}.parquet", "employees_${EXEC_PLAN_TS}.parquet"], ), input_tables=["hr/departments", "hr/employees"], ) def write_files( departments: TableFrameSpec, employees: TableFrameSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return (departments, employees) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ Two objects written under `/hr`, one per input table: ``` ┌─────────────────────┬────────────────┬───────┬────────┐ │ object │ from table │ rows │ bytes │ ├─────────────────────┼────────────────┼───────┼────────┤ │ departments.parquet │ hr/departments │ 14 │ 2.1 KB │ ├─────────────────────┼────────────────┼───────┼────────┤ │ employees.parquet │ hr/employees │ 1,204 │ 86 KB │ └─────────────────────┴────────────────┴───────┴────────┘ ``` Input tables are written as `collection/table`, since a subscriber reads from collections other than its own. A subscriber declares no `output_tables`: it returns one frame per destination slot, and returning `None` for a slot writes nothing that run. Without `trigger_by`, the subscriber runs whenever any of its input tables gets a new commit, so an export stays current without a schedule. --- # Subscribe data to BigQuery > Writes Tabsdata tables to BigQuery with a subscriber: BigQueryDest connection config, GCP service account credentials, if_table_exists and schema_evolution options, and merge pointers. Source: https://docs.tabsdata.com/tutorials/subscribers/bigquery Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) ## Write tables out to BigQuery One entry in `tables` per input table, in order. `if_table_exists="replace"` rebuilds each destination table from the frame, so a changed schema does not fail the run. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you write the sales vendors and items tables out to bigquery Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ That is a subscriber, so it goes in a destination collection rather than a source one. The guide and connection template are loaded. Three values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - destination address and credentials - target tables ❯everything you need is in Read 2 files, called tabsdata 2 times (ctrl+o to expand) ⏺ Credentials found. Reading back what is not secret: project acme-analytics, dataset warehouse, collection warehouse. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **warehouse** created from it in group `destinations`. Connection check passed. Called tabsdata 2 times (ctrl+o to expand) ⏺ The subscriber, registered in `warehouse`: ```python from tabsdatak.api import subscriber, TableFrameSpec from tabsdatak.conn.bigquery import BigQueryDest @subscriber( destination=BigQueryDest( tables=["vendors", "items"], if_table_exists="replace", schema_evolution="strict", ), input_tables=["sales/vendors", "sales/items"], ) def write_sales( vendors: TableFrameSpec, items: TableFrameSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return (vendors, items) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ Two tables written in **warehouse**, one per input table: ``` ┌─────────┬───────────────┬───────┐ │ table │ from table │ rows │ ├─────────┼───────────────┼───────┤ │ vendors │ sales/vendors │ 312 │ ├─────────┼───────────────┼───────┤ │ items │ sales/items │ 8,940 │ └─────────┴───────────────┴───────┘ ``` Input tables are written as `collection/table`, since a subscriber reads from collections other than its own. A subscriber declares no `output_tables`: it returns one frame per destination slot, and returning `None` for a slot writes nothing that run. Without `trigger_by`, the subscriber runs whenever any of its input tables gets a new commit, so an export stays current without a schedule. ## Merge changes into the destination To apply inserts, updates, and deletes by key, see [Merging into SQL destinations](https://docs.tabsdata.com/guide/functions/sql-merge). The guide covers merge configuration, schema evolution, and retry watermarks. --- # Subscribe data to ClickHouse > Writes Tabsdata subscriber output to ClickHouse tables using ClickHouseDest, with S3 staging, if_table_exists and schema_evolution options, and the tabsdata[clickhouse] install requirement. Source: https://docs.tabsdata.com/tutorials/subscribers/clickhouse Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) ## Write tables out to ClickHouse One entry in `tables` per input table, in order. `if_table_exists="replace"` rebuilds each destination table from the frame, so a changed schema does not fail the run. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you write the sales vendors and items tables out to clickhouse Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ That is a subscriber, so it goes in a destination collection rather than a source one. The guide and connection template are loaded. Three values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - destination address and credentials - target tables ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ Both credential sets found, the database one and the staging one. Reading back what is not secret: clickhouse.internal:8123, database analytics, staging in acme-staging, collection analytics. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **analytics** created from it in group `destinations`. Connection check passed. Called tabsdata 2 times (ctrl+o to expand) ⏺ The subscriber, registered in `analytics`: ```python from tabsdatak.api import subscriber, TableFrameSpec from tabsdatak.conn.clickhouse import ClickHouseDest @subscriber( destination=ClickHouseDest( tables=["vendors", "items"], if_table_exists="replace", schema_evolution="strict", ), input_tables=["sales/vendors", "sales/items"], ) def write_sales( vendors: TableFrameSpec, items: TableFrameSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return (vendors, items) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. The frames were staged as Parquet in acme-staging, then loaded. ⏺ Two tables written in **analytics**, one per input table: ``` ┌─────────┬───────────────┬───────┐ │ table │ from table │ rows │ ├─────────┼───────────────┼───────┤ │ vendors │ sales/vendors │ 312 │ ├─────────┼───────────────┼───────┤ │ items │ sales/items │ 8,940 │ └─────────┴───────────────┴───────┘ ``` Input tables are written as `collection/table`, since a subscriber reads from collections other than its own. A subscriber declares no `output_tables`: it returns one frame per destination slot, and returning `None` for a slot writes nothing that run. Without `trigger_by`, the subscriber runs whenever any of its input tables gets a new commit, so an export stays current without a schedule. The connector ships separately from the core package, so `pip install 'tabsdata[clickhouse]'` has to have run in the environment the server uses. --- # Subscribe data to Databricks > Writes Tabsdata tables to Databricks with a subscriber using DatabricksDest, covering connection spec fields (host_url, token, volume, warehouse, catalog, schema), table options, and merging by key. Source: https://docs.tabsdata.com/tutorials/subscribers/databricks Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) ## Write tables out to Databricks One entry in `tables` per input table, in order. `if_table_exists="replace"` rebuilds each destination table from the frame, so a changed schema does not fail the run. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you write the sales vendors and items tables out to databricks Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ That is a subscriber, so it goes in a destination collection rather than a source one. The guide and connection template are loaded. Three values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - destination address and credentials - target tables ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ Credentials found. Reading back what is not secret: volume main.default.tabsdata\_stage, warehouse analytics\_wh, catalog main, collection warehouse. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **warehouse** created from it in group `destinations`. Connection check passed. Called tabsdata 2 times (ctrl+o to expand) ⏺ The subscriber, registered in `warehouse`: ```python from tabsdatak.api import subscriber, TableFrameSpec from tabsdatak.conn.databricks import DatabricksDest @subscriber( destination=DatabricksDest( tables=["vendors", "items"], if_table_exists="append", schema_evolution="update", ), input_tables=["sales/vendors", "sales/items"], ) def write_sales( vendors: TableFrameSpec, items: TableFrameSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return (vendors, items) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ Two tables written in **warehouse**, one per input table: ``` ┌─────────┬───────────────┬───────┐ │ table │ from table │ rows │ ├─────────┼───────────────┼───────┤ │ vendors │ sales/vendors │ 312 │ ├─────────┼───────────────┼───────┤ │ items │ sales/items │ 8,940 │ └─────────┴───────────────┴───────┘ ``` Input tables are written as `collection/table`, since a subscriber reads from collections other than its own. A subscriber declares no `output_tables`: it returns one frame per destination slot, and returning `None` for a slot writes nothing that run. Without `trigger_by`, the subscriber runs whenever any of its input tables gets a new commit, so an export stays current without a schedule. ## Merge changes into the destination To apply inserts, updates, and deletes by key, see [Merging into SQL destinations](https://docs.tabsdata.com/guide/functions/sql-merge). The guide covers merge configuration, schema evolution, and retry watermarks. --- # Subscribe data to Google Cloud Storage > Covers writing Tabsdata tables to Google Cloud Storage as Parquet objects via a subscriber with GCSDest, including connection YAML, service account credentials, paths, and input tables. Source: https://docs.tabsdata.com/tutorials/subscribers/gcp Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) ## Write tables out to Google Cloud Storage One entry in `paths` per input table, in order, so each table lands as its own Parquet object. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you write the hr departments and employees tables out to google cloud storage as parquet Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ That is a subscriber, so it goes in a destination collection rather than a source one. The guide and connection template are loaded. Three values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - destination address and credentials - target objects ❯everything you need is in Read 2 files, called tabsdata 2 times (ctrl+o to expand) ⏺ Credentials found. Reading back what is not secret: bucket acme-exports, project acme-analytics, base path /hr, collection exports. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **exports** created from it in group `destinations`. Connection check passed. Called tabsdata 2 times (ctrl+o to expand) ⏺ The subscriber, registered in `exports`: ```python from tabsdatak.api import subscriber, TableFrameSpec from tabsdatak.conn.gcs import GCSDest @subscriber( destination=GCSDest( paths=["departments_${EXEC_PLAN_TS}.parquet", "employees_${EXEC_PLAN_TS}.parquet"], ), input_tables=["hr/departments", "hr/employees"], ) def write_files( departments: TableFrameSpec, employees: TableFrameSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return (departments, employees) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ Two objects written under `/hr`, one per input table: ``` ┌─────────────────────┬────────────────┬───────┬────────┐ │ object │ from table │ rows │ bytes │ ├─────────────────────┼────────────────┼───────┼────────┤ │ departments.parquet │ hr/departments │ 14 │ 2.1 KB │ ├─────────────────────┼────────────────┼───────┼────────┤ │ employees.parquet │ hr/employees │ 1,204 │ 86 KB │ └─────────────────────┴────────────────┴───────┴────────┘ ``` Input tables are written as `collection/table`, since a subscriber reads from collections other than its own. A subscriber declares no `output_tables`: it returns one frame per destination slot, and returning `None` for a slot writes nothing that run. Without `trigger_by`, the subscriber runs whenever any of its input tables gets a new commit, so an export stays current without a schedule. --- # Subscribe data to Iceberg on AWS Glue > Subscriber connector that writes Tabsdata tables to Amazon S3 as Parquet files and registers them in AWS Glue Iceberg tables, with catalog, partition, and schema options. Source: https://docs.tabsdata.com/tutorials/subscribers/aws-iceberg Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) You can use this built-in Tabsdata subscriber to write tables from the Tabsdata server to Amazon S3 as parquet files and add those files to AWS Glue Iceberg tables. ## Example (Subscriber - AWS Glue - Iceberg tables) Here is an example subscriber named `write_employees`. It reads departments table and multiple employees tables from Tabsdata. Subsequently, it writes the tables to the output HR folder. This subscriber executes automatically as soon as a new commit occurs on any of its input tables. ```python import tabsdata as td s3_credentials = td.S3AccessKeyCredentials( # HashiCorp is a type of secret store, ideal for production use cases. # For rapid testing, you can refer to the credential values directly. You can also use environment variables as detailed here: https://docs.tabsdata.com/latest/guide/secrets_management/env_variables/main.html. aws_access_key_id=td.HashiCorpSecret("path-to-secret","S3_ACCESS_KEY"), aws_secret_access_key=td.HashiCorpSecret("path-to-secret","S3_SECRET_KEY"), ) catalog_definition = { "name": "default", "type": "glue", "client.region": "us-east-1", } @td.subscriber( tables=["departments", "employees_1", "employees_2"], destination=td.S3Destination( [ "s3://opt/hr/departments.parquet", "s3://opt/hr/employees_*.parquet", ], credentials = s3_credentials, region = "us-east-1", catalog = td.AWSGlue( definition=catalog_definition, tables=[ "glue_dbname.table", "glue_dbname.table", "glue_dbname.table", ], credentials = s3_credentials, .... ), ), trigger_by=["", ""] ), def write_employees(tf1: td.TableFrame, tf2: td.TableFrame, tf3: td.TableFrame): return tf1, tf2, tf3 ``` Where: `S3_ACCESS_KEY` is the value of your Amazon S3 access key. `S3_SECRET_KEY` is the value of your Amazon S3 secret key. note - After defining the function, you need to register it with a Tabsdata collection and execute it. For more information, see Register a Function and Execute a Function. ## Setup (Subscriber - AWS Glue - Iceberg tables) The following code uses placeholder values for defining a subscriber to write tables from the Tabsdata server to Amazon S3 as parquet files and add those files to AWS Glue Iceberg tables. ```python import tabsdata as td s3_credentials = td.S3AccessKeyCredentials( aws_access_key_id=td.HashiCorpSecret("path-to-secret","S3_ACCESS_KEY"), aws_secret_access_key=td.HashiCorpSecret("path-to-secret","S3_SECRET_KEY"), ) catalog_definition = { "name": "default", "type": "glue", "client.region": "", } @td.subscriber( tables=["", ""], destination=td.S3Destination( ["s3://", "s3://"], credentials = s3_credentials, region = "", catalog = td.AWSGlue( definition = catalog_definition, tables = ["",""], credentials = s3_credentials, auto_create_at = ["",""], if_table_exists = "", partitioned_table = "", schema_strategy = "", ), ), trigger_by=["", ""]), def (: td.TableFrame, : td.TableFrame): return , ``` note - After defining the function, you need to register it with a Tabsdata collection and execute it. For more information, see Register a Function and Execute a Function. Following properties are defined in the setup code above: ### tables ``, ``… are the names of the Tabsdata tables to be written to the external system. ### destination #### `` ``, ``… are the full system directory paths to the parquet files to write. Please note files must be Parquet files. You can specify as many file paths as needed. You can also use the following variables within path to files to make the files unique or identifiable by a Tabsdata ID. e.g. `td-iceberg/customers-$EXPORT_TIMESTAMP.parquet` as used in [this Tabsdata tutorial](https://github.com/tabsdata/tutorials/blob/main/t03_csv_iceberg_pub_sub/subscriber.py). | Field | Description | | --- | --- | | $EXECUTION\_ID | The execution plan ID. | | $EXPORT\_TIMESTAMP | Datetime the function started executing in epoch millis. | | $FUNCTION\_RUN\_ID | The function run ID. | | $SCHEDULER\_TIMESTAMP | Datetime when the scheduler set the function run to running, in epoch millis. | | $TRIGGER\_TIMESTAMP | Datetime when the execution plan was triggered, in epoch millis. | | $TRANSACTION\_ID | The transaction ID. | You can define the destination files in the following ways: **File Path** To write by file path where the file extension is included as part of the file path, define the destination as follows: ```python destination=td.S3Destination(["s3://","s3://"], s3_credentials), ``` ``… have the extensions of the file included in the file name as part of the path. **File Format** To write files by file format where format is declared separately and not in the file name, define the destination as follows: ```python destination=td.S3Destination([ "s3://", "s3://", ], s3_credentials, format="parquet"), ``` `""`, `""`… don't have the extension in the file name. The extension to all files is mentioned separately in `format` property. #### credentials A subscriber needs credentials to write files from Amazon S3. Here the value is defined using a variable `s3_credentials`. The variable is an object of class `S3AccessKeyCredentials` with following values. `S3_ACCESS_KEY` is the value of your Amazon S3 access key. `S3_SECRET_KEY` is the value of your Amazon S3 secret key. `path-to-secret` is the name of the default key value store in HashiCorp where the credential values are required to be stored. You can use different ways to store the credentials which are highlighted here in the documentation. #### region `` is where your S3 bucket is located. #### catalog This property is used to define the properties of your catalog. ##### definition This property defines the catalog. It is of dictionary format, and matches the [pyiceberg catalog definition](https://py.iceberg.apache.org/configuration/#catalogs). The value is simply passed through. ##### tables These are the names of the Iceberg tables we want to add the parquet files to. They need to have the same number as the parquet files being created in S3. ##### auto\_create\_at If you don't specify `auto_create_at` the table must exist in the Iceberg catalog. If you want the publisher to create the table if it does not exist, in the `auto_create_at` you have to provide an S3 URI where the entry point of Iceberg metadata of the table will be created. Typically you would put the base path where you write the parquet files. e.g. the path would look like: `s3://`. ##### \[Optional\] if\_table\_exists This is an optional property to define the strategy to follow when the table already exists. `replace` will create a new table in Iceberg, overwriting the existing one, and `append` will append to the existing data in the table. Defaults to `append`. ##### partitioned\_table This option indicates that the Iceberg table definition corresponds to a partitioned table. If the Iceberg table is a partitioned table, the subscriber function should return an array of TableFrames with each of those TableFrames containing data for one partition only. It is the responsibility of the user publisher creator to partition the data accordingly before returning it. Partition tables can only be used with already existing (and partitioned) Iceberg tables. `auto_create_at` is not supported, and when using partition tables the `if_table_exists` strategy must be `append`. ##### schema\_strategy Schema strategy indicates if the schema in iceberg will be modified (augmented) if necessary to be compatible with the parquet files being uploaded. If using `strict`, the function will fail if the schemas differ. #### Important Points to note: - Existing tables in AWS Glue can have their schema updated (augmented) to match the written parquet files. - Tables can be automatically created in AWS Glue using Tabsdata table schema. - The parquet files are registered with AWS Glue as external files for a table. - Future versions will add support for replacing partitions in Iceberg partitioned tables. ### None as an input and output A subscriber may receive and return a `None` value instead of TableFrames. When a subscriber receives a `None` value instead of a TableFrame it means that the requested table dependency version does not exist. When a subscriber returns a `None` value instead of a TableFrame it means there is no new data to write to the external system. This helps in avoiding the creation of multiple copies of the same data. ### \[Optional\] trigger\_by ``, ``… are the names of the tables in the Tabsdata server. A new commit to any of these tables triggers the subscriber. All listed trigger tables must exist in the server before registering the subscriber. Defining trigger tables is optional. If you don't define the `trigger_by` property, the subscriber will be triggered by any of its input tables. If you define the `trigger_by` property, then only those tables listed in the property can automatically trigger the subscriber. For more information, see Working with Triggers. ### `` `` is the name for the subscriber that you are configuring. ### `` `` governs the processing performed by the subscriber. You can specify function logic to be a simple write or to perform additional processing as needed. For more information about the function logic that you can include, see Working with Tables. ``, ``… are the names for the variables that temporarily store source data for processing. ``, ``… are the output from the function that are written to the external system. ### Data Drift Support This section talks about how Tabsdata handles data drift in the output data for this Subscriber connector. The behaviour depends on how `schema_strategy` is configured in the subscriber. When `schema_strategy=update` is configured, the schema gets updated is the new commit from the subscriber has a different schema form the previous versions. When `schema_strategy=strict` is configured, the subscriber does not modify the existing schema. Any schema changes (added, removed, or modified columns) will cause execution to fail. In this mode, the subscriber function must ensure consistent column definitions across executions e.g., by explicitly specifying the columns to export. Also, this applies only if the subscriber function does not contain schema-dependent logic. If such logic exists, changes in the schema may conflict with the function's expectations, potentially leading to execution errors outside of the configuration of `schema_strategy`. --- # Subscribe data to local files > Writes Tabsdata tables to local Parquet files using a subscriber with LocalFileDest, covering paths, input_tables, output handling, and trigger_by. Source: https://docs.tabsdata.com/tutorials/subscribers/local-file-system Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) ## Write tables out to local files One entry in `paths` per input table, in order, so each table lands as its own Parquet object. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you write the hr departments and employees tables out to disk as parquet Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ That is a subscriber, so it goes in a destination collection rather than a source one. The guide and connection template are loaded. Three values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - destination address and credentials - target objects ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ No credentials needed for a local destination, so nothing here is secret. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **exports** created from it in group `destinations`. Connection check passed. Called tabsdata 2 times (ctrl+o to expand) ⏺ The subscriber, registered in `exports`: ```python from tabsdatak.api import subscriber, TableFrameSpec from tabsdatak.conn.localfile import LocalFileDest @subscriber( destination=LocalFileDest( paths=["departments_${EXEC_PLAN_TS}.parquet", "employees_${EXEC_PLAN_TS}.parquet"], ), input_tables=["hr/departments", "hr/employees"], ) def write_files( departments: TableFrameSpec, employees: TableFrameSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return (departments, employees) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ Two objects written under `/data/exports`, one per input table: ``` ┌─────────────────────┬────────────────┬───────┬────────┐ │ object │ from table │ rows │ bytes │ ├─────────────────────┼────────────────┼───────┼────────┤ │ departments.parquet │ hr/departments │ 14 │ 2.1 KB │ ├─────────────────────┼────────────────┼───────┼────────┤ │ employees.parquet │ hr/employees │ 1,204 │ 86 KB │ └─────────────────────┴────────────────┴───────┴────────┘ ``` Input tables are written as `collection/table`, since a subscriber reads from collections other than its own. A subscriber declares no `output_tables`: it returns one frame per destination slot, and returning `None` for a slot writes nothing that run. Without `trigger_by`, the subscriber runs whenever any of its input tables gets a new commit, so an export stays current without a schedule. --- # Subscribe data to MariaDB > Writes Tabsdata input tables to MariaDB with the @subscriber decorator and MySQLDest connector, covering table mapping, if_table_exists="replace", connection YAML, and the mysql-connector-python driver requirement. Source: https://docs.tabsdata.com/tutorials/subscribers/mariadb Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) warning The MariaDB connector needs the `mysql-connector-python` driver, which Tabsdata does not redistribute. See [Installing Third-Party Database Drivers](https://docs.tabsdata.com/server-admin/third-party-database-drivers) to add it to your Tabsdata server. ## Write tables out to MariaDB One entry in `tables` per input table, in order. `if_table_exists="replace"` rebuilds each destination table from the frame, so a changed schema does not fail the run. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you write the sales vendors and items tables out to mariadb Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ That is a subscriber, so it goes in a destination collection rather than a source one. The guide and connection template are loaded. Three values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - destination address and credentials - target tables ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ Credentials found. Reading back what is not secret: uri mysql://maria.internal:3306/warehouse, collection warehouse, tables vendors and items. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **warehouse** created from it in group `destinations`. Connection check passed. Called tabsdata 2 times (ctrl+o to expand) ⏺ The subscriber, registered in `warehouse`: ```python from tabsdatak.api import subscriber, TableFrameSpec from tabsdatak.conn.mysql import MySQLDest @subscriber( destination=MySQLDest( tables=["vendors", "items"], if_table_exists="replace", ), input_tables=["sales/vendors", "sales/items"], ) def write_sales( vendors: TableFrameSpec, items: TableFrameSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return (vendors, items) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ Two tables written in **warehouse**, one per input table: ``` ┌─────────┬───────────────┬───────┐ │ table │ from table │ rows │ ├─────────┼───────────────┼───────┤ │ vendors │ sales/vendors │ 312 │ ├─────────┼───────────────┼───────┤ │ items │ sales/items │ 8,940 │ └─────────┴───────────────┴───────┘ ``` Input tables are written as `collection/table`, since a subscriber reads from collections other than its own. A subscriber declares no `output_tables`: it returns one frame per destination slot, and returning `None` for a slot writes nothing that run. Without `trigger_by`, the subscriber runs whenever any of its input tables gets a new commit, so an export stays current without a schedule. --- # Subscribe data to MongoDB > Covers writing Tabsdata input tables to MongoDB with the @subscriber decorator and MongoDBDest, including collections_with_ids, if_collection_exists, update_existing, and connection setup. Source: https://docs.tabsdata.com/tutorials/subscribers/mongodb Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) ## Write tables out to MongoDB One entry in `collections` per input table, in order. `if_collection_exists` decides whether an existing collection is replaced or added to. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you write the sales vendors and items tables out to mongodb Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ That is a subscriber, so it goes in a destination collection rather than a source one. The guide and connection template are loaded. Three values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - destination address and credentials - target collections ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ Credentials found. Reading back what is not secret: uri mongodb://db.internal:27017/warehouse, collection warehouse, target collections vendors and items. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **warehouse** created from it in group `destinations`. Connection check passed. Called tabsdata 2 times (ctrl+o to expand) ⏺ The subscriber, registered in `warehouse`: ```python from tabsdatak.api import subscriber, TableFrameSpec from tabsdatak.conn.mongodb import MongoDBDest @subscriber( destination=MongoDBDest( collections_with_ids=[ ("sales_db.vendors", None), ("sales_db.items", "item_id"), ], if_collection_exists="replace", update_existing=True, ), input_tables=["sales/vendors", "sales/items"], ) def write_sales( vendors: TableFrameSpec, items: TableFrameSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return (vendors, items) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ Two collections written in **warehouse**, one per input table: ``` ┌────────────┬───────────────┬───────────┐ │ collection │ from table │ documents │ ├────────────┼───────────────┼───────────┤ │ vendors │ sales/vendors │ 312 │ ├────────────┼───────────────┼───────────┤ │ items │ sales/items │ 8,940 │ └────────────┴───────────────┴───────────┘ ``` Input tables are written as `collection/table`, since a subscriber reads from collections other than its own. A subscriber declares no `output_tables`: it returns one frame per destination slot, and returning `None` for a slot writes nothing that run. Without `trigger_by`, the subscriber runs whenever any of its input tables gets a new commit, so an export stays current without a schedule. --- # Subscribe data to SQL Server > Writes Tabsdata tables to SQL Server with the @subscriber decorator and MSSQLDest, covering connection URI and credentials, if_table_exists options, and the collection/table input format. Source: https://docs.tabsdata.com/tutorials/subscribers/mssql Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) ## Write tables out to SQL Server One entry in `tables` per input table, in order. `if_table_exists="replace"` rebuilds each destination table from the frame, so a changed schema does not fail the run. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you write the sales vendors and items tables out to sql server Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ That is a subscriber, so it goes in a destination collection rather than a source one. The guide and connection template are loaded. Three values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - destination address and credentials - target tables ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ Credentials found. Reading back what is not secret: db.internal:1433/warehouse over ODBC Driver 18, collection warehouse, tables vendors and items. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **warehouse** created from it in group `destinations`. Connection check passed. Called tabsdata 2 times (ctrl+o to expand) ⏺ The subscriber, registered in `warehouse`: ```python from tabsdatak.api import subscriber, TableFrameSpec from tabsdatak.conn.mssql import MSSQLDest @subscriber( destination=MSSQLDest( tables=["dbo.vendors", "dbo.items"], if_table_exists="replace", ), input_tables=["sales/vendors", "sales/items"], ) def write_sales( vendors: TableFrameSpec, items: TableFrameSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return (vendors, items) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ Two tables written in **warehouse**, one per input table: ``` ┌─────────┬───────────────┬───────┐ │ table │ from table │ rows │ ├─────────┼───────────────┼───────┤ │ vendors │ sales/vendors │ 312 │ ├─────────┼───────────────┼───────┤ │ items │ sales/items │ 8,940 │ └─────────┴───────────────┴───────┘ ``` Input tables are written as `collection/table`, since a subscriber reads from collections other than its own. A subscriber declares no `output_tables`: it returns one frame per destination slot, and returning `None` for a slot writes nothing that run. Without `trigger_by`, the subscriber runs whenever any of its input tables gets a new commit, so an export stays current without a schedule. --- # Subscribe data to MySQL > Writes Tabsdata table frames to MySQL tables through the MySQL destination connector, covering the connection setup, the `MySQLDest` options such as `if_table_exists`, and the `@subscriber` decorator. Source: https://docs.tabsdata.com/tutorials/subscribers/mysql Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) warning The MySQL connector needs the `mysql-connector-python` driver, which Tabsdata does not redistribute. See [Installing Third-Party Database Drivers](https://docs.tabsdata.com/server-admin/third-party-database-drivers) to add it to your Tabsdata server. ## Write tables out to MySQL One entry in `tables` per input table, in order. `if_table_exists="replace"` rebuilds each destination table from the frame, so a changed schema does not fail the run. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you write the sales vendors and items tables out to mysql Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ That is a subscriber, so it goes in a destination collection rather than a source one. The guide and connection template are loaded. Three values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - destination address and credentials - target tables ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ Credentials found. Reading back what is not secret: uri mysql://db.internal:3306/warehouse, collection warehouse, tables vendors and items. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **warehouse** created from it in group `destinations`. Connection check passed. Called tabsdata 2 times (ctrl+o to expand) ⏺ The subscriber, registered in `warehouse`: ```python from tabsdatak.api import subscriber, TableFrameSpec from tabsdatak.conn.mysql import MySQLDest @subscriber( destination=MySQLDest( tables=["vendors", "items"], if_table_exists="replace", ), input_tables=["sales/vendors", "sales/items"], ) def write_sales( vendors: TableFrameSpec, items: TableFrameSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return (vendors, items) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ Two tables written in **warehouse**, one per input table: ``` ┌─────────┬───────────────┬───────┐ │ table │ from table │ rows │ ├─────────┼───────────────┼───────┤ │ vendors │ sales/vendors │ 312 │ ├─────────┼───────────────┼───────┤ │ items │ sales/items │ 8,940 │ └─────────┴───────────────┴───────┘ ``` Input tables are written as `collection/table`, since a subscriber reads from collections other than its own. A subscriber declares no `output_tables`: it returns one frame per destination slot, and returning `None` for a slot writes nothing that run. Without `trigger_by`, the subscriber runs whenever any of its input tables gets a new commit, so an export stays current without a schedule. --- # Subscribe data to Oracle > Writes Tabsdata input tables to Oracle with the `OracleDest` destination, covering the `oracledb` driver requirement, `if_table_exists` replace behavior, and `@subscriber` setup. Source: https://docs.tabsdata.com/tutorials/subscribers/oracle Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) warning The Oracle connector needs the `oracledb` driver, which Tabsdata does not redistribute. See [Installing Third-Party Database Drivers](https://docs.tabsdata.com/server-admin/third-party-database-drivers) to add it to your Tabsdata server. ## Write tables out to Oracle One entry in `tables` per input table, in order. `if_table_exists="replace"` rebuilds each destination table from the frame, so a changed schema does not fail the run. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you write the sales vendors and items tables out to oracle Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ That is a subscriber, so it goes in a destination collection rather than a source one. The guide and connection template are loaded. Three values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - destination address and credentials - target tables ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ Credentials found. Reading back what is not secret: uri oracle://db.internal:1521/warehouse, collection warehouse, tables vendors and items. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **warehouse** created from it in group `destinations`. Connection check passed. Called tabsdata 2 times (ctrl+o to expand) ⏺ The subscriber, registered in `warehouse`: ```python from tabsdatak.api import subscriber, TableFrameSpec from tabsdatak.conn.oracle import OracleDest @subscriber( destination=OracleDest( tables=["VENDORS", "ITEMS"], if_table_exists="replace", ), input_tables=["sales/vendors", "sales/items"], ) def write_sales( vendors: TableFrameSpec, items: TableFrameSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return (vendors, items) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ Two tables written in **warehouse**, one per input table: ``` ┌─────────┬───────────────┬───────┐ │ table │ from table │ rows │ ├─────────┼───────────────┼───────┤ │ vendors │ sales/vendors │ 312 │ ├─────────┼───────────────┼───────┤ │ items │ sales/items │ 8,940 │ └─────────┴───────────────┴───────┘ ``` Input tables are written as `collection/table`, since a subscriber reads from collections other than its own. A subscriber declares no `output_tables`: it returns one frame per destination slot, and returning `None` for a slot writes nothing that run. Without `trigger_by`, the subscriber runs whenever any of its input tables gets a new commit, so an export stays current without a schedule. --- # Subscribe data to Postgres > Documents how to write Tabsdata input tables out to a PostgreSQL destination with the `@subscriber` decorator and PostgresDest, including if_table_exists options and the psycopg2-binary driver requirement. Source: https://docs.tabsdata.com/tutorials/subscribers/postgres Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) warning The PostgreSQL connector needs the `psycopg2-binary` driver, which Tabsdata does not redistribute. See [Installing Third-Party Database Drivers](https://docs.tabsdata.com/server-admin/third-party-database-drivers) to add it to your Tabsdata server. ## Write tables out to Postgres One entry in `tables` per input table, in order. `if_table_exists="replace"` rebuilds each destination table from the frame, so a changed schema does not fail the run. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you write the sales vendors and items tables out to postgres Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ That is a subscriber, so it goes in a destination collection rather than a source one. The guide and connection template are loaded. Three values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - destination address and credentials - target tables ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ Credentials found. Reading back what is not secret: uri postgres://db.internal:5432/warehouse, collection warehouse, tables vendors and items. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **warehouse** created from it in group `destinations`. Connection check passed. Called tabsdata 2 times (ctrl+o to expand) ⏺ The subscriber, registered in `warehouse`: ```python from tabsdatak.api import subscriber, TableFrameSpec from tabsdatak.conn.postgres import PostgresDest @subscriber( destination=PostgresDest( tables=["vendors", "items"], if_table_exists="replace", ), input_tables=["sales/vendors", "sales/items"], ) def write_sales( vendors: TableFrameSpec, items: TableFrameSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return (vendors, items) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ Two tables written in **warehouse**, one per input table: ``` ┌─────────┬───────────────┬───────┐ │ table │ from table │ rows │ ├─────────┼───────────────┼───────┤ │ vendors │ sales/vendors │ 312 │ ├─────────┼───────────────┼───────┤ │ items │ sales/items │ 8,940 │ └─────────┴───────────────┴───────┘ ``` Input tables are written as `collection/table`, since a subscriber reads from collections other than its own. A subscriber declares no `output_tables`: it returns one frame per destination slot, and returning `None` for a slot writes nothing that run. Without `trigger_by`, the subscriber runs whenever any of its input tables gets a new commit, so an export stays current without a schedule. --- # Subscribe data to Snowflake > Writes Tabsdata tables out to Snowflake using a subscriber with SnowflakeDest, covering connection YAML, the tables and if_table_exists options, and secrets handling. Source: https://docs.tabsdata.com/tutorials/subscribers/snowflake Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) ## Write tables out to Snowflake One entry in `tables` per input table, in order. `if_table_exists="replace"` rebuilds each destination table from the frame, so a changed schema does not fail the run. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you write the sales vendors and items tables out to snowflake Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ That is a subscriber, so it goes in a destination collection rather than a source one. The guide and connection template are loaded. Three values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - destination address and credentials - target tables ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ Credentials found. Reading back what is not secret: account acme-prod, warehouse LOAD\_WH, database ANALYTICS, collection warehouse. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **warehouse** created from it in group `destinations`. Connection check passed. Called tabsdata 2 times (ctrl+o to expand) ⏺ The subscriber, registered in `warehouse`: ```python from tabsdatak.api import subscriber, TableFrameSpec from tabsdatak.conn.snowflake import SnowflakeDest @subscriber( destination=SnowflakeDest( tables=["vendors", "items"], if_table_exists="append", ), input_tables=["sales/vendors", "sales/items"], ) def write_sales( vendors: TableFrameSpec, items: TableFrameSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return (vendors, items) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ Two tables written in **warehouse**, one per input table: ``` ┌─────────┬───────────────┬───────┐ │ table │ from table │ rows │ ├─────────┼───────────────┼───────┤ │ vendors │ sales/vendors │ 312 │ ├─────────┼───────────────┼───────┤ │ items │ sales/items │ 8,940 │ └─────────┴───────────────┴───────┘ ``` Input tables are written as `collection/table`, since a subscriber reads from collections other than its own. A subscriber declares no `output_tables`: it returns one frame per destination slot, and returning `None` for a slot writes nothing that run. Without `trigger_by`, the subscriber runs whenever any of its input tables gets a new commit, so an export stays current without a schedule. ## Merge changes into the destination To apply inserts, updates, and deletes by key, see [Merging into SQL destinations](https://docs.tabsdata.com/guide/functions/sql-merge). The guide covers merge configuration, schema evolution, and retry watermarks. --- # Subscribe data to StarRocks > Writes Tabsdata tables to StarRocks with the StarRocksDest connector, covering the subscriber decorator, if_table_exists and schema_evolution options, S3 staging, and tabsdata[starrocks] install. Source: https://docs.tabsdata.com/tutorials/subscribers/starrocks Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) ## Write tables out to StarRocks One entry in `tables` per input table, in order. `if_table_exists="replace"` rebuilds each destination table from the frame, so a changed schema does not fail the run. warning There are several ways to pass secrets into Tabsdata. See [Secrets Management](https://docs.tabsdata.com/guide/secrets-management/) for more info if you are planning to have an LLM register secrets into Tabsdata's credential store. AI agent session ❯Can you write the sales vendors and items tables out to starrocks Listed 1 directory, called tabsdata 3 times (ctrl+o to expand) ⏺ That is a subscriber, so it goes in a destination collection rather than a source one. The guide and connection template are loaded. Three values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - destination address and credentials - target tables ❯everything you need is in Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ Both credential sets found, the database one and the staging one. Reading back what is not secret: starrocks.internal:9030, database analytics, staging in acme-staging, collection analytics. Called tabsdata 4 times (ctrl+o to expand) ⏺ Connection written to , and collection **analytics** created from it in group `destinations`. Connection check passed. Called tabsdata 2 times (ctrl+o to expand) ⏺ The subscriber, registered in `analytics`: ```python from tabsdatak.api import subscriber, TableFrameSpec from tabsdatak.conn.starrocks import StarRocksDest @subscriber( destination=StarRocksDest( tables=["vendors", "items"], if_table_exists="append", schema_evolution="strict", ), input_tables=["sales/vendors", "sales/items"], ) def write_sales( vendors: TableFrameSpec, items: TableFrameSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return (vendors, items) ``` Called tabsdata 6 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. The frames were staged as Parquet in acme-staging, then loaded. ⏺ Two tables written in **analytics**, one per input table: ``` ┌─────────┬───────────────┬───────┐ │ table │ from table │ rows │ ├─────────┼───────────────┼───────┤ │ vendors │ sales/vendors │ 312 │ ├─────────┼───────────────┼───────┤ │ items │ sales/items │ 8,940 │ └─────────┴───────────────┴───────┘ ``` Input tables are written as `collection/table`, since a subscriber reads from collections other than its own. A subscriber declares no `output_tables`: it returns one frame per destination slot, and returning `None` for a slot writes nothing that run. Without `trigger_by`, the subscriber runs whenever any of its input tables gets a new commit, so an export stays current without a schedule. The connector ships separately from the core package, so `pip install 'tabsdata[starrocks]'` has to have run in the environment the server uses. --- # Register tables in Unity Catalog > Tabsdata subscriber that writes Tabsdata tables to Unity Catalog managed Delta tables in Databricks via DatabricksDestination, covering its host, token, volume, warehouse, and schema options. Source: https://docs.tabsdata.com/tutorials/catalogs/unity-catalog You can use this built-in Tabsdata subscriber to write to Unity-Catalog managed delta tables in Databricks. ## Installing the connector To work with the Databricks connector you are required to install the dependencies separately. This is to manage the size of the core package. Please run the following command in your terminal to install the dependencies. ```bash $ pip install 'tabsdata[databricks]' ``` ## Example (Subscriber - Databricks) The following example creates a subscriber named `write_sales`. It writes two Tabsdata tables to the database. The subscriber is automatically triggered by a new commit to any of its input tables, and writes data to the database without any modification. ```python import tabsdata as td @td.subscriber( tables=["vendors", "items"], destination=td.DatabricksDestination( host_url = "https://adb-1234567890123456.7.azuredatabricks.net", token = "dapi1234567890abcdef1234567890abcd", tables = ["vendors", "items"], volume = "main_volume", catalog = "hive_metastore", schema = "default", warehouse = "my_sql_warehouse", warehouse_id = null, if_table_exists = "append", schema_strategy = "update" ) ) def write_sales(tf1: td.TableFrame, tf2: td.TableFrame): return tf1, tf2 ``` note - After defining the subscriber, you need to register it with a Tabsdata collection. For more information, see Register a Function. ## Setup (Subscriber - Databricks) The following code uses placeholder values for defining a subscriber that reads Tabsdata tables and writes to Databricks and Unity Catalog: ```python import tabsdata as td @td.subscriber( tables=["", ""], destination=td.DatabricksDestination( host_url: str, # Databricks URL token: str|Secret, # Databricks Personal Account Token (PAT) tables: list[str]|str, # Tables to be created in Databricks volume: str, # Name of the Databricks volume catalog: str|None, # Catalog name (optional, can be null if the table name has the catalog name.) schema: str|None, # Schema name (optional, can be null if the table name has the schema name.) warehouse: str|None, # SQL warehouse name (use either this or warehouse_id) warehouse_id: str|None, # Use null if warehouse name is provided if_table_exists: str = "append" # append or replace schema_strategy: str = "update" # update or strict ), trigger_by=["", ""], ) def (:td.TableFrame, :td.TableFrame): return , ``` note - After defining the subscriber, you need to register it with a Tabsdata collection. For more information, see Register a Function. Following properties are defined in the code above: ### tables ``, ``… are the names of the Tabsdata tables to be written to the external system. ### destination - **host\_url**: Databricks URL - **token**: Databricks Personal Account Token (PAT). The user owning the PAT requires ‘USE CATALOG’, ‘USE SCHEMA’, ‘READ VOLUME’ & ‘WRITE VOLUME’ privileges. - **tables**: Tables to be created in Databricks. Each table name provided must either be fully qualified in the form of ‘catalog.schema.table\_name’ or have the information provided through the ‘catalog’ or the ‘schema’ parameters. For example, if a table of the form ‘schema.table\_name’ is provided, the ‘catalog’ parameter must also be provided. If a table of the form ‘table\_name’ is provided, both ‘catalog’ and ‘schema’ must be provided. - **volume**: Name of the Databricks volume. The connector uses an existing Databricks managed volume. The files written to the volume are deleted after connector finishes writing the data to the Databricks table. - **catalog**: Catalog name (optional, can be null if the table name has the catalog name.) - **schema**: Schema name (optional, can be null if the table name has the schema name.) - **warehouse**: Warehouse name (use either this or warehouse\_id) that will be used for uploading data to Databricks tables. The user has to have privileges to use the warehouse. - **warehouse\_id**: Warehouse Id. Use null if warehouse name is provided. - \[optional\] **if\_table\_exists**: This is an optional property to define the strategy to follow when the table already exists. ‘replace’ will create overwrite the existing table, and ‘append’ will append to the existing data in the table. Defaults to ‘append’. - **schema\_strategy**: The is to mention the strategy to follow when appending to a table with an existing schema. ‘update’ will update the schema with the possible new columns that might exist in the TableFrame. ‘strict’ will not modify the schema, and will fail if there is any difference. ### None as an input and output A subscriber may receive and return a `None` value instead of TableFrames. When a subscriber receives a `None` value instead of a TableFrame it means that the requested table dependency version does not exist. When a subscriber returns a `None` value instead of a TableFrame it means there is no new data to write to the external system. This helps in avoiding the creation of multiple copies of the same data. ### \[Optional\] trigger\_by ``, ``… are the names of the tables in the Tabsdata server. A new commit to any of these tables triggers the subscriber. All listed trigger tables must exist in the server before registering the subscriber. Defining trigger tables is optional. If you don’t define the `trigger_by` property, the subscriber will be triggered by any of its input tables. If you define the `trigger_by` property, then only those tables listed in the property can automatically trigger the subscriber. For more information, see Working with Triggers. ### `` `` is the name for the subscriber that you are configuring. ### `` `` governs the processing performed by the subscriber. You can specify function logic to be a simple write or to perform additional processing as needed. For more information about the function logic that you can include, see Working with Tables. ``, ``… are the names for the variables that temporarily store source data for processing. ``, ``… are the output from the function that are written to the external system. --- # Manually Trigger a Function > Explains how to trigger a Tabsdata function or publisher manually to run it on demand, including scheduled functions, with the default no-trigger behavior. Source: https://docs.tabsdata.com/tutorials/trigger-setup/manual Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) A manual trigger runs a function on demand. Any function can be triggered this way, including one that already carries a schedule, which makes it the way to test a function before trusting it to a cron mask. A publisher with no trigger configured runs only when triggered manually. That is the default for publishers, since nothing inside Tabsdata can signal that an external system has new data. AI agent session ❯run the read\_orders publisher in the sales collection Called tabsdata 2 times (ctrl+o to expand) ⏺ Triggering **read\_orders**. The plan covers three functions, since `summarize` reads `orders` and `write_report` reads `orders_by_region`. Called tabsdata 4 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ All three ran in dependency order: ``` ┌──────────────┬─────────────┬───────────┐ │ function │ kind │ outcome │ ├──────────────┼─────────────┼───────────┤ │ read_orders │ publisher │ committed │ ├──────────────┼─────────────┼───────────┤ │ summarize │ transformer │ committed │ ├──────────────┼─────────────┼───────────┤ │ write_report │ subscriber │ committed │ └──────────────┴─────────────┴───────────┘ ``` ⏺ Nothing commits unless every function in the plan succeeds, so a failure in `summarize` would have left `orders` at the version it held beforehand. --- # Schedule a Cron Trigger > Schedules a Tabsdata publisher to run at fixed times with a CronTrigger crontab mask read in UTC, for sources with no upstream change signal. Source: https://docs.tabsdata.com/tutorials/trigger-setup/cron Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) A scheduled trigger runs a function at fixed times whether or not any upstream data changed. It is the right trigger for a publisher, which has nothing inside Tabsdata to signal that its external system has new rows. Tabsdata reads every crontab schedule in **UTC**. AI agent session ❯make read\_orders in the sales collection run at the top of every hour ⏺ Needs a schedule on the publisher. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — a `CronTrigger` with the mask `0 * * * *`, which fires at minute 0 of every hour. Tabsdata reads the mask in UTC. Called tabsdata 3 times (ctrl+o to expand) ⏺ The publisher, registered in sales: ```python @publisher( source=MySQLSrc( queries=["SELECT * FROM orders"], ), output_tables=["orders"], trigger_by=CronTrigger("0 * * * *"), ) def read_orders(orders: TableFrameSpec) -> TableFrameSpec: return orders ``` Called tabsdata 2 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ sales/orders is live — the schedule starts now and fires at the next hour boundary: ``` ┌──────────┬─────────────────┐ │ field │ value │ ├──────────┼─────────────────┤ │ function │ read_orders │ ├──────────┼─────────────────┤ │ mask │ 0 * * * * │ ├──────────┼─────────────────┤ │ timezone │ UTC │ ├──────────┼─────────────────┤ │ enabled │ true │ ├──────────┼─────────────────┤ │ next run │ top of the hour │ └──────────┴─────────────────┘ ``` --- # Trigger a Function from a Table Commit > Controls which input tables fire a Tabsdata function on commit using the trigger_by parameter, including an empty list to disable automatic triggering. Source: https://docs.tabsdata.com/tutorials/trigger-setup/table-commit Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) By default, functions trigger when any of their input tables receives new data. The `trigger_by` parameter lets users specify a subset of input tables that should trigger the function instead, or disable automatic triggering entirely. AI agent session ❯make a transformer that reads the orders and regions tables and totals orders by region, but set it to only trigger on commits to the orders table ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — trigger\_by narrows the default trigger set to orders alone. regions is still read on every run and still supplies its current data, it just no longer starts one. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in sales: ```python @transformer( input_tables=["orders", "regions"], output_tables=["orders_by_region"], trigger_by=["orders"], ) def summarize(orders: TableFrameSpec, regions: TableFrameSpec) -> TableFrameSpec: return orders.group_by("region").agg(col("amount").sum()) ``` ⏺ This is not about avoiding duplicate runs. A plan that commits both tables runs the transformer once either way, since the plan covers every downstream function and orders them by dependency rather than firing one run per committed table. AI agent session ❯make a transformer that reads the orders and regions tables and totals orders by region, and only trigger it off commits to the purchases table ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — trigger\_by names purchases alone. input\_tables and trigger\_by are independent lists: orders and regions are both read but silent, and purchases starts a run despite never being read. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in sales: ```python @transformer( input_tables=["orders", "regions"], output_tables=["orders_by_region"], trigger_by=["purchases"], ) def summarize(orders: TableFrameSpec, regions: TableFrameSpec) -> TableFrameSpec: return orders.group_by("region").agg(col("amount").sum()) ``` ⏺ A publisher uses the same mechanism. It has no input tables at all, so naming a table in trigger\_by is the only way a Tabsdata commit can start it. ⏺ `collection/table` references work within one project. A reference to a table in another project reports `Dependency table '' does not exist`, because row-level security hides it rather than reporting it as forbidden. AI agent session ❯make a transformer that reads the orders and regions tables and totals orders by region, but don't trigger it automatically at all — I'll run it manually ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — an empty trigger\_by. Orders and regions are both still read as inputs, and neither commit starts a run any more. Called tabsdata 3 times (ctrl+o to expand) ⏺ The transformer, registered in sales: ```python @transformer( input_tables=["orders", "regions"], output_tables=["orders_by_region"], trigger_by=[], ) def summarize(orders: TableFrameSpec, regions: TableFrameSpec) -> TableFrameSpec: return orders.group_by("region").agg(col("amount").sum()) ``` ⏺ This runs only when triggered directly, with `tdk fn trigger`. --- # Trigger off CDC in an SQL Database > Covers CDC stream sources for MySQL, PostgreSQL, and IBM Db2 (`MySQLCdcStreamSrc`, `PostgresCdcStreamSrc`, `Db2CdcStreamSrc`) with `start_from`, replication slots, and batch timing for `@stream_publisher`. Source: https://docs.tabsdata.com/tutorials/trigger-setup/cdc Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) A CDC stream source reads a database's replication log and emits one Tabsdata table per tracked table. Every insert, update, and delete arrives, rather than whatever a `SELECT` happens to see when it runs. Three databases have a CDC source, each with its own connection type: | Connection type | Database | Class | | --- | --- | --- | | `mysql-cdc` | MySQL | `MySQLCdcStreamSrc` | | `postgres-cdc` | PostgreSQL | `PostgresCdcStreamSrc` | | `db2-cdc` | IBM Db2 | `Db2CdcStreamSrc` | What runs the publisher is a **batch** rather than a clock or an upstream commit. The stream worker reads the log continuously, and when a batch is due, by the `batch_*` parameters of `@stream_publisher`, it hands the changes captured since the previous batch to the publisher. The database URI and its credentials live on the collection's connection, not in the function, so the collection has to exist with a CDC connection bound to it before a stream publisher can be registered there. AI agent session ❯stream changes from the shop.products and shop.stock tables in mysql ⏺ Needs a stream publisher, and a source collection with a `mysql-cdc` connection to hold it. Four values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - database URI and credentials - tables to track - where in the log to start ❯see Read 1 file, called tabsdata 2 times (ctrl+o to expand) ⏺ Password found. Reading back what is not secret: mysql://db.internal:3306/shop, tracking shop.products and shop.stock, starting at the tail, collection inventory. Called tabsdata 3 times (ctrl+o to expand) ⏺ Collection **inventory** created in group `sources`, with a `MySQLCdcStreamSrcConn` to db.internal:3306. Connection check passed and binlog access confirmed. ⏺ Generating — `MySQLCdcStreamSrc` tracking both tables. A CDC source emits one output table per tracked table, so this one has two. Called tabsdata 3 times (ctrl+o to expand) ⏺ The stream publisher, registered in inventory: ```python @stream_publisher( source=MySQLCdcStreamSrc( tables=["shop.products", "shop.stock"], start_from="tail", ), batch_max_secs=300, output_tables=["products", "stock"], ) def read_changes( products: TableFramesSpec, stock: TableFramesSpec, ) -> tuple[TableFrameSpec, TableFrameSpec]: return concat(products), concat(stock) ``` Called tabsdata 2 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ inventory/products and inventory/stock are live — the worker is reading the binlog from the current position: ``` ┌───────────────┬──────────────┬───────┐ │ tracked table │ output table │ start │ ├───────────────┼──────────────┼───────┤ │ shop.products │ products │ tail │ ├───────────────┼──────────────┼───────┤ │ shop.stock │ stock │ tail │ └───────────────┴──────────────┴───────┘ ``` One output table per tracked table, in order, and one input slot per output table. The function declares no transaction context, since it runs inside the runtime's stream loop rather than per-transaction. ## Where the stream starts `start_from` decides what the worker reads first. `"full"` starts with a full load of the tracked tables and then switches to the changes. For PostgreSQL, `"head"` starts at the oldest retained WAL, `"tail"` starts at the current LSN, and an `LsnPosition` starts at a position you name. Starting at the tail skips the history and streams only what arrives from now on. ## PostgreSQL needs a replication slot PostgreSQL logical replication is slot-based, so its source takes fields the other two do not. AI agent session ❯set the same thing up against postgres instead Called tabsdata 3 times (ctrl+o to expand) ⏺ Switched to `PostgresCdcStreamSrc`, which needs a logical replication slot. The stream publisher now sets `replication_slot`, and `replication_slot_behavior` defaults to reusing an existing slot rather than creating one. ``` ┌───────────────────────────┬──────────────┐ │ field │ value │ ├───────────────────────────┼──────────────┤ │ replication_slot │ tabsdata_cdc │ ├───────────────────────────┼──────────────┤ │ replication_slot_behavior │ reuse │ ├───────────────────────────┼──────────────┤ │ columns_format │ columns │ ├───────────────────────────┼──────────────┤ │ flatten_columns │ true │ └───────────────────────────┴──────────────┘ ``` ⏺ `publication_name` is optional. `columns_format` also accepts `struct` and `map` if you would rather keep the change payload nested. ## Batch timing `batch_max_secs=300` hands the changes to the publisher at least every 5 minutes while they arrive. `batch_max_rows` or `batch_max_bytes` make a batch due sooner once that many rows or bytes are waiting, but never sooner than `batch_min_secs`, which is at least 60 seconds. See [Batch parameters](https://docs.tabsdata.com/guide/functions/stream-publishers#batch-parameters). ## Drivers warning MySQL CDC needs the `mysql-connector-python` driver and PostgreSQL CDC needs the `psycopg2-binary` driver, which Tabsdata does not redistribute. Add the driver with `tdkserver venv update`, as described in [Installing Third-Party Database Drivers](https://docs.tabsdata.com/server-admin/third-party-database-drivers). ## What to do next [Trigger off a Kafka Stream](https://docs.tabsdata.com/tutorials/trigger-setup/kafka) covers the fourth stream source, which uses the same batch parameters against a topic. See [Stream Triggers](https://docs.tabsdata.com/guide/triggers/stream) for how a batch becomes a run, and [MySQL CDC](https://docs.tabsdata.com/guide/connectors/mysql-cdc) for every option of the source. --- # Trigger off a Kafka Stream > Covers triggering a stream publisher from a Kafka topic with KafkaStreamSrc: the kafka-topic-in connection, required servers and group_id, schema, batch_min/max_secs/rows/bytes, and dlq_topic. Source: https://docs.tabsdata.com/tutorials/trigger-setup/kafka Before you begin [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) or [Start a Server](https://docs.tabsdata.com/server-admin/start) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) A Kafka stream source consumes one topic continuously and writes it into one Tabsdata table, named in `output_tables`. What runs the publisher is neither a clock nor an upstream commit but a **batch**: the stream worker reads messages as they arrive, and when a batch is due, by the `batch_*` parameters of `@stream_publisher`, it hands the messages read since the previous batch to the publisher. A stream publisher therefore takes no trigger at all. The brokers and the consumer group live on the collection's connection, not in the function, so the collection has to exist with a `kafka-topic-in` connection bound to it before a stream publisher can be registered there. `servers` and `group_id` are required. A broker that takes SASL needs credentials alongside them, and one that does not takes none. AI agent session ❯read the clicks topic from kafka into a table ⏺ Needs a stream publisher, and a source collection with a Kafka connection to hold it. Five values left, then I generate and register. Reply with them one per line, or point me at a file that has them: - collection name - bootstrap servers - consumer group - topic - message format and schema ❯it's all in Read 2 files, called tabsdata 2 times (ctrl+o to expand) ⏺ Schema read from ~/clicks.json. No credentials in the file, so the connection goes out unauthenticated: `credentials` is optional on a Kafka connection, and this broker takes no SASL. Reading back: broker.internal:9092, group tabsdata\_clicks, topic clicks, format json, collection events. Called tabsdata 3 times (ctrl+o to expand) ⏺ Collection **events** created in group `sources`, with a `KafkaStreamSrcConn` to broker.internal:9092. ⏺ Generating — `KafkaStreamSrc` on the `clicks` topic. A Kafka source has a single slot, so there is one output table, and `concat` folds the batch's frames into it. Called tabsdata 3 times (ctrl+o to expand) ⏺ The stream publisher, registered in events: ```python @stream_publisher( source=KafkaStreamSrc( topic="clicks", data_format="json", schema=CLICK_SCHEMA, ), batch_max_secs=300, output_tables=["clicks"], ) def read_clicks(clicks: TableFramesSpec) -> TableFrameSpec: return concat(clicks, how="diagonal_relaxed") ``` Called tabsdata 2 times (ctrl+o to expand) ⏺ Execution committed, 0 failures. ⏺ events/clicks is live — the worker is consuming, and the first batch lands within 5 minutes: ``` ┌────────────────┬───────────┐ │ field │ value │ ├────────────────┼───────────┤ │ topic │ clicks │ ├────────────────┼───────────┤ │ format │ json │ ├────────────────┼───────────┤ │ output table │ clicks │ ├────────────────┼───────────┤ │ batch_max_secs │ 300 │ ├────────────────┼───────────┤ │ worker │ consuming │ └────────────────┴───────────┘ ``` `schema` is required. Without a schema registry, every JSON message is validated against it. The function takes one input slot per source, always in array mode, and declares no transaction context: it runs inside the runtime's stream loop rather than per-transaction. ## When a batch is due Once `batch_min_secs` have passed since the previous batch (60 seconds, and never less), a batch is due when `batch_max_secs` have passed, or when at least `batch_max_rows` messages or `batch_max_bytes` bytes are waiting. At least one of the three is required. AI agent session ❯hand over a batch as soon as 10,000 clicks are waiting Called tabsdata 3 times (ctrl+o to expand) ⏺ Added `batch_max_rows=10_000`. A batch is now due once 10,000 messages are waiting, or after 5 minutes, whichever comes first, and never sooner than a minute after the previous one. ```python batch_max_secs=300, batch_max_rows=10_000, ``` See [Batch parameters](https://docs.tabsdata.com/guide/functions/stream-publishers#batch-parameters) for every rule. ## Messages that fail to deserialize Set a dead letter topic on the source with `dlq_topic`, and messages that cannot be decoded are written there while the stream carries on. Without one, such a message stops the stream. ## What to do next [Trigger off CDC in an SQL Database](https://docs.tabsdata.com/tutorials/trigger-setup/cdc) covers the other three stream sources, which use the same batch parameters against a database log. See [Stream Triggers](https://docs.tabsdata.com/guide/triggers/stream) for how a batch becomes a run, and [Kafka](https://docs.tabsdata.com/guide/connectors/kafka) for every option of the source and the connection. --- # Create a Project > Shows how to create a Tabsdata project from an agent session via MCP, covering the project name, description, ID, and the active-project setting. Source: https://docs.tabsdata.com/tutorials/managing-projects/create Before you begin [Install Tabsdata](https://docs.tabsdata.com/server-admin/install) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) AI agent session ❯create a project called finance for subscription billing Called tabsdata 2 times (ctrl+o to expand) ⏺ **finance** is created and set as the active project. ``` ┌─────────────┬────────────────────────────┐ │ field │ value │ ├─────────────┼────────────────────────────┤ │ name │ finance │ ├─────────────┼────────────────────────────┤ │ description │ Subscription billing │ ├─────────────┼────────────────────────────┤ │ id │ 06fvco21l5q73d44ecrdii68k0 │ ├─────────────┼────────────────────────────┤ │ contents │ empty — 0 collections │ └─────────────┴────────────────────────────┘ ``` --- # Update a Project > Updates a Tabsdata project's name and description through an MCP agent session, noting that the project id and its groups and collections are preserved on rename. Source: https://docs.tabsdata.com/tutorials/managing-projects/update Before you begin [Install Tabsdata](https://docs.tabsdata.com/server-admin/install) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) [Create a Project](https://docs.tabsdata.com/tutorials/managing-projects/create) AI agent session ❯rename the finance project to billing and describe it as subscription billing and invoicing Called tabsdata 2 times (ctrl+o to expand) ⏺ Renamed. The groups and collections inside came along untouched, and the id is unchanged, so anything holding the id still resolves. ``` ┌───────────────┬────────────────────────────────────┐ │ field │ value │ ├───────────────┼────────────────────────────────────┤ │ name │ billing │ ├───────────────┼────────────────────────────────────┤ │ previous name │ finance │ ├───────────────┼────────────────────────────────────┤ │ description │ Subscription billing and invoicing │ ├───────────────┼────────────────────────────────────┤ │ id │ 06fvco21l5q73d44ecrdii68k0 │ └───────────────┴────────────────────────────────────┘ ``` ⏺ One thing to know: the old name is free again the moment the rename commits, so a second project could take `finance` straight away. --- # Delete a Project > Shows how to delete a Tabsdata project through the MCP agent, including the confirmation step and the removal of its groups and collections, with prerequisites for installing and creating projects. Source: https://docs.tabsdata.com/tutorials/managing-projects/delete Before you begin [Install Tabsdata](https://docs.tabsdata.com/server-admin/install) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) [Create a Project](https://docs.tabsdata.com/tutorials/managing-projects/create) AI agent session ❯delete the billing project Called tabsdata 1 time (ctrl+o to expand) ⏺ This project holds 3 groups and 2 collections. All of them are deleted with it. Confirm by writing `delete`. ❯delete Called tabsdata 2 times (ctrl+o to expand) ⏺ **billing** is gone, and the name is free for reuse. ``` ┌─────────────────────┬───────────┐ │ field │ value │ ├─────────────────────┼───────────┤ │ deleted │ billing │ ├─────────────────────┼───────────┤ │ groups removed │ 3 │ ├─────────────────────┼───────────┤ │ collections removed │ 2 │ ├─────────────────────┼───────────┤ │ name now │ available │ └─────────────────────┴───────────┘ ``` --- # Create a Group > Shows how to create a group in a Tabsdata project via the MCP agent, covering the name and description fields, the data product collection type, and the seeded sources and destinations groups. Source: https://docs.tabsdata.com/tutorials/managing-groups/create Before you begin [Install Tabsdata](https://docs.tabsdata.com/server-admin/install) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) [Create a Project](https://docs.tabsdata.com/tutorials/managing-projects/create) AI agent session ❯create a group called bronze for the raw ingest tables Called tabsdata 2 times (ctrl+o to expand) ⏺ **bronze** is created in the finance project, alongside the three groups that came with it. ``` ┌─────────────────┬───────────────────┐ │ field │ value │ ├─────────────────┼───────────────────┤ │ name │ bronze │ ├─────────────────┼───────────────────┤ │ description │ Raw ingest tables │ ├─────────────────┼───────────────────┤ │ project │ finance │ ├─────────────────┼───────────────────┤ │ collection type │ data product │ ├─────────────────┼───────────────────┤ │ collections │ 0 │ └─────────────────┴───────────────────┘ ``` ⏺ A collection created in `bronze` is a data product collection, the same as one created in `default`: it holds transformers and carries no connection. Only the two seeded groups `sources` and `destinations` produce connection-backed collections. --- # Update a Group > Updates an existing Tabsdata group's name and description, with notes on which system groups (default, sources, destinations) cannot be renamed. Source: https://docs.tabsdata.com/tutorials/managing-groups/update Before you begin [Install Tabsdata](https://docs.tabsdata.com/server-admin/install) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) [Create a Project](https://docs.tabsdata.com/tutorials/managing-projects/create) [Create a Group](https://docs.tabsdata.com/tutorials/managing-groups/create) AI agent session ❯rename the bronze group to raw and describe it as landing tables, no transforms Called tabsdata 2 times (ctrl+o to expand) ⏺ Renamed. The collections in it kept their type, since type is fixed at creation and a rename does not touch it. ``` ┌───────────────┬───────────────────────────────┐ │ field │ value │ ├───────────────┼───────────────────────────────┤ │ name │ raw │ ├───────────────┼───────────────────────────────┤ │ previous name │ bronze │ ├───────────────┼───────────────────────────────┤ │ description │ Landing tables, no transforms │ ├───────────────┼───────────────────────────────┤ │ project │ finance │ ├───────────────┼───────────────────────────────┤ │ collections │ 2 │ └───────────────┴───────────────────────────────┘ ``` ⏺ Worth knowing: `default`, `sources` and `destinations` are system groups and will refuse this. Renaming one is not something the server allows. --- # Delete a Group > Explains how to delete a Tabsdata group, including that deleting it also removes its collections and their contents, and that dependent functions in other collections are frozen. Source: https://docs.tabsdata.com/tutorials/managing-groups/delete Before you begin [Install Tabsdata](https://docs.tabsdata.com/server-admin/install) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) [Create a Project](https://docs.tabsdata.com/tutorials/managing-projects/create) [Create a Group](https://docs.tabsdata.com/tutorials/managing-groups/create) AI agent session ❯delete the raw group Called tabsdata 1 time (ctrl+o to expand) ⏺ This group holds 2 collections. Deleting it deletes them and everything they contain. Confirm by writing `delete`. ❯delete Called tabsdata 2 times (ctrl+o to expand) ⏺ **raw** is gone, along with its 2 collections. ``` ┌─────────────────────┬───────────┐ │ field │ value │ ├─────────────────────┼───────────┤ │ deleted │ raw │ ├─────────────────────┼───────────┤ │ project │ finance │ ├─────────────────────┼───────────┤ │ collections removed │ 2 │ ├─────────────────────┼───────────┤ │ name now │ available │ └─────────────────────┴───────────┘ ``` ⏺ Functions in other collections that read tables from those collections are now frozen rather than deleted: they stay registered and stop running until you re-register them against tables that exist. --- # Create a Collection > Explains how to create a Tabsdata collection through the MCP agent, covering collection types (data product vs. source or destination), the default group, connection requirements, and storage path. Wait, that's over 160 characters and too vague. Here is the corrected sentence: Creates a Tabsdata collection through the agent, covering data product versus source or destination types, group and connection rules, and the storage path field. Source: https://docs.tabsdata.com/tutorials/managing-collections/create Before you begin [Install Tabsdata](https://docs.tabsdata.com/server-admin/install) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) [Create a Project](https://docs.tabsdata.com/tutorials/managing-projects/create) AI agent session ❯create a collection called subscriptions for the billing transformers Called tabsdata 2 times (ctrl+o to expand) ⏺ **subscriptions** is created in the `default` group, which makes it a data product collection: it holds transformers and carries no connection. ``` ┌──────────────┬──────────────────────┐ │ field │ value │ ├──────────────┼──────────────────────┤ │ name │ subscriptions │ ├──────────────┼──────────────────────┤ │ description │ Billing transformers │ ├──────────────┼──────────────────────┤ │ project │ finance │ ├──────────────┼──────────────────────┤ │ group │ default │ ├──────────────┼──────────────────────┤ │ type │ data product │ ├──────────────┼──────────────────────┤ │ connection │ none │ ├──────────────┼──────────────────────┤ │ storage path │ / │ └──────────────┴──────────────────────┘ ``` ⏺ Ask for a source or destination collection instead and I would need a connection document with it, because those are created with their connection rather than given one later. The group you name and the connection you pass have to agree, and a mismatch is an error rather than a silent choice. --- # Update a Collection > Explains how to rename a Tabsdata collection and update its description using an agent session, noting that a rename does not change the group and that a source or destination collection's connection is renamed with it. Wait, that's over 160 characters and too long. Corrected: Renames a Tabsdata collection and updates its description, with notes that the group cannot change and that renaming a source or destination collection also renames its connection. Source: https://docs.tabsdata.com/tutorials/managing-collections/update Before you begin [Install Tabsdata](https://docs.tabsdata.com/server-admin/install) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) [Create a Project](https://docs.tabsdata.com/tutorials/managing-projects/create) [Create a Collection](https://docs.tabsdata.com/tutorials/managing-collections/create) AI agent session ❯rename the subscriptions collection to billing and describe it as invoicing and dunning transformers Called tabsdata 2 times (ctrl+o to expand) ⏺ Renamed. The functions and tables inside kept their names; only the collection changed. ``` ┌───────────────┬────────────────────────────────────┐ │ field │ value │ ├───────────────┼────────────────────────────────────┤ │ name │ billing │ ├───────────────┼────────────────────────────────────┤ │ previous name │ subscriptions │ ├───────────────┼────────────────────────────────────┤ │ description │ Invoicing and dunning transformers │ ├───────────────┼────────────────────────────────────┤ │ group │ default │ ├───────────────┼────────────────────────────────────┤ │ type │ data product │ ├───────────────┼────────────────────────────────────┤ │ connection │ none │ └───────────────┴────────────────────────────────────┘ ``` ⏺ Two things this does not do. It cannot move the collection to another group, so the type stays what it was. And on a source or destination collection a rename renames the connection too, since the connection is named by its collection. --- # Delete a Collection > Deletes a Tabsdata collection and its functions and tables, showing the confirmation prompt, the dependent functions that get frozen in other collections, and the name becoming reusable. Source: https://docs.tabsdata.com/tutorials/managing-collections/delete Before you begin [Install Tabsdata](https://docs.tabsdata.com/server-admin/install) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) [Create a Project](https://docs.tabsdata.com/tutorials/managing-projects/create) [Create a Collection](https://docs.tabsdata.com/tutorials/managing-collections/create) AI agent session ❯delete the billing collection Called tabsdata 1 time (ctrl+o to expand) ⏺ This collection holds 3 functions and 4 tables, and 1 function in another collection reads one of those tables. Confirm by writing `delete`. ❯delete Called tabsdata 2 times (ctrl+o to expand) ⏺ **billing** is gone. The name is free again immediately, so you can recreate it under the same name straight away. ``` ┌────────────────────────────┬───────────┐ │ field │ value │ ├────────────────────────────┼───────────┤ │ deleted │ billing │ ├────────────────────────────┼───────────┤ │ functions deleted │ 3 │ ├────────────────────────────┼───────────┤ │ tables deleted │ 4 │ ├────────────────────────────┼───────────┤ │ functions frozen elsewhere │ 1 │ ├────────────────────────────┼───────────┤ │ name now │ available │ └────────────────────────────┴───────────┘ ``` ⏺ The frozen function is the one to look at: it stays registered and stops running until it is re-registered against tables that exist. Functions outside the collection are frozen, not deleted, so nothing you built elsewhere is thrown away. --- # Register a Function > Registers a Tabsdata function from a Python file into a collection, covering the register command, the name, description, and collection options, and how versions and updates behave. Source: https://docs.tabsdata.com/tutorials/managing-functions/register Before you begin [Install Tabsdata](https://docs.tabsdata.com/server-admin/install) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) [Create a Collection](https://docs.tabsdata.com/tutorials/managing-collections/create) AI agent session ❯register a transformer that totals orders by region into the reporting collection ⏺ Needs a transformer. Getting its guide. Called tabsdata 2 times (ctrl+o to expand) ⏺ Generating — group\_by on region with a summed amount. Called tabsdata 3 times (ctrl+o to expand) ⏺ Registered as **summarize** in `reporting`, description "Rolls orders up by region": ``` ┌────────────┬───────────┐ │ field │ value │ ├────────────┼───────────┤ │ name │ summarize │ ├────────────┼───────────┤ │ collection │ reporting │ ├────────────┼───────────┤ │ state │ active │ ├────────────┼───────────┤ │ version │ 1 │ └────────────┴───────────┘ ``` ⏺ Registering the same file again creates a new version rather than erroring, as long as I pass update. Without it, a second registration under the same name is rejected. --- # Delete a Function > Explains how to delete a registered Tabsdata function, noting that its output tables remain and downstream functions are frozen rather than deleted. Source: https://docs.tabsdata.com/tutorials/managing-functions/delete Before you begin [Install Tabsdata](https://docs.tabsdata.com/server-admin/install) [Connect to MCP](https://docs.tabsdata.com/guide/connect-agent) [Register a Function](https://docs.tabsdata.com/tutorials/managing-functions/register) AI agent session ❯delete the summarize function from reporting Called tabsdata 2 times (ctrl+o to expand) ⏺ Deleted **summarize** from `reporting`. The tables it wrote are still there — deleting a function does not delete its tables. ⏺ Anything downstream that read from those tables is now frozen rather than deleted: it stays registered and stops running until it is re-registered against tables that still exist. --- # Install Tabsdata > Installs the tabsdata Python package and tabsdata-agent into a virtual environment using uv, pip, conda, pyenv, or venv on Linux, macOS, and Windows. Source: https://docs.tabsdata.com/server-admin/install Tabsdata requires the `tabsdata` python package, which is installable through `pip` and requires **no admin or root privileges**. Requirements - Machine with a supported OS - Linux (x86) - macOS (Apple Silicon/ARM64) - Windows (x86) - **Python 3.12.13 or later** - An LLM API key from **Anthropic** or **OpenAI (Required for AI features)** ### Create a Python virtual environment Using a virtual environment provides a layer of isolation and prevents interference from other python environments on your machine. - macOS / Linux - Windows (PowerShell) - conda - uv - pyenv - venv ```bash conda create -n tabsdata python=3.12 -y conda activate tabsdata ``` ```bash uv venv --python 3.12.13 .venv source .venv/bin/activate ``` ```bash pyenv install 3.12.13 pyenv local 3.12.13 python -m venv .venv source .venv/bin/activate ``` ```bash python3 -m venv .venv source .venv/bin/activate ``` - conda - uv - pyenv - venv ```powershell conda create -n tabsdata python=3.12 -y conda activate tabsdata ``` ```powershell uv venv --python 3.12.13 .venv .venv\Scripts\activate ``` ```powershell pyenv install 3.12.13 pyenv local 3.12.13 python -m venv .venv .venv\Scripts\activate ``` ```powershell python -m venv .venv .venv\Scripts\activate ``` ### Install Tabsdata Install `tabsdata-agent` alongside `tabsdata[all]` to enable AI features. - uv - pip ```bash uv pip install "tabsdata[all]" tabsdata-agent ``` ```bash pip install "tabsdata[all]" tabsdata-agent ``` Some Notes on Tabsdata Server Providers Tabsdata runs on Kubernetes, and the local deployment uses `kminus`: a lightweight provider that runs pods as native processes in your Python environment instead of Docker containers, so it needs no root or administrator access. Continue with [Deploy on k-minus](https://docs.tabsdata.com/server-admin/kminus), or deploy on [AWS EKS](https://docs.tabsdata.com/server-admin/eks-install), [Azure AKS](https://docs.tabsdata.com/server-admin/aks-install), or [Google GKE](https://docs.tabsdata.com/server-admin/gke-install). --- # Install Third-Party Database Drivers > Install third-party database drivers (mysql-connector-python, psycopg2-binary, oracledb) for MySQL, PostgreSQL, Oracle and CDC connectors using a requirements file and `tdkserver venv update`. Source: https://docs.tabsdata.com/server-admin/third-party-database-drivers Some Tabsdata connectors depend on Python database drivers that Tabsdata cannot redistribute because of their licensing terms. These drivers are **not** included in the Tabsdata distribution, and are not installed by default. If you use one of the affected connectors, you must install its driver explicitly as part of the Tabsdata installation. ## Which drivers do I need? | Connector | Required package | | --- | --- | | MySQL, MySQL CDC, MariaDB | `mysql-connector-python` | | PostgreSQL, PostgreSQL CDC | `psycopg2-binary` | | Oracle | `oracledb` | ## Declaring the drivers The Tabsdata package must be installed twice when creating a Tabsdata server: once in your own terminal before running `tdkserver quickstart`, and once more by the Tabsdata server's own shell process when it is being created. As a result, driver installation instructions must be passed as a plain [pip requirements file](https://pip.pypa.io/en/stable/reference/requirements-file-format/) listing the drivers and their versions that should be installed on Tabsdata's shell process. ## Adding a driver to a Tabsdata server Use `tdkserver venv update` to add or change database drivers. Step 1Create a requirements file with your additional drivers requirements-td.txt ```txt mysql-connector-python==9.6.0 psycopg2-binary==2.9.12 oracledb==3.2.0 ``` Step 2pass the file path into the `tdkserver venv update` command Tabsdata InstanceRequirements File ```bash tdkserver venv update --instance --name fn --requirements tdkserver venv update --instance --name ag --requirements ``` You need to add the requirements to both `fn` and `ag`, otherwise the Tabsdata AI Agent and MCP server won't be able to access your external systems to fetch metadata and sample data. Until the driver is present in the environment that executes functions, any connector that needs it will fail at run time. note When using Kminus (quickstart), Tabsdata is restarted as part of this update. When using EKS, Tabsdata is not restarted. ## EKS installation For an EKS installation the requirements file lives inside the instance's init assets. Step 1Initialise the instance ```bash tdkserver init ``` Step 2Edit the requirements file in the instance config directory Tabsdata Instance ```bash ~/.tabsdata/instances//init/assets/configs/requirements-td.txt ``` Add the necessary database drivers as shown in [Declaring the drivers](#declaring-the-drivers): requirements-td.txt ```txt mysql-connector-python==9.6.0 psycopg2-binary==2.9.12 oracledb==3.2.0 ``` Step 3Continue with the rest of the Tabsdata EKS setup instructions See [Install on EKS](https://docs.tabsdata.com/server-admin/eks-install). The drivers are baked into the function runner environment during setup, so they are available to connector functions at execution time. ## Troubleshooting **A connector fails with a missing-module error.** The driver for that connector is not installed in the environment running the function. Confirm that the package is listed in the requirements file, that `tdkserver venv update` ran against both `fn` and `ag` (or, for EKS, that the file is in the instance configs directory), and that the update has been re-run since the file changed. **The driver fails to build during installation.** Prefer the binary distributions named above (`psycopg2-binary` rather than `psycopg2`). If a build is still attempted, the installation environment is missing the compiler or client-library headers that the driver's source distribution needs. --- # Connect to MCP > Registers the Tabsdata MCP server (tabsdata-agent) with Claude Code or Codex using `claude mcp add` or `codex mcp add`, with shell-specific commands for Linux, macOS, PowerShell, and cmd.exe. Source: https://docs.tabsdata.com/server-admin/connect-agent Before you begin [Install Tabsdata](https://docs.tabsdata.com/server-admin/install) Tabsdata's agentic workflow uses an MCP server so a coding agent builds and runs your workflows for you. Install it into your agent before deploying a server. Claude Code and Codex are currently supported. Start the agent from a shell whose virtual environment has Tabsdata installed. note The MCP server needs `tabsdata-agent`, which ships as its own package and is not yet pulled in by `tabsdata[all]`. Install it if you have not done so yet: - uv - pip ```bash uv pip install tabsdata-agent ``` ```bash pip install tabsdata-agent ``` [Install Tabsdata](https://docs.tabsdata.com/server-admin/install) covers this as part of the full install, so skip this if you followed that page. - Claude Code - Codex ```bash claude mcp add tabsdata -- python -m tabsdatak.ag.mcp - ``` - Linux / macOS - Windows PowerShell - Windows cmd.exe ```bash codex mcp add tabsdata -- "$(python -c 'import sys; print(sys.executable)')" -m tabsdatak.ag.mcp - ``` ```powershell codex mcp add tabsdata -- (python -c "import sys; print(sys.executable)") -m tabsdatak.ag.mcp - ``` ```bat for /f "delims=" %i in ('python -c "import sys; print(sys.executable)"') do codex mcp add tabsdata -- "%i" -m tabsdatak.ag.mcp - ``` --- # Deploy Tabsdata on k-minus > Quickstart a local Tabsdata Server or manage its k-minus lifecycle step by step. Source: https://docs.tabsdata.com/server-admin/kminus k-minus is Tabsdata's default deployment provider. It runs Kubernetes-style pods as native processes in a Python environment instead of Docker containers. Use Quickstart to deploy with the default configuration. Use the individual lifecycle commands when you want to edit the configuration before creating and starting the Server. Before you begin [Install Tabsdata](https://docs.tabsdata.com/server-admin/install) ## Quickstart a Server `tdkserver quickstart` bundles [initializing](#initialize-a-server), [creating](#create-a-server), and [starting](#start-a-server) a Tabsdata server into a single command, using a default configuration for compute and storage. ### Start the Tabsdata server `tdkserver quickstart` initializes and deploys a Tabsdata Instance. The command should take around 30 seconds to fully execute. - Anthropic - OpenAI ```bash export ANTHROPIC_KEY= tdkserver quickstart --config anthropic-key=env:ANTHROPIC_KEY ``` ```bash export OPENAI_KEY= tdkserver quickstart --config openai-key=env:OPENAI_KEY ``` note If you plan to use the MySQL or PostgreSQL CDC connectors, they need database drivers that Tabsdata cannot redistribute. After the quickstart completes, add them with `tdkserver venv update`, as described in [Installing Third-Party Database Drivers](https://docs.tabsdata.com/server-admin/third-party-database-drivers): ```bash tdkserver venv update --instance --name fn --requirements tdkserver venv update --instance --name ag --requirements ``` ### Check Tabsdata server status View all your Tabsdata Instances ```bash tdkserver list ``` Output ``` Instances ┌──────────┬──────────────────────────────────────────────────┬──────────┬─────────┬─────────┬─────────┬─────────────────┬───────────────┬───────────────────────┐ │ Name │ Folder │ Provider │ Version │ State │ Status │ API Server Port │ Database Port │ API Server URL │ ├──────────┼──────────────────────────────────────────────────┼──────────┼─────────┼─────────┼─────────┼─────────────────┼───────────────┼───────────────────────┤ │ tabsdata │ ~/.tabsdata/instances/tabsdata │ kminus │ 2.0.2 │ created │ Running │ 2457 │ 2460 │ http://localhost:2457 │ └──────────┴──────────────────────────────────────────────────┴──────────┴─────────┴─────────┴─────────┴─────────────────┴───────────────┴───────────────────────┘ ``` View the status of the Tabsdata Instance you just deployed. ```bash tdkserver status ``` Output ``` Deployments ┌───────────────┬─────────┬───────┬─────┐ │ Name │ Status │ Ready │ Age │ ├───────────────┼─────────┼───────┼─────┤ │ srv-aiagent │ Running │ 1/1 │ 15s │ │ srv-apiserver │ Running │ 1/1 │ 15s │ │ srv-database │ Running │ 1/1 │ 24s │ └───────────────┴─────────┴───────┴─────┘ Pods ┌────────────────────────────────┬─────────┬───────┬─────┐ │ Name │ Status │ Ready │ Age │ ├────────────────────────────────┼─────────┼───────┼─────┤ │ srv-aiagent-7476c88484-4klx8 │ Running │ 2/2 │ 15s │ │ srv-apiserver-56bbbb559d-8cnxl │ Running │ 1/1 │ 15s │ │ srv-database-fd4f8f85-dsxcj │ Running │ 1/1 │ 24s │ └────────────────────────────────┴─────────┴───────┴─────┘ CronJobs ┌──────────────────┬─────────┬─────────────┬─────────────────┬─────┐ │ Name │ Status │ Expression │ Explanation │ Age │ ├──────────────────┼─────────┼─────────────┼─────────────────┼─────┤ │ srv-aiembeddings │ Running │ */5 * * * * │ Every 5 minutes │ 6s │ │ srv-aimetadata │ Running │ */5 * * * * │ Every 5 minutes │ 6s │ └──────────────────┴─────────┴─────────────┴─────────────────┴─────┘ Jobs ┌────────────────┬──────────┬─────────────┬─────┐ │ Name │ Status │ Completions │ Age │ ├────────────────┼──────────┼─────────────┼─────┤ │ ini-inspector │ Complete │ 1/1 │ 16s │ │ ini-venvloader │ Complete │ 1/1 │ 17s │ └────────────────┴──────────┴─────────────┴─────┘ ``` tip `tdkserver` commands usually require an `--instance` flag that specifies the specific instance you want to interact with. If no `--instance` flag is passed, Tabsdata defaults the instance name to `tabsdata` ### Log in ```bash tdk login --server localhost --user admin --password tabsdata ``` Or access the UI at [http://localhost:2457/login](http://localhost:2457/login) ## Initialize a Server `tdkserver init` produces an editable config folder for a Tabsdata instance within `~/.tabsdata`, pre-filled with default values. ### Initialize an instance ```bash tdkserver init --instance myinstance --provider kminus ``` Output ``` About to initialize tabsdata instance: - instance.....: myinstance - provider.....: kminus - log level....: debug (default) Proceed with the initialization? [y/yes/n/no]: y ▶ Creating tabsdata instance folder ✔ Created tabsdata instance folder at ~/.tabsdata/instances/myinstance (0s) ▶ Initializing tabsdata templates ✔ Initialized tabsdata templates in ~/.tabsdata/instances/myinstance/init/input (0s) ▶ Writing tabsdata instance settings ✔ Wrote tabsdata instance settings (0s) Initialized tabsdata instance 'myinstance'. - state........: initialized - version......: 2.0.0 - log level....: debug - provider.....: kminus - instance name: myinstance - kubeconfig...: ~/.tabsdata/instances/myinstance/cluster/config/kubeconfig.yaml - namespace....: default ╭─────────────────────────────────────────────────────────────────────╮ │ Inspect and edit the config in the instance's init folder, then run │ │ 'tdkserver create --instance myinstance' to create the instance. │ ╰─────────────────────────────────────────────────────────────────────╯ ``` The config folder is written to `~/.tabsdata/instances/myinstance/`. The default configuration is what is used by `tdkserver quickstart` which bundles initializing, creating, and starting a server Edit the config files the command produced, if you need to change anything. ## Create a Server After [initializing](#initialize-a-server) a server, `tdkserver create` builds an instance from the config folder that [`tdkserver init`](#initialize-a-server) produced. Step 1Set the LLM API key `tdkserver create` builds the aiagent pod, so the LLM key has to be in the environment before running it, not before [`tdkserver init`](#initialize-a-server): - Anthropic - OpenAI ```bash export ANTHROPIC_KEY="sk-ant-..." ``` ```bash export OPENAI_KEY="sk-..." ``` Step 2Create an instance ```bash tdkserver create --instance myinstance ``` A server's name is unique and is used internally for identification. The name provided to `--instance` is used to find the config folder created during initialization. Step 3Check the instance was created ```bash tdkserver list ``` Output ``` Instances ┌──────────┬──────────────────────────────────────────────────┬──────────┬─────────┬─────────┬─────────┬─────────────────┬───────────────┬───────────────────────┐ │ Name │ Folder │ Provider │ Version │ State │ Status │ API Server Port │ Database Port │ API Server URL │ ├──────────┼──────────────────────────────────────────────────┼──────────┼─────────┼─────────┼─────────┼─────────────────┼───────────────┼───────────────────────┤ │ tabsdata │ ~/.tabsdata/instances/tabsdata │ kminus │ 2.0.2 │ created │ Stopped │ 2457 │ 2460 │ http://localhost:2457 │ └──────────┴──────────────────────────────────────────────────┴──────────┴─────────┴─────────┴─────────┴─────────────────┴───────────────┴───────────────────────┘ ``` The instance appears with its name and state. It is not running yet. ### Ports Every service in the instance takes a port, assigned sequentially from a single base value. `--port` sets that base, defaulting to `2457`: ```bash tdkserver create --instance myinstance --port 3000 ``` Two instances on one machine need base ports far enough apart that their ranges do not overlap. The instance keeps its configuration across stops and starts, so creating it is a one-time step. See [`tdkserver create`](https://docs.tabsdata.com/cli/tdkserver/create) for the full option list. ## Start a Server `tdkserver start` brings up the services of an instance that [`tdkserver create`](#create-a-server) already built. Until the instance is running, `tdk` has nothing to connect to. Step 1Start an instance ```bash tdkserver start --instance myinstance ``` ## Stop a Server `tdkserver stop` shuts down a running instance's services and leaves everything else in place. The instance stays registered, its configuration is untouched, and the tables and function code it holds survive, so a later [start](#start-a-server) picks up where the stop left off. Step 1Stop the instance ```bash tdkserver stop --instance myinstance ``` The command prints the values it resolved and waits for confirmation. Pass `--yes` to skip the prompt. Step 2Confirm it stopped ```bash tdkserver status --instance myinstance ``` ### Running functions Stopping the server stops its workers with it, so a function run in flight does not finish. Nothing partial is committed: a run only commits when every function in its execution plan succeeds, so an interrupted run leaves the tables at the version they held beforehand. Check for active work before stopping: ```bash tdk fnrun list --status running ``` ### Picking up a configuration change A configuration edit reaches a running instance through a restart rather than a stop: ```bash tdkserver restart --instance myinstance ``` See [`tdkserver stop`](https://docs.tabsdata.com/cli/tdkserver/stop) for the full option list. ## Delete a Server To permanently remove the instance, follow [Delete a Server](https://docs.tabsdata.com/server-admin/delete). --- # Install Tabsdata on AWS EKS > Installs Tabsdata on AWS EKS: priority classes, kubeconfig access, RDS PostgreSQL and S3 config files, tdkserver init/assess/create/start, port-forward, and login. Source: https://docs.tabsdata.com/server-admin/eks-install [AWS Services Setup for Tabsdata on EKS](#aws-services-setup-for-tabsdata-on-eks) is included below. ## Before You Begin See the AWS Services Setup for Tabsdata on EKS documentation for prerequisite configuration. ## Overview Tabsdata runs on Kubernetes as deployments containing an API server, AI agent, and supporting services within a single namespace. User functions execute as individual Kubernetes jobs that start, run, and exit. Table data resides in AWS S3 while metadata lives in AWS RDS for PostgreSQL. This installation is performed by the Tabsdata administrator. ## Requirements - A Windows (x86), macOS (ARM64), or Linux (x86) client machine - Python 3.12.13 installed - The AWS CLI (`aws`) and `kubectl` - AWS EKS Kubernetes cluster with x86 nodes - AWS RDS for PostgreSQL instance running version 18.4 in the same region as EKS - AWS S3 bucket, preferably in the same region - Network access between the EKS cluster and the RDS instance, and between the client and the EKS cluster ### Priority Classes Tabsdata pods name two priority classes, `tabsdata-service` for the instance and `tabsdata-function` for the runs it triggers. They are cluster-wide, so a cluster administrator creates them once; the service class must hold the higher value: ```yaml apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: tabsdata-service value: 1000 globalDefault: false description: Tabsdata instance pods --- apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: tabsdata-function value: 100 globalDefault: false description: Pods of the runs a Tabsdata instance triggers ``` ```text kubectl --kubeconfig apply -f tabsdata-priority-classes.yaml ``` ### Cluster Access Fetch the cluster credentials into your kubeconfig: ```text aws eks update-kubeconfig --region --name --kubeconfig kubectl --kubeconfig config get-contexts ``` The context is named after the cluster's ARN, `arn:aws:eks:::cluster/`. ## Install Tabsdata Packages From a machine with EKS cluster access, create a Python virtual environment and install: **Using uv:** ```text uv pip install "tabsdata[all]" tabsdata-agent ``` **Using pip:** ```text pip install "tabsdata[all]" tabsdata-agent ``` **Note:** MySQL and PostgreSQL CDC connectors require database drivers that Tabsdata cannot redistribute. See "Installing Third-Party Database Drivers" if needed before proceeding. ## Configuration Values Needed Gather these before starting: | Category | Variable | Description | | --- | --- | --- | | AWS RDS for PostgreSQL | `` | Hostname of RDS instance | | | `` | Port of RDS instance | | | `` | Database for Tabsdata installation | | | `` | Connection username | | | `` | Connection password | | AWS S3 | `` | Bucket for data storage | | | `` | Bucket region | | | `` | Access key with full bucket access | | | `` | Secret key for access key | | AWS EKS | `` | Region of the cluster | | | `` | Target cluster | | | `` | Empty reserved namespace | | | `` | Path to kubeconfig file | | LLM Provider | `` / `` | LLM API key if AI integration enabled | | Chosen During Install | `` | Tabsdata instance name | ### Database TLS Nothing to fetch. AWS publishes the RDS certificate bundle, which ships in the Tabsdata images; Tabsdata connects with `verify-ca` against it. ## Configure Tabsdata ### Prepare the Namespace Create the namespace; `tdkserver` never creates one: ```text kubectl --kubeconfig create namespace ``` ### Prepare Configuration ```text tdkserver init --instance --provider eks --kubeconfig --namespace ``` This creates `~/.tabsdata/instances//` with configuration files to edit in `~/.tabsdata/instances//init/input/configs/`. ### Configure Database Edit `dbmanager.database.yaml` with RDS information. Leave `ssl_mode` and `ssl_root_cert` as they are; Tabsdata fills them: ```yaml database_server: postgres: host: port: database: username: password: ssl_mode: ${{system:tdk_database_ssl_mode}} ssl_root_cert: ${{system:tdk_database_ssl_root_cert}} pool: admin: min_connections: 1 max_connections: 1 acquire_timeout: 30 max_lifetime: 1800 idle_timeout: 30 test_before_acquire: true ``` Use a host the pods in the cluster can reach: the RDS endpoint resolves to its in-VPC address from inside the VPC. The `apiserver.database.yaml` file requires no editing; Tabsdata generates and sets credentials automatically for the `_apiserver` and `_aiagent` roles. ### Configure S3 Storage Edit `apiserver.storage.yaml` with S3 details for three mounts (ROOT, BUNDLES, IMAGES). They can share the same bucket with different paths: ```yaml storage: mounts: - id: ROOT path: / uri: s3:///root/ options: aws_region: aws_access_key_id: aws_secret_access_key: - id: BUNDLES path: /bundles uri: s3:///bundles/ options: aws_region: aws_access_key_id: aws_secret_access_key: - id: IMAGES path: /images uri: s3:///images/ options: aws_region: aws_access_key_id: aws_secret_access_key: ``` **Warning:** S3 URIs must end with `/`. ### Configure AI Agent Edit `aiagent.yaml` to enable LLM integration with Anthropic or OpenAI API keys: ```yaml port: ${{system:tdk_aiagent_public_port}} allow_writes: true ai_providers: openai: api_key: model: openai/gpt-5.4 anthropic: api_key: model: anthropic/claude-sonnet-4-6 hybrid_search: embedder: fastembed embedding_model: BAAI/bge-small-en-v1.5 device: cpu cache_dir: null offline: false ``` ### Preflight Check Verify configuration completeness: ```text tdkserver config check --instance --init ``` ### Assess the Instance Before creating the instance, check that it can be created: ```text tdkserver assess --instance ``` `assess` changes nothing. It checks everything `create` and the first `start` depend on, and reports each check as passed, failed, warning, skipped (could not run) or waived (does not apply): | Area | What is checked | | --- | --- | | Local registry | The instance is initialized and not yet created; `init` was run by this Tabsdata version and edition; the init folder differs from the templates only in filled placeholders; no mandatory placeholder is left | | Cluster access | The program the kubeconfig runs to authenticate (`aws`) is present; the API server answers; the namespace exists | | Operator rights | Your identity has the verbs an instance needs in the namespace | | Scheduling | The priority classes `tabsdata-service` and `tabsdata-function` exist, the service one higher; a ready node carries the labels the service pods select; a node or pool can serve the function pods; a pool carries the taint the function pods tolerate | | Database | The CA the pods verify against is in place; the database answers and accepts the credentials, over TLS; the role has `CREATEROLE` and `CREATEDB` and does not bypass row-level security; the role owns its database; the database holds no schemas from an earlier instance | | Storage | Each mount carries the options its scheme takes; each bucket answers and accepts the credentials | | Coherence | The package index the function environments use answers; no secret is both supplied and generated; the AI agent has a model it can call | It exits 0 when nothing stands in the way of `create`, and 1 when a check failed or was skipped. Add `--explain` to print, after the report, what every check asks. Fix what it reports, then run it again. **Note:** The database and bucket checks run from the client machine. An RDS instance only reachable from inside the VPC shows as a warning there, not a failure; wrong credentials still fail. ### Create Instance ```text tdkserver create --instance ``` Verify state: ```text tdkserver inspect --instance ``` ### Start Instance The first start builds the Python environments and sets up the database, so it takes several minutes. ```text tdkserver start --instance tdkserver status --instance ``` Every service pod should reach `Running`: ```text kubectl --kubeconfig --namespace get pods ``` ### Access Tabsdata Use port forwarding for verification: ```text kubectl port-forward --kubeconfig --namespace service/net-apiserver 2457:2457 ``` Access at `http://127.0.0.1:2457`. Port 2457 is default; adjust if using a different base port. **Note:** Port forwarding is for verification only. It is a single stream through the cluster's control plane and drops when the pods restart; restart it when that happens. Properly expose Tabsdata before production use. ### Log In ```text tdk login --server --user admin --password tabsdata tdk status ``` ### Post-Installation Cleanup Delete local setup files containing sensitive credentials: ```text rm -rf ~/.tabsdata/instances//init ``` ## Expose Tabsdata Outside Cluster The instance runs but is only accessible via port forwarding. See AWS Services Setup documentation for HTTPS exposure instructions. ## Next Steps Run your first data integration per "Run Your First Data Integration" documentation. ## Appendix: Sizing Service and Function Pods Configure sizing in the instance's `configs/tabsdata.yaml`. Service pods cover the instance components (API server, AI agent, embeddings, metadata); function pods cover per-execution pods. ```yaml pods: service: priority: tabsdata-service resources: cpu: "2" memory: 1Gi disk: 16Gi placement: selectors: {} tolerations: [] apiserver: resources: cpu: "2" memory: 4Gi disk: 16Gi aiagent: resources: cpu: "2" memory: 8Gi disk: 64Gi function: priority: tabsdata-function resources: cpu: "1" memory: 16Gi disk: 64Gi placement: selectors: {} tolerations: - key: workload value: batch effect: NoSchedule ``` Set values based on actual component footprints and confirm node pools can satisfy the largest single pod. A tolerations list replaces the one Tabsdata would apply, so it must be complete. Use `tdkserver config get` and `tdkserver config put` to manage the file. ## AWS Services Setup for Tabsdata on EKS Tabsdata on EKS runs as long-running deployment pods in a single namespace, and dispatches every publisher, subscriber and transformer execution as its own Kubernetes job. It keeps no state in the cluster: table data lives in AWS S3 and all metadata lives in AWS RDS for PostgreSQL. This document covers the AWS-side setup those three services need before Tabsdata can be installed, and how to expose the instance once it is running. The values produced here are the ones the Tabsdata admin needs at hand for [Install Tabsdata on AWS EKS](#overview). ### Before you begin These instructions assume you have knowledge of, and full access to, the AWS IAM, EKS, RDS and S3 services, or that your AWS admins do. Each section indicates the type of admin access it requires. It is recommended you read through the whole setup before you begin, so you can gather all the necessary information and have it ready. ### Prepare the EKS cluster (EKS admin) Tabsdata must be installed into an existing cluster ``, in an empty namespace. Create an EKS namespace ``. #### Node scheduling (EKS admin) Karpenter must be installed in the cluster and given the AWS resources it needs: an IAM role for the controller (IRSA or EKS Pod Identity) allowing it to run instances, create launch templates and pass the node role; a node IAM role admitted to the cluster via an EKS access entry or `aws-auth`; the SQS interruption queue with its EventBridge rules; and `karpenter.sh/discovery` tags on the subnets and security groups the `EC2NodeClass` selects. Because Karpenter cannot run on a node it manages, a small fixed managed node group must exist beforehand, outside the pools, to hold it and the cluster addons. Two `EC2NodeClass`/`NodePool` pairs are then created, distinguished mainly by local disk. The services pool is untainted, sized for the long-running instance pods, with the instance store exposed to the kubelet, expiry disabled and consolidation enabled so the fleet shrinks. The functions pool is sized for the substantially larger disk a run stages, with an explicit memory floor since that does not follow from vCPU or disk, and is tainted so a function pod that lost its toleration fails visibly rather than running slowly on a small machine. All sizes in the reference pools are initial suggested values. Set them from the instance's own requests and expected concurrency and volumetry. Pool and node class names are free; placement follows the pods' resource requests, so function pods land on the functions pool because only it provisions machines that large. Three strings are binding and must match exactly. Two `PriorityClass` objects named `tabsdata-service` and `tabsdata-function` must exist, with the service class ranked higher, because the API server rejects a pod naming a class that is absent. The functions pool's taint key, value and effect must be identical to `pods.function.placement.tolerations` in the instance's `configs/tabsdata.yaml`, edited via `tdkserver get-cfg` / `tdkserver put-cfg`. Resource requests and the `aut-services` disruption budget arrive with the instance and need no administrator action. #### Map your AWS user to a Kubernetes group (EKS admin) Map the AWS username `` that will install and manage Tabsdata to the Kubernetes group `` in the cluster ``, region ``. #### Create the namespace role and binding (EKS admin) This role grants Tabsdata the permissions it needs inside its own namespace. ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: namespace: rules: - apiGroups: [ "" ] resources: [ "pods" ] verbs: [ "get", "list", "watch", "create", "update", "patch", "delete", "deletecollection" ] - apiGroups: [ "" ] resources: [ "services", "configmaps", "secrets", "serviceaccounts" ] verbs: [ "get", "list", "watch", "create", "update", "patch", "delete" ] - apiGroups: [ "" ] resources: [ "pods/log" ] verbs: [ "get", "list" ] - apiGroups: [ "" ] resources: [ "pods/portforward" ] verbs: [ "create" ] - apiGroups: [ "apps" ] resources: [ "deployments" ] verbs: [ "get", "list", "watch", "create", "update", "patch", "delete", "deletecollection" ] - apiGroups: [ "policy" ] resources: [ "poddisruptionbudgets" ] verbs: [ "get", "list", "create", "update", "delete" ] - apiGroups: [ "networking.k8s.io" ] resources: [ "ingresses" ] verbs: [ "get", "list", "watch", "create", "update", "patch", "delete" ] - apiGroups: [ "batch" ] resources: [ "jobs", "cronjobs" ] verbs: [ "get", "list", "watch", "create", "update", "patch", "delete", "deletecollection" ] - apiGroups: [ "rbac.authorization.k8s.io" ] resources: [ "roles", "rolebindings" ] verbs: [ "get", "list", "create", "update", "delete" ] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: namespace: subjects: - kind: Group name: apiGroup: rbac.authorization.k8s.io roleRef: kind: Role name: apiGroup: rbac.authorization.k8s.io ``` #### Create the cluster role and binding (EKS admin) A few of the checks Tabsdata runs cannot be expressed in a namespaced role, so they need a cluster role. All of them are read-only, and none grants any access beyond reading. Create the cluster role `` and bind it to the group ``. ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: rules: - apiGroups: [ "" ] resources: [ "namespaces", "services", "nodes" ] verbs: [ "get", "list" ] - apiGroups: [ "karpenter.sh" ] resources: [ "nodepools" ] verbs: [ "get", "list" ] ``` Finally, provide a `` file to the Tabsdata operator so the installation steps below can reach the cluster. #### Verify your permissions (Tabsdata admin) Run these checks with the `` you were given, before going any further. ```bash kubectl auth can-i create deployments.apps kubectl auth can-i create cronjobs.batch kubectl auth can-i create secrets kubectl auth can-i create services kubectl auth can-i create roles.rbac.authorization.k8s.io kubectl auth can-i create serviceaccounts kubectl auth can-i get pods/log kubectl auth can-i deletecollection pods kubectl auth can-i deletecollection deployments.apps kubectl auth can-i deletecollection jobs.batch kubectl auth can-i deletecollection cronjobs.batch kubectl auth can-i create pods/portforward kubectl auth can-i create ingresses.networking.k8s.io kubectl auth can-i create poddisruptionbudgets.policy kubectl auth can-i list namespaces kubectl auth can-i list services --all-namespaces kubectl auth can-i list nodes kubectl auth can-i list nodepools.karpenter.sh ``` Every command must answer `yes`. If any command fails or answers `no`, check with your AWS EKS administrator before proceeding. ### Tabsdata database (RDS admin) Tabsdata stores its own state and the metadata for user data in PostgreSQL. Create an **AWS RDS for PostgreSQL** 18.4 instance with a database ``, using password authentication with `` and ``. `` must have the following permissions: | Requirement | Kind | Statement | | --- | --- | --- | | `CONNECT` on `` | privilege | `GRANT CONNECT ON DATABASE TO ;` | | `CREATEROLE` | role attribute | `ALTER ROLE CREATEROLE;` | | `CREATE` on `` | privilege | `GRANT CREATE ON DATABASE TO ;` | The RDS master username already satisfies all three, and owns any database it creates. The Tabsdata installation will create the necessary database schemas, roles and grants. #### What the installation creates (RDS admin) One schema per Tabsdata component, inside ``: | Schema | Component | | --- | --- | | `_apiserver` | API server | | `_aiagent` | AI agent | Each schema holds that component's tables, indexes, views, triggers, row-level-security policies and baseline data. Three PostgreSQL roles. Roles are cluster-global, so their names must be unique across the whole RDS instance, which is why they derive from ``: | Role | Login | Password | Owns | | --- | --- | --- | --- | | `_apiserver` | `LOGIN` | generated automatically | the API server schema | | `_aiagent` | `LOGIN` | generated automatically | the AI agent schema | | `_app` | `NOLOGIN` | none | nothing | The two login roles are the identities Tabsdata authenticates as, one per component. Their passwords are generated and set automatically during installation. `_app` is the role every Tabsdata connection then switches to, and the one row-level security is enforced against. It is shared by both components, owns nothing, and cannot be reached directly: it has no login and no password, and is used only internally by Tabsdata. ### Tabsdata cloud storage (S3 admin) Tabsdata uses AWS S3 bucket(s) for data storage. You need an S3 bucket `` in region ``, and full-access credentials for it: `` and ``. ### Configure access from outside the cluster (over HTTPS) (EKS admin) note None of this applies to local access via `kubectl port-forward`, as described in [Install Tabsdata on AWS EKS](#overview). Once installed, Tabsdata is running inside the cluster, but is not reachable from outside it. Public access is delivered by two AWS components layered on the cluster. In EKS, the apiserver Service is fronted by an `alb`\-class Ingress, which the AWS Load Balancer Controller turns into an internet-facing Application Load Balancer targeting pod IPs, so the controller must already be installed with an IAM role permitting ELBv2 management, and the VPC's public subnets must carry the `kubernetes.io/role/elb` tag in every availability zone in use. That ALB listens on HTTP only, and its security group is locked to the AWS-managed CloudFront origin-facing prefix list (read from EC2), so it is not reachable directly. In front of it, a CloudFront distribution uses the ALB as a custom origin and serves viewers over HTTPS on its own `*.cloudfront.net` hostname, which is why no Route 53 record or ACM certificate is needed. Deployment of the distribution takes five to ten minutes; after that the instance answers only through CloudFront. Use the distribution's `*.cloudfront.net` URL as ``, and point your browser at it. --- # Install Tabsdata on Azure AKS > Installs Tabsdata on Azure AKS: required Azure resources, priority classes, node pools, database and Blob storage configuration, tdkserver init/assess/create/start, and verification. Source: https://docs.tabsdata.com/server-admin/aks-install ## Before You Begin This page assumes the Azure resources below already exist: an AKS cluster, an Azure Database for PostgreSQL flexible server, and a storage account with a Blob container. ## Overview Tabsdata runs on Kubernetes as deployments containing an API server, AI agent, and supporting services within a single namespace. User functions execute as individual Kubernetes jobs that start, run, and exit. Table data resides in Azure Blob Storage while metadata lives in Azure Database for PostgreSQL. This installation is performed by the Tabsdata administrator. ## Requirements - A Windows (x86), macOS (ARM64), or Linux (x86) client machine - Python 3.12.13 installed - The Azure CLI (`az`) and `kubectl` - Azure AKS Kubernetes cluster with x86 nodes - Azure Database for PostgreSQL **Flexible Server** running version 18, in the same region as AKS (Single Server is not supported) - Azure storage account with a Blob container, preferably in the same region - Network access between the AKS cluster and the flexible server (VNet integration or a private endpoint), and between the client and the AKS cluster ### Priority Classes Tabsdata pods name two priority classes, `tabsdata-service` for the instance and `tabsdata-function` for the runs it triggers. They are cluster-wide, so a cluster administrator creates them once; the service class must hold the higher value: ```yaml apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: tabsdata-service value: 1000 globalDefault: false description: Tabsdata instance pods --- apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: tabsdata-function value: 100 globalDefault: false description: Pods of the runs a Tabsdata instance triggers ``` ```text kubectl --kubeconfig apply -f tabsdata-priority-classes.yaml ``` ### Node Pools The AKS configuration schedules pods by node label. Service pods are placed on nodes labeled `role=tabsdata-services`, function pods on nodes labeled `role=tabsdata-functions`. Function pods tolerate the `workload=batch:NoSchedule` taint, so the function pool can be reserved for them: ```text az aks nodepool add --subscription --resource-group \ --cluster-name --name tdservices \ --labels role=tabsdata-services az aks nodepool add --subscription --resource-group \ --cluster-name --name tdfunctions \ --labels role=tabsdata-functions \ --node-taints workload=batch:NoSchedule ``` Size the VM sizes for the largest single pod (see the sizing appendix). To place pods differently, edit `placement.selectors` in `tabsdata.yaml` instead. ### Cluster Access Fetch the cluster credentials into your kubeconfig. Name the subscription explicitly: the cluster's subscription is not necessarily the one `az account show` reports. ```text az aks get-credentials --subscription --resource-group --name kubectl --kubeconfig config get-contexts ``` The context is named after the cluster, ``. Its API server endpoint is a `*.azmk8s.io` host; check it with `kubectl --kubeconfig config view --minify -o jsonpath='{.clusters[0].cluster.server}'`. ## Install Tabsdata Packages From a machine with AKS cluster access, create a Python virtual environment and install: **Using uv:** ```text uv pip install "tabsdata[all]" tabsdata-agent ``` **Using pip:** ```text pip install "tabsdata[all]" tabsdata-agent ``` **Note:** MySQL and PostgreSQL CDC connectors require database drivers that Tabsdata cannot redistribute. See "Installing Third-Party Database Drivers" if needed before proceeding. ## Configuration Values Needed Gather these before starting: | Category | Variable | Description | | --- | --- | --- | | Azure Database for PostgreSQL | `` | Host of the flexible server, `.postgres.database.azure.com` | | | `` | Port of the flexible server (5432) | | | `` | Database for Tabsdata installation | | | `` | Administrator username (plain name, not `user@server`) | | | `` | Connection password | | Azure Blob Storage | `` | Storage account name | | | `` | Storage account access key | | | `` | Blob container for data storage | | Azure AKS | `` | Subscription of the cluster | | | `` | Resource group of the cluster | | | `` | Target cluster | | | `` | Empty reserved namespace | | | `` | Path to kubeconfig file | | LLM Provider | `` / `` | LLM API key if AI integration enabled | | Chosen During Install | `` | Tabsdata instance name | ### Database TLS Nothing to fetch. Azure signs flexible servers with a public root (DigiCert Global Root G2), which ships in the Tabsdata images; Tabsdata connects with `verify-ca` against it. If your server chains to a different root (Microsoft RSA Root CA 2017 on some servers), see [https://learn.microsoft.com/azure/postgresql/flexible-server/concepts-networking-ssl-tls](https://learn.microsoft.com/azure/postgresql/flexible-server/concepts-networking-ssl-tls). ## Configure Tabsdata ### Prepare the Namespace Create the namespace; `tdkserver` never creates one: ```text kubectl --kubeconfig create namespace ``` ### Prepare Configuration ```text tdkserver init --instance --provider aks --kubeconfig --namespace ``` This creates `~/.tabsdata/instances//` with configuration files to edit in `~/.tabsdata/instances//init/input/configs/`. ### Configure Database Edit `dbmanager.database.yaml` with the flexible server information. Leave `ssl_mode` and `ssl_root_cert` as they are; Tabsdata fills them: ```yaml database_server: postgres: host: port: database: username: password: ssl_mode: ${{system:tdk_database_ssl_mode}} ssl_root_cert: ${{system:tdk_database_ssl_root_cert}} pool: admin: min_connections: 1 max_connections: 1 acquire_timeout: 30 max_lifetime: 1800 idle_timeout: 30 test_before_acquire: true ``` Use a host the pods in the cluster can reach: with VNet integration or a private endpoint, the server's private name resolves inside the VNet. The `apiserver.database.yaml` file requires no editing; Tabsdata generates and sets credentials automatically for the `_apiserver` and `_aiagent` roles. ### Configure Azure Blob Storage Edit `apiserver.storage.yaml` with Azure Blob details for three mounts (ROOT, BUNDLES, IMAGES). They can share the same container with different paths: ```yaml storage: mounts: - id: ROOT path: / uri: az:///root/ options: azure_storage_account_name: azure_storage_account_key: - id: BUNDLES path: /bundles uri: az:///bundles/ options: azure_storage_account_name: azure_storage_account_key: - id: IMAGES path: /images uri: az:///images/ options: azure_storage_account_name: azure_storage_account_key: ``` **Warning:** Azure Blob URIs must end with `/`. ### Configure AI Agent Edit `aiagent.yaml` to enable LLM integration with Anthropic or OpenAI API keys: ```yaml port: ${{system:tdk_aiagent_public_port}} allow_writes: true ai_providers: openai: api_key: model: openai/gpt-5.4 anthropic: api_key: model: anthropic/claude-sonnet-4-6 hybrid_search: embedder: fastembed embedding_model: BAAI/bge-small-en-v1.5 device: cpu cache_dir: null offline: false ``` ### Preflight Check Verify configuration completeness: ```text tdkserver config check --instance --init ``` ### Assess the Instance Before creating the instance, check that it can be created: ```text tdkserver assess --instance ``` `assess` changes nothing. It checks everything `create` and the first `start` depend on, and reports each check as passed, failed, warning, skipped (could not run) or waived (does not apply): | Area | What is checked | | --- | --- | | Local registry | The instance is initialized and not yet created; `init` was run by this Tabsdata version and edition; the init folder differs from the templates only in filled placeholders; no mandatory placeholder is left | | Cluster access | The program the kubeconfig runs to authenticate (`kubelogin`, when the cluster uses Microsoft Entra ID) is present; the API server answers; the namespace exists | | Operator rights | Your identity has the verbs an instance needs in the namespace | | Scheduling | The priority classes `tabsdata-service` and `tabsdata-function` exist, the service one higher; a ready node carries the labels the service pods select; a node or pool can serve the function pods; a pool carries the taint the function pods tolerate | | Database | The CA the pods verify against is in place; the database answers and accepts the credentials, over TLS; the role has `CREATEROLE` and `CREATEDB` and does not bypass row-level security; the role owns its database; the database holds no schemas from an earlier instance | | Storage | Each mount carries the options its scheme takes; each bucket answers and accepts the credentials | | Coherence | The package index the function environments use answers; no secret is both supplied and generated; the AI agent has a model it can call | It exits 0 when nothing stands in the way of `create`, and 1 when a check failed or was skipped. Add `--explain` to print, after the report, what every check asks. Fix what it reports, then run it again. **Note:** The database and bucket checks run from the client machine. A flexible server only reachable inside the VNet shows as a warning there, not a failure; wrong credentials still fail. ### Create Instance ```text tdkserver create --instance ``` Verify state: ```text tdkserver inspect --instance ``` ### Start Instance The first start builds the Python environments and sets up the database, so it takes several minutes. ```text tdkserver start --instance tdkserver status --instance ``` Every service pod should reach `Running`: ```text kubectl --kubeconfig --namespace get pods ``` A pod stuck in `Pending` usually means no node carries the label its `placement.selectors` asks for (see Node Pools), or no node is large enough for its resources. ### Access Tabsdata Use port forwarding for verification: ```text kubectl port-forward --kubeconfig --namespace service/net-apiserver 2457:2457 ``` Access at `http://127.0.0.1:2457`. Port 2457 is default; adjust if using a different base port. **Note:** Port forwarding is for verification only. It is a single stream through the cluster's `konnectivity-agent`, and on AKS it drops when node autoprovisioning replaces the node that agent runs on (`error: lost connection to pod`); restart it when that happens. Properly expose Tabsdata before production use. ### Log In ```text tdk login --server --user admin --password tabsdata tdk status ``` ### Post-Installation Cleanup Delete local setup files containing sensitive credentials: ```text rm -rf ~/.tabsdata/instances//init ``` ## Expose Tabsdata Outside Cluster The instance runs but is only accessible via port forwarding. Expose `service/net-apiserver` over HTTPS, for example with the Application Gateway for Containers or an ingress controller and a TLS certificate, before production use. ## Next Steps Run your first data integration per "Run Your First Data Integration" documentation. ## Appendix: Sizing Service and Function Pods Configure sizing in the instance's `configs/tabsdata.yaml`. Service pods cover the instance components (API server, AI agent, embeddings, metadata); function pods cover per-execution pods. On AKS the placement selects nodes by the `role` label: ```yaml pods: service: priority: tabsdata-service resources: cpu: "2" memory: 1Gi disk: 16Gi placement: selectors: role: tabsdata-services tolerations: [] apiserver: resources: cpu: "2" memory: 4Gi disk: 16Gi aiagent: resources: cpu: "2" memory: 8Gi disk: 64Gi function: priority: tabsdata-function resources: cpu: "1" memory: 16Gi disk: 64Gi placement: selectors: role: tabsdata-functions tolerations: - key: workload value: batch effect: NoSchedule ``` Set values based on actual component footprints and confirm node pools can satisfy the largest single pod. A tolerations list replaces the one Tabsdata would apply, so it must be complete. Use `tdkserver config get` and `tdkserver config put` to manage the file. --- # Install Tabsdata on Google GKE > Installs Tabsdata on Google GKE with Cloud SQL for PostgreSQL and GCS: priority classes, node pools, tdkserver init/assess/create/start, GKE database and storage config. Source: https://docs.tabsdata.com/server-admin/gke-install ## Before You Begin This page assumes the Google Cloud resources below already exist: a GKE cluster, a Cloud SQL for PostgreSQL instance, a GCS bucket and a service account that can use the bucket. ## Overview Tabsdata runs on Kubernetes as deployments containing an API server, AI agent, and supporting services within a single namespace. User functions execute as individual Kubernetes jobs that start, run, and exit. Table data resides in Google Cloud Storage (GCS) while metadata lives in Cloud SQL for PostgreSQL. This installation is performed by the Tabsdata administrator. ## Requirements - A Windows (x86), macOS (ARM64), or Linux (x86) client machine - Python 3.12.13 installed - `gcloud` with the `gke-gcloud-auth-plugin` component, and `kubectl` - Google GKE Kubernetes cluster with x86 nodes - Cloud SQL for PostgreSQL instance running version 18, in the same region and VPC as the GKE cluster, with a private IP - GCS bucket, preferably in the same region - A service account with read/write access to the bucket (for example `roles/storage.objectAdmin` on the bucket), and a JSON key for it - Network access between the GKE cluster and the Cloud SQL instance, and between the client and the GKE cluster ### Priority Classes Tabsdata pods name two priority classes, `tabsdata-service` for the instance and `tabsdata-function` for the runs it triggers. They are cluster-wide, so a cluster administrator creates them once; the service class must hold the higher value: ```yaml apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: tabsdata-service value: 1000 globalDefault: false description: Tabsdata instance pods --- apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: tabsdata-function value: 100 globalDefault: false description: Pods of the runs a Tabsdata instance triggers ``` ```text kubectl --kubeconfig apply -f tabsdata-priority-classes.yaml ``` ### Node Pools The GKE configuration schedules pods by node label. Service pods are placed on nodes labeled `role=tabsdata-services`, function pods on nodes labeled `role=tabsdata-functions`. Function pods tolerate the `workload=batch:NoSchedule` taint, so the function pool can be reserved for them: ```text gcloud container node-pools create tabsdata-services \ --cluster --region --project \ --node-labels=role=tabsdata-services gcloud container node-pools create tabsdata-functions \ --cluster --region --project \ --node-labels=role=tabsdata-functions \ --node-taints=workload=batch:NoSchedule ``` Size the machine types for the largest single pod (see the sizing appendix). To place pods differently, edit `placement.selectors` in `tabsdata.yaml` instead. ### Cluster Access Fetch the cluster credentials into your kubeconfig: ```text gcloud components install gke-gcloud-auth-plugin gcloud container clusters get-credentials --region --project kubectl --kubeconfig config get-contexts ``` The context is named `gke___`. Without the auth plugin on `PATH`, every `kubectl` call fails with a credentials error. ## Install Tabsdata Packages From a machine with GKE cluster access, create a Python virtual environment and install: **Using uv:** ```text uv pip install "tabsdata[all]" tabsdata-agent ``` **Using pip:** ```text pip install "tabsdata[all]" tabsdata-agent ``` **Note:** MySQL and PostgreSQL CDC connectors require database drivers that Tabsdata cannot redistribute. See "Installing Third-Party Database Drivers" if needed before proceeding. ## Configuration Values Needed Gather these before starting: | Category | Variable | Description | | --- | --- | --- | | Cloud SQL for PostgreSQL | `` | Private IP of the Cloud SQL instance | | | `` | Port of the Cloud SQL instance (5432) | | | `` | Cloud SQL instance ID | | | `` | Database for Tabsdata installation | | | `` | Connection username (for example `postgres`) | | | `` | Connection password | | | `` | The instance's server CA certificate | | GCS | `` | Bucket for data storage | | | `` | JSON key of a service account with full bucket access | | Google GKE | `` | Project of the cluster | | | `` | Region of the cluster | | | `` | Target cluster | | | `` | Empty reserved namespace | | | `` | Path to kubeconfig file | | LLM Provider | `` / `` | LLM API key if AI integration enabled | | Chosen During Install | `` | Tabsdata instance name | ### Cloud SQL Server CA Cloud SQL signs each instance with a CA of its own, so there is no public bundle to download. Fetch the instance's certificate once: ```text gcloud sql instances describe --project \ --format='value(serverCaCert.cert)' > server-ca.pem ``` Tabsdata connects with `verify-ca`: the chain is verified, the host name is not. A Cloud SQL certificate names the instance, not its IP address, so `verify-full` against an IP would fail. If the instance's CA is rotated, update the certificate in the instance's configuration. ## Configure Tabsdata ### Prepare the Namespace Create the namespace; `tdkserver` never creates one: ```text kubectl --kubeconfig create namespace ``` ### Prepare Configuration ```text tdkserver init --instance --provider gke --kubeconfig --namespace ``` This creates `~/.tabsdata/instances//` with configuration files to edit in `~/.tabsdata/instances//init/input/configs/`. ### Configure Database Edit `dbmanager.database.yaml` with Cloud SQL information. Unlike EKS and AKS, GKE takes the server CA certificate inline, as `ssl_root_cert_pem`: paste the contents of `server-ca.pem` as a block, indented under the key: ```yaml database_server: postgres: host: port: database: username: password: ssl_mode: ${{system:tdk_database_ssl_mode}} ssl_root_cert_pem: | -----BEGIN CERTIFICATE----- MIIDfzCCAmegAwIBAgIBADANBgkqhkiG9w0BAQsFADB3MS0wKwYDVQQuEyQ... ... -----END CERTIFICATE----- pool: admin: min_connections: 1 max_connections: 1 acquire_timeout: 30 max_lifetime: 1800 idle_timeout: 30 test_before_acquire: true ``` Use the instance's **private** IP as ``: it is the pods in the cluster that connect, not the client machine. The `apiserver.database.yaml` file requires no editing; Tabsdata generates and sets credentials automatically for the `_apiserver` and `_aiagent` roles. ### Configure GCS Storage Edit `apiserver.storage.yaml` with GCS details for three mounts (ROOT, BUNDLES, IMAGES). They can share the same bucket with different paths. The service account key is the JSON document itself, as a block: ```yaml storage: mounts: - id: ROOT path: / uri: gs:///root/ options: google_service_account_key: | { "type": "service_account", "project_id": "", "private_key_id": "...", "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n", "client_email": "@.iam.gserviceaccount.com", "client_id": "..." } - id: BUNDLES path: /bundles uri: gs:///bundles/ options: google_service_account_key: | - id: IMAGES path: /images uri: gs:///images/ options: google_service_account_key: | ``` **Warning:** GCS URIs must end with `/`. ### Configure AI Agent Edit `aiagent.yaml` to enable LLM integration with Anthropic or OpenAI API keys: ```yaml port: ${{system:tdk_aiagent_public_port}} allow_writes: true ai_providers: openai: api_key: model: openai/gpt-5.4 anthropic: api_key: model: anthropic/claude-sonnet-4-6 hybrid_search: embedder: fastembed embedding_model: BAAI/bge-small-en-v1.5 device: cpu cache_dir: null offline: false ``` ### Preflight Check Verify configuration completeness: ```text tdkserver config check --instance --init ``` ### Assess the Instance Before creating the instance, check that it can be created: ```text tdkserver assess --instance ``` `assess` changes nothing. It checks everything `create` and the first `start` depend on, and reports each check as passed, failed, warning, skipped (could not run) or waived (does not apply): | Area | What is checked | | --- | --- | | Local registry | The instance is initialized and not yet created; `init` was run by this Tabsdata version and edition; the init folder differs from the templates only in filled placeholders; no mandatory placeholder is left | | Cluster access | The program the kubeconfig runs to authenticate (`gke-gcloud-auth-plugin`) is present; the API server answers; the namespace exists | | Operator rights | Your identity has the verbs an instance needs in the namespace | | Scheduling | The priority classes `tabsdata-service` and `tabsdata-function` exist, the service one higher; a ready node carries the labels the service pods select; a node or pool can serve the function pods; a pool carries the taint the function pods tolerate | | Database | The CA the pods verify against is in place; the database answers and accepts the credentials, over TLS; the role has `CREATEROLE` and `CREATEDB` and does not bypass row-level security; the role owns its database; the database holds no schemas from an earlier instance | | Storage | Each mount carries the options its scheme takes; each bucket answers and accepts the credentials | | Coherence | The package index the function environments use answers; no secret is both supplied and generated; the AI agent has a model it can call | It exits 0 when nothing stands in the way of `create`, and 1 when a check failed or was skipped. Add `--explain` to print, after the report, what every check asks. Fix what it reports, then run it again. **Note:** The database and bucket checks run from the client machine. A Cloud SQL instance only reachable on its private IP shows as a warning there, not a failure; wrong credentials still fail. ### Create Instance ```text tdkserver create --instance ``` Verify state: ```text tdkserver inspect --instance ``` ### Start Instance The first start builds the Python environments and sets up the database, so it takes several minutes. ```text tdkserver start --instance tdkserver status --instance ``` Every service pod should reach `Running`: ```text kubectl --kubeconfig --namespace get pods ``` A pod stuck in `Pending` usually means no node carries the label its `placement.selectors` asks for (see Node Pools), or no node is large enough for its resources. ### Access Tabsdata Use port forwarding for verification: ```text kubectl port-forward --kubeconfig --namespace service/net-apiserver 2457:2457 ``` Access at `http://127.0.0.1:2457`. Port 2457 is default; adjust if using a different base port. **Note:** Port forwarding is for verification only. It is a single stream through the cluster's control plane and drops when the pods restart or the node behind it is replaced. Properly expose Tabsdata before production use. ### Log In ```text tdk login --server --user admin --password tabsdata tdk status ``` ### Post-Installation Cleanup Delete local setup files containing sensitive credentials (the database password, the server CA and the service account key): ```text rm -rf ~/.tabsdata/instances//init rm server-ca.pem ``` ## Expose Tabsdata Outside Cluster The instance runs but is only accessible via port forwarding. Expose `service/net-apiserver` over HTTPS, for example with a GKE Ingress or Gateway and a Google-managed certificate, before production use. ## Next Steps Run your first data integration per "Run Your First Data Integration" documentation. ## Appendix: Sizing Service and Function Pods Configure sizing in the instance's `configs/tabsdata.yaml`. Service pods cover the instance components (API server, AI agent, embeddings, metadata); function pods cover per-execution pods. On GKE the placement selects nodes by the `role` label: ```yaml pods: service: priority: tabsdata-service resources: cpu: "2" memory: 1Gi disk: 16Gi placement: selectors: role: tabsdata-services tolerations: [] apiserver: resources: cpu: "2" memory: 4Gi disk: 16Gi aiagent: resources: cpu: "2" memory: 8Gi disk: 64Gi function: priority: tabsdata-function resources: cpu: "1" memory: 16Gi disk: 64Gi placement: selectors: role: tabsdata-functions tolerations: - key: workload value: batch effect: NoSchedule ``` Set values based on actual component footprints and confirm node pools can satisfy the largest single pod. A tolerations list replaces the one Tabsdata would apply, so it must be complete. Use `tdkserver config get` and `tdkserver config put` to manage the file. --- # Quickstart a Server > Quickstart a Tabsdata server with `tdkserver quickstart`, covering the default kminus, eks, aks and gke providers, the `--config`, `--fill` and `--yes` options, and login. Source: https://docs.tabsdata.com/server-admin/quickstart Before you begin [Install Tabsdata](https://docs.tabsdata.com/server-admin/install) `tdkserver quickstart` bundles [initializing](https://docs.tabsdata.com/server-admin/initialize), [creating](https://docs.tabsdata.com/server-admin/create), and [starting](https://docs.tabsdata.com/server-admin/start) a Tabsdata server into a single command, using a default configuration for compute and storage. ### Start the Tabsdata server `tdkserver quickstart` initializes and deploys a Tabsdata Instance. The command should take around 30 seconds to fully execute. - Anthropic - OpenAI ```bash export ANTHROPIC_KEY= tdkserver quickstart --config anthropic-key=env:ANTHROPIC_KEY ``` ```bash export OPENAI_KEY= tdkserver quickstart --config openai-key=env:OPENAI_KEY ``` note If you plan to use the MySQL or PostgreSQL CDC connectors, they need database drivers that Tabsdata cannot redistribute. After the quickstart completes, add them with `tdkserver venv update`, as described in [Installing Third-Party Database Drivers](https://docs.tabsdata.com/server-admin/third-party-database-drivers): ```bash tdkserver venv update --instance --name fn --requirements tdkserver venv update --instance --name ag --requirements ``` ### Check Tabsdata server status View all your Tabsdata Instances ```bash tdkserver list ``` Output ``` Instances ┌──────────┬──────────────────────────────────────────────────┬──────────┬─────────┬─────────┬─────────┬─────────────────┬───────────────┬───────────────────────┐ │ Name │ Folder │ Provider │ Version │ State │ Status │ API Server Port │ Database Port │ API Server URL │ ├──────────┼──────────────────────────────────────────────────┼──────────┼─────────┼─────────┼─────────┼─────────────────┼───────────────┼───────────────────────┤ │ tabsdata │ ~/.tabsdata/instances/tabsdata │ kminus │ 2.0.2 │ created │ Running │ 2457 │ 2460 │ http://localhost:2457 │ └──────────┴──────────────────────────────────────────────────┴──────────┴─────────┴─────────┴─────────┴─────────────────┴───────────────┴───────────────────────┘ ``` View the status of the Tabsdata Instance you just deployed. ```bash tdkserver status ``` Output ``` Deployments ┌───────────────┬─────────┬───────┬─────┐ │ Name │ Status │ Ready │ Age │ ├───────────────┼─────────┼───────┼─────┤ │ srv-aiagent │ Running │ 1/1 │ 15s │ │ srv-apiserver │ Running │ 1/1 │ 15s │ │ srv-database │ Running │ 1/1 │ 24s │ └───────────────┴─────────┴───────┴─────┘ Pods ┌────────────────────────────────┬─────────┬───────┬─────┐ │ Name │ Status │ Ready │ Age │ ├────────────────────────────────┼─────────┼───────┼─────┤ │ srv-aiagent-7476c88484-4klx8 │ Running │ 2/2 │ 15s │ │ srv-apiserver-56bbbb559d-8cnxl │ Running │ 1/1 │ 15s │ │ srv-database-fd4f8f85-dsxcj │ Running │ 1/1 │ 24s │ └────────────────────────────────┴─────────┴───────┴─────┘ CronJobs ┌──────────────────┬─────────┬─────────────┬─────────────────┬─────┐ │ Name │ Status │ Expression │ Explanation │ Age │ ├──────────────────┼─────────┼─────────────┼─────────────────┼─────┤ │ srv-aiembeddings │ Running │ */5 * * * * │ Every 5 minutes │ 6s │ │ srv-aimetadata │ Running │ */5 * * * * │ Every 5 minutes │ 6s │ └──────────────────┴─────────┴─────────────┴─────────────────┴─────┘ Jobs ┌────────────────┬──────────┬─────────────┬─────┐ │ Name │ Status │ Completions │ Age │ ├────────────────┼──────────┼─────────────┼─────┤ │ ini-inspector │ Complete │ 1/1 │ 16s │ │ ini-venvloader │ Complete │ 1/1 │ 17s │ └────────────────┴──────────┴─────────────┴─────┘ ``` tip `tdkserver` commands usually require an `--instance` flag that specifies the specific instance you want to interact with. If no `--instance` flag is passed, Tabsdata defaults the instance name to `tabsdata` ### Log in ```bash tdk login --server localhost --user admin --password tabsdata ``` Or access the UI at [http://localhost:2457/login](http://localhost:2457/login) ## What is Quickstart `quickstart` bundles initializing, creating, and running a server using a default configuration. For custom deployment, do [Initialize a Server](https://docs.tabsdata.com/server-admin/initialize), [Create a Server](https://docs.tabsdata.com/server-admin/create), and [Start a Server](https://docs.tabsdata.com/server-admin/start). ## Settings and cloud providers `--config` sets a value of the instance configuration, as `ADDRESS=VALUE` or `@FILE`, and can be repeated. A value that starts with `env:` is read from an environment variable. Every setting is checked before the instance folder is created. ```bash tdkserver quickstart --config tabsdata:pods.service.resources.memory=8Gi ``` A custom Python package index is set the same way: ```bash tdkserver quickstart --config tabsdata:pods.global.envs.UV_INDEX=http://localhost:8000/+simple ``` `--provider` takes `kminus`, `eks`, `aks` or `gke`. Without it, `quickstart` uses the provider set for the session with `tdkserver set`, and falls back to `kminus`. A cloud provider needs one of `--kubeconfig`, `--context` or `--current`. `--fill` opens the provider's configuration stub in an editor in the terminal before anything is created. `Ctrl+Q` saves and continues, `Ctrl+S` saves, and `Esc` cancels. The filled stub is kept in `/init/fill/stub.yaml`. ```bash tdkserver quickstart --provider gke --current --fill ``` `--fill` needs an interactive terminal. For an unattended install, copy the stubs into the Tabsdata home folder with `tdkserver stubs`, fill one in, and pass it with `--config @FILE`: ```bash tdkserver stubs ``` ```bash tdkserver quickstart --provider gke --context my-gke-context --config @settings.yaml --yes ``` If `quickstart` fails on a new instance, the partial instance is moved to trash. A `kminus` instance is stopped first, and a cloud instance has its cluster resources deleted first. `--anthropic-key`, `--openai-key` and `--uv-index` still work, with a deprecation warning, and are converted into the matching `--config` setting. `--reqs-td`, `--reqs-fn` and `--reqs-ag` are no longer accepted; add drivers with `tdkserver venv update` instead. See [`tdkserver quickstart`](https://docs.tabsdata.com/cli/tdkserver/quickstart) for every option. --- # Initialize a Server > Covers `tdkserver init`, which writes an editable config folder under ~/.tabsdata/instances/ with default settings, using options like --instance and --provider, before creating and starting a server. Source: https://docs.tabsdata.com/server-admin/initialize Before you begin [Install Tabsdata](https://docs.tabsdata.com/server-admin/install) `tdkserver init` produces an editable config folder for a Tabsdata instance within `~/.tabsdata`, pre-filled with default values. ### Initialize an instance ```bash tdkserver init --instance myinstance --provider kminus ``` Output ``` About to initialize tabsdata instance: - instance.....: myinstance - provider.....: kminus - log level....: debug (default) Proceed with the initialization? [y/yes/n/no]: y ▶ Creating tabsdata instance folder ✔ Created tabsdata instance folder at ~/.tabsdata/instances/myinstance (0s) ▶ Initializing tabsdata templates ✔ Initialized tabsdata templates in ~/.tabsdata/instances/myinstance/init/input (0s) ▶ Writing tabsdata instance settings ✔ Wrote tabsdata instance settings (0s) Initialized tabsdata instance 'myinstance'. - state........: initialized - version......: 2.0.0 - log level....: debug - provider.....: kminus - instance name: myinstance - kubeconfig...: ~/.tabsdata/instances/myinstance/cluster/config/kubeconfig.yaml - namespace....: default ╭─────────────────────────────────────────────────────────────────────╮ │ Inspect and edit the config in the instance's init folder, then run │ │ 'tdkserver create --instance myinstance' to create the instance. │ ╰─────────────────────────────────────────────────────────────────────╯ ``` The config folder is written to `~/.tabsdata/instances/myinstance/`. The default configuration is what is used by `tdkserver quickstart` which bundles initializing, creating, and starting a server ## What to do next 1. Edit the config files the command produced, if you need to change anything. 2. [Create a Server](https://docs.tabsdata.com/server-admin/create) using the config 3. [Start a Server](https://docs.tabsdata.com/server-admin/start). To accept the defaults and get all three in one command, use [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) instead. Deploying on AWS EKS starts here too, with values for the database, storage and cluster. See [Install Tabsdata on AWS EKS](https://docs.tabsdata.com/server-admin/eks-install). --- # Create a Server > Covers `tdkserver create --instance`, which builds a server instance from the init config folder, setting the LLM API key, port assignment with `--port`, and checking with `tdkserver list`. Source: https://docs.tabsdata.com/server-admin/create Before you begin [Install Tabsdata](https://docs.tabsdata.com/server-admin/install) [Initialize a Server](https://docs.tabsdata.com/server-admin/initialize) After [initializing](https://docs.tabsdata.com/server-admin/initialize) a server, `tdkserver create` builds an instance from the config folder that [`tdkserver init`](https://docs.tabsdata.com/server-admin/initialize) produced. Step 1Set the LLM API key `tdkserver create` builds the aiagent pod, so the LLM key has to be in the environment before running it, not before [`tdkserver init`](https://docs.tabsdata.com/server-admin/initialize): - Anthropic - OpenAI ```bash export ANTHROPIC_KEY="sk-ant-..." ``` ```bash export OPENAI_KEY="sk-..." ``` Step 2Create an instance ```bash tdkserver create --instance myinstance ``` A server's name is unique and is used internally for identification. The name provided to `--instance` is used to find the config folder created during initialization. Step 3Check the instance was created ```bash tdkserver list ``` Output ``` Instances ┌──────────┬──────────────────────────────────────────────────┬──────────┬─────────┬─────────┬─────────┬─────────────────┬───────────────┬───────────────────────┐ │ Name │ Folder │ Provider │ Version │ State │ Status │ API Server Port │ Database Port │ API Server URL │ ├──────────┼──────────────────────────────────────────────────┼──────────┼─────────┼─────────┼─────────┼─────────────────┼───────────────┼───────────────────────┤ │ tabsdata │ ~/.tabsdata/instances/tabsdata │ kminus │ 2.0.2 │ created │ Stopped │ 2457 │ 2460 │ http://localhost:2457 │ └──────────┴──────────────────────────────────────────────────┴──────────┴─────────┴─────────┴─────────┴─────────────────┴───────────────┴───────────────────────┘ ``` The instance appears with its name and state. It is not running yet. ## Ports Every service in the instance takes a port, assigned sequentially from a single base value. `--port` sets that base, defaulting to `2457`: ```bash tdkserver create --instance myinstance --port 3000 ``` Two instances on one machine need base ports far enough apart that their ranges do not overlap. ## What to do next [Start a Server](https://docs.tabsdata.com/server-admin/start). The instance keeps its configuration across stops and starts, so creating it is a one-time step. See [`tdkserver create`](https://docs.tabsdata.com/cli/tdkserver/create) for the full option list. --- # Start a Server > Describes starting a Tabsdata server instance with the `tdkserver start` command, including the `--instance` option, after an instance has been built by `tdkserver create`. Source: https://docs.tabsdata.com/server-admin/start Before you begin [Install Tabsdata](https://docs.tabsdata.com/server-admin/install) [Initialize a Server](https://docs.tabsdata.com/server-admin/initialize) [Create a Server](https://docs.tabsdata.com/server-admin/create) `tdkserver start` brings up the services of an instance that [`tdkserver create`](https://docs.tabsdata.com/server-admin/create) already built. Until the instance is running, `tdk` has nothing to connect to. Step 1Start an instance ```bash tdkserver start --instance myinstance ``` --- # Stop a Server > Explains how to stop a running Tabsdata server with `tdkserver stop` while keeping its registration, configuration, tables and functions, plus checking runs with `tdk fnrun list` and restarting after config changes. Source: https://docs.tabsdata.com/server-admin/stop Before you begin [Install Tabsdata](https://docs.tabsdata.com/server-admin/install) [Initialize a Server](https://docs.tabsdata.com/server-admin/initialize) [Create a Server](https://docs.tabsdata.com/server-admin/create) [Start a Server](https://docs.tabsdata.com/server-admin/start) `tdkserver stop` shuts down a running instance's services and leaves everything else in place. The instance stays registered, its configuration is untouched, and the tables and function code it holds survive, so a later [start](https://docs.tabsdata.com/server-admin/start) picks up where the stop left off. Step 1Stop the instance ```bash tdkserver stop --instance myinstance ``` The command prints the values it resolved and waits for confirmation. Pass `--yes` to skip the prompt. Step 2Confirm it stopped ```bash tdkserver status --instance myinstance ``` ## Running functions Stopping the server stops its workers with it, so a function run in flight does not finish. Nothing partial is committed: a run only commits when every function in its execution plan succeeds, so an interrupted run leaves the tables at the version they held beforehand. Check for active work before stopping: ```bash tdk fnrun list --status running ``` ## Picking up a configuration change A configuration edit reaches a running instance through a restart rather than a stop: ```bash tdkserver restart --instance myinstance ``` ## What to do next [Start a Server](https://docs.tabsdata.com/server-admin/start) to bring the same instance back up, or [Delete a Server](https://docs.tabsdata.com/server-admin/delete) to discard it. See [`tdkserver stop`](https://docs.tabsdata.com/cli/tdkserver/stop) for the full option list. --- # Delete a Server > Deletes a local kminus server instance with `tdkserver trash`, covering the stop and confirmation steps, the irreversible data loss, and the EKS exception. Source: https://docs.tabsdata.com/server-admin/delete Before you begin [Install Tabsdata](https://docs.tabsdata.com/server-admin/install) [Initialize a Server](https://docs.tabsdata.com/server-admin/initialize) [Create a Server](https://docs.tabsdata.com/server-admin/create) [Start a Server](https://docs.tabsdata.com/server-admin/start) [Stop a Server](https://docs.tabsdata.com/server-admin/stop) warning Trashing an instance discards everything it holds: the projects inside it, the tables and every version of their data, the registered function code, and the execution history. This cannot be reversed. `tdkserver trash` removes a local kminus instance and its data. The command is `trash` rather than `delete`, and it applies to local kminus instances. An EKS deployment is torn down through its own uninstall path instead. Step 1Stop the instance ```bash tdkserver stop --instance myinstance ``` Step 2Trash the instance ```bash tdkserver trash --instance myinstance ``` The command prints a summary of what it is about to discard and waits for confirmation. Read the instance name in that summary before agreeing to it, since there is nothing to recover afterward. Step 3Confirm the instance is gone ```bash tdkserver list ``` The instance no longer appears, and its name is free for reuse. ## Keeping the data Nothing in the instance survives the trash, so anything worth keeping has to leave the server first. A subscriber writes a table out to an external system, which is the supported way to get data off an instance. See [Subscribers](https://docs.tabsdata.com/guide/functions/subscribers). ## What to do next [Quickstart a Server](https://docs.tabsdata.com/server-admin/quickstart) for a fresh instance with default configuration, or [Initialize a Server](https://docs.tabsdata.com/server-admin/initialize) to set the configuration yourself. See [`tdkserver trash`](https://docs.tabsdata.com/cli/tdkserver/trash) for the full option list.