Skip to main content
GuideServerConfigure and deploy Tabsdata servers on your machine.TutorialsConfigure data integration workflows within a running Tabsdata server.Advanced TutorialsBuild end-to-end workflows between two specific systems.API ReferenceCLI ReferenceRelease Notes
Version: 2.0.2

Functions

A Tabsdata Function is Tabsdata's operational unit of work and defines a single step that reads data from or writes data to a Tabsdata Table.

Tabsdata Functions build on the familiar model of a regular Python function, where data comes in through a parameter, the function performs work on that parameter, and the result is returned through the return statement.

Tabsdata extends this model to data integration by abstracting data sources and destinations into the simple parameters and return statement of a Python function.

Users define their inputs, which can be a SQL query against a database, a file in object storage, or a Tabsdata Table. They then define where they want the output data to be written, whether that is a database table, a file in object storage, or another Tabsdata Table.

Tabsdata handles retrieving, converting, and moving the data between those systems. While the data is being passed through the Function, Tabsdata materializes it as a DataFrame object that users can transform through an expression-based API.

This allows users to work with tabular data independently of where it is physically stored or how it needs to be moved between systems.

Publisher: Database to Python to Tabsdata Table.@td.publisher(input=database,output=['orders'],)def clean_orders(orders):cleaned = ...return cleanedDatabaseTabsdata Tableorders

Function types​

There are three types of Tabsdata functions:

  1. publishers: read data from external systems and publish data into Tabsdata Tables
  2. transformers: read data from Tabsdata Tables and write data into Tabsdata Tables
  3. subscribers: read data from Tabsdata Tables and subscribe data into Tabsdata Tables
Publishers
External SystemTabsdata Table

Reads an external system and writes Tabsdata Tables.

Transformers
Tabsdata TableTabsdata Table

Reads Tabsdata Tables and writes new Tabsdata Tables.

Subscribers
Tabsdata TableExternal System

Reads Tabsdata Tables and writes an external system.

Inputs and outputs​

The decorator maps declared inputs to the Python callable's parameters. It also maps returned values to output Tables or to destination slots in an external system. The number and order of parameters and returned values must match the corresponding declarations.

Publishers and Subscribers use a Connector for the system-specific read or write operation. The Collection's Connection supplies the configuration and credentials used by that Connector. Transformers work entirely with Tabsdata Tables and do not use an external Connection.

Output Tables belong to the Collection where the Function is registered. Input Table references can identify Tables in other Collections within the same Project, allowing Functions to connect into a larger workflow.

Triggers​

A Trigger determines when a Function starts. A Function without trigger_by runs only when manually triggered. Set trigger_by to a schedule or one or more Table references when the Function should run automatically.

Registration​

Registration sends the Function definition, source code, and required bundle metadata to the Server. A Function name must be unique within its Collection.

Register a decorated callable from a Python file with:

tdk fn register --coll <collection_name> --path <path_to_python_file>::<function_name>

Pass --update when the same command should update an existing Function with that name. Registration records the definition; it does not execute the Function.

See Register a Function for the full procedure.

Execution​

A manual Trigger can start any registered Function:

tdk fn trigger --coll <collection_name> --name <function_name>

The Server evaluates the Function's dependencies and creates an Execution Plan. Each Function included in that plan gets a Function Run, and a Worker executes each attempt. A successful run that produces Tables creates new Table Versions; the Server keeps the registered Function definition after the Worker exits.

Function code does not run in the CLI process that sends the command. The CLI requests the Trigger, while the Server coordinates and records the execution.