Functions

Functions are Tabsdata's unit of work. Each Function performs a single discrete step of moving data between a source and a destination.

Sources and destinations can be either external systems, such as databases, data warehouses, and object stores, or internal Tabsdata Tables.

Publishers

External SystemTabsdata Table

Reads data from an external system and writes the data to one or more Tabsdata Tables.

Transformers

Tabsdata TableTabsdata Table

Reads data from one or more Tabsdata Tables and writes data into one or more Tabsdata Tables

Subscribers

Tabsdata TableExternal System

Reads data from one or more Tabsdata Tables and writes data into an external system.

Defining Inputs and Outputs

A Function decorator defines where data is read from and where the results are written.

For Publishers and Subscribers, the source or destination is defined through a Connector, a python class that contains the system-specific logic needed to communicate with that type of external system.

The decorator also maps inputs to the Function body's arguments and maps returned values to output Tables.

Configuring Triggers

Triggers cause a function to execute. By default, transformers and subscribers are triggered automatically whenever one of their input tables commits a new version. Publishers have no input tables, so they have no default trigger, give them one explicitly with trigger_by:

A publisher can also run on a schedule instead of waiting on a table commit. Pick the mode that matches when the read should happen.

Runs on a schedule

Use CronTrigger to run the publisher on a fixed schedule instead of waiting on another table to commit.

read_world.pyCron

from tabsdatak.api import publisher, TableFrameSpec, CronTrigger
from tabsdatak.conn.postgres import PostgresSrc


@publisher(
    source=PostgresSrc(
        queries=[
            "SELECT * FROM country",
            "SELECT * FROM city",
        ],
    ),
    output_tables=["country", "city"],
    trigger_by=CronTrigger("0 * * * *"),
)
def read_world(country: TableFrameSpec, city: TableFrameSpec) -> tuple[TableFrameSpec, TableFrameSpec]:
    return (country, city)

For more information, see Working with Triggers.

Registering a Function

After you build the python file for your function, you must register into a collection in the Tabsdata server using the tdk fn register CLI command.

To register a function, use the following command in a CLI:

$ tdk fn register --coll <collection_name> --path <path_to_python_file>::<function_name> --update

Where:

• <collection_name> is the name of the collection in the Tabsdata server to register the function with.
• <path_to_python_file> is the path to the Python file containing the function. It can be absolute system path or relative to the CLI directory from which the command is run.
• <function_name> is the name of the function as defined in the Python file.
• --update flag updates the function if the function with the same name already exists.

Execute a Function

Once executed, a function takes the input data, processes it as specified, and writes the output as configured.

To execute a function manually, use the following command in a CLI:

$ tdk fn trigger --coll <collection_name> --name <function_name>

Where:

• <collection_name> is the name of the collection in the Tabsdata server that the function is registered with.
• <function_name> is the name of the function to be executed. This name is defined when the function is registered.

The function execution happens by initiating a trigger. A trigger can be manual or automated. Manual triggers are initiated by a CLI command from a user. Automated triggers are initiated by the creation of a new table commit on a trigger table associated with the function. Regardless of the trigger defined in a function, you can always manually execute a function from CLI.

For more information on triggers, see Working with Triggers.