Collections

Collections store Functions, Tables, and Connections.

Collection Purpose, Behaviour and Architecture varies depending on whether they store Publisher, Transformers, or Subscribers

Transformer Collections

These collections store Transformer functions, which work exclusively inside the Tabsdata Instance and do not make any calls outside the Server environment

Transformer Collections can store one or more Tabsdata Functions and do not require a Connection to be bound to it.

The main purpose of Transformer collections is that they define commit behavior during Execution Plans. For more information, view [Pristine State] and [Transactions]

tdk collection create --name dataprod --group default

collection · world

collection · insights · data product

Table

country

//world

Committed

Table

city

//world

Committed

Transformer

find_big_cities

Table

big_cities

//insights

No versions yet

Publisher Collections

Publisher Collections are specialized for handling Publisher Functions that have to connect to and ingest data from external systems into Tabsdata.

While Publishers define things like the queries to run, what Tabsdata Tables to write the data into, and transformation logic, Sensitive Data like passwords are decoupled into a Connection resource.

Publisher Collections bind and enable Functions to access and use credentials stored within this Connection resource

Unlike Transformer Collections, Publisher Collections can only hold one Publisher at a time and must also have a Connection bound to it.

tdk collection create --name input --group sources --conn-file conn-input.yaml

Tabsdata server

collection · world

Database

PostgreSQL

PostgreSQL · external

Reachable

Connection

conn-postgres.yaml

Publisher

read_world

Table

country

//world

No versions yet

Table

city

//world

No versions yet

Subscriber Collections

Subscriber Collections are specialized for handling Subscriber Functions that have to connect to write data from Tabsdata into external systems.

While Subscribers define things like what Tabsdata Tables to read from and Trigger Behavior, Sensitive Data like passwords are decoupled into a Connection resource.

Subscriber Collections bind Connections and Functions together and enable Functions to access and use credentials stored within this Connection resource to complete it's write operations.

Unlike Transformer Collections, Subscriber Collections can only hold one Subscriber at a time and must also have a Connection bound to it.

tdk collection create --name output   --group destinations --conn-file conn-output.yaml

Tabsdata server

collection · warehouse

Table

vendors

//sales

Committed

Table

items

//sales

Committed

Connection

conn-mysql-dest.yaml

Subscriber

write_sales

Database

MySQL

MySQL · external

Not yet synced