Collections

A groups related and together. Depending on the type of function, collections serve different purposes in your workflow.

For and , which move data into and out of external systems, the collection holds the those functions use to access the external system. This keeps connection configuration and sensitive credentials out of the function code and allows multiple functions in the same collection to use the same connection.

do not interact with external systems, so do not need a connection. Instead, their main purpose is to group related functions and tables and define a . The transaction boundary determines which table changes are committed or rolled back together during an .

Transaction boundaries apply to publisher and as well.

Collection types

Tabsdata has for , for , and for .

s3_srcSOURCES

Tabsdata›Collections›s3_src

Connection

The external system connection this collection uses to read data from.

1id: 06fvnth9dlrr3f3svp62ihn6mk
2connection:
3kind: connectionDef
4apiVersion: '1.0'
5type: tabsdatak.conn.s3:S3SrcConn
6spec:
7  bucket: str:tabsdata-benchmarking
8  region: str:us-east-2
9  credentials:
10    kind: awsAccessSecretKeyCredentials
11    apiVersion: '1.0'
12    type: tabsdatak.conn.common.types:AwsAccessSecretKey
13    spec:
14      access_key_id: ':secret:'
15      secret_access_key: ':secret:'
16  base_path: str:/
17name: s3_src
18secret_names:
19- SECRET_CONN__CREDENTIALS_SPEC_ACCESS_KEY_ID
20- SECRET_CONN__CREDENTIALS_SPEC_SECRET_ACCESS_KEY
21var_names: []

Source collections

A contains a single , the that use it, and the those publishers produce. All publishers in the use the same connection.

The publisher defines what data to read from the external system and what tables to write into Tabsdata. The connection provides the configuration and credentials needed to access that system. You define the connection when creating the collection. Publishers get that connection from their collection, without specifying it in their code.

For example, the s3_src collection above has one S3 connection and two publishers. publish_orders reads order files and publish_customers reads customer files. Both use the same bucket and credentials from the collection's connection.

note

In Tabsdata 1.x, you specified the connection in the publisher. It now comes from the collection hosting the publisher.

Tabsdata server

collection · world

Database

PostgreSQL

PostgreSQL · external

Reachable

Connection

conn-postgres.yaml

Publisher

read_world

Table

country

//world

No versions yet

Table

city

//world

No versions yet

Transformer collections

A contains and the they produce. It does not have an external , since transformers read and write data within Tabsdata.

A can contain multiple transformers, and those transformers can read input tables from other collections. Their output tables belong to the collection where they are registered.

Grouping transformers in the same collection determines which table updates are committed together. When an runs multiple transformers in that collection, their table changes become available at the same time.

collection · world

collection · insights · data product

Table

country

//world

Committed

Table

city

//world

Committed

Transformer

find_big_cities

Table

big_cities

//insights

No versions yet

Destination collections

A contains a single and the that use it. All subscribers in the use the same connection to write to the external system.

The subscriber defines which Tabsdata to read and what data to write to the destination. As with , the connection comes from the collection hosting the .

Tabsdata server

collection · warehouse

Table

vendors

//sales

Committed

Table

items

//sales

Committed

Connection

conn-mysql-dest.yaml

Subscriber

write_sales

Database

MySQL

MySQL · external

Not yet synced

Collections and transactions

Every acts as a for its . When an changes multiple tables in the same collection, those changes are committed together, meaning they become available at the same time.

An execution plan can span multiple collections, each with its own transaction.

See and for more information.