Groups

Groups are an organizational layer that separate workflow resources across two dimensions: where they move data to and form, and the type of transformations they perform on data.

Separating Extract, Transform, and Load

Not all parts of a data integration workflow serve the same purpose:

• some operations ingest data from internal systems
• some operations clean and transform data in-transit
• some operations load data into external systems

Groups organize workflow steps into these three buckets.

When a project is created in Tabsdata, it automatically comes with three groups out-of-the-box, each group responsible for handling a single portion of ETL

1. The sources group exclusively stores Publisher Functions and handles data ingestion into Tabsdata.
2. The default group exclusively stores Transformer Functions and handles internal data transformation within Tabsdata.
3. The destinations group exclusively stores Subscriber Functions and handles data load out of Tabsdata.

Groups separate workflow resources based on type of data movement

Database

postgres

Tabsdata Server

sources

Publisher

load_users

Table

users

default

Transformer

rank_users

Table

top_users

destinations

Subscriber

sync_users

Warehouse

snowflake

By creating a layer of separation between extract, transform, and load, groups enable abstraction of source and destination data into a standardized tabular format.

Separating Bronze, Silver, and Gold

When transforming data, not every operation has the same purpose. For example, Tabsdata uses the Medallion Architecture for Data Integration, meaning data transformation is divided into three layers:

• Bronze: Layer that ingests raw source data from external systems with the goal of having as close of a representation to data in the external system
• Silver: Layer that cleans, dedupes, normalizes, and joins related entities
• Gold: Layer that applies granular, business-specific transformations and creates "consumer-ready" tables.

The source and destination groups are unique, hardcoded resources that require all ingest and load operations to exist within them. However, users can create as many Transformer groups as needed for their workflows.

A medallion architecture can be represented by five groups within Tabsdata: two built-in groups (sources, destinations) and three transformer groups (bronze, silver, gold).

External systems

GroupsourcesPublishers

Ingest from external systems.

Built in

GroupbronzeTransformers

Raw data, kept as close to the external system as possible.

User created

GroupsilverTransformers

Cleaned, deduped, normalized, and joined.

User created

GroupgoldTransformers

Business-specific transformations, consumer-ready tables.

User created

GroupdestinationsSubscribers

Write out to external systems.

Built in

External systems

Every arrow between two groups is a Tabsdata table, so the data stays inside the server. Only the top and bottom arrows cross the boundary: the top one reads from an external system, the bottom one writes back out to one.

Watching data move between groups

The Projects tab in the Tabsdata UI visualizes data flow across the groups within your project, providing insights into how many rows and MB of data pass through each group and how many tables that data spans across.

The diagram above shows a workflow that reads 200 MB of data from AWS S3 into two bronze tables. The data is then cleaned and processed through silver and gold tables before 30 MB of derived data is written to Snowflake.