Skip to main content
GuideTutorialsAdvanced TutorialsAPI ReferenceCLI ReferenceRelease Notes
Version: 2.0.0

Groups

Groups store resources such as Collections, Functions, and Tables. Groups help semantically organize resources inside your projects

Separating Extract, Transform, and Load​

Not all parts of your workflow serve the same purpose:

  • Publishers write data into your Tabsdata Server
  • Subscribers write data out of your Tabsdata Server
  • Transformers move and transform data inside a Tabsdata Server

Groups help separate these three types of operations

When a project is created in Tabsdata, it comes with three groups out-of-the-box:

  1. sources that exclusively stores Publisher Functions.
  2. destinations that exclusively stores Subscriber Functions.
  3. default that stores Transformer Functions.
Each group holds the functions that do one kind of work
Database
postgres
Tabsdata Server
sources
Publisher
load_users
Table
users
default
Transformer
rank_users
Table
top_users
destinations
Subscriber
sync_users
Warehouse
snowflake

By creating a layer of separation between inter-server operations (Extract, Load) and intra-server (Transform) operations, groups help abstract your source data into a standardized tabular format for easy data transformation.

Building your Medallion​

Factoring out Ingest and Load, the Transformation aspect of ETL is often differentiated by semantic purpose.

Tabsdata uses the Medallion Architecture for Data Integration, meaning data transformation is divided into three layers:

  • Bronze: Layer that ingests raw source data from external systems with the goal of having as close of a representation to data in the external system
  • Silver: Layer that cleans, dedupes, normalizes, and joins related entities
  • Gold: Layer that applies granular, business-specific transformations and creates "consumer-ready" tables.

However, users can create as many Transformer groups as needed for their workflows.

Put together, a medallion architecture can be represented by five groups: two built-in groups (sources, destinations) and three transformer groups (bronze, silver, gold).

External systems
GroupsourcesPublishers

Ingest from external systems.

Built in
GroupbronzeTransformers

Raw data, kept as close to the external system as possible.

User created
GroupsilverTransformers

Cleaned, deduped, normalized, and joined.

User created
GroupgoldTransformers

Business-specific transformations, consumer-ready tables.

User created
GroupdestinationsSubscribers

Write out to external systems.

Built in
External systems
Every arrow between two groups is a Tabsdata table, so the data stays inside the server. Only the top and bottom arrows cross the boundary: the top one reads from an external system, the bottom one writes back out to one.

How to Create a Group​

tdk group create --name bronze --proj myproject