Connections

A Connection is a resource that stores the configuration and credentials needed to access an external system.

Although Functions like Publishers and Subscribers need to connect to external systems to read and write data, Functions themselves do not contain sensitive credentials.

Credentials are instead defined in Connections, which are attached to Collections as .yaml files and provide Functions within the Collection with access to the required credentials during execution.

Connection Structure

Connections are defined using .yaml files. The fields in a Connection depend on the external system and Connection type.

For example, an S3 Connection stores the bucket, region, and credentials needed to access files in an S3 bucket.

conn-s3.yaml

kind: connectionDef
apiVersion: '1.0'
type: tabsdatak.conn.s3:S3SrcConn
spec:
  bucket: 'acme-ingest'
  region: 'us-east-1'
  credentials:
  kind: awsAccessSecretKeyCredentials
  apiVersion: '1.0'
  type: tabsdatak.conn.common.types:AwsAccessSecretKey
  spec:
    access_key_id: '$secret:AWS_ACCESS_KEY_ID'
    secret_access_key: '$secret:AWS_SECRET_ACCESS_KEY'

• required
• required
• required
• optional

Connections and Collections

Connections are bound to Publisher and Subscriber Collections.

A Publisher Collection binds a Publisher to an input Connection. The Publisher defines what data to read from the external system and what Tables to create in Tabsdata, while the Connection provides the configuration and credentials needed to access that system during the Publisher's execution.

A Subscriber Collection binds a Subscriber to an output Connection. The Subscriber defines which Tabsdata Tables to read and what data to write, while the Connection provides the configuration and credentials needed to access the destination system during the Subscriber's execution.

A Publisher or Subscriber Collection has exactly one Connection, and it is required. It can hold any number of Functions, and they all reach their external system through that one Connection.

Credential Security

When configuring a connection , a credential can be passed directly as secret:<value>, or as a reference to an environment variable, written as $secret:<VARIABLE NAME>.

Regardless of the method used, when the connection file is registered into Tabsdata, Tabsdata dehydrates the document by extracting any secrets into a separate secret store and storing a reference to the secret in the file.

note

For security, once credentials are stored in Tabsdata, they can only be accessed by a Tabsdata Function at runtime.