Connections

A Connection describes an external system and how to authenticate with it. It stores the endpoint, credentials, and configuration needed to access databases, cloud storage, message queues, and other external systems.

Connections are attached to Collections as .yaml files. Every Function within the Collection shares the same Connection.

Publisher Functions use the Collection's Connection to ingest data from external systems, while Subscriber Functions use it to write data back to external systems.

How to Create a Connection

Find the Connection Type

tdk connection types

Type names read <system>-<access>-<direction>, so the direction is part of the name: s3-file-in is an S3 source, s3-file-out an S3 destination, mysql-sql-in a MySQL source. This command needs no server.

Generate a Blank Connection Template

tdk connection template --type s3-file-in --file conn-s3.yaml

Fill in Connection Template values

conn-s3.yaml

kind: connectionDef
apiVersion: '1.0'
type: tabsdatak.conn.s3:S3SrcConn
spec:
  bucket: 'acme-ingest'
  region: 'us-east-1'
  credentials:
  kind: awsAccessSecretKeyCredentials
  apiVersion: '1.0'
  type: tabsdatak.conn.common.types:AwsAccessSecretKey
  spec:
    access_key_id: '$secret:AWS_ACCESS_KEY_ID'
    secret_access_key: '$secret:AWS_SECRET_ACCESS_KEY'

• required
• required
• required
• optional

warning

For security, credentials must be stored as environmental variables in your terminal and have the variable name passed into the connection template.

Check the File

tdk connection validate --file conn-s3.yaml

This resolves every $secret: reference, reports which environment variables it read, and checks the document against the connector's schema. Still no server involved: the next step is the first one that talks to one.

Bind the .yaml file to a Collection

Create a New Collection

tdk collection create --name input --group sources --conn-file conn-s3.yaml

Edit an Existing Collection

tdk collection update --name input --group sources --conn-file conn-s3.yaml

warning

A connection's credentials can be replaced at any time. Its type cannot. Switching connectors means deleting the collection and creating it again.

The connection's direction already implies the group, so --group is checked rather than obeyed: an s3-file-in connection makes a sources collection, and passing --group destinations is an error. A collection with no connection takes no --conn-file at all.

Build a Tabsdata Function

The function names the files to read and the tables to write. It carries no bucket, no region and no keys, because those come from the connection you just bound to the collection.

read_files.py

from tabsdatak.api import publisher, TableFrameSpec, TableFramesSpec
from tabsdatak.conn.s3 import S3Src

@publisher(
  source=S3Src(
        paths=[
            "departments.csv",
            "employees.csv",
      ],
    ),
  output_tables=["departments", "employees"],
)
def read_files(
  departments: TableFramesSpec,
  employees: TableFramesSpec,
) -> tuple[TableFrameSpec, TableFrameSpec]:
  return (departments[0], employees[0])

• required
• optional
• optional
• optional

Register the Function to the Collection

tdk fn register --coll input --path read_files.py::read_files

Values and secrets

Each value under spec carries a small prefix that says what it is:

Form	Meaning	Where the value ends up
str:<value>	a plain value	stays in the stored connection
secret:<value>	a literal secret value	extracted into the secret store
$secret:<ENV>	a secret read from environment variable ENV	extracted into the secret store

Passwords, access keys, and tokens are secret-only fields, so they take secret:<value> or $secret:<ENV> and nothing else. A URI, a bucket, or a region is a string-or-secret field and takes all three forms, where str: keeps the value in the connection file and the other two move it into the secret store instead.

When and where $secret:<ENV> is resolved. It is resolved once, at the moment you run tdk collection create --conn-file or tdk collection update --conn-file. The value is read from the environment variable ENV in the shell where you run that tdk command: your own machine, not the server. So ENV has to be exported in that shell at that moment.

The resolved value is then stored as a secret like any other, and the environment variable is not consulted again afterwards, neither on later runs of the function nor on the server.

For example, export the variables in the same shell, then create the collection:

export AWS_ACCESS_KEY_ID="AKIAEXAMPLE"
export AWS_SECRET_ACCESS_KEY="s3cr3t"

tdk collection create --name input --group sources --conn-file conn-s3.yaml

with the connection file referencing them:

      access_key_id: '$secret:AWS_ACCESS_KEY_ID'
      secret_access_key: '$secret:AWS_SECRET_ACCESS_KEY'

The literal keys never appear in conn-s3.yaml. The names AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are not stored anywhere either. Only their resolved values are stored, as secrets.

tip

A few fields take a plain value written directly, with no str: prefix (MySQL's uri is one). The connector's own guide shows the exact form; if tdk connection validate rejects a str:-prefixed value, write it plain instead.

How secrets are handled

When you create or update a collection with a connection file, the client:

1. resolves every $secret:<ENV> reference from the environment;
2. validates the document against the connector's schema;
3. dehydrates it: every secret value is stripped out into a separate secret payload, and the stored document keeps only a :secret: marker in its place.

The server then stores the secret values separately from the connection document. The stored document never contains a secret value, and reads never return one.

Viewing a connection

tdk collection connection --name input

Shows the connection's type and the names of its secrets and variables, never the values. That also means there is no way to pull a connection down, edit it, and push it back, so keep your local copy of the YAML or generate a fresh template and refill it.

Updating a connection and rotating secrets

Replace a collection's connection, whether for a new address or rotated credentials, by passing a new file to update:

tdk collection update --name input --conn-file conn-s3.yaml

The new file is validated and dehydrated exactly as on create, and its secrets replace the stored ones. Renaming a source or destination collection renames its connection along with it, since the two share a name.

Command reference

Command	What it does	Needs a server
tdk connection types	List the connection types installed connectors provide.	no
tdk connection template --type T --file F	Write a fill-in template for type T.	no
tdk connection validate --file F	Validate a connection document.	no
tdk collection create --group G --conn-file F	Create a collection with a connection.	yes
tdk collection update --conn-file F	Replace a collection's connection / rotate secrets.	yes
tdk collection connection --name N	Show a collection's connection (names only).	yes

Every command has -h for its full options. For the rest of the tdk surface, see the CLI Guide.