Postgres

The Postgres connector lets Tabsdata run queries against Postgres and write tables into Postgres.

The PostgresSrc connector can be used by a publisher function to read data from Postgres into a Tabsdata table.

The PostgresDest connector can be used by a subscriber function to write data from a Tabsdata table into Postgres.

Publisher

The PostgresSrc connector can be used by a publisher function to read data from Postgres into a Tabsdata table. See the full publisher walkthrough

Connection

Postgres publishers use PostgresSrcConn to define the database connection and credentials used by the publisher.

conn-postgres.yaml

kind: connectionDef
apiVersion: '1.0'
type: tabsdatak.conn.postgres:PostgresSrcConn
spec:
  uri: 'postgres://db.internal:5432/world'
  credentials:
  kind: userPasswordCredentials
  apiVersion: '1.0'
  type: tabsdatak.conn.common.types:UserPassword
  spec:
    user: 'secret:tabsdata_reader'
    password: 'secret:example-password'

• required
• required
• optional
• optional

Connection Config Parameters

uri

The SQLAlchemy-style connection URI: postgres://host:port/database. Written as a plain value here, but accepts secret: too.

credentials

The database user and password, spliced into the URI at connect time. UserPassword requires a user and password.

entity_whitelist

Limits which source tables are discovered (registered in the collection's table catalog) when the source collection is created: a YAML list of regular expressions, not a comma-separated string. A table is kept if it matches any pattern. Omit it entirely to discover every table in the database.

conn_cfg

Driver-side options forwarded to the connection as SQLAlchemy connect_args.

Publisher

Configure PostgresSrc as the source of a publisher function to select the queries that Tabsdata runs against Postgres.

publish_orders.py

from tabsdatak.api import publisher
from tabsdatak.conn.postgres import PostgresSrc

@publisher(
  source=PostgresSrc(
        queries=[
            "SELECT * FROM orders",
      ],
    ),
  output_tables=["orders"],
)
def publish_orders(orders):
  return orders

• required
• optional
• optional

Function Config Parameters

queries

Defines the SQL queries the publisher runs against Postgres.

Each element in queries represents one source slot and maps positionally to an argument in the publisher function.

For example:

PostgresSrc(
    queries=[
        "SELECT * FROM country",
        "SELECT * FROM city",
    ],
)

initial_values

Values substituted into {placeholder}-style tokens inside queries, so one publisher can read the same tables out of a different schema or catalog without the SQL changing.

src_cfg

Sets additional source configuration.

Subscriber

The PostgresDest connector can be used by a subscriber function to write data from a Tabsdata table into Postgres. See the full subscriber walkthrough

Connection

Postgres subscribers use PostgresDestConn to define the database connection and credentials used by the subscriber.

conn-postgres.yaml

kind: connectionDef
apiVersion: '1.0'
type: tabsdatak.conn.postgres:PostgresDestConn
spec:
  uri: 'postgres://db.internal:5432/warehouse'
  credentials:
  kind: userPasswordCredentials
  apiVersion: '1.0'
  type: tabsdatak.conn.common.types:UserPassword
  spec:
    user: 'secret:tabsdata_reader'
    password: 'secret:example-password'

• required
• required
• optional
• optional

Connection Config Parameters

uri

The SQLAlchemy-style connection URI: postgres://host:port/database. Written as a plain value here, but accepts secret: too.

credentials

The database user and password, spliced into the URI at connect time. UserPassword requires a user and password.

entity_whitelist

Limits which destination tables are discovered (registered in the collection's table catalog) when the destination collection is created: a YAML list of regular expressions, not a comma-separated string. A table is kept if it matches any pattern. Omit it entirely to discover every table in the database.

conn_cfg

Driver-side options forwarded to the connection as SQLAlchemy connect_args.

Subscriber

Configure PostgresDest as the destination of a subscriber function to define which tables Tabsdata writes in Postgres.

write_orders.py

from tabsdatak.api import subscriber
from tabsdatak.conn.postgres import PostgresDest

@subscriber(
  destination=PostgresDest(
        tables=[
            "orders",
      ],
    ),
  input_tables=["orders"],
)
def write_orders(orders):
  return orders

• required
• optional
• optional

Function Config Parameters

tables

Defines the destination tables the subscriber writes to in Postgres.

Each element in tables represents one destination slot and maps positionally to a value returned by the subscriber function.

For example:

PostgresDest(
    tables=[
        "vendors",
        "items",
    ],
)

if_table_exists

Controls what happens when the destination table already has rows. "append" (the default) keeps inserting into the existing rows. "replace" truncates the table before inserting. Neither mode changes the destination table's schema, so the returned frame's columns need to match the table Postgres already has.

dest_cfg

Sets additional destination configuration.