Databricks

The Databricks connector lets Tabsdata run queries against Databricks and write tables into Databricks.

The DatabricksSrc connector can be used by a publisher function to read data from Databricks into a Tabsdata table.

The DatabricksDest connector can be used by a subscriber function to write data from a Tabsdata table into Databricks.

Publishers and Subscribers

The DatabricksSrc connector can be used by a publisher function to read data from Databricks into a Tabsdata table. See the full publisher walkthrough

Connection

Databricks publishers use DatabricksSrcConn to define the workspace, token, and warehouse used by the publisher.

conn-databricks.yaml

kind: connectionDef
apiVersion: '1.0'
type: tabsdatak.conn.databricks:DatabricksSrcConn
spec:
  host_url: 'secret:https://dbc-00000000.cloud.databricks.com'
  token: 'secret:dapi0example0token0000'
  volume: 'main.default.tabsdata_stage'

• required
• required
• required
• optional
• optional
• optional
• optional

Connection Config Parameters

host_url

The Databricks workspace URL, stored as a secret.

token

The Databricks personal access token used for workspace access.

volume

The managed Databricks volume the connector uses to stage files during reads and writes.

warehouse

The SQL warehouse name. Use this or warehouse_id, not both.

warehouse_id

An alternative to warehouse. The two fields are mutually exclusive.

catalog

The default catalog for partially qualified table names.

schema

The default schema. Requires catalog to be set.

Publisher

Configure DatabricksSrc as the source of a publisher function to select the queries that Tabsdata runs against Databricks.

publish_orders.py

from tabsdatak.api import publisher
from tabsdatak.conn.databricks import DatabricksSrc

@publisher(
  source=DatabricksSrc(
        queries=[
            "SELECT * FROM {catalog}.{schema}.orders",
      ],
    ),
  output_tables=["orders"],
)
def publish_orders(orders):
  return orders

• required
• optional
• optional

Function Config Parameters

queries

Defines the SQL queries the publisher runs against Databricks.

Each element in queries represents one source slot and maps positionally to an argument in the publisher function.

A query can carry {'{placeholder}'} names that initial_values fills in, so one publisher reads the same tables out of a different catalog or schema without the SQL changing:

DatabricksSrc(
    queries=[
        "SELECT * FROM {catalog}.{schema}.country",
        "SELECT * FROM {catalog}.{schema}.city",
    ],
    initial_values={"catalog": "main", "schema": "default"},
)

initial_values

Values substituted into {'{placeholder}'}-style tokens inside queries.

src_cfg

Sets additional source configuration.