Databricks

The Databricks connector lets Tabsdata run queries against Databricks and write tables into Databricks.

The DatabricksSrc connector can be used by a publisher function to read data from Databricks into a Tabsdata table.

The DatabricksDest connector can be used by a subscriber function to write data from a Tabsdata table into Databricks.

Publisher

The DatabricksSrc connector can be used by a publisher function to read data from Databricks into a Tabsdata table. See the full publisher walkthrough

Connection

Databricks publishers use DatabricksSrcConn to define the workspace, token, and warehouse used by the publisher.

conn-databricks.yaml

kind: connectionDef
apiVersion: '1.0'
type: tabsdatak.conn.databricks:DatabricksSrcConn
spec:
  host_url: 'secret:https://dbc-00000000.cloud.databricks.com'
  token: 'secret:dapi0example0token0000'
  volume: 'main.default.tabsdata_stage'

• required
• required
• required
• optional
• optional
• optional
• optional

Connection Config Parameters

host_url

The Databricks workspace URL, stored as a secret.

token

The Databricks personal access token used for workspace access.

volume

The managed Databricks volume the connector uses to stage files during reads and writes.

warehouse

The SQL warehouse name. Use this or warehouse_id, not both.

warehouse_id

An alternative to warehouse. The two fields are mutually exclusive.

catalog

The default catalog for partially qualified table names.

schema

The default schema. Requires catalog to be set.

Publisher

Configure DatabricksSrc as the source of a publisher function to select the queries that Tabsdata runs against Databricks.

publish_orders.py

from tabsdatak.api import publisher
from tabsdatak.conn.databricks import DatabricksSrc

@publisher(
  source=DatabricksSrc(
        queries=[
            "SELECT * FROM main.default.orders",
      ],
    ),
  output_tables=["orders"],
)
def publish_orders(orders):
  return orders

• required
• optional
• optional

Function Config Parameters

queries

Defines the SQL queries the publisher runs against Databricks.

Each element in queries represents one source slot and maps positionally to an argument in the publisher function.

initial_values

The first-run values for named parameters in queries, written :name. On later runs each parameter takes the value the publisher stored with ctx.set_attr(name, value), so a query can read only the rows added since the last run.

Parameters bind values such as IDs and timestamps, not identifiers. Catalog, schema and table names have to be written into the query, or set with catalog and schema on the connection.

For example:

from tabsdatak.api import TableFrameSpec, TrxCtx, publisher
from tabsdatak.conn.databricks import DatabricksSrc


@publisher(
    source=DatabricksSrc(
        queries=["SELECT * FROM main.default.orders WHERE id > :last_id ORDER BY id"],
        initial_values={"last_id": 0},
    ),
    output_tables=["orders"],
)
def publish_orders(orders: TableFrameSpec, ctx: TrxCtx) -> TableFrameSpec:
    row = orders.max_for("id")
    if row is not None and row["id"] is not None:
        ctx.set_attr("last_id", int(row["id"]))
    return orders

src_cfg

Sets additional source configuration.

Subscriber

The DatabricksDest connector can be used by a subscriber function to write data from a Tabsdata table into Databricks. See the full subscriber walkthrough

Connection

Databricks subscribers use DatabricksDestConn to define the workspace, token, and warehouse used by the subscriber.

conn-databricks.yaml

kind: connectionDef
apiVersion: '1.0'
type: tabsdatak.conn.databricks:DatabricksDestConn
spec:
  host_url: 'secret:https://dbc-00000000.cloud.databricks.com'
  token: 'secret:dapi0example0token0000'
  volume: 'main.default.tabsdata_stage'

• required
• required
• required
• optional
• optional
• optional
• optional

Connection Config Parameters

host_url

The Databricks workspace URL, stored as a secret.

token

The Databricks personal access token used for writes.

volume

The managed volume used for staged files during COPY INTO.

warehouse

The SQL warehouse name. Use this or warehouse_id, not both.

warehouse_id

An alternative to warehouse. The two fields are mutually exclusive.

catalog

The default catalog for partially qualified table names.

schema

The default schema. Requires catalog to be set.

Subscriber

For merge writes, schema evolution, and retry watermarks, see Merging into SQL destinations.

Configure DatabricksDest as the destination of a subscriber function to define which tables Tabsdata writes in Databricks.

write_orders.py

from tabsdatak.api import subscriber
from tabsdatak.conn.databricks import DatabricksDest

@subscriber(
  destination=DatabricksDest(
        tables=[
            "orders",
      ],
    ),
  input_tables=["orders"],
)
def write_orders(orders):
  return orders

• required
• optional
• optional
• optional

Function Config Parameters

tables

Defines the destination tables the subscriber writes to in Databricks.

Each element in tables represents one destination slot and maps positionally to a value returned by the subscriber function. Destination table names can be fully qualified or completed from the connection's catalog/schema defaults.

if_table_exists

"merge" applies inserts, updates, and deletes by key and requires a merge_cfg list. See Merge configuration for a complete example.

Controls what happens when the destination table already exists. "append" (the default) adds rows. "replace" overwrites the table.

schema_evolution

Controls how new columns in the returned frame are handled. "update" forwards mergeSchema=true to COPY INTO so new columns are added. "strict" is the default and leaves existing data columns unchanged. See Merging into SQL destinations for merge-specific behavior.

dest_cfg

Sets additional destination configuration.