Skip to main content
GuideServerConfigure and deploy Tabsdata servers on your machine.TutorialsConfigure data integration workflows within a running Tabsdata server.Advanced TutorialsBuild end-to-end workflows between two specific systems.API ReferenceCLI ReferenceRelease Notes
Version: 2.0.2

Databricks

The Databricks connector lets Tabsdata run queries against Databricks and write tables into Databricks.

The DatabricksSrc connector can be used by a publisher function to read data from Databricks into a Tabsdata table.

The DatabricksDest connector can be used by a subscriber function to write data from a Tabsdata table into Databricks.

Publisher​

The DatabricksSrc connector can be used by a publisher function to read data from Databricks into a Tabsdata table. See the full publisher walkthrough

Connection​

Databricks publishers use DatabricksSrcConn to define the workspace, token, and warehouse used by the publisher.

conn-databricks.yaml
kind: connectionDef
apiVersion: '1.0'
type: tabsdatak.conn.databricks:DatabricksSrcConn
spec:
  host_url: 'secret:https://dbc-00000000.cloud.databricks.com'
  token: 'secret:dapi0example0token0000'
  volume: 'main.default.tabsdata_stage'
  • required
  • required
  • required
  • optional
  • optional
  • optional
  • optional

Connection Config Parameters​

host_url​

The Databricks workspace URL, stored as a secret.

token​

The Databricks personal access token used for workspace access.

volume​

The managed Databricks volume the connector uses to stage files during reads and writes.

warehouse​

The SQL warehouse name. Use this or warehouse_id, not both.

warehouse_id​

An alternative to warehouse. The two fields are mutually exclusive.

catalog​

The default catalog for partially qualified table names.

schema​

The default schema. Requires catalog to be set.

Publisher​

Configure DatabricksSrc as the source of a publisher function to select the queries that Tabsdata runs against Databricks.

publish_orders.py
from tabsdatak.api import publisher
from tabsdatak.conn.databricks import DatabricksSrc

@publisher(
  source=DatabricksSrc(
        queries=[
            "SELECT * FROM {catalog}.{schema}.orders",
      ],
    ),
  output_tables=["orders"],
)
def publish_orders(orders):
  return orders
  • required
  • optional
  • optional

Function Config Parameters​

queries​

Defines the SQL queries the publisher runs against Databricks.

Each element in queries represents one source slot and maps positionally to an argument in the publisher function.

A query can carry {'{placeholder}'} names that initial_values fills in, so one publisher reads the same tables out of a different catalog or schema without the SQL changing:

DatabricksSrc(
    queries=[
        "SELECT * FROM {catalog}.{schema}.country",
        "SELECT * FROM {catalog}.{schema}.city",
    ],
    initial_values={"catalog": "main", "schema": "default"},
)

initial_values​

Values substituted into {'{placeholder}'}-style tokens inside queries.

src_cfg​

Sets additional source configuration.

Subscriber​

The DatabricksDest connector can be used by a subscriber function to write data from a Tabsdata table into Databricks. See the full subscriber walkthrough

Connection​

Databricks subscribers use DatabricksDestConn to define the workspace, token, and warehouse used by the subscriber.

conn-databricks.yaml
kind: connectionDef
apiVersion: '1.0'
type: tabsdatak.conn.databricks:DatabricksDestConn
spec:
  host_url: 'secret:https://dbc-00000000.cloud.databricks.com'
  token: 'secret:dapi0example0token0000'
  volume: 'main.default.tabsdata_stage'
  • required
  • required
  • required
  • optional
  • optional
  • optional
  • optional

Connection Config Parameters​

host_url​

The Databricks workspace URL, stored as a secret.

token​

The Databricks personal access token used for writes.

volume​

The managed volume used for staged files during COPY INTO.

warehouse​

The SQL warehouse name. Use this or warehouse_id, not both.

warehouse_id​

An alternative to warehouse. The two fields are mutually exclusive.

catalog​

The default catalog for partially qualified table names.

schema​

The default schema. Requires catalog to be set.

Subscriber​

Configure DatabricksDest as the destination of a subscriber function to define which tables Tabsdata writes in Databricks.

write_orders.py
from tabsdatak.api import subscriber
from tabsdatak.conn.databricks import DatabricksDest

@subscriber(
  destination=DatabricksDest(
        tables=[
            "orders",
      ],
    ),
  input_tables=["orders"],
)
def write_orders(orders):
  return orders
  • required
  • optional
  • optional
  • optional

Function Config Parameters​

tables​

Defines the destination tables the subscriber writes to in Databricks.

Each element in tables represents one destination slot and maps positionally to a value returned by the subscriber function. Destination table names can be fully qualified or completed from the connection's catalog/schema defaults.

if_table_exists​

Controls what happens when the destination table already exists. "append" (the default) adds rows. "replace" overwrites the table.

schema_evolution​

Controls how new columns in the returned frame are handled. "update" (the default) forwards mergeSchema=true to COPY INTO so new columns are added. "strict" rejects schema changes.

dest_cfg​

Sets additional destination configuration.