Version: 2.0.0

Subscribe data to Databricks

Write tables out to Databricks

One entry in tables per input table, in order. if_table_exists="replace" rebuilds each destination table from the frame, so a changed schema does not fail the run.

warning

There are several ways to pass secrets into Tabsdata. See Secrets Management for more info if you are planning to have an LLM register secrets into Tabsdata's credential store.

Can you write the sales vendors and items tables out to databricks

Listed 1 directory, called tabsdata 3 times (ctrl+o to expand)

That is a subscriber, so it goes in a destination collection rather than a source one. The guide and connection template are loaded. Three values left, then I generate and register. Reply with them one per line, or point me at a file that has them:

collection name
destination address and credentials
target tables

everything you need is in

Read 1 file, called tabsdata 2 times (ctrl+o to expand)

Credentials found. Reading back what is not secret: volume main.default.tabsdata_stage, warehouse analytics_wh, catalog main, collection warehouse.

Called tabsdata 4 times (ctrl+o to expand)

Connection written to , and collection warehouse created from it in group destinations. Connection check passed.

Called tabsdata 2 times (ctrl+o to expand)

The subscriber, registered in warehouse:

from tabsdatak.api import subscriber, TableFrameSpec
from tabsdatak.conn.databricks import DatabricksDest


@subscriber(
  destination=DatabricksDest(
      tables=["vendors", "items"],
      if_table_exists="append",
      schema_evolution="update",
  ),
  input_tables=["sales/vendors", "sales/items"],
)
def write_sales(
  vendors: TableFrameSpec,
  items: TableFrameSpec,
) -> tuple[TableFrameSpec, TableFrameSpec]:
  return (vendors, items)

Called tabsdata 6 times (ctrl+o to expand)

Execution committed, 0 failures.

Two tables written in warehouse, one per input table:

┌─────────┬───────────────┬───────┐
│  table  │  from table   │ rows  │
├─────────┼───────────────┼───────┤
│ vendors │ sales/vendors │ 312   │
├─────────┼───────────────┼───────┤
│ items   │ sales/items   │ 8,940 │
└─────────┴───────────────┴───────┘

Input tables are written as collection/table, since a subscriber reads from collections other than its own. A subscriber declares no output_tables: it returns one frame per destination slot, and returning None for a slot writes nothing that run.

Without trigger_by, the subscriber runs whenever any of its input tables gets a new commit, so an export stays current without a schedule.

Write tables out to Databricks​

Write tables out to Databricks