DatabricksDest

class

class DatabricksDest(
    tables: DatabricksTableNamesSpec,
    if_table_exists: IfTableExistsSpec = 'append',
    schema_evolution: SchemaEvolutionSpec = 'update',
    dest_cfg: DestCfgSpec | None = None,
)

Bases: Dest

Categories: destination

Databricks destination for @subscriber -- one output slot per table.

Examples

Overwrite a Databricks table on each run:

@subscriber(
    destination=DatabricksDest(
        tables=["cap_replace"],
        if_table_exists="replace",
    ),
    input_tables=["caps_input/cap_rows"],
)
def publish(rows: TableFrameSpec) -> TableFrameSpec:
    return rows

Append rows and let COPY INTO add new columns via schema_evolution:

@subscriber(
    destination=DatabricksDest(
        tables=["cap_evolve"],
        if_table_exists="append",
        schema_evolution="update",
    ),
    input_tables=["caps_input/cap_evo_v2"],
)
def evolve(rows: TableFrameSpec) -> TableFrameSpec:
    return rows

Parameters

parameter

tablesDatabricksTableNamesSpec (list[str])

Target table names, one write slot per entry. Each is a 1-, 2- or 3-part name (table, schema.table or catalog.schema.table); backtick-quote a part for special characters. Partial names are completed from the connection's catalog / schema defaults.

parameter

if_table_existsIfTableExistsSpec (Literal['append', 'replace'])

"append" (the default) adds rows; "replace" overwrites the table.

parameter

schema_evolutionSchemaEvolutionSpec (Literal['update', 'strict'])

"update" (the default) forwards mergeSchema=true to COPY INTO so new columns are added; "strict" rejects schema changes.

parameter

dest_cfgDestCfgSpec | None (Mapping[Literal['databricks.logging_level'], Any] | None)

Optional connector config; the only supported key is databricks.logging_level.