DatabricksDestination
class DatabricksDestination(
host_url: str,
token: str | Secret,
tables: list[str] | str,
volume: str,
catalog: str | None = None,
schema: str | None = None,
warehouse: str | None = None,
warehouse_id: str | None = None,
if_table_exists: IfTableExistStrategySpec = 'append',
schema_strategy: SchemaStrategySpec = 'update',
**kwargs,
)
Bases: DestinationPlugin
Categories: destination
Databricks based data outputs.
Initializes the DatabricksDestination with the configuration desired to store the data.
Parameters
host_urlDatabricks URL
tokenDatabricks Personal Account Token (PAT). The user owning the PAT requires ‘USE CATALOG’, ‘USE SCHEMA’, ‘READ VOLUME’ & ‘WRITE VOLUME’ privileges.
tablesTables to be created in Databricks. Each table name provided must either be fully qualified in the form of ‘catalog.schema.table_name’ or have the information provided through the ‘catalog’ or the ‘schema’ parameters. For example, if a table of the form ‘schema.table_name’ is provided, the ‘catalog’ parameter must also be provided. If a table of the form ‘table_name’ is provided, both ‘catalog’ and ‘schema’ must be provided.
volumeName of the Databricks volume. The connector uses an existing Databricks managed volume. The files written to the volume are deleted after connector finishes writing the data to the Databricks table.
catalogCatalog name (optional, can be null if the table name has the catalog name.)
schemaSchema name (optional, can be null if the table name has the schema name.)
warehouseWarehouse name (use either this or warehouse_id) that will be used for uploading data to Databricks tables. The user has to have privileges to use the warehouse.
warehouse_idWarehouse Id. Use null if warehouse name is provided.
if_table_existsThis is an optional property to define the strategy to follow when the table already exists. ‘replace’ will create overwrite the existing table, and ‘append’ will append to the existing data in the table. Defaults to ‘append’.
schema_strategyThe is to mention the strategy to follow when appending to a table with an existing schema. ‘update’ will update the schema with the possible new columns that might exist in the TableFrame. ‘strict’ will not modify the schema, and will fail if there is any difference.
Properties
host_urlstrif_table_existsIfTableExistStrategySpecstr: The strategy to follow when the table already exists.
schema_strategySchemaStrategySpecstr: The strategy to follow when appending to a table with an existing schema.
tablesList[str]tokenSecretMethods
chunkdef chunk(working_dir: str, *results: LazyFrame | None) -> List[None | str]
Trigger the exporting of the data to local parquet chunks. This method will receive the resulting data from the user function and must store it in the local system as parquet files, using the working_dir. Note: This method should not materialize the data, it should only store it in the local system.
Parameters:
working_dirstrThe folder where any files generated must be stored (this refers to temporary files that will be deleted after the execution of the plugin, not the final destination of the data)
resultsThe data to be exported. It is a list of polars LazyFrames or None.
Returns:
A list of the intermediate files created
streamdef stream(working_dir: str, *results: List[LazyFrame | None] | LazyFrame | None)
Trigger the exporting of the data. This method will receive the resulting data from the user function and must store it in the desired location. Note: this method might materialize the data provided in a single chunk generated by the chunk function if invoked, so chunks should be of an appropriate size.
Parameters:
working_dirstrThe folder where any intermediate files generated must be stored (this refers to temporary files that will be deleted after the execution of the plugin, not the final destination of the data)
resultsThe data to be exported. It is a list of polars LazyFrames or None.
Returns:
None
writedef write(files)
This method is used to write the files to the databricks. It is called from the stream method, and it is not intended to be called directly.