DatabricksSource

classView source ↗

class DatabricksSource(
    host_url: str,
    token: str | Secret,
    queries: list[str] | str,
    initial_values: dict | None = None,
    warehouse: str | None = None,
    warehouse_id: str | None = None,
    **kwargs,
)

Bases: SourcePlugin

Categories: source

Databricks based data inputs.

Initializes the DatabricksSource with the configuration desired to read the data.

Parameters

parameter

host_url

Databricks URL

parameter

token

Databricks Personal Account Token (PAT). The user owning the PAT requires 'USE CATALOG', 'USE SCHEMA' & 'SELECT' privileges.

parameter

queries

SQL query or queries to execute. Each query will produce one output. Queries can contain placeholders in the form ':param_name' that will be replaced with the values from 'initial_values'.

parameter

initial_values

Initial values for query parameters. Placeholders in queries of the form ':param_name' will be replaced with the corresponding value. These values are updated between runs to support incremental ingestion. Defaults to None.

parameter

warehouse

Warehouse name (use either this or warehouse_id) that will be used for executing queries. The user has to have privileges to use the warehouse.

parameter

warehouse_id

Warehouse Id. Use null if warehouse name is provided.

Properties

property

host_urlstr

property

initial_valuesdict

The initial values for the parameters in the SQL queries.

property

queriesList[str]

property

tokenSecret

Methods

method

View source ↗

chunk

def chunk(working_dir: str) -> str | List[str]

Trigger the import of the data. This must be implemented in any class that inherits from this class unless directly implementing streaming. The method will receive a folder where it must store the data as parquet files, and return a list of the paths of the files created. This files will then be loaded and mapped to the dataset function in positional order, so if you want file.parquet to be the first argument of the dataset function, you must return it first. If you want a parameter to receive multiple files, return a list of the paths. For example, you would give the following return to provide a first argument with a single file and a second argument with two files:return ["file1.parquet", ["file2.parquet", "file3.parquet"]]

Parameters:

parameter

working_dirstr

The folder where the files must be stored

Returns:

Union[str, Tuple[str, ...], List[str]]: The path of the file(s) created, in the order they must be mapped to the dataset function