DruidDest

classView source ↗

class DruidDest(
    conn: DruidConn,
    destination_tables: str | list[str],
    if_table_exists: IfTableExistStrategySpec = 'append',
)

Bases: DestinationPlugin

Categories: destination

Destination plugin for writing data to Apache Druid using cloud-staged bulk loading.

Data is first written as Parquet files to the configured cloud storage staging area (S3, Azure Blob, or GCS) and then ingested into Druid via a native batch index_parallel task submitted to the Overlord REST API.

Initialize a DruidDest instance for writing data to Apache Druid.

Parameters

parameter

conn

The Druid connection configuration, including both the Overlord parameters and the cloud staging configuration.

parameter

destination_tables

The datasource(s) to load data into. A single datasource name may be provided as a string; multiple names must be a list of strings.

parameter

if_table_exists

The strategy to follow when the datasource already exists. Defaults to 'append'. * 'append' — add new segments alongside existing ones. * 'replace' — drop existing segments and replace with new data.

Properties

property

connDruidConn

The Druid connection configuration.

property

destination_tableslist[str]

The datasource(s) to load data into.

property

if_table_existsIfTableExistStrategySpec

The strategy to follow when the datasource already exists.

Methods

method

View source ↗

stream

def stream(working_dir: str, *results: list[LazyFrame | None])

Write each result as a Parquet file and bulk load it into Druid via the cloud staging area.

Parameters:

parameter

working_dir

Temporary directory for intermediate Parquet files.

parameter

results

One polars LazyFrame (or None) per destination datasource, in the same order as destination_tables.