ClickHouseDest

classView source ↗

class ClickHouseDest(
    conn: ClickHouseConn,
    destination_tables: str | list[str],
    if_table_exists: IfTableExistStrategySpec = 'append',
    schema_evolution: SchemaEvolutionSpec = 'none',
)

Bases: DestinationPlugin

Categories: destination

Destination plugin for writing data to ClickHouse using cloud-staged bulk loading.

Data is first written as Parquet files to the configured cloud storage staging area (S3, Azure Blob, or GCS) and then ingested into ClickHouse via its native table functions (s3(), azureBlobStorage()).

Initialize a ClickHouseDest instance for writing data to ClickHouse.

Parameters

parameter

conn

The ClickHouse connection configuration, including both the server parameters and the cloud staging configuration.

parameter

destination_tables

The table(s) to load data into. A single table name may be provided as a string; multiple tables must be a list of strings.

parameter

if_table_exists

The strategy to follow when the table already exists. Defaults to 'append'. * 'append' — insert new rows into the existing table. * 'replace' — truncate the table before loading new rows.

parameter

schema_evolution

The schema evolution strategy to use when the incoming data schema differs from the existing table schema. Defaults to 'none'. * 'none' — raise an error if the schema does not match. * 'iceberg' — apply Iceberg-style schema evolution (add columns, drop columns, widen types, make columns nullable).

Properties

property

connClickHouseConn

The ClickHouse connection configuration.

property

destination_tableslist[str]

The table(s) to load data into.

property

if_table_existsIfTableExistStrategySpec

The strategy to follow when the table already exists.

property

schema_evolutionSchemaEvolutionSpec

The schema evolution strategy.

Methods

method

View source ↗

stream

def stream(working_dir: str, *results: list[LazyFrame | None])

Write each result as a Parquet file and bulk load it into ClickHouse via the cloud staging area.

Parameters:

parameter

working_dir

Temporary directory for intermediate Parquet files.

parameter

results

One polars LazyFrame (or None) per destination table, in the same order as destination_tables.