DruidConn

classView source ↗

class DruidConn(
    overlord_url: str,
    staging: StagingConfig,
    time_column: str | None = None,
    segment_granularity: SegmentGranularitySpec = 'ALL',
    max_concurrent_subtasks: int = 1,
    max_rows_per_segment: int = 5000000,
    max_rows_in_memory: int = 1000000,
    http_timeout_sec: int = 300,
    credentials: UserPasswordCredentials | None = None,
)

Bases: Conn

Categories: connections

Represents a connection configuration for Apache Druid with cloud-staged bulk loading.

Data is first uploaded to the configured cloud staging area (S3, Azure Blob, or GCS) and then ingested into Druid via the Overlord REST API using a native batch index_parallel ingestion task.

Represents a connection configuration for Apache Druid with cloud-staged bulk loading.

Data is first uploaded to the configured cloud staging area (S3, Azure Blob, or GCS) and then ingested into Druid via the Overlord REST API using a native batch index_parallel ingestion task.

Parameters

parameter

overlord_url

Base URL of the Druid Overlord REST API (e.g. "http://druid-overlord:8090").

parameter

staging

Cloud storage staging configuration. Provide one of: * :class:~tabsdata._io.cloud_staging.common.storage_config.S3Config — stage files in Amazon S3. * :class:~tabsdata._io.cloud_staging.common.storage_config.AzureConfig — stage files in Azure Blob Storage. * :class:~tabsdata._io.cloud_staging.common.storage_config.GCSConfig — stage files in Google Cloud Storage.

parameter

time_column

Name of the PyArrow column to use as Druid's required __time primary timestamp. When None a constant value is used (2000-01-01T00:00:00.000Z). Defaults to None.

parameter

segment_granularity

Time-based partitioning granularity. One of "ALL", "DAY", "HOUR", "MONTH", or "YEAR". Defaults to "ALL".

parameter

max_concurrent_subtasks

Maximum number of parallel sub-tasks inside the index_parallel ingestion task. When set to 1 the supervisor runs inline without spawning subtasks. Defaults to 1.

parameter

max_rows_per_segment

Dynamic partitioning threshold — maximum number of rows per segment. Defaults to 5_000_000.

parameter

max_rows_in_memory

In-memory buffer size during ingestion. Defaults to 1_000_000.

parameter

http_timeout_sec

Timeout in seconds for HTTP requests to the Overlord API. Defaults to 300.

parameter

credentials

:class:~tabsdata.UserPasswordCredentials for HTTP Basic authentication against the Overlord API, or None (the default) to skip authentication.

Properties

property

credentialsUserPasswordCredentials | None

The credentials used for HTTP Basic authentication. None if no credentials were provided.

property

http_timeout_secint

Timeout in seconds for HTTP requests to the Overlord API.

property

max_concurrent_subtasksint

Maximum number of parallel sub-tasks inside the ingestion task.

property

max_rows_in_memoryint

In-memory buffer size during ingestion.

property

max_rows_per_segmentint

Maximum number of rows per segment (dynamic partitioning threshold).

property

overlord_urlstr

The base URL of the Druid Overlord REST API.

property

segment_granularitySegmentGranularitySpec

Time-based partitioning granularity for new segments.

property

stagingStagingConfig

The cloud storage staging configuration (S3Config, AzureConfig, or GCSConfig).

property

time_columnstr | None

The PyArrow column to use as Druid's __time primary timestamp. None if a constant value should be used.