BigQueryDest

classView source ↗

class BigQueryDest(
    conn: BigQueryConn,
    tables: TableSpec | None = None,
    if_table_exists: IfTableExistStrategySpec = 'append',
    schema_strategy: SchemaStrategySpec = 'update',
)

Bases: DestinationPlugin

Categories: destination

BigQuery based data outputs. The data is first stored in parquet files in a GCS bucket, and then loaded into the BigQuery tables.

Initializes the BigQueryDest with the configuration desired to store the data.

Parameters

parameter

conn

The BigQuery connection configuration.

parameter

tables

The table(s) to store the data in. If multiple tables are provided, they must be provided as a list. If None, the table names will be those of the input tables for the function. Defaults to None.

parameter

if_table_exists

The strategy to follow when the table already exists. - ‘append’ will append to an existing table. - ‘replace’ will create a new table, overwriting an existing one. Defaults to 'append'.

parameter

schema_strategy

The strategy to follow for the schema when the table already exists. - ‘update’ will update the schema with the possible new columns that might exist in the TableFrame. - ‘strict’ will not modify the schema, and will fail if there is any difference. Defaults to 'update'.

Properties

property

connBigQueryConn

property

if_table_existsIfTableExistStrategySpec

str: The strategy to follow when the table already exists.

property

schema_strategySchemaStrategySpec

str: The strategy to follow when appending to a table with an existing schema.

property

tableslist[str] | None

Methods

method

View source ↗

stream

def stream(working_dir: str, *results: list[LazyFrame | None] | LazyFrame | None)

Trigger the exporting of the data. This method will receive the resulting data from the user function and must store it in the desired location. Note: this method might materialize the data provided in a single chunk generated by the chunk function if invoked, so chunks should be of an appropriate size.

Parameters:

parameter

working_dirstr

The folder where any intermediate files generated must be stored (this refers to temporary files that will be deleted after the execution of the plugin, not the final destination of the data)

parameter

results

The data to be exported. It is a list of polars LazyFrames or None.

Returns:

None