SnowflakeDestination
class SnowflakeDestination(
connection_parameters: dict,
destination_table: List[str] | str,
if_table_exists: Literal['append', 'replace'] = 'append',
stage: str | None = None,
**kwargs,
)
Bases: DestinationPlugin
Categories: destination
Snowflake based data outputs.
Initializes the SnowflakeDestination with the configuration desired to store the data.
Parameters
connection_parametersFollowing values are to be defined in connection parameters. account: Snowflake account user: Snowflake user password: Snowflake user’s Personal Access Token (PAT). Password for programmatic access is not supported anymore. role: Snowflake role database: Snowflake database schema: Snowflake schema warehouse: Snowflake warehouse
destination_tableNames of the destination tables to be created, replaced, or appended in Snowflake.
if_table_existsThis is an optional property to define the strategy to follow when the table already exists. ‘replace’ will create overwrite the existing table, and ‘append’ will append to the existing data in the table. Defaults to ‘append’.
stageSnowflake stage name. If not provided, a temporary one will be created.
Properties
destination_tablelist[str]Get the destination table(s) where the data will be stored.
Returns: str | list[str]: The destination table(s).
if_table_existsLiteral['append', 'replace']Returns the value of the if_table_exists property. This property determines what to do if the table already exists.
Methods
chunkdef chunk(working_dir: str, *results: LazyFrame | None) -> List[None | str]
Trigger the exporting of the data to local parquet chunks. This method will receive the resulting data from the user function and must store it in the local system as parquet files, using the working_dir. Note: This method should not materialize the data, it should only store it in the local system.
Parameters:
working_dirstrThe folder where any files generated must be stored (this refers to temporary files that will be deleted after the execution of the plugin, not the final destination of the data)
resultsThe data to be exported. It is a list of polars LazyFrames or None.
Returns:
A list of the intermediate files created
streamdef stream(working_dir: str, *results: List[LazyFrame | None] | LazyFrame | NoneType)
Trigger the exporting of the data. This method will receive the resulting data from the user function and must store it in the desired location. Note: this method might materialize the data provided in a single chunk generated by the chunk function if invoked, so chunks should be of an appropriate size.
Parameters:
working_dirstrThe folder where any intermediate files generated must be stored (this refers to temporary files that will be deleted after the execution of the plugin, not the final destination of the data)
resultsThe data to be exported. It is a list of polars LazyFrames or None.
Returns:
None
writedef write(files)
This method is used to write the files to the database. It is called from the stream method, and it is not intended to be called directly.