MSSQLDestination

class

class MSSQLDestination(
    connection_string: str,
    destination_table: str | list[str],
    credentials: dict | UserPasswordCredentials | None = None,
    server: str | Secret = None,
    database: str | Secret = None,
    driver: str | Secret = None,
    if_table_exists: IfTableExistStrategySpec = 'append',
    chunk_size: int = 50000,
    **kwargs,
)

Bases: DestinationPlugin

Categories: destination

Destination plugin for Microsoft SQL Server.

Initialize the MSSQLDestination with a connection string and a table name.

Parameters

parameter

connection_stringstr

The connection string to connect to the database.

parameter

destination_tablestr | List[str]

The table(s) to create. If multiple tables are provided, they must be provided as a list.

parameter

if_table_exists\{'append', 'replace'\}

The strategy to follow when the table already exists. - ‘replace’ will create a new database table, overwriting an existing one. - ‘append’ will append to an existing table.

Properties

property

connection_stringstr

Get the connection string for the database.

Returns: str: The connection string.

property

credentialsUserPasswordCredentials | None

The credentials required to access Microsoft SQL Server. If no credentials were provided, it will return None.

property

destination_tablelist[str]

Get the destination table(s) where the data will be stored.

Returns: list[str]: The destination table(s).

property

if_table_existsIfTableExistStrategySpec

Returns the value of the if_table_exists property. This property determines what to do if the table already exists.

Methods

method

chunk

def chunk(working_dir: str, *results: LazyFrame | None) -> list[None | str]

Trigger the exporting of the data to local parquet chunks. This method will receive the resulting data from the user function and must store it in the local system as parquet files, using the working_dir. Note: This method should not materialize the data, it should only store it in the local system.

Parameters:

parameter

working_dirstr

The folder where any files generated must be stored (this refers to temporary files that will be deleted after the execution of the plugin, not the final destination of the data)

parameter

results

The data to be exported. It is a list of polars LazyFrames or None.

Returns:

A list of the intermediate files created

method

stream

def stream(working_dir: str, *results: list[LazyFrame | None] | LazyFrame | None)

Trigger the exporting of the data. This method will receive the resulting data from the user function and must store it in the desired location. Note: this method might materialize the data provided in a single chunk generated by the chunk function if invoked, so chunks should be of an appropriate size.

Parameters:

parameter

working_dirstr

The folder where any intermediate files generated must be stored (this refers to temporary files that will be deleted after the execution of the plugin, not the final destination of the data)

parameter

results

The data to be exported. It is a list of polars LazyFrames or None.

Returns:

None

method

write

def write(files)

This method is used to write the files to the database. It is called from the stream method, and it is not intended to be called directly.