DestinationPlugin

classView source ↗

class DestinationPlugin(*args, **kwargs)

Categories: plugin

Parent class for subscriber connectors.

Methods:

trigger_output(working_dir, *args, **kwargs): Trigger the exporting of the data. This function will receive the resulting data from the dataset function and must store it in the desired location.

Methods

method

View source ↗

chunk

def chunk(
    working_dir: str,
    *results: VALID_PLUGIN_RESULT,
) -> List[str | List[str] | List[List[str]]]

Trigger the exporting of the data to local parquet chunks. This method will receive the resulting data from the user function and must store it in the local system as parquet files, using the working_dir. Note: This method should not materialize the data, it should only store it in the local system.

Parameters:

parameter

working_dirstr

The folder where any files generated must be stored (this refers to temporary files that will be deleted after the execution of the plugin, not the final destination of the data)

parameter

results

The data to be exported. It is a list of polars LazyFrames or None.

Returns:

A list of the intermediate files created

method

View source ↗

stream

def stream(working_dir: str, *results: VALID_PLUGIN_RESULT)

Trigger the exporting of the data. This method will receive the resulting data from the user function and must store it in the desired location. Note: this method might materialize the data provided in a single chunk generated by the chunk function if invoked, so chunks should be of an appropriate size.

Parameters:

parameter

working_dirstr

The folder where any intermediate files generated must be stored (this refers to temporary files that will be deleted after the execution of the plugin, not the final destination of the data)

parameter

results

The data to be exported. It is a list of polars LazyFrames or None.

Returns:

None

method

View source ↗

write

def write(files: List[str | List[str] | List[List[str]]])

Given a file or a list of files, write to the desired destination. Note: this method might materialize the data in the files it receives, so chunks should be of an appropriate size.

Parameters:

parameter

filesstr

The file or files to be stored in the final destination.