SourcePlugin

classView source ↗

class SourcePlugin(*args, **kwargs)

Categories: plugin

Parent class for input plugins.

Methods: chunk(working_dir: str) -> Union[str, Tuple[str, ...], List[str]] Trigger the import of the data. The method will receive a folder where it must store the data as parquet files, and return a list of the paths of the files created. This files will then be loaded and mapped to the dataset function in positional order, so if you want file.parquet to be the first argument of the dataset function, you must return it first. If you want a parameter to receive multiple files, return a list of the paths. For example, you would give the following return to provide a first argument with a single file and a second argument with two files: return ["file1.parquet", ["file2.parquet", "file3.parquet"]]

Properties

property

initial_valuesdict

Return a dictionary with the initial values to be stored after execution of the plugin. They will be accessible in the next execution of the plugin. The dictionary must have the parameter names as keys and the initial values as values, all the type string.

Returns: dict: A dictionary with the initial values of the parameters of the plugin.

Methods

method

View source ↗

chunk

def chunk(working_dir: str) -> str | Tuple[str, ...] | List[str]

Trigger the import of the data. This must be implemented in any class that inherits from this class unless directly implementing streaming. The method will receive a folder where it must store the data as parquet files, and return a list of the paths of the files created. This files will then be loaded and mapped to the dataset function in positional order, so if you want file.parquet to be the first argument of the dataset function, you must return it first. If you want a parameter to receive multiple files, return a list of the paths. For example, you would give the following return to provide a first argument with a single file and a second argument with two files: return ["file1.parquet", ["file2.parquet", "file3.parquet"]]

Parameters:

parameter

working_dirstr

The folder where the files must be stored

Returns:

Union[str, Tuple[str, ...], List[str]]: The path of the file(s) created, in the order they must be mapped to the dataset function

method

View source ↗

stream

def stream(working_dir: str) -> List[TableFrame | None | List[TableFrame | None]]