S3Source

classView source ↗

class S3Source(
    uri: str | List[str],
    credentials: dict | S3Credentials,
    format: str | dict | FileFormat = None,
    initial_last_modified: str | datetime = None,
    region: str = None,
)

Bases: SourcePlugin

Categories: source

Class for managing the configuration of S3-file-based data inputs.

Attributes: format (FileFormat): The format of the file. If not provided, it will be inferred from the file extension of the data. uri (str | List[str]): The URI of the files with format: 's3://path/to/files'. It can be a single URI or a list of URIs. credentials (S3Credentials): The credentials required to access the S3 bucket. initial_last_modified (str | datetime): If provided, only the files modified after this date and time will be considered.

Initializes the S3Source with the given URI and the credentials required to access the S3 bucket, and optionally a format and date and time after which the files were modified.

Parameters

parameter

uristr | List[str]

The URI of the files with format: 's3://path/to/files'. It can be a single URI or a list of URIs.

parameter

credentialsdict | S3Credentials

The credentials required to access the S3 bucket. Can be a dictionary or a S3Credentials object.

parameter

formatstr | dict | FileFormat, optional

The format of the file. If not provided, it will be inferred from the file extension of the data. Can be either a string with the format, a FileFormat object or a dictionary with the format as the 'type' key and any additional format-specific information. Currently supported formats are 'csv', 'parquet', 'ndjson', 'jsonl' and 'log'.

parameter

initial_last_modifiedstr | datetime, optional

If provided, only the files modified after this date and time will be considered. The date and time can be provided as a string in ISO 8601 format or as a datetime object. If no timezone is provided, UTC will be assumed.

parameter

regionstr, optional

The region where the S3 bucket is located. If not provided, the default AWS region will be used.

Raises

InputConfigurationError FormatConfigurationError

Properties

property

credentialsS3Credentials

The credentials required to access the S3 bucket.

property

formatFileFormat

The format of the file. If not provided, it will be inferred from the file.

property

initial_last_modifiedstr

The date and time after which the files were modified.

property

initial_valuesdict

Return a dictionary with the initial values to be stored after execution of the plugin. They will be accessible in the next execution of the plugin. The dictionary must have the parameter names as keys and the initial values as values, all the type string.

Returns: dict: A dictionary with the initial values of the parameters of the plugin.

property

regionstr | None

str: The region where the S3 bucket is located.

property

uristr | List[str]

The URI of the files with format: 's3://path/to/files'.

class

View source ↗

SupportedFormats

class SupportedFormats(*args, **kwds)

Bases: Enum

Enum for the supported formats for the S3Source.

attribute

csv

attribute

ndjson

attribute

log

attribute

parquet

Methods

method

View source ↗

chunk

def chunk(working_dir: str) -> list[str | None | list[str | None]]

Trigger the import of the data. This must be implemented in any class that inherits from this class unless directly implementing streaming. The method will receive a folder where it must store the data as parquet files, and return a list of the paths of the files created. This files will then be loaded and mapped to the dataset function in positional order, so if you want file.parquet to be the first argument of the dataset function, you must return it first. If you want a parameter to receive multiple files, return a list of the paths. For example, you would give the following return to provide a first argument with a single file and a second argument with two files: return ["file1.parquet", ["file2.parquet", "file3.parquet"]]

Parameters:

parameter

working_dirstr

The folder where the files must be stored

Returns:

Union[str, Tuple[str, ...], List[str]]: The path of the file(s) created, in the order they must be mapped to the dataset function