tabsdata.S3Source#

class S3Source(uri: str | list[str], credentials: S3Credentials, format: str | FileFormat = None, initial_last_modified: str | datetime = None, region: str = None)[source]#

Bases: SourcePlugin

Class for managing the configuration of S3-file-based data inputs.

format#

The format of the file. If not provided, it will be inferred from the file extension of the data.

Type:: FileFormat

uri#

The URI of the files with format: ‘s3://path/to/files’. It can be a single URI or a list of URIs.

Type:: str | list[str]

credentials#

The credentials required to access the S3 bucket.

Type:: S3Credentials

initial_last_modified#

If provided, only the files modified after this date and time will be considered.

Type:: str | datetime

__init__(uri: str | list[str], credentials: S3Credentials, format: str | FileFormat = None, initial_last_modified: str | datetime = None, region: str = None)[source]#

Initializes the S3Source with the given URI and the credentials required to: access the S3 bucket, and optionally a format and date and time after which the files were modified.

Parameters:

uri (str | list[str]) – The URI of the files with format: ‘s3://path/to/files’. It can be a single URI or a list of URIs.
credentials (S3Credentials) – The credentials required to access the S3 bucket. Must be a S3Credentials object.
format (str | FileFormat, optional) – The format of the file. If not provided, it will be inferred from the file extension of the data. Can be either a string with the format or a FileFormat object. Currently supported formats are ‘csv’, ‘parquet’, ‘avro’, ‘ndjson’, ‘jsonl’ and ‘log’.
initial_last_modified (str | datetime, optional) – If provided, only the files modified after this date and time will be considered. The date and time can be provided as a string in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601) or as a datetime object. If no timezone is provided, UTC will be assumed.
region (str, optional) – The region where the S3 bucket is located. If not provided, the default AWS region will be used.

Raises:

InputConfigurationError –
FormatConfigurationError –

Methods

`__init__`(uri, credentials[, format, ...])	Initializes the S3Source with the given URI and the credentials required to
`chunk`(working_dir)	Trigger the import of the data. This must be implemented in any class that
`stream`(working_dir)

Attributes

`credentials`	The credentials required to access the S3 bucket.
`format`	The format of the file.
`initial_last_modified`	The date and time after which the files were modified.
`initial_values`	Return a dictionary with the initial values to be stored after execution of the plugin.
`region`	The region where the S3 bucket is located.
`uri`	's3://path/to/files'.