tabsdata.S3Destination#

class S3Destination(uri: str | list[str], credentials: S3Credentials, format: str | FileFormat = None, region: str = None, catalog: AWSGlue = None)[source]#

Bases: DestinationPlugin

Class for managing the configuration of S3-file-based data outputs.

format#

The format of the file. If not provided, it will be inferred from the file extension.

Type:: FileFormat

uri#

The URI of the files with format: ‘s3://path/to/files’. It can be a single URI or a list of URIs.

Type:: str | list[str]

credentials#

The credentials required to access the S3 bucket.

Type:: S3Credentials

__init__(uri: str | list[str], credentials: S3Credentials, format: str | FileFormat = None, region: str = None, catalog: AWSGlue = None)[source]#

Initializes the S3Destination with the given URI and the credentials required to: access the S3 bucket, and optionally a format and date and time after which the files were modified.

Parameters:

uri (str | list[str]) – The URI of the files with format: ‘s3://path/to/files’. It can be a single URI or a list of URIs.
credentials (S3Credentials) – The credentials required to access the S3 bucket. Must be a S3Credentials object.
format (str | FileFormat, optional) – The format of the file. If not provided, it will be inferred from the file extension of the data. Can be either a string with the format or a FileFormat object. Currently supported formats are ‘csv’, ‘parquet’, ‘ndjson’ and ‘jsonl’.
region (str, optional) – The region where the S3 bucket is located. If not provided, the default AWS region will be used.

Raises:

OutputConfigurationError –
FormatConfigurationError –

Methods

`__init__`(uri, credentials[, format, region, ...])	Initializes the S3Destination with the given URI and the credentials required to
`chunk`(working_dir, *results)	Trigger the exporting of the data to local parquet chunks. This method will
`stream`(working_dir, *results)	Trigger the exporting of the data. This method will receive the resulting data
`write`(files)	Given a file or a list of files, write to the desired destination. Note: this

Attributes

`allow_fragments`	Whether to allow fragments in the output.
`catalog`	The catalog to store the data in.
`credentials`	The credentials required to access the S3 bucket.
`format`	The format of the file.
`region`	The region where the S3 bucket is located.
`uri`	's3://path/to/files'.