tabsdata.S3Destination#

class S3Destination(uri: str | List[str], credentials: dict | S3Credentials, format: str | dict | FileFormat = None, region: str = None, catalog: dict | AWSGlue = None)[source]#

Bases: Output

Class for managing the configuration of S3-file-based data outputs.

format#

The format of the file. If not provided, it will be inferred from the file extension.

Type:

FileFormat

uri#

The URI of the files with format: ‘s3://path/to/files’. It can be a single URI or a list of URIs.

Type:

str | List[str]

credentials#

The credentials required to access the S3 bucket.

Type:

S3Credentials

to_dict()[source]#

Converts the S3Destination object to a dictionary.

__init__(uri: str | List[str], credentials: dict | S3Credentials, format: str | dict | FileFormat = None, region: str = None, catalog: dict | AWSGlue = None)[source]#
Initializes the S3Destination with the given URI and the credentials required to

access the S3 bucket, and optionally a format and date and time after which the files were modified.

Parameters:

  • uri (str | List[str]) – The URI of the files with format: ‘s3://path/to/files’. It can be a single URI or a list of URIs.

  • credentials (dict | S3Credentials) – The credentials required to access the S3 bucket. Can be a dictionary or a S3Credentials object.

  • format (str | dict | FileFormat, optional) – The format of the file. If not provided, it will be inferred from the file extension of the data. Can be either a string with the format, a FileFormat object or a dictionary with the format as the ‘type’ key and any additional format-specific information. Currently supported formats are ‘csv’, ‘parquet’, ‘ndjson’ and ‘jsonl’.

  • region (str, optional) – The region where the S3 bucket is located. If not provided, the default AWS region will be used.

Raises:

  • OutputConfigurationError –

  • FormatConfigurationError –

Methods

__init__(uri, credentials[, format, region, ...])

Initializes the S3Destination with the given URI and the credentials required to

Attributes

CATALOG_KEY

allow_fragments

Whether to allow fragments in the output.

catalog

The catalog to store the data in.

credentials

The credentials required to access the S3 bucket.

format

The format of the file.

region

The region where the S3 bucket is located.

uri

's3://path/to/files'.