S3Destination

classView source ↗

class S3Destination(
    uri: str | List[str],
    credentials: dict | S3Credentials,
    format: str | dict | FileFormat = None,
    region: str = None,
    catalog: dict | AWSGlue = None,
)

Bases: Output

Categories: destination

Class for managing the configuration of S3-file-based data outputs.

Attributes: format (FileFormat): The format of the file. If not provided, it will be inferred from the file extension. uri (str | List[str]): The URI of the files with format: 's3://path/to/files'. It can be a single URI or a list of URIs. credentials (S3Credentials): The credentials required to access the S3 bucket.

Methods: to_dict(): Converts the S3Destination object to a dictionary.

Initializes the S3Destination with the given URI and the credentials required to access the S3 bucket, and optionally a format and date and time after which the files were modified.

Parameters

parameter

uristr | List[str]

The URI of the files with format: 's3://path/to/files'. It can be a single URI or a list of URIs.

parameter

credentialsdict | S3Credentials

The credentials required to access the S3 bucket. Can be a dictionary or a S3Credentials object.

parameter

formatstr | dict | FileFormat, optional

The format of the file. If not provided, it will be inferred from the file extension of the data. Can be either a string with the format, a FileFormat object or a dictionary with the format as the 'type' key and any additional format-specific information. Currently supported formats are 'csv', 'parquet', 'ndjson' and 'jsonl'.

parameter

regionstr, optional

The region where the S3 bucket is located. If not provided, the default AWS region will be used.

Raises

OutputConfigurationError FormatConfigurationError

Properties

property

allow_fragmentsbool

Whether to allow fragments in the output.

property

catalogAWSGlue

Catalog: The catalog to store the data in.

property

credentialsS3Credentials

The credentials required to access the S3 bucket.

property

formatFileFormat

The format of the file. If not provided, it will be inferred from the file.

property

regionstr | None

str: The region where the S3 bucket is located.

property

uristr | List[str]

The URI of the files with format: 's3://path/to/files'.

class

View source ↗

SupportedFormats

class SupportedFormats(*args, **kwds)

Bases: Enum

Enum for the supported formats for the S3Destination.

attribute

csv

attribute

ndjson

attribute

parquet

Methods

method

View source ↗

to_dict

def to_dict() -> dict

Converts the S3Destination object to a dictionary with all the relevant information.

Returns:

dict: A dictionary with the relevant information of the S3Destination object.