S3Dest

class

class S3Dest(
    paths: DestPathsSpec,
    format: DestFileFormat = DestFileFormat.AUTO,
    format_cfg: DestFormatCfgSpec | None = None,
    dest_cfg: DestCfgSpec | None = None,
)

Bases: Dest

Categories: destination

S3 destination for @subscriber -- one target file per path.

Examples

Subscribe rows to S3, choosing the output file format:

@subscriber(
    destination=S3Dest(
        paths=["caps_out.json"],
        format=DestFileFormat.JSON,
    ),
    input_tables=["caps_input/cap_rows"],
)
def write_json(rows: TableFrameSpec) -> TableFrameSpec:
    return rows

Write CSV, tuning the writer through format_cfg:

@subscriber(
    destination=S3Dest(
        paths=["caps_out.csv"],
        format=DestFileFormat.CSV,
        format_cfg={
            DestFileFormat.CSV: {
                "separator": ";",
                "include_header": True,
            }
        },
    ),
    input_tables=["caps_input/cap_rows"],
)
def write_csv(rows: TableFrameSpec) -> TableFrameSpec:
    return rows

Parameters

parameter

pathsDestPathsSpec (list[str])

Target paths relative to the connection's base_path, one write slot per entry. Each path is relative (no leading /), must not end with /, and must not contain empty segments (//). Globs are not allowed; the tokens ${EXEC_PLAN_ID}, ${TRX_ID} and ${EXEC_PLAN_TS} are substituted at write time. Backslashes and NUL are not allowed.

parameter

formatDestFileFormat

Output format -- one of AUTO, CSV, JSON, AVRO, PARQUET. AUTO (the default) infers it from the file extension.

parameter

format_cfgDestFormatCfgSpec | None (dict[DestFileFormat, dict[Literal['separator', 'quote_char', 'eol_char', 'include_header', 'datetime_format', 'date_format', 'time_format', 'float_scientific', 'float_precision', 'null_value', 'quote_style', 'maintain_order'], Any] | dict] | None)

Optional per-format writer options (dest-side keys only), keyed by the DestFileFormat.

parameter

dest_cfgDestCfgSpec | None (Mapping[Literal['tabsdata.file.chunk_size'], Any] | None)

Optional connector config; supports the key tabsdata.file.chunk_size.

Methods

method

validate

def validate()

Cross-field validation of format_cfg (dest side).

Called by the framework; raises S3ValidateException on failure.