GCSSrc

class

class GCSSrc(
    paths: SrcPathsSpec,
    format: SrcFileFormat = SrcFileFormat.AUTO,
    format_cfg: SrcFormatCfgSpec | None = None,
    initial_last_modified: datetime | None = None,
    src_cfg: SrcCfgSpec | None = None,
)

Bases: Src

Categories: source

GCS source for @publisher -- one output slot per path.

Every table this source publishes automatically carries a @td.file.path metadata column: the gs:// object URI the row was read from. A path that globs several files reports the specific file each row came from. Setting the src_cfg key tabsdata.src_metadata.drop to True leaves the column off.

Examples

Read a single object, selecting the CSV format explicitly:

@publisher(
    source=GCSSrc(paths=["export.csv"], format=SrcFileFormat.CSV),
    output_tables=["rows"],
)
def ingest(rows: TableFramesSpec) -> TableFrameSpec:
    return rows[0] if rows else None

Incremental read: only objects modified at or after the cutoff are imported, and the cutoff advances each run:

@publisher(
    source=GCSSrc(
        paths=["parquet/orders.parquet"],
        initial_last_modified=datetime(2024, 1, 1, tzinfo=timezone.utc),
    ),
    output_tables=["orders"],
)
def ingest_new(orders: TableFramesSpec) -> TableFrameSpec:
    return orders[0] if orders else None

Parameters

parameter

pathsSrcPathsSpec (list[str])

Source paths relative to the connection's base_path, one output slot per entry. Each path is relative (no leading /), must not end with /, and must not contain empty segments (//). A path may use a * glob in its final segment only; backslashes, NUL and ${...} tokens are not allowed.

parameter

formatSrcFileFormat

File format -- one of AUTO, CSV, JSON, AVRO, PARQUET, LOG. AUTO (the default) infers it from the file extension.

parameter

format_cfgSrcFormatCfgSpec | None (dict[SrcFileFormat, dict[Literal['separator', 'quote_char', 'eol_char', 'encoding', 'null_values', 'missing_is_null', 'truncate_ragged_lines', 'comment_prefix', 'try_parse_dates', 'decimal_comma', 'has_header', 'skip_rows', 'skip_rows_after_header', 'raise_if_empty', 'ignore_errors'], Any] | dict] | None)

Optional per-format reader options (source-side keys only), keyed by the SrcFileFormat.

parameter

initial_last_modifieddatetime | None

Optional cutoff for incremental ingestion. The first run imports only files modified at or after this timestamp; the cutoff then advances each run so only newer files are re-imported.

parameter

src_cfgSrcCfgSpec | None (Mapping[Literal['tabsdata.file.chunk_size', 'tabsdata.src_metadata.drop'], Any] | None)

Optional connector config; supported keys are tabsdata.file.chunk_size and tabsdata.src_metadata.drop.

Methods

method

validate

def validate()

Cross-field validation of format_cfg (source side).

Called by the framework; raises GCSValidateException on failure.