LocalFileSrc

class

class LocalFileSrc(
    paths: SrcPathsSpec,
    format: SrcFileFormat = SrcFileFormat.AUTO,
    format_cfg: SrcFormatCfgSpec | None = None,
    initial_last_modified: datetime | None = None,
    src_cfg: SrcCfgSpec | None = None,
)

Bases: Src

Categories: source

LocalFile source for @publisher -- one output slot per path.

Every table this source publishes automatically carries a @td.file.path metadata column: the local path the row was read from. A path that globs several files reports the specific file each row came from. Setting the src_cfg key tabsdata.src_metadata.drop to True leaves the column off.

Examples

Publish one file, selecting the reader format explicitly. Each path is an array slot, so its frames arrive as a list:

@publisher(
    source=LocalFileSrc(
        paths=["cities.csv"],
        format=SrcFileFormat.CSV,
    ),
    output_tables=["cities"],
)
def ingest(rows: TableFramesSpec) -> TableFrameSpec:
    return rows[0] if rows else None

Incremental ingestion: initial_last_modified imports only files at or after the cutoff, advancing it on each run:

@publisher(
    source=LocalFileSrc(
        paths=["daily/*.parquet"],
        initial_last_modified=datetime(2024, 1, 1, tzinfo=timezone.utc),
    ),
    output_tables=["new_files"],
)
def ingest_new(rows: TableFramesSpec) -> TableFrameSpec:
    return rows[0] if rows else None

Parameters

parameter

pathsSrcPathsSpec (list[str])

Source paths relative to the connection's base_path, one output slot per entry. Each path is relative (no leading /), must not end with /, and must not contain empty segments (//). A path may use a * glob in its final segment only; backslashes, NUL and ${...} tokens are not allowed.

parameter

formatSrcFileFormat

File format -- one of AUTO, CSV, JSON, AVRO, PARQUET, LOG. AUTO (the default) infers it from the file extension.

parameter

format_cfgSrcFormatCfgSpec | None (dict[SrcFileFormat, dict[Literal['separator', 'quote_char', 'eol_char', 'encoding', 'null_values', 'missing_is_null', 'truncate_ragged_lines', 'comment_prefix', 'try_parse_dates', 'decimal_comma', 'has_header', 'skip_rows', 'skip_rows_after_header', 'raise_if_empty', 'ignore_errors'], Any] | dict] | None)

Optional per-format reader options (source-side keys only), keyed by the SrcFileFormat.

parameter

initial_last_modifieddatetime | None

Optional cutoff for incremental ingestion. The first run imports only files modified at or after this timestamp; the cutoff then advances each run so only newer files are re-imported.

parameter

src_cfgSrcCfgSpec | None (Mapping[Literal['tabsdata.file.chunk_size', 'tabsdata.src_metadata.drop'], Any] | None)

Optional connector config; supported keys are tabsdata.file.chunk_size and tabsdata.src_metadata.drop.

Methods

method

validate

def validate()

Cross-field validation of format_cfg (source side).

Called by the framework; raises LocalFileValidateException on failure.