LocalFileSource

classView source ↗

class LocalFileSource(
    path: str | List[str],
    format: str | dict | FileFormat = None,
    initial_last_modified: str | datetime = None,
)

Bases: Input

Categories: source

Class for managing the configuration of local-file-based data inputs.

Attributes: format (FileFormat): The format of the file. If not provided, it will be inferred from the file extension of the data. path (str | List[str]): The path where the files can be found. It can be a single path or a list of paths. initial_last_modified (str | None): If not None, only the files modified after this date and time will be considered.

Methods: to_dict(): Converts the LocalFileSource object to a dictionary.

Initializes the LocalFileSource with the given path, and optionally a format and a date and time after which the files were modified.

Parameters

parameter

pathstr | List[str]

The path where the files can be found. It can be a single path or a list of paths.

parameter

formatstr | dict | FileFormat, optional

The format of the file. If not provided, it will be inferred from the file extension of the data. Can be either a string with the format, a FileFormat object or a dictionary with the format as the 'type' key and any additional format-specific information. Currently supported formats are 'csv', 'parquet', 'ndjson', 'jsonl' and 'log'.

parameter

initial_last_modifiedstr | datetime.datetime, optional

If provided, only the files modified after this date and time will be considered. The date and time can be provided as a string in ISO 8601 format or as a datetime object. If no timezone is provided, UTC will be assumed.

Raises

InputConfigurationError FormatConfigurationError

Properties

property

formatFileFormat

The format of the file or files. If not provided, it will be inferred from the file extension in the path.

property

initial_last_modifiedstr

The date and time after which the files were modified.

property

pathstr | List[str]

The path or paths to the files to load.

class

View source ↗

SupportedFormats

class SupportedFormats(*args, **kwds)

Bases: Enum

Enum for the supported formats for the LocalFileSource.

attribute

csv

attribute

ndjson

attribute

log

attribute

parquet

Methods

method

View source ↗

to_dict

def to_dict() -> dict

Converts the LocalFileSource object to a dictionary with all the relevant information.

Returns:

dict: A dictionary with the relevant information of the LocalFileSource object.