LocalFileDestination

classView source ↗

class LocalFileDestination(
    path: str | List[str],
    format: str | dict | FileFormat = None,
)

Bases: Output

Categories: destination

Abstract base class for managing data output configurations.

Initializes the LocalFileDestination with the given path; and optionally a format.

Parameters

parameter

pathstr | List[str]

The path where the files must be stored. It can be a single path or a list of paths.

parameter

formatstr | dict | FileFormat, optional

The format of the file. If not provided, it will be inferred from the file extension of the data. Can be either a string with the format, a FileFormat object or a dictionary with the format as the 'type' key and any additional format-specific information. Currently supported formats are 'csv', 'parquet', 'ndjson' and 'jsonl'.

Raises

OutputConfigurationError FormatConfigurationError

Properties

property

allow_fragmentsbool

Whether to allow fragments in the output.

property

formatFileFormat

The format of the file or files. If not provided, it will be inferred from the file extension in the path.

property

pathstr | List[str]

The path or paths to store the files.

class

View source ↗

SupportedFormats

class SupportedFormats(*args, **kwds)

Bases: Enum

Enum for the supported formats for the LocalFileDestination.

attribute

csv

attribute

ndjson

attribute

parquet

Methods

method

View source ↗

to_dict

def to_dict() -> dict

Converts the LocalFileDestination object to a dictionary with all the relevant information.

Returns:

dict: A dictionary with the relevant information of the LocalFileDestination object.