AzureDestination

classView source ↗

class AzureDestination(
    uri: str | list[str],
    credentials: AzureCredentials,
    format: str | FileFormat = None,
)

Bases: DestinationPlugin

Categories: destination

Class for managing the configuration of Azure-file-based data outputs.

Attributes: format (FileFormat): The format of the file to be created. If not provided, it will be inferred from the file extension. uri (str | list[str]): The URI of the files with format: 'az://path/to/files'. It can be a single URI or a list of URIs. credentials (AzureCredentials): The credentials required to access Azure.

Initializes the AzureDestination with the given URI and the credentials required to access Azure; and optionally a format.

Parameters

parameter

uristr | list[str]

The URI of the files to export with format: 'az://path/to/files'. It can be a single URI or a list of URIs.

parameter

credentialsAzureCredentials

The credentials required to access Azure. Must be an AzureCredentials object.

parameter

formatstr | FileFormat, optional

The format of the file. If not provided, it will be inferred from the file extension. Can be either a string with the format or a FileFormat object. Currently supported formats are 'csv', 'parquet', 'ndjson' and 'jsonl'.

Raises

OutputConfigurationError FormatConfigurationError

Properties

property

allow_fragmentsbool

Whether to allow fragments in the output.

property

credentialsAzureCredentials

The credentials required to access Azure.

property

formatFileFormat

The format of the file. If not provided, it will be inferred from the file extension of the URI.

property

uristr | list[str]

The URI of the files with format: 'az://path/to/files'.

class

View source ↗

SupportedFormats

class SupportedFormats(*args, **kwds)

Bases: Enum

Enum for the supported formats for the AzureDestination.

attribute

avro

attribute

csv

attribute

ndjson

attribute

parquet

Methods

method

View source ↗

chunk

def chunk(working_dir: str, *results)

Trigger the exporting of the data to local parquet chunks. This method will receive the resulting data from the user function and must store it in the local system as parquet files, using the working_dir. Note: This method should not materialize the data, it should only store it in the local system.

Parameters:

parameter

working_dirstr

The folder where any files generated must be stored (this refers to temporary files that will be deleted after the execution of the plugin, not the final destination of the data)

parameter

results

The data to be exported. It is a list of polars LazyFrames or None.

Returns:

A list of the intermediate files created

method

View source ↗

stream

def stream(working_dir: str, *results)

Trigger the exporting of the data. This method will receive the resulting data from the user function and must store it in the desired location. Note: this method might materialize the data provided in a single chunk generated by the chunk function if invoked, so chunks should be of an appropriate size.

Parameters:

parameter

working_dirstr

The folder where any intermediate files generated must be stored (this refers to temporary files that will be deleted after the execution of the plugin, not the final destination of the data)

parameter

results

The data to be exported. It is a list of polars LazyFrames or None.

Returns:

None