Azure Blob Storage

The Azure Blob Storage connector lets Tabsdata read files from and write files to Azure Blob Storage.

The AzureBlobSrc connector can be used by a publisher function to read data from Azure Blob Storage into a Tabsdata table.

The AzureBlobDest connector can be used by a subscriber function to write data from a Tabsdata table into Azure Blob Storage.

Supported formats

Format	Publish	Subscribe
AUTO	Yes	Yes
CSV	Yes	Yes
JSON	Yes	Yes
AVRO	Yes	Yes
PARQUET	Yes	Yes
LOG	Yes	No

Publisher

The AzureBlobSrc connector can be used by a publisher function to read data from Azure Blob Storage into a Tabsdata table. See the full publisher walkthrough

Connection

Azure Blob Storage publishers use AzureBlobSrcConn to define the container and credentials used by the publisher.

conn-azure.yaml

kind: connectionDef
apiVersion: '1.0'
type: tabsdatak.conn.azureblob:AzureBlobSrcConn
spec:
  container: 'acme-ingest'
  credentials:
  kind: azureAccountKeyCredentials
  apiVersion: '1.0'
  type: tabsdatak.conn.common.types:AzureAccountKey
  spec:
    account_name: 'secret:mystorageaccount'
    account_key: 'secret:8Zs0RExAmPlEkEy0000=='

• required
• required
• optional
• optional

Connection Config Parameters

container

The name of the Azure Blob container that contains the files to publish.

credentials

The Azure storage-account credentials used to access the container. AzureAccountKey requires an account_name and account_key.

base_path

Sets the base path within the container. All paths configured in AzureBlobSrc are relative to this path.

base_path starts with /, cannot end with /, and cannot contain empty path segments. The default is /.

conn_cfg

Optional connector configuration overrides.

Publisher

Configure AzureBlobSrc as the source of a publisher function to select the files that Tabsdata reads from Azure Blob Storage.

publish_orders.py

from tabsdatak.api import publisher
from tabsdatak.conn.azureblob import AzureBlobSrc, SrcFileFormat

@publisher(
  source=AzureBlobSrc(
        paths=[
            "incoming/orders.parquet",
      ],
    ),
  output_tables=["orders"],
)
def publish_orders(orders):
  return orders

• required
• optional
• optional
• optional
• optional

Function Config Parameters

paths

Defines the files that the publisher reads from Azure Blob Storage.

Each path is relative to the base_path configured in AzureBlobSrcConn. Source paths:

• cannot begin or end with /
• cannot contain empty path segments
• can contain a * glob in the final path segment

For example:

AzureBlobSrc(
    paths=[
        "orders/2026/*.parquet",
        "customers/customers.csv",
    ]
)

Each element in paths represents one source slot and maps positionally to an argument in the publisher function.

When a path uses a glob to match multiple files, the matching files are provided through the corresponding publisher input.

format

Sets the format used to read the source files.

Supported formats are AUTO, CSV, JSON, AVRO, PARQUET, and LOG. The default is AUTO, which determines the format from the file extension.

format_cfg

Sets format-specific reader options. Configuration is keyed by SrcFileFormat.

initial_last_modified

Sets the initial cutoff for incremental ingestion.

On the first run, Tabsdata imports files modified at or after the specified timestamp. The cutoff then advances so subsequent runs only import newer files.

src_cfg

Sets additional source configuration.

The following options are supported:

Option	Description
tabsdata.file.chunk_size	Controls the size of chunks used when reading files.
tabsdata.src_metadata.drop	When True, excludes the @td.file.path metadata column.

By default, rows read from Azure Blob Storage include a @td.file.path metadata column containing the source file's URI.

Subscriber

The AzureBlobDest connector can be used by a subscriber function to write data from a Tabsdata table into Azure Blob Storage. See the full subscriber walkthrough

Connection

Azure Blob Storage subscribers use AzureBlobDestConn to define the container and credentials used by the subscriber.

conn-azure.yaml

kind: connectionDef
apiVersion: '1.0'
type: tabsdatak.conn.azureblob:AzureBlobDestConn
spec:
  container: 'acme-exports'
  credentials:
  kind: azureAccountKeyCredentials
  apiVersion: '1.0'
  type: tabsdatak.conn.common.types:AzureAccountKey
  spec:
    account_name: 'secret:mystorageaccount'
    account_key: 'secret:8Zs0RExAmPlEkEy0000=='

• required
• required
• optional
• optional

Connection Config Parameters

container

The name of the Azure Blob container where the subscriber writes files.

credentials

The Azure storage-account credentials used to access the container. AzureAccountKey requires an account_name and account_key.

base_path

Sets the base path within the container. All paths configured in AzureBlobDest are relative to this path.

base_path starts with /, cannot end with /, and cannot contain empty path segments. The default is /.

conn_cfg

Optional connector configuration overrides.

Subscriber

Configure AzureBlobDest as the destination of a subscriber function to define where Tabsdata writes files in Azure Blob Storage.

write_orders.py

from tabsdatak.api import subscriber
from tabsdatak.conn.azureblob import AzureBlobDest, DestFileFormat

@subscriber(
  destination=AzureBlobDest(
        paths=[
            "exports/orders.parquet",
      ],
    ),
  input_tables=["orders"],
)
def write_orders(orders):
  return orders

• required
• optional
• optional
• optional

Function Config Parameters

paths

Defines where the subscriber writes files in Azure Blob Storage.

Each path is relative to the base_path configured in AzureBlobDestConn. Destination paths:

• cannot begin or end with /
• cannot contain empty path segments
• cannot contain globs

Each element in paths represents one destination slot and maps positionally to a value returned by the subscriber function.

Destination paths can contain variables that Tabsdata substitutes when the subscriber runs:

Variable	Value
${EXEC_PLAN_ID}	Current execution plan ID.
${TRX_ID}	Current transaction ID.
${EXEC_PLAN_TS}	Timestamp when the execution plan started.

For example:

AzureBlobDest(
    paths=[
        "exports/${EXEC_PLAN_ID}/orders.parquet",
    ]
)

Dynamic paths can be used to create a new object for each execution instead of overwriting an existing object.

format

Sets the format used to write the destination file.

Supported formats are AUTO, CSV, JSON, AVRO, and PARQUET. The default is AUTO, which determines the format from the destination file extension.

format_cfg

Sets format-specific writer options. Configuration is keyed by DestFileFormat.

dest_cfg

Sets additional destination configuration.

The tabsdata.file.chunk_size option controls the size of chunks used when writing files.