Data

class

class Data(df: Any, chunk_size: int | None = None)

Categories: source

One dataframe emitted by DataGenSrc, optionally chunked.

A Data wraps a Polars or Pandas dataframe (a Pandas frame is converted to Polars on construction) plus a per-run chunk_size:

• chunk_size=None -- the whole dataframe is emitted on every run.
• chunk_size=N -- N rows are emitted per run, advancing an offset kept in the plugin's per-flow state; once the offset reaches the end the slot yields None (no more rows).

On the wire a Data serialises to {"parquet": <base64>, "chunk_size": <int|null>}: the dataframe is written to parquet and base64-encoded on serialisation, and decoded back to a Polars frame on deserialisation so the plugin can materialise it to parquet for its output. Like Secret, it round-trips via its own pydantic core schema rather than the descriptor envelope.

Parameters

parameter

dfAny

A Polars or Pandas DataFrame to emit.

parameter

chunk_sizeint | None

Optional positive row count per run; None (the default) emits the whole frame every run.

Properties

property

frameAny

The wrapped dataframe as a Polars DataFrame.

Methods

method

parquet_b64

def parquet_b64() -> str

The dataframe as base64-encoded parquet (computed once).