MySQLCdcStreamSrc

class

class MySQLCdcStreamSrc(
    tables: list[MySQLTableNameSpec],
    start_from: MySQLCdcStartFromSpec,
    columns_format: CdcColumnsFormatSpec = 'columns',
    flatten_columns: bool = True,
    src_cfg: CdcSrcCfgSpec | None = None,
)

Bases: StreamSrc

Categories: source

MySQL CDC stream source for @stream_publisher.

A streaming change-data-capture source: it emits one tableframe-batch per binlog flush across the tracked tables. The decorated @stream_publisher function receives the flushed batch as a list of frames and must NOT declare a ctx parameter.

Parameters

parameter

tableslist[MySQLTableNameSpec] (list[str])

Qualified schema.table names to track.

parameter

start_fromMySQLCdcStartFromSpec (Literal['full'] | Literal['head', 'tail'] | GtidPosition | BinlogPosition | TimestampPosition)

Where to begin reading when no prior offset is persisted -- "full", the "head" / "tail" keyword, a GtidPosition, a BinlogPosition or a TimestampPosition. Required.

"full" first emits a full snapshot of every tracked table (rows marked as inserts, preceded by a LOAD control row per table) and then captures changes from the exact position of that snapshot.

To resynchronise a destination later, keep start_from="full" and reset the function's stage (tdk fn update --reset-stage): a fresh stage holds no stream state, so the stream starts again with the snapshot and emits the LOAD rows first. A stage reset with any other start_from restarts there and sends no LOAD row.

Beyond the privileges plain CDC needs, the connecting user must hold SELECT on every tracked table, since the snapshot reads the tables themselves rather than the binlog, and, where the server grants it, RELOAD (to briefly take the global read lock that pins the snapshot to a binlog position). The server must run with gtid_mode=ON, and the tracked tables must be InnoDB. The lock is held only while the position is read, not while the tables are scanned. Managed services such as Amazon RDS and Aurora refuse the global read lock, and the initial load still runs there with the same transactional consistency.

parameter

columns_formatCdcColumnsFormatSpec (Literal['columns', 'struct', 'map'])

Layout for the new (and old, when present) columns payload on each change row -- "columns" (default), "struct" or "map".

parameter

flatten_columnsbool

When True (default), the new columns are inlined as top-level columns, so the downstream consumer sees an ordinary tabular row and columns_format then only affects the old-columns layout.

parameter

src_cfgCdcSrcCfgSpec | None (Mapping[Literal['tabsdata.cdc.poll.max_secs', 'mysql.cdc.logging_level', 'tabsdata.cdc.buffer.max_rows', 'tabsdata.cdc.buffer.max_bytes', 'tabsdata.cdc.buffer.max_secs', 'tabsdata.cdc.poll.idle_secs', 'tabsdata.cdc.max_large_text_length', 'tabsdata.cdc.fetch_size', 'tabsdata.cdc.chunk_size'], Any] | None)

Optional config mapping over the allowed CDC keys only -- buffer thresholds (the batch limits are the @stream_publisher batch settings) and polling cadence. The plugin falls back to its built-in defaults for missing keys. A byte threshold is measured on every change and slows ingestion down; bounding by rows is the cheap option and the preferred one.