PostgresSrc
class PostgresSrc(
conn: PostgresConn,
queries: QueriesSpec,
initial_values: dict[str, Any] | None = None,
transactional: bool = True,
chunk_size: PositiveInt = 100000,
loader: PostgresSrcLoaderSpec = 'polars_sqlalchemy',
schema_overrides: SchemaOverrideSpec = None,
)
Bases: SourcePlugin
Categories: source
Source plugin for reading data from Postgres.
Initialize a PostgresSrc instance for reading data from Postgres.
Parameters
connThe Postgres connection configuration.
queriesThe SQL query or list of queries to execute.
initial_valuesInitial values for query parameters. Defaults to None.
transactionalWhether to use transactions for reading to ensure consistency. Defaults to True.
chunk_sizeThe chunk size in records for reading large queries in batches. Defaults to 100000.
loaderThe data processing loader to use for executing SQL imports.
schema_overridesType mapping overrides for the columns of the queries. It can be a single mapping, a dictionary of mappings or a list of mappings. It can be a single dictionary mapping only when the queries is a single query. If using a dictionary of mappings, the keys must match any of names of the Tabsdata tables produced by the publisher. If using a list of mappings, the list of mappings must have the same size as the queries list; for queries there is no need for mapping the corresponding element in the list can be None. The mapping dictionary must have column names from the corresponding query as keys and the TableFrame DataType to cast the column to. The type casting from the original column type to the desired type must be supported.
Properties
chunk_sizeintThe chunk size in records for reading large queries in batches.
connPostgresConnThe Postgres connection configuration.
initial_valuesdictThe initial values for the parameters in the SQL queries.
loaderPostgresSrcLoaderSpecThe data processing loader to use for executing SQL imports.
querieslist[str]The SQL queries to execute to read data.
schema_overrideslist[dict[str, DataType]] | NoneThe schema overrides for the tables being read.
transactionalboolWhether to use transactions for reading to ensure consistency.
Methods
streamdef stream(working_dir: str) -> list[TableFrame]
Execute the query and load the results as a list of TableFrames.
Parameters:
working_dirThe directory where the Parquet files will be saved.
Returns:
A list of TableFrames containing the query results.