tabsdata.dataquality

class AllOk( none_is_ok: bool = False, nan_is_ok: bool = False, *, tags: str | list[str] | None = None, )

Bases: BoolCriteria

Categories:: dq-operator

A boolean criteria that selects records where all boolean classifiers pass (True).

class AnyFailed( none_is_ok: bool = False, nan_is_ok: bool = False, *, tags: str | list[str] | None = None, )

Bases: BoolCriteria

Categories:: dq-operator

A boolean criteria that selects records where at least one boolean classifier fails (False).

Bases: Classifier, ABC

Abstract base class for classifiers that produce a boolean result (True or False).

classmethod supported_dtypes() → FrozenSet[Type]: Returns the set of data types supported by boolean classifiers.

classmethod BoolClassifier.supported_dtypes() → FrozenSet[Type]: Returns the set of data types supported by boolean classifiers.

class BoolCriteria( none_is_ok: bool, nan_is_ok: bool, *, tags: str | list[str] | None = None, )

Bases: Criteria, ABC

A boolean criteria to select records based on classifier results.

property nan_is_ok: bool: Returns if NaN values should be considered OK.

property none_is_ok: bool: Returns if None values should be considered OK.

property tags: list[str] | None: Returns the list of tags associated with this criteria.

Bases: Classifier, ABC

A base class for categorizers.

classmethod Categorizer.supported_dtypes() → FrozenSet[Type]: Returns the set of data types supported by this classifier.

class CategoryCriteria( column_name: str, bins: Annotated[int, Strict(strict=True), FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=0), Le(le=100)])] | Literal['none', 'nan', 'underflow', 'overflow'] | Annotated[list[Annotated[int, Strict(strict=True), FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=0), Le(le=100)])] | Literal['none', 'nan', 'underflow', 'overflow']], Strict], )

Bases: Criteria, ABC

A category criteria to select records based on classifier results.

property bins: AbstractSet[Annotated[int, Strict(strict=True), FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=0), Le(le=100)])] | Literal['none', 'nan', 'underflow', 'overflow']]: Returns a set of bins.

property column_name: str: Returns the column name to check for values.

Bases: ABC

Abstract base class for all data quality classifiers.

A classifier is applied to one or more columns to produce a classification result, which is stored in a new column. It supports tags to allow data quality operations to selectively act on specific classifiers.

property column_names: list[tuple[str, str | None]] | None: Returns the list of columns the classifier applies to, including optional destination column names.

property on_missing_column: Literal['ignore', 'fail']: Returns what do if the column is missing.

property on_wrong_type: Literal['ignore', 'fail']: Returns what do if the column type is incompatible with the classifier.

property on_wrong_value: Literal['ignore', 'fail']: Returns what do if the column type is incompatible with the classifier.

classmethod supported_dtypes() → FrozenSet[Type]: Returns the set of data types supported by this classifier.

property tags: list[str] | None: Returns the list of tags associated with the classifier.

classmethod Classifier.supported_dtypes() → FrozenSet[Type]: Returns the set of data types supported by this classifier.

class Criteria

Bases: ABC

Defines the conditions for selecting records based on classifier results.

A Criteria object specifies which rows of a table should be considered for a data quality operation. This can be based on the outcomes of boolean classifiers or the binning results of categorical classifiers.

class DataQuality( table: str, classifiers: Classifier | list[Classifier], operators: Operator | list[Operator], )

Bases: OnTablesAction

Categories:: dq-action

Represents a data quality action to be performed on a table.

This class encapsulates a set of data quality rules, defined by classifiers and operators, that are applied to a specified table. It serves as a container for a complete data quality workflow.

property classifiers: list[Classifier]: Returns the list of classifiers used in this data quality action.

property operators: list[Operator]: Returns the list of operators that will be executed based on the classifier results.

property table: str: Returns the name of the table targeted by this data quality action.

Bases: MatchClassifier

Categories:: dq-classifier

A boolean classifier that checks if a value does not match a regex pattern.

classmethod DoesNotMatch.supported_dtypes() → FrozenSet[Type]: Returns the set of data types supported by Matches classifier.

class Enrich( to_table: str | None = None, *, tags: str | list[str] | None = None, )

Bases: TagOperator

Categories:: dq-operator

An operator that adds the data quality classifier columns to the original table frame. It does not remove any row from the original table.

property to_table: str | None: Returns the destination table with the enriched data quality classifier columns. If None, the original table is used.

class ExponentialScale( min_val: Annotated[int, Strict(strict=True)] | Annotated[float, Strict(strict=True)], max_val: Annotated[int, Strict(strict=True)] | Annotated[float, Strict(strict=True)], base: Annotated[float, Strict(strict=True), FieldInfo(annotation=NoneType, required=True, metadata=[Gt(gt=0)])] | None = None, bins: Annotated[int, Strict(strict=True), FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=1), Le(le=100)])] = 100, use_bin_zero: bool = False, )

Bases: NonIdentityScale

Categories:: dq-classifier

An exponential scale.

property base: Annotated[float, Strict(strict=True), FieldInfo(annotation=NoneType, required=True, metadata=[Gt(gt=0)])]: Returns the base of the exponential scale.

class Fail( criteria: Criteria, threshold: Threshold, )

Bases: CriteriaOperator

Categories:: dq-operator

An operator that fails the current transaction if a data quality threshold is met.

property threshold: Threshold: Returns the threshold for this Fail operation.

class Filter( criteria: Criteria, to_table: str | None = None, include_quality_columns: Literal['none', 'criteria', 'all'] = 'none', )

Bases: CriteriaOperator

Categories:: dq-operator

An operator that filters rows from a table based on data quality criteria.

The filtered rows are removed from the table being processed. They can either be discarded or redirected to another table.

property include_quality_columns: Literal['none', 'criteria', 'all']: Returns the data quality columns to included in the table with filtered out data.

property to_table: str | None: Returns the destination table for filtered rows, or None if they are discarded.

class HasLength( column_names: str | tuple[str, str | None] | Annotated[list[str], Strict] | Annotated[list[tuple[str, str | None]], Strict] | None = None, min_len: Annotated[int, Strict(strict=True), FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=0)])] = 0, max_len: Annotated[int, Strict(strict=True), FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=0)])] = 9223372036854775807, on_missing_column: Literal['ignore', 'fail'] = 'ignore', on_wrong_type: Literal['ignore', 'fail'] = 'ignore', *, tags: str | list[str] | None = None, )