Deequ

Deequ is a library built on top of Apache Spark for defining “unit tests for data”, which measure data quality in large datasets. Constraints such as completeness, uniqueness, value ranges or patterns are declared on a dataset and verified when the data is processed.

Besides constraint verification, Deequ computes data quality metrics, suggests constraints by profiling the data, and stores metrics to detect anomalies over time. Releases are built for specific Apache Spark versions. Python users can use PyDeequ, a Python interface for Deequ.

Deequ is developed by Amazon Web Services (awslabs) and is open source under the Apache 2.0 licence.