DataComPy

DataComPy is a Python package to compare two DataFrames or tables. It supports Pandas, Polars, Spark (classic and Spark Connect) and Snowflake.

It was originally created as a replacement for SAS PROC COMPARE. Beyond a simple equality check, it joins the two datasets on key columns, reports rows and columns present in only one of them, and gives statistics on mismatched values, with configurable tolerances for numeric comparisons. This makes it useful to validate data migrations, ETL pipelines or the output of data processing code.

DataComPy is developed by Capital One and is free and open source under the Apache 2.0 licence. It is installed with pip or conda.