Compare Two Parquet Files

Compare two Parquet files by a key column and see every added, removed and changed row and cell. Free, runs in your browser, nothing is uploaded.

Quick answer: Compare two datasets and see what was added, removed or changed.

Related: Data Comparison (CSV and Excel Diff) · Compare Two CSV Files · Compare Two Excel Files · Compare Two JSON Files · Parquet Guides and Free Parquet Tools

How to use the Compare Two Parquet Files

Compare two datasets and see what was added, removed or changed.

Data Comparison shows exactly what changed between two versions of a file. Pick a key column, or use the first column or the first two columns, and it lists rows that were added, removed or changed, down to the cell. Use it to check an export against last month, to review a data migration, or to find which values a script changed. CSV, Excel, JSON, Parquet and Avro are supported, and the two files can be different formats.

  1. Load dataset A and B
  2. Pick or type the key column
  3. Review and download the differences

Tips

Why compare Parquet files

Parquet is what pipelines write, and you cannot open it in a text editor to check it. Comparing two Parquet files answers the questions that come up after a pipeline change: did the new run produce the same rows as the old one, which records are new, and which values moved. Pick a key column such as an id and every row is matched to its twin in the other file.

The result lists added, removed and changed rows, and for each changed row the columns that differ, with the old and new value side by side. You can download the differences.

Types and size

Parquet keeps real types, so numbers and dates are compared as numbers and dates. Integers and decimals compare by value, and a timestamp at midnight matches the same date. Files are read in your browser, so very large Parquet files depend on your computer memory: for files of several hundred megabytes, compare a sample or a filtered part.

Example

Input: jan.parquet + feb.parquet, key = id

Output: Added 3 · Removed 2 · Changed 5

Browse by topic

Who uses Data Utility

Frequently asked questions · Data format benchmark

Frequently asked questions

How do I compare two Parquet files?

Load both .parquet files, choose a key column such as id and read the added, removed and changed rows. You can download the differences. No Spark, Python or parquet-tools needed.

Can I compare a Parquet file with a CSV?

Yes. Any two supported formats can be compared, for example a Parquet file against the CSV it was exported from. Values are compared by meaning, so 10 and 10.0 match.

Are my Parquet files uploaded?

No. The files are read in your browser and never leave your computer.

Which file formats can I compare?

CSV, TSV, Excel (.xlsx), JSON, Parquet and Avro, in any combination. For example, compare a Parquet file with the CSV it was made from. Nested JSON and Avro fields are flattened into columns such as address.city.

Will 10 in a CSV match 10 in a Parquet file?

Yes. Values are compared by meaning, not by how they are written: 10 equals 10.0, true equals "true", and a date equals the same date in any format. A value only counts as changed if it really differs.

Can Data Comparison do change data capture (CDC)?

It does the snapshot version of CDC for files. Compare the earlier and the latest export by key: added rows are inserts, changed rows are updates and removed rows are deletes. It does not read database logs or run in real time.

How does Data Comparison find differences?

It matches rows by a key column, or by the first two columns, and shows which rows were added, removed or changed, down to the cell. It works with CSV and Excel files.

Related pages

Topic guides

Guides