Compare Two Avro Files

Compare two Avro files by a key field and list the added, removed and changed records. Reads the embedded schema, runs in your browser. Free.

Quick answer: Compare two datasets and see what was added, removed or changed.

Related: Data Comparison (CSV and Excel Diff) · Compare Two CSV Files · Compare Two Excel Files · Compare Two Parquet Files · Avro Guides and Free Avro Tools

How to use the Compare Two Avro Files

Compare two datasets and see what was added, removed or changed.

Data Comparison shows exactly what changed between two versions of a file. Pick a key column, or use the first column or the first two columns, and it lists rows that were added, removed or changed, down to the cell. Use it to check an export against last month, to review a data migration, or to find which values a script changed. CSV, Excel, JSON, Parquet and Avro are supported, and the two files can be different formats.

  1. Load dataset A and B
  2. Pick or type the key column
  3. Review and download the differences

Tips

Comparing Avro by record

Avro files carry their schema, so the tool reads the field names and types from the file and shows each record as a row. Pick a key field, such as an event id, and it lists the records that were added, removed or changed between two files, for example two days of an event log or a topic exported before and after a change.

Nested records, arrays and maps are flattened into columns, and date and timestamp types are read as dates.

Field names

Avro does not allow a dot in a field name, so a nested field such as address.city is stored as address_city. When you compare an Avro file with a CSV or JSON file, columns that differ only by case or punctuation are matched automatically, and the page lists the matches so nothing is hidden.

Example

Input: events-1.avro + events-2.avro, key = event_id

Output: Added 3 · Removed 2 · Changed 5

Browse by topic

Who uses Data Utility

Frequently asked questions · Data format benchmark

Frequently asked questions

How do I compare two Avro files?

Load both .avro files, choose a key field and read the added, removed and changed records. The schema is read from the file, and nested records are flattened into columns.

Which Avro files can it read?

Files with no compression or with deflate. Nested records, arrays, maps, enums, unions and date and timestamp types are read.

Why does an Avro column name use an underscore where my CSV has a dot?

Avro field names cannot contain dots, so address.city is stored as address_city. When you compare Avro with another format, columns that differ only by case or punctuation are matched and the page tells you which.

Which file formats can I compare?

CSV, TSV, Excel (.xlsx), JSON, Parquet and Avro, in any combination. For example, compare a Parquet file with the CSV it was made from. Nested JSON and Avro fields are flattened into columns such as address.city.

Will 10 in a CSV match 10 in a Parquet file?

Yes. Values are compared by meaning, not by how they are written: 10 equals 10.0, true equals "true", and a date equals the same date in any format. A value only counts as changed if it really differs.

Can Data Comparison do change data capture (CDC)?

It does the snapshot version of CDC for files. Compare the earlier and the latest export by key: added rows are inserts, changed rows are updates and removed rows are deletes. It does not read database logs or run in real time.

How does Data Comparison find differences?

It matches rows by a key column, or by the first two columns, and shows which rows were added, removed or changed, down to the cell. It works with CSV and Excel files.

Related pages

Topic guides

Guides