Data Lineage for Avro Files: Convert and Keep a Record

Avro is common in data pipelines and hard to open by hand. Here is how it fits into a lineage you can show.

Open the Data Lineage

Short answer: Avro is handled by the Format Converter. Open an Avro file there, convert it to CSV, JSON or Parquet, and that step is recorded in Data Lineage like any other tool's. After that, the converted file can go through the other tools, and the chain keeps growing.

Where Avro fits

Avro is supported in the Format Converter only. Other tools, such as Data Transformation and Data Masking, work with CSV, JSON, Parquet and Excel. So a typical Avro workflow is: convert first, then clean or mask the converted file.

A typical chain

  • Avro file opened and converted to Parquet or CSV in the Format Converter.
  • The converted file filtered or joined in Data Transformation.
  • Sensitive columns masked in Data Masking.
  • The result downloaded with its lineage report.

What the lineage shows

  • The Avro input and where it came from.
  • The conversion step and its settings.
  • Every later step and its row count.
  • What happened to each column along the chain.

Why it is useful

Avro files usually come from a pipeline you cannot see into. When you convert one to look at it, a lineage records that the table you now hold came from that Avro file, and what you did to it afterwards.

Limits

  • Avro is a Format Converter format. The Result + lineage download offers CSV, JSON, Parquet and Excel.
  • Only steps done in Data Utility are recorded, not how the Avro file was produced upstream.
  • History is kept in your browser. Download the report to keep it.

Open the Data Lineage

Related tool pages

  • Data Lineage Version History: Keep a version history of your data lineage and compare any two versions: steps, settings, columns and row counts. Export a report. Free and private.
  • Free Data Lineage Tool: A free data lineage tool for files: transform data, see input, transformations and result as a graph, and download a shareable HTML report. No signup.
  • Data Lineage Report as HTML: Download a data lineage report as one HTML page: a tree from the input file through every transformation to the result, with a SHA-256 checksum.
  • Share Your Data Together with Its Lineage: Send a result file and the proof of how it was made in one ZIP: your data, an HTML lineage report and a JSON copy. Free, made in your browser.
  • Data Audit Trail for CSV and Excel Files: Keep an audit trail of what was done to a CSV or Excel file: conversions, cleaning, joins and masking, with the settings used. Free, in your browser.
  • Data Provenance Tool for Files: Record where a data file came from and every transformation applied to it. See it as a graph and export an HTML provenance report. Free, no upload.

Also try

How to use the Data Lineage: step by step

  1. Open the Avro file in the Format Converter.
  2. Convert it and save or download the result.
  3. Continue in other tools from Recent files if needed.
  4. Open Data Lineage and download Result + lineage.

When to use the Data Lineage

  • Record converting an Avro file to Parquet
  • Show what happened after an Avro export was opened
  • Document a CSV to Avro conversion
  • Explain an Avro to JSON step

Example

Input: customers.csv → Data Masking → Format Converter

Output: customers.json + LINEAGE.html (input → transformations → result)

Frequently asked questions

Can I track the lineage of an Avro file?

Yes. Convert it in the Format Converter and the step is recorded, along with every later step.

Which tools read Avro?

The Format Converter. The other tools work with CSV, JSON, Parquet and Excel.

Is my Avro file uploaded?

No. Conversion runs in your browser.

Does it record how the Avro file was made?

No. It records steps done in Data Utility only.

Tip: Convert Avro first, then use the other tools on the converted file.