Generate Test Data for ETL and Pipeline Testing

A pipeline that has only seen clean sample data will break on the first real file. Here is how to test it harder.

Open the Schema Data Generator

ETL tests need volume, the right shape and awkward cases. Production copies bring privacy risk and are hard to refresh. A small hand-made file misses scale problems.

Use the Schema Data Generator with the CREATE TABLE of the source table to get rows that fit each column type and rule, up to 500,000, and download CSV, JSON, Parquet or Excel in the format your pipeline reads. If you have a real sample you are allowed to use, the Synthetic Data Generator can create new rows that follow its patterns.

Then break it on purpose. Add rows with nulls, wrong types, duplicate keys, very long values and unusual characters, and check how the job behaves: does it reject, quarantine or fail? Compare input and output counts with the Data Profiler, and use Data Comparison to compare an expected output file with the real one.

Remember what generated data cannot do. It does not reproduce real data skew or the odd values found in production. Use it to test mechanics and error handling, and still run a check on a real sample before you go live.

Open the Schema Data Generator

Related tool pages

  • Generate Data from CREATE TABLE: Paste a CREATE TABLE and get up to 500,000 rows of realistic test data that fit every column type and rule. Free, and runs in your browser.
  • JSON Schema Data Generator: Paste a JSON Schema and generate realistic sample data that follows it: formats, enums, ranges and nested objects. Free, in your browser.

How to use the Schema Data Generator: step by step

  1. Paste the source CREATE TABLE into the Schema Data Generator.
  2. Download the file in your pipeline format.
  3. Add bad rows by hand.
  4. Run the job and compare outputs with Data Comparison.

When to use the Schema Data Generator

  • Test a loader at volume
  • Check error handling with bad rows
  • Compare expected and actual output
  • Create files in the formats a job reads

Example

Input: CREATE TABLE users (id INT PRIMARY KEY, email VARCHAR(100), age INT);

Output: 1, ada.chen7@example.com, 34

Frequently asked questions

Which databases are supported?

CREATE TABLE from PostgreSQL, MySQL, SQL Server, Oracle and SQLite. Comments, quotes and schema names are handled.

Are foreign keys checked?

No. A foreign key column gets a number from 1 to 100, so load the parent table with ids 1 to 100 first.

Which JSON Schema versions work?

Draft 7 and newer for the common keywords listed above. Unknown keywords are ignored.

Is the data real?

No. It is random and fictional, so it is safe to share.

Tip: Test what the job does with a bad row: reject, quarantine or fail.