Testing data pipelines with realistic data is something I\u2019ve struggled with in several projects. In many environments, we can\u2019t use production data because of privacy constraints, and small handcrafted datasets rarely capture the complexity of real schemas (relationships, constraints, distributions, etc.). <\/p>\n
I\u2019ve been experimenting with a schema-driven approach to synthetic data generation and wanted to get feedback from others working on data engineering systems. <\/p>\n
The idea is to treat the **schema as the source of truth** and attach generation rules to it. From that, you can generate datasets that mirror the structure of production systems while remaining reproducible. <\/p>\n
Some of the design ideas I\u2019ve been exploring: <\/p>\n
\u2022 define tables, columns, and relationships in a schema definition <\/p>\n
\u2022 attach generation rules per column (faker, uuid, sequence, range, weighted choices, etc.) <\/p>\n
\u2022 validate schemas before generating data <\/p>\n
\u2022 generate datasets with a run manifest that records configuration and schema version <\/p>\n
\u2022 track lineage so datasets can be reproduced later <\/p>\n
I built a small open-source tool around this idea while experimenting with the approach. <\/p>\n
Tech stack is fairly straightforward: <\/p>\n
Python (FastAPI) for the backend and a small React\/Next.js UI for editing schemas and running generation jobs. <\/p>\n
If you\u2019ve worked on similar problems, I\u2019m curious about a few things: <\/p>\n
\u2022 How do you currently generate realistic test data for pipelines? <\/p>\n
\u2022 Do you rely on anonymised production data, synthetic data, or fixtures? <\/p>\n
\u2022 What features would you expect from a synthetic data tool used in data engineering workflows? <\/p>\n
Repo for reference if anyone wants to look at the implementation: <\/p>\n
[https:\/\/github.com\/ojasshukla01\/data-forge](https:\/\/github.com\/ojasshukla01\/data-forge)<\/a><\/p>\n<\/div>\n submitted by \/u\/Business-Quantity-15 <\/a>
[link]<\/a><\/span> [comments]<\/a><\/span><\/p>
