{"id":31884,"date":"2024-12-13T12:27:27","date_gmt":"2024-12-13T11:27:27","guid":{"rendered":"https:\/\/www.graviton.at\/letterswaplibrary\/can-we-automate-data-quality-assessment-process-for-small-datasets\/"},"modified":"2024-12-13T12:27:27","modified_gmt":"2024-12-13T11:27:27","slug":"can-we-automate-data-quality-assessment-process-for-small-datasets","status":"publish","type":"post","link":"https:\/\/www.graviton.at\/letterswaplibrary\/can-we-automate-data-quality-assessment-process-for-small-datasets\/","title":{"rendered":"Can We Automate Data Quality Assessment Process For Small Datasets?"},"content":{"rendered":"

<\/p>\n

\n

Recently, my friend and I have been thinking of working on a side project (for our portfolios) to automate data quality assessment for small tabular datasets that you often find in kaggle.<\/p>\n

We acknowledge that such a tool can’t be 100% accurate but it can definitely help nontech people and tech people to get started with working on their datasets. We aim to have a platform where the user will upload a dataset, the system will identify anomalies and give suggestions to the user with different ways to fix that anomaly (e.g. imputation of missing value, fixing an email that doesn’t follow the email pattern, etc).<\/p>\n

I would love to discuss the project further and get your thoughts on it. We have been researching similar projects and we found Cocoon, they use proceed column by column, and for each column they have a series of anomalies to fix using an LLM. But we want to have statistical methods for numerical columns, and use LLM only when it’s needed. Can anyone help?<\/p>\n<\/div>\n

submitted by \/u\/Better_Resource_4765 <\/a>
[link]<\/a><\/span> [comments]<\/a><\/span><\/p>