Hello Redditors!<\/p>\n
I’m excited to share Datalab<\/strong> \u2014 a linter<\/em> for datasets.<\/p>\n I recently published a blog<\/a> introducing Datalab<\/strong> and an open-source<\/a> Python implementation that is easy-to-use for all data types (image, text, tabular, audio, etc). For data scientists, I\u2019ve made a quick Jupyter tutorial<\/a> to run Datalab<\/strong> on your own data.<\/p>\n All of us that have dealt with real-world data know it\u2019s full of various issues like label errors, outliers, (near) duplicates, drift, etc. One line of open-source code datalab.find_issues() automatically detects all of these issues.<\/p>\n In Software 2.0, data is the new code, models are the new compiler, and manually-defined data validation is the new unit test. Datalab<\/strong> combines any ML model with novel data quality algorithms to provide a linter<\/em> for this Software 2.0 stack that automatically analyzes a dataset for \u201cbugs\u201d. Unlike data validation<\/em>, which runs checks that you manually define via domain knowledge, Datalab adaptively checks for the issues that most commonly occur in real-world ML datasets without you having to specify their potential form. Whereas traditional dataset checks are based on simple statistics\/histograms, Datalab\u2019s checks consider all the pertinent information learned by your trained ML model.<\/p>\n Hope Datalab helps you automatically check your dataset for issues that may negatively impact subsequent modeling — it’s so easy to use you have no excuse not to \ud83d\ude1b<\/p>\n Let me know your thoughts!<\/p>\n<\/div>\n submitted by \/u\/jonas__m <\/a>
[link]<\/a><\/span> [comments]<\/a><\/span><\/p>
