I checked out the word prevalence dataset<\/a> of 62,000 lemmas. But it has some limitations:<\/p>\n It hasn’t been updated since 2019.<\/p>\n<\/li>\n It misses modern terms like TikTok.<\/p>\n<\/li>\n It doesn’t cover phrases.<\/p>\n<\/li>\n<\/ul>\n I’ve scored<\/a> about a million English entries from Wiktionary for recognizability. I built this for a pun tool<\/a>. But I want to use the data<\/a> for a new language project.<\/p>\n The dataset is too bloated because it’s full of inflected forms. Even if I set the recognizability threshold at 50 percent, I’m still looking at 100K words and 100K phrases. Going through a list that size is a waste of time. I need to filter the data through the English lemmas category<\/a> from Wiktionary and split the single words from the multi-word phrases into separate lists.<\/p>\n Since the hard part of scoring is done, the rest should be easy peasy lemma squeezy. I just want to avoid reinventing the wheel if I can.<\/p>\n Before I spin up a separate repository to handle this, I’m checking if a similar dataset already exists. Has anyone seen a project that offers this?<\/p>\n<\/div>\n submitted by \/u\/8ta4 <\/a> \n
[link]<\/a><\/span> [comments]<\/a><\/span><\/p>
