Hey all,<\/p>\n
I just released a dataset I’ve been working on: a sentence-level corpus extracted from the entire Hebrew Wikipedia. It’s up on HuggingFace now:<\/p>\n
https:\/\/huggingface.co\/datasets\/tomron87\/hebrew-wikipedia-sentences-corpus<\/a><\/p>\n Why this exists:<\/strong> Hebrew is seriously underrepresented in open NLP resources. If you’ve ever tried to find a clean, large-scale Hebrew sentence corpus for downstream tasks, you know the options are… limited. I wanted something usable for language modeling, sentence similarity, NER, text classification, and benchmarking embedding models, so I built it.<\/p>\n What’s in it:<\/strong><\/p>\n Pipeline overview:<\/strong> Articles were fetched through the MediaWiki API, then run through a rule-based sentence splitter that handles Hebrew-specific abbreviations and edge cases. Deduplication was done at both the exact level (SHA-256 hashing) and near-duplicate level (MinHash).<\/p>\n I think this could be useful for anyone working on Hebrew NLP or multilingual models where Hebrew is one of the target languages. It’s also a decent foundation for building evaluation benchmarks.<\/p>\n I’d love feedback. If you see issues with the data quality, have ideas for additional metadata (POS tags, named entities, topic labels), or think of other use cases, I’m all ears. This is v1 and I want to make it better.<\/p>\n<\/div>\n submitted by \/u\/tomron87 <\/a> \n
[link]<\/a><\/span> [comments]<\/a><\/span><\/p>
