Hey fellow datasets enthusiasts!<\/p>\n
I’ve developed a robust public data collection engine that’s been quietly amassing an impressive dataset, and I’m curious about its potential applications and demand.<\/p>\n
The Dataset<\/h1>\n
Scale<\/strong>: Over 2 billion data points, with 10 million added per day (4 billion per year at our current rate) Sources<\/strong>: Diverse and challenging public social media sources (X, Reddit, BlueSky, Youtube, Mastodon, Lemmy, TradingView, bitointalk, jeuxvideo.com<\/a>, etc.) (6000+ sources) Collection<\/strong>: Near real-time capture Rich<\/strong>: Structured, and annotated with translation, emotions, sentiment, top_keywords, topics. <\/p>\n We are an emerging, small startup, and of course I’m not trying to do self promotion, so won’t write the link\/name (PM me for that).<\/p>\n I was thinking of opening datasets on Hugginface. I could do several & in various forms, I wanted to know what this community would be most interested in?<\/p>\n Possibilities are:<\/p>\n – A full slice of 1 day of data, with all annotated\/attributes<\/p>\n – A sampled set of 1 source (for example X dataset, Reddit dataset) up to like 10 million items<\/p>\n – etc.<\/p>\n What would be interesting to you all? We want to do a genuine gift to the Open Source community, especially since Twitter\/X shut down its free API & locked out 99.99% of OSINT\/researchers.<\/p>\n<\/div>\n submitted by \/u\/askolein <\/a>
[link]<\/a><\/span> [comments]<\/a><\/span><\/p>
