I scanned the Tranco top 1,000,000 sites (June 2026) and recorded, per domain, which anti-bot/WAF vendor protects it and whether a plain request gets challenged. Releasing it as open data.

– 998,497 probed, 818,614 reachable

– Fields: domain, rank, reachable, protected, vendor(s), kind (waf/captcha/bot_management/…), difficulty band, block reason, enforcement, CAPTCHA type, final URL, status, probed_at — names only, no PII

– Plus a top-50k “deep-page census” (86,792 rows) with a page_type field (homepage vs product/listing/profile)

– License: CC BY 4.0

Headline: 53.5% of reachable sites run a managed anti-bot/WAF (Cloudflare ~45%), but only 9.8% actively challenged the request. The busiest sites run the least (top-1k 44% → long tail 54%).

Dataset (gzipped JSONL + sample + summary.json): https://github.com/Crawlora-org/anti-bot-adoption-index-data

Open-source detector CLI: go install github.com/Crawlora-org/crawlora-antibot@latest

I need dataset of all LinkedIn profiles. I know there are some paid sources for this but I want a free source. Reason I want a free source is because it makes no sense to pay for data, if I have to pay for data why can’t I then just sell that data for half price to other people after buying it ?

Hey,

I’m planning to add broad support for daily updated fund holdings!

Problem: SEC N-PORT data lags behind a LOOOOONG time when it comes to fund holdings.

Solution: Funds actually release holdings with much more up-to-date information on their website. It’s just a huge hassle to actually fetch them reliably.

If I were to say that I have found a reliable way to pull this off for a large and expanding set of funds, would you be interested in that kind of data?

Built a site that uses the Kaggle international football results dataset to compute Elo ratings and championship probabilities for World Cup 2026 in real time.
Layered on top: AI-generated match reports combining live data with news sentiment via OpenRouter.
Site: skorradar.live — the methodology is explained in the About section. Curious if anyone has thoughts on improving the Elo calibration for tournament play vs. friendlies.

hi! like my title states, I was wondering whether anyone has any good datasets of driver drowsiness or just drowsiness in general for south asian people? or Asians, actually, because my project is catered to a more minor demographic in my country (Sri Lanka). it would also be a major advantage if any of you could also help with datasets that have driver fatigue data in low-light conditions, or with people wearing glasses / sunglasses.

thank you! I’d really appreciate it 🙂

Disclosure: I work with a team that collects and licenses paid robotics training datasets.

I’ve been speaking with robotics teams about human demonstration data, and every team seems to evaluate it differently.

Some only need egocentric video, while others require synchronized wrist views, task labels, collection metadata and licensing documentation.

We currently have small evaluation samples from the US, UK and Australia, covering:

• Egocentric demonstrations
• Egocentric + two wrist views
• Task and step labels
• Country and collection metadata

The small evaluation samples are free, but the complete datasets and custom collection services are paid.

For teams working on robot manipulation or embodied AI, what do you normally check first?

Camera coverage, task diversity, collection country, metadata quality or licensing?

I’m mainly trying to understand what makes a sample genuinely useful before preparing more of them.

📊 Filipino Recipe Dataset — 1,574 Recipes I've compiled a clean, structured dataset of Filipino recipes scraped from a top Filipino recipe site. Perfect for food tech startups, recipe apps, meal planners, nutrition analysis, or AI training data. What's included: • 1,574 recipes spanning 2009–2026 • Complete ingredients list with measurements (every recipe) • Step-by-step cooking instructions (every recipe) • Full nutritional data per serving: calories, protein, fat, carbs, fiber, sugar, sodium, etc. (97% of recipes) • Prep time, cook time, total time • YouTube video links (31% of recipes) • User ratings and vote counts (28% of recipes) • Categories, cuisines, and keywords • High-resolution image URLs Data format: Clean JSON, ready to import into any application or database. Use cases: - Build a Filipino recipe search engine or mobile app - Train a recipe recommendation model - Analyze Filipino cuisine nutrition trends - Power a meal planning or grocery list tool - Academic research on Southeast Asian food culture DM me if interested. Can provide a sample file upon request. 

I am writing a thesis. For this I need a data set which includes the effects of environmental conditions on solar panel energy output. This includes things like cloud cover temperature wind precipitation atmospheric pressure etc.

​

If anyone knows where I can get a large data set with all of this, I’d appreciate it.

Sharing a dataset I built. Disclosure: this is my project. Free to download and use.

https://huggingface.co/datasets/fineset-io/model-merging-papers

Stats:

– 381 records, 2021–2026

– Sources: arXiv + Semantic Scholar, cross-referenced by arxiv_id and DOI

– quality_score: 0-1, citation-normalized

Fields: id, title, abstract, authors, categories, published_date,

citation_count, quality_score, has_code, code_url, venue

The most-cited paper in the set is “Model soups: averaging weights of multiple

fine-tuned models improves accuracy without increasing inference time” (1,565 citations,

2022); if you’re doing any merging work this is probably already in your reading list,

but the rest of the dataset has 380 more.

109 papers have code repos; filter has_code=true if you want reproducible implementations.

Built with FineSet (fineset.io). Sign up free to get daily-refreshed datasets on your own topic.

Just finished a few weeks of testing proxy providers for a project that needs accurate location data. pulling localized pricing, so if the geo is wrong the whole thing is useless.
Short version: Most of the advertised coverage numbers are pretty meaningless. had requests that allegedly originated from some cities in completely different areas. not like a little bit off, like wrong country level off on a couple of them.
Across all of the providers I tested, ASN targeting was far more reliable than city targeting. If you need location accuracy that’s probably where to start rather than trusting city-level claims.

One provider did truly better than the rest on consistency. Happy to chat through what I found if anyone has the same problem.

Preferably from the big American vendors if possible (ex. target, walmart, costco, safeway, albertsons, etc.). Need this info for OCR work. It’s also fine if the grocery receipts are part of a dataset that includes all kinds of receipts.

I’m researching skilled labor shortages in construction and related industries.

Looking for public or commercial datasets covering:

• Electricians
• Project Managers
• Construction workforce demographics
• Apprenticeship enrollment
• Retirement risk
• Regional wage inflation
• Infrastructure project activity

Any recommendations beyond BLS, Census, ACS, and OEWS?

Hello,

I’m doing some research and need help getting recent public statistics from the EMISS portal on foreign nationals entering the Russian Federation. The portal is unfortunately not accessible from my location. The site is fedstat[dot]ru.

Specifically looking for the dataset titled approximately:
“Численность иностранных граждан, въехавших в Российскую Федерацию, по странам гражданства и целям поездок”

Filtered by Tajikistan as country of citizenship, for at least 2024–2025.

If anyone has access and can export the Excel table, I would be very grateful if you could share it! Спасибо вам большое!!

We are having a hard time finding videos for our thesis. We visited most of the social media platforms and so far, we still haven’t managed to reach our goal. Maybe you guys can recommend me an archive website or something.

TL;DR: short-horizon crypto direction on Polymarket is NOT where the edge is. I tested entries at candle birth, mid-candle, T-180s, T-60s, and the “deathbed” — the book charges ~fair value (incl. fees) at every static point. The deathbed-favorite edge people mention is basically closed (priced to ~$1.00).

​

Where edge actually showed up: execution/structure (nested-market consistency, late-window favorites with real leads), favourite-longshot bias (longshots overpriced, favorites underpriced), and LP rewards — not price prediction.

​

Happy to share the fair-value model I used (driftless P(up) from realized move + time + vol) in the comments. Anyone found the same?

I am currently working on a paper. So I need atleast 1 year of uber movement dataset of any city possible. Any suggestions? Found in kaggale but could find only 2017 oct to 2017 november. So can someone please help me with it

DeepTech Venture Capital Firms — firm websites, investment stages, sectors, office locations, and portfolio links. Structured from recent funding activity.

https://deeptechvclist.com

Helloo

I have been working in the branch of lip reading for a while now. Currently there are about 100000 videos with youtube ids, start time, and end time of the clip. I am constantly working to reduce the friction in the dataset — as we cannot share the actual video clips from youtube — by adding download scripts and the actual transcripts in the near future.

I have transcripts ready of about 80000 videos. The rest are yet to be made but since the dataset is constantly expanding (150,000 ish by end of day), transcripts would lack behind until I am done with the actual videos.

Also trying to figure out how to not get rate-limited when downloading the videos from youtube using yt-dlp. If anyone knows, please enlighten me a bit 🙂.

My core aim is to make this a standard like LRS2,LRW,LRS3 etc.

I will soon add a commercial subset in the dataset. Made from youtube videos which specifically allow commercial use so if someone wants to make a hardware out of it and bring it into the market, they can wholeheartedly do so :D.

That’s mostly it.

Have a look at the dataset if you would like to 😀

huggingface.co/datasets/Rizul2159/WildVid-LIP

There isnt much right now on it. Just a csv file with 115k videos with their ids and timestamps but soon there would be a lot more than that.

I’m 18, from Tunisia, and I built this because nobody else had.

Tunisian Darija is what 12 million Tunisians actually speak. Not Modern Standard Arabic. Not Moroccan. A separate dialect that borrows from Arabic, French, Italian, and Amazigh, written online in Arabizi Latin letters with numbers for Arabic sounds (3→ع, 7→ح, 9→ق, 5→خ).

When I searched for a parallel corpus to build a translation model, I found nothing. TUNIZI covers sentiment analysis. TunBERT does dialect classification. But zero parallel datasets existed for Tunisian Darija-to-English translation. Not one.

So I built the first one from scratch with no funding, no university affiliation, no mentor, and no institutional support. Just me, a laptop, and the language I grew up speaking.

The first 500 pairs came from my own memory as a native speaker, covering 50 categories of real Tunisian daily life cafe culture, Ramadan traditions, wedding customs, bac exam stress, barbershop talk, louage rides, haggling at the medina, football arguments, bureaucracy nightmares, olive harvest season, Friday afternoon naps, and more. Zero automated generation. Every pair hand-written and validated.

Then I left my desk and started collecting from real people:

• My father’s childhood memories growing up in Ain Draham, a mountain village in northwestern Tunisia the scent of the forest, nearly getting bitten by a snake, his cousin falling off his uncle’s horse
• My grandmother’s stories about her father’s farm cows, sheep, thieves stealing the neighbors’ animals at night, and her father calmly finishing his morning prayer before stepping outside to check
• An elderly man from Siliana I met at a cafe who speaks a dialect I barely recognized — words I had to ask about, rhythms I’d never heard

Every pair is provenance-tagged with its source: self, family-father, family-grandmother, community-siliana. Every collection session is logged with date, place, speaker context, and consent status.

I excluded an entire session of data because I hadn’t established consent before the conversation began. The language was rich. I threw it all away anyway. A dataset built on trust means sometimes throwing away good data.

What this dataset has that scraped corpora don’t:

• Regional dialect diversity: urban , mountain Ain Draham, rural Siliana
• Generational variation: grandmother’s speech vs mine
• Provenance: every pair traces to a known speaker, region, and context
• Documented ethics: consent logged, exclusions documented, no anonymous mass scraping

I trained the first Tunisian Darija-to-English translation model on this dataset a 15.6M parameter Transformer built from scratch on an RTX 3050 (4GB VRAM). v1 BLEU: 3.89 on a held-out test set. Low, but the first benchmark ever measured for this language. A published ACL researcher who found my work on Reddit said it’s ‘basically guaranteed to be novel.’

I’m heading toward 1,000+ pairs through continued community collection and will be presenting this research at Tunisia’s AI National Summit (AINS 4.0) later this month the first high schooler to ever present at the event.

The dataset is CC BY-NC-SA 4.0 and public on HuggingFace. 110+ downloads so far.

If you work on low-resource NLP, Arabic dialect processing, or sociolinguistic data it’s yours.

HuggingFace: huggingface.co/datasets/Dhiadev-tn/tunisian-darija-english
Full pipeline + model: github.com/Dhiadev-tn/darija-translator

Sharing a dataset of 233 used car listings I pulled from AutoTrader.ca this week. All records are from dealer listings (no private sellers, so no personal contact info).

Fields per record (PII removed from this sample):

• Price (CAD, formatted + numeric + average market price for comparison)
• Specs: make, model, year, trim, body type, drivetrain, transmission, color, displacement, doors, cylinders
• Mileage (formatted + numeric km)
• Location: city, postal code, latitude, longitude
• Equipment by category: comfort, safety, entertainment, extras
• History: accident-free flag, Carfax URL, rental flag
• Images: URLs (1280×960)

Sample (3 records, contact fields removed):

[ { "data_source": "AutoTrader.ca", "ad_id": "264a7bb7-5b85-4b0c-9420-b87783a41389", "make": "Mazda", "model": "CX-5", "year": 2024, "trim": "Signature AWD – BOSE Sound", "body_type": "SUV", "status": "Used", "price_cad": 39900, "price_formatted": "$ 39,900", "average_market_price": 37600, "mileage_km": 29454, "mileage_formatted": "29,454 km", "transmission": "Automatic", "drivetrain": "All Wheel Drive", "exterior_color": "Red", "interior_color": "Brown", "fuel_type": "Gasoline", "displacement": "2,500 cc", "doors": 4, "cylinders": 4, "city": "NORTH VANCOUVER", "zip_code": "V7P 3R8", "country": "CA", "latitude": 49.3165, "longitude": -123.09942, "seller_name": "Morrey Mazda of the Northshore", "dealer_google_rating": 4.5, "accident_free": true, "comfort_equipment": ["Automatic climate control", "Cruise control", "Heads-up display", "Heated steering wheel", "Navigation system"], "safety_equipment": ["Adaptive Cruise Control", "Electronic stability control", "Lane departure warning system"], "image_count": 34, "created_timestamp": "2026-04-18T07:43:14.098Z" }, { "data_source": "AutoTrader.ca", "ad_id": "ec42fc58-8459-457c-a9a8-54638894a694", "make": "Mazda", "model": "CX-5", "year": 2024, "trim": "GS AWD | Heated Leather", "body_type": "SUV", "status": "Used", "price_cad": 27994, "price_formatted": "$ 27,994", "average_market_price": 30300, "mileage_km": 49984, "mileage_formatted": "49,984 km", "transmission": "Automatic", "drivetrain": "All Wheel Drive", "exterior_color": "Grey", "fuel_type": "Gasoline", "doors": 4, "cylinders": 4, "city": "Fredericton", "zip_code": "E3C 1N8", "country": "CA", "latitude": 45.94504, "longitude": -66.68895, "seller_name": "ReCar", "dealer_google_rating": 4.5, "accident_free": true, "comfort_equipment": ["Air conditioning", "Cruise control", "Leather steering wheel", "Power windows"], "safety_equipment": ["Anti-lock braking system (ABS)", "Electronic stability control", "Traction control"], "image_count": 18, "created_timestamp": "2026-04-24T19:47:48.215Z" }, { "data_source": "AutoTrader.ca", "ad_id": "bd822421-6d67-47ac-a079-69b129aea48f", "make": "Mazda", "model": "CX-5", "year": 2024, "trim": "GS", "body_type": "SUV", "status": "Used", "price_cad": 31757, "price_formatted": "$ 31,757", "average_market_price": 30000, "mileage_km": 66855, "mileage_formatted": "66,855 km", "transmission": "Automatic", "drivetrain": "All Wheel Drive", "exterior_color": "White", "fuel_type": "Gasoline", "doors": 4, "cylinders": 4, "seats": 5, "city": "Mississauga", "zip_code": "L5L1X3", "country": "CA", "latitude": 43.53093, "longitude": -79.67701, "seller_name": "Erin Mills Mazda", "dealer_google_rating": 4.2, "accident_free": true, "carfax_url": "https://vhr.carfax.ca/?id=2GpEicFIk9VsxXw/rcTLBLxhbymmt8Oz", "image_count": 19, "created_timestamp": "2026-04-02T09:26:07.098Z" } ] 

Collected via AutoTrader.ca’s public search pages. Happy to share more records or answer questions about the fields.

Sharing a dataset I recorded because nothing like it seems to exist publicly: the order book
of Polymarket’s 5-minute crypto up/down markets, sampled once per second.

• ~89,000 markets across 7 coins (BTC, ETH, SOL, XRP, DOGE, HYPE, BNB)
• ~26.8M per-second rows (~300 per market), Mar–May 2026, UTC
• Two Parquet tables per coin, joined on `condition_id`: `markets` (one row per 5-min market) and `ticks` (one row per second)
• Per tick: best bid/ask, resting sizes, and bid-side 5¢ depth for both the Up and Down outcome – ~725MB total, 99.8%+ coverage, no duplicates
• Licence: CC0 (public domain)

Caveats up front: fixed window (collection ended 18 May 2026), outcome is inferred from
the final tick rather than read on-chain, ask-side depth isn’t recorded, and there are ~1.5h
of collector outages over the span (shared across all coins, so collector hiccups rather
than market-data loss). Full data dictionary and coverage audit are in the write-up.

Hugging Face: https://huggingface.co/datasets/kachoio/polymarket-5-minute-crypto-up-down-markets
Kaggle: https://www.kaggle.com/datasets/kachoio/polymarket-5-minute-crypto-updown-markets
Write-up (schema, provenance, limitations): https://kacho.io/polymarket-5min-crypto-dataset

**Self-promotion**

Hi r/remotesensing,

I’m sharing DeepPVMapper, an open-source tool we developed to detect and characterize rooftop PV systems from very high-resolution aerial imagery (IGN orthophotos, 20cm).

What’s available:

• Model weights on HuggingFace: huggingface.co/gabrielkasmi/bdappv-models
• Interactive demo (no GPU, ~1 min/km²): huggingface.co/spaces/gabrielkasmi/deeppvmapper
• Training dataset (45k+ images, segmentation masks): huggingface.co/datasets/gabrielkasmi/bdappv
• Full detections for France (~500k systems, GeoJSON): https://zenodo.org/records/19188878
• Code: github.com/gabrielkasmi/deeppvmapper

What it does:
Detects rooftop PV panels and estimates surface area, installed capacity, tilt and azimuth. Deployed at national scale across France — evaluation against official registries (RTE, RNI) revealed 10% missing capacity nationally.

The repo has been refactored and is open to contributions. Happy to discuss methodology, limitations, or potential extensions.

Project page: gabrielkasmi.github.io/deeppvmapper

Disclosure: I built this project and this is my own API/product. It has free and paid access tiers. I’m sharing it here because I think the data engineering approach may be useful, and I’m looking for technical feedback.

I built Trace Jobs Core, a job postings data API built around a simple idea: Do not guess.

A lot of job data pipelines end up doing some combination of:

• scraping HTML pages
• parsing unstable frontend output
• using models to extract fields
• guessing missing/ambiguous values
• deduplicating after the fact

I took a different approach.

The pipeline ingests job postings from public machine-readable sources, translates them into a Schema.org JobPosting format, applies only deterministic normalization where the source provides clear structure, and preserves original values when fields are ambiguous.

Current system:

• 9,800+ structured feeds
• ~13k new postings/day
• daily refresh
• Schema.org JobPosting records
• SHA-256 based deduplication
• RFC 8785 canonicalization
• original upstream values preserved when normalization is uncertain

The goal is not to create a “smart” interpretation layer. The goal is to provide stable, predictable data and leave interpretation to the downstream user.

A future enrichment layer could exist separately, but it would remain separate from the source-faithful data layer.

Examples (HTML + JSON responses refreshed daily):
https://kaleh.net/trace/examples.html

Documentation:
https://kaleh.net/trace/docs.html

Project overview:
https://kaleh.net/trace/

I would especially appreciate feedback on:

• dataset design
• normalization strategies
• preserving source fidelity
• handling schema differences between providers
• what fields/data would make this more useful

Thanks!

So I have access to a vast network of farms and farm workers and have been looking into collecting videos to sell as data sets to AI labs etc. I’ve done research and noticed that it’s hard to find quality data sets specifically in agriculture. A lot of the video data is either from a vehicle moving at a higher speed (which also lacks hand to object interaction) or is simply a birds eye view. I realized I have an opportunity and have started working on it and sending basic outreach to dataset licensing and a few agtech startups. I was curious if anyone has experience in this sort of field?

For video gathering I’ve already found and set up a set of glasses that are able to get the job done. I’ve tested them and have sample videos ready. If you have any advice or tips that would greatly appreciated!

I ran a benchmark for a tool I built and figured the dataset might be useful to others. It took ~$100 of API credits to produce.

The test is simple: I give the agent a document describing a piece of code it can’t directly see, then record whether it double-checks the doc against the real code or just takes the doc’s word for it. The doc is sometimes accurate and sometimes out of date, so the data captures how each model handles documentation it can and can’t trust. The writeup covers what I found; the dataset lets you check it or look for your own patterns.

Dataset
Outcome

Star the repo if it’s useful. Cheers.