Made a resource for student researchers aiming to ideate, or improve upon their existing science fair projects.

Hey everyone,

I am working on an indie web game called FeatFighter where players link artists together through their collaborations (like a music version of Six Degrees of Separation).

My goal is to cover music artists globally across all genres. To make the gameplay smooth, I need a massive initial database of tracks with explicitly linked primary and featured artists.

I would love to get your advice, ideas, or recommendations on how to approach this:

1. Are there existing open datasets, dumps, or graph database exports (like MusicBrainz, Wikidata, or Kaggle projects) that already capture artist collaborations well on a global scale?

2. If you had to build a global music collaboration graph today, what strategy would you use for the initial data load to avoid getting blocked by rate limits on streaming APIs?

Any suggestions, links, or technical insights on how to tackle this would be super helpful.

Thanks a lot for your help!

i’m not sure if this is the right thread for this question but where can i find a list of available usernames? i’ve been going back and fourth with support about a username that’s showing available but can’t claim it due to it being “reserved” according to support.

I released a public crypto microstructure dataset for liquidity-stress prediction. I’m looking for feedback on whether the features/labels are useful.

Check it out here: https://huggingface.co/datasets/Goooddy/crypto-lob-stream

I’m looking to build out 30 different use cases for 30 days where I pick up real data from multiple domains for eg – finance, technology, gaming etc. I want to run the data through a business reporting tool I came across and want to convince my peers/seniors in my firm that it’s a great product.

My real question – which all domains should I focus on and where can I real messy data for each of those domains. Any help is appreciated. Thanks.

Hi all,

I was recently working on a machine learning research project and I came across the WILDS paper. The dataset IWildCam seems very interesting for my project and I wanted to experiment with it. Unfortunately I found that currently the download page seems broken. Am I missing something or is that really the case? That would be very unfortunate.

Has anyone recently used that dataset?

I’m building a structured dataset of real-world locations represented in video games.

The current dataset contains close to 3,000 records from more than 380 games.

Two example records look like this:

record_id,game_title,release_year,franchise,primary_genre,location_name,country_iso,latitude,longitude,location_type,setting_status,coordinate_method,verification_level

acw12_0001,Assassin’s Creed,2007.0,assassins_creed,Action Adventure,Jerusalem,IL,31.7683,35.2137,city,real,representative,verified

bra49_0006,2014 FIFA World Cup Brazil,2014.0,fifa_world_cup,Sports,Arena das Dunas,BR,-5.8267,-35.2125,stadium,real,exact,corroborated

The two rows illustrate an important distinction: Arena das Dunas can be represented by an exact point, while Jerusalem is represented by a point that stands for a larger city.

I would appreciate feedback from people who work with public datasets:

1. Should a public preview include source URLs and evidence, or is it reasonable to keep those in a separate research layer?
2. Which additional information would make the records easier to validate or reuse?
3. What kinds of analyses or applications would you personally build with this dataset?

The goal is to improve the data structure and documentation through feedback from people who might actually use it.

Thanks for your feedback 🙂

we are doing research that involves training our model on really big dataset of diffusion technique based synthetic images. but I am unable to trace appropriate free and active Creative common data hubs for it.
please anyone help me work around this

I run freehire, an open-source job aggregator that ingests postings from dozens of ATS platforms (Greenhouse, Lever, Ashby, Workday, etc.), normalizes them into one schema, and runs each through a facet pipeline.

Just exported the whole catalogue to Hugging Face: freehire-jobs — 6,041,471 postings.

Each row is a raw posting (title, company, description, URL, source, posted_at) plus derived facets:

– Dictionary-only (deterministic, no LLM): skills, seniority, category, work_mode, posting_language, employment_type, education_level, english_level, experience_years_min

– Dictionary-first, LLM-filled: countries, regions, cities

– LLM-only: salary_min/max, currency, period — pay is stated in every format imaginable, no dictionary handles that

Company rows carry their own facets too (industries, HQ country, size, YC batch/stage where applicable).

Format: 20 gzip JSONL shards, split by row range for easy streaming/resuming.

Backstory: I originally wanted to train a cheap classifier for seniority/category tagging instead of paying for an LLM call per job. Turned out the dictionary+LLM pipeline already in prod beats a from-scratch classifier by enough that finishing the classifier wasn’t worth it — so I’m sharing the data instead.

Data includes:

lien_number, debtor_name, address, business_owner_name(s), filing_date, status, secured_party, lien_id

I need last 2 yrs of DAX LTP data, 5 second time frame would be perfect but if not even 1 min or 5 min is fine. If you know where I can collect this data from please lmk. Thank you.

Hi everyone, First time posting here. Can i get some datasets in excel to clean and analyze it? I’m just practicing it to enhance my skills. And also where can i get more datasets aside kaggle and creating from chatgpt? Thanks!

Data. Like a professional excel sheet for like ram prices with relation to Al. This is for my college project. It need to have numbers atleast 25 samples. It is for economics.

Hey y’all,

I’m Luke, and I built Stackra for some context. Not selling a ding o dang thing here, just looking to see what this dataset is really capable of.

The corpus is 15 million sites pulled from HTTP, CommonCrawl, CrUX, and Overture.

I have a few thousand elements mapped out, like analytics, tools, vendors, link value, site category, schema, plugins, you name it. It runs back to 2024, and I’ve started putting some of this data user facing.

I find things like being able to identify that the market for funeral homes is dominated by two vendors, that I can identify site directories and their value/trajectory, that custom built websites pass CVW at a higher rate than WP, but rarely crack the top rankings without investor backing.

Weird little stuff like an emulator site that has been highjacked by porn/gambling sites. A popular listing tool that claims their links provide DR, but are explicitly UTM and nofollow. Pinpointing an ICP for a small business from clustering shared linked domains.

I’m curious to see what it can do, what would you want to ask?

hey, working on a research thing where we estimate flood depth from traffic cameras using signs/poles as reference.

problem is i can find live flood cams (atxfloods, sunny day flooding, san diego cams, fl511 etc) but almost nothing archived where the road is actually flooded AND a sign/pole is clear enough to measure from.

if anyone’s seen a dataset, old webcam dumps, youtube clips, or even just a few screenshots like that, drop a link. would help a lot.

I’m researching publicly available and commercial datasets for grocery prices across Europe.

– Has anyone worked with data like this?

– What sources did you end up using, and what were their limitations?

Hello everyone! I would like to ask, I am currently building a platform, which will contain large amount of datasets about a particular topic. The main purpose of the platform will be datasets. I would like to ask, how do I monetise and make money off the datasets? I am also considering of open sourcing the datasets, with a copyleft license.

I just wrapped up work on a vehicle speed dataset for the major European road network, built from Sentinel-2 imagery and spanning 2022 to 2026 (~22M samples).

dataset: https://zenodo.org/records/21484806
pipeline: https://gitlab.heigit.org/traffic-flow-estimator/nibblergb

Questions, feedback, and thoughts on how to use it are more than welcome.

Happy exploring around and finding out!

There are many amazing open datasets available today, but I’m curious about the gaps that still exist.

If you could create or request one high quality public dataset that anyone could use, what would it be?

Hi, I am having trouble getting datasets from the NOAA search tool. Every time I pick a station and request some data, NOAA emails me something like:

Order #[Redacted] (Custom GHCN-Daily CSV)

Can someone guide me to a station that doesn’t send you an empty set of data? Anything helps, since I’m just using it to learn some programming.

Thank you.

Disclosure: I built this. It powers quizbandiere.it, my free flag quiz.

Country datasets are everywhere; flag colors usually aren’t, and when they are they’re hand-written heraldic descriptions. I extract them from the rendered SVG: rasterize at 64×48, classify pixels by hue, merge antialiasing shades, drop anything under 2.5% of the area, then order the result along whichever axis separates the color centroids most. So a vertical tricolor comes out left-to-right and a horizontal one top-to-bottom — Italy is ["verde", "bianco", "rosso"], not sorted by area. Coats of arms and crosses fall back to area order.

I built it to generate believable wrong answers in a quiz. It works well as a similarity key for “which flags get confused with which”.

250 entries (195 independent states, 55 territories): ISO 3166-1 alpha-2/3, official name, capital, region, area, borders, currencies, languages, TLD, calling code, emoji, colors, SVG URL.

• JSON: https://quizbandiere.it/dati/bandiere.json
• CSV: https://quizbandiere.it/dati/bandiere.csv
• Docs: https://quizbandiere.it/dati/

Served with Access-Control-Allow-Origin: *, so it works from a browser. The license and attribution are inside the JSON, not just on the page.

Limitations

• Everything is in Italian, including the color names (verde, azzurro).
• Colors are perceptual, not official. No hex codes, no heraldic terms.
• No flag ratios and no historical flags.
• Derived work: base data is mledoze/countries, I added the Italian layer and the color extraction. Contested names and borders are inherited from upstream, not adjudicated by me.
• Snapshot, not a feed. The aggiornato field tells you how stale your copy is.

License: ODbL 1.0 — not a choice, mledoze/countries is ODbL and share-alike. Artwork is lipis/flag-icons (MIT), re-hosted.

Corrections welcome, especially on flags where the color order looks wrong. That’s where my pipeline is most likely to be quietly incorrect.

🚨

4,000 interviews across the Niger Delta.
260 of them were conducted with nobody.

A research team was running a corruption-reporting experiment across 106 communities in Nigeria.
The data was flowing in.
The numbers looked fine.

But something nagged at them.
Some surveyors were finishing interviews far faster than anything seen in training.
Too fast to be real conversations.

So the team added something quiet to the survey app.
Audio recordings of the surveyors reading the questions aloud.

Then they listened.

13.83% of the surveys failed the check.
Some recordings were completely silent.
Others captured surveyors chatting with family and friends — while entering fake responses into the tablet.

Interviews with nobody.
Answers from no one.

260 of 1,880 surveys were flagged and thrown out.
And here’s the part worth remembering: once surveyors knew the audio was reviewed nightly, fabrication dropped sharply.

The lesson isn’t just about catching fraud.
It’s that a cheap audio snippet is one of the strongest deterrents against wholesale fabrication ever tested.
People behave differently when the tape is rolling.

If your field data has never been listened to — literally listened to — how sure are you about what’s in it?

Based on: Gomila, Littman, Blair & Paluck, “The Audio Check” (Social Psychological and Personality Science, 2017)
#FieldResearch #DataQuality #SurveyFraud #MEL #AudioAudit

#FieldResearch #DataQuality #SurveyFraud #MEL #AudioAudit

Hello, we are building a system for baby cries detection in a confined space such as a room or hallway via CCTV cameras. However, we are unable to source the baby cries dataset. I tried to contact some DayCare and submitted an application upon their request but was denied due to parental privacy reasons.

We have a working system, but the model is way poor as it is only trained on a few examples and fails at CCTV distance as the baby is too far.

Disclosure: this is my own dataset and I sell access to it. Four days are free, linked below, so you can check the schema before deciding anything.

I run a collector against Polymarket’s public API for the 5-minute and 15-minute up/down markets on BTC, ETH, SOL and XRP. It has been running since 26 June 2026. What has accumulated so far:

• 58,600 markets (14,653 per asset — roughly 11k five-minute, 3.7k fifteen-minute)
• ~5.69 billion rows across six tables
• One Parquet file per asset, per table, per UTC day, each with a manifest carrying sha256, row count and interval bounds

Tables: book_snaps (best bid/ask), book_depth (ladder levels, ~88 per event), clob_trades and clob_trades_ws (trades with taker side, the second carries tx hash and fee for VPIN), spot_klines, book_meta.

Limitations, because they decide whether this is useful to you:

• book_snaps is top of book only. There are level and size columns but they are 0 in every row — they exist to keep the schema stable across tables.
• book_depth starts 20 July 2026. Fifteen days of the archive have it; everything before is quotes, trades and spot candles.
• The clock changes partway through. Up to 20 July the timestamp is receipt time rounded to the second, so intra-second ordering is lost and cannot be recovered. From 21 July it is exchange time in milliseconds.
• Per-day uptime runs 95.8% to 100%. Every day carries its own measured figure.

Free sample (original source): 26-29 June, four full days, no payment and no signup — https://huggingface.co/datasets/astrnvk/polymarket-orderbook-data-preview

Paid access covers 30 June onward and is priced per day ($5/day for short ranges down to $3/day past a month), with incomplete days billed proportionally lower — a day missing depth costs about half. Happy to answer schema questions here either way.

A dataset problem I keep running into: the starting data is often small, messy, and inconsistent, but the expected output still needs to be high quality.

This is common when the source is not a clean benchmark dataset, but real-world material: PDFs, internal documents, database exports, tables, reports, web pages, logs, or mixed text files. There may be useful signal inside, but it is usually buried under formatting noise, duplicates, missing metadata, broken structure, and uneven quality.

One approach I find useful is to treat dataset construction as a pipeline of small operators, rather than one large script.

For example, a pipeline might look like this:

• convert files or URLs into markdown/text
• split the content into chunks
• clean noisy text
• normalize fields with pandas-style operators
• remove duplicates with hash, MinHash, SimHash, or semantic deduplication
• filter low-quality samples with rule-based or LLM-based filters
• select representative samples with embedding-based methods like K-Center Greedy
• synthesize QA pairs or SFT samples
• evaluate generated samples for quality, alignment, and verifiability
• export the result into formats for RAG, fine-tuning, or evaluation

The important part is that each step is inspectable. If the final dataset is bad, you can trace whether the issue came from extraction, chunking, cleaning, synthesis, filtering, or evaluation.

This is especially useful when the raw data is limited. With small datasets, every bad transformation hurts more. A pipeline makes it easier to preserve signal, reduce noise, and add synthetic examples only where they are actually helpful.

This is the path behind OpenDCAI/DataFlow: using reusable operators and composable pipelines to produce high-quality data from messy sources, and recently it can also be used with coding agents like Codex to build custom data pipelines through natural language interaction.