Hello all,

I’m sharing what is possibly the biggest dataset of what technologies new websites are launching with. I have a self-curated catalogue of 48k+ technologies. My discovery pipeline is adding ~80k new websites a day, we typically add sites on the day they appear on the internet.

The site is free to browse and explore but we do have paid options.

Please check it out and let me know what you think, always happy to hear feedback:

https://stackscope.dev

Created a dataset of fund returns on Snowflake. It’s quite extensive showing both trailing and calendar year returns for every mutual fund and ETF since inception.

You can fetch a mutual funds trailing annualized performance every trading day going back 15+ years.

Each trailing returns row shows: 1D, 1W, 1M, 2M, 3M, 1Y, 2Y, 3Y, 4Y, 5Y, 7Y, 10Y, 12Y, 15Y, and earliest available inception. I usually never see websites (even Morningstar) provide this breakdown which I think is useful when exploring different economic cycles.

Calendar year returns go back as far as possible as well.

Data is updated daily.

disclosure per rule 1: i work at Schema Labs. we train models on tables instead of text. there is a link at the bottom because this sub allows one. i am here for the answer to the question at the end.

here is the claim with the numbers attached because i would rather be argued with than believed.

take a standard tabular benchmark. strip every header off it. replace price and age and zip with positional tokens so there is nothing left but values. comparable models drop about 7 points of mean ROC-AUC. ours goes 0.9224 to 0.9230. flat.

that is not a claim that we win on clean data. those models start ahead of us when the headers are good. it is a claim that we do not read the headers at all, which only matters because production tables do not have good ones. they have val_b and metric_14 and a four-character code from a system nobody has logged into since 2019.

two more. on sector identification we name the correct industry of a dataset we have never seen, from raw values alone, at 86.3% top-1 out of 10,000 possible sectors. on multi-table entity matching we lead published state of the art across all six standard benchmarks with zero shared keys between tables, including a 1.9x improvement over the previous best on one of them. on the geospatial benchmark our lead is 0.07 F1 which i would not want read as more than it is.

every one of those is our own harness, run under each benchmark’s published protocol, one sealed configuration, no per-dataset tuning. competitor figures are third-party published values we did not re-run. nobody independent has replicated us. we are not on the public leaderboard because entering it requires shipping a runnable wrapper and we do not distribute weights. i put all of that up front every time rather than waiting for someone to ask.

which is the actual question.

we are closed. no pip install, no weights, no local option. none of that changes for at least six months. to try it you make an account, you put a card on file, you upload your data. i think that single fact is the biggest thing standing between us and everyone reading this.

so would you use a thing like this at all. not would you like it to exist, everybody likes things existing. would you upload a real table to a hosted model you cannot inspect, run by a company you had not heard of ninety seconds ago. if the answer is no i want to know what moves it. a named customer. a SOC 2. a benchmark you ran yourself on your own data. an on-prem story you would never actually take up but need to hear. or is the honest answer that nothing moves it and closed is closed.

and if you would rather test the claim than discuss it: send me two tables nobody at your company can explain. no account, no card, nothing to sign up for. i run them and send back what each column appears to be with a confidence score on every one. i flag the ones we got wrong as well, because those are the interesting half and you should see them before you decide anything.

anonymised is fine. structure and distributions are what matter.

https://www.schemalabs.ai/

I paid $765 for his advertised “1990–2027 Full Specs” automotive database with 117 columns. What I actually received was a database where the detailed specs/packages essentially stop around March 2020, meaning newer vehicles are missing the “full specs” the product is advertised as providing.

When I confronted him, he acknowledged that information is missing, but his response was basically: why am I focusing on the missing data instead of the substantial amount of data that IS included? The problem is, I specifically paid for a database advertised as having “1990–2027 Full Specs.”

If you’re considering buying from DatabaseAtlas, be VERY careful and verify exactly what data is actually available for newer vehicles before paying.
I would never have paid $765 if this had been clearly disclosed upfront.

Disclosure per rule 1: I built and maintain this dataset, and it powers a site I run (globenomad.com), which is affiliate-funded. The dataset itself carries no affiliate or tracking links.

One row per programme, not per country: Thailand has six routes and they are six rows. Fields: income requirement (USD plus the government’s own wording), fees, max stay, renewal, residency and citizenship path, tax residency trigger, family provisions, application logistics, official source URL, verified date.

Two things worth knowing before you use it:

• A null means the government has not published that rule. It never means “no”.
• Programmes that closed, were announced and never opened, or never existed (Cayman Islands, Peru, Qatar, Vietnam) are kept with an honest status. Most visa sites delete those.

Formats: JSON (canonical, with nested requirements/steps/FAQs) and a flat CSV of the scalar columns.

Original source, GitHub (always current): https://github.com/MSeutin/digital-nomad-visa-data Kaggle mirror, with a worked-example notebook: https://www.kaggle.com/datasets/frenchmike/digital-nomad-visa-dataset-53-sourced-programmes Method and licence: https://globenomad.com/data

Licence is CC BY 4.0, attribution is the only condition. Official pages are re-fetched weekly and confirmed changes are dated on a public changelog. Happy to answer questions about any record.

Trackmania players have been building and sharing maps for about 20 years through Mania Exchange, which has a public API. There was no packaged dataset for any of it, so I made one.

Contents: 850k+ maps across Trackmania Nations Forever and Trackmania 2020. Each map is decomposed into its block sequence with positions and rotations, so the level structure is directly usable rather than sitting in a binary blob. Replays are parsed into telemetry and aligned to the blocks the run passes through, so you get both the level and how it plays.

Format: Parquet, roughly 30GB. Data dictionary included.

License / source: everything comes from the public Mania Exchange API. Maps are community-created.

Possible uses: procedural level generation, level design analysis, player behaviour modelling, map recommendation, or just exploring what 20 years of community map-making looks like in aggregate. I used it for generation myself, but that’s one angle out of many.

Links: 📖Article (EN): https://the-odd-dataguy.com/en/blog/2026/08/13/trackmania-dataset/ 📖Article (FR): https://the-odd-dataguy.com/fr/blog/2026/08/13/trackmania-dataset/ 😇Hugging Face (Dataset): https://huggingface.co/datasets/jeanmidev/trackmania-community-tracks-and-telemetry 🔵Kaggle: https://www.kaggle.com/datasets/jeanmidev/trackmania-community-tracks-and-telemetry/data

Disclosure: I work in data at Ubisoft in Canada, but I have no connection to Nadeo. Built at home on my own time, resources and dime, just something for the Trackmania community and for anyone curious, whether game telemetry is your thing or not.

Open to feedback or ideas (you can also used discussions on HF/Kaggle)

Hello everyone! This may be a really dumb question but I am doing a small research project on population distribution in major cities in El Salvador. I am having a really hard time finding accurate data, especially since I am looking for data from back during the civil war (1979-1992), as well as current population data. I have found general population data from their most recent census, as well as general population data from the CIA factbook. But as for city by city, I cant find anything outside of the data USAID HAD before we all know who took that away. So, if you have any suggestions or different things for me to think about I would appreciate it. Thanks!

Curious if anyone here has worked with gameplay data for training or eval, what’s been useful vs. not worth the effort?

This dataset has all entry from wikidata who has Japanese citizenship and a wiki article, 231k in total. The dataset is on Kaggle.

A dataset like this can only be built from data that’s already public, and Wikipedia is the largest source of that kind of public personal information. There are structured people datasets out there, but none cover as many Japanese entries as this one, which makes this the largest dataset of its kind.

I Queried via SPARQL against the QLever Wikidata endpoint, then cleaned/normalized with Python (name cleaning/splitting, BMI calculation, etc). dataset is in CSV format, 65MB. with multiple entry column separated with “|”

Sample rows (two people picked so together they cover every column):

qid: Q160847
kanji: 東條 英機
hiragana: とうじょう ひでき
description: 日本の陸軍軍人、政治家、第40代内閣総理大臣(1884-1948)
gender: M
birth_year: 1884
death_year: 1948
age: 64
death_age: 64
label: 東條英機
name_en: Hideki Tojo
is_japanese_name: True
readings: とうじょう ひでき
occupations: 士官|政治家|外交官
birth_places: 麹町区
alma_maters: 東京陸軍幼年学校|陸軍大学校|陸軍中央幼年学校|陸軍士官学校
positions_held: 内務大臣|内閣総理大臣|外務大臣|軍需省|陸軍省|文部省|参謀本部|農商務卿
death_places: 巣鴨拘置所
death_causes: 縊死
death_manners: 死刑
fathers: 東條英教
mothers: 東條千歳
spouses: 東條かつ子
children: 東條敏夫|東條輝雄|東條満喜枝
awards: 大礼記念章|チュラチョームクラーオ勲章|…(18 total)
parties: 大政翼賛会
notable_works: 東條英機の遺言|大詔を拝し奉りて
military_branches: 大日本帝国陸軍|関東軍
military_ranks: 陸軍大将
wiki_url: https://ja.wikipedia.org/wiki/東條英機
image_url: http://commons.wikimedia.org/wiki/Special:FilePath/Hideki%20Tojo.jpg

qid: Q11467665
kanji: 山咲 トオル
hiragana: やまざき とおる
description: 日本の漫画家、タレント
gender: M
birth_year: 1969
age: 57
label: 山咲トオル
name_en: Tōru Yamazaki
is_japanese_name: True
readings: やまざき とおる
occupations: タレント|日本の漫画家
birth_places: 東京都
genres: ホラー漫画
height_cm: 170.0
weight_kg: 57.0
bmi: 19.7
siblings: 中沢初絵
wiki_url: https://ja.wikipedia.org/wiki/山咲トオル

Potential Use Cases & As a Dataset

limitations: The least-filled column, military_ranks, has only ~3k rows filled, while the average filled-column count per row is 15.25 out of 37 (41.2%). No rows are dropped to keep this the full Wiki-derived dataset. But I added is_japanese_name column so you can reliably excludes non-Japanese names (virtually no false negatives, ~0.1% false positives). About 1% of rows have a reversed family/given order in the hiragana column.

Crawler code : Kaggle notebook / GitHub. This dataset will be updated automatically with the crawler.

Access vehicle specifications, history reports, images, valuations, market listings, and automotive market insights through powerful APIs.

been thinking about why finding training data is still such a slog in 2026. someone at berkeley recently built a tool just to search kaggle, huggingface and data . gov at the same time, because doing it by hand was too slow. the fact that this is a tool people have to build says a lot

the usual explanation is fragmentation, which is true but kind of surface level. the deeper thing is incentives. on most data marketplaces the vendor pays to be listed, and almost nobody works on commission. so the platform is optimized for whoever pays for placement, and whether the data is actually findable or usable ends up an afterthought. the money doesn’t come from discovery, so discovery gets neglected

feels like it should run the other way. a request-first marketplace, where the buyer posts what they actually need and providers come back with offers. kind of like how taxis worked before the apps. you announce where you’re going, drivers see it and take it if the terms work for them. the buyer sets the direction and providers respond to it

this matters more as requests get weirder. the more specialized and one-off your data ask is, the lower the odds that any single provider already has exactly that sitting on a shelf. a static catalog struggles with that kind of ask. something you can post and have people bid on fits it a lot better

genuinely curious what people here think. has anyone seen a marketplace that works request-first, or runs on commission instead of paid placement? and if you’ve bought data before, did you actually find it through a marketplace, or did you end up emailing providers directly because search never surfaced the right thing?

disclosure, i work at Titan Network and we’re on the provider side of this, so it’s something i think about a lot. not pitching anything, btw

Any of y’all know where to find an oil dataset with DDRs, time series data, with good amount of completion? I tried the volve equinor one, but the formation tops for like ~70% is NULL,

Hello!

This dataset I have created last year has been downloaded over 20 000 times already! It’s the biggest open club football dataset in the world, including over 240 000 football matches’ data such as scores, stats, form, Elo, odds and more, all updated up to 09/26!

The dataset can be used for training AI models, creating visualizations, or just for personal data exploration 🙂

If anyone wants to dive deep into leveraging bookmaker’s odds, I have also created a dataset specifically for that with 2000 matches and 41 betting markets. It’s behind a paywall for a symbolic price because it supports my future work and allows for other free datasets to be updated: https://odds.adamgabor.eu/

All of the links:

Kaggle: https://www.kaggle.com/datasets/adamgbor/club-football-match-data-2000-2025/data

Github: https://github.com/xgabora/Club-Football-Match-Data

HuggingFace: https://huggingface.co/datasets/xgabora/club-football-match-data

Odds Dataset: https://odds.adamgabor.eu/

Hi, I’m looking for a free dataset or API for Indian packaged food products.

Does anyone know of a good source? Thanks!

I wanted a per-product count of the seven FD&C dyes still certified for use in US food, and it turns out the Open Food Facts bulk export is enough on its own. Posting the method and the numbers because I could not find this cut published anywhere.

Source: the Open Food Facts full CSV export, https://world.openfoodfacts.org/data. I used the 2026-08-31 snapshot, 1.275 GB gzipped, 4,535,553 rows. Filter to countries_tags containing en:united-states and require a non-empty ingredients_text, which leaves 444,943 products. Then match the ingredient text against the US number form, the common chemical name and the E number for each dye.

The part I did not expect is that the naming is regular. I went in braced for the inulin problem, where a third of the products containing an ingredient never print its common name and you spend the afternoon chasing synonyms. It is not there. Of the 6,259 products matching any Red 3 form, 23 do not use the plain “Red 3” wording, which is 0.4%. Red 40 is 96 of 28,591, or 0.3%. Yellow 5 is the worst at 0.8%, because tartrazine still turns up. A regex on the number form catches essentially all of it, which is not how ingredient text usually behaves.

There is a regulatory reason for that. 21 CFR 101.22(k)(1) requires a colour additive subject to certification to be declared by the name given in the applicable regulation, and it explicitly allows dropping the “FD&C” prefix and the “No.” So “Red 40” is not shorthand, it is the compliant form. Only colours not subject to certification may be declared under (k)(2) as “Artificial Color”, “Artificial Color Added” or “Color Added”.

That flips something I had assumed going in. 16,768 products use one of those generic phrases, and 10,571 of them also name a certified dye, so in those the generic phrase is covering something else in the same product. 3,914 use a generic phrase and name no colourant at all, and by (k)(2) whatever those are, they are not one of the seven.

Where the label does go dark is (k)(3), which says colouring added to butter, cheese and ice cream need not be declared at all. Most producers declare anyway: 27.5% of the 12,080 cheese records name a natural colourant, mostly annatto. But the exemption is real and it is invisible from the ingredient list.

On timing, FDA revoked the Red 3 authorization on 2025-01-15 under the Delaney Clause and gave food manufacturers until 2027-01-15 to reformulate, so this snapshot sits four months out from the deadline. The 6,256 products span 1,073 distinct first-brands, so it is not one catalogue duplicated.

Limitations, and the second one is large.

• Open Food Facts is crowd contributed. This is a sample of what people scanned, not the US shelf, and popular products are over-represented.
• Records go stale. Only 17.3% of the Red 3 rows were modified after the revocation date, against 43.0% of all rows. Restrict to rows touched since then and Red 3 falls from 1.41% to 0.57%. I am quoting both because neither is the honest single number: the full count includes labels that have already changed, the restricted count drops products nobody has rescanned.
• A string match cannot see inside “natural flavors” or a sub-ingredient it does not parse, so every figure here is a floor.
• I hand-read 12 random Red 3 matches looking for false positives and found none. They are all “fd&c red #3”, “red 3 lake” and similar.

The gap I cannot close from this export is retail availability. Does anyone know a source that would let me weight this by what is actually stocked, rather than by whoever happened to scan it?

Disclosure per rule 1: I write an iOS app called Snack Check that reads these dyes off a barcode, which is why the parser was already sitting there. I am not linking it, and the export above is the only link in this post.

Disclosure: I built this and it is hosted on my own site, quant500.com. It is free, CC0, no account and no API key. I am posting it because the time-of-day column does not seem to exist anywhere else for free, and because the defects in it are worth more discussion than the coverage.

What it is. Every Form 8-K carrying item 2.02 filed by an S&P 500 company: 64,938 rows, one per filing, covering 63,969 distinct announcements (group by cik + announcement_date) from 808 companies, 2003-04-25 to 2026-09-01. Sixteen columns. Every row carries the accession number and a direct link to the filing on sec.gov, so any line can be checked at source.

Original source is SEC EDGAR (data.sec.gov/submissions). This is a derived file, rebuilt daily.

The column that seems to be missing elsewhere is the clock time, and with it the session the news could first be traded in: 46.4% before the open, 41.8% after the close, 11.7% during the session. 99.8% resolve to a New York time.

Four defects, because you would find them anyway:

1. The 8-K cover-page date is typed by the filer and is often wrong. Micron enters the fiscal quarter end there in 14 of its 94 announcements; a 2004 Walmart filing declares the event as happening in 2001. Where the cover date sits more than three days from the EDGAR entry, the EDGAR date is used and the row carries date_uncertain = yes. That is 3,569 rows.

2. EDGAR’s acceptance timestamp ends in “Z” but is not always UTC. Measured on this file: 924 rows carry a raw hour between 00:00 and 05:59, impossible if the stamp were already Eastern because EDGAR is closed then, and 4,835 carry one between 06:00 and 09:59, impossible if it were UTC. Both conventions are genuinely present.

3. And the one I cannot fix. I resolve that convention per company, which is probably the wrong unit: the share of announcements from companies classified as “already New York” falls steadily from 24.1% in 2003 to 1.0% in 2026. A fixed company attribute should not drift like that, so the convention likely belongs to the filing agent or to the era. It is declared in the file header and unresolved. If anyone here has parsed EDGAR at scale and has seen this, I would like to know.

4. The filing does not always arrive on the day of the announcement. In 7,695 rows (11.8%) filing_date is later than announcement_date, by a single day in 6,265 of them. For those rows the session and the clock time describe the day the filing arrived, not the day the news broke. So if you group by announcement_date and read the session column without also reading filing_date, you place 11.8% of the sample in the wrong trading session. Use filing_date to know which day the session refers to.

Limits: nothing before 2003-03-28, when the 8-K had no dedicated earnings item; item 2.02 only exists from 2004-08-23, and earlier filings come in under the old item 12 and are marked rule = 12; and the session recorded is the one in which the 8-K reached EDGAR, not the one in which the press release went out, which is earlier.

CSV: https://quant500.com/api/descarga/anuncios.csv

Method and caveats: https://quant500.com/blog/2026-09-04-conjunto-de-datos-resultados

Browsable: https://quant500.com/earnings-date

Disclosure: I run genchijapan.com. This is my own dataset.

What it is

A table of the rules foreign visitors keep asking about in Japan — is Gion closed to photos, can you drink on the street in Shibuya, is there a fine for smoking in Osaka, what the tax-free refund rules actually say — with the things that are usually missing from English answers:

• level — national law / prefectural ordinance / city ordinance / district rule with no statutory force (e.g. a neighbourhood council’s posted ¥10,000 “fine”)
• authority — who issued it
• penalty and since — fine amount and effective date
• source — URL of the primary source (e-Gov statute text, ministry or city page) or, where the issuer has no web page, press coverage
• verified — date a human last checked the row against the source

Manners are excluded on purpose: they can’t be sourced.

Honest limits

• 17 rows across 7 areas. Small. Rows are added as I verify them, not scraped.
• A few rows (mainly Gion) cite press instead of the issuing body, because the district council publishes nothing online. source_label tells you which.
• verified is a date, not a guarantee. Rules change.

Links

• GitHub (JSON + CSV + schema): https://github.com/genchijapan/japan-tourist-rules
• Live JSON, CORS-enabled: https://genchijapan.com/rules.json

CC BY 4.0. Corrections with a source URL are the most useful thing you can send me.

Hey guys. I’ve been training reinforcement learning models and pulling my hair out trying to get clean, historical order book depth data without paying institutional API fees. I finally set up my own pipeline to capture L2 depth and imbalance for SOL. I formatted it specifically for backtesting and feature engineering. You can grab the dataset here:https://imbalancelabs.com/datasets/sol-historical-orderbook. What specific L2 features (besides standard imbalance and spread) are you feeding into your models lately?

Adaption has launched a new solution – Invent a Dataset – that allows you to to describe in a few words what you want a model to learn and it turns that objective into a structured, training-ready [Synthetic] dataset. Most synthetic-data tools automate generation only after someone has defined the schema, task distribution, and generation strategy. Invent a Dataset starts one level earlier: with the behavior the model should learn.

Recently I’m stuck hard on my research at finding extremely similar datasets, but should still be different. It seems that even datasets in the same semantic field differs drastically inside an LLM’s internal representations. Any good way of finding datasets this similar?

By the way I’m especially interested in datasets that are at least 5,000 samples large, which narrows the candidates greatly.

Hi, I was looking for this dataset ‘BIWI Kinect Head Pose’. But it is no longer available for the public, If anyone has the dataset downloaded please DM me.

Hi everyone,

I’m shutting down a crypto market research / ML project that I’ve been running since late 2025, and I’ve decided to release a free 7-day sample of the market data I collected for anyone interested in quantitative research, market microstructure, backtesting or machine learning.

The sample contains 81,579 ADA/USDC market snapshots with 97 columns.

What’s included

• 20 bid + 20 ask Level-2 order book levels
• Bid/ask prices and quantities
• Best bid / best ask
• Mid price and spread
• Latest spot trade price and quantity
• Trade direction
• Futures open interest
• Funding rate
• Mark price
• Order-book imbalance
• Market depth at different distances from mid price
• Liquidity wall ratio
• Book pressure

The collector runs on a nominal 5-second polling cycle. Because API calls and processing occur between snapshots, the actual median interval in this sample is approximately 6.17 seconds.

The original timestamps are preserved and the public sample has not been artificially interpolated or resampled.

Free download

Hugging Face:
rfab85/crypto-5s-market-data-adausdc-sample · Datasets at Hugging Face

Kaggle:
ADA/USDC High-Frequency Market Microstructure Data

I’d be genuinely interested to hear what people working with L2/order-book data think of the schema and what features you would derive from it.

has anyone ever needed govt datasets for a project and spent significant time finding/accessing it? or is it just me