Produced by https://www.landrecords.us by sourcing directly from State and County governments, and enriched with public data from USGS, BLM, and EPA.

Disclosure first: I’m one of the maintainers of PortalJS, an open source framework for data portals, so this is my own project.

The problem we kept hitting: you have a folder of CSVs you want to publish properly. “Properly” increasingly means machine readable metadata, DCAT feeds so data.gov or data.europa.eu can harvest you, and lately Croissant so ML tooling can load your datasets. The usual answer is “run CKAN”, which is a lot of infrastructure if all you have is fifty files that update quarterly.

Our approach is a static site that does the metadata work at build time. You describe each dataset once (it’s a Frictionless Data Package under the hood) and the build emits DCAT catalog feeds in JSON-LD, Turtle and RDF/XML, with application profiles for DCAT 2/3, DCAT-AP, DCAT-US, GeoDCAT-AP and Croissant, plus autodiscovery links and SHACL conformance checks against the EU ITB validator where the profile requires it. Static hosting, no runtime, harvesters just fetch the feeds.

If you use Claude Code there are skills that do the assembly:

/portaljs-new-portal "Auckland Council open data portal" /portaljs-add-dataset ./air-quality.csv /portaljs-add-dcat profiles=dcat-ap 

But it’s plain Next.js you can edit by hand, the skills are a convenience. Repo: https://github.com/datopian/portaljs

What I’d like to hear from people who publish datasets: does anyone actually see harvest or referral traffic that you can attribute to DCAT or Croissant metadata? I have a decent theory of why it matters (national portal harvesting, dataset search, agents reading catalogs) but I’m curious what it looks like in practice versus a clean CSV and a good README.

Disclosure: I built DietlyAPI, one of the APIs compared below. I used the providers’ public documentation and included the calculations so people can challenge them. This is not a speed or nutrition-accuracy benchmark.

I compared the closest common operation: searching for a food or looking up a product/barcode. Recipe search, image recognition, natural-language parsing, and other premium operations are not directly equivalent.

Prices and limits checked in July 2026.

Open Food Facts

Price: Free

Limit: 10 search requests/minute/IP and 15 product reads/minute/IP

Monthly allocation: No fixed monthly quota published

Best when you need open, downloadable worldwide product data. The hosted search API is not intended for search-as-you-type at scale. Open Food Facts recommends using its exports or running your own backend for high-volume applications.

USDA FoodData Central

Price: Free

Limit: 1,000 requests/hour/IP

Equivalent sustained average: 16.7 requests/minute

Authentication: Free data.gov API key required

Best for authoritative US reference data. The data is public domain under CC0 and can also be downloaded. Its search behavior and geographic focus differ from a worldwide consumer-product autocomplete API.

FatSecret Basic

Price: Free

Included quota: 5,000 calls/day

Equivalent sustained average: 3.47 requests/minute

Equivalent monthly capacity: 150,000 calls over 30 days

FatSecret also offers Premier Free to qualifying startups, nonprofits, and students. That tier advertises unlimited calls but requires verification, attribution, and is limited to US data. Paid international access is quote-based.

Edamam Basic

Price: $14/month

Peak limit: 50 food and nutrition requests/minute

Included quota: 100,000 calls/month

Equivalent sustained average: 2.31 requests/minute

Edamam Core costs $69/month for 750,000 calls and 100 requests/minute. Plus costs $299/month for 5 million calls and 300 requests/minute.

Edamam is particularly strong when natural-language parsing, serving measures, diet filters, allergy filters, and image recognition matter. Its caching and attribution requirements should be reviewed carefully.

Spoonacular Cook

Price: $29/month

Peak limit: 5 requests/second, or 300/minute

Included quota: 1,500 points/day

Optimistic monthly ceiling: 45,000 one-point calls over 30 days

Equivalent sustained average: at most 1.04 one-point calls/minute

Spoonacular uses points rather than ordinary requests. A call usually costs one point plus an amount based on the number of results, while some endpoints cost more. It is strongest for recipe and ingredient workflows, but its advertised requests-per-second limit should not be confused with the included daily capacity.

Nutritionix

Price: Contact/quote

Published numeric limit: I could not find a current public figure

Published quota: I could not find a current public figure

Nutritionix focuses on natural-language food logging, instant search, barcode lookup, nutrients, and exercise parsing. Its documentation currently describes a database of more than 600,000 foods. I would request a current quote instead of relying on old third-party pricing.

DietlyAPI Pro

Price: €27/month

Peak limit: 500 requests/minute/account

Monthly call allocation: None

Theoretical 30-day ceiling at the RPM limit: 21.6 million calls

Approximate subscription price per theoretical million calls: €1.25

DietlyAPI Scale

Price: €92/month

Peak limit: 3,000 requests/minute/account

Monthly call allocation: None

Theoretical 30-day ceiling at the RPM limit: 129.6 million calls

Approximate subscription price per theoretical million calls: €0.71

Dietly also allows anonymous non-commercial reads at 30 requests/minute/IP with a small fairness delay.

The Dietly numbers are theoretical RPM ceilings, not an SLA or a promise that an application should run continuously at the limit. They show that the plans are rate-capped instead of monthly-credit-capped.

Dietly currently focuses on fuzzy and confidence-aware food search, stable food IDs, barcode lookup, and structured nutrition across more than 4.2 million indexed foods. Its catalog is primarily derived from Open Food Facts and retains the relevant attribution and share-alike obligations.

It does not currently provide USDA data, recipe discovery, food-image recognition, or Edamam-style natural-language parsing.

My conclusion

There is no honest universal winner:

– Open Food Facts is best when open and downloadable worldwide product data matters most.

– USDA is best for authoritative US reference data and permissive reuse.

– FatSecret is compelling for eligible startups or businesses needing verified country-specific datasets.

– Edamam is strong for NLP, measures, and diet/allergy features.

– Spoonacular is strong for recipes and ingredient workflows.

– Nutritionix is worth evaluating when natural-language food logging is central, but you need a current quote.

– DietlyAPI has the strongest published price-to-throughput ratio in this comparison for straightforward food search and barcode lookup.

The biggest lesson was that peak requests per minute can be misleading. A service may advertise hundreds of requests per minute while its daily or monthly quota only supports one or two requests per minute when averaged over the complete billing period.

Create structured data datasets, infinite possibilities (companies, people, places, products, markets, regulations, the obscure and the everyday): You can define the topic/theme of the desired data, define the output fields, generate a sample record with test lookup values, and then when ready generate an API to it that we host and that you can integrate to or call from anywhere, as well as append the defined dataset to another file [self-promotion]: https://custom-data-wizard.interzoid.com/

Includes: – Economics across continents.

Historical Tick Data & OHLCV data (all time frames)

• Options data including Greeks
• Live streaming Options & CandleSticks
• 24H tick replay
• Fixed Income
• ETF
• Yields
• Futures
• Commodities
• FX
• Crypto
• Derivates
• Indices
• Level 3 OrderBook data ( obviously theres no point unless your an institution that can realistically afford this Post use, then using this would be pointless, unless your just a researcher)

https://londonstrategicedge.com/data

i have zero affiliation, but what i came across when i gave up searching the entire internet for datasets

Hi everyone! I recently published an open flight dataset on Zenodo containing simulated Antonov An-32 flight data focused on accelerated stall events. The dataset was created as part of my undergraduate research and is intended for machine learning, anomaly detection, time-series forecasting, and flight dynamics research.

It includes:

• Flight state variables and control inputs.
• Normal and accelerated stall scenarios.
• Time-series data suitable for deep learning models.
• DOI and open access via Zenodo.

I’d be happy to answer any questions or receive feedback!

https://zenodo.org/records/20359823

I published a free sample of 5,000 food records for anyone working on nutrition search, calorie tracking, barcode matching, data cleaning, or food-related ML experiments.

Dataset:

https://www.kaggle.com/datasets/dietly/dietly-food-sample-5000-foods-calories-and-macros

Each row can include:

• – food name and brand
• – barcode
• – category
• – serving size
• – calories
• – protein, fat and carbohydrates
• – fiber and sugar
• – sodium, saturated fat and cholesterol
• – potassium
• – source and confidence metadata

The data is primarily derived from Open Food Facts and is provided with its ODbL provenance and attribution requirements.

Important limitations:

• – nutrition fields can be missing
• – community-contributed labels can be incorrect or outdated
• – coverage varies by product and country
• – this is a 5,000-row sample, not the complete catalog
• – it should not be treated as medical or laboratory-verified data

Disclosure: I created this sample while building DietlyAPI, a hosted search and barcode API over a much larger indexed catalog. The downloadable sample itself is available without an API signup.

I’d especially appreciate feedback about the schema: which additional fields or export format would make the dataset more useful?

Hi community,

I recently published a research-based synthetic Chronic Kidney Disease (CKD) dataset on Kaggle after spending several weeks studying clinical guidelines and epidemiological literature.

The motivation came from a common challenge I encountered: many publicly available CKD datasets contain only a few hundred patient records and a limited number of clinical variables, making them less suitable for building and evaluating modern machine learning models.

Dataset Highlights

• 200,000 synthetic patient records

• 82 clinically meaningful features

• Research-informed design using published clinical guidelines and epidemiological evidence

• Covers demographics, lifestyle, medical history, vital signs, kidney biomarkers, medications, frailty, healthcare utilization, and clinical outcomes

• Includes CKD stage, kidney failure risk, dialysis requirement, and hospitalization risk

• Machine learning and healthcare analytics ready

The dataset is completely synthetic and contains no real patient information. It was created for educational purposes, machine learning experiments, healthcare analytics, and research.

I’d really appreciate feedback from the community.

Some questions I’d love your thoughts on:

• Are there any important CKD-related variables you think are missing?

• What types of ML or analytics projects would you build with this dataset?

• What would you improve in a future version?

Kaggle Dataset:

https://www.kaggle.com/datasets/mohankrishnathalla/chronic-kidney-disease-risk-dataset-2026

Thanks for taking the time to check it out. I’m happy to answer questions about the design process or discuss future improvements.

Hi everyone,

As part of my PhD thesis focusing on the integration of BIM, AI, and IoT for predictive HVAC systems management, I am looking to find a dataset specific to the Moroccan context.

I am searching for historical time-series data that ideally includes:

Building energy consumption (preferably commercial, office, or institutional buildings)

Associated meteorological/weather data

HVAC parameters (indoor temperatures, setpoints, flow rates, fan/pump speeds, etc.)

If you know of any open-source Moroccan repositories (like past projects from IRESEN, AMEE, or local universities) or if you could point me toward professionals, researchers, or facilities management teams who might share anonymized data, I would be incredibly grateful.

Thank you in advance for your help!

Hi everyone,

As part of my PhD thesis focusing on the integration of BIM, AI, and IoT for predictive HVAC systems management, I am looking to find a dataset specific to the Moroccan context.

I am searching for historical time-series data that ideally includes:

Building energy consumption (preferably commercial, office, or institutional buildings)

Associated meteorological/weather data

HVAC parameters (indoor temperatures, setpoints, flow rates, fan/pump speeds, etc.)

If you know of any open-source Moroccan repositories (like past projects from IRESEN, AMEE, or local universities) or if you could point me toward professionals, researchers, or facilities management teams who might share anonymized data, I would be incredibly grateful.

Thank you in advance for your help!

Looking to connect with teams and individuals collecting egocentric data across North America, LATAM, Asia, and Europe.

If you’ve got data or are mid-collection, let’s talk.

GitHub link: here

Licensed under MIT.

Includes:

• States and Union Territories
• Districts
• Sub-districts (Tehsils/Taluks)
• Latitude/Longitude
• Bounding boxes (where available)

Sources:

• iGOD (India Portal) for administrative divisions
• OpenStreetMap Photon for geocoding

The repository also includes scripts to regenerate the dataset. The script can also be incremented to get the block-level data.

I don’t understand scraping infrastructure.

I can make 10 fake YouTube accounts and try to scrape Koala 36M but it’s not possible. It takes like 100-1000VMs to actually do this scraping in time

Large companies don’t publish anything. They have 10s of millions scale videos and don’t even put of 10M.

Does anyone have any advice on this? Im training video models and world models.

Hello, here’s a little python script I made to download data from Reddit community dumps. I isolated it from a larger project, it fetches data for the subreddits you input, output format CSV or JSON (with the actual threads).
https://github.com/Tryhard-cs/reddit-download-tool

Note: It will download all pre-2026 data (for the subreddits) if you include a time interval that has pre2026 data (downloads the data from academic torrent archives)

For post2026 data it will only download that interval, but download speed is very low since it’s community APIs (which I am very grateful for and that should be the case to prevent abuse)

Sources: https://academictorrents.com/details/3e3f64dee22dc304cdd2546254ca1f8e8ae542b4 (made by u/Watchful1 I believe)
https://github.com/ArthurHeitmann/arctic_shift

I remember testing the data coverage manually by comparing to the Reddit website and being satisfied, but my main purpose was more for analytics / research and tracking evolution of stuff overtime / by category and not for a precise total amount. I can’t do any guarantees honestly.

I may add more categories / PRAW support in the future.

PS: Looking for someone to test it and tell me if it works for them / if the install instructions are clear enough, if this is helpful for you and you tried it and you encountered a problem send me a message I’ll fix it right away.

The project is primarily intended to help the police in my city monitor situations that could potentially lead to harm to individuals or the public. It would analyze public posts and public comments to identify content that may indicate emerging threats or dangerous situations. In a way, it can be considered a sentiment analysis system designed for law enforcement.

However, the project requires a live data feed. Reddit has effectively stopped providing the level of API access needed for this use case, Meta’s APIs (Facebook and Instagram) are too restrictive to be useful, and Twitter (X) API access is paid. At the moment, the only options I can think of are web scraping or using downloaded datasets.

The problem with downloaded datasets is that the project is specific to a single city Surat, India so historical datasets may not contain the localized, real-time information needed for effective monitoring.

Kept running into the same problem: messy CSVs with duplicate rows, stray whitespace, and broken email fields, and no fast way to clean them without spinning up a script every time.

So I built CSVCleaner (hackiom.xyz) — drop a file in and it removes duplicates, trims whitespace, and validates emails right in your browser. Nothing gets uploaded to a server, so it works fully offline and there’s zero signup friction.

Still actively building this out, so I’d love feedback on what other cleanup features would be useful — currently thinking about type detection, column renaming, and null handling next.

Happy to answer questions about how it works under the hood.

Hi everyone,

I’m a local entrepreneur doing research on the medical waste management industry and came across an IBISWorld report that would be extremely helpful for my research:

Medical Waste Disposal Services in the US (NAICS OD4182)

https://www.ibisworld.com/united-states/industry/medical-waste-disposal-services/4182/

I know many universities provide IBISWorld access through their libraries, so I was wondering if anyone at UMass Lowell could check whether it’s available through the university.

If you have access and would be willing to help, I’d really appreciate it. Even a PDF export or screenshots of the sections on market share, barriers to entry, competitive landscape, and financial benchmarks would be incredibly helpful.

Thanks in advance!

An interesting project based on a massive dataset of machine learning papers!

Sharing an open food and nutrition dataset I built, free and under ODbL. It started because I kept needing clean food data for a calorie-tracking app and couldn’t find anything that was open, clean and multilingual all at once, the open stuff out there is either English-only or pretty messy, and the genuinely multilingual options are paywalled APIs like FatSecret or Edamam.

The basics:

• ~9,800 base foods, each with localized names in ~46 languages
• Nutrition values from OpenNutrition’s open data (ODbL, credited); the layer I actually built is the localization on top
• Not a Google-Translate pass, food names get tricky (“peperoni” vs “pepperoni”), so real disambiguation, normalization and cross-language matching
• Format: JSON Lines (.jsonl), ~25 MB compressed, ODbL
• Download: https://leana.app/en/data-sources/
• Browse live, no download: https://leana.app/en/foods (live search covers 5 languages for now: EN, IT, ES, FR, DE; the full download already has ~46)

It’s an ongoing project and I’ll keep adding foods and languages over time. No repo yet, so if you spot something wrong or want a language added, just drop a comment here or send me a DM and I’ll fold it in.

Mostly I’d just love some general feedback, and honestly whether something like this feels useful to you at all or not. Thanks for your attention!

Hi all! I’ve been working on ML/Robotics research for a while and often work with HDF5, Parquet, and Zarr files. Personally, I love the myHDF5 viewer, but there’s no good equivalent for Parquet and Zarr, and switching between different sites also gets annoying. So, I built a tool that provides a unified solution.

It currently supports viewing several formats, including HDF5, Parquet, Zarr, Arrow, JSONL, NumPy, TFRecord, etc. I’m hoping to add more features/formats depending on what people find useful!

It’s free to use with no sign-up required. I’d love for people to try it out: https://viewkit.app/

Everything is loaded and parsed locally in your browser (WebAssembly + JS), so your data always remains on your machine. It’s also built to remain responsive on big files via efficient reading, caching, and prefetching. Traversing through data files actually feels faster than existing solutions like myHDF5 with simple caching/prefetching strategies. It also supports some common data types that existing viewers don’t support (e.g. float16, complex numbers for HDF5).

I’d appreciate any feedback (feel free to comment or send a message through the website). Looking forward to supporting additional features/file formats that the community finds useful!

For context, this is for a pre-algebra curriculum as a math teacher. I want to give students more opportunities to examine, interpret, and model real-world data, using skills like slope and line of best fit to draw conclusions and make predictions.

I feel like I waste so many hours searching the internet for good, public data sets that are relevant and interesting for 12-14 year olds. Do you know of any good “database” websites that can serve as a central, starting point?

I work on a training log w/ statistical models (HR and/or watts based) and found this amazing dataset – GoldenCheetah shared their workout data where each athlete’s data is a single zip file that contains a summary level description (aggregates, metrics and so on) as a JSON file and additionally, all workout files are stored as CSV files. The CSV files contain second by second sample data from athlete workouts for; Heartrate, Cadence, Power, Distance and Altitude.

It’s perfect for both predictive and generative models experiments.

Sharing a dataset I’ve been building: daily LLM inference outputs on stock market forecasting, captured before outcomes were known, so predictions can’t be reconstructed with hindsight.

What’s in it: 90+ days of runs (Feb 17 – May 19, 2026, ongoing) for Gemini 2.5 Flash with Google Search grounding, temperature 0.2 Multi-model coverage: 2.5 Pro, 2.5 Flash Lite, and 3 Flash Preview also included Per-run: 10-trading-day price lookahead, sentiment, confidence score, full reasoning trace, cited search snippets ~3,655 rows total, 211MB, fully documented schema with a Colab quickstart notebook for hydrating ground truth yourself

Why it might be useful: most LLM benchmark datasets test on static, already-resolved questions. This one is structured so ground truth genuinely didn’t exist at generation time — useful for studying calibration (ECE), hallucination patterns, and confidence-vs-accuracy relationships under real uncertainty instead of retrospective fitting.

Note on compliance: realized prices and news text aren’t redistributed (licensing reasons) — there’s a hydration script to populate those fields yourself with your own data source, or you can just inspect pre-computed outcome comparisons and results on the companion site (glassballai.com/results).

Note Evaluation: Some tickers have very low run counts due to interrupted tracking or individual tracking runs that are not part of the fixed set of tracked stocks. They are included for full transparency and factor into the global metrics, but their individual ticker-level stats should be ignored due to high variance.

Published on Hugging Face under CC-BY-NC-4.0: huggingface.co/datasets/louidev/glassballai

Happy to answer questions about the collection methodology or the metrics computed on top of it.

(Disclosure: I built and run both of these. Full self-promo disclosure per rule 1.)

Two small datasets I maintain and publish as static JSON, both free, no key, no rate limit, CC BY (just cite the source):

observatory.mordo.ai/data.json – joins real eBay sold prices for used GPUs (refreshed twice monthly) with community llama.cpp benchmark results run on the same models across cards, to compute the metric nobody else publishes: tokens/sec per $100. Current standout is the ~$80 Tesla P100 at ~73 tok/s per $100 on 7-8B models. Most price trackers have zero performance data, most benchmark sites have zero price data, this joins them.

registry.mordo.ai/data.json – a permanent, sourced record of smart-home devices bricked or degraded by a vendor cloud shutdown (Revolv, Wink, Nest Secure, Dropcam, Logitech’s Squeezebox/UE SmartRadio, 16 entries so far), plus a smaller device catalog scored on how much they depend on the cloud vs. working fully local. Every tombstone entry links its sources- no unsourced claims.

Both regenerate nightly off a SQLite backend, no manual curation lag. Repo (Python + SQLite + cron, no Docker required): github.com/tekzer0/instruments -PRs welcome if you know of a cloud-death or device that should be tracked.

Happy to answer questions on the methodology (how prices/benchmarks get matched, how tombstones get sourced/verified) or take requests for fields to add to the JSON.