Hello, here’s a little python script I made to download data from Reddit community dumps. I isolated it from a larger project, it fetches data for the subreddits you input, output format CSV or JSON (with the actual threads).
https://github.com/Tryhard-cs/reddit-download-tool

Note: It will download all pre-2026 data (for the subreddits) if you include a time interval that has pre2026 data (downloads the data from academic torrent archives)

For post2026 data it will only download that interval, but download speed is very low since it’s community APIs (which I am very grateful for and that should be the case to prevent abuse)

Sources: https://academictorrents.com/details/3e3f64dee22dc304cdd2546254ca1f8e8ae542b4 (made by u/Watchful1 I believe)
https://github.com/ArthurHeitmann/arctic_shift

I remember testing the data coverage manually by comparing to the Reddit website and being satisfied, but my main purpose was more for analytics / research and tracking evolution of stuff overtime / by category and not for a precise total amount. I can’t do any guarantees honestly.

I may add more categories / PRAW support in the future.

PS: Looking for someone to test it and tell me if it works for them / if the install instructions are clear enough, if this is helpful for you and you tried it and you encountered a problem send me a message I’ll fix it right away.

The project is primarily intended to help the police in my city monitor situations that could potentially lead to harm to individuals or the public. It would analyze public posts and public comments to identify content that may indicate emerging threats or dangerous situations. In a way, it can be considered a sentiment analysis system designed for law enforcement.

However, the project requires a live data feed. Reddit has effectively stopped providing the level of API access needed for this use case, Meta’s APIs (Facebook and Instagram) are too restrictive to be useful, and Twitter (X) API access is paid. At the moment, the only options I can think of are web scraping or using downloaded datasets.

The problem with downloaded datasets is that the project is specific to a single city Surat, India so historical datasets may not contain the localized, real-time information needed for effective monitoring.

Kept running into the same problem: messy CSVs with duplicate rows, stray whitespace, and broken email fields, and no fast way to clean them without spinning up a script every time.

So I built CSVCleaner (hackiom.xyz) — drop a file in and it removes duplicates, trims whitespace, and validates emails right in your browser. Nothing gets uploaded to a server, so it works fully offline and there’s zero signup friction.

Still actively building this out, so I’d love feedback on what other cleanup features would be useful — currently thinking about type detection, column renaming, and null handling next.

Happy to answer questions about how it works under the hood.

Hi everyone,

I’m a local entrepreneur doing research on the medical waste management industry and came across an IBISWorld report that would be extremely helpful for my research:

Medical Waste Disposal Services in the US (NAICS OD4182)

https://www.ibisworld.com/united-states/industry/medical-waste-disposal-services/4182/

I know many universities provide IBISWorld access through their libraries, so I was wondering if anyone at UMass Lowell could check whether it’s available through the university.

If you have access and would be willing to help, I’d really appreciate it. Even a PDF export or screenshots of the sections on market share, barriers to entry, competitive landscape, and financial benchmarks would be incredibly helpful.

Thanks in advance!

An interesting project based on a massive dataset of machine learning papers!

Sharing an open food and nutrition dataset I built, free and under ODbL. It started because I kept needing clean food data for a calorie-tracking app and couldn’t find anything that was open, clean and multilingual all at once, the open stuff out there is either English-only or pretty messy, and the genuinely multilingual options are paywalled APIs like FatSecret or Edamam.

The basics:

• ~9,800 base foods, each with localized names in ~46 languages
• Nutrition values from OpenNutrition’s open data (ODbL, credited); the layer I actually built is the localization on top
• Not a Google-Translate pass, food names get tricky (“peperoni” vs “pepperoni”), so real disambiguation, normalization and cross-language matching
• Format: JSON Lines (.jsonl), ~25 MB compressed, ODbL
• Download: https://leana.app/en/data-sources/
• Browse live, no download: https://leana.app/en/foods (live search covers 5 languages for now: EN, IT, ES, FR, DE; the full download already has ~46)

It’s an ongoing project and I’ll keep adding foods and languages over time. No repo yet, so if you spot something wrong or want a language added, just drop a comment here or send me a DM and I’ll fold it in.

Mostly I’d just love some general feedback, and honestly whether something like this feels useful to you at all or not. Thanks for your attention!

Hi all! I’ve been working on ML/Robotics research for a while and often work with HDF5, Parquet, and Zarr files. Personally, I love the myHDF5 viewer, but there’s no good equivalent for Parquet and Zarr, and switching between different sites also gets annoying. So, I built a tool that provides a unified solution.

It currently supports viewing several formats, including HDF5, Parquet, Zarr, Arrow, JSONL, NumPy, TFRecord, etc. I’m hoping to add more features/formats depending on what people find useful!

It’s free to use with no sign-up required. I’d love for people to try it out: https://viewkit.app/

Everything is loaded and parsed locally in your browser (WebAssembly + JS), so your data always remains on your machine. It’s also built to remain responsive on big files via efficient reading, caching, and prefetching. Traversing through data files actually feels faster than existing solutions like myHDF5 with simple caching/prefetching strategies. It also supports some common data types that existing viewers don’t support (e.g. float16, complex numbers for HDF5).

I’d appreciate any feedback (feel free to comment or send a message through the website). Looking forward to supporting additional features/file formats that the community finds useful!

For context, this is for a pre-algebra curriculum as a math teacher. I want to give students more opportunities to examine, interpret, and model real-world data, using skills like slope and line of best fit to draw conclusions and make predictions.

I feel like I waste so many hours searching the internet for good, public data sets that are relevant and interesting for 12-14 year olds. Do you know of any good “database” websites that can serve as a central, starting point?

I work on a training log w/ statistical models (HR and/or watts based) and found this amazing dataset – GoldenCheetah shared their workout data where each athlete’s data is a single zip file that contains a summary level description (aggregates, metrics and so on) as a JSON file and additionally, all workout files are stored as CSV files. The CSV files contain second by second sample data from athlete workouts for; Heartrate, Cadence, Power, Distance and Altitude.

It’s perfect for both predictive and generative models experiments.

Sharing a dataset I’ve been building: daily LLM inference outputs on stock market forecasting, captured before outcomes were known, so predictions can’t be reconstructed with hindsight.

What’s in it: 90+ days of runs (Feb 17 – May 19, 2026, ongoing) for Gemini 2.5 Flash with Google Search grounding, temperature 0.2 Multi-model coverage: 2.5 Pro, 2.5 Flash Lite, and 3 Flash Preview also included Per-run: 10-trading-day price lookahead, sentiment, confidence score, full reasoning trace, cited search snippets ~3,655 rows total, 211MB, fully documented schema with a Colab quickstart notebook for hydrating ground truth yourself

Why it might be useful: most LLM benchmark datasets test on static, already-resolved questions. This one is structured so ground truth genuinely didn’t exist at generation time — useful for studying calibration (ECE), hallucination patterns, and confidence-vs-accuracy relationships under real uncertainty instead of retrospective fitting.

Note on compliance: realized prices and news text aren’t redistributed (licensing reasons) — there’s a hydration script to populate those fields yourself with your own data source, or you can just inspect pre-computed outcome comparisons and results on the companion site (glassballai.com/results).

Note Evaluation: Some tickers have very low run counts due to interrupted tracking or individual tracking runs that are not part of the fixed set of tracked stocks. They are included for full transparency and factor into the global metrics, but their individual ticker-level stats should be ignored due to high variance.

Published on Hugging Face under CC-BY-NC-4.0: huggingface.co/datasets/louidev/glassballai

Happy to answer questions about the collection methodology or the metrics computed on top of it.

(Disclosure: I built and run both of these. Full self-promo disclosure per rule 1.)

Two small datasets I maintain and publish as static JSON, both free, no key, no rate limit, CC BY (just cite the source):

observatory.mordo.ai/data.json – joins real eBay sold prices for used GPUs (refreshed twice monthly) with community llama.cpp benchmark results run on the same models across cards, to compute the metric nobody else publishes: tokens/sec per $100. Current standout is the ~$80 Tesla P100 at ~73 tok/s per $100 on 7-8B models. Most price trackers have zero performance data, most benchmark sites have zero price data, this joins them.

registry.mordo.ai/data.json – a permanent, sourced record of smart-home devices bricked or degraded by a vendor cloud shutdown (Revolv, Wink, Nest Secure, Dropcam, Logitech’s Squeezebox/UE SmartRadio, 16 entries so far), plus a smaller device catalog scored on how much they depend on the cloud vs. working fully local. Every tombstone entry links its sources- no unsourced claims.

Both regenerate nightly off a SQLite backend, no manual curation lag. Repo (Python + SQLite + cron, no Docker required): github.com/tekzer0/instruments -PRs welcome if you know of a cloud-death or device that should be tracked.

Happy to answer questions on the methodology (how prices/benchmarks get matched, how tombstones get sourced/verified) or take requests for fields to add to the JSON.

Hi r/datasets,

I’ve uploaded a free sample pack containing 100K rows split across 3 different fully synthetic, high-quality datasets tailored for ML research, data analysis, and pipeline testing:

1. Credit Risk (Improved): Synthesized credit bureau features, debt-to-income ratios, and loan default targets.
2. Finance Transactions: Mock banking/transaction histories with categorical merchant types and amounts.
3. Ecommerce Customer Behavior: Session duration, cart adds, purchase history, and user drop-off flows.

Download Link:
You can access the dataset files directly on Hugging Face:
👉 Synthetic Datasets Free Samples (100k)

Note: Since these are synthetic, they contain no real personally identifiable information (PII) or real-world entities.

Any feedback or requests for specific domains you’d like to see next would be highly appreciated! Enjoy!

USDA’s Local Food Portal is the canonical US farmers-market dataset, but the raw feed is rough: truncated names, stale records, a state= filter that substring-matches state names (querying “WA” returns Delaware rows), and thousands of missing websites/hours.

I cleaned and enriched it: deduplicated to 8,863 real markets (record-level), backfilled website coverage to ~50% from the live API + state sources, and added season/SNAP/organic fields. It’s CC-BY 4.0 as CSV/JSON.

Archived with DOI: https://doi.org/10.5281/zenodo.21360372

Disclosure: I run harvestlymarkets.com, the directory built on this data — full methodology and the same downloads live at https://harvestlymarkets.com/data-sources/. Personal contact names/emails are stripped from the redistributable; business fields come from USDA’s public feed.

Happy to answer questions about the data-quality issues — the state-filter substring bug was a fun one to find.

The core idea: take a real document (SEC filings, contracts, enterprise email), verify by exhaustive normalized scan that a specific plausible fact is not in it, then ask about that fact. The honest answer is “not in the document.” We ran six frontier models on these with zero abstention coaching and they asserted made up answers 11% to 44% of the time. The full per model table is on the dataset card with raw logs and API errors disclosed.

What’s in it: 2,889 certified absence rows, 3,088 span verified extractive QA rows, a 127K token packed long context task set, and a split minted only from SEC filings dated after every major model’s training cutoff. That fresh split regenerates monthly, so it stays impossible to have trained on, by construction.

Every row carries a certificate you can re-check yourself in a few lines of python, the audit snippet is on the card. When our own audits flag something, like extractive answers that are guessable from world knowledge (about 1.6% of them), we label it instead of quietly deleting it.

Also worth knowing before you trust us: a reviewer caught one of our splits being weaker than claimed this week. We re-audited every row the same night, withdrew the split with per row evidence committed to the repo, tightened the protocol, and reshipped only the rows that survive everything. The full trail is in the audits folder, judge for yourself.

License CC BY 4.0. Generation was an Apache 2.0 open weight model on our own hardware, the claim is the verification layer, not the generation. Held out versions never get published so they can’t leak into training data. If anyone wants a sealed diagnostic run against their own model or domain (25 items, free, about a day), contact is on the card.

https://huggingface.co/datasets/SovNodeAI/certified-document-qa

Hi r/MachineLearning,

I’m building a collection of high-quality synthetic datasets for different ML use cases.

Currently have samples ready for:
– Finance / Credit Risk
– Transactions & Fraud
– Ecommerce Customer Behavior

**100K row free samples** are available for anyone who wants to test them.

Question for the community:
What domain or specific problem would you like to see a synthetic dataset for? (Healthcare, Marketing, Insurance, Time-series, etc.)

Comment below with your use case or interests, and I’ll prioritize accordingly + send you the current free samples if you want.

Looking forward to your ideas!

Does anyone have an archived copy of the bulk UK Energy Performance Certificate (EPC) dataset from the old government Open Data Communities portal (epc.opendatacommunities.org)?

Context: On 30 May 2026, the UK government migrated the service to a new platform. In doing so, they completely dropped all pre-2012 certificates from the public register due to them being “expired” and based on older methodologies.

Why I need it: I’m doing property energy analysis, and removing that 2008-2011 dataset creates data gaps for properties that haven’t been sold or rented in the last 14 years.

The original ZIP file was about 5.6GB. I am looking for a snapshot from late 2025 or early 2026 before the site was taken offline.

If anyone has a magnet link, a torrent, or is willing to share a cloud drive link to the original CSVs or a Parquet equivalent, I’d really appreciate it.

I wanted to share a project I’ve been working on called EBBC OpenData, which is a public API and dataset designed to promote Open Science and support bibliometric, scientometric, and informetric analyses. You can find the full project and source code in the repository at https://github.com/GabrielBaiano/EBBC-OpenData

This project provides structured metadata from the publications of the Encontro Brasileiro de Bibliometria e Cientometria (EBBC), which is one of the main events on metric studies of information in Brazil. Through this API and dataset, you can easily query detailed information about authors and their academic networks, articles and papers (including titles, abstracts, and publication years), institutions associated with the research, keywords, thematic trends, as well as references and citations.

The core metadata and documentation are currently being organized, and I am actively working on translating the API documentation and dataset fields into English and Spanish to make the project fully accessible to the global research community.

Since this is an ongoing project, I would highly appreciate your thoughts and feedback. I am especially interested in knowing what features or endpoints would make this more useful for your research, any suggestions you might have regarding the data structure or documentation, and any general tips on best practices for open-data APIs. Please feel free to check out the GitHub repository, open an issue, or leave a comment below. Thanks for your support!

I’ve been deep-fine-tuning small local models (BitNet 1.58, Qwen 1.7B/4B, Gemma-4) into a real on-device assistant with actual tool-calling. The hard part was never the training loop, it’s the data. Scraped

ShareGPT dumps collapse small models into formulaic slop and teach tool syntax that doesn’t match your runtime. And for full/deep fine-tuning (not LoRA), bad data is fatal: a small model trained on redundant, single-voice data monocultures hard.

So I built a pipeline where every example has to survive a set of hard gates before it’s allowed in.

Sharing how it works, because I rarely see people talk about the controls, only the volume.

The core: hand-written “gold seed” → multi-teacher expansion

– The seed is hand-written, one example at a time, in a neutral model-agnostic format ({messages, tools}).

– It’s rendered per dialect: ChatML for Qwen/BitNet, native Gemma tool-call format for Gemma. Same data, correct syntax for each target.

– From a curated seed it expands to hundreds of thousands of examples on demand — volume comes from multiple teacher models of different families (anti style-collapse), each example tagged with the teacher that produced it. You scale to the size you actually need.

The gates (this is the value)

– Anti-formula: blocks over-used openers/closers at ingest + global frequency caps; any phrase repeated too often anywhere gets flagged. This is what stops a full fine-tune from collapsing into one voice.

– Semantic dedup (BGE-M3): near-duplicate detection, not byte-match. On one 9k combinatorial corpus it found 43% near-duplicates, paraphrases expansion would’ve amplified. Keeps 1 per cluster, with a coverage guard that never drops the sole example teaching a capability.

– Flow gate: multi-turn tool-call integrity: every assistant tool_call is followed by exactly its results, no orphans, no pending calls, ends with a real answer.

– Dialect gate: every tool-call is round-tripped through each target model’s syntax and rejected if it doesn’t parse back identical. train == runtime, guaranteed.

– Coverage: every tool taught above threshold; catalog-sync rejects phantom (hallucinated) and untaught tools.

– Vision routing: vision examples go only to vision-capable models; text-only models never see image context they can’t use at inference.

– Judge routing — teacher outputs that pass go to SFT; the ones that fail become KTO negatives (preference signal, not trash).

Built for deep fine-tuning, at scale

The whole point of the gates is that you can full-fine-tune a small model on this without it collapsing and expand a small hand-verified seed to 100k+ examples while keeping every one of those checks green.

Tool-calling, multi-turn, charts/tables/HTML, per-model vision, reasoning, all gate-verified, neutral format so you render to your own dialect.

What it trains

Liara — a local personal AI with 24 real tools (email, calendar, files, notes, web, weather, charts), english and italian-first + multilingual, that resists prompt-injection while still handling legitimate reasoning-test / structured-output prompts (the distinction most classifiers get wrong).

– Liara app: https://nothumanallowed.com/local

– Tooling / code: https://github.com/adoslabsproject-gif/Liara-toolkit

Hello!
I’ve spent a couple of months polishing a series of tools designed to analyze Reddit data to create statistics on various topics. A lot of examples on my profile but I’ll put a couple here:
1- Tracking the best method outreach method for freelancers accross reddit discussions
2- Tracking changing sentiment over time on different AI Models
3- Summarizing Reddit opinions on game engines from discussions on game dev subreddits

And much more, you can see more examples on my Reddit profile (some were better received than others).

Here’s what I offer:
– Filtering for niche topics on a list of subreddits: You pick a list of subreddits and select a goal for this study, I’ll arrange NLP and LLMs models to filter for your niche topic out of the thousands of irrelevant posts.
– Extracting json data from the relevant posts and comments, the specific schema for extraction is customizable; You may extract sentiment, entities (dollar amount, days,names, etc..), categories of comparison and much more.
– Cleaning the data into analysis ready data-sets with hierarchical topic modeling for categories if desired
– I can also perform simple analysis on the data for you and provide clean looking charts (made with flourish)

Price : Just the compute, around 10$ for 40-30k rows (depends on the complexity of the topic)

If you or someone you know might be interested don’t hesitate to reach out!

i’m looking for an infrastructure to ingest an entire 600,000+ archive if the 45k samples look good to them.
We have just finished staging over 45,000 fashion, portrait, and lifestyle images on AWS S3, and they are ready for immediate review.

A quick overview of our datasets:

Clear Ownership: We captured and own this entire collection (2002–2026), meaning you get an unbroken chain of title and fully signed commercial releases.

Privacy Options: We offer both the original unedited files and anonymized batches where faces have been neutralized to simplify your compliance.

Full S3 Availability: While the links below cover our 45k sample sets, our complete 600,000+ RAW image archive is already fully staged in our secure S3 buckets and ready for immediate, direct transfer.

Final year CS student here, targeting data science and analytics roles for campus placements.

Been struggling with this question while building my portfolio: does it matter whether your project uses real messy data vs synthetic/clean data?

Real datasets from Kaggle feel either too cleaned already or the same recycled projects everyone does. But synthetic data feels hollow because the hard part — cleaning, feature engineering, deriving meaningful columns from raw data — is already done for you. You’re basically just visualizing something someone else already solved.

Specifically for BI/dashboard projects — if you use synthetic data, the dashboard looks clean and professional but there’s no real discovery or insight because the data was designed to be dashboarded. Nothing surprising comes out of it.

Also practically — if an interviewer asks “where did you get this dataset?” what’s the right answer? Saying “I generated it synthetically” feels like admitting you took the easy route. But lying about the source is obviously wrong. Is there a way to frame synthetic data usage that doesn’t sound like you avoided the hard part?

At the same time I’ve heard people say interviewers care more about what you built on top of the data than where it came from. But isn’t handling bad data literally the core skill in DS?

For people who’ve interviewed at analytics/DS companies or done hiring — how much does data source actually matter? Is a well-executed project on synthetic data better than a mediocre project on real messy data? Or does using synthetic data automatically signal you avoided the hard part?

Please tell me how to join it, guide me.

Hi r/datasets,

I built Energtx, a global energy data platform that standardizes publicly available energy, electricity, emissions, and climate-related datasets.

Current coverage includes:

• 235,000+ structured records

• 170+ indicators

• 106 countries

• Historical data from 1960 to 2025

• CSV, JSON, and XLSX downloads

• Filtering by country, indicator, source, and year

The platform includes data on electricity generation, renewable energy, primary energy consumption, CO₂ emissions, electricity access, nuclear power, carbon pricing, and oil, gas, and coal markets.

The underlying data is compiled from public sources such as the World Bank, Ember, Our World in Data, the Energy Institute, EIA, OECD, IAEA, and Climate TRACE. Source attribution is included with the datasets.

Dataset explorer:

https://energtx.com/datasets

Disclosure: I built and maintain Energtx. The platform is free to browse and does not require registration.

Feedback on the dataset structure, metadata, indicators, and download formats would be useful.

r/ramen
Title: I mapped 62,173 ramen shops in Japan into a free English/romaji API