The concept is simple, 4 Large Language Models, 1 prompt, you’re either matched with a human or an LLM. It’s a Turing Test and and I really need the data and have no way of getting it. I worked my ass off creating this website and I’d be forever grateful if you spent 5 minutes of your time to play a few rounds. Here’s the link: https://the-imitation-project.vercel.app/

I am building a metadata-only index for AI image discovery packs and wanted feedback from people who actually use datasets.

Current shape:

• one JSONL record per image
• prompt fragments when available
• source URL and creator/source attribution fields
• safety labels
• category/style tags
• pack manifests for small curated image sets
• no upstream image files included in the first pass

Example manifest and records are here: https://generatedgallery.com/index/manifest.json https://generatedgallery.com/index/generated-gallery.sample.json

Protocol notes: https://generatedgallery.com/protocol

The use case is prompt research, moodboards, model eval sets, and image discovery where provenance does not get stripped away.

What fields would make this more useful before I publish a larger metadata-only dataset repo?

Especially now with AI crawlers, MCP servers, and retrieval-based systems becoming more common.

Please see title. Im looking to send mail to property owners or management companies in NYC that have buildings with active DOB violations. If someone could help me or direct me in regards to this that would be very much appreciated. Thank you!

I built a large-scale scraping system that can extract data from thousands of sources simultaneously, bypass anti-bot protection, and convert unstructured formats (PDFs, scanned docs, complex HTML) into clean structured datasets.

What public datasets should exist but don’t because:

• Data is scattered across too many jurisdictions (every state/county has their own portal) • No one has aggregated it yet • It’s in PDFs or hard-to-parse formats • Sites actively block automated access 

Not looking to sell—genuinely trying to understand what public data would be valuable if someone aggregated it. If there’s demand, I might build and release it.

Hey r/datasets,

I got tired of manually copying data tables or dealing with messy HTML structures when trying to feed data into my personal scripts and models.

To solve this, I built and hosted a lightweight cloud API that automatically scrapes public web pages, isolates the tables/data grids, and packages everything into an organized, nested JSON matrix.

I wanted to share it here for anyone looking to automate their data gathering pipelines. I set up a free testing tier on RapidAPI that gives you 50 free requests a month to play around with it:

https://rapidapi.com/patcicci4/api/housing-and-wikipedia-data-scraper

Let me know if you test it out or have any feedback on extra features I should add to the parser!

So guys, Im still uploading, about 150GB, about 1.1 billion replies, most from Brazil users (pt-br)

Also give a look at https://github.com/rodrigosf672/orkut-pydataglobal2025 and https://snap.stanford.edu/data/com-Orkut.html

So this one is just raw data, for now, I will later do ML analysis on this, if anyone want to write a paper together about it DM me.

Anyway on HF SalatielJordao/orkut-communities

Open-source dataset of US professions. Two levels:

130 profession profiles in data/professions/us/profiles/. Each is a JSON with 7 sections – daily routine, regulations, tools, jargon, career levels + fears, community channels, labor market. All sourced from .gov, law.cornell.edu, BLS, and professional associations with source URLs attached to every fact. Built by running 7 targeted WebSearch queries per profession.

25 of those profiles also have generated pain bundles in data/professions/us/pains/. 8-15 inferred recurring pains per profession, each paired with a typed spec for the AI tool that would solve it (calculator with inputs/outputs/formula, checklist with steps and statutory refs, document template with variables, reference lookup keys, LLM advisor decision criteria). Generated by feeding the profile to Opus with a deductive system prompt – no web search at the generation step.

Sample of what comes out, from data/professions/us/pains/us-lawyers.json:

• Billable Hours & Fee Calculation (calculator)
• Statute of Limitations Lookup (reference)
• IOLTA Trust Account Reconciliation (calculator)
• Engagement Letter Drafting (template)
• Court Filing Deadline Calculator (calculator)
• … 8 more

And from data/professions/us/pains/us-auto-detailers.json:

• Cost-plus detail job pricing calculator (calculator, includes 2026 IRS mileage rate)
• EPA stormwater compliance checklist (checklist, $64,618/day Clean Water Act exposure)
• California Car Wash Act registration + surety bond (checklist, Labor Code §§ 2050-2067)
• Vehicle intake / pre-inspection form generator (template)
• Quarterly self-employment tax estimator (calculator, 15.3% SE tax)
• … 8 more

Each pain entry has: title, problem (2-3 sentences), affected segment, frequency, time_waste_h, money_risk_usd, source SCOPE section, skill_type, and a typed skill_spec matching the type. Schema docs in data/professions/us/_FORMAT.md.

Backstory: extending an MIT pain-mining repo I’d been running (court records based, B2B angle). Court records don’t have profession-level pain because professionals don’t litigate their own workflow tedium. Switched to web search for regulatory facts + offline LLM deduction for what’s painful given those facts.

Honest positioning: discovery dataset, not validated pain register. Pains are inferred from regulation + daily routine, not from real users complaining. Plausible starting points for customer-development interviews, not conclusions.

Both pipeline stages are in prompts/profession-scan/ so the dataset is fully regenerable. Country-aware – works for any country with adequate online regulatory data.

Repo: https://github.com/AyanbekDos/unfairgaps-os Cleanest single file to open: https://github.com/AyanbekDos/unfairgaps-os/blob/main/data/professions/us/pains/us-auto-detailers.json

MIT. PRs welcome for the remaining 105 profiles or non-US countries.

I built a free searchable site for public lead service line inventory records:

https://leadserviceline.org/

It aggregates public records from state, city, utility, spreadsheet, and PDF sources into address, water system, city, and state lookup pages.

Caveat: the records are only as good as the public inventories they come from. They can be incomplete, outdated, or wrong, and this is not a water test or a replacement for checking with a local utility.

Right now it is a website. If there is demand, I would like to add an API or bulk data access so people can pull the data directly.

I am looking for a live database or dataset that contains public health information of cases related to pandemics, epidemics, etc.

Can be archived or live and regularly updated.

I am currently in the process of completing the Data Analysis Google Course on Couresa. I was wondering if there was any feedback anyone who has completed it can give.

I am wanting to get into data analysis and change my career.

Any tips?

Full disclosure [self-promotion]: I’m the solo builder. Happy to answer questions about the data, methodology, or entity resolution approach.

I built FastDOL, a platform that links federal workplace enforcement records across agencies into a single employer profile. The government publishes this data, but each agency has its own database, its own identifiers, and its own terrible search UI.

The cross-agency dataset links enforcement records from OSHA, WHD, MSHA, EPA, EEOC, OFCCP, OFLC, and others at the employer level with parent-company rollup. The interesting finding: employers cited by 3+ agencies have a 3.4x higher worker fatality rate than employers cited by 1-2 agencies.

Four open datasets available so far, all CC BY 4.0:

• Cross-Agency Federal Violations by Employer (~2.3M rows)
• OSHA Construction Enforcement by Employer (377K rows)
• OSHA Citations Q1 2026 (28,827 rows, citation-level)
• WHD Wage Theft Enforcement Actions by Employer

All hosted on Hugging Face, Kaggle, and Zenodo with DOIs. Full schema, methodology, and BibTeX on the canonical pages: https://www.fastdol.com/datasets

I’ve released two large-scale multilingual pretraining datasets on Hugging Face, built from the HPLT v3 high-quality web crawl. Both are CC0 licensed (public domain) and ready to use with 🤗 Datasets.

📦 Indic HPLT v1

~9.8M documents | ~8.4B estimated tokens | 11 languages
🔗 https://huggingface.co/datasets/AM0908/indic-hplt-v1

Covers: Hindi, Bengali, Punjabi, Urdu, Tamil, Telugu, Marathi, Gujarati, Malayalam, Kannada, English

📦 Indic HPLT v2 (larger successor)

~34.6M documents | ~25.5B estimated tokens | 14 languages |
🔗 https://huggingface.co/datasets/AM0908/indic-hplt-v2

Adds Nepali, Odia, and Assamese on top of v1, with ~3.5× more documents overall.

🔧 How it was built

• Source: HPLT v3 sorted shards (top-scoring documents by WDS quality score)
• Quality filters: 50–100K chars/doc, max 50% non-alphabetic chars, min avg word length 2.0
• Deduplication: SHA-256 exact dedup on all languages + MinHash LSH near-dedup on English (Jaccard ≥ 0.7)
• Pipeline code: https://github.com/ashtok/multilingual-hplt-corpus

I’m not sure if this is the right subreddit (if not, please let me know where this would fit better).

At my company, our SharePoint contains roughly 21,000 folders/files combined, with some paths going as deep as 13 levels.

As an intern, I was tasked with creating a flowchart that lists all folder names and filenames while showing the hierarchy/path structure.

I was advised to focus on just one root folder (out of ~30 total) for now, but even that single folder contains around 13,000 items.

What management ultimately wants is a visual way to understand:

– what files/folders exist

– whether things are stored in the correct location

– what can be moved or deleted

– how the structure could be reorganized

The reorganization decisions themselves are for management to make, my task is mainly to provide a usable visual representation of the structure.

I’m struggling to figure out the best approach.

So far I’ve:

– tried generating HTML visualizations with AI using file paths

– considered using Microsoft Visio

– considered assigning codenames/IDs to folders with a separate legend for reference

But with 13,000 items, every approach still feels too cluttered and difficult to navigate.

I’m also hesitant to use third-party tools/sites because this involves company information.

Does anyone have suggestions on how to approach this in a practical way?

Hey everyone,

(Disclosure: I built this dataset and pipeline myself).

I created a strict Python pipeline to solve the time-drift issue with public financial news APIs. I scraped 400+ high-impact crypto news events (Nov 2025 – May 2026) and mapped their exact UTC publication timestamps directly to 1-minute Binance BTC/USDT candles.

The dataset provides clean T0 anchors and forward-mapped price snapshots (T+5m, T+15m) so you can backtest event-driven volatility decay without look-ahead bias.

The open-source sample and the EDA notebook just received a Bronze medal on Kaggle! You can download the free sample, check out the methodology, and see the visual volatility decay analysis here:

https://www.kaggle.com/datasets/yevheniipylypchuk/bitcoin-news-vs-1m-btc-price-action-2025-26

(Note regarding Rule 5: The Kaggle link above provides a free sample for EDA and initial modeling. If you find the methodology sound and need the full unredacted 6-month historical dataset for heavy backtesting, I do sell the complete version on my Gumroad. You can find that link inside the Kaggle notebook).

Let me know if you have any questions about the timezone synchronization or the scraping logic!

Hey guys

I’m building a physical 3-node cluster (1 Master, 2 Workers, Docker Swarm) for a backend class. I need to distribute a heavy workload to process massive text/JSON data, but I want the final presentation to be actually funny. No boring corporate data!!!!

I’m looking for ideas on what exactly to analyze. I want to calculate crazy metrics, find weird patterns, etc

I was thinking on:
• Analyzing League of Legends chat logs but it is meh

The dataset needs to be easy to find (Kaggle, Hugging Face, APIs) but large enough to justify parallel processing on a cluster pleaaaase

Any crazy ideas or dataset links? Thanks! 😀

If you happen to be in the e-commerce space, please checkout my latest dataset of trending ideas. I researched and compiled a list of 10+ spreadsheets of trending searches on Google that could translate into great product ideas.

All data has been verified against paid Google research tools like SemRush, Ahrefs etc… and while nobody has 100% accurate search volume numbers, this dataset is pretty close to what Google offers.

Also includes Amazon best seller products.

kaizap.com ~ is where you can learn more.

Disclaimer: I own kaizap.com and it’s a paid product, however, I also do provide a free sample dataset as well.

1000 Venture Capital Firms writing seed checks.

Includes:

• investment stages
• sectors
• firm websites
• portfolio links
• office locations

Structured from recent funding activity.

Updated monthly.

https://seedvclist.com

Hey everyone,

I’m trying to look at job satisfaction data across different occupations.

I’m planning on using the General Social Survey, mapping satjob (work satisfaction) against occ10 (2010 Census Occupation).

I attempted to build a cross-tabulation directly in the NORC GSS Data Explorer website, but it flags occ10 as a continuous variable rather than categorical and throws the error: “Continuous Variables (Except Year) are not allowed” when I try to drop it into the tabulation builder.

Could anyone let me know how to fix issue?

Thanks for the help!

Hey everyone!

I’m planning to scrape a few popular job sites to build datasets for Kaggle and need your help picking 3-4 industries to focus on. Since the community is already flooded with standard data roles (DA/DS/ML/DE/etc.), I want to target 3-4 completely different industries for each dataset separately. These could be related to your research, work, or any other areas you think would be useful. The plan is to scrape the data weekly for at least a year, with possible continuation if there’s enough interest. Some initial ideas are government, finance, and food/hospitality, but I’m open to suggestions.

Let me know your thoughts!