https://github.com/leakyhose/open-trivia-dataset

Pulled it from open trivia database, they lock the questions behind an API call that only returns 50 each time. Ran a script that repeatedly calls it, storing the questions and sorting them by difficulty and category.

I’m working on a project that required accurate hierarchical Japanese location data
(prefecture → city/ward/town/village).

Since most publicly available datasets were outdated, inconsistent, or missing entries,
I compiled a clean version from multiple official sources.

It includes:

• 1 country
• 47 prefectures
• 1,700+ municipalities
• consistent hierarchical IDs
• UTF-8, machine-friendly
• suitable for forms, address validation, GIS, ML, and location-based apps

If anyone is interested, I’m happy to provide details or export it as CSV / SQL.

The full JSON dataset is available here (paid):
https://makotocroco.gumroad.com/l/japan-locations

(self-promotion: this is my own dataset)

Hi everyone,

I’ve been working as a structural engineer for about 10 years (Germany, RC design).
Over the last few years I’ve noticed something very surprising in AI/ML:

We have datasets for almost everything — but none for real structural engineering drawings.

These drawings are extremely challenging for machine learning due to:

• dense, overlapping geometry
• structural symbols and reinforcement notation
• dimensions, leaders, section markers
• multi-layer technical detailing
• scale-dependent information
• mixed text + geometry + symbols

Because of this, they are highly relevant for:

• OCR / document understanding
• object detection
• layout analysis
• symbol recognition
• segmentation
• BIM automation
• engineering-focused CV research

So I started building a series of datasets of real reinforced-concrete drawings, created specifically for ML tasks.

Each dataset contains:

• 25 PDF engineering drawings (Columns 50 PDF)
• 25 PNG images (1200 dpi) (Columns 50 PDF)
• one structural category per dataset (RC beams, walls, foundations, columns, precast columns, etc.)

So far I’ve released 6 datasets:

• RC Beams V1
• RC Columns V1
• RC Foundations V1
• RC Precast Columns V1
• RC Walls V1
• RC Walls V2

All datasets, including sample images, can be viewed here:

👉 [https://huggingface.co/PNEngineeringDatasets]()

I’d be happy to hear any feedback, suggestions or use cases you think could be valuable for ML research in this domain.

Disclaimer: this is my own dataset project; posting once for visibility.

Hello guys. Any idea where I could get a free dataset containing monolingual dictionary (word- definition pairs in the same language) in multiple languages? I got english from kaikki(wiktionary) but it is missing other language ‘senses’. WordNet might be no good, since I need sensible definitions. I’m considering making it myself from the wiktionary dumps of different languages, but I thought it might be better to ask first

GICS (The Global Industry Classification Standard from MSCI) and ICB (Industry Classification Benchmark from FTSE/LSE/Dow Jones) seem to dominate the securities industry sector data market.

There are alternatives available from players such at ICE, but in all cases, they are proprietary, and as far as i can tell pretty much identical.

11 top level sectors, which are then split into more and more granular sub-categories.

I’m fairly certain that nobody really has any use for the most granular sub-sectors which contain >160 sectors… But the high and mid level classifications would be really useful.

You can theoretically grab sector weightings data from Yahoo Finance by ticker code… But i’d ideally like to be able to use either Sedol or ISIN to look values up.

I’m sure there are others who would like something like this, so before i think about trying to create my own gizmo for it i was wondering if anybody has done anything similar?

Would you please help me finding publicly available multimodal (image, audio or sensors) healthcare related datasets for novel research?

Interlock is a safety layer for production AI stacks that does three things: detects degradation/hazard, refuses or degrades responses when confidence is low, and records cryptographically verifiable evidence of the intervention. The repo includes middleware (Express, FastAPI), adapters for 6 vector DBs, CI-driven stress chamber tests, benchmarks, and certified badges with signatures. Repo & quickstart: https://github.com/CULPRITCHAOS/Interlock

What’s novel / useful from an ML perspective

Formal primitives (Hazard, Reflex, Guard, State, Confidence, Trust Decay) to reason about operating envelopes for LLM/RAG systems.

Stress-chamber + production-simulation CI workflows that inject latency/errors to evaluate recovery & cascade risk.

Evidence-over-claims approach: signed artifacts that let you prove interventions happened and why — useful for audits, incident triage, and model governance.

Restart continuity: protection survives process restarts (addresses anti-amnesia).

Key experimental results (from v5.3 README)

False negative rate: 0% in validated scenarios

False positive rate: 4.0% (tradeoff to reduce silent corruption)

Recovery time mean: 52.3s, P95 ≈ 58.3s

Zero cascading failures & zero data loss in tests

What you can find in the repo

Middleware for Express and FastAPI to add Interlock to existing stacks

Stress chamber scripts that run protected vs control comparative experiments

Benchmark suite and artifact retention of evidence and certification badges

Live-monitor reference service and scripts to reproduce injected failures

Documentation: primitives, validation artifacts, case study, and live incidents

Why this matters for ML ops & research

Bridges the gap between research on LLM calibration / confidence and production safety tooling.

Provides a repeatable evaluation pipeline for failure‑survivability and impact analysis (including economic impact reports).

Enables measurable trade-offs (false positives vs safety) with reproducible artifacts to tune policies.

Suggested experiments or avenues for feedback

Calibration strategies that reduce FPR while keeping FN≈0

Alternative reflex actions (partial answer + flagged sections vs full refusal)

Integration with downstream retraining / feedback loops using forensic logs

Domain-specific thresholds (healthcare / finance) and legal/compliance validation

This is MY FIRST INFRA PROJECT and a new coder. Any suggestions or feedback I’d GREATLY APPRECIATE IT!

Hello,

I want to get some opinions and recommendations on statistical methods that could be used for my analysis.

The plan is to collect data through a survey and a database search. The results will be used as input and output for Data Envelopment Analysis (DEA). The target of the survey is a decision-making unit (DMU).

There are eight input items and two output items. The score for the input items will be based on the survey responses received. For output items, the score will be calculated using data from public databases.

Each item comprises questions with different types of answers. These include yes/no questions, questions where you select one of statements 1–5, and numerical questions. The number of questions for each item varies depending on its specific characteristics.

This is how I grade each answer and calculate the total score for each item.

Scoring answers:

Type A question: yes/no, YES is given score 3, NO is given score 1

Type B question: A score from 1 to 5 is given based on the score of the selected answer

Type C question: numerical question. The number (n) will be given a score based on the calculation of the mean/median of all the collected answers. If n < Q2, the score is 1; if n = Q2, the score is 2; and if n > Q2, the score is 3.

I then sum up the grades from all the questions in each item. The final score for an item is = total grade/max grade*5 (I set the highest score for an item as 5)

A radar chart for a DMU will be developed showing the scores of the 8 input items.

For the output items:

The data is derived from a public database. I classify the data from each DMU into one of four groups based on quality.

For the scoring:

1. derive the frequency number from database
2. calculate the median for each group
3. set the grade as 1 to 3 (same as the type C question)

4.Because I want to give different weights to each group so that the data from the high-quality group contributes more to the total score. A multiplication factor depending on the group will be applied to each grade, as follows:

Output1

This is how I set the input and output values for each DMU.

Question:

1. Is this kind of scoring acceptable, even when there are different types of questions for each input item?
2. Is there a scientific method that can be applied here? For example, how should the score for each answer be set? I have found papers that use scoring in their surveys, but their questions are usually of the same type, producing the same type of answer (e.g. a Likert scale).

Any comments or advice would be appreciated, also if anyone can recommend me any references that would be awesome.

Thank you.
marlee

I need a chat dataset to train a model like these friends or virtual girlfriend I want it to be able to enter into a conversation in turns

Hello I’m looking for a tomato leaf dataset for environmental conditions such as high/low humidity and lightning for my thesis. Most of the datasets on web focuses on diseases. Can anyone help please, thanks!

We’re undergraduate Machine Learning students working on a crop disease generation project using CGANs, aimed at supporting global sustainability. 🌱

Right now, we’re looking for wheat images of yellow rust disease along with metadata like region, severity, and time range for model training and evaluation.

If you know of any public datasets, research projects, or institutional resources, or even just pointers on where to look, we’d really appreciate your guidance.

Thanks so much for your help! Any leads will be credited in our project.

I have a large database/wordlist containing probably every English dictionary word plus many additional ones like brand names, but this naturally includes many words no longer in use. I need to cut down the size of the list, but since too many words have been added to it to start from scratch, my plan is to obtain a corpus of only archaic words and use these as negatives to remove from the main wordlist. Does such a corpus/wordlist exist anywhere in text form, even it’s just a word per line? Thank you in advance, any help is much appreciated.

We are building an automotive vehicle check platform focused on the European market and we are looking for a long-term technical collaborator, not a one-off freelancer.

Our goal is to collect, structure, and expose automotive-related data that can be included in vehicle history / verification reports.

We are particularly interested in sourcing and integrating:

• Vehicle recalls / technical campaigns / service recalls, using public sources such as RAPEX (EU Safety Gate)

• Commercial use status (e.g. taxi, ride-hailing, fleet usage), where this can be inferred from public or correlatable data

• Safety ratings, especially Euro NCAP (free source)

• Any other publicly available or correlatable automotive data that adds real value to a vehicle check report

What we are looking for:

• Experience with data extraction, web scraping, or data engineering

• Ability to deliver structured data (JSON / database) and ideally expose it via API

• Focus on data quality, reliability, and long-term maintainability

• Interest in a long-term collaboration, not short-term gigs

Context:

• European market focus

• Product-oriented project with real-world usage

If this sounds interesting, feel free to comment or send a DM with a short intro and relevant experience.

I am trying to define what makes a dataset “enterprise-ready” versus just a dump of files. Regarding structure, do you generally prefer one monolithic archive or segmented collections with manifests? I’m also looking for best practices on taxonomy. How do you expect keywords and tags to be formatted for the easiest integration into your systems?

One of the biggest friction points seems to be legal clarity. What is the clearest way to express restrictions, such as allowed uses, no redistribution, or retention limits, so that engineers can understand them without needing a lawyer to parse the file every time?

If you have seen examples of “gold standard” dataset documentation that handles this perfectly, I would love to see them.

Thanks again guys for the help!

Hi, I’m working on a university project building an AI-based interview system (technical + HR). I’m specifically looking for datasets related to interview questions, interview responses, or behavioral/self-awareness analysis that could be mapped to concepts like the Johari Window (Open/Blind/Hidden/Unknown).

Most public datasets I’ve found focus only on question generation, not behavioral or self-awareness labeling.
If anyone knows of relevant datasets, research papers, or even similar projects, I’d really appreciate pointers.

Thanks!

# r/datasets – ScrapeGraphAI 100k Post

Announcing ScrapeGraphAI 100k – a dataset of 100,000 real-world structured extraction examples from the open-source ScrapeGraphAI library:

https://huggingface.co/datasets/scrapegraphai/scrapegraphai-100k

What’s Inside:

This is raw production data – not synthetic, not toy problems. Derived from 9 million PostHog events collected from real users of ScrapeGraphAI during Q2-Q3 2025.

Every example includes:

– `prompt`: Actual user instructions sent to the LLM

– `schema`: JSON schema defining expected output structure

– `response`: What the LLM actually returned

– `content`: Source web content (markdown)

– `llm_model`: Which model was used (89% gpt-4o-mini)

– `source`: Source URL

– `execution_time`: Real timing data

– `response_is_valid`: Ground truth validation (avg 93% valid)

Schema Complexity Metrics:

– `schema_depth`: Nesting levels (typically 2-4, max ~7)

– `schema_keys`: Number of fields (typically 5-15, max 40+)

– `schema_elements`: Total structural pieces

– `schema_cyclomatic_complexity`: Branching complexity from `oneOf`, `anyOf`, etc.

– `schema_complexity_score`: Weighted aggregate difficulty metric

All metrics based on [SLOT: Structuring the Output of LLMs](https://arxiv.org/abs/2505.04016v1)

Data Quality:

– Heavily balanced: Cleaned from 9M raw events to 100k diverse examples

– Real-world distribution: Includes simple extractions and gnarly complex schemas

– Validation annotations: `response_is_valid` field tells you when LLMs fail

– Complexity correlation: More complex schemas = lower validation rates (thresholds identified)

Key Findings:

– 93% average validation rate across all schemas

– Complex schemas cause noticeable degradation (non-linear drop-off)

– Response size heavily correlates with execution time

– 90% of schemas have <20 keys and depth <5

– Top 10% contain the truly difficult extraction tasks

Use Cases:

– Fine-tuning models for structured data extraction

– Analyzing LLM failure patterns on complex schemas

– Understanding real-world schema complexity distribution

– Benchmarking extraction accuracy and speed

– Training models that handle edge cases better

– Studying correlation between schema complexity and output validity

The Real Story:

This dataset reflects actual open-source usage patterns – not pre-filtered or curated. You see the mess:

– Schema duplication (some schemas used millions of times)

– Diverse complexity levels (from simple price extraction to full articles)

– Real failure cases (7% of responses don’t match their schemas)

– Validation is syntactic only (semantically wrong but valid JSON passes)

Load It:

from datasets import load_dataset dataset = load_dataset("scrapegraphai/sgai-100k") 

This is the kind of dataset that’s actually useful for ML work – messy, real, and representative of actual problems people solve.

Hi,

Loking for a dataset / API that contains information about Football players, their nationalities, clubs they played at, their coaches and their individual & team trophies.

Most of the API-s / Datasets out there are either, oriented on the football and game tactical analysis, or transfer market, so I could not find reliable data source.

Tried Transfermarkt data but it has a lot of inaccuracies, and it has limited history. Need something rather comprehensive.

Any tips?

I have been trying to find a way to get into the Fortnite kernel so that I can record myself playing and have the automatic annotations, hopefully, as well as the perfect character representation from reading the memory.

Any tips to get around easy Anti-Cheat? no injection just reading.

I’m trying to identify repositories that are growing the fastest in GitHub and came across gharchive.org. Has anyone used this before / have a better solution?

I’ve gone down a rabbit hole trying to analyze the “Athlete ROI” of different zip codes. Basically, I want to build a heatmap that shows which high schools are statistically over-performing at sending kids to college on athletic scholarships (specifically D1/D2 commits). My theory is that there are “hidden gem” public schools that produce just as many elite athletes as the $50k/year private academies, but the data is impossible to visualize because it’s all locked in individual profiles. I’ve looked at MaxPreps, 247Sports, and Rivals, but they are designed for tracking single players, not analyzing school output at scale. The Question: Does anyone know of an aggregate dataset (or a paid API) that links: High School Name / Zip Code Total Commits per year (broken down by D1 vs D2 if possible) Sport Category

I’m trying to avoid writing a scraper to crawl 20,000 school pages if a clean database already exists. Has anyone worked with recruitment data like this before?

TL;DR: I am aggregating all public releases regarding the Epstein estate (House Oversight docs, DOJ disclosures, flight logs, multimedia) into one repository. While I finish processing the data (OCR and Whisper transcription), I have opened my Dropbox for public access to the raw files.

This archive aims to be a unified resource for OSINT analysis and research. It expands on previous dumps by combining the recent November 2025 House Oversight releases with the DOJ’s “First Phase” declassification.

• Note: I am still in the process of uploading some of the larger media files, so keep checking back. However, it currently contains ALL the raw pdf’s from every source (fbi, house/senate, doj, etc), including the most recent (tho heavily redacted) release

To avoid bots scraping, the Dropbox is password protected, but you can access it via password. The pass is my username for my github account, theelderemo

I am currently running a pipeline to process these files to make them fully searchable:

OCR: Extracting high-fidelity text from the raw PDFs.

Transcription: Using OpenAI Whisper to generate transcripts for all audio and video evidence.

Once the processing is complete, the structured dataset will be hosted on Hugging Face, and I will be releasing a Gradio app to make searching the index user friendly.

Please Watch or Star the GitHub repository. That is where I will post the updates, the link to the final Hugging Face dataset, and the search app once they are live.

Github Repo

Dropbox with all files

Original Repo for 20k Emails (this contains the november dataset and gradio search app)

content warning: This repository contains graphic and highly sensitive material regarding sexual abuse, exploitation, and violence. It also contains unverified allegations. discretion is strongly advised.

EDIT: apparantly subfolders are not being publicly shared for some reason, so only the top parent folder is shared in dropbox. I’m cloning them to my google drive. Be patient with me, lol. I’ll update the dropbox link to the drive link once it’s done. It’s over 150gb.

Here’s the link for the google drive
It is being updated via a script in colab cloning my dropbox to the drive, so each refresh will have new folders/docs.

For now, here’s individual share links for each subfolder:

https://www.dropbox.com/scl/fo/mu2ebqnutbehj5ix063hi/AO_gd0QCu7dopIc5KulYqcs?rlkey=eoqzz5a8x9v1qsjotxmwax8ed&st=7d5tjzjq&dl=0

https://www.dropbox.com/scl/fo/lhdne8ebxvih4z9y83aqj/ACFUeplO_SCiCYF6PLVQTNE?rlkey=miisoobzylco8hzhc8yjtfbim&st=3k6uha26&dl=0

https://www.dropbox.com/scl/fo/xmgoirs4n1cjobpu45wgo/AH-YxKPuoecKz2cvrV24xtA?rlkey=6dmiuieavbifgucvtmhxg5oz2&st=fm2lceeb&dl=0

https://www.dropbox.com/scl/fo/nommub0xf7yw1uvnzzu6s/ACPTR-QCmzRj_-YXUFnONws?rlkey=zf0e1l0tggxagphvl8z0qj1j2&st=hlsvrqf8&dl=0

https://www.dropbox.com/scl/fo/q4sjrvwfemg3uwx63kgiz/AP_HvwExmO7YxYD32Nixvwg?rlkey=ygb0w2ardd1vud5tknr2xf6zv&st=y0pyxhv3&dl=0

https://www.dropbox.com/scl/fo/va3f0oraph91wljz2dhst/AFkaQGsAPDWad4U9gg8_8Ag?rlkey=hjkyqs6q9hqjttf8dvot6c5w4&st=vd1f6rk1&dl=0

https://www.dropbox.com/scl/fo/k3hwoqmax72un20ok70cy/AHmkB7YPXV_6xRLtDRNxPVQ?rlkey=7ak8w1dm2iyzvjxuqjxd5qsoo&st=uroug8x1&dl=0

Okay, so here’s the deal – I somehow ended up with this massive file that’s got like a million lines of what looks like Israeli car data. It’s all separated by these pipe characters (|) and has Hebrew writing mixed in. From what I can tell by looking at it, it’s got stuff about different cars – models, years, engine info, all that – but written out in Hebrew. Kinda wild.

02263039|0650|P|ñåáàøå éôï|0226|GP3ELCC|XV|XV|1.6 PREMIUM|5|14|2016|FB 16

02258339|0650|P|ñåáàøå éôï|0247|GP7ELUC|XV|XV|2.0I|5|14|2016|FB 20

02279939|0650|P|ñåáàøå éôï|0253|SJ5DL7C|FORESTER|FORESTER|2.0XS|5|14|2017|FB 20

02247639|0650|P|ñåáàøå éôï|0243|GP7ELTC|XV|XV|2.0 PREMIUM|5|14|2016|FB 20

01851239|0650|P|ñåáàøå éôï|0228|GP7ELUC|XV|XV|2.0I|1|14|2017|FB 20

What I’ve Figured Out:

• Pipe-delimited format
• Column 4: Hebrew vehicle descriptions (decodes to makes/models like Honda CR-V, Seat, BMW)
• Column 12: Year (1999-2017+)
• Column 13: Engine codes (G4LC = Hyundai/Kia 1.4L, etc.)
• Columns 10-11: Likely cylinders and engine displacement
• ISO-8859-8 encoding for Hebrew

Questions for the Community:

1. Does anyone recognize this specific data format or structure?
2. What industries would find this data most valuable?
3. Any creative but legitimate applications for this type of automotive dataset?
4. What are the best ways to process/enhance this data?
5. Any Israeli-specific considerations I should know about?
6. Has anyone worked with similar automotive data commercially?
7. What might the other columns represent (1-3, 5-9)?

I have technical skills (Python, SQL, APIs) to work with this but need domain knowledge about what’s actually valuable here and how to properly interpret the structure.

Not looking to share the full dataset publicly, but happy to provide more samples if helpful for analysis. Interested in legitimate applications and technical insights.

Thanks for any help!

It includes batsman, bowler, matches related different files if u like the dataset dont forget to upvote it

In my past job, and I want to as vague as possible, there was a need for data manipulation/migration/backup numerous times over the cours of say 2 years. There were almost no safety standards at place in handling the data, I couldn’t believe some of the tasks I was assigned by the management, for example I was supposed to back them up on my local machine temporarily, etc.

I don’t want to go more into detail and possibly get anyone (myself included) in trouble.

I was just curious, how much would be the data worth on open (and possibly black) market? I had no intention of betraying anyone but I wondered this for a couple of years now just being in awe how much the management was risking in trusting several people without having no protocols at place. I am pretty sure our contracts had no clauses about leaking data etc.

The data contained about 5-7000 mortgage details over a span of 5-7 years and its entire screening process (a very complex data model) – the applicants’ health reports based on their medical records, their specific and verified assets and liquidity, verified income, liabilities, the property information etc., banking information, contact information. Anything that would be required in a screening process for a mortgage was basically in the dataset. Lots of sensitive and personal information.

I don’t want to specify the country exactly, you may consider it was either USA, UK, or Canada.

And just to clarify, I would never do anything illegal with the data as I appreciated the people and had no intention of going to jail.