Disclosure: this is our own dataset.

Our dataset consists of 362 translation segments annotated by 48 professional linguists (not crowdsourced) across 16 language pairs.

MT systems evaluated: EuroLLM-22B, Qwen3-235B, TranslateGemma-12B.

Language pairs (all from English): Arabic (MSA, Egyptian, Moroccan, Saudi), Belarusian, French, German, Hmong, Italian, Japanese, Korean, Polish, Portuguese (Brazilian and European), Russian, Ukrainian.

Each segment includes full MQM error annotations:

• error category (accuracy, fluency, terminology, etc.)
• severity level (minor, major, critical)
• exact error span in the text
• multiple annotators per segment for inter-annotator agreement analysis

Methodology follows WMT guidelines. Kendall’s τ = 0.317 on IAA – roughly 2.6x what typical WMT campaigns report.

It may be useful for MT evaluation research and benchmarking translation quality.

Dataset: https://huggingface.co/datasets/alconost/mqm-translation-gold

Happy to answer questions about the annotation process!

Around 8 years ago, we had the idea of using geographic data (census, accidents, crimes) in our models — and it ended up being a top 3 predictor.

Since then, I’ve rebuilt that postcode/zip code-level dataset at every company I’ve worked at, with great results across a range of models.

The trouble is that this dataset is difficult to create (In my case, UK):

• data is spread across multiple sources (ONS, crime, transport, etc.)
• everything comes at different geographic levels (OA / LSOA / MSOA / coordinates)
• even within a country, sources differ (e.g. England vs Scotland)
• and maintaining it over time is even worse, since formats keep changing

Which probably explains why a lot of teams don’t really invest in this properly, even though the signal is there.

After running into this a few times, a few of us ended up putting together a reusable postcode feature set for Great Britain, to avoid rebuilding it from scratch.

If anyone’s interested, happy to share more details (including a sample).

https://www.gb-postcode-dataset.co.uk/

(Note: dataset is Great Britain only)

When people start learning data analytics, one common problem is they don’t have access to real company datasets.

I recently researched several practical ways beginners can still practice real data skills like SQL, Excel, and dashboards.

Some useful approaches include:

• Using public datasets from Kaggle or government portals

• Creating sample business datasets for practice

• Participating in Kaggle competitions

• Recreating dashboards from sample datasets

These methods help simulate real work scenarios and build a strong portfolio.

I also wrote a detailed guide explaining practical ways to practice data skills even without real company data.

One pattern we kept seeing while working with LLM systems:

The assistant sounds correct…
but nothing actually happens.

Example:

“Your issue has been escalated and your ticket has been created.”

But in reality:

• No ticket was created
• No tool was triggered
• No structured action happened
• The user walks away thinking it’s done

This feels like a core gap in how most datasets are designed.

Most training data focuses on: → response quality
→ tone
→ conversational ability

But in real systems, what matters is: → deciding what to do
→ routing correctly
→ triggering tools
→ executing workflows reliably

We’ve been exploring this through a dataset approach focused on action-oriented behavior:

• retrieval vs answer decisions
• tool usage + structured outputs
• multi-step workflows
• real-world execution patterns

The goal isn’t to make models sound better, but to make them actually do the right thing inside a system.

Curious how others here are handling this:

• Are you training explicitly for action / tool behavior?
• Or relying on prompting + system design?
• Where do most failures show up for you?

Would love to hear how people are approaching this in production.

Are there any datasets about datasets that could tell what is the average/mean size of all possibly known datasets. I know this is somehow a very unrealistic question but I’m interested to know if there are known conducted research about it.

Chart overview — 5 panels of real NVDA data

What it is

ULTRA is a flat CSV dataset that aligns three data layers on the same 15-minute timestamp:

• GDELT (~1,256 cols): The full GCAM emotional spectrum — WordNet Affect, SentiWordNet, Harvard IV, AFINN, Loughran-McDonald financial sentiment, Moral Foundations, plus geopolitical events (GoldsteinScale, QuadClass, CAMEO codes), media mentions, entity extraction, and macro themes.
• AI Analysis (18 cols): Contextual sentiment from Gemini — not word-counting, but actual comprehension of why sentiment is negative (export controls vs earnings miss vs CEO departure). Includes impact, novelty, actionability, narrative codes, and binary flags.
• Price (16 cols): Multi-source OHLCV from Polygon.io + Twelve Data, VWAP, trade count, cross-source mean and spread, 15-min return.

96 timestamps per day. Currently covering the Magnificent Seven (AAPL, AMZN, GOOG, META, MSFT, NVDA, TSLA).

Free sample + data dictionary

Full day of NVDA data (Jan 2, 2026) — all 1,437 columns, 96 rows. No paywall, no signup.

→ Sample CSV: marketsignal.solutions/data/samples/ULTRA_sample_NVDA.csv → Data Dictionary: marketsignal.solutions/data/samples/ULTRA_DataDictionary.txt

Quick load:

import pandas as pd df = pd.read_csv("ULTRA_sample_NVDA.csv") print(f"{df.shape[1]} columns, {df.shape[0]} timestamps") # AI sentiment + price at market open cols = ["meta_timestamp", "ai_sentiment_score", "ai_impact_score", "ai_narrative_primary_code", "poly_close", "price_return_15m"] print(df[df["poly_close"].notna()][cols].head(10).to_string(index=False)) 

Why I built it

GDELT is incredible — it’s the world’s largest open news database. But it’s raw, unfiltered, and has no ticker mapping. If you want to use it for quant research, you need months of pipeline engineering just to get it into a usable format.

I built the pipeline that: 1. Ingests 3 GDELT streams every 15 minutes (GKG, Events, Mentions) 2. Matches articles to S&P 100 tickers via org-name resolution 3. Parses all 1,256 GCAM dimensions per ticker 4. Runs Gemini AI on every batch for contextual analysis 5. Fuses with multi-source verified price data

The result is a single CSV you can pd.read_csv() and start researching.

What I’m NOT claiming

• This is not “beat the market” data. It’s research-grade alternative data.
• GDELT is open/public — I didn’t create it. I created the pipeline, the AI layer, and the fusion.
• Coverage is currently 7 tickers (Mag 7). S&P 100 expansion is in progress.
• The AI layer depends on Gemini — it’s contextual NLP, not proprietary.

Pricing

$99/month for the Mag 7 live feed. Details at marketsignal.solutions.

Happy to answer any questions about the data, the pipeline, or the methodology.

This dataset is for research purposes. Past patterns do not guarantee future performance.

So… for my project, i want to train a cnn, and i need a dataset consist of user distance (preferably cm) from the device (eg. Laptop, PC, phone). Please help if found any good one!

I am a figurative artist based in New York with work in the collections of the Metropolitan Museum of Art, MoMA, SFMOMA, and the British Museum. I recently published my catalog raisonne as an open dataset on Hugging Face.

What is in it:

∙ Roughly 3,000 to 4,000 documented works currently, spanning 1970s to present ∙ Media includes oil on canvas, works on paper, drawings, etchings, lithographs, and digital works ∙ Metadata fields: catalog number, title, year, medium, dimensions, collection, copyright holder, license, view type ∙ Images derived from 4x5 large format transparencies, medium format slides, and high resolution photography ∙ License: CC-BY-NC-4.0, free for research and non-commercial use 

What makes it unusual:

Most fine art image datasets are scraped, aggregated, or institutionally compiled. This one is published directly by the artist, with metadata mapped from original physical archive records accumulated over fifty years. Every work is fully documented and provenance is intact. It is artist-controlled from the ground up.

The dataset currently represents roughly half my total output. I will keep adding works as scanning continues. It is a living dataset, not a static dump.

It has had over 2,500 downloads in its first week on Hugging Face.

Looking for:

Researchers or developers working with art image datasets who want to discuss potential uses or collaborations. Also interested in connecting with anyone building tools for visual archive navigation, as the Hugging Face default viewer is not adequate for this kind of dataset.

Dataset: huggingface.co/datasets/Hafftka/michael-hafftka-catalog-raisonne

Hey. Just launched a platform with cleaned, formatted data ready to pipe straight into model training, i swear im not trying to promote.

All 12,500 datasets are CC0 and free to download manually. The API just handles bulk or incremental access so you don’t have to write the data pipeline yourself.

And im giving away API access to 5 people who are actively training something. No catch, just want real feedback.

Drop a comment or DM if you’re building something.

hi, i am David, and I need an advise

I am currently developing a data monetization platform, i am still working on the development, but mainly everything is going on the road

What i am worry about is that, in order to prove the platform, the concept and the workflow is actually viable, i am making a research myself, making all the work the platform would do, manually myself

The reason behind this, is because in the past i have already made a blog like website thought for developers and had to leave the project, for no people visited it, and in general even the ones mildly interested eventually leave, having to close everything; I didn´t want that to happen again so i took that decision

Many weeks have passed and in order to prove the platform is viable and to have a proper deployment, i have at least to have 1 dataset buyer and 50 volunteers who i am paying to participate, i have successfully confirmed 5 people to be volunteers in this time and contacted many possible dataset buyers, i have contacted from ai researchers to teachers from various universities, i got some curious replies, asking about the platform and the project on its own, i even got an email from a Standford professor saying the platform sounds like a really valuable resource and will tell his students if someone is interested, but after that no one replied, I keep looking everyday for possible buyers and email them to outstretch, look in forums, post on reddit and other platforms, but not really finding anyone; this problem also applies for the volunteers, however i could ease it a bit since i am using a survey platform and got those 5 who i talked earlier and expecting it to keep getting some more

All this process as been done in parallel with the development of the platform, since i am working alone i tried using antigravity to help with bugs and extra features

it made development more bearable

That is the place i am rn, i don´t wanna end the project, but its squeezing me

What should i do?

I am in need of a large string of english prose, like a book or blog post, that makes use of all 26 letters that is consistent to how often they’re used over all (x, z, q used uncommonly but still included)

hi everyone I am an AI/ML student and currently I am building a project that detects littered garbage by people in public places and calls out people for violating civic responsibility and raise a real time alaram but the catch is this will be detected through IP cameras so I need a valid set of data for the model to detect the garbage that people litter.

please help…

AI-analyzed news sentiment on silver — here’s my free dataset. Feel free to leave your opinion on the quality.

https://www.opendatabay.com/data/financial/b732efe7-3db9-4de1-86e1-32ee2a4828d0

Disclosure: I’m the creator of this dataset / founder of MarketSignal Solutions.

Hi all, I’m looking for a dataset that contains payment statement descriptors and ideally their related merchant.

For example: “AMZN*MARKETPLACE” -> “Amazon”, or “STEAMGAMES.COM 12345” -> “Steam”.

Any help is appreciated

Just finished uploading some new datasets and thought some people here might be interested in some free data. Most files have millions of rows.

Files in /data hosted on GitHub:

https://github.com/appgoblin-dev/appgoblin-data These files are smaller due to GitHub file size limits.

• live_store_apps.tsv.xz: Apps’ store_ids that are currently live on the Google Play and Apple App Store. This TSV includes names and categories.

• store_apps.tsv.xz: All Appgoblin’s known 4m+ Android and iOS app store ids. Many of which are no longer live on the app stores.

• store_apps_metrics.tsv.xz (limited): ~2m ‘Live’ apps only with installs and total ratings. For full apps and metrics see larger hosted one below.

Larger files hosted on AppGoblin:

Download links are free on https://appgoblin.info/free-app-datasets but you’ll need login to see the download URLs:

• store_apps_metrics.tsv.xz, This is all 5m+ apps with with installs, ratings, app rating, release date, store last updated and several other app meta data.

• descriptions.tsv.xz: English language store app descriptions, based on the latest crawls. English language here are apps that were queried for en and checked once for mostly english output, but may still contain non english languages.

Other datasets?

Let me know if there are other datasets you’d like exports of.

I merged the World Happiness Report 2017 with data most happiness analyses never touch: the Schulz et al. (2019, Science) Kinship Intensity Index (cousin marriage, polygyny, lineage, clan structure), historical Western and Eastern Church exposure, religion shares, Yale Environmental Performance Index, Women Peace & Security Index, and World Bank climate data.

One CSV, 155 countries, 34 variables, ready to use. All open-license sources except the EIU Democracy Index (available separately via Our World in Data).

Comes with three companion notebooks: EDA with distance correlation and variable clustering, hierarchical regression, and a HARKing tutorial showing how a seductive GDP satiation pattern fails bootstrap testing.

Dataset: https://www.kaggle.com/datasets/mycarta/world-happiness-2017-kinship-and-climate

Hi all, I’m releasing a dataset of 518,255 Vietnamese legal documents I collected and processed as a personal research project.

Why it matters: Vietnamese is a low-resource language in the legal NLP space. There’s no comparable open dataset of this scale for Vietnamese law.

What’s inside: – Document types: Decisions, Official Letters, Resolutions, Circulars, Laws, … – 2,393 unique issuing authorities – Full text converted from HTML → Markdown – Metadata: title, date, legal type, sector tags, issuing body, signers

Two configs (join on id): – metadata — 9 columns, ~82 MB – content — full text, ~3.6 GB

🔗 https://huggingface.co/datasets/th1nhng0/vietnamese-legal-documents

Happy to answer questions about the collection pipeline!

Hey all,

I’m building an LLM router (a system that routes each incoming prompt to the cheapest model likely to pass, rather than always sending everything to GPT-4). The core idea: if a prompt is simple enough for Mistral-7B, why pay for GPT-4?

I’m currently using the RouterBench dataset a lot. These kinds of data are incredibly valuable because you get multiple model outputs for the exact same prompts, plus metadata like cost/quality, which makes it much easier to experiment with routing strategies and selection policies.

I’m wondering: are there other public datasets or benchmarks that provide:

• The same prompt / input evaluated by several different LLMs
• Full model outputs (not just scores)
• Ideally with some form of human or automated quality labels

They don’t have to be as big or polished as RouterBench, but anything in this spirit (evaluation logs, comparison datasets, crowdsourced model outputs, etc.) would be super helpful. Links to GitHub, Hugging Face datasets, papers with released generations, or hosted eval platforms that export data are all welcome.

If you’ve built your own multi-model eval logs and are open to sharing or partially anonymizing them, I’d also love to hear about that.

Thanks!

theres a bunch of great search tools for the epstein files now (jmail, sifter labs, epstein graph) but they all work the same way.. you type keywords and scroll through results

im thinking about building something different. a conversational layer where u just ask questions by voice or text and it pulls relevant docs with page-level citations across all the datasets. like talking to someone who read everything

i already have infrastructure for this. we built a similar system for 965 holocaust survivor testimonies so the RAG pipeline and voice interface exist. have some free budget to make this a public good project. probably a week to adapt it

before i commit the time:

1. is there a gap here or are existing tools enough
2. what kind of queries would be most useful
3. any specific datasets to prioritize first (doj batches, flight logs, deposition transcripts?)

if theres real interest ill build it

Hi everyone,

I’m currently working on an image deraining project and I need the Rain100H (CVPR 2017 old version) dataset. Specifically, both the training and test sets.

I found the dataset listed here:
https://github.com/nnUyi/DerainZoo/blob/master/DerainDatasets.md
(under Rain100H_CVPR2017 old version)

But the download links are hosted on Baidu Pan, and I’m running into a big issue:

• I’m based in Europe
• I can’t create a Baidu account (no Chinese phone number)
• Most download tools / scripts don’t work anymore without login
• Online “downloaders” either don’t load or require payment for large files

So right now I’m basically stuck…

What I’m looking for:

• Is there a working mirror (Google Drive, Hugging Face, etc.) for the original Rain100H dataset?
• Or would someone with Baidu access be willing to download and reupload just the Rain100H folders?
• Any reliable workaround that still works in 2026?

I’d really appreciate any help. This dataset seems widely used, so I’m surprised how hard it is to access from outside China.

Thanks a lot in advance!

We’ve just opened our Discord community for people working with datasets, LLM training, and AI systems.

This space is meant to be genuinely useful — not just announcements, but ongoing value for anyone building in this area.

Here’s what you can expect inside:

• Regular updates on new datasets (behavioral, conversational, structured, agent workflows)
• Discussions around dataset design, fine-tuning, and real-world LLM systems
• Insights and breakdowns of what’s actually working in production AI
• Early access to what we’re building with DinoDS
• A growing marketplace where you can explore and purchase high-quality datasets
• Opportunities to collaborate, share feedback, and even contribute datasets

Whether you’re training models, building agents, or just exploring this space — you’ll find people working on similar problems here.

Join us: https://discord.gg/3CKKy4h9

kept hitting the same wall every time, find a dataset then spend ages getting it into a usable state before I could do anything with it.

so I built Neurvance. every dataset is already cleaned, formatted and ready for model training. you can browse and download for free or use an API key if you need bulk or incremental access.

genuinely curious what types of datasets would be most useful to people here, still at the stage where feedback shapes what I build next

neurvance.com