I want to get this DR(eye)VE dataset from AImage Lab
https://aimagelab-legacy.ing.unimore.it/imagelab/page.asp?IdPage=8

But the form on this site doesnt seem to work. So I tried contacting them through the methods in their new website
https://aimagelab.unimore.it/contacts/

But no responses to emails and even calls are stuck in a automatic response loop in Italian.
Does anyone have this dataset, or a similar one or know how I could ontain this via AImage Lab?

Any support is welcome! Thank you.

hi! So I have this dataset where the objective is to predict a student’s health risk, but I’m a lil confused about how to handle the null values. These are the % of null values for the columns:

 id 0.000000 health_condition 0.000000 sleep_duration 11.012943 heart_rate 1.135073 bmi 2.013946 calorie_expenditure 7.658878 step_count 2.016554 exercise_duration 1.000017 water_intake 6.300211 diet_type 1.000017 stress_level 12.000064 sleep_quality 8.452690 physical_activity_level 5.306715 smoking_alcohol 4.141791 gender 3.097141 dtype: float64id 

What would you recommend I do for these values? If I were to drop the columns <5%, I would be losing nearly 100,000 values (out of 700,000) which I don’t think is all that good. I thought of using K-means to fill the null BMI values but I don’t know.

I would appreciate any advice! Thanks 🙂

Hello everyone, I’m working on my university thesis and I was wondering if any of you would be willing to share access to datasets pertaining to the movie industry, specifically datasets containing Yearly box office revenue or amount of tickets sold for specific movie genres, more precisely for the horror genre. I started some searches already but the promising leads I found were pretty expensive.

I’m trying to find a database of surnames for use in writing/testing code that converts an author name (e.g, “Stan Sieler”) into a sortable/alphabetizable name (e.g, “Sieler, Stan”).

Many surnames are compound (“de Camp”, “Cartwright-Chickering” (bonus for people who recognize that one!), some with and some without hypens, and some with more than two words.

The U.S. Census database isn’t useful to me … they compress all last names, removing spaces.

(I’m ignoring people like “Arthur Conan Doyle”, whose last name at birth was “Doyle”, but later adopted the practice of using “Conan Doyle” as his surname … confusing librarians around the world 🙂

Any pointers appreciated, thanks!

The PSID has tracked the same American families since 1968 across income,

health, housing, wealth, education, and depression. It’s one of the most

powerful public datasets in social science, but the raw files arrive with no

meaningful column names and require a codebook crosswalk just to understand

what you have.

PSID-SHELF (from U-Michigan) reorganized the data into 34 topic areas with

real variable names. There’s now a browser app built on top of it — search

across all 34 topics in plain English and see sample data immediately. No

download, no account, no setup. Link in comments.

There’s also a local track that produces 34 clean CSVs from your own SHELF

download, ready for pandas, R, or Excel.

Happy to answer questions.

I’m currently working on a cosmetics/skincare startup and one thing I’ve been struggling with is finding reliable market data. Whenever I need information like market size, growth rates, consumer trends, pricing, competitor analysis, retailer performance, ingredient trends, or industry forecasts, I end up finding reports that cost anywhere from hundreds to thousands of dollars.

For those of you who regularly work with market research or data:

Which platforms do you actually use?
Which ones are worth paying for?
Are there any hidden gems that professionals use but aren’t widely known?
How do startups without huge research budgets access high-quality data?
Do you combine multiple sources (government data, retail data, consumer surveys, Google Trends, etc.) instead of relying on one platform?
I’m particularly interested in the beauty, cosmetics, skincare, and consumer products industries, but I’m also curious about general-purpose research platforms.

I’d love to hear what professionals, analysts, consultants, or founders use in their day-to-day work.

Was poking at the SF market and put together a clean dataset of homes + condos currently listed: list price, price/sqft, sqft, beds/baths, year built, lot size, agent, and the Redfin link for each.
A few things that jumped out:
– Median list price is ~$1.27M, median $980/sqft
– Cheapest thing on the market: a $369k 523-sqft condo at 601 Van Ness
– Priciest: a $6.6M unit at 188 Minna — which works out to $3,256/sqft lol
– Year built ranges from 1884 to 2021, which is very SF

CSV/XLSX here if anyone wants to take a look at it: https://docs.google.com/spreadsheets/d/17BhnTFkWtN6cI9Yn9f0BgPcLF6sVEk9T/edit?usp=sharing&ouid=108885207033845537587&rtpof=true&sd=true

Made it with an open-source tool called Bigset where you basically describe the dataset in a sentence and it goes and pulls + verifies the data from the live web.

Happy to pull a different slice if people want -by neighborhood, condos only, under $1M, whatever.

Hello folks i need an advice from DBAs.
I’m building a gentle data extractor from dabases.

What’s the most important metric that can confirm that ongoing data extraction is not harmful for database?

Started out with a question, how do I spend $300 in free GCC credits, and how much could I do with it. I started with figuring out how to query HTTP Archives, pulling CRuX data to correlate sites, and learning a bit about BigQuery along the way. I went from ~12 million total sites and pared that down to 1.5 million that I could verify were live, had enough data to be able to classify/categorize, and then built a front end to access the highlights.

So far, I’ve been focused on identifying key business segments with missing opportunities, classic one click misses, some schema mapping for business type, and wondering why in the world any sane business owner would use Weebly.

What would YOU want to know?

Sharing a dataset resource that may be useful for researchers, data scientists, and investigators.

Image-Meta has indexed the embedded metadata (EXIF/IPTC/XMP) from ~720 million publicly accessible images using ExifTool. The data is queryable via a REST API & web rather than a bulk download.

**What’s in the dataset:**

– Camera make, model, and serial number

– Author, copyright, rights, title, description

– GPS coordinates (where present, subject to strict TOS/paid tier not publicly free available)

– Software chain

– Creation, modification, and index dates

– Filename and document ID

– Creation, Modify Date, Date Found

– Extra JSON supplimental metadata in full per image

**Potential research uses:**

– Camera device attribution studies

– Metadata privacy/leakage research

– Image provenance and disinformation analysis

– Geospatial studies using embedded GPS

– Timeline reconstruction of image publication

**Access:**

Web or Queryable via REST API with field-level boolean search, date ranges,

https://image-meta.com

API docs: https://image-meta.com/api-docs

I just released a free sample dataset of 50 traditional recipes from 25 countries.
Each recipe includes:
Ingredients
Step-by-step instructions
Prep time & cook time
Serving size
Format: JSON
The full dataset contains 1,925 recipes from 194 countries and is available on HuggingFace under the name:
“FoodieAtlas World Traditional Recipes Dataset”
Disclosure: I am the creator of this dataset.

Please someone provide me with this, there is no risk for you with this so no need to worry about safety.

Finding a clean, structured list of financial instruments has always been harder than it should be. Bloomberg sells it. Refinitiv sells it. Yahoo Finance gives you a search bar. If you want “all biotech companies listed in Germany” or “all fixed income ETFs from Vanguard” as a filterable dataset, you’re usually either scraping something or paying for a data vendor. I’ve spent the last few years building and maintaining a free alternative.

The Finance Database covers seven asset classes across 300,000+ symbols:

Each equity record includes: symbol, name, currency, sector, industry group, industry, exchange, market, country, city, market cap tier, ISIN, CUSIP, FIGI, composite FIGI, share class FIGI, and website. ETFs and funds carry family, category group, and category instead of GICS-style classification. Every record has what you need to cross-reference against other data sources.

The data is an aggregation of publicly available sources – no paid API required to use the database itself. It is community-maintained, MIT-licensed, and lives on GitHub as CSV files you can open in Excel if that’s your preference.

The Python package gives you structured filtering and text search:

“`python

Install via: pip install financedatabase -U

import financedatabase as fd

equities = fd.Equities()

All semiconductor companies in Taiwan on primary listings only

equities.select( country=’Taiwan’, industry=’Semiconductors’, only_primary_listing=True )

Free-text search: robotics or automation companies on the Frankfurt exchange

equities.search( summary=[‘Robotics’, ‘Automation’], index=’.F’ )

Explore what’s available before filtering

fd.show_options(‘equities’) “`

The show_options call is useful before you filter – it returns every distinct value per column without loading the full dataset, so you can scope your query without memory overhead.

For anyone doing universe construction for backtests or systematic strategies, the ISIN/FIGI coverage is the most practical part. You can pull a filtered symbol list here and pipe it directly into your price data provider.

The database is not a price or fundamentals source – that’s intentional. Metadata and categorization data is the hard part to get for free and I’ve built a seperate tool for that, the Finance Toolkit.

GitHub page: https://github.com/JerBouma/FinanceDatabase

Most of the work isn’t building charts.

It’s preparing the data before the analysis even starts.

Cleaning files.
Fixing formats.
Validating data.
Checking structures.
Transforming datasets.

The better your preparation process is, the easier the actual analysis becomes.

What part of data preparation do you find most annoying?

I’ve spent the last few weeks working on a pipeline to solve a problem that has frustrated me (and likely other researchers) for a while: working with arXiv source files at scale.

If you have ever tried to analyze the LaTeX source code of arXiv papers, you have probably run into two major roadblocks:

1. The Egress Tax: arXiv’s official bulk S3 bucket is configured as “requester-pays.” If you try to download the complete 5 TB corpus to any machine outside of the AWS us-east-1 region, you get hit with standard egress fees. At $0.09 per GB, a single full download can cost over $450 in bandwidth alone.
2. Unpacking Pain: The raw S3 data is packaged as hundreds of nested .tar archives containing gzipped payloads of individual papers. Extracting these, parsing the inner LaTeX code, and matching the files with their JSON metadata snapshots is quite CPU-intensive and requires a lot of boilerplate ingestion code.

To make this easier, I built a pipeline that runs inside AWS us-east-1 (where transfer is free), pulls the raw source files, unpacks them, matches them with the official metadata, and bundles them into ready-to-query Parquet partitions.

• HuggingFace Dataset Link: https://huggingface.co/datasets/scholarweave/arxiv

What is inside:

Each row represents a single paper and contains both the official metadata and the parsed source files:

• Core Metadata: id, title, authors, abstract, doi, categories, license, versions, etc.
• latex (Large String): The parsed, compiled LaTeX source code from the paper. I wrote a parser to bundle the primary .tex, .bib, and .sty files into a single, readable Markdown-style tree structure.

Maintenance & Syncing:

• Monthly Updates: I plan to sync the pipeline once a month to capture new uploads.
• Resilient Syncing: I maintain an XML manifest file in the HuggingFace repository (arxiv_parquet_manifest.xml) that maps each Parquet partition to its size, MD5 checksum, and the raw S3 .tar source files used to generate it. This should make incremental syncing or troubleshooting much easier.

If you are working on NLP, training LLMs on scientific text, analyzing citation networks, or doing sociolinguistic research, hopefully this saves you some time and cloud budget.

I built a clean, structured dataset of all 78 Rider-Waite tarot card meanings. Each entry has upright + reversed interpretations plus separate love / career / general context fields, so it’s usable for NLP, recommender experiments, or hobby projects.

Released open with a permanent DOI so it’s citable.

– Hugging Face: https://huggingface.co/datasets/Blacik/deckaura-tarot-card-meanings

– DOI (Zenodo): https://doi.org/10.5281/zenodo.19475329

Happy to take feedback on the schema or labeling. If anyone uses it in a project I’d love to see what you build.

This is for sale

Hey there! I am the founder of Brickroad, a frontier AI lab building agentic infrastructure for data provisioning.

Super excited to share that source streaming is now live on Brickroad. Set your search parameters once, and your agent runs continuously, notifying you the moment a new data supplier comes online.

For those who rely on data to get a performance edge, the directories, the catalogs, the curated lists of “alternative data providers” — they are useful, but they are lagging indicators of alpha. A vendor only lands in one of these catalogs after they have built a website, hired a salesperson, and shopped themselves to enough buyers that an analyst notices. By then, the first ten funds, AI labs, and corporates have already signed contracts. The information edge has dissipated into consensus.

We launched the Information Frontier Agent to compress that lag. A Source Stream is a continuous, agent-driven feed of novel data suppliers that match a thesis you define. The agent runs in the background indefinitely, scanning the complete corpus of its resources to find new suppliers that fit your criteria. Every time it finds a new supplier, the agent notifies you and logs the source into your lead table.

It’s free to trial – we’d love your feedback.

Hi , I’m looking for datasets which consist of rov images or any sort of deep sea footage to train a model.

I built a dataset joining USAspending federal contract awards to seven federal enforcement databases at the contractor level: OSHA, WHD, MSHA, EPA ECHO, NLRB, SEC, the UVA Corporate Prosecution Registry, and the SAM.gov debarment list. 5,557 contractors with documented violations, $3.19T in lifetime federal contracts, 758 OSHA-investigated fatalities.

The novel slice is the multi-agency overlap. Roughly 2000 contractors appear in 2+ federal enforcement databases. 500 in 3+. 70 in 4+. Topping the 4+ cohort by lifetime contract value: Raytheon ($68B, OSHA + WHD + NLRB + SEC + UVA), GE ($47B, same five), Merck, Microsoft, Austal USA, Marinette Marine.

Hugging Face: https://huggingface.co/datasets/FastDOLz/Federal-Contractor-Violations-Dataset

Kaggle: https://www.kaggle.com/datasets/benturneroffice365/federal-contractor-violations-dataset

Zenodo DOI (all versions): https://doi.org/10.5281/zenodo.20777627

Methodology + limitations: https://www.fastdol.com/methodology

CC-BY-4.0.

disclosure: I run FastDOL (https://www.fastdol.com), a federal workplace-enforcement search by employer, where this corpus comes from. Free for individual lookups; the dataset is one of several full extracts.

Hello Friends,

I am interested in exploring the data on the fires that have happened in NYC for different spatiotemporal analysis. I came across the following datasets from the open data platforms:

[Fire Incident Dispatch Data from NYC open data](https://data.cityofnewyork.us/Public-Safety/Fire-Incident-Dispatch-Data/8m42-w767/about\_data)

[Incidents Responded to by Fire Companies (NYFIR)](https://data.cityofnewyork.us/Public-Safety/Incidents-Responded-to-by-Fire-Companies/tm6d-hbzd/about\_data)

[NFIR](https://fema.hub.arcgis.com/search?collection=dataset&tags=nfirs)

What I noticed is that there is a lot of inconsistencies across these datasets, and the volume of the data dramatically decreases from dispatch to NYFIR an NFIR.
Please share your experiences how you guys handle this datasets for more granular analysis.

Disclosure: I’m the author of the open-source project (trove) that parses and repackages these. Original government sources are linked below; my bundles are at the end. MIT code, public-domain data, nothing paid.

Two public-domain US healthcare datasets that get cited constantly but are painful to use in raw form:

1. FDA novel drug approvals, 2021–2024 — 218 drugs (192 CDER NMEs + 26 CBER cell & gene therapies). Each row: application number, sponsor, approval date, indication, regulatory center, and a deep link to the approval-package docs.

Original sources:

– CDER Novel Drug Approvals: https://www.fda.gov/drugs/development-approval-process-drugs/novel-drug-approvals-fda

– CBER Approved Cellular and Gene Therapy Products: https://www.fda.gov/vaccines-blood-biologics/cellular-gene-therapy-products/approved-cellular-and-gene-therapy-products

– Drugs@FDA: https://www.fda.gov/drugsatfda

1. Nonprofit hospital charity-care reporting, TY2022 — 1,295 nonprofit hospital systems, with CMS HCRIS Worksheet S-10 and IRS Form 990 Schedule H side by side. Both lines are meant to capture the cost of care for patients who couldn’t pay, but the rules diverge, so the two numbers often disagree. Each row also carries a CDC Social Vulnerability Index county percentile and a deep link to the 990 on ProPublica.

Original sources:

– CMS HCRIS (Hospital 2552-10 cost reports): https://www.cms.gov/data-research/statistics-trends-and-reports/cost-reports/hospital-2552-2010-form

– IRS Form 990 series XML downloads: https://www.irs.gov/charities-non-profits/form-990-series-downloads

– CDC Social Vulnerability Index 2022: https://www.atsdr.cdc.gov/place-health/php/svi/index.html

– ProPublica Nonprofit Explorer (where the 990 deep links point): https://projects.propublica.org/nonprofits/

What I added on top: parsing the raw formats (headerless 100k-row HCRIS CSVs, IRS bulk-XML ZIPs, hundreds of FDA PDF directories) into tidy Parquet/JSON/CSV, plus a CCN↔EIN crosswalk that joins the two hospital filings.

My packaged bundles + parsers (self-promo — I built this): https://github.com/cbetz/trove — browsable lookup at https://troveproject.com

Happy to answer questions about the parsing or add fields people want!

I was really curious about the amount of product hunt launches over the years, and how AI/LLMs have affected the amount and topic of the launches. I scraped this dataset using their API.

I also built a small dashboard to visualize the trends: https://producthunt.homek8s.com/trends