Hello everyone, hope you are all doing well.I live in Brazil, where Polymarket is currently geoblocked (ironically, sports betting sites work completely fine here, go figure). I am looking to extract Polymarket data to incorporate into my predictive models. Prediction markets serve as an excellent proxy for public sentiment, such as forecasting the final outcome of a World Cup match.

I considered using a VPN, but I know Polymarket actively blocks them. Does anyone know of an alternative repository on GitHub, Hugging Face, Kaggle, or Google Cloud BigQuery that hosts historical Polymarket data (order books, transaction-level data, or market resolution history)?

Ideally, I am looking for structured formats like .csv, .parquet, or public SQL tables so I can bypass the local geo-restriction. Any leads or links to open-source data dumps would be highly appreciated.

I scraped the census data files and arranged them in a kaggle dataset. Also added a notebook for quick-start. There are attributes on demography and housing. Unfortunately the attributes are all in german, but I did not want to change the original data with half assed translations (LLMS will do a much better job in explaining what is what than I could anyways). I think this is a very neat geo dataset with interesting correlations.

Hi,

I made a new sub that’s tangentially related to this one and want to advertise it but don’t want to break any rules. Can any mods tell me the rules?

I built NorthTick after getting tired of patchy Canadian market data. It is local Parquet files, not an API: - TSX/TSXV/CSE/NEO OHLCV - Daily data back to 1993 - Minute bars from 2020 - Ticker metadata included - Free sample available Site: https://northtick.ca Disclosure: I built it. 

I built API for SEC 13F data with 57 endpoints as a site project. The data collection and the collectors are running on AWS, so the resources can be expended. I am thinking to add more data and endpoints. It is for free if someone wants to try it out.

So I was trying to figure out what Android app to build next, and the first step was doing some market research. I wanted to see what apps were already out there, so I ended up creating this dataset.

It contains app data across the top 10 fastest-growing Android categories. If you’re planning to build an Android app or just want to analyze the market, feel free to use it.

It also includes my GitHub repo, so you can customize it and scrape whatever kind of app data you want.

I want to get this DR(eye)VE dataset from AImage Lab
https://aimagelab-legacy.ing.unimore.it/imagelab/page.asp?IdPage=8

But the form on this site doesnt seem to work. So I tried contacting them through the methods in their new website
https://aimagelab.unimore.it/contacts/

But no responses to emails and even calls are stuck in a automatic response loop in Italian.
Does anyone have this dataset, or a similar one or know how I could ontain this via AImage Lab?

Any support is welcome! Thank you.

hi! So I have this dataset where the objective is to predict a student’s health risk, but I’m a lil confused about how to handle the null values. These are the % of null values for the columns:

 id 0.000000 health_condition 0.000000 sleep_duration 11.012943 heart_rate 1.135073 bmi 2.013946 calorie_expenditure 7.658878 step_count 2.016554 exercise_duration 1.000017 water_intake 6.300211 diet_type 1.000017 stress_level 12.000064 sleep_quality 8.452690 physical_activity_level 5.306715 smoking_alcohol 4.141791 gender 3.097141 dtype: float64id 

What would you recommend I do for these values? If I were to drop the columns <5%, I would be losing nearly 100,000 values (out of 700,000) which I don’t think is all that good. I thought of using K-means to fill the null BMI values but I don’t know.

I would appreciate any advice! Thanks 🙂

Hello everyone, I’m working on my university thesis and I was wondering if any of you would be willing to share access to datasets pertaining to the movie industry, specifically datasets containing Yearly box office revenue or amount of tickets sold for specific movie genres, more precisely for the horror genre. I started some searches already but the promising leads I found were pretty expensive.

I’m trying to find a database of surnames for use in writing/testing code that converts an author name (e.g, “Stan Sieler”) into a sortable/alphabetizable name (e.g, “Sieler, Stan”).

Many surnames are compound (“de Camp”, “Cartwright-Chickering” (bonus for people who recognize that one!), some with and some without hypens, and some with more than two words.

The U.S. Census database isn’t useful to me … they compress all last names, removing spaces.

(I’m ignoring people like “Arthur Conan Doyle”, whose last name at birth was “Doyle”, but later adopted the practice of using “Conan Doyle” as his surname … confusing librarians around the world 🙂

Any pointers appreciated, thanks!

The PSID has tracked the same American families since 1968 across income,

health, housing, wealth, education, and depression. It’s one of the most

powerful public datasets in social science, but the raw files arrive with no

meaningful column names and require a codebook crosswalk just to understand

what you have.

PSID-SHELF (from U-Michigan) reorganized the data into 34 topic areas with

real variable names. There’s now a browser app built on top of it — search

across all 34 topics in plain English and see sample data immediately. No

download, no account, no setup. Link in comments.

There’s also a local track that produces 34 clean CSVs from your own SHELF

download, ready for pandas, R, or Excel.

Happy to answer questions.

I’m currently working on a cosmetics/skincare startup and one thing I’ve been struggling with is finding reliable market data. Whenever I need information like market size, growth rates, consumer trends, pricing, competitor analysis, retailer performance, ingredient trends, or industry forecasts, I end up finding reports that cost anywhere from hundreds to thousands of dollars.

For those of you who regularly work with market research or data:

Which platforms do you actually use?
Which ones are worth paying for?
Are there any hidden gems that professionals use but aren’t widely known?
How do startups without huge research budgets access high-quality data?
Do you combine multiple sources (government data, retail data, consumer surveys, Google Trends, etc.) instead of relying on one platform?
I’m particularly interested in the beauty, cosmetics, skincare, and consumer products industries, but I’m also curious about general-purpose research platforms.

I’d love to hear what professionals, analysts, consultants, or founders use in their day-to-day work.

Was poking at the SF market and put together a clean dataset of homes + condos currently listed: list price, price/sqft, sqft, beds/baths, year built, lot size, agent, and the Redfin link for each.
A few things that jumped out:
– Median list price is ~$1.27M, median $980/sqft
– Cheapest thing on the market: a $369k 523-sqft condo at 601 Van Ness
– Priciest: a $6.6M unit at 188 Minna — which works out to $3,256/sqft lol
– Year built ranges from 1884 to 2021, which is very SF

CSV/XLSX here if anyone wants to take a look at it: https://docs.google.com/spreadsheets/d/17BhnTFkWtN6cI9Yn9f0BgPcLF6sVEk9T/edit?usp=sharing&ouid=108885207033845537587&rtpof=true&sd=true

Made it with an open-source tool called Bigset where you basically describe the dataset in a sentence and it goes and pulls + verifies the data from the live web.

Happy to pull a different slice if people want -by neighborhood, condos only, under $1M, whatever.

Hello folks i need an advice from DBAs.
I’m building a gentle data extractor from dabases.

What’s the most important metric that can confirm that ongoing data extraction is not harmful for database?

Started out with a question, how do I spend $300 in free GCC credits, and how much could I do with it. I started with figuring out how to query HTTP Archives, pulling CRuX data to correlate sites, and learning a bit about BigQuery along the way. I went from ~12 million total sites and pared that down to 1.5 million that I could verify were live, had enough data to be able to classify/categorize, and then built a front end to access the highlights.

So far, I’ve been focused on identifying key business segments with missing opportunities, classic one click misses, some schema mapping for business type, and wondering why in the world any sane business owner would use Weebly.

What would YOU want to know?

Sharing a dataset resource that may be useful for researchers, data scientists, and investigators.

Image-Meta has indexed the embedded metadata (EXIF/IPTC/XMP) from ~720 million publicly accessible images using ExifTool. The data is queryable via a REST API & web rather than a bulk download.

**What’s in the dataset:**

– Camera make, model, and serial number

– Author, copyright, rights, title, description

– GPS coordinates (where present, subject to strict TOS/paid tier not publicly free available)

– Software chain

– Creation, modification, and index dates

– Filename and document ID

– Creation, Modify Date, Date Found

– Extra JSON supplimental metadata in full per image

**Potential research uses:**

– Camera device attribution studies

– Metadata privacy/leakage research

– Image provenance and disinformation analysis

– Geospatial studies using embedded GPS

– Timeline reconstruction of image publication

**Access:**

Web or Queryable via REST API with field-level boolean search, date ranges,

https://image-meta.com

API docs: https://image-meta.com/api-docs

I just released a free sample dataset of 50 traditional recipes from 25 countries.
Each recipe includes:
Ingredients
Step-by-step instructions
Prep time & cook time
Serving size
Format: JSON
The full dataset contains 1,925 recipes from 194 countries and is available on HuggingFace under the name:
“FoodieAtlas World Traditional Recipes Dataset”
Disclosure: I am the creator of this dataset.

Please someone provide me with this, there is no risk for you with this so no need to worry about safety.

Finding a clean, structured list of financial instruments has always been harder than it should be. Bloomberg sells it. Refinitiv sells it. Yahoo Finance gives you a search bar. If you want “all biotech companies listed in Germany” or “all fixed income ETFs from Vanguard” as a filterable dataset, you’re usually either scraping something or paying for a data vendor. I’ve spent the last few years building and maintaining a free alternative.

The Finance Database covers seven asset classes across 300,000+ symbols:

Each equity record includes: symbol, name, currency, sector, industry group, industry, exchange, market, country, city, market cap tier, ISIN, CUSIP, FIGI, composite FIGI, share class FIGI, and website. ETFs and funds carry family, category group, and category instead of GICS-style classification. Every record has what you need to cross-reference against other data sources.

The data is an aggregation of publicly available sources – no paid API required to use the database itself. It is community-maintained, MIT-licensed, and lives on GitHub as CSV files you can open in Excel if that’s your preference.

The Python package gives you structured filtering and text search:

“`python

Install via: pip install financedatabase -U

import financedatabase as fd

equities = fd.Equities()

All semiconductor companies in Taiwan on primary listings only

equities.select( country=’Taiwan’, industry=’Semiconductors’, only_primary_listing=True )

Free-text search: robotics or automation companies on the Frankfurt exchange

equities.search( summary=[‘Robotics’, ‘Automation’], index=’.F’ )

Explore what’s available before filtering

fd.show_options(‘equities’) “`

The show_options call is useful before you filter – it returns every distinct value per column without loading the full dataset, so you can scope your query without memory overhead.

For anyone doing universe construction for backtests or systematic strategies, the ISIN/FIGI coverage is the most practical part. You can pull a filtered symbol list here and pipe it directly into your price data provider.

The database is not a price or fundamentals source – that’s intentional. Metadata and categorization data is the hard part to get for free and I’ve built a seperate tool for that, the Finance Toolkit.

GitHub page: https://github.com/JerBouma/FinanceDatabase

Most of the work isn’t building charts.

It’s preparing the data before the analysis even starts.

Cleaning files.
Fixing formats.
Validating data.
Checking structures.
Transforming datasets.

The better your preparation process is, the easier the actual analysis becomes.

What part of data preparation do you find most annoying?

I’ve spent the last few weeks working on a pipeline to solve a problem that has frustrated me (and likely other researchers) for a while: working with arXiv source files at scale.

If you have ever tried to analyze the LaTeX source code of arXiv papers, you have probably run into two major roadblocks:

1. The Egress Tax: arXiv’s official bulk S3 bucket is configured as “requester-pays.” If you try to download the complete 5 TB corpus to any machine outside of the AWS us-east-1 region, you get hit with standard egress fees. At $0.09 per GB, a single full download can cost over $450 in bandwidth alone.
2. Unpacking Pain: The raw S3 data is packaged as hundreds of nested .tar archives containing gzipped payloads of individual papers. Extracting these, parsing the inner LaTeX code, and matching the files with their JSON metadata snapshots is quite CPU-intensive and requires a lot of boilerplate ingestion code.

To make this easier, I built a pipeline that runs inside AWS us-east-1 (where transfer is free), pulls the raw source files, unpacks them, matches them with the official metadata, and bundles them into ready-to-query Parquet partitions.

• HuggingFace Dataset Link: https://huggingface.co/datasets/scholarweave/arxiv

What is inside:

Each row represents a single paper and contains both the official metadata and the parsed source files:

• Core Metadata: id, title, authors, abstract, doi, categories, license, versions, etc.
• latex (Large String): The parsed, compiled LaTeX source code from the paper. I wrote a parser to bundle the primary .tex, .bib, and .sty files into a single, readable Markdown-style tree structure.

Maintenance & Syncing:

• Monthly Updates: I plan to sync the pipeline once a month to capture new uploads.
• Resilient Syncing: I maintain an XML manifest file in the HuggingFace repository (arxiv_parquet_manifest.xml) that maps each Parquet partition to its size, MD5 checksum, and the raw S3 .tar source files used to generate it. This should make incremental syncing or troubleshooting much easier.

If you are working on NLP, training LLMs on scientific text, analyzing citation networks, or doing sociolinguistic research, hopefully this saves you some time and cloud budget.

I built a clean, structured dataset of all 78 Rider-Waite tarot card meanings. Each entry has upright + reversed interpretations plus separate love / career / general context fields, so it’s usable for NLP, recommender experiments, or hobby projects.

Released open with a permanent DOI so it’s citable.

– Hugging Face: https://huggingface.co/datasets/Blacik/deckaura-tarot-card-meanings

– DOI (Zenodo): https://doi.org/10.5281/zenodo.19475329

Happy to take feedback on the schema or labeling. If anyone uses it in a project I’d love to see what you build.