I have a little personal project that I’d like to build and I see there are a number of APIs available around the Internet (RapidAPI, apify, etc.)

Is there a known best-in-class API that provides flight information/pricing from most airlines, can discriminate by coach/business, and offer information on hotel availability and pricing too?

A while ago I tried an API from RapidAPI, but quickly discovered that it wasn’t bringing in a lot of stuff from lesser-known airlines (Copa, smaller Euro carriers, etc). I’d like to build this on top of something solid, but that doesn’t require me to buy millions of calls a month since this is a personal project.

Hello all, I’m looking for recommendations to use IP addresses (source and destination) in my Random Forest classification model.

When images are run through social media platforms, they are resized, re-encoded, and pushed through the platform’s codec. In assessing a deepfake detector model, it’s important to ensure the model remains robust across real world platforms.

I built a dataset of varied image formats that mimic the image adjustments made by these popular platforms and tested some open source models on it.

Dataset, Model Results

Hi everyone
We are doing some very interesting data capture work and turn it into 4D. We are based in New York City Brooklyn.

We need some help with volunteers to capture themselves.

If anyone interested please don’t hesitate to reach out to me
😀

I am looking for Airbnb data for research on short-term rental markets. I am especially interested in listings and listing-level data, ideally covering several years so I can analyze changes over time. I am looking for information such as price, location, size, number of guests, minimum stay / length of stay, and other basic listing characteristics.
The geographic scope I am interested in includes tourist coastal cities in Poland, such as Gdańsk, Sopot, and Kołobrzeg, as well as selected cities abroad, such as Dubrovnik, Split, and Rijeka.
The Inside Airbnb website primarily features data for the US. It doesn’t list any Polish cities.

If anyone has access to such data, knows where it can be obtained, or has worked with similar datasets before, I would be very grateful for any contact, advice, or suggestions.

I’ve been looking into first-person (egocentric) video datasets for activity recognition and multimodal learning research.

A few challenges that seem to come up repeatedly are:

Motion blur
Rapid viewpoint changes
Occlusions from hands and objects
Long video sequences
Annotation consistency

For people who have worked with these datasets:

Which datasets have been the most useful?
What limitations did you encounter?
How well do current datasets generalize to real-world applications?
Are there any newer datasets you’d recommend exploring?

I’d appreciate hearing about experiences from both research and production environments.

Sharing a dataset I work on. REFUTE is an Apache-2.0 benchmark for testing whether models can critique recent science summaries with calibrated, evidence-grounded judgment.

Configs: – refute_soundness — judge-free split (no LLM judge needed to score) – refute_hard_60 / refute_120 — harder vignettes

Each item: a paper summary (some with planted flaws / overclaims / missing evidence) + gold labels, with confidence targets scored using Brier (a strictly proper rule), so calibration is measured rather than just accuracy.

License: Apache-2.0 Load: load_dataset(“BGPT-OFFICIAL/refute”, “refute_soundness”) Dataset: https://huggingface.co/datasets/BGPT-OFFICIAL/refute Leaderboard: https://huggingface.co/spaces/BGPT-OFFICIAL/refute-leaderboard

Happy to answer questions about how it was constructed and labeled.

https://mcinerney.ai/writings/i-open-sourced-my-ufc-prediction-model-weights-and-database/ – lessons on using the data for modeling over the last several years.

Here’s my gigantic database of UFC stats including hour by hour odds data of fights over the past 5 years. There’s an AGENTS.md/CLAUDE.md/README.md that are optimized for CC or codex to analyze the data.

Each row has:
→ equity / investment terms
→ program dates and location
→ focus area
→ notable alumni

Built with BigSet (Open-Source Dataset Builder, Powered by TinyFish)

Thank me later 🙂

Hi everyone,

I work at PredictLeads, where we collect and maintain company datasets focused on business signals.

Our Jobs Dataset currently includes:

• 271.3 million job openings detected since 2018
• 8.9 million active job openings with job descriptions available
• Historical hiring activity and trends
• Company-level hiring signals
• API and bulk data access

Documentation:

https://docs.predictleads.com/api_endpoints/job_openings_dataset

In addition to jobs data, we also provide datasets covering:

• Technologies
• News Events
• Funding Events
• Company Data
• Website Changes
• GitHub Activity
• And more

One thing that makes us a bit different is that we don’t focus on building a platform. We’re a data provider focused primarily on data quality, coverage, and making the data easy to integrate into your existing workflows, data warehouses, CRMs, or enrichment pipelines.

Happy to answer any questions about coverage, use cases, APIs, or data delivery formats.

PyPI: https://pypi.org/project/crimedatasets/
GitHub: https://github.com/lightbluetitan/crimedatasets-py
Docs: https://lightbluetitan.github.io/crimedatasets-py/
pip install crimedatasets
The crimedatasets package provides a comprehensive collection of crime-related datasets from around the world. It includes extensive data on topics such as mass shootings, hate crimes, incarceration statistics, serial killers, corruption indexes, law enforcement data, criminal justice metrics, drug overdoses, and prison facilities.

reposting this here. But I’ve built out a crawler that obtains live job listings across 5.6 million US company websites, and continuously updates a monthly pool of job listing data.

I’ve seen other people doing this on reddit but refusing to be transparent and actually share their datasets for download.

My airflow dags complete a full crawling cycle of all companies and their associated job boards in under 24 hours. This is on a windows machine and modest home network so my operating costs are near zero.

This data will remain forever free @ jobdatapool.com

I’ve been working with a cross-national dataset where all the variable labels and value labels are in a foreign language. Renaming them manually is tedious and error-prone, especially with 200+ variables.

I know I can write a do-file to relabel everything but that still requires me to know what the foreign labels mean and manually enter English equivalents one by one.

Is there any tool or workflow that handles this automatically? Ideally something that takes the .dta file, translates the metadata, and returns a clean English-labeled file without touching the underlying data

Been working on a gap I kept running into: getting official,

daily FX rates for West African countries programmatically.

The World Bank has this data but with a 6-12 month lag.

Everything else is either paywalled or scraped from aggregators

with no attribution.

So I built an actor that pulls directly from the issuing

central banks — CBN Nigeria, Bank of Ghana, BCEAO for the 8

WAEMU nations, and Banco de Cabo Verde. 11 countries, 4

currencies, history back to 1996 in some cases.

A few things I found interesting while building it:

The 8 WAEMU countries (Côte d’Ivoire, Senegal, Mali etc.)

share a currency pegged to the euro by treaty since 1999 —

at exactly 655.957 XOF/EUR, never changed. There’s no

independently set USD rate, it’s mathematically derived from

the ECB daily reference rate.

Every output record carries the source bank, URL, retrieval

timestamp and licence note — CBN explicitly grants permission

to copy with attribution which made things cleaner legally.

Available here if useful: https://apify.com/malmon/west-africa-fx-rates

Happy to answer questions about coverage or methodology.

Hey everyone,

I’m working on a business/company dataset and I’d really appreciate honest feedback from people who care about datasets, data quality, structure, and usefulness.

Just to be clear, this is not meant to be an ad. I’m not trying to sell anything here. I’m genuinely looking for advice on whether the data is useful, what’s missing, and what would make it more valuable as a dataset.

The idea is to build a structured dataset of business profiles over time. Right now, each company profile can include things like:

• company name
• website
• industry
• sector
• location/headquarters
• short description
• related business details where available
• confidence indicators
• sources/references where possible

The longer-term plan is for the dataset to improve and grow as more businesses are searched and evaluated. But before I keep building in that direction, I’d really like people to look at what it currently returns and tell me whether it’s actually useful from a data perspective.

There’s a free live search page here where you can test the current output:

https://fastbusinessapi.com/trial-search/

I’d really appreciate feedback on things like:

• whether the fields are useful
• whether the structure makes sense
• what fields are missing
• whether the data feels trustworthy
• what would make this more useful as a dataset
• what would make you not use or trust it
• whether this type of dataset has value if it grows over time

Again, this is genuinely not intended as advertising. I’m asking because I want honest feedback from people who understand datasets before I spend more time building the wrong thing.

Any criticism, advice, or suggestions would be really appreciated.

The usdatasets package provides a comprehensive collection of datasets focused on the United States. It includes extensive data on topics such as crime and public safety, political history, economic indicators, education, public health, natural disasters, demographics, infrastructure, sports, and cultural events.
https://pypi.org/project/usdatasets/
https://lightbluetitan.github.io/usdatasets-py/

I can’t find any articles talking about overall human populations. I’ve just had this question while researching about ancient human life, natural selection, genetics, stuff like that. Do most people reproduce? Is it more 50/50? Ik our population is increasing still, but people are also living longer. From a childfree perspective, it seems that like 80% of the population has kids, but I’m probably not very accurate there lol.

I just launched a specialized vocal pack built specifically for indie game devs, gamified UIs, fitness apps, and conversational AI tools. The links below are to the [10-word] sample pack, which is available for download now! The complete pack includes 100 single-word vocal tokens such as Success, Level, Win, Combo, Wow, and Boost.

Specs:

• Studio-Grade Audio: This audio is completely dry and background-reverb-free.
• Pro Calibration: Standardized to -23 LUFS with a strict -1.0 dB True Peak ceiling with zero clipping or distortion.
• Pipeline Ready: It includes a fully aligned mapping file for immediate ingestion.

If you would like to test the vocal quality in your project, check out the evaluation samples here:

• Hugging Face: https://huggingface.co/datasets/MarieDeVox/high-energy-ui-vocal-dataset
• GitHub: https://github.com/MarieDeVox/high-energy-ui-vocal-dataset

I will be releasing a few more of these micro vocal packs, including a bundle item! Let me know if you check it out or if you would like something for your personal task!

Hi there,

I wanted to share a multilingual version of OpenAI’s HealthBench dataset. It’s currently available in 32 languages, spoken by 5+ billion people.

Languages:

Amharic, Arabic, Bengali, Brazilian Portuguese, Chinese, Dutch, Estonian, Finnish, French, German, Hausa, Hindi, Indonesian, Italian, Japanese, Korean, Latvian, Lithuanian, Malay, Norwegian, Persian, Polish, Russian, Somali, Spanish, Swahili, Swedish, Thai, Turkish, Ukrainian, Urdu, and Vietnamese.

Dataset link: https://huggingface.co/datasets/projetogabi/healthbench-multilingual

Cheers

Free account, Python SDK.

https://mozilladatacollective.com/

Hey guys, I’m working on a project and trying to figure out what data sources I’m still missing.

Still looking for good sources for:
State and local contract awards (DOTs, municipalities, utilities, etc.)
Utility interconnection queues (ERCOT, PJM, MISO, CAISO, SPP)
Data center / semiconductor / battery plant / LNG project tracking
Construction wage data by metro
Trade workforce retirement/aging data

Any suggestions or ideas?

Disclosure: I’m the creator of the website.

I have always considered this type of data useful, but I was never satisfied with the available alternatives, mainly for three reasons: 1) lack of transparency regarding the source, or the use of crowdsourced data; 2) missing, incomplete or unclear methodology; and 3) comparisons between data that are not always truly comparable.

That is why I decided to create this website: citycostatlas.com

All data has its source indicated — most of it comes from public institutions — and the methodology used to obtain the data is explained clearly. I try to ensure that the data being compared is actually comparable; when it is not fully comparable, this is indicated — for example, when comparing the sale price per m² of a house in the City of Helsinki with the value for the Greater City Area of Madrid, because they do not represent the same geographical/statistical area.

In this first version, I chose to include the capital cities of the European Union and some key costs: sale price per m² of apartments and houses, monthly rents for different dwelling types, household gas consumption between 20 GJ and 199 GJ, household electricity consumption between 2,500 kWh and 4,999 kWh, and water based on an annual consumption of 120 m³. The gas and electricity bands were chosen because they are intermediate, standardised household consumption categories used for comparison between countries. For water, I used 120 m³/year as a practical benchmark to make tariffs with different structures more comparable.

Suggestions, additional information or any errors you notice are welcome. Please contact: [migralept@gmail.com](mailto:migralept@gmail.com)

hi all,

i’m having trouble finding a dataset to download that has minimum wage data by US state, along with the federal minimum wage and real vs nominal numbers. I found one that goes up to 2020, but i’m looking to go to 2024. i’ve been looking around on github and google but can’t find anything yet, and i don’t know how to scrape the table off the DOL website. can anyone please help me out? thanks

Over the past few months, I’ve been working on a high-scale scraping pipeline to aggregate listings directly from company job boards and applicant tracking systems. Mapping over 100,000 distinct companies to their career pages turned out to be a massive engineering headache, but it’s finally stable.

The result is a unified database of more than 2 million active job postings, which I’m opening up to everyone for free. I am running daily delta refreshes to keep it current.

Dataset Overview

• Scale: 2M+ active job listings across 100,000+ unique companies.
• Format: Parquet. (To keep storage costs to minimum)
• Core Fields: job_title, company_name, company_website, job_description, location, post_date, and the original tracking URL. For more detailed info check here.
• Update Cadence: Refreshed daily straight from the source.

Why I Built This

Finding a clean, scaled, and up-to-date job dataset is surprisingly difficult. Most available options are either heavily gatekept by expensive subscription APIs or restricted to a single job board like LinkedIn. By scraping the actual employer sites directly, this collection sidesteps the noise and captures a much cleaner cross-section of the live market.

How to Access It

I set up a dedicated project space where you can grab the data directly: Open Job data

Let me know what kind of analysis or projects you end up running with it. If you have questions about the engineering architecture behind handling this scale, or ideas for specific fields you’d like to see enriched next, let’s discuss in the comments.