Hi r/datasets,

I’ve uploaded a free sample pack containing 100K rows split across 3 different fully synthetic, high-quality datasets tailored for ML research, data analysis, and pipeline testing:

1. Credit Risk (Improved): Synthesized credit bureau features, debt-to-income ratios, and loan default targets.
2. Finance Transactions: Mock banking/transaction histories with categorical merchant types and amounts.
3. Ecommerce Customer Behavior: Session duration, cart adds, purchase history, and user drop-off flows.

Download Link:
You can access the dataset files directly on Hugging Face:
👉 Synthetic Datasets Free Samples (100k)

Note: Since these are synthetic, they contain no real personally identifiable information (PII) or real-world entities.

Any feedback or requests for specific domains you’d like to see next would be highly appreciated! Enjoy!

USDA’s Local Food Portal is the canonical US farmers-market dataset, but the raw feed is rough: truncated names, stale records, a state= filter that substring-matches state names (querying “WA” returns Delaware rows), and thousands of missing websites/hours.

I cleaned and enriched it: deduplicated to 8,863 real markets (record-level), backfilled website coverage to ~50% from the live API + state sources, and added season/SNAP/organic fields. It’s CC-BY 4.0 as CSV/JSON.

Archived with DOI: https://doi.org/10.5281/zenodo.21360372

Disclosure: I run harvestlymarkets.com, the directory built on this data — full methodology and the same downloads live at https://harvestlymarkets.com/data-sources/. Personal contact names/emails are stripped from the redistributable; business fields come from USDA’s public feed.

Happy to answer questions about the data-quality issues — the state-filter substring bug was a fun one to find.

The core idea: take a real document (SEC filings, contracts, enterprise email), verify by exhaustive normalized scan that a specific plausible fact is not in it, then ask about that fact. The honest answer is “not in the document.” We ran six frontier models on these with zero abstention coaching and they asserted made up answers 11% to 44% of the time. The full per model table is on the dataset card with raw logs and API errors disclosed.

What’s in it: 2,889 certified absence rows, 3,088 span verified extractive QA rows, a 127K token packed long context task set, and a split minted only from SEC filings dated after every major model’s training cutoff. That fresh split regenerates monthly, so it stays impossible to have trained on, by construction.

Every row carries a certificate you can re-check yourself in a few lines of python, the audit snippet is on the card. When our own audits flag something, like extractive answers that are guessable from world knowledge (about 1.6% of them), we label it instead of quietly deleting it.

Also worth knowing before you trust us: a reviewer caught one of our splits being weaker than claimed this week. We re-audited every row the same night, withdrew the split with per row evidence committed to the repo, tightened the protocol, and reshipped only the rows that survive everything. The full trail is in the audits folder, judge for yourself.

License CC BY 4.0. Generation was an Apache 2.0 open weight model on our own hardware, the claim is the verification layer, not the generation. Held out versions never get published so they can’t leak into training data. If anyone wants a sealed diagnostic run against their own model or domain (25 items, free, about a day), contact is on the card.

https://huggingface.co/datasets/SovNodeAI/certified-document-qa

Hi r/MachineLearning,

I’m building a collection of high-quality synthetic datasets for different ML use cases.

Currently have samples ready for:
– Finance / Credit Risk
– Transactions & Fraud
– Ecommerce Customer Behavior

**100K row free samples** are available for anyone who wants to test them.

Question for the community:
What domain or specific problem would you like to see a synthetic dataset for? (Healthcare, Marketing, Insurance, Time-series, etc.)

Comment below with your use case or interests, and I’ll prioritize accordingly + send you the current free samples if you want.

Looking forward to your ideas!

Does anyone have an archived copy of the bulk UK Energy Performance Certificate (EPC) dataset from the old government Open Data Communities portal (epc.opendatacommunities.org)?

Context: On 30 May 2026, the UK government migrated the service to a new platform. In doing so, they completely dropped all pre-2012 certificates from the public register due to them being “expired” and based on older methodologies.

Why I need it: I’m doing property energy analysis, and removing that 2008-2011 dataset creates data gaps for properties that haven’t been sold or rented in the last 14 years.

The original ZIP file was about 5.6GB. I am looking for a snapshot from late 2025 or early 2026 before the site was taken offline.

If anyone has a magnet link, a torrent, or is willing to share a cloud drive link to the original CSVs or a Parquet equivalent, I’d really appreciate it.

I wanted to share a project I’ve been working on called EBBC OpenData, which is a public API and dataset designed to promote Open Science and support bibliometric, scientometric, and informetric analyses. You can find the full project and source code in the repository at https://github.com/GabrielBaiano/EBBC-OpenData

This project provides structured metadata from the publications of the Encontro Brasileiro de Bibliometria e Cientometria (EBBC), which is one of the main events on metric studies of information in Brazil. Through this API and dataset, you can easily query detailed information about authors and their academic networks, articles and papers (including titles, abstracts, and publication years), institutions associated with the research, keywords, thematic trends, as well as references and citations.

The core metadata and documentation are currently being organized, and I am actively working on translating the API documentation and dataset fields into English and Spanish to make the project fully accessible to the global research community.

Since this is an ongoing project, I would highly appreciate your thoughts and feedback. I am especially interested in knowing what features or endpoints would make this more useful for your research, any suggestions you might have regarding the data structure or documentation, and any general tips on best practices for open-data APIs. Please feel free to check out the GitHub repository, open an issue, or leave a comment below. Thanks for your support!

I’ve been deep-fine-tuning small local models (BitNet 1.58, Qwen 1.7B/4B, Gemma-4) into a real on-device assistant with actual tool-calling. The hard part was never the training loop, it’s the data. Scraped

ShareGPT dumps collapse small models into formulaic slop and teach tool syntax that doesn’t match your runtime. And for full/deep fine-tuning (not LoRA), bad data is fatal: a small model trained on redundant, single-voice data monocultures hard.

So I built a pipeline where every example has to survive a set of hard gates before it’s allowed in.

Sharing how it works, because I rarely see people talk about the controls, only the volume.

The core: hand-written “gold seed” → multi-teacher expansion

– The seed is hand-written, one example at a time, in a neutral model-agnostic format ({messages, tools}).

– It’s rendered per dialect: ChatML for Qwen/BitNet, native Gemma tool-call format for Gemma. Same data, correct syntax for each target.

– From a curated seed it expands to hundreds of thousands of examples on demand — volume comes from multiple teacher models of different families (anti style-collapse), each example tagged with the teacher that produced it. You scale to the size you actually need.

The gates (this is the value)

– Anti-formula: blocks over-used openers/closers at ingest + global frequency caps; any phrase repeated too often anywhere gets flagged. This is what stops a full fine-tune from collapsing into one voice.

– Semantic dedup (BGE-M3): near-duplicate detection, not byte-match. On one 9k combinatorial corpus it found 43% near-duplicates, paraphrases expansion would’ve amplified. Keeps 1 per cluster, with a coverage guard that never drops the sole example teaching a capability.

– Flow gate: multi-turn tool-call integrity: every assistant tool_call is followed by exactly its results, no orphans, no pending calls, ends with a real answer.

– Dialect gate: every tool-call is round-tripped through each target model’s syntax and rejected if it doesn’t parse back identical. train == runtime, guaranteed.

– Coverage: every tool taught above threshold; catalog-sync rejects phantom (hallucinated) and untaught tools.

– Vision routing: vision examples go only to vision-capable models; text-only models never see image context they can’t use at inference.

– Judge routing — teacher outputs that pass go to SFT; the ones that fail become KTO negatives (preference signal, not trash).

Built for deep fine-tuning, at scale

The whole point of the gates is that you can full-fine-tune a small model on this without it collapsing and expand a small hand-verified seed to 100k+ examples while keeping every one of those checks green.

Tool-calling, multi-turn, charts/tables/HTML, per-model vision, reasoning, all gate-verified, neutral format so you render to your own dialect.

What it trains

Liara — a local personal AI with 24 real tools (email, calendar, files, notes, web, weather, charts), english and italian-first + multilingual, that resists prompt-injection while still handling legitimate reasoning-test / structured-output prompts (the distinction most classifiers get wrong).

– Liara app: https://nothumanallowed.com/local

– Tooling / code: https://github.com/adoslabsproject-gif/Liara-toolkit

Hello!
I’ve spent a couple of months polishing a series of tools designed to analyze Reddit data to create statistics on various topics. A lot of examples on my profile but I’ll put a couple here:
1- Tracking the best method outreach method for freelancers accross reddit discussions
2- Tracking changing sentiment over time on different AI Models
3- Summarizing Reddit opinions on game engines from discussions on game dev subreddits

And much more, you can see more examples on my Reddit profile (some were better received than others).

Here’s what I offer:
– Filtering for niche topics on a list of subreddits: You pick a list of subreddits and select a goal for this study, I’ll arrange NLP and LLMs models to filter for your niche topic out of the thousands of irrelevant posts.
– Extracting json data from the relevant posts and comments, the specific schema for extraction is customizable; You may extract sentiment, entities (dollar amount, days,names, etc..), categories of comparison and much more.
– Cleaning the data into analysis ready data-sets with hierarchical topic modeling for categories if desired
– I can also perform simple analysis on the data for you and provide clean looking charts (made with flourish)

Price : Just the compute, around 10$ for 40-30k rows (depends on the complexity of the topic)

If you or someone you know might be interested don’t hesitate to reach out!

i’m looking for an infrastructure to ingest an entire 600,000+ archive if the 45k samples look good to them.
We have just finished staging over 45,000 fashion, portrait, and lifestyle images on AWS S3, and they are ready for immediate review.

A quick overview of our datasets:

Clear Ownership: We captured and own this entire collection (2002–2026), meaning you get an unbroken chain of title and fully signed commercial releases.

Privacy Options: We offer both the original unedited files and anonymized batches where faces have been neutralized to simplify your compliance.

Full S3 Availability: While the links below cover our 45k sample sets, our complete 600,000+ RAW image archive is already fully staged in our secure S3 buckets and ready for immediate, direct transfer.

Final year CS student here, targeting data science and analytics roles for campus placements.

Been struggling with this question while building my portfolio: does it matter whether your project uses real messy data vs synthetic/clean data?

Real datasets from Kaggle feel either too cleaned already or the same recycled projects everyone does. But synthetic data feels hollow because the hard part — cleaning, feature engineering, deriving meaningful columns from raw data — is already done for you. You’re basically just visualizing something someone else already solved.

Specifically for BI/dashboard projects — if you use synthetic data, the dashboard looks clean and professional but there’s no real discovery or insight because the data was designed to be dashboarded. Nothing surprising comes out of it.

Also practically — if an interviewer asks “where did you get this dataset?” what’s the right answer? Saying “I generated it synthetically” feels like admitting you took the easy route. But lying about the source is obviously wrong. Is there a way to frame synthetic data usage that doesn’t sound like you avoided the hard part?

At the same time I’ve heard people say interviewers care more about what you built on top of the data than where it came from. But isn’t handling bad data literally the core skill in DS?

For people who’ve interviewed at analytics/DS companies or done hiring — how much does data source actually matter? Is a well-executed project on synthetic data better than a mediocre project on real messy data? Or does using synthetic data automatically signal you avoided the hard part?

Please tell me how to join it, guide me.

Hi r/datasets,

I built Energtx, a global energy data platform that standardizes publicly available energy, electricity, emissions, and climate-related datasets.

Current coverage includes:

• 235,000+ structured records

• 170+ indicators

• 106 countries

• Historical data from 1960 to 2025

• CSV, JSON, and XLSX downloads

• Filtering by country, indicator, source, and year

The platform includes data on electricity generation, renewable energy, primary energy consumption, CO₂ emissions, electricity access, nuclear power, carbon pricing, and oil, gas, and coal markets.

The underlying data is compiled from public sources such as the World Bank, Ember, Our World in Data, the Energy Institute, EIA, OECD, IAEA, and Climate TRACE. Source attribution is included with the datasets.

Dataset explorer:

https://energtx.com/datasets

Disclosure: I built and maintain Energtx. The platform is free to browse and does not require registration.

Feedback on the dataset structure, metadata, indicators, and download formats would be useful.

r/ramen
Title: I mapped 62,173 ramen shops in Japan into a free English/romaji API

For the minute by minute bars data, columns are:

“symbol”, “timestamp”, “open”, “high”, “low”, “close”, “volume”, “vwap”, “trade_count”, “spy_close”, “iv”, “delta”, “gamma”, “theta”, “vega”, “rho”

For tick_by_tick (all individual trades executed) columns are:

“symbol”, “timestamp”, “price”, “size”, “exchange”, “conditions”, “spy_close”, “iv”, “delta”, “gamma”, “theta”, “vega”, “rho”

It goes back a few years, depending on the ticker.

Hi everyone,

I’ve been working on a data-centric pipeline for constructing SFT and KTO datasets for small language models, targeting models ranging from a 1.58B ternary model up to 12B parameters (with a particular focus on the 1.5B–4B range), using an Italian tool-calling assistant (“Liara”) as a case study.

Instead of focusing on model architecture, the goal is to reduce common failure modes through dataset construction itself:

• tool over-calling
• style collapse
• excessive verbosity
• semantic redundancy
• memory inconsistencies

The pipeline currently includes:

• typed validation outcomes (PASS / Soft Reject / Hard Reject / Warning)
• semantic + structural deduplication
• multi-teacher generation
• dataset lineage and versioning
• regression set
• dataset health dashboard
• capability-based dataset profiling for different model sizes
• typed routing into SFT, KTO-negative, or discard
• Soft Reject examples are not discarded by default: they undergo additional validation and, if confirmed, are reused as KTO-negative examples rather than being treated as unusable data.

The current specification describes the methodology. The implementation is underway, and the experimental validation is currently running.

I’d love feedback from people who have built or maintained instruction datasets:

• Which parts seem genuinely useful?
• Which ideas already exist in other pipelines?
• What ablation studies would you expect before considering this publishable?

I’m currently generating the gold seed dataset, which is the most time-consuming part of the pipeline and is expected to take around 10 days at the planned scale. Once that’s complete, I’ll publish the implementation, the ablation results, and the evaluation so the methodology can be assessed based on experimental evidence rather than design alone.

In the meantime, I’d really appreciate any feedback or suggestions on the pipeline itself.

A lot of IPO research starts as manual filing review: S-1s, F-1s, amendments, 424B4s, effectiveness notices, ticker changes, exchange hints, and post-listing performance. That works for one company. It breaks down if you want to build watchlists, backtests, screens, or systematic models.

The IPOGrid API is meant to make that workflow structured.

Docs:

https://ipogrid.com/docs

API reference:

https://ipogrid.com/api/v1/docs

OpenAPI:

https://ipogrid.com/api/v1/openapi.json

What you can pull

IPOGrid exposes structured company, filing, news, chart, article, and outcome data. The useful modeling chain is usually:

companies → filings → extracted terms / consensus → outcomes / news

That means you can start with an issuer, resolve it to a durable CIK, add filing events, add deal terms, then join to post-listing outcomes. IPOGrid’s docs specifically recommend using CIK as the durable issuer key because tickers can drift, collide, or point to units and warrants instead of the common-stock IPO.

Example feature families

Here are the kinds of features I’d extract for a financial model.

Issuer identity features

CIK

issuer name

ticker hint

resolved ticker

exchange

sector

issuer kind

market family

deal type

operating company vs SPAC vs fund vs follow-on

These are basic filters, but they matter. Mixing operating IPOs, SPACs, funds, direct listings, and follow-ons in the same model is usually garbage-in, garbage-out.

Filing timeline features

first registration date

latest amendment date

number of amendments

days from initial filing to effectiveness

days from latest amendment to pricing

presence of S-1, F-1, S-11, 424B4, 424B1, 8-A, EFFECT

final prospectus filed or not

recent filing activity count

filing recency bucket

The docs call out SEC filing events such as registrations, amendments, final prospectuses, effectiveness notices, and 8-A registrations as explicit research surfaces.

Deal term features

offer price

price range low

price range high

range midpoint

shares offered

gross proceeds

underwriters

use of proceeds

unit details

warrant terms

range revision direction

range revision magnitude

priced above range / in range / below range

IPOGrid exposes extracted offer price, range, shares, proceeds, underwriters, unit details, warrants, and use-of-proceeds fields so you do not have to reopen every filing manually.

Consensus / listing features

resolved listing date

resolved exchange

resolved offer price

resolved shares

resolved proceeds

latest terms snapshot

classification

consensus ticker

consensus listing fields

The API supports company lookups with includes such as latest_terms, classification, consensus, filings, news, and financial_snapshot.

Outcome features

trading start date

first close

1D return

week-one close

week-one return

current performance

offer-to-first-close return

offer-to-week-one return

IPOGrid has outcome rows for first close, 1D return, week-one close, and post-listing performance.

Market context features

gross proceeds by week

prospectus counts

final prospectus counts

first-day returns by sector

filing activity by form family

sector-level IPO volume

recent IPO temperature

The chart API supports metrics such as gross proceeds, prospectus counts, final prospectus counts, and first-day returns grouped by sector, form family, or status over fixed or trailing windows.

Basic API calls

Use an API key for the richer endpoints:

export IPOGRID_API_KEY="your_key_here" 

Find companies and include the useful joins:

curl -H "Authorization: Bearer $IPOGRID_API_KEY"  "https://ipogrid.com/api/v1/companies?scope=all&kind=operating&include=latest_terms,classification,consensus" 

Fetch company detail by ticker or CIK:

curl -H "Authorization: Bearer $IPOGRID_API_KEY"  "https://ipogrid.com/api/v1/companies/spcx?include=filings,news,latest_terms,classification,consensus,financial_snapshot" 

Pull recent final prospectus filings:

curl -H "Authorization: Bearer $IPOGRID_API_KEY"  "https://ipogrid.com/api/v1/filings?form_type=424B4&filing_date_from=2026-06-01&include=company,terms" 

Pull recent outcomes:

curl -H "Authorization: Bearer $IPOGRID_API_KEY"  "https://ipogrid.com/api/v1/outcomes?trading_start_date_from=2026-06-01&limit=100" 

Pull public articles without auth:

curl "https://ipogrid.com/api/v1/articles?limit=30" 

The public article feed works without authentication; deeper company, filing, and outcome data uses API key bearer auth.

Backtest shape

A clean IPO backtest should freeze the cohort before looking at outcomes.

For example:

1. Select operating-company IPOs only.
2. Require a final prospectus or effective status.
3. Exclude SPACs, funds, vehicles, direct listings, and follow-ons.
4. Resolve each issuer to CIK.
5. Pull only filing and term data available as of the cohort date.
6. Join outcomes after trading starts.
7. Measure first-day and week-one returns against the offer price.

IPOGrid’s research workflow docs make the same basic point: freeze the cohort definition first, then join market bars or outcomes only after the cohort date to avoid leaking future information.

Example model table

A row in a modeling dataset might look like this:

cik company_name issuer_kind sector exchange form_family initial_filing_date latest_amendment_date final_prospectus_date effective_date trading_start_date days_initial_to_effective days_final_prospectus_to_trade amendment_count offer_price range_low range_high range_midpoint priced_vs_range shares_offered gross_proceeds underwriter_count has_warrants is_unit_deal first_close day1_return_pct week1_close week1_return_pct sector_ipo_count_26w sector_avg_day1_return_26w 

That table can feed normal financial modeling workflows: screening, regression, ranking, clustering, risk controls, or event studies.

Caveats

Do not treat latest terms as historical truth. Latest snapshots are useful for current watchlists, but historical tests should use the filings and market data that existed around the test date. IPOGrid’s freshness docs explicitly warn not to silently fill missing historical fields with today’s values.

Also separate clocks carefully. SEC filing dates, SEC acceptance times, pricing dates, and trading dates are different events. A Friday filing, Monday effectiveness notice, and Tuesday first trade should not be collapsed into one timestamp.

Finally, ticker logic needs care. Units, warrants, and share classes can trade separately. A ticker hint is not always the same thing as the security your model is trying to study. For anything serious, resolve to CIK first, then verify the traded instrument.

Practical use cases

A few useful models you can build from this:

IPO readiness screen: find effective or recently amended operating IPOs with exchange listing signals and extracted terms.

Pricing-change model: compare initial range, revised range, final offer price, and sector conditions.

Day-one return model: join final prospectus terms to first close and sector-level IPO context.

Week-one fade model: compare first close to week-one close.

Filing-momentum model: count amendments, form changes, and final prospectus timing.

Sector heat model: use chart data to track issuance volume, proceeds, and first-day returns by sector.

The important part is not just having IPO data. It is having issuer, filing, term, and outcome data joined in a way that does not leak future information into the model.

Hello,

Just want to drop a project I think others might find helpful.

This one’s for my fellow GIS people (and anyone else of course). I’ve put together a resource for free datasets that are local to the NoVA region. All data is pulled from public/open sources and each data set comes with a DOI number via zenodo if you need citations.

Figured this could be useful for anyone whose capstone or thesis is focused in the NoVA region

TLDR: free NoVA data sets, with new sets every morning (typically before 630am), no signups or other nonsense.

data library is here: [https://keystonegis.com/data-library](https://keystonegis.com/data-library)

if you rather pull from zenodo itself: [https://zenodo.org/search?q=metadata.creators.person\_or\_org.name%3A%22Keystone%20GIS%22&l=list&p=1&s=10&sort=bestmatch](https://zenodo.org/search?q=metadata.creators.person_or_org.name%3A%22Keystone%20GIS%22&l=list&p=1&s=10&sort=bestmatch)

I’m curious how people are currently handling ZIP code to Census Tract (and vice versa) conversions in their workflows.

A few questions:

• What tool or service are you using to convert ZIP codes to Census Tracts (or Census Tracts back to ZIP codes)?
• What’s your actual business use case? (Market research, direct mail, demographics, healthcare, real estate, site selection, etc.)
• Do you need the conversion as a one-time lookup, or are you doing it in bulk?

I’m asking because I’ve noticed the process can be surprisingly manual, especially when you need to enrich hundreds or thousands of records.

I’m considering building a very simple tool where you can drag and drop an Excel or Google Sheet, and in less than five minutes it converts ZIP ↔ Tract (and potentially enriches the data with Census demographics) without needing to write code or use GIS software.

Would something like that actually save you time, or are your current tools already good enough? If it wouldn’t be useful, I’d love to know why.

Interested to hear how everyone is solving this today.

[Disclaimer – Freely accessible]

The Earth Data Hub distributes global Climate Reanalysis such as ECMWF’s ERA5 and multi-decadal Climate Projections such as the Destination Earth’s Climate Adaptation Digital Twin in Zarr format.
Any Zarr-compatible tool can access these datasets with just a few lines of code.

Try it out:

import xarray as xr EDH_API_KEY = "your_EDH_api_key" # can be found at: https://earthdatahub.destine.eu/account-settings xr.open_dataset( f"https://edh:{EDH_API_KEY}@api.earthdatahub.destine.eu/era5/era5-single-levels-atmosphere-v0.zarr", chunks={}, engine="zarr", ) 

Im trying to put together a clean corpus for an ai reasoning test right now (mostly to distract myself from how badly im doing at my driving school lessons this week tbh) and I downloaded this massive “pre-cleaned” open-source corporate registry dataset

The data hygiene is just offensively bad. half the rows are misaligned and the contact columns are filled with literal regex nightmares and placeholder junk. I ended up having to pipe the whole thing through mail tester just to strip out the dead syntax and fake emails so my script wouldn’t crash every five seconds

why do people upload these 10GB csv files to github without doing even basic sanitization first? my laptop fan is literally screaming right now.

A few of us trade vol on Deribit day-to-day and kept running into the same wall: the options/tick data good enough to actually backtest on is priced like it’s meant for funds with five-figure data budgets, not for someone running their own book. Tardis is genuinely solid, but at that price it’s out of reach if you’re not a fund.

So we built Volar: minute-level BTC, ETH, and SOL options chains, computed Greeks, SVI-fitted vol surfaces per tenor, and a dense 40-month BTC archive (2021-06 → 2024-09) at per-minute resolution, not daily aggregates. Every row is source-tagged (live capture vs. historical vs. modeled) so you always know what you’re actually looking at.

Pricing, to be upfront about it: free Sandbox tier if you just want to poke at the schema and sample data first, no card needed. Pro is $99/mo for live BTC/ETH/SOL plus a 90-day rolling BTC window; the full historic archive is on the annual plan. Didn’t want to bury that distinction since I know this crowd will check.

Genuinely interested in feedback, especially from anyone who’s tried to backtest a crypto vol strategy and hit walls with existing data, what’s missing, what’s annoying, what would actually make you trust a smaller vendor over an established one. Happy to answer anything on the data/methodology side too.

(Disclosure: I’m one of the people building this, didn’t want to post without saying so upfront.)

Looking for individuals willing to participate in a Supplier Review so that we can start to “Grade” suppliers.[ ](https://docs.google.com/forms/d/e/1FAIpQLSeaxdW1LqTHzK2GWDee_fF56ZBfVxYRo3mfFgI1YJphO29fCg/viewform)If youd like to be part of the community please join [Skool here](https://www.skool.com/ncunderground-7525/about).

With how rough the job market is, I can’t land a job despite looking for so long. So I am trying to start another data project with Python, SQL, Alteryx, Power BI to add to my portfolio.

However, I do not want to use synthetic datasets or those from Kaggle. Is there any platform that you can extract your personal historical data from? I thought about my credit card transactions but apparently that is not accessible for security reasons. Thank you!

I am building a fuel price dataset that tracks any type of gas prices at gas station in multiple countries such as France, Spain, Italy, Mexico and others. Would love to get feedback on potentital interest and what level of granularity would be interesting for you.

Hello everyone, I’m working on my Final Year Project and I am looking for any road traffic dataset available (for free) that contains numeric information. (Most importantly Timestamps or Date with Time and Vehicle counts, even if each vehicle type has a separate count, I will consider it). I need this in numeric format (also data must be big like 1000+ rows because I keep finding smaller 20-40 rows of data on kaggle, need bigger for better training) in order to apply Time Series using FB Prophet in order to predict the future traffic flow (historical data needed to demonstrate a prototype model before moving to real cameras). If anyone knows where I can find one please share and thanks in advance!

Something I keep running into when observing and working with long-term climate datasets: the data exists, it’s publicly available and can be easily accessed, reasonably well-documented, and yet asking a simple cross-decade question still takes an unreasonable amount of work to actually answer.

For example if I want to ask something like “how has the distribution of extreme precipitation events shifted since 1980 across the Gulf Coast?”, it will take forever to actually find an answer. Starting off with finding the right NOAA dataset, then figuring out which version covers your full time range, dealing with the fact that station coverage changed significantly in the 90s, reconciling pre and post-satellite era measurement methodologies, writing custom scripts to normalize format differences across dataset versions, then finally doing the actual analysis.

Do you think with the AI wave we are seeing, this entire process or any part of it will be improved? NOAA NCEI alone archives over 229 terabytes of data every month across 130+ observing platforms, but the part of analyzing and understanding those data, I feel like, still lags behind.

Are you also mostly doing Python + manual joins and custom preprocessing per dataset? Has anything changed in how you approach cross-decade multi-source queries, or is it still bespoke work every time?