For the minute by minute bars data, columns are:

“symbol”, “timestamp”, “open”, “high”, “low”, “close”, “volume”, “vwap”, “trade_count”, “spy_close”, “iv”, “delta”, “gamma”, “theta”, “vega”, “rho”

For tick_by_tick (all individual trades executed) columns are:

“symbol”, “timestamp”, “price”, “size”, “exchange”, “conditions”, “spy_close”, “iv”, “delta”, “gamma”, “theta”, “vega”, “rho”

It goes back a few years, depending on the ticker.

Hi everyone,

I’ve been working on a data-centric pipeline for constructing SFT and KTO datasets for small language models, targeting models ranging from a 1.58B ternary model up to 12B parameters (with a particular focus on the 1.5B–4B range), using an Italian tool-calling assistant (“Liara”) as a case study.

Instead of focusing on model architecture, the goal is to reduce common failure modes through dataset construction itself:

• tool over-calling
• style collapse
• excessive verbosity
• semantic redundancy
• memory inconsistencies

The pipeline currently includes:

• typed validation outcomes (PASS / Soft Reject / Hard Reject / Warning)
• semantic + structural deduplication
• multi-teacher generation
• dataset lineage and versioning
• regression set
• dataset health dashboard
• capability-based dataset profiling for different model sizes
• typed routing into SFT, KTO-negative, or discard
• Soft Reject examples are not discarded by default: they undergo additional validation and, if confirmed, are reused as KTO-negative examples rather than being treated as unusable data.

The current specification describes the methodology. The implementation is underway, and the experimental validation is currently running.

I’d love feedback from people who have built or maintained instruction datasets:

• Which parts seem genuinely useful?
• Which ideas already exist in other pipelines?
• What ablation studies would you expect before considering this publishable?

I’m currently generating the gold seed dataset, which is the most time-consuming part of the pipeline and is expected to take around 10 days at the planned scale. Once that’s complete, I’ll publish the implementation, the ablation results, and the evaluation so the methodology can be assessed based on experimental evidence rather than design alone.

In the meantime, I’d really appreciate any feedback or suggestions on the pipeline itself.

A lot of IPO research starts as manual filing review: S-1s, F-1s, amendments, 424B4s, effectiveness notices, ticker changes, exchange hints, and post-listing performance. That works for one company. It breaks down if you want to build watchlists, backtests, screens, or systematic models.

The IPOGrid API is meant to make that workflow structured.

Docs:

https://ipogrid.com/docs

API reference:

https://ipogrid.com/api/v1/docs

OpenAPI:

https://ipogrid.com/api/v1/openapi.json

What you can pull

IPOGrid exposes structured company, filing, news, chart, article, and outcome data. The useful modeling chain is usually:

companies → filings → extracted terms / consensus → outcomes / news

That means you can start with an issuer, resolve it to a durable CIK, add filing events, add deal terms, then join to post-listing outcomes. IPOGrid’s docs specifically recommend using CIK as the durable issuer key because tickers can drift, collide, or point to units and warrants instead of the common-stock IPO.

Example feature families

Here are the kinds of features I’d extract for a financial model.

Issuer identity features

CIK

issuer name

ticker hint

resolved ticker

exchange

sector

issuer kind

market family

deal type

operating company vs SPAC vs fund vs follow-on

These are basic filters, but they matter. Mixing operating IPOs, SPACs, funds, direct listings, and follow-ons in the same model is usually garbage-in, garbage-out.

Filing timeline features

first registration date

latest amendment date

number of amendments

days from initial filing to effectiveness

days from latest amendment to pricing

presence of S-1, F-1, S-11, 424B4, 424B1, 8-A, EFFECT

final prospectus filed or not

recent filing activity count

filing recency bucket

The docs call out SEC filing events such as registrations, amendments, final prospectuses, effectiveness notices, and 8-A registrations as explicit research surfaces.

Deal term features

offer price

price range low

price range high

range midpoint

shares offered

gross proceeds

underwriters

use of proceeds

unit details

warrant terms

range revision direction

range revision magnitude

priced above range / in range / below range

IPOGrid exposes extracted offer price, range, shares, proceeds, underwriters, unit details, warrants, and use-of-proceeds fields so you do not have to reopen every filing manually.

Consensus / listing features

resolved listing date

resolved exchange

resolved offer price

resolved shares

resolved proceeds

latest terms snapshot

classification

consensus ticker

consensus listing fields

The API supports company lookups with includes such as latest_terms, classification, consensus, filings, news, and financial_snapshot.

Outcome features

trading start date

first close

1D return

week-one close

week-one return

current performance

offer-to-first-close return

offer-to-week-one return

IPOGrid has outcome rows for first close, 1D return, week-one close, and post-listing performance.

Market context features

gross proceeds by week

prospectus counts

final prospectus counts

first-day returns by sector

filing activity by form family

sector-level IPO volume

recent IPO temperature

The chart API supports metrics such as gross proceeds, prospectus counts, final prospectus counts, and first-day returns grouped by sector, form family, or status over fixed or trailing windows.

Basic API calls

Use an API key for the richer endpoints:

export IPOGRID_API_KEY="your_key_here" 

Find companies and include the useful joins:

curl -H "Authorization: Bearer $IPOGRID_API_KEY"  "https://ipogrid.com/api/v1/companies?scope=all&kind=operating&include=latest_terms,classification,consensus" 

Fetch company detail by ticker or CIK:

curl -H "Authorization: Bearer $IPOGRID_API_KEY"  "https://ipogrid.com/api/v1/companies/spcx?include=filings,news,latest_terms,classification,consensus,financial_snapshot" 

Pull recent final prospectus filings:

curl -H "Authorization: Bearer $IPOGRID_API_KEY"  "https://ipogrid.com/api/v1/filings?form_type=424B4&filing_date_from=2026-06-01&include=company,terms" 

Pull recent outcomes:

curl -H "Authorization: Bearer $IPOGRID_API_KEY"  "https://ipogrid.com/api/v1/outcomes?trading_start_date_from=2026-06-01&limit=100" 

Pull public articles without auth:

curl "https://ipogrid.com/api/v1/articles?limit=30" 

The public article feed works without authentication; deeper company, filing, and outcome data uses API key bearer auth.

Backtest shape

A clean IPO backtest should freeze the cohort before looking at outcomes.

For example:

1. Select operating-company IPOs only.
2. Require a final prospectus or effective status.
3. Exclude SPACs, funds, vehicles, direct listings, and follow-ons.
4. Resolve each issuer to CIK.
5. Pull only filing and term data available as of the cohort date.
6. Join outcomes after trading starts.
7. Measure first-day and week-one returns against the offer price.

IPOGrid’s research workflow docs make the same basic point: freeze the cohort definition first, then join market bars or outcomes only after the cohort date to avoid leaking future information.

Example model table

A row in a modeling dataset might look like this:

cik company_name issuer_kind sector exchange form_family initial_filing_date latest_amendment_date final_prospectus_date effective_date trading_start_date days_initial_to_effective days_final_prospectus_to_trade amendment_count offer_price range_low range_high range_midpoint priced_vs_range shares_offered gross_proceeds underwriter_count has_warrants is_unit_deal first_close day1_return_pct week1_close week1_return_pct sector_ipo_count_26w sector_avg_day1_return_26w 

That table can feed normal financial modeling workflows: screening, regression, ranking, clustering, risk controls, or event studies.

Caveats

Do not treat latest terms as historical truth. Latest snapshots are useful for current watchlists, but historical tests should use the filings and market data that existed around the test date. IPOGrid’s freshness docs explicitly warn not to silently fill missing historical fields with today’s values.

Also separate clocks carefully. SEC filing dates, SEC acceptance times, pricing dates, and trading dates are different events. A Friday filing, Monday effectiveness notice, and Tuesday first trade should not be collapsed into one timestamp.

Finally, ticker logic needs care. Units, warrants, and share classes can trade separately. A ticker hint is not always the same thing as the security your model is trying to study. For anything serious, resolve to CIK first, then verify the traded instrument.

Practical use cases

A few useful models you can build from this:

IPO readiness screen: find effective or recently amended operating IPOs with exchange listing signals and extracted terms.

Pricing-change model: compare initial range, revised range, final offer price, and sector conditions.

Day-one return model: join final prospectus terms to first close and sector-level IPO context.

Week-one fade model: compare first close to week-one close.

Filing-momentum model: count amendments, form changes, and final prospectus timing.

Sector heat model: use chart data to track issuance volume, proceeds, and first-day returns by sector.

The important part is not just having IPO data. It is having issuer, filing, term, and outcome data joined in a way that does not leak future information into the model.

Hello,

Just want to drop a project I think others might find helpful.

This one’s for my fellow GIS people (and anyone else of course). I’ve put together a resource for free datasets that are local to the NoVA region. All data is pulled from public/open sources and each data set comes with a DOI number via zenodo if you need citations.

Figured this could be useful for anyone whose capstone or thesis is focused in the NoVA region

TLDR: free NoVA data sets, with new sets every morning (typically before 630am), no signups or other nonsense.

data library is here: [https://keystonegis.com/data-library](https://keystonegis.com/data-library)

if you rather pull from zenodo itself: [https://zenodo.org/search?q=metadata.creators.person\_or\_org.name%3A%22Keystone%20GIS%22&l=list&p=1&s=10&sort=bestmatch](https://zenodo.org/search?q=metadata.creators.person_or_org.name%3A%22Keystone%20GIS%22&l=list&p=1&s=10&sort=bestmatch)

I’m curious how people are currently handling ZIP code to Census Tract (and vice versa) conversions in their workflows.

A few questions:

• What tool or service are you using to convert ZIP codes to Census Tracts (or Census Tracts back to ZIP codes)?
• What’s your actual business use case? (Market research, direct mail, demographics, healthcare, real estate, site selection, etc.)
• Do you need the conversion as a one-time lookup, or are you doing it in bulk?

I’m asking because I’ve noticed the process can be surprisingly manual, especially when you need to enrich hundreds or thousands of records.

I’m considering building a very simple tool where you can drag and drop an Excel or Google Sheet, and in less than five minutes it converts ZIP ↔ Tract (and potentially enriches the data with Census demographics) without needing to write code or use GIS software.

Would something like that actually save you time, or are your current tools already good enough? If it wouldn’t be useful, I’d love to know why.

Interested to hear how everyone is solving this today.

[Disclaimer – Freely accessible]

The Earth Data Hub distributes global Climate Reanalysis such as ECMWF’s ERA5 and multi-decadal Climate Projections such as the Destination Earth’s Climate Adaptation Digital Twin in Zarr format.
Any Zarr-compatible tool can access these datasets with just a few lines of code.

Try it out:

import xarray as xr EDH_API_KEY = "your_EDH_api_key" # can be found at: https://earthdatahub.destine.eu/account-settings xr.open_dataset( f"https://edh:{EDH_API_KEY}@api.earthdatahub.destine.eu/era5/era5-single-levels-atmosphere-v0.zarr", chunks={}, engine="zarr", ) 

Im trying to put together a clean corpus for an ai reasoning test right now (mostly to distract myself from how badly im doing at my driving school lessons this week tbh) and I downloaded this massive “pre-cleaned” open-source corporate registry dataset

The data hygiene is just offensively bad. half the rows are misaligned and the contact columns are filled with literal regex nightmares and placeholder junk. I ended up having to pipe the whole thing through mail tester just to strip out the dead syntax and fake emails so my script wouldn’t crash every five seconds

why do people upload these 10GB csv files to github without doing even basic sanitization first? my laptop fan is literally screaming right now.

A few of us trade vol on Deribit day-to-day and kept running into the same wall: the options/tick data good enough to actually backtest on is priced like it’s meant for funds with five-figure data budgets, not for someone running their own book. Tardis is genuinely solid, but at that price it’s out of reach if you’re not a fund.

So we built Volar: minute-level BTC, ETH, and SOL options chains, computed Greeks, SVI-fitted vol surfaces per tenor, and a dense 40-month BTC archive (2021-06 → 2024-09) at per-minute resolution, not daily aggregates. Every row is source-tagged (live capture vs. historical vs. modeled) so you always know what you’re actually looking at.

Pricing, to be upfront about it: free Sandbox tier if you just want to poke at the schema and sample data first, no card needed. Pro is $99/mo for live BTC/ETH/SOL plus a 90-day rolling BTC window; the full historic archive is on the annual plan. Didn’t want to bury that distinction since I know this crowd will check.

Genuinely interested in feedback, especially from anyone who’s tried to backtest a crypto vol strategy and hit walls with existing data, what’s missing, what’s annoying, what would actually make you trust a smaller vendor over an established one. Happy to answer anything on the data/methodology side too.

(Disclosure: I’m one of the people building this, didn’t want to post without saying so upfront.)

Looking for individuals willing to participate in a Supplier Review so that we can start to “Grade” suppliers.[ ](https://docs.google.com/forms/d/e/1FAIpQLSeaxdW1LqTHzK2GWDee_fF56ZBfVxYRo3mfFgI1YJphO29fCg/viewform)If youd like to be part of the community please join [Skool here](https://www.skool.com/ncunderground-7525/about).

With how rough the job market is, I can’t land a job despite looking for so long. So I am trying to start another data project with Python, SQL, Alteryx, Power BI to add to my portfolio.

However, I do not want to use synthetic datasets or those from Kaggle. Is there any platform that you can extract your personal historical data from? I thought about my credit card transactions but apparently that is not accessible for security reasons. Thank you!

I am building a fuel price dataset that tracks any type of gas prices at gas station in multiple countries such as France, Spain, Italy, Mexico and others. Would love to get feedback on potentital interest and what level of granularity would be interesting for you.

Hello everyone, I’m working on my Final Year Project and I am looking for any road traffic dataset available (for free) that contains numeric information. (Most importantly Timestamps or Date with Time and Vehicle counts, even if each vehicle type has a separate count, I will consider it). I need this in numeric format (also data must be big like 1000+ rows because I keep finding smaller 20-40 rows of data on kaggle, need bigger for better training) in order to apply Time Series using FB Prophet in order to predict the future traffic flow (historical data needed to demonstrate a prototype model before moving to real cameras). If anyone knows where I can find one please share and thanks in advance!

Something I keep running into when observing and working with long-term climate datasets: the data exists, it’s publicly available and can be easily accessed, reasonably well-documented, and yet asking a simple cross-decade question still takes an unreasonable amount of work to actually answer.

For example if I want to ask something like “how has the distribution of extreme precipitation events shifted since 1980 across the Gulf Coast?”, it will take forever to actually find an answer. Starting off with finding the right NOAA dataset, then figuring out which version covers your full time range, dealing with the fact that station coverage changed significantly in the 90s, reconciling pre and post-satellite era measurement methodologies, writing custom scripts to normalize format differences across dataset versions, then finally doing the actual analysis.

Do you think with the AI wave we are seeing, this entire process or any part of it will be improved? NOAA NCEI alone archives over 229 terabytes of data every month across 130+ observing platforms, but the part of analyzing and understanding those data, I feel like, still lags behind.

Are you also mostly doing Python + manual joins and custom preprocessing per dataset? Has anything changed in how you approach cross-decade multi-source queries, or is it still bespoke work every time?

The app is for students that help them:

– find which study programs you can apply for based on your exam track

– or discover careers you’re interested in and the study programs related to them 🙂

## Data

The data is still incomplete due to the lack of clear sources, but you can get what’s already there (Madagascar datasets are the only ones for now).

But mostly, you’re welcome to contribute 🙂

Repo: https://github.com/gigasandwich/giga-roadmap

Data (json) are stored in `/data`

App: https://roadmap.gigas.app

Hi everyone) I’m doing a systematic review and unfortunately don’t have institutional access to embase.
can some please run the search for my and export the results in csv/RIS format?

The search prompt is:

(“Boron Neutron Capture Therapy”[MeSH] OR “boron neutron capture therapy”[Title/Abstract] OR BNCT[Title/Abstract])

Thank you so much!

https://huggingface.co/datasets/ministere-culture/comparia-fr-arena

I am hoping to speak with someone actively sourcing datasets from the film industry. I am not selling anything. I am not asking for anything other than 15 min of your time to geek out over how you’re doing it.

Please delete if not allowed

Disclosure: I built and run GribStream, so this is self-promotion. It is a commercial/freemium API, but there is a free tier, and new free accounts currently get an intro quota boost so people can run real tests before deciding if it is useful.

The data itself is not mine. The original sources are public weather and climate feeds from NOAA/NCEP, ECMWF, Copernicus/ERA5, NOMADS, and public cloud archives. GribStream is a unified API and indexing layer on top of those datasets.

The idea is simple: instead of learning a different access pattern for every weather model feed, you can query many of them through the same API.

It covers common, broad-use datasets like:

• GFS global forecasts
• IFS deterministic and ensemble forecasts
• HRRR high-resolution US forecasts
• NBM forecast blends
• ERA5 reanalysis
• RTMA/URMA surface analyses
• GEFS ensemble forecasts

And also more specialized datasets, for example:

• AQM/NAQFC air-quality guidance for ozone and PM2.5
• GTGN aviation turbulence nowcasts
• aviation icing and turbulence feeds
• SPC severe-weather probability products
• wave, chemistry, UV index, and seasonal forecast products
• AI weather model feeds and archives like AIFS, AIGFS/AIGEFS, GraphCastGFS, and FourCastNetGFS

What the API is mainly useful for:

• pulling time series for one point or thousands of points
• comparing forecast model runs over time
• backtesting with “what was known at the time” cutoffs
• querying multiple variables, levels, model runs, and ensemble members
• getting data back as JSON, CSV, or NDJSON without building a custom weather-data pipeline first

A practical example: if you have 500 solar sites, farms, airports, ships, stores, or insurance exposure locations, you can ask for historical forecasts or recent model data at those coordinates directly, instead of separately wrangling GFS, HRRR, NBM, IFS, ERA5, etc.

Links:

GribStream: https://gribstream.com/

Model catalog: https://gribstream.com/models

Original/public source examples:

https://nomads.ncep.noaa.gov/

https://registry.opendata.aws/collab/noaa/

https://www.ecmwf.int/en/forecasts/dataset/open-data

https://cds.climate.copernicus.eu/

I’d be grateful for feedback from people who use weather, climate, aviation, energy, agriculture, logistics, or environmental datasets. Are there public weather datasets you wish were easier to query? Would you be looking for bulk exports? Interested in being able to setup notifications for weather data events?

Happy to answer questions. I’m trying to make the public model data easier to use while still being clear about where the original data comes from.

So, you guys might know I did $70k of inference on GPT-5.5 extra high. I’m actually willing to sell all of that (to help train models, etc.) for maybe $500-700. If you want, I can reinforce this by selling GPT-5.6 Sol data when that comes out too. We could even have a longer-term partnership! Anyone wanna try?

[mega.cartoon834@passinbox.com](mailto:mega.cartoon834@passinbox.com)

Anyone know where to find historical influencer data? Stuff like what account age, how many followers they had in the past, etc.

I am building a YOLO-based PCB reverse engineering pipeline for academic research. I desperately need the FICS-PCB dataset (hosted on TrustHub) to scale my component detection model, but it is locked behind an authentication wall. I have emailed the authors but am waiting on a response. Looking for a mirror or anyone with TrustHub access.

The Engineering Context

I am currently working on an automated Printed Circuit Board (PCB) reverse engineering and hardware assurance pipeline. The end goal is automated Bill of Materials (BoM) extraction.

Initially, I replicated the classical image processing pipeline from Kleber et al. (2017). While I got decent IC detection using rigid OpenCV heuristics (HSV masking, Otsu thresholding, morphological transformations), the pipeline was far too brittle. Any change in PCB substrate color or environmental lighting required manual parameter tuning.

I recently pivoted the component detection stage to a Convolutional Neural Network (Ultralytics YOLO).

The Problem: Data Starvation

The YOLO architecture completely bypassed the need for manual CV parameter tuning and successfully isolated primary SoCs (97%+ confidence) against complex backgrounds. However, I am hitting a massive data starvation wall.

To make this model generalize across edge cases and minority classes, I need high-volume, annotated data.

The Roadblock

The FICS-PCB: A Multi-Modal Image Dataset (Lu et al., 2020) is exactly what I need. It contains 9,912 PCB sample images and over 77,000 component annotations.

• The dataset is hosted onTrustHub.
• It is locked behind an ID/password authentication barrier.
• I have already sent a formal request from my institutional email to the principal investigators (University of Florida), but I am waiting on approval and my research sprint is currently bottlenecked.

I’m sharing an interest in datasets related to image metadata — especially EXIF, IPTC, and XMP fields — for use in forensic analysis, provenance research, search/indexing, and large-scale metadata extraction workflows.

I’m specifically looking for datasets that include one or more of the following:

• Original image files with metadata intact.
• Paired image + metadata exports.
• Large collections suitable for testing extraction, indexing, normalization, or deduplication pipelines.
• Real-world examples that include camera data, timestamps, geotags, creator info, editing history, and embedded tags.
• Datasets useful for studying metadata loss across platforms or image-processing tools.

If anyone knows of public datasets, archives, or research corpora in this area, I’d appreciate recommendations. I’m especially interested in datasets that are legal to analyze and can be used for technical experimentation.

Disclosure: I work on image-meta.com, which is relevant to this topic.