Category: Datatards

Here you can observe the biggest nerds in the world in their natural habitat, longing for data sets. Not that it isn’t interesting, i’m interested. Maybe they know where the chix are. But what do they need it for? World domination?

[Data Licensing] 2,000+ Real Indian B2B Sales Conversations With Transcripts + Outcomes — Looking For AI Companies Actively Acquiring This Type Of Data

I’m exploring a potential commercial data-licensing opportunity and looking to connect specifically with AI companies, dataset buyers, speech/voice AI teams, AI-agent companies, or data brokers actively sourcing proprietary conversational data.
I may have access to 2,000+ real-world Indian B2B sales conversations, with the actual inventory potentially significantly larger.
Current dataset characteristics:
• ~40 min average conversation length
• Hindi / Hinglish / English
• Video + audio
• Transcripts available
• Real customer ↔ salesperson interactions
• Natural questions, objections, negotiation and follow-ups
• Linked business outcomes: demo / won / lost
• Indian B2B sales context
The interesting part isn’t just the audio.
The conversations can potentially be structured as:
Customer intent → objection/question → salesperson response → follow-up → sales outcome
This could potentially support:
• Voice AI / conversational AI
• AI sales agents
• Speech & multilingual model development
• Indian-language / Hinglish evaluation
• Sales-agent benchmarking
• Conversation-quality evaluation
• Human preference / response datasets
• Training realistic AI sales workflows
I’m not looking for generic feedback or people interested in datasets in theory.
I’m specifically interested in connecting with people who currently acquire, license, commission or broker proprietary AI training/evaluation data.
I can provide a high-level dataset specification to qualified/interested parties.
No recordings or identifiable customer information will be posted publicly. Any commercial licensing would be subject to verification of ownership, consent, privacy and applicable data rights.
If your company is actively sourcing this type of data, comment or DM me with what you are currently looking for (modality, language, hours, annotation requirements, licensing model, etc.).

submitted by /u/EconomyLayer2854
[link] [comments]

Data Creation Tools – Africa Cameroon

Hi everyone,

I work in international cooperation and I’m currently working with a major city in Cameroon.

We’re primarily working on a project for an urban observatory oriented to GIS data.

Financial and Human Resources are limited, so we’re looking for tools that could help us create data on the longterm. We are currently gathering data from local stakeholders and OpenStreetMap.

I’ve identified different open source or affordable tools, such as:

– AI segmentation by terra lab https://terra-lab.ai/fr/ai-segmentation
– Google Open Buildings : https://sites.research.google/gr/open-buildings/
-DEM comparator : https://github.com/nadiopt-cell/dem\_comparator

I would like to know about any other tools such as this (probably with AI). With the current data situation, even if the data is not 100% precise, it’s better than nothing.

Thanks !

submitted by /u/Revolutionary-Two780
[link] [comments]

Question Of Legal Importance – How To Interpret Profile_privacy_changes.html From Instagram Data Set?

Hi friends,

I have had a private Instagram page for about a year now. Someone has been harassing me, and attached a screenshot of my private Instagram story I posted in an anonymous email to my in-laws, which proves that my harasser is, or at least was, one of my followers.

We are preparing legal action, and I am trying to collect proof that my Instagram account was private on the date the screenshot of the story was made. The only issue is that the bulk data I downloaded is contradictory:

– profile_privacy_changes.html indicates that my profile switched to public on Sep 26, nothing else. This seems to indicate that my profile is currently public, but I am certain the exact opposite happened. I also certainly didn’t change my privacy settings after downloading my Instagram data.

– profile_activity.html is the only other record of the profile privacy settings change, but only says “Is profile private? True”

– personal_information.html also correctly displays that my account currently private.

Does someone know if this is a bug and whether switched to public in profile_privacy_changes.html actually means the opposite, namely that I switched my profile from public to private?

Thank you!

submitted by /u/Key_Measurement6787
[link] [comments]

I’m Stuck Finding Usable Historical Data For A Bayesian PR Risk Model — Looking For Advice On How To Proceed

Hi everyone,

I’m a student working on a research project on risk-aware GitHub PR review. I’m doing the project mostly on my own and I don’t have access to a research lab, large compute budget, or people who can manually annotate thousands of PRs, so I’m trying to find a practical approach that I can actually finish.

The idea is to take a GitHub PR and estimate four types of risk:

  1. Bug / correctness

  2. Security

  3. Compatibility

  4. Cross-system / integration

The architecture I’m working with has four separate risk models. They share the same PR characteristics/features, but each risk model has its own historical data, prior, and evidence.

My main problem is the historical data needed for those priors.

At first, I looked for a single PR dataset where I could get reliable PR-level outcomes for all four risks. I couldn’t find one.

I then tried looking for separate datasets for each individual risk model. I thought this would solve the problem, but I keep finding datasets where the labels look relevant at first but don’t actually represent the outcome I need.

For example, SEVRA-plus looked very promising for the Security model:

https://huggingface.co/datasets/RedAI4Code/SEVRA-plus

It contains security-related PR examples with vulnerability/CWE information, but the malicious PRs are deliberately constructed by reversing real CVE security fixes. So although they are useful for evaluating or studying security vulnerabilities, I don’t think I can use their class distribution directly as a real-world prior for ordinary GitHub PRs.

I’ve run into similar issues with other datasets:

– some label the linked issue rather than the PR implementation,

– some label review comments rather than actual PR outcomes,

– some contain artificially constructed vulnerable/failing PRs,

– some only give merge/close status, which doesn’t tell me whether the PR itself was buggy, vulnerable, incompatible, etc.

The distinction between the issue and the PR is especially important for what I am trying to do.

For example, imagine a maintainer opens a security issue, someone creates a PR to fix it, but the PR implementation itself contains a correctness bug and gets rejected. For my problem, I need to know the nature of the PR, not simply inherit the security label from the original issue.

Similarly, a PR could be opened to fix a small bug, get merged, and then later cause a compatibility problem. Again, I care about what happened because of the PR implementation, not just why the PR was originally opened.

Because I couldn’t find a dataset that directly gives me what I need, I tried a practical compromise.

I took 96 real PRs from SWE-Review-Chat, filtered them for sufficient evidence, and used an LLM to annotate the four risk states from the information available in the PR record, such as the description, review discussion, diff context, tests, and lifecycle information.

I’m treating these as weak/model-assisted labels rather than independent ground truth.

The resulting usable outcomes are:

Bug:

30 present / 9 absent

Security:

1 present / 7 absent

Compatibility:

5 present / 12 absent

Cross-system:

4 present / 8 absent

So now I feel like I’ve hit a wall.

I can keep searching for datasets, but so far I haven’t found anything that solves the underlying problem. I also don’t have the resources to manually establish reliable ground truth for thousands of PRs.

I’m therefore looking for advice on “how I should move forward from here”.

Should I continue with the small real dataset I have and explicitly model the uncertainty caused by the sparse risks?

Should I rely on LLM-assisted annotations of real PRs as a practical research compromise, or is there a better low-resource approach that I am missing?

Or is there a completely different way of constructing the historical priors that would make more sense for this problem?

I’m not looking for a perfect dataset at this point. I’m mainly looking for a practical and defensible way to move forward given that I’m a student doing this alone with limited time and resources.

If anyone has worked on GitHub PR datasets, Mining Software Repositories, empirical software engineering, code-review research, or Bayesian risk modelling, I would really appreciate any advice on what you would do in this situation.

Thanks!

submitted by /u/Accomplished-Fun4629
[link] [comments]

Where To Bulk-download Genuinely Public Domain Images, And Why “no Known Copyright Restrictions” Doesn’t Mean Public Domain

Every few weeks someone asks where to get public domain images at volume. The source list part is easy and it’s below. The part that actually costs people money is the rights, so there’s more on that.

I do this for a living, I build and licence image archives, and I’ve had to unpick most of these the hard way. Not legal advice.

The sources

Smithsonian Open Access is the cleanest of the lot. Genuinely CC0, commercial use explicitly allowed, millions of assets. Free key via api.data.gov, though honestly you can skip the API and just take the weekly JSON dumps off their GitHub or AWS. Anything marked “usage conditions apply” is not free, ignore those.

NARA is the big one for anything historic. Tens of millions of records, most of it CC0. The catalog API doesn’t need a key for basic search and the whole catalogue is sitting in an AWS S3 open-data bucket if you want it all. Do check the “Use Restriction(s)” field, a small number of donated items are restricted and they’re mixed in with everything else.

NASA needs no key whatsoever. images-api.nasa.gov/search, then /asset/{id} for the full res. Handy trick, if the record has no copyright field it’s public domain, if it has one it’s third party and you drop it.

DVIDS has the most explicit wording of any of them, their API terms say outright that the assets are free for commercial use. Free key. Two irritations though. You get 50 results a page and only the first 1,000 hits per query, so you have to slice everything by date or branch, and some of the assets are contractor shot or “courtesy photos” which are absolutely not public domain. Check the copyright field on every one.

Library of Congress, add ?fo=json to the end of more or less any search or item URL and you get the record back as JSON. Throttle yourself, they rate limit hard and you can’t page past 100,000 results.

Something on LoC that isn’t in their docs and took me ages to work out. You can build the image URL straight from the digital ID rather than scraping item pages. An ID like fsac.1a35373 gives you

https://tile.loc.gov/storage-services/service/pnp/fsac/1a35000/1a35300/1a35373v.jpg 

Collection, 5 digit block, 3 digit block, then the ID with a size letter on the end. r small, v service copy, u.tif master.

Wikimedia Commons is per file licensing so filter before you download, not after. comload is the tool worth knowing, it pulls a whole category and its subcategories with the rights metadata attached. PetScan if you want to intersect a category with a licence. Keep PD-USGov, PD-old and CC0.

Europeana with &reusability=open. Internet Archive through the ia CLI.

Now the bit that catches people

“No known copyright restrictions” is not a licence. It’s the institution telling you what they believe, with no warranty attached. It can mean they own it and won’t chase you, or that they think they can authorise use. Neither of those is public domain. The Public Domain Mark is the same, Creative Commons say themselves it isn’t a legal instrument and anyone can apply it to anything. CC0 is the exception, that one is a proper legal waiver.

Second thing, US government means US federal. 17 USC 105 covers USAF, Navy, NASA and the rest. It does not cover foreign governments, it does not cover US state or city government, and it does not cover contractors working for any of them. The contractor one gets people all the time because the image sits in a federal archive looking federal.

Third, and this is the one nobody sees coming. In 1996 the URAA restored US copyright to a pile of foreign works that had lapsed there on technicalities, provided they were still protected at home on 1 January 1996. So a 1950s British photo that looks perfectly US public domain can be protected until 95 years after publication. If you’re outside the US and your users aren’t, you want it clear both ends.

Fourth, public domain clears the photograph and nothing else. Trademarks, logos, liveries, all separate. Military insignia, separate. Identifiable people carry publicity and privacy rights and a photo from 1943 has no model release. Plenty of genuinely public domain images are still editorial only in practice.

Current US cutoff is anything published in 1930 or earlier, and it moves on a year every January.

Last thing

Capture provenance while you’re downloading, not afterwards. One row per file, source, permalink, source ID, creator, date, the exact rights wording, the rights URL, and when you pulled it. For API sources just keep the raw JSON.

You can’t reconstruct any of that later. I’ve tried. And filter on the machine readable rights field before download rather than eyeballing it, at any real volume you’ll get it wrong otherwise.

Disclosure per rule 1, I run an image archive and licence datasets commercially so I’ve obviously got an interest in this area. No link, nothing for sale.

submitted by /u/piemags_aviation
[link] [comments]

What’s Your Actual Process When You Get A Table With No Docs And Nothing That Joins To Anything You Have?

Partner sends over an export last month. 63 columns, maybe 20 named something a human could read. Rest are val_b, metric_14, one that’s just “flag3.” No ID matching anything in our warehouse, no data dictionary, guy who built it left in 2023.

Two days gone. Almost none of it pipeline work. Just me squinting at distributions going “this one’s between 0 and 1 and 40% null so it’s probably a rate of something,” then hand-checking matches to see if I was making it up. Which I was, twice.

So what do you actually do here:

Profile everything up front, or start joining and work backwards from the wreckage? I go back and forth. Profiling is probably right but I get impatient.

At what point do you just email whoever sent the file? Held out way too long on this one, mostly pride.

Anything better than fuzzy matching on a few fields and hoping, when there’s no shared key at all?

That last one is where I’m stuck. Not the matching, the knowing when to stop trusting it. A 60% confident match is worse than none because it doesn’t fail loudly, it quietly wires two wrong things together and you find out four dashboards later when someone asks why the numbers look weird.

Curious if anyone has a real system or if it’s vibes and coffee for everyone.

submitted by /u/donewitheverything26
[link] [comments]

Navigating China Family Panel Studies (CFPS): An Unofficial Guide For Non-Chinese Speaker

As a bilingual undergraduate user of China Family Panel Studies (CFPS), I have noticed that the English documentations are very scattered across multiple portals, which might be a barrier for non-Chinese speaking researchers attempting to use the data. Here is a navigation guide I compiled to hopefully mitigate some language barrier.

Disclaimer: I am in no way affiliated with CFPS or Peking University, this guide is unofficial and is not guaranteed to be 100% accurate. I will however try my best to provide the most accurate information possible.

Last Update 27 August.

Introduction of CFPS (quoted from the website)

“China Family Panel Studies (CFPS) is a nationally representative, biennial longitudinal survey of Chinese communities, families, and individuals launched in 2010 by the Institute of Social Science Survey (ISSS) of Peking University, China. The CFPS is designed to collect individual-, family-, and community-level longitudinal data in contemporary China. The studies focus on the economic, as well as the non-economic, wellbeing of the Chinese population, with a wealth of information covering such topics as economic activities, education outcomes, family dynamics and relationships, migration, and health.”

##

Home Page

https://www.isss.pku.edu.cn/cfps/ or https://www.isss.pku.edu.cn/cfps/en/index.htm

This is the page that appears when you search China Family Panel Studies (CFPS). Do note that the two language versions are seemingly hosted on two separate websites, and I believe they are not updated in parallel. Some documents and information are only available in the Chinese website and not the English website.

I will illustrate the discrepancies using side-by-side comparison table in the following. I will point out major discrepancies that will likely affect non-Chinese speakers’ navigation of the panel data, instead of pointing out every single tiny difference.

Discrepancies in “News 相关新闻” sub-page

Chinese Version English Version
Last Update 2026.07.07 2025.10.24
Entries 108 14
Covered Content Type Fieldwork & operations /
Data releases Data releases (fewer details)
Academic events Academic events (selective)
Calls for proposals Calls for proposals (in some years)
Training & recruitment /

In general, the English website covers selective summary highlighting key data releases and international engagement; while the Chinese website includes comprehensive internal/domestic record of all CFPS activities.

Discrepancies in “Documentation 文档中心” sub-page

Chinese Version English Version
Questionnaire 调查问卷 (Both have 2010-2022 full questionnaires ) https://www.isss.pku.edu.cn/cfps/wdzx/tcwj/index.htm
2008 & 2009 pilot versions /
A variable navigation website listed (http://surveysearch.akng.net/) /
A side-by-side comparison of 2010 to 2022 questionnaire change (titled: CFPS2010-2022历年问卷内容汇总表) /
User’s Manual 用户手册 (Least discrepancy) https://www.isss.pku.edu.cn/cfps/wdzx/yhsc/index.htm
1 Manual (2017.7.30) 3 Manuals (latest is updated on 2025.10.10)
Documentation 数据文档 (Largest discrepancy, English website only had the “Codebook and SAS” sub-page) https://www.isss.pku.edu.cn/cfps/wdzx/sjwd/index.htm
Measures 量表 Detailed summary of all (psychometric) measures/ scales used in 2010-2022 /
A side-by-side comparison of measures/ scales used in 2010-2022 /
Coding Scheme for Text Data 文本编码 Occupation coding (职业编码): CFPS has very detailed categorization of various occupation using CCSO and ISCO /
Illness coding (疾病编码), Death coding (死亡编码), Undergraduate Institution coding /
Occupation Coding 职业编码 Unsure how this one is different from the occupation coding under “coding scheme” subpage, I believe they are the same thing. /
Logic Flowchart 逻辑流程图 2018, 2020, 2022 flowcharts Given the complex skip logic across hundreds of questions in CFPS (e.g., if answer to Q1 = 5, skip to Q3), logic flowchart visualization is provided to help users identify missing values better. /
Codebook & SAS label 2010-2022 codebook and SAS label & Composite variables across waves Available under the “Data Description” sub-page · Same
Technical Reports技术报告 46 technical documents on few areas: Survey Design & Sampling, Data Cleaning & Variable Construction, Weighting & Socioeconomic Measures, Special Topics & Methodological Innovations, Wave-Specific Data Introductions & Cleaning /

Discrepancies in “FAQ 常见问题” sub-page

Chinese Version English Version
Coverage 13 categories in FAQ 3 categories in FAQ
Registration & Download, Design & Data Collection, Family Relationship Database, Income & Expenditure, Cross-Database Linkage, Data Updates, Publication-Related, Weights, Missing Values, Address, Occupation, Scales/Inventories, Other Data Acquisition, Data Use, Publish with CFPS

Discrepancies in “Contact Us 联系我们” sub-page

Chinese Version English Version
Basic Contact Email & Address Email & Address
Additional Services Free Data Consultation Services 数据咨询 (Online or Face-to-Face) https://www.isss.pku.edu.cn/cfps/lxwm/sjzx/index.htm · /
Paid Personalized Services 个性化服务 /

##

The Most Efficient Navigation

For non-Chinese speaking users, the most efficient path will probably be:

Register Directly at the Data Centre

URL: https://cfpsdata.pku.edu.cn/#/home

· Why? The English and Chinese versions here are better synchronized than the main website.

· What You’ll Find:

  • User’s Manual
  • Measures/Scales documentation
  • Coding Schemes for Text Data
  • Constructed Variables documentation
  • Flowcharts for Questionnaires (logic flow)

All of these documents are in Chinese – but they are available here, unlike on the main English website.

Bonus: The Data Centre also contains tutorial videos for data users (also in Chinese).

Main Portals of CFPS

Portal URL What It Has
Website (Chinese) www.isss.pku.edu.cn/cfps/ Comprehensive news, documentation, FAQ, contact services
Website (English) www.isss.pku.edu.cn/cfps/en/ Project overview, news (selective), basic navigation
CFPS’s Data Centre https://cfpsdata.pku.edu.cn/#/home The actual data and documentation—better bilingual sync
Peking University Open Research Data Platform https://opendata.pku.edu.cn/dataverse/CFPS Accounts are not linked to the CFPS’s data centre.

Summary Table

Task Best Path Language
Register & Download Data Data Centre English & Chinese (better sync)
Questionnaires, User’s Manual English Website Documentation Sub-page English
Other Technical Documents (Measures, Coding Scheme) Data Centre Chinese
Technical Reports Chinese Website Documentation Sub-page Chinese only
News & Announcement English Website for major updates; Chinese Website for details Both
FAQ Chinese Website FAQ Chinese
Tutorial Videos Data Centre Chinese

Hopefully it helps! Please tell me if I made any mistakes.

submitted by /u/Fast_Temperature5983
[link] [comments]

Request For Your United Airlines Flight/MileagePlus Data

Hello! I’m working on a project and I’d like to have some more real flight data from real people.

Personally I’ve found that United Airlines makes this really easy

United Flight Activity
https://www.united.com/en/us/account/activity

Download activity button reference photo

https://cdn.phototourl.com/free/2026-08-27-979545a3-7965-43de-9407-4ba44ca44259.jpg

I will be eternally grateful and publish the dataset on Kaggle with your permission!
(Probably just the flights, not the Mileage plus data)

Reference photo of the cool project I’m working on

https://cdn.phototourl.com/free/2026-08-27-88d3ca03-259b-436b-a83e-4aa79726cedc.png

P.S. I’m probably going to give you an alias of a famous (but deceased) painter, author, director, filmmaker, etc. But if you want to decide your own alias that would be cool too!

Note: I checked my own United-My Activity.csv and did NOT see any PII

submitted by /u/need-to-vacation
[link] [comments]

[Self-Promotion] Follow Up On The Canadian Federal Contract Data. Expiry Dates Are A Floor, Not Real End Date.

Follow up to my post a couple of weeks ago about the Government of Canada proactive disclosure of contracts.

The most useful thing I have learned since came from a retired federal procurement officer. The end date in this data only covers the period a department has actually committed to. A lot of contracts are a short base period with option years attached, and those options are not published until somebody exercises them. So the end date is a floor. It moves quietly and nothing announces it.

You can see the effect once you know to look for it. Contracts that have never been amended run a median of 380 days. Contracts with at least one amendment run 1,087. The amendment rows are where exercised options surface. Roughly 23% of currently running contracts have been amended at least once.

A few other things worth knowing if you pull this source yourself. The full download is 1,313,544 rows, which collapses to 1,100,130 contracts once amendment rows are folded into their parent. Vendor names are rough; 211,165 raw spellings reduce to 149,331 identities after normalising. Category labels are inconsistent enough that one category can appear twice under different spellings, for example “Accounting and audit services” sitting alongside “ACCOUNTING & AUDIT SERVICES”.

Bidder counts are incomplete. Of the 26,023 contracts still running, 21,462 report one. Of those, 16,123 drew a single bid or none.

One thing I am not publishing is a dump. A meaningful number of vendor records are individual people rather than companies, and those names are withheld. The ingest code is public if you want to build your own copy. Source is the proactive disclosure dataset under the Open Government Licence.

If anyone here has already tackled the vendor name normalisation on this dataset, I would like to compare notes on it.

submitted by /u/kapnkrispy
[link] [comments]

Can LLMs Read Paywalled Articles Or News? If Not, How Do They Stay Up To Date And Provide The Latest Information?

Can LLMs bypass paywalls imposed by publications such as Bloomberg, The Economist, and the Financial Times? If they cannot, how can they stay up to date with real-time information? And if they can bypass paywalls, by whatever means, wouldn’t that expose them to potential litigation? Otherwise, wouldn’t this limitation render LLMs ineffective for providing real-time information?

submitted by /u/manuspresso
[link] [comments]

Looking For Advice: How Can I License A Large Swahili Voice Dataset For AI Training?

I run a call center in Dar es Salaam, Tanzania, and we have been operating for about five years.

Over that period, we have accumulated thousands of hours of naturally occurring Swahili call-center conversations. The calls are recorded with customer consent, and we can anonymize/redact personal and sensitive information before any potential licensing.

I’m exploring whether this type of data could be valuable for companies developing:
– Speech recognition / ASR
– Conversational AI
– Voice agents
– Multilingual AI models
– Swahili language models

I’ve started looking into companies such as Defined.ai and Appen, but I’d like to hear from people who have experience in AI training-data licensing or selling speech datasets.

What would be the best way to approach this market?

Specifically:
Which companies or marketplaces should I contact?
Is licensing the dataset non-exclusively generally better than selling it outright?
What documentation/metadata would buyers expect?
What kind of pricing model is typical for large speech datasets?

I’m not looking to publish any of the actual recordings publicly. I’m mainly looking for advice from people who understand the AI data marketplace.
Thanks.

submitted by /u/gillsaint
[link] [comments]

I Scraped/tracked 35 US Cash Advance Apps’ Fees And Terms For Months — Cleaned It Up Into An Open Dataset

Been maintaining a spreadsheet on cash advance apps (Dave, Earnin, Brigit, MoneyLion, etc.) for a side project comparing fees and terms, since most of these apps make it genuinely hard to figure out what you’re actually paying. Fees range very widely by app, and I wanted to make it easier for people with already-stretched finances to find the lowest cost products. Finally cleaned it up enough to make public.

35 apps, covering advance ranges, subscription vs. tip-based models, state availability, and a few other fields. CSV and JSON, CC-BY licensed, updated periodically as terms change (fintech fee structures shift more than you’d think).

Live on Hugging Face: https://huggingface.co/datasets/overdraftapps/cash-advance-apps

Full disclosure, I run a cash advance comparison site and this data feeds into it, methodology and a bit more context is linked in the dataset card if anyone wants the full picture. Happy to answer questions about specific fields or how anything was sourced. If anyone finds a fee that’s changed since I last verified it, genuinely tell me, half of these apps update pricing with zero announcement.

submitted by /u/Money_Seaweed_1895
[link] [comments]

Where Do You Guys Find Datasets For Real World Data Science Projects?

I’m trying to build a proper data science/ML project, but I’m having a hard time finding a dataset that is big enough and not already used by everyone.
For example, there are datasets like the UK Online Retail dataset, Olist, and other popular sales/retail datasets. They’re good datasets, but I see them being used in a lot of projects already.
I don’t want to just download a dataset, do some EDA, train a model and put it on my resume. I want to build something around an actual business problem, where I have to figure out what the problem is, analyze the data, come up with useful insights, maybe build a model, and actually explain how it could help the business.
So where do you guys usually find datasets for this?
Should I try to find data from smaller companies, government sources, APIs, research papers, etc.? Or is it okay to create my own dataset using AI/cloud tools and then create a realistic business problem around it?
For example, if I create a large synthetic sales dataset, could I create a realistic business scenario around it and then treat it like a real project — forecasting sales, understanding customer behavior, optimizing inventory, etc.?
Would that be considered a decent portfolio project, or is using real-world data much better?
I’d mainly like to hear from people who have built projects for their portfolios or have experience hiring for data science/ML roles. Where do you actually get your data from when you want to build something that’s not the same Kaggle project everyone has already done?

submitted by /u/Informal-Dog2741
[link] [comments]

[OC] Optical Music Recognition Benchmark: Ground Truth, Three Engines’ Raw Output, And The Scorer (2.4 MB, CC BY)

Releasing the full artefacts behind an optical music recognition (OMR) benchmark, so the published figures can be recomputed rather than trusted.

**Contents**
– Ground-truth MusicXML for 31 engraved pages from public-domain sources
– The same 31 pages after a deterministic phone-photograph degradation
– Raw MusicXML output from three engines on both suites: a commercial pipeline, Audiveris 5.11.0, oemer 0.1.x
– Transcriptions of 60 systems from OLiMPiC (ICDAR 2024) by two engines
– `score.py` — the scorer, Python standard library only, no dependencies
– `photo_sim.py` — the degradation model
– Per-piece results as JSON, plus the exact competitor CLI invocations

**Metric:** strict F1 at note level — pitch, onset and duration must all match, with a 1/64-note tolerance. Notes only; rests, clefs, key and time signatures, repeats and dynamics are not scored.

**Size:** 2.4 MB. `python3 score.py .` reproduces the published table in about ten seconds.

**Licence:** CC BY 4.0 for the engraved notation, transcriptions and scripts. The OLiMPiC-derived transcriptions inherit that dataset’s CC BY-SA; its ground truth is not redistributed.

**Disclosure:** I work on one of the three engines. The scorer contains no engine-specific branching and a published erratum corrected a flaw that had been penalising a competitor.

https://www.trysheetly.com/compare/omr-benchmark

submitted by /u/alyimsa
[link] [comments]

Before Labeling 10,000 Images, Label The Same 100 Twice

One of the cheapest ways to catch dataset problems is to run a small annotation pilot before scaling.

Give two annotators the same 50–100 representative images — including occlusions, cropped objects, unusual angles, blur, and borderline classes — and compare where they disagree.

The disagreements usually reveal that the problem isn’t the annotators. The task itself is underspecified: should they label the visible or full object? When is an object too occluded? What should happen when two class definitions overlap?

Fix those decisions in the guidelines, repeat the pilot, and only then start labeling thousands of images.

It’s less exciting than auto-labeling, but it can prevent a large dataset from becoming consistently inconsistent.

Do you run this kind of agreement check before larger annotation projects? How many images are usually enough to expose problems?

submitted by /u/onesunnysunday
[link] [comments]

I Built A Dataset Tracking Daily US Job Postings Across 44 Sectors Since 2020 (Indeed Hiring Lab Data, CC BY 4.0)

What it is
Daily and sector-level tracking of the US job market since February 1, 2020 (pre-pandemic baseline) — how hiring demand shifted through COVID, the 2021–22 boom, and the tech slowdown after. Includes a “sector summary” file with the headline numbers already computed for you: net change since baseline, pandemic trough/peak, and days to recovery per sector.

Source
Underlying job postings data is from Indeed Hiring Lab’s Job Postings Tracker, licensed CC BY 4.0. I built and included an original sector-to-SOC-2018-major-group crosswalk (Indeed’s category system doesn’t map 1:1 to the standard occupational taxonomy, so this fills that gap) — that mapping is my own work, not part of Indeed’s data.

Size/scope
44 occupational sectors, all 51 states (incl. DC), daily observations from Feb 1, 2020 through today. National, sector-level, and state-level files, plus the pre-computed summary stats file.

How it was built
Fully scripted pipeline (Python/pandas) — one command downloads the raw data and rebuilds every output file from scratch. Nothing manually edited or copy-pasted, so it’s reproducible and easy to refresh later since Indeed updates their source weekly.

Links Dataset: https://www.kaggle.com/datasets/adizz321/us-job-market-shift-since-2020-postings-by-sector Notebook with visualizations (recovery-speed chart, sector winners/losers, state-level breakdown): https://www.kaggle.com/code/adizz321/the-job-market-inversion-what-changed-since-2020

submitted by /u/Firm_Ad9420
[link] [comments]

Is There Any Dataset For Human Detection With OBB Annotations?

Is there any dataset for human detection with OBB annotations?

I’m doing my program with yolo and it’s about human detection with obb and i’m a beginner. But i can’t find dataset to train. Can only find human&hbb, and all those obb ones i saw are for vehicles or sth like that.

So does anyone know about this?

And also, i found a HIT-UAV dataset, but it’s thermal dataset.Though i thought about using this and just do a thermal one and saw the rotation part. But no matter how i tried to fix, it’s still not obb dataset and error everytime…🤔

submitted by /u/MindyOctant
[link] [comments]

Dataset: 18,208 Salary Changes Made To Job Postings After They Went Live

collected by hitting the public json endpoints for greenhouse, ashby and lever across 998 companies, once a day, and diffing each posting against the previous snapshot. no auth needed, these are the boards the companies serve to their own careers pages.

what the diff has caught so far:
salary_changed 18,208
department_changed 15,188
closed 4,357
title_changed 700
location_changed 364
reopened 98

51,061 postings total, 46,761 currently open. 22,820 of the open ones carry a pay range, which is 48.8%.

two things worth knowing if you want to do this yourself. greenhouse has no structured salary field at all, the range is written into the description html and it is double escaped, so a naive parse gets you nothing and a slightly less naive one gets you garbage. and a dollar sign is not a currency, i nearly recorded a taipei role in TWD as a $700k job.

the salary_changed rows are the ones i’d point at. nobody keeps the before, so as far as i know this is the only place the previous band exists once the company overwrites it.

built off a job tracker i run, link in the comments if the method is useful to anyone.

submitted by /u/Significant_Camp4148
[link] [comments]

[Self-promotion] Free Chest-worn Human Activity Dataset: 180 Recordings, 518,300 Samples At 50 Hz

Disclosure: I’m one of the people behind Aidlab.

If you’re working on human activity recognition, exercise classification, or repetition counting, we’ve published AIDLAB-HAR on Hugging Face under CC BY 4.0.

It contains 180 chest-worn recordings and 518,300 samples at 50 Hz. The signals include 3-axis acceleration, orientation quaternions, 16 activity labels, and repetition markers.

We packaged it into three configurations that work directly with the datasets library: recordings, signals, and annotations. The raw v3 archive, cleaning notes, and preparation script are included too.

One detail we found while preparing the release: the paper’s overview mentions 15 activities, while the archive contains 16 filename labels. We documented the discrepancy and preserved the source labels.

Dataset: https://huggingface.co/datasets/aidlab-wearables/AIDLAB-HAR

Paper: https://doi.org/10.3390/s24123891

submitted by /u/Guzikk
[link] [comments]