I’ve been searching for about 2 hours for specific data regarding poultry exports from the US to either Europe in general or Germany specifically. I am looking for the years 1960-1970, more specifically 1962, 63, and 64 which seem to be unfindable. I’ve found this for 1961 on AgEcon but I can’t find past that. I also have found it for 1967 and onwards but again have the gap in the years I specifically need. I am able to find this for poultry broiler/young chicken exports in pounds, which is helpful, but not in the dollar amount that I need. Any ideas where to look further?

I am new to this. Extremely new to this. I’m working on a university capstone project that requires coding news headlines to compare trends in content with some other thing that’s unimportant right now.

I’ve been trying to figure out a way to scrape headlines from local news outlets (ABC 7, FOX 5, NY Post, etc— I’m not picky lol) from 2021 to 2024 (or any year within those, I’m more than happy to reduce the scope). I had some luck with scraping a month’s worth of daily headlines in 2024 of ABC 7 using Internet Archive, but it didn’t translate over well to NBC 4 or CBS 2. And IA can be finicky with taking lots of data.

Basically I’m trying to find major headlines from local news outlets daily, at about 9 AM EST, from 2021 – 2024. I’m okay with getting creative. Any suggestions or ideas??

eta: i do know the NYT API

Hi everyone,
I’m a Ph.D. student currently working on a funded project with my advisor using PRAMS data.

I applied through the PRAMS website, and after getting approved, I was only able to download the Phase 8 dataset without the core file. Unfortunately, my account was later blocked for some reason.

Since then, I’ve been in contact with the PRAMS data manager, but it’s already been over three months without resolution. I completely understand that they may be dealing with internal issues and it’s not necessarily their fault.

That said, the deadline for our project’s progress report is fast approaching, and I can no longer afford to just wait for a response.

If anyone has previously downloaded the Phase 8 data with the core file, or knows of any way to access it, I’d deeply appreciate it if you could share or point me in the right direction.

Thank you so much in advance and I really hope everything gets back to normal soon.

Can anyone recommend where to find datasets with genetics data which are suitable for PCA (like studying haplogroups or similar)? Any recommendations are appreciated.

Try to see if it makes sense to do this project or if it is not worth it.

I’m sharing a unique dataset of 12,000 IQ test participants collected via my own test platform. It includes First Name, Last Name, Email, and Score.
This was collected with full user consent.
If you’re interested in using it for academic or research purposes, feel free to DM me.

Tittle says it. I’m looking for images that I can use to train my model on. Any help would be appreciated.

I have a huge database of indian contact numbers and I’m looking forward to sell them some of the contacts are random but I also have niche contact numbers like from specific housing societies or govt. officials, how and to whom should I sell these to I don’t know much people and any leads will be appreciated.

years ago (before the current generative AI wave) I’d seen this person start a website for crowdsourced image annotations, I thought that was a great idea so I tried to support by becoming a user, when I had spare moments I’d go annotate. Killed a lot of time doing that during pandemic lockdowns etc. There around 300,000 polygonal outlines here accumulated over many years. to view them you must search for specific labels ; there’s a few hundred listed in the system and a backlog of new label requests hidden from public view. there is an export feature

https://imagemonkey.io

example .. roads/pavements in street scenes (“rework” mode will show you outlines, you can also go to “dataset->explore” to browse or export)

https://imagemonkey.io/annotate?mode=browse&view=unified&query=road%7Cpavement&search_option=rework

it’s also possible to get the annotations out in batches via a python API

https://github.com/ImageMonkey/imagemonkey-libs/blob/master/python/snippets/export.py

i’m worried the owner might get disheartened from a sense of futility (so few contributors, and now there are really powerful foundation models available including image to text)

but I figure “every little helps”, it would be useful to get this data out into a format or location where it can feed back into training, maybe even if it’s obscure and not yet in training sets it could be used for benchmarking or testing other models

When the site was started the author imagined a tool for automatically fine-tuning some vision nets for specific labels, I’d wanted to broaden it to become more general. the label list did grow and there’s probably a couple of hundred more that would make sense to make ‘live’

There’s also an aspect that these generative AI models get accused of theft, so the more deliberate voluntary data there is out there the better. I’d guess that you could mix image annotations somehow into the pretraining data for multimodal models, right? I’m also aware that you can reduce the number of images needed to train image-generators if you have polygonal annotations aswell as image/descriptions-text pairs.

Just before the diffusion craze kicked off I’d had some attempts at trying to train small vision nets myself from scratch (rtx3080) but could only get so far. When stable diffusion came out I figured my own attemtps to train things were futile.

Here’s a thread where I documented my training attempt for the site owner

https://github.com/ImageMonkey/imagemonkey-core/issues/300 – in here you’ll see some visualisations of the annotations (the usual color coded overlays)

I think these labels today could be generalised by using an NLP model to turn the labels into vector embeddings (cluster similar labels or train image to embedding, etc)

The annotations would probably want to be converted to some better known format that could be loaded into other tools. they are available in his json format.

can anyone advise on how to get this effort fed back into some kind of visible community benefit?

Hey everyone!

I know it’s been a long minute since my original call‑for‑clips – life got hectic and the project had to sit on the back burner a bit longer than I’d hoped. 😅 Thanks for bearing with me!

What’s new?

• The dataset is live on Hugging Face and ready for download or contribution.
• First models are on the way (starting with build‑plate identification) – but I can’t promise an exact release timeline yet. Life still throws curveballs!

🔗 Dataset page: https://huggingface.co/datasets/v2thegreat/bambu-timelapse-dataset

What’s inside?

• 627 timelapse videos from P1/X1 printers
• 81 full‑length camera recordings straight off the printer cam
• Thumbnails + CSV metadata for quick indexing
• CC‑BY‑4.0 license – free for hobby, research, and even commercial use with proper attribution

Why bother?

• It’s the first fully open corpus of Bambu timelapses; most prior failure‑detection work never shares raw data.
• Bambu Lab printers are everywhere, so the footage mirrors real‑world conditions.
• Great sandbox for manufacturing / QA projects—failure classification, anomaly detection, build‑plate detection, and more.

Contribute your clips

1. Open a Pull Request on the repo (originals/timelapses/<your_id>/).
2. If PRs aren’t your jam, DM me and we’ll arrange a transfer link.
3. Please crop or blur anything private; aim for bed‑only views.

Skill level

If you know some Python and basic ML, this is a perfect intermediate project to dive into computer vision. Total beginners can still poke around with the sample code, but training solid models will take a bit of experience.

Thanks again for everyone’s patience and for the clips already shared—can’t wait to see what the community builds with this!

Was looking for datasets with nutrition content in mind and perhaps feed efficiency rate but now I realized I’m struggling to find any dataset related to egg size, shell hardness, and contents. I’m checking FSIS and USDA but most studies are focused around incidences of contamination and the like rather than product quality, perhaps due to only having “standards,” but that means they should have the data somewhere and I just can’t find it, right…? Please help 🙏

Hi all, I am looking for a dataset of classified pics of car repairs to help automate insurance claims. Thank you very much!

Been messing with automation + AI for over a year along side with my team and ended up building a system that scraped 300 million+ leads from LinkedIn. Used a mix of:

• Multiple Sales Nav accounts
• Rotating proxies & custom scripts
• Headless browsers & queue-based servers
• ChatGPT for data cleaning & enrichment

Honestly, the setup was painful at times (LinkedIn doesn’t play nice), but the results were wild. If you’re into large-scale scraping, lead gen, or just curious how this stuff works under the hood, happy to chat.

I packaged everything into a cleaned database way cheaper than ZoomInfo/Apollo if anyone ever needs it. It’s up at Leadady .com, one-time payment, no fluff.

Working on training a model for a hobby project.

Does anyone know of a newer available dataset of investment data in startups?

Thank you

Hi r/datasets,

I’m excited to share a new dataset I’ve created and uploaded to the Hugging Face Hub: Generated-Recovery-Support-Dialogues.

https://huggingface.co/datasets/filippo19741974/Generated-Recovery-Support-Dialogues

About the Dataset:

This dataset contains ~1100 synthetic conversational examples in English between a user discussing addiction recovery and an AI assistant. The AI responses were generated following guidelines to be empathetic, supportive, non-judgmental, and aligned with principles from therapeutic approaches like Motivational Interviewing (MI), ACT, RPT, and the Transtheoretical Model (TTM).

The data is structured into 11 files, each focusing on a specific theme or stage of recovery (e.g., Ambivalence, Managing Negative Thoughts, Relapse Prevention, TTM Stages – Precontemplation to Maintenance).

Format:

JSONL (one JSON object per line)

Each line follows the structure: {“messages”: [{“role”: “system/user/assistant”, “content”: “…”}]}

Size: Approximately 1100 examples total.

License: MIT

Intended Use:

This dataset is intended for researchers and developers working on:

Fine-tuning conversational AI models for empathetic and supportive interactions.

NLP research in mental health support contexts (specifically addiction recovery).

Dialogue modeling for sensitive topics.

Important Disclaimer:

Please be aware that this dataset is entirely synthetic. It was generated based on prompts and guidelines, not real user interactions. It should NOT be used for actual diagnosis, treatment, or as a replacement for professional medical or psychological advice. Ethical considerations are paramount when working with data related to sensitive topics like addiction recovery.

I hope this dataset proves useful for the community. Feedback and questions are welcome!

Does anyone know where I can find a person level data-set for anything health related?

Hi everybody!

I am currently building a model that analyze the customer service calls and evaluate the agents for my college class. I wonder what is the most well-known, free, recommended datasets to use for this? I am currently looking for test data for model evaluations.

We are very new with the model training and testing so please drop your recommendations below..

Thank you so much.

This is my idea for helping AI deal with contradiction and paradox and judge not deterministic truth.

from datasets import load_dataset

ds = load_dataset(“AmarAleksandr/LudusRecursiveV5”)

https://huggingface.co/datasets/AmarAleksandr/LudusRecursiveV5/tree/main

Any feedback, even if it’s “this sucks and is nothing” is helpful.

Thank you for your time

Hi, for a course I am required to find and pick a raw and unprocessed dataset with a minimum of 1 million records, another constraint that I have is that this data needs to be tabular. Additionally, The data set should not be an already fully processed data product. Good examples of raw and unprocessed data are JSON/XML files from the web. These records can’t immediately be put into a structured table without processing.

The goal for me is to turn the unprocessed source into a data product, and example that was given: Preparing Wikipedia data dumps so that they can be used for graph query processing.

So far I have been browsing the following two resources:

• https://github.com/awesomedata/awesome-public-datasets
• https://huggingface.co/datasets

I am looking for additional sources for potential datasets, and tips or hints are welcome!

What can i say?

Please also see if you can help at r/datahoarders

Details of all spending by the council over £500. Already contains 123 CSV files – spending data since 2010. Updated regularly by the council.

I’m currently working on an AI project focused on architecture and need access to plans for properties such as plots, apartments, houses, and more. Could anyone assist me in finding an open-source dataset for this purpose? If such a dataset isn’t available, I’d appreciate guidance on how to gather this data from the internet or other sources.

Your insights and suggestions would be greatly appreciated!

Hi everyone,

I’m currently working on my final project titled “The Evolution of Social Media Engagement: Trends Before, During, and After the COVID-19 Pandemic.”

I’m specifically looking for free datasets that align with this topic, but I’ve been having trouble finding ones that are accessible without high costs — especially as a full-time college student. Ideally, I need to be able to download the data as CSV files so I can import them into Tableau for visualizations and analysis.

Here are a few research questions I’m focusing on:

1. How did engagement levels on major social media platforms change between the early and later stages of the pandemic?
2. What patterns in user engagement (e.g., time of day or week) can be observed during peak COVID-19 months?
3. Did social media engagement decline as vaccines became widely available and lockdowns began to ease?

I’ve already found a couple of datasets on Kaggle (linked below), and I may use some information from gs.statcounter, though that data seems a bit too broad for my needs.

If anyone knows of any other relevant free data sources, or has suggestions on where I could look, I’d really appreciate it!

Kaggle dataset 1

Kaggle Dataset 2

We scraped the Shopify GraphQL docs with code examples so you can experiment with codegen. Enjoy!

https://github.com/lsd-so/Shopify-GraphQL-Spec

Hey everyone,

Just wanted to share a Company Search API we built at my last company — designed specifically for autocomplete inputs, dropdowns, or even basic enrichment features when working with company data.

What it does:

• Input a partial company name, get back relevant company suggestions
• Returns clean data: name, domain, location, etc.
• Super lightweight and fast — ideal for frontend autocompletes

Use cases:

• Autocomplete field for company name in signup or onboarding forms
• CRM tools or internal dashboards that need quick lookup
• Prototyping tools that need basic company info without going full LinkedIn mode

Let me know what features you’d love to see added or if you’re working on something similar!

I have a web scraping task, but i faced some issues, some of URLs (sites) have HTML structure changes, so once it scraped i got that it is JavaScript-heavy site, and the content is loaded dynamically that lead to the script may stop working anyone can help me or give me a list of URLs that can be easily scraped for text data? or if anyone have a task for web scraping can help me? with python, requests, and beautifulsoup

I found a hard-to-come-by dataset that would be very useful a current project but I absolutely cannot afford Gigasheet’s pricing :/

Context: I have a dataset of company owned products like: Name: Company A, Address: 5th avenue, Product: A. Company A inc, Address: New york, Product B. Company A inc. , Address, 5th avenue New York, product C.

I have 400 million entries like these. As you can see, addresses and names are in inconsistent formats. I have another dataset that will be me ground truth for companies. It has a clean name for the company along with it’s parsed address.

The objective is to match the records from the table with inconsistent formats to the ground truth, so that each product is linked to a clean company.

Questions and help: – i was thinking to use google geocoding api to parse the addresses and get geocoding. Then use the geocoding to perform distance search between my my addresses and ground truth BUT i don’t have the geocoding in the ground truth dataset. So, i would like to find another method to match parsed addresses without using geocoding.

• Ideally, i would like to be able to input my parsed address and the name (maybe along with some other features like industry of activity) and get returned the top matching candidates from the ground truth dataset with a score between 0 and 1. Which approach would you suggest that fits big size datasets?

• The method should be able to handle cases were one of my addresses could be: company A, address: Washington (meaning an approximate address that is just a city for example, sometimes the country is not even specified). I will receive several parsed addresses from this candidate as Washington is vague. What is the best practice in such cases? As the google api won’t return a single result, what can i do?

• My addresses are from all around the world, do you know if google api can handle the whole world? Would a language model be better at parsing for some regions?

Help would be very much appreciated, thank you guys.