I am looking for new vehicle data at the state (or zip code) x year (or month) x vehicle make. In particular, I am interested in the count of vehicle lease or buy at the level. It does not have to recent. A few years or historical data is fine.

I’m working on a multimodal machine learning pipeline that combines image data with structured/genomic-like data for prediction task. I’m looking for publicly available datasets where MRI/Image data and Genomic/Structured data are explicitly paired for the same individual/subject. My ideal scenario would be human cancer (like Glioblastoma Multiforme, where I know TCGA exists), but given recent data access changes (e.g., TCIA policies), I’m open to other domains that fit this multimodal structure:

What I’m looking for (prioritized):

Human Medical Data (e.g., Cancer): MRI/Image: Brain MRI (T1, T1Gd, T2, FLAIR). Genomic: Gene expression, mutations, methylation. Crucial: Data must be for the same patients, linked by ID (like TCGA IDs).

I’m aware of TCGA-GBM via TCIA/GDC, but access to the BraTS-TCGA-GBM imaging seems to be undergoing changes as of July 2025. Any direct links or advice on navigating the updated TCIA/NIH Data Commons policies for this specific type of paired data would be incredibly helpful.

Animal Data:

Image: Animal MRI, X-rays, photos/video frames of animals (e.g., for health monitoring, behavior).

Genomic/Structured: Genetic markers, physiological sensor data (temp, heart rate), behavioral data (activity), environmental data (pen conditions), individual animal ID/metadata.

Crucial: Paired for the same individual animal.

I understand animal MRI+genomics is rare publicly, so I’m also open to other imaging (e.g., photos) combined with structured data.

Plant Data:

Image: Photos of plant leaves/stems/fruits (e.g., disease symptoms, growth).

Structured: Environmental sensor data (temp, humidity, soil pH), plant species/cultivar genetics, agronomic metadata. Crucial: Paired for the same plant specimen/plot.

I’m aware of PlantVillage for images, but seeking datasets that explicitly combine images with structured non-image data per plant.

What I’m NOT looking for:

Datasets with only images or only genomic/structured data.

Datasets where pairing would require significant, unreliable manual matching.

Data that requires extremely complex or exclusive access permissions (unless it’s the only viable option and the process is clearly outlined).

Any pointers to specific datasets, data repositories, research groups known for sharing such data, or advice on current access methods for TCGA-linked imaging would be immensely appreciated!

Thank you!

Looking for a list of 3,000 Shopify store owners based in India. Need basic contact info (email + first name + last name + mobile).

Payment: UPI/PhonePe/Gpay

Just need fresh, real contacts of active Shopify stores operating in India.

Fast deal if the data is legit and clean.

If you already have such a list or can source it quickly, feel free to DM me. Happy to close this ASAP.

Hey r/datasets,

Built a PDF table extraction tool for my own analysis work. Got tired of copying data by hand when creating datasets. The breaking point was a 250-page quarterly report where all the tables were screenshots.

Trained it on 100 million table cells from public datasets (FinTabNet, TableBank, PubTables-1M, WebTables, etc). Now it pulls structured data from PDFs that typically require manual extraction. Academic papers with supplementary data tables, government statistical reports, historical documents with scanned tables, handwritten edits, corporate filings with embedded data. Straight into Excel/CSV. No merged cells. No cleanup. Just structured data ready for analysis.

So now I’m here trying to understand how this fits into dataset creation workflows beyond my own use case.

The tool: https://sheetops.io

The challenge: People like the results, but I need to understand how this fits into data collection pipelines. While many datasets exist pre-structured, tons of valuable data is still locked in PDFs. Right now I’ve got a solid engine that needs to fit where data professionals actually work.

Here’s what I’m hoping to learn:

* What types of data are you extracting from PDFs for datasets?

* How do you currently handle PDF table extraction? (Manual, crowdsourcing, other tools?)

* What format do you need the output in? (CSV, JSON, direct to database?)

* What would make this worth integrating into your data pipeline?

The tool handles things most extractors fail on. Tables split across pages, rotated scanned documents, complex nested structures, handwritten data collection forms. Started with English docs, now supports 70+ languages for international data collection.

I’m offering free processing for anyone willing to share their dataset creation workflow. Built it for myself, but want it to work for the data community.

Would love your feedback. Fire away.

Hello r/datasets! I am trying to retrieve data from Chinese authors published on Baidu, but unfortunately, I found out that I cannot access the website from Europe. Neither do any of the authors respond to my request. I know there are mirror sites such as Baidu Erranium available, but it is a bit intransparent who is behind them. I was therefore wondering whether any of you have figured a way out on how to safely retrieve data from Baidu?

Hello everyone. I am working on a paper currently, for which I need access to Bloomberg’s ESG Disclosure Scores for companies in the NIFTY50 index for the years 2016 to 2025. I just need the company name, Bloomberg ticker, and the ESG disclosure score.

Unfortunately, my institution doesn’t have access to a Bloomberg Terminal, and of course, it is not affordable for me. If anyone here (student, researcher, or finance professional) has access through their employer, institution or any other way, and can help me with this, I would be extremely grateful.

I want to clarify that this is purely for academic purposes. If you’re willing to help or can guide me, please DM or comment. Thank you in advance 🙏

Hi all — I’m developing a project focused on mapping emotional drift, tone arcs, and symbolic resonance across time in text (e.g., journals, interviews, dialogue, narratives). It’s an experimental system designed to simulate how emotional memory and narrative coherence evolve — including decay, rebound, and symbolic shifts.

I’m looking for public or open datasets that include:

• Emotion or sentiment annotations (even basic: joy/sadness/anger/etc.)
• Time-sequenced or multi-turn data (dialogue, diaries, long-form text)
• Any datasets involving metaphor, archetype, or tone transition labeling
• Reddit threads, interview logs, or scripted conversations welcome

This is currently an open exploratory project, though I may pursue formal publication or applied use down the line. I’m not seeking commercial leads—just trying to find relevant data to push the theory forward.

Thanks in advance for any suggestions!

I am fairly new to ML and I’ve been wanting to fine tune a model (T5-base/large) with my own dataset. There are a few problems i’ve been encountering:

1. Writing a script to scrape different websites but it comes with a lot of noise.

2. I need to write a different script for different websites

3. Some data that are scraped could be wrong or incomplete

4. I’ve tried manually checking a few thousand samples and come to a conclusion that I shouldn’t have wasted my time in the first place.

5. Sometimes the script works but a different html format in the same website led to noise in my samples where I would not have realised unless I manually go through all the samples.

Solutions i’ve tried:
1. Using ChatGPT to generate samples. (The generated samples are not good enough for fine tuning and most of them are repetitive.)

1. Manually adding sample (takes fucking forever idk why I even tried this should’ve been obvious, but I was desperate)

2. Write a mini script to scrape from each source (works to an extent, I have to keep writing a new script and the data scraped are also noisy.)

3. Tried using regex to clean the data but some of them are too noisy and random to properly clean (It works, but about 20-30% of the data are still extremely noisy and im not sure how i can clean them)

4. I’ve tried looking on huggingface and other websites but couldn’t exactly find the data im looking for and even if it did its insufficient. (tbf I also wanted to collect data on my own to see how it works)

So, my question is: Is there any way where I am able to get clean data easier? What kind of crawlers/scripts I can use to help me automate this process? Or more precisely I want to know what’s the go to solution/technique that is used to collect data.

Hi,

i am looking for news apis that provide the full content of the news with good coverage of german/austrian news.

anyone knows a good source?

Are you using event-driven setups with Kafka or something similar, or full real-time streaming?

Trying to figure out if real-time data setups are actually worth it over event-driven ones. Event-driven seems simpler, but real-time sounds nice on paper.

What are you using? I also wrote a comparing them (it is in the comments), but still I am curious.

Hello,

I need SMS data related to delivery time OTP…., I am creating a small tool which forwards sms(otp) to a family member, when one is not home.

i want SMS data to classify which SMS have OTP at the time of delivery

You can comment if you want to help….

(You need not to give the real OTP, I am interest in the Pattern of the message)

Hey everyone I’m looking for Nike sales predictions datasets for my class project, I looked everywhere online, do anyone have any clue?

I know business insider has one, but everything else is a pdf from the handwritten log. Thank you!

Hey everyone,

I made a tool called TableDrip. It lets you pull tables out of PDFs and export them to CSV, Excel, or JSON fast.

If you’ve ever had to clean up tables from PDFs just to get them into a usable format for analysis or ML, you know how annoying that is. TableDrip handles the messy part so you can get straight to the data.

Would love to hear any feedback or ideas to make it better for real-world workflows.

Does anyone know of any source to find “passes” by fighter or fight? I’ve looked at all of the stat sites and datasets that people have already put together and can’t seem to find this anywhere. I know ufcstats had it years ago and then removed it and now keep it under wraps.

Hi everyone,
I’m trying to find a dataset that contains first names by country, ideally sorted by popularity or frequency – something similar to what census.name offers (they have a paid database of 1.5M+ names across 200+ countries).

Does anyone know of:

• A free alternative
• A mirror or archived version of the census.name database
• Or any large dataset with realistic global first names?

Open to Kaggle, GitHub, or even academic/public resources.
Thanks in advance for any leads!

Sharing original research recently collected by a quant + qual survey of 1,000 consumers nationwide (US) trying to better understand current consumer sentiment, and how consumer spending habits have or have not changed in the past year due to things like inflation/shrinkflation, tariff concerns, higher cost of living and more.

In a Highlight survey taken the week of July 7, 2025, we polled our proprietary panel of nationwide consumers, achieving 1,000 completions with an even gender split (500 men and 500 women).

Among other questions, we asked them: In terms of your personal finances, how do you feel today compared with this time last year?

62% of respondents said money feels somewhat or much tighter than a year ago, while only 10% said money feels somewhat or much easier than a year ago. Over a quarter of respondents (28%) say that money feels about the same as compared with this time last year.

In an open-ended question, respondents were given the opportunity to describe how their consumption habits and saving strategies have changed in their own words. Highlight asked: Thinking about your everyday routines, purchases, or habits–is there anything you’re doing now that you weren’t doing a year ago? Here’s the full breakdown of respondents’ qualitative responses:

No/Not really: This or similar phrases like “Nope it’s the same,” “No changes,” “nothing,” “I don’t think so,” or “everything is basically the same” appears 93 times. This indicates a significant portion of the respondents haven’t changed their habits much.

“I shop the same overall.” – She/her, 47 years old, North Carolina

Exercising more/Working out more: This theme appears 47 times. Many respondents mentioned exercising, working out, going to the gym, walking more, or increasing physical activity.

“Drinking more iced coffee, working out more, traveling less, reading audiobooks more.” – He/him, 36 years old, Illinois

Eating healthier/Better food choices: This theme appears 39 times. Responses include eating healthier, eating more vegetables, focusing on protein, buying organic, or making healthier food choices.

“I’m eating better. I’m putting better stuff in my body. I’m working out more. Also I’m buying different things that I need for a healthier life.” – He/him, 43 years old, Texas

Budgeting/Saving money/More conscious of spending/Looking for sales: This broad category appears 65 times. Many people are trying to save money, be more budget-conscious, look for sales, use coupons, or buy less.

“[I’m] budgeting better. Picked up a second job.” – He/him, 39 years old, Tennessee

Shopping online more: This response appears 25 times.

“I visit Sam’s Club more often for bulk purchases and savings. I also shop online more frequently for pick up or shipped items from CVS.” – She/her, 61 years old, Florida

Cooking more/Eating at home more: This theme appears 14 times.

“I’m watching my money more as things get more expensive. We’re also eating out less as restaurant prices have risen tremendously.” – She/her, 58 years old, Pennsylvania

In this same Highlight survey of 1,000 Americans, we also asked respondents: What are you doing to better manage your spending?

In a multiple choice question where respondents were invited to select all that apply, this is how panelists responded, from most popular to least popular responses:

• 67% of respondents are eating at home more often
• 57% are shopping sales more actively
• 55% are buying fewer non-essential products
• 54% are holding off on major purchases (e.g., tech, furniture)
• 43% are avoiding eating out
• 39% are switching to more affordable brands
• 33% are canceling subscriptions
• 32% are traveling less
• 30% are choosing private label/store brands
• 29% are buying in bulk
• 23% are using budgeting apps or tracking spending more closely
• 17% are cutting back on wellness and/or beauty spending
• 9% said none of the above

In a multiple choice question, Highlight asked respondents: Which of the following, if any, are you not willing to sacrifice–even when budgets are tight? (Select up to three.) These were their answers, from most to least popular:

• 42% of respondents are not willing to give up high-quality food & beverages
• 39% say they are not willing to give up their self-care and wellness routines
• 31% don’t want to give up their streaming services or other entertainment
• 30% say they won’t part with their preferred brands
• 29% won’t give up travel or experiences
• 23% said they won’t give up products that make them feel good or confident
• 15% said they won’t give up conveniences like delivery
• 7% said they won’t give up products that support sustainability of ethics

Highlight also gave respondents the opportunity to say what habits they are not willing to change or products they are not willing to give up in their own words.

Overall, the qualitative results mirrored the quantitative: Consumers mentioned over and over again that they are unwilling to give up buying food, especially healthy, quality, or favorite foods.

While respondents across genders agreed high-quality food is their non-negotiable item, women most frequently mentioned their unwillingness to give up coffee specifically. Their open-ended responses mentioned iced coffee, Starbucks, Dunkin, “good coffee,” “homemade coffee,” and other specific brands.

“I MUST have my favorite coffee even though it’s more expensive even now.” – She/her, 61 years old, Iowa

Women respondents were also more likely to mention these topics in their open-ended answers:

• Specifically, healthy food was mentioned approximately 40 times, often paired with words like “quality,” “organic,” and “produce.”
• Personal care and self-care purchases were mentioned approximately 30 times, including terms like manicures, skincare, hair care, beauty, and nails.
• Pets and pet products (dog food, cat food, vet care, pet supplies and more) were mentioned approximately 30 times.

“I still buy extra healthy food. The healthier the food, the more it will cost. I will not buy cheap food.” – She/her, 66 years old, Arizona

“Hair color and nail appointments.” – She/her, 55 years old, Texas

“My dog’s food and heartworm medication. I will always make sure to buy her the good healthy food she is on and make sure she has her heartworm medication to take each month.” – She/her, 25 years old, Florida

Male respondents also placed a premium on high-quality food and eating well. When it comes to themes that were repeated most frequently in their open-ended responses, nothing else came close to quality food, which was mentioned upwards of 60 times.

“I will still purchase organic produce and look for items that are healthier.” – He/him, 43 years old, Arizona

But when we look at the honorable mentions, a few stand out:

• Men do not want to part with their streaming services, television, and other entertainment (mentioned approximately 20 times)
• Men also mentioned travel, vacations, and getaways as a non-negotiable (mentioned approximately 20 times)
• Men mentioned not wanting to give up purchases that support a healthy lifestyle (eating, gym, working out), but mentioned this less frequently than female respondents did (approximately 15 times versus 40 for women)

“I pay for a number of TV streaming services that I would feel deprived not to have.” – He/him, 55 years old, Texas

“My grocery bill and gym membership.” – He/him, 47 years old, Oregon

“We still go on trips and vacations.” – He/him, 50 years old, New York

“My kid’s favorite snack: She loves Takis. They’re a bit expensive but I give up things for her. She is all that matters.” – He/him, 40 years old, North Carolina

Original source

A gold mine of info on optimizing Parquet: https://huggingface.co/blog/parquet-cdc

Here is the idea: chunk and deduplicate your data and you will speed up uploads and downloads

Hugging Face uses this to speed up data workflows on their platform (they use a dedupe-based storage called Xet).

Pretty excited by this. It looks like it can really speed up data workflows, especially operations like append/delete/edit/insert. Happy to have this enabled for Hugging Face where the AI datasets community is amazing too. What do you think ?

I have a Statista subscription. Reach out if you need access to any data or report. I can help you for a cheap price!

I’m a researcher working on model-agnostic meta-learning (MAML) for personalized music recommendation. I urgently need access to either the LFM‑2b or LFM‑1b dataset, which used to be hosted by JKU Linz but has since been removed due to licensing constraints.

I’ve already checked Kaggle, GitHub, Zenodo, and official sources, no mirrors exist.

If anyone has a copy and is willing to share (for research use only), please DM me or point me to a working archive/mirror.
Alternatively, any help with locating subsets or working alternatives would also be appreciated.

Thanks in advance.

https://tinyurl.com/3can6b4y

I’ve been working on a few projects recently where I needed structured data from e-commerce and social media sites (like prices, product descriptions, user reviews, etc.). I used to rely on my own scrapers with BeautifulSoup or Scrapy, but as you know, many sites now have rate-limiting, bot detection, or constantly changing layouts.

Lately, I’ve experimented with Bright Data to access web data from different regions/IPs — mostly for testing, not large-scale production. It worked surprisingly well, but I’m curious:

🔹 What sources or services are you all using when you need consistent or hard-to-access datasets from the web?

🔹 Any experiences with open APIs, rotating proxies, or maybe even public datasets that saved you a ton of work?

Would love to hear your approach, especially for projects where the public datasets don’t quite cut it.

I have never tried to train an ai model before .I need some datasets on car sounds and images ,damaged and good .this is for a personal project. Also any advice on how to approach this field 😅?

Hi, everyone. which is a good free tool for Dataset Analizer?

Finishing a data visualization class and I need to find two separate, but related data sets. One has to have at least 300 records and 4 fields, the other has to have 100 records and 3 fields. I have to show something happening over time, and a geographical component. I’ve been searching for hours and am obviously not creative enough. Any help is deeply appreciated.

Preferably categorically divided on the level of sleep debt or number of hours.

Would appreciate it, as I have not been able to find any at all which are publicly available.

I am not looking for fatigue detection datasets as mainly that is what I have found.

Thanks so much!

I deal in import-export data and have direct sources with customs, allowing me to provide accurate and verified data based on your specific needs.

You can get a sample dataset, based on your product or HSN code. This will help you understand what kind of information you’ll receive. If it’s beneficial, I can then share the complete data as per your requirement—whether it’s for a particular company, product, or all exports/imports to specific countries.

This data is usually expensive due to its value, but I offer it at negotiable prices based on the number of rows your HSN code fetches in a given month

If you want a clearer picture, feel free to dm. I can also search specific companies—who they exported to, what quantity, and which countries what amount.

Let me know how you’d like to proceed, lets grow our business together.

I pay huge yearly fees for getting the import export data for my own company and thought if I could recover a small bit by helping others. And get the service in a winwin

I am trying to train a T5 model to be able to learn and generate Data Structure questions but I am not sure if the format of the data I scraped is correctly formatted. I’ve trained it without context and its generating questions that are barebones or not properly formatted and it is also not generating questions that make sense. What do I need to do to fix this problem?

Im training my model with this code:

from transformers import T5ForConditionalGeneration from transformers import T5Tokenizer from transformers import Seq2SeqTrainer, Seq2SeqTrainingArguments from datasets import Dataset import json def main(): global tokenizer with open('./datasets/final.json', 'r', encoding='utf-8') as f: data = json.load(f) dataset = Dataset.from_list(data) dataset = dataset.train_test_split(test_size=0.1) tokenizer = T5Tokenizer.from_pretrained("google/flan-t5-base") model = T5ForConditionalGeneration.from_pretrained("google/flan-t5-base") tokenized = dataset.map(tokenize, batched=True) tokenized_train = tokenized["train"].shuffle(seed=42) tokenized_eval = tokenized["test"].shuffle(seed=42) training_args = Seq2SeqTrainingArguments( output_dir="./outputs_T5", per_device_train_batch_size=4, per_device_eval_batch_size=4, num_train_epochs=10, save_strategy="epoch", learning_rate=5e-5, predict_with_generate=True, logging_dir="./logs_bart", ) trainer = Seq2SeqTrainer( model=model, args=training_args, train_dataset=tokenized_train, eval_dataset=tokenized_eval, tokenizer=tokenizer, compute_metrics=compute_metrics ) trainer.train() eval_results = trainer.evaluate() print(eval_results) def compute_metrics(eval_preds): predictions, labels = eval_preds decoded_preds = tokenizer.batch_decode(predictions, skip_special_tokens=True) decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True) exact_matches = sum(p.strip() == l.strip() for p, l in zip(decoded_preds, decoded_labels)) return {"accuracy": exact_matches / len(decoded_preds)} def tokenize(examples): global tokenizer model_inputs = tokenizer(examples["input_text"], max_length=128, truncation=True, padding="max_length") with tokenizer.as_target_tokenizer(): labels = tokenizer(examples["target_text"], max_length=128, truncation=True, padding="max_length") model_inputs["labels"] = labels["input_ids"] return model_inputs if __name__ == "__main__": main() 

and heres how my dataset currently looks like

{ "input_text": "Topic: GraphnDifficulty: EasynContext: The kth power of an adjacency matrix gives the number of paths of length k between any two vertices in a graph. Each entry A^k[i][j] equals the number of such paths from i to j.nTask: Generate a multiple-choice question on the given topic and difficulty using the provided context.", "target_text": "Let A be an adjacency matrix of a graph G. The ijth entry in the matrix AK , gives, , Choices: ['A\nThe number of paths of length K from vertex Vi to vertex \n Vj.', 'B\nShortest path of K edges from vertex Vi to vertex Vj.', 'C\nLength of a Eulerian path from vertex Vi to vertex Vj.', 'D\nLength of a Hamiltonian cycle from vertex Vi to vertex \n Vj.'], Answer: BnShortest path of K edges from vertex Vi to vertex Vj." }, { "input_text": "Topic: TreenDifficulty: EasynContext: In an AVL tree, after inserting a node, the balance factor of nodes along the path to the root may need to be updated. This ensures the tree remains balanced.nTask: Generate a multiple-choice question on the given topic and difficulty using the provided context.", "target_text": "To restore the AVL property after inserting a element, we start at the insertion point and move towards root of that tree. is this statement true?na) truenb) falsennnAnswer: a" }, { "input_text": "Topic: TreenDifficulty: EasynContext: AA-Trees and Red-Black Trees are both self-balancing binary search trees. They have similar properties and performance characteristics.nTask: Generate a multiple-choice question on the given topic and difficulty using the provided context.", "target_text": "Which of the following trees is similar to that of an AA-Tree?na) Splay Treenb) B+ Treenc) AVL Treend) Red-Black TreennnAnswer: d" }, { "input_text": "Topic: TheorynDifficulty: EasynContext: In hashing theory, probe sequences like linear and quadratic probing determine how collisions are resolved. Expression evaluation and conversion also fall under theory topics, such as converting infix to postfix using stacks.nTask: Generate a multiple-choice question on the given topic and difficulty using the provided context.", "target_text": "What would be the Prefix notation for the given equation?nna) ^^^ABCDnb) ^A^B^CDnc) ABCD^^^nd) AB^C^DnnAnswer: b" }, { "input_text": "Topic: TheorynDifficulty: EasynContext: Linked list manipulations require careful updates of pointers. The given code removes the first node in a circular list and returns its value.nTask: Generate a multiple-choice question on the given topic and difficulty using the provided context.", "target_text": "What is the functionality of the following code? Choose the most appropriate answer.nnpublic int function() {n if(head == null) return Integer.MIN_VALUE;n int var;n Node temp = head;n while(temp.getNext() != head) temp = temp.getNext();n if(temp == head) {n var = head.getItem();n head = null;n return var;n }n temp.setNext(head.getNext());n var = head.getItem();n head = head.getNext();n return var;n}nna) Return data from the end of the listnb) Returns the data and deletes the node at the end of the listnc) Returns the data from the beginning of the listnd) Returns the data and deletes the node from the beginning of the listnnAnswer: d" }, { "input_text": "Topic: ArraynDifficulty: EasynContext: Breadth First Traversal (BFS) is implemented using a queue. This data structure allows level-order traversal in graphs or trees.nTask: Generate a multiple-choice question on the given topic and difficulty using the provided context.", "target_text": "The data structure required for Breadth First Traversal on a graph is?na) Stacknb) Arraync) Queuend) TreennnAnswer: c" },