I’ve been scraping for the past week most of the data present in metal-archives website. I extracted 180k entries worth of metal bands, their labels and soon, the discographies of each band. Let me know what you think and if there’s anything i can improve.

https://www.kaggle.com/datasets/guimacrlh/every-metal-archives-band-october-2024/data?select=metal_bands_roster.csv

Before I try to figure out how to do a scraper for https://leginfo.legislature.ca.gov/faces/codes.xhtml I wanted to see if there is any downloadable dataset that includes California statutes (really only need Penal and Evidence Code)? Prefer a PDF but I’ll take anything.

DISCO-10M was removed by huggingface and wiped from the internet. I cannot find any other site than https://www.atyun.com/datasets/info/DISCOX/DISCO-10M.html?lang=en

which I cant signup for, I’ve tried a US number / UK number and a Chinese number

Im desperate yall. please help or dm if you hve anything

hi guys i’ve made this cool thing! go check it!

https://github.com/leocalle-swag/autolabel-tool

Hey guys,

I’ve built an AI tool to help people extract data from the web. I need to test my tool and learn more about the different use cases that people have, so I’m willing to extract web data for free for anyone that wants it!

as title

Anybody aware of public county-level 2024 presidential election results datasets, downloadable as CSV or accessible via free API? I’m specifically looking for total number of votes by county for each party.

HI everyone, I am searching for a dataset about homonal imbalance in women for a project. Data set may or should contain physical symptoms,age, height, weight, BMI, food habbit, hormonal test results and other clinical features. Thanks in advance.

I am still trying to understand how can I find proper datasets, everytime I need to look for something, I am lost. Any help highly appreciated! Thank you in advance.

I’m working on a small project to demonstrate the effects of T1 and T2 weighting on a PD-weighted image or a phantom image.

For example, I aim to recreate a T1 contrast between tissues on a PD image of the heart following the signal equation for MRI.

I’ve been searching for example pictures but haven’t had much luck. I’ve tried resources like the Cardiac Atlas Project, open-access papers, raw K-space data, and phantom images.

Does anyone have suggestions on where I might find what I need?

Hi all! I’m interested in finding APIs or open datasets that provide average travel costs for various destinations worldwide, including things like flight prices and daily expenses.

Ideally, I’m looking for options that cover multiple countries and can provide reasonable cost estimates for different types of travelers (budget, mid-range, etc.).

Any recommendations for APIs (like Skyscanner, Amadeus, etc.) or public datasets you’ve found useful? Also curious about any insights on pricing or request limits if you’ve worked with them. Thanks in advance!

I’ve been learning how to locally finetune and wanted to create a dataset that involve using my conversations I had with LLM’s like GPT and Claude. I know that dataset’s usually have an input output format and some variations of metadata and instructions along with it but how does one actually finetune data that requires previous context?

Like lets say initially my Chat would go somewhere in the lines like this:

Input: What is a bird?

Output: A bird is…

Input: Why do they fly?

Output: They fly because…

In this context the AI knows what I am referring to based on my previous input. But how would I implement the previous context on a dataset? Because the issue is that if I just include “Why do they fly?” as an isolated input, the model wouldn’t have the context about birds from the previous exchange and therefore assumes the input “Why do they fly?” have to associate generally with birds (possibly ignoring that the user could refer to a plane, etc..

I initially combine the previous output and the current input together but I feel like that method would only train the model to associate that previous output to be included with the input in order to get the current output. Another method was to nest the conversation spanning multiple input output pairs but utilizing that method wouldn’t be scalable since some of my conversations span 50 chats long.

Is there a much more efficient way for me to handle a dataset that utilizes previous context? The model I would be using to train for now is Llama 3.1 8b as it will be small enough to train fast and test if this dataset approach beneficial

Luxxify: Ulta Makeup Reviews

Hey everyone,

I recently released an open source dataset containing Ulta makeup products and its corresponding reviews!

Custom Created Kaggle Dataset via Webscraping: Luxxify: Ulta Makeup Reviews

Feel free to use the dataset I created for your own projects!

Webscraping Process

Web Scraping: Product and review data are scraped from Ulta, which is a popular e-commerce site for cosmetics. This raw data serves as the foundation for a robust recommendation engine, with a custom scraper built using requests, Selenium, and BeautifulSoup4. Selenium was used to perform button click and scroll interactions on the Ulta site to dynamically load data. I then used requests to access specific URLs from XHR GET requests. Finally, I used BeautifulSoup4 for scraping static text data. Leveraging PostgreSQL UDFs For Feature Extraction: For data management, I chose PostgreSQL so that I could clean the scraped data from Ulta. This data was originally stored in a complex JSON which needed to be unrolled in Postgres.

As an example, I made a recommender model using this dataset which benefited greatly from its richness and diversity.

To use the Luxxify Makeup Recommender click on this link: https://luxxify.streamlit.app/

I’d greatly appreciate any suggestions and feedback 🙂

Link to GitHub Repo

Hey data folks! I’ve put together an advent calendar of SQL challenges that might interest anyone who enjoys exploring and manipulating datasets with SQL.

Each day features a different Christmas themed dataset with an interesting problem to solve (all the data is synthetic).

The challenges focus on different ways to analyze and transform these datasets using SQL. For example, finding unusual patterns, calculating rolling averages, or discovering hidden relationships in the data.

While the problems use synthetic data, I tried to create interesting scenarios that reflect real-world data analysis situations.

Starting December 1st at adventofsql.com – (totally free) and you’re welcome to use the included datasets for your own projects.

I’d love to hear what kinds of problems you find most interesting to work on, or if you have suggestions for interesting data scenarios!

Hey everyone,

I am a contractor that uses statista data from time to time to get data from specific sectors and execute my consulting work with a bit more ease.

Well, I decided to have a baby! Yay!

This meant I was going to be out of work for a while. I also noticed that Statista started charging me for simply being able to see the data sets they used to offer for free so I decided it was time to cancel with the intention of re-subscribing when my baby goes into daycare.

So I went to the portal to see where I could cancel and you can’t. But you do have a customer service rep.

Ok great so I email them.

Nothing.

I wait a week and email again. Nothing.

So then I started emailing everyone I could – nothing.

Then I pay to call long distance to Europe after weeks of not hearing anything back and I end up getting a human in Germany who indicated that they cannot help I have to contact the rep.

So at this point I was charged 2 months worth of fees just going through the process of cancellation.

At this point I let the guy know that this was against consumer protection laws in my country (which it is). At that point, he emailed someone internally to allow me to cancel after a lot of hassle.

They credited ONE of the two months I was charged even though I had been requesting cancellation. Either way, I was just glad my credit card was no longer held hostage.

I just wanted to share this information in case anyone else had been using Statista. This is a really sketchy practice and wanted to call it out.

As the title suggests, I’m looking for funny datasets, like one containing only puns.

I’m also interested in character-trait-specific humor, such as a dataset filled with funny and outrageous conspiracy theories or self-deprecating, dark humor.

Any humorous datasets that could turn an LLM into a joke machine are welcome!

Hi everyone,

I’m looking for the DISCO-10M: A Large-Scale Music Dataset. It was previously available through Huggingface, but it is not there anymore. Someone who can share a copy?

Hi everyone! I wanted to share a benchmark we developed for testing our LLM-based symptom checker app. We built this because existing static benchmarks (like MedQA, PubMedQA) didn’t fully capture the real-world utility of our app. With no suitable benchmark available, we created our own and are open-sourcing it in the spirit of transparency.

GitHub: https://github.com/medaks/symptomcheck-bench

Quick Summary:

We call it SymptomCheck Bench because it tests the core functionality of symptom checker apps—extracting symptoms through text-based conversations and generating possible diagnoses. It’s designed to evaluate how well an LLM-based agent can perform this task in a simulated setting.

The benchmark has three main components:

Patient Simulator: Responds to agent questions based on clinical vignettes. Symptom Checker Agent: Gathers information (limited to 12 questions) to form a diagnosis. Evaluator agent: Compares symptom checker diagnoses against the ground truth diagnosis.

Key Features:

400 clinical vignettes from a study comparing commercial symptom checkers. Multiple LLM support (GPT series, Mistral, Claude, DeepSeek) Auto-evaluation system validated against human medical experts

We know it’s not perfect, but we believe it’s a step in the right direction for more realistic medical AI evaluation. Would love to hear your thoughts and suggestions for improvement!

Hello everyone,

I’m working on an image segmentation project aimed at aiding rescue missions by detecting human bodies in underwater crash site images. Specifically, the goal is to identify and segment human figures from underwater images, which could be instrumental in emergency response and recovery operations.

I’m reaching out to see if anyone has, or knows of, a dataset that includes underwater human imagery, especially from crash sites or similar scenarios. Ideally, the dataset would contain varied conditions like different lighting, depths, and visibility to better simulate real-world underwater environments.

If such a dataset isn’t readily available, any resources, advice on data collection, or possible collaboration opportunities to create one would be greatly appreciated! I’m open to any suggestions, as I understand this is a unique and challenging request.

Thank you in advance for any help you can provide!

This is for a university project. Thus far I’ve tried Guidestar, the American hospital directory, CMS, and more to no avail. I am really struggling to obtain any data but am passionate about this topic (and unfamiliar with datasets lol). Looking for financials and/or patient outcomes. I would really appreciate anything!

Hi everyone,

I’m working on a project that requires datasets related to two areas:

1. Soil characteristics: I need data on soil and whether the soil is suitable for farming or not. 2. Water consumption: Datasets that track water usage, ideally in agriculture, industrial settings, or residential homes. Information on seasonal or regional usage trends would be especially helpful.

If anyone knows where I could find reliable datasets for these, or if you’ve come across anything similar in your own work, I’d really appreciate your guidance. Thanks in advance for any recommendations or resources!

Detect PII and PHI with Gretel’s latest synthetic dataset and fine-tuned NER models 🚀:
– 50k train / 5k validation / 5k test examples
– 40 PII/PHI types
– Diverse real world industry contexts
– Apache 2.0

Dataset: https://huggingface.co/datasets/gretelai/gretel-pii-masking-en-v1
Fine-tuned GliNER PII/PHI models: https://huggingface.co/gretelai/gretel-gliner-bi-large-v1.0
Blog / docs: https://gretel.ai/blog/gliner-models-for-pii-detection

Hey r/Datasets! We’re excited to announce K2Q, a newly curated dataset collection for anyone working with visually rich documents and large language models (LLMs) in document understanding. If you want to push the boundaries on how models handle complex, natural prompt-response queries, K2Q could be the dataset you’ve been looking for! The paper can be found here and is accepted to the Empirical Methods in Natural Language Processing (EMNLP) Conference.

What’s K2Q All About?

As LLMs continue to expand into document understanding, the need for prompt-based datasets is growing fast. Most existing datasets rely on basic templates like “What is the value for {key}?”, which don’t fully reflect the varied, nuanced questions encountered in real-world use. K2Q steps in to fill this gap by:

Converting five Key Information Extraction (KIE) datasets into a diverse, prompt-response format with multi-entity, extractive, and boolean questions. Using bespoke templates that better capture the types of prompts LLMs face in real applications.

Why Use K2Q?

Our empirical studies on generative models show that K2Q’s diversity significantly boosts model robustness and performance compared to simpler, template-based datasets.

Who Can Benefit from K2Q?

Researchers and practitioners can use K2Q to:

Test zero-shot or fine-tuned models with realistic, challenging questions. Improve model performance on KIE tasks through diverse prompt-response training. Contribute to future studies on data quality for generative model training.

📄 Dataset & Paper: K2Q will be presented at the Findings of EMNLP, so feel free to dive into our paper for in-depth analyses and results! We’d love to see K2Q inspire your own projects and findings in Document AI.

Hi all,

As the title describes, I am looking for a timeseries sales data set of atleast 3 years with minimum of 10 different products. The dataset should be monthly, weekly or daily.

Can someone recommend me one? I am really struggling to find one on Kaggle.

Hope you guys can help me out!!

Hi,
Has anyone used the Mushroom Observer dataset for image classification? Unless I’m getting something badly wrong, they all reference image IDs but do not supply the images.
i think the images can be gathered through the API using the image ID but they do not want you to scrape them this way.
Does anyone have any experience woerkin with it? It’s for an image classification application.

Hey 👋 i

Ads data published in vanityfair magazines published from 1913 to November 2024.

Data Format:

{ [year]: { year: “1913”, issues: [{ id: “issue’s month”, ads: [ articleKey: “articleKey”, issueKye: “issueKey”, title: “Ad title”, slug: “ad-slug”, coverDate: “coverDate”, pageRange: “page number on which ad was published”, wordCount: “word count” ] }] } }

Link: Google Drive

NOTE: VF was shutdown in 1936 and relaunched in 1983, so in-between years data isn’t available.

Hello everyone!

I’m currently working on a research project aimed at improving early-stage detection of ovarian cancer using deep learning applied to ultrasound images. Right now, I’m in the dataset collection phase and have encountered some challenges in finding accessible datasets.

I’ve come across the PLCO and MMOTU datasets:

PLCO requires a project proposal to gain access, which I’m considering but may take some time. MMOTU offers segmentation data but doesn’t include the full range of diagnostic images needed for my work.

After reviewing literature, I’ve noticed that many researchers use clinical study datasets that are private, hospital-specific patient data, or other datasets that aren’t publicly available.

If anyone here has worked on similar projects or faced these challenges, I’d be very grateful for any pointers! Specifically, I’m looking for:

Publicly accessible ultrasound datasets focused on ovarian or gynecological cancers Datasets that may be available through author requests or by contacting relevant organizations

Thanks in advance for any guidance or resources you can share!