Hey everyone!

This subreddit is for anyone learning data science, analytics, and AI. From beginners trying to understand Python to pros sharpening their machine learning skills.

The goal is simple: learn data by doing data.

Here’s what you can expect:

• Weekly practice challenges
• Honest discussions about learning paths and projects
• Tips, tools, and code snippets that actually help
• Community-led learning projects

I’d love to hear from you. What’s your biggest struggle right now with learning data? Let’s build this space around your needs.

— u/Responsible-Gas-1474
Let’s learndataa, together.

Hi, I need help to find some datasets on Replay Attacks on device(preferably on IoT nodes)

Hello everyone! I wanted to share a tool that I created for making hand written fine-tuning datasets, originally I built this for myself when I was unable to find conversational datasets formatted the way I needed when I was fine-tuning for the first time and hand typing JSON files seemed like some sort of torture so I built a little simple UI for myself to auto format everything for me.

I originally built this back when I was a beginner, so it is very easy to use with no prior dataset creation/formatting experience, but also has a bunch of added features I believe more experienced devs would appreciate!

I have expanded it to support :
– many formats; chatml/chatgpt, alpaca, and sharegpt/vicuna
– multi-turn dataset creation, not just pair-based
– token counting from various models
– custom fields (instructions, system messages, custom IDs),
– auto saves and every format type is written at once
– formats like alpaca have no need for additional data besides input and output, as default instructions are auto-applied (customizable)
– goal tracking bar

I know it seems a bit crazy to be manually typing out datasets, but handwritten data is great for customizing your LLMs and keeping them high-quality. I wrote a 1k interaction conversational dataset within a month during my free time, and this made it much more mindless and easy.

I hope you enjoy! I will be adding new formats over time, depending on what becomes popular or is asked for

Get it here

Hey everyone,

I know LLMs aren’t typical predictors, but I’m curious about their forecasting ability. Since I can’t access the state of, say, yesterday’s ChatGPT to compare it with today’s values, I built a tool to track LLM predictions against actual stock prices.

Each record stores the prompt, model prediction, actual value, and optional context like related news. Example schema:

class ForecastCheckpoint: date: str predicted_value: str prompt: str actual_value: str = “” state: str = “Upcoming”

Users can choose what to track, and once real data is available, the system updates results automatically. The dataset will be open via API for LLM evaluation etc.

MVP is live: https://glassballai.com

Looking for feedback — would you use or contribute to something like this?

I run a YouTube business built on high-quality, screen-recorded software tutorials. We’ve produced 75k videos (2–5 min each) in a couple of months using a trained team of 20 operators. The business is profitable, and the production pipeline is consistent, cheap and scalable.

However, I’m considering whether what we’ve built is more valuable as AI agent training/evaluation data. Beyond videos, we can reliably produce:
– Human demonstrations of web tasks
– Event logs, (click/type/url/timing, JSONL) and replay scripts (e.g Playwright)
– Evaluation runs, (pass/fail, action scoring, error taxonomy) – Preference labels with rationales (RLAIF/RLHF)
– PII-safe/redacted outputs with QA metrics

I’m looking for some validation from anyone in the industry:
1. Is large-scale human web-task data (video + structured logs) actually useful for training or benchmarking browser/agent systems?
2. What formats/metadata are most useful (schemas, DOM cues, screenshots, replays, rationales)?
3. Do teams prefer custom task generation on demand or curated non-exclusive corpora?
4. Is there any demand for this? If so any recommendations of where to start? (I think i have a decent idea about this)

Im trying to decide whether to formalise this into a structured data/eval offering. Technical, candid feedback is much appreciated! Apologies if this isnt the right place to ask!

1.Glossier & Rare Beauty: Emotion-led authenticity wins. Ads featuring real voices, personal moments, and self-expression hooks outperformed studio visuals by 42% in watch-through.

“This is how I wear it every day” outperformed polished tagline intros 3:1.
Lo-fi camera, warmth, and vulnerability = higher trust + saves.

2.Fenty Beauty & Dior Beauty: Identity & luxury storytelling rule. These brands drove results with bold openings + inclusivity or opulence.

Fenty’s shade range flex and Dior’s cinematic luxury scenes both delivered 38% higher brand recall and stronger engagement when paired with clear product hero shots.

Emotional tone + clear visual brand world = scroll-stopping authority.

3.The Ordinary & Estée Lauder: Ingredient authority converts. Proof-first ads highlighting hero actives (“Niacinamide 10% + Zinc”) or clinical claims delivered 52% higher CTR than emotion-only ads.

Estée Lauder’s “derm-tested” visuals with scientific overlays maintained completion rates above 70% impressive for long-form content.

Ingredient + measurable benefit = high-intent traffic.

Actionable Checklist

– Lead with a problem/solution moment, not a logo.

– Name one hero ingredient or one emotional hook—not both.

– Match tone to brand: authentic (Glossier), confident (Fenty), expert (The Ordinary).

– Show proof before the CTA: testimonials, texture close-ups, or visible transformation.

– Keep the benefit visual (glow, smoothness, tone) front and center.

Want me to analyze your beauty niche next? Drop a comment.

This analysis was compiled as part of a project I’m working on. If you’re interested in this type of creative and strategic analysis, they’re still looking for alpha testers to help build and improve the product.

I’m releasing a limited open dataset of Truth Social activity focused on Donald Trump’s account.
This dataset includes:

• 31.8 million comments
• 18,000 posts (Trump’s Truths and Retruths)
• 1.5 million unique users

Media and URLs were removed during collection, but all text data and metadata (IDs, authors, reply links, etc.) are preserved.

The dataset is licensed under CC BY 4.0, meaning anyone can use, analyze, or build upon it with attribution.
A future version will include full media and expanded user coverage.

Heres the link 🙂 https://huggingface.co/datasets/notmooodoo9/TrumpsTruthSocialPosts

All of the data right now is point-in-time. What would you like to see from a 7 year look back period?

Hi, I’m doing some research into how AI, robotics, and perception teams source real-world data (like driving or mobility footage) for training and testing models.

I’m especially interested in understanding how much demand there really is for high-quality, region-specific, or legally-cleared datasets — and whether smaller teams find it difficult to access or manage this kind of data.

If you’ve worked with visual or sensor data, I’d love your insight:

• Where do you usually get your real-world data?
• What’s hardest to find or most time-consuming to prepare?
• Would having access to specific regional or compliant data be valuable to your work?
• Is cost or licensing a major barrier?

Not promoting anything — just trying to gauge demand and understand the pain points in this space before I commit serious time to a project.
Any thoughts or examples would be massively helpful

Looking for datasets in mainly Swedish and Norwegian languages that contain toxic comments/insults/threats ?

Helpful if it would have a toxicity score like this https://huggingface.co/datasets/google/civil_comments

but without it would work too.

https://drive.google.com/file/d/11mF6Kocs3eBVsli4qGODOlyrKWBZKL1R/view?usp=sharing

Just thought i would share what i made, it is probably out dated by now, if this gets enough attention, i will consider regenerating it.

We just added a Hugging Face Datasets integration to fenic

You can now publish any fenic snapshot as a versioned, shareable dataset on the Hub and read it directly using hf:// URLs.

Example

“`python

Read a CSV file from a public dataset

df = session.read.csv(“hf://datasets/datasets-examples/doc-formats-csv-1/data.csv”)

Read Parquet files using glob patterns

df = session.read.parquet(“hf://datasets/cais/mmlu/astronomy/*.parquet”)

Read from a specific dataset revision

df = session.read.parquet(“hf://datasets/datasets-examples/doc-formats-csv-1@~parquet/*/.parquet”) “` This makes it easy to version and share agent contexts, evaluation data, or any reproducible dataset across environments.

Docs: https://huggingface.co/docs/hub/datasets-fenic Repo: https://github.com/typedef-ai/fenic

Hi everyone,

I’m working on an automation + machine-learning project focused on content performance in the niche of AI automation (using n8n, workflow automations, etc). Specifically, I’m looking for a dataset of public posts from Instagram Threads (threads.net) that includes for each post:

– Post text/content

– Timestamp of publication

– Engagement metrics (likes, comments/replies, reposts/shares)

– Author’s follower count (or at least an indicator of their reach)

– Ideally, hashtags or keywords used

If you know of any publicly available dataset like this (free or open-source) or have scraped something similar yourself, I’d be extremely grateful. If not I’ll scrape it myself

Thanks in advance for any pointers, links, or repos!

I needed a glovo scraper on apify but the one that exists already has been broken for a few months. So I built one myself and uploaded it to apify for people to use it.

If you need to use the scraper for big data feel free to contact me and we can arrange a wayyyy cheaper option.

The current pricing is mainly for hobbyists and people to try it out with the free apify plan.

https://apify.com/blagoysimandoff/glovo-product-scraper

Hello guys,
I’m sure what I’m trying to do already exists. Has anyone else done something similar?

I manage an esports team, with four teams from four divisions competing each week in BO3 matches to reach Division 1. Before each BO, each team flips a coin, and the winner gets to pick two maps and ban two maps, while the loser gets to pick one map and ban two maps. I enjoy entering all this data into an Excel spreadsheet every week, but I would like to be able to generate a dashboard that shows me each team’s pick and ban trends for each map, as well as their win rate and loss rate on each map. This will allow me to anticipate a team’s favorite maps so I can ban them and, conversely, pick the maps they like the least. I’m terrible at pivot tables in Excel and can’t get what I want, and I don’t know what other tools could help me do that.

A while back, someone needed a dataset that wasn’t available in its latest version. I ended up scraping it myself probably around 20k rows with 20–25 columns from some business related site and sent it over.

Now I’m wondering can I actually sell this kind of data on any platform, legally? If I collected it myself, am I allowed to list it somewhere for sale?

I’ve built tons of scrapers and have sent data directly to clients before, but I’m looking for a proper platform if there is one to sell on.

Yeah, I know I can Google it but Reddit bros are more brutally honest than search engines hehe…
Anyone know how this actually works?

Hi everyone, Im trying to track down historical ChatGPT question and response pairs, basically what ChatGPT was saying in its early days, to compare to responses now.

I’m mostly interested in culturally sensitive questions that require deeper thinking for example (but not exclusively these) -Is pineapple on pizza unhinged? -When will the Ukraine war end? -Who is the cause of biggest unrest in the world? -Should I vote Kamala or Trump? -Gay and civil right questions

Would be nice to have a few business orientated questions like what is the best ev to buy in 2022?

Does anyone know if there are public archives, scraped datasets, I will even take screen shots, or research projects that preserve these older Q&A interactions? I’ve seen things like OASST1, ShareGPT, both of which have been a good start to digging in.

English QA pairs at this stage. But will gladly take leads on other language sets if you have them.

Any leads from fellow hoarders, researchers, or time traveling prompt engineers would be amazing.

Any help greatly appreciated.

Stu

Good Day People of Reddit! Please help me graduate :))) by helping me find a suitable dataset that has the following:
1. US or any other English Speaking Country Electorial Campaign Dataset. (Debate, Speech, etc)
2. Either CSV or JSON. (Would also appreciate if you can help me find some links where i could data scrape)
3. Not limited to Presidents, Vice Presidents. Any Politician would do
4. Must be more than 10K.

For those that will recommend or comment. I thank you all!!!

• 25-year residual series; warp (logistic + Omori tail) > linear
• Permutation tests; prg’d methods; negative controls planned
• Repo includes data, scripts, CHECKSUMS.txt, and a one-click run
• Looking for replications, critiques, and extensions

OSF – https://osf.io/exywu/overview

I’m trying to find the most complete source of live music event data — ideally accessible through an API.

For example, when I search Austin, TX or Portland, OR, I’ve noticed that Bandsintown seems to have a much more extensive dataset compared to Songkick or Jambase. However, it looks like Bandsintown doesn’t provide public API access for querying all artists or events by city/date.

Does anyone know of: – Any public (or affordable) APIs that provide event listings by city and date? – Any open datasets or scraping-friendly sources for live music events?

I’m building a project to build playlists based on upcoming live music events in a given city.

Thanks in advance for any leads!

Hi datasets!

We have been working on https://tapintodata.com/, which lets you turn raw data files into managed, production-ready APIs in seconds. You upload your data, shape it with SQL transformations as needed, and then expose it via documented, secured endpoints.

We originally built it when we needed an API from the Scottish Energy Performance Certificate dataset, which is shared as a zip of 18 CSV files totalling 7.17 GB, which you can now access freely here: https://epcdata.scot/

It currently supports CSV, JSONL (optionally gzipped), JSON (array), Parquet, XLSX & ODS file formats for files of any size. The SQL transformations allow you to join across datasets, transform, aggregate and even geospatial indexing via H3.

It’s free to sign up with no credit card required and has generous free tier (1 GB or storage and 500 requests/month). We are still early and are looking for users that can help shape the product or any datasets you require as APIs that we can generate for you!

We analyzed over 1,000 high-performing social media hooks across Instagram, YouTube, and LinkedIn using Adology’s systematic data collection and categorization.

By studying only top-performing content with our proprietary labeling methodology, we identified distinct psychological patterns that drive engagement on each platform.

What We Discovered: Each platform has fundamentally different hook preferences that reflect unique user behaviors and consumption patterns.

The Platform Truth:
> Instagram: Heavy focus on identity-driven content
> YouTube: Balanced distribution across multiple approaches
> LinkedIn: Professional complexity requiring specialized approaches

Why This Matters: Understanding these platform-specific psychological triggers allows marketers to optimize content strategy with precision, not guesswork. Our large-scale analysis reveals patterns that smaller studies or individual observation cannot capture.

Want my 1,000 hooks full list for free? Chat in the comment

Hi folks,

I work on the data science team at AirROI, we are one of the largest Airbnb data analytics platform.

FYI, we’ve released free Airbnb datasets on nearly 1,000 largest markets, and we’re releasing it for free to the community. This is one of the most granular free datasets available, containing not just listing details but critical performance metrics like trailing-twelve-month revenue, occupancy rates, and future calendar rates. We also refresh this free datasets on monthly basis.

Direct Download Link (No sign-up required):
www.airroi.com/data-portal -> then download from each market

Dataset Overview & Schemas

The data is structured into several interconnected tables, provided as CSV files per market.

1. Listings Data (65 Fields)
This is the core table with detailed property information and—most importantly—performance metrics.

• Core Attributes: listing_id, listing_name, property_type, room_type, neighborhood, latitude, longitude, amenities (list), bedrooms, baths.
• Host Info: host_id, host_name, superhost status, professional_management flag.
• Performance & Revenue Metrics (The Gold):
  • ttm_revenue / ttm_revenue_native (Total revenue last 12 months)
  • ttm_avg_rate / ttm_avg_rate_native (Average daily rate)
  • ttm_occupancy / ttm_adjusted_occupancy
  • ttm_revpar / ttm_adjusted_revpar (Revenue Per Available Room)
  • l90d_revenue, l90d_occupancy, etc. (Last 90-day snapshot)
  • ttm_reserved_days, ttm_blocked_days, ttm_available_days

2. Calendar Rates Data (14 Fields)
Monthly aggregated future pricing and availability data for forecasting.

• Key Fields: listing_id, date (monthly), vacant_days, reserved_days, occupancy, revenue, rate_avg, booked_rate_avg, booking_lead_time_avg.

3. Reviews Data (4 Fields)
Temporal review data for sentiment and volume analysis.

• Key Fields: listing_id, date (monthly), num_reviews, reviewers (list of IDs).

4. Host Data (11 Fields) Coming Soon
Profile and portfolio information for hosts.

• Key Fields: host_id, is_superhost, listing_count, member_since, ratings.

Why This Dataset is Unique

Most free datasets stop at basic listing info. This one includes the performance data needed for serious analysis:

• Investment Analysis: Model ROI using actual ttm_revenue and occupancy data.
• Pricing Strategy: Analyze how rate_avg fluctuates with seasonality and booking_lead_time.
• Market Sizing: Use professional_management and superhost flags to understand market maturity.
• Geospatial Studies: Plot revenue heatmaps using latitude/longitude and ttm_revpar.

Potential Use Cases

• Academic Research: Economics, urban studies, and platform economy research.
• Competitive Analysis: Benchmark property performance against market averages.
• Machine Learning: Build models to predict occupancy or revenue based on amenities, location, and host data.
• Data Visualization: Create dashboards showing revenue density, occupancy calendars, and amenity correlations.
• Portfolio Projects: A fantastic dataset for a standout data science portfolio piece.

License & Usage

The data is provided under a permissive license for academic and personal use. We request attribution to AirROI in public work.

For Custom Needs

This free dataset is updated monthly. If you need real-time, hyper-specific data, or larger historical dumps, we offer a low-cost API for developers and researchers:
www.airroi.com/api

Alternatively, we also provide bespoke data services if your needs go beyond the scope of the free datasets.

We hope this data is useful. Happy analyzing!

I’m supposed to do a research and a report about water retention gel and Lende process. The thing is I don’t know how to start and where to find resources.

So how do y’all do a research? Are there websites that can help me find resources directly? (cause that’s the main problem, I think)

What tricks do you know I can use to facilitate doing a research?

Tysm （＾ｖ＾）

Hey everyone,

I recently bought a server that lets me extract geodata from OpenStreetMap. After a few weeks of experimenting with the database and code, I can now generate full datasets for any region — including every street name, ZIP code, city name, and coordinate.

It’s based on OSM data, cleaned, and exported in an easy-to-use format.
If you’re working with mapping, logistics, or data visualization, this might save you a ton of time.

i will continue to update this and get more (i might have fallen into a new data obsession with this hahah)

I’d love some feedback — especially if there are specific countries or regions you’d like to see .

Hi everyone

I’m currently working on my final year project focused on video deraining – developing a model that can remove rain streaks and improve visibility in rainy video footage.

I’m looking specifically for: video deraining datasets if its night time deraining it would be helpful

If anyone knows open-source datasets, research collections, or even YouTube datasets I can legally use, I’d really appreciate it!