Repo: https://github.com/Datalore-ai/datalore-localgen-cli

Hi everyone,

During my internship I built a small terminal tool that could generate fine tuning datasets from real world data using deep research. I later open sourced it and recently built a version that works fully offline on local files like PDFs DOCX TXT or even JPGs.

I shared this update a few days ago and it was really cool to see the response. It got around 50 stars and so many thoughtful suggestions. Really grateful to everyone who checked it out.

One suggestion that came up a lot was if it can handle multiple files at once. So I integrated that. Now you can just point it at a directory path and it will process everything inside extract text find relevant parts with semantic search apply your schema or instructions and output a clean dataset.

Another common request was around privacy like supporting local LLMs such as Ollama instead of relying only on external APIs. That is definitely something we want to explore next.

We are two students juggling college with this side project so sorry for the slow updates but every piece of feedback has been super motivating. Since it is open source contributions are very welcome and if anyone wants to jump in we would be really really grateful.

so i wanna scrape every business name registered on google in an entire city or state but scraping it directly through selenium does not seem like a good idea even with proxies so is there is any dataset like this for a city like Delhi so that i don’t need to scrape entirety of google maps i need id to train a model for text classification any viable way i can do this?

A huge chunk of US jobs never reach boards at all they sit only on internal career pages.

So I built an AI crawler that goes straight to the source: 70k+ corporate websites.
It collects and cleans the data automatically. Here’s what I found (US only):

You can explore and apply to all these jobs for free here: laboro.co

Hello,
Kindly let me know where I can get low quality football datasets for player detection and analysis. I am working on optimizing a model for African grassroots football. Datasets on Kaggle are done on green astro turf pitches with good cameras and I want to optimize a model for low quality and low resource settings.

Hi everyone,

I’m working on a project where I need a dataset that contains numbers (like 4–8 digit sequences, phone numbers, PINs, etc.) along with some measure of how easy they are to remember.

For example, numbers like 1234 or 7777 are obviously easier to recall than something like 9274, but I need structured data where each number has a “memorability” score (human-rated or algorithmically assigned).

I’ve been searching, but I haven’t found any existing dataset that directly covers this. Before I go ahead and build a synthetic dataset (based on repetition, patterns, palindromes, chunking, etc.), I wanted to check:

• Does such a dataset already exist in psychology, telecom, or cognitive science research?
• If not, has anyone here worked on generating similar “memorability” metrics for numbers?
• Any tips on crowdsourcing this kind of data (e.g., survey setups)?

Any leads or references would be super helpful

Thanks in advance!

I created a tool that extracts books and metadata from Project Gutenberg, the online repository for public domain books, with options for filtering by keyword, category, and language. It outputs structured JSON or CSV for analysis.

Repo link: Project Gutenberg Scraper.

Useful for NLP projects, training data, or text mining experiments.

Looking for affordable, reliable nationwide data for comps. Need both:

• Sold properties (6–12 months history: price, date, address, beds, baths, sqft, lot size, year built, type).
• Active listings (list price, DOM, beds/baths, sqft, property type, location).
• Nationwide coverage preferred (not just one MLS).
• Property details (beds, baths, sqft, lot size, year built, assessed value, taxes).
• API access so it can plug into an app.

Constraints:

• Budget: under $200/month.
• Not an agent → no direct MLS access.
• Needs to be consistent + credible for trend analysis.

If you’ve used a provider that balances accuracy, cost, and coverage, I’d love your recommendations.

I’ve just released The Stack Processed V2, a carefully curated version of The Stack dataset optimized for training robust multi-language code models.

📊 Key Stats:

• 468GB of high-quality code
• 91.3% syntax validation rate (vs ~70% in raw Stack)
• ~10,000 files per language (perfectly balanced)
• 8 major languages: Python, JavaScript, Java, C++, Ruby, PHP, Swift, Shell
• Parquet format for 3x faster loading
• 271 downloads in first month

🎯 What Makes It Different:

Unlike raw scraped datasets that are heavily imbalanced (some languages have millions of files, others just thousands), this dataset ensures equal representation for each language. This prevents model bias toward overrepresented languages.

Processing Pipeline:

1. Syntax validation (removed 8.7% invalid code)
2. Deduplication
3. Quality scoring based on comments, structure, patterns
4. Balanced sampling to ~10k files per language
5. Optimized Parquet format

📈 Performance Impact:

Early testing shows models trained on this dataset achieve:

• +15% accuracy on syntax validation tasks
• +8% improvement on cross-language transfer
• 2x faster convergence compared to raw Stack

🔗 Resources:

• Dataset: https://huggingface.co/datasets/vinsblack/The_Stack_Processed-v2
• Interactive Demo: [Colab Notebook Link]
• License: Apache 2.0

💭 Use Cases:

Perfect for:

• Pre-training multi-language code models
• Fine-tuning for code completion
• Cross-language understanding research
• Educational purposes

Looking for feedback! What features would you like to see in v3? More languages? Different sampling strategies? Enterprise patterns focus?

Happy to answer any questions about the curation process or technical details.

Hi,

Like many of you, I’ve often found that while US Census data is incredibly valuable, it can be a real pain to access for quick, specific queries. With the official QuickFacts tool being down for a while, this has become even more apparent.

So, our team and I built a couple of free tools to try and solve this. I wanted to share them with you all to get your feedback.

The tools are:

• The County Explorer: A simple, at-a-glance dashboard for a snapshot of any US county. Good for a quick baseline.
  • Link: https://counties.cambium.ai/
• Cambium AI: The main tool. It’s a conversational AI that lets you ask detailed questions in plain English and get instant answers.
  • Link: https://app.cambium.ai/

Examples of what you can ask the chat:

• “What is the median household income in Los Angeles County, CA?”
• “Compare the percentage of renters in Seattle, WA, and Portland, OR”
• “Which county in Florida has the highest population over 65?”

Data Source: All the data comes directly from the American Community Survey (ACS) 5-year estimates and IPUMS. We’re planning to add more datasets in the future.

This is a work in progress and would genuinely love to hear your thoughts, feedback, or any features you’d like to see (yes, an API is on the roadmap!).

Thanks!

Hey everyone,

I’m an intern at a new AI startup, and my current task is to collect, store, and organize data for a project where the end goal is to build an archetype after-sales (SAV) agent for financial institutions.

I’m focusing on 3 banks and an insurance company . My first step was scraping their websites, mainly FAQ pages and product descriptions (loans, cards, accounts, insurance policies). The problem is:

• Their websites are often outdated, with little useful product/service info.
• Most of the content is just news, press releases, and conferences (which seems irrelevant for an after-sales agent).
• Their social media is also mostly marketing and event announcements.

This left me with a small and incomplete dataset that doesn’t look sufficient for training a useful customer support AI. When I raised this, my supervisor suggested scraping everything (history, news, events, conferences), but I’m not convinced that this is valuable for a customer-facing SAV agent.

So my questions are:

• What kinds of data do people usually collect to build an AI agent for after-sales service (in banking/insurance)?
• How is this data typically organized/divided (e.g., FAQs, workflows, escalation cases)?
• Where else (beyond the official sites) should I look for useful, domain-specific data that actually helps the AI answer real customer questions?

Any advice, examples, or references would be hugely appreciated .

I have a dataset of 1.1 billion rss feeds and two others, one with 337 million and another with 45 million. Now that i have it I’ve realised ive got no use for it, does anyone know if there’s a way to get rid of it, free or paid to a company who might benefit from it like Dataminr or some data ingesting giant?

Hi everyone,

I’m currently working on a business analytics project as part of my academic work at IIT Madras, and I’m seeking access to Point of Sale (POS) data or any related sales/transactional datasets from any business.

Purpose: The data will be used strictly for educational and analytical purposes to explore trends, build predictive models, and derive business insights.

What I’m looking for:

->POS data (product ID, timestamp, quantity, price, etc.)

->Inventory or stock movement records

->Sales by region, time, or category

If you or your organization is willing to help, or if you can point me in the right direction, I’d be incredibly grateful! I’m also open to signing NDAs or any data use agreements as needed.

Any suggestions are also welcomed
Thank You

Hello everyone,

I’m a third-year undergrad student pursuing a degree in Artificial Intelligence and Machine Learning. For my Deep Learning course project, I’m planning to build a model that detects plastic litter both on the ground and in water.

I’m specifically looking for dataset suggestions — preferably satellite or aerial imagery datasets — that could help with training and testing such a model.

If you know of any publicly available datasets, research projects, or organizations that might share relevant data, I’d greatly appreciate your recommendations.

Thanks in advance!

Hello, I am building a chord sound classifier for my system. I badly need dataset for the following chords A, Cm, D, E, Fm, and Gm. Do you guys know where to find dataset for these chords?

Hey everyone,

I put together an API to make it easier to get historical OHLCV stock prices and full financial statements (income, balance sheet, cash flow) without scraping or manual downloads.

The API:

• Returns quarterly reports in JSON format
• Provides complete price history for any US stock
• Is accessible via RapidAPI for easy integration

Could you give me some feedback on:

• Any missing data fields
• How easy it is to integrate into Python/JS workflows
• Other endpoints you’d want added

Here is the link : https://rapidapi.com/vincentbourgeois33/api/macrotrends-finance1

Thanks for checking it out!

Hey everyone,
I’m learning data science and want to build my skills by working on real-world data. If you have any messy datasets (CSV, Excel, Google Sheets) that need:

• Cleaning (removing duplicates, handling missing values, etc.)
• Structuring
• Basic analysis or summary
• Visualizations (charts, graphs)

…I’d be happy to do it completely free — no catch.

You get clean data and maybe some cool insights. I get practice and accountability.
Drop me a message or comment below if you’re interested — I’ll handle only a couple of small projects each week to give proper focus.

Thanks!

Hello, I need a dataset of active ingredient synonyms for a project. Can you help?

Hi everyone!
I’m working on a final year project related to sentiment analysis on students, aiming to explore aspects like mental health, teacher behavior, course feedback, class schedules, and academic stress.

I’m looking for a dataset that contains:

• Student responses or posts (can be survey-based, forum discussions, or open-ended feedback)
• Labeled sentiments (positive/neutral/negative) or at least raw text suitable for labeling
• Data like year of study, age range, CGPA, course/subject, etc.

Does anyone know of such a dataset or where I might find something similar (publicly available or open for research use)? Any help or direction is greatly appreciated!

Thanks in advance!

for eg , let say Fusariosis (Fusarium infections) or Candida auris Infection , i wanted to train my model on these diseases for a research paper but no good dataset till now , if anyone can help me thanks
if not , then i will just increase the saturation , rotate them , add noise and do stuff like that to train

I compiled a clean, ready-to-use dataset of 50+ leads for Facebook ad targeting. I built it because I couldn’t find one that was up-to-date. Here’s a sample: [Google Drive Link]. Let me know if you find it useful. Feedback is most welcome.

I swear if I see one more portfolio project analyzing Titanic survival rates, I’m going to start rooting for the iceberg.

In actual work, 80% of the job is cleaning messy, inconsistent, incomplete data. But every public dataset I find seems to be already scrubbed within an inch of its life. Missing values? Weird formats? Duplicate entries?

I want datasets that force me to:
– Untangle inconsistent date formats
– Deal with text fields full of typos
– Handle missing data in a way that actually matters for the outcome
– Merge disparate sources that almost match but not quite

My problem is, most companies won’t share their raw internal data for obvious reasons, scraping can get into legal gray areas, and public APIs are often rate-limited or return squeaky clean data.

The difficulty of finding data sources is comparable to that of interpreting the data. I’ve been using beyz to practice explaining my data cleaning and decision, but it’s not as compelling without a genuinely messy dataset to showcase.

So where are you all finding realistic, sector-specific, gloriously imperfect datasets? Bonus points if they reflect actual business problems and can be tackled in under a few weeks.

I’m doing a history project on British cars, and I need datasets regarding car sales in Britain going back to at least the 50s, on cars like the Mini, Rolls Royces and Aston Martins. I’ve poked around a bit already, but I can’t find anything that goes back far enough. I want to be able to reference the data sets to see how various forms of advertising (like TV commercials or celebrity endorsement) affected car sales. Would love some help putting all this together!

So i have to make a VQG model that takes image containing geometrical shapes can be multiple and to generate questions like how many type of shapes are there, which is the biggest shape, what color is the square of etc So i have the images now the questions are left i was thinking of annotating the images like types of shapes, color,size etc and use them in some scripts for question like What is (shape_name) color etc So what are your suggestion what to annotate or how to make questions? Thanks

I’m working on a project that requires a dataset containing body images paired with accurate body fat percentage measurements.

I’ve found several DEXA scan datasets, but they only include anthropometric data and no images. I’ve also scraped a number of publicly available images and estimated body fat visually, but I’m looking for a more accurate dataset.

If anyone can recommend an existing dataset or suggest ways to acquire such data, I’d really appreciate it.

I spent years listening to every song to ever get to number one on the Billboard Hot 100. Along the way, I built a massive dataset about every song. I turned that listening journey into a data-driven history of popular music that will be out soon, but I’m hoping that people can use the data in novel ways!