I’ve seen many posts about API’s to track flight prices but is there anything out there that tracks on time/delayed arrivals and departures?

For instance, some states in the United States show a preference for ham during Thanksgiving while others prefer turkey.

Are there any datasets with similar data to generate insights?

Is there a strong need to fine-tune agents with tool calling? How would I construct a dataset to do so (please explain with an example). Thanks in advance

Hey everyone!

I’ve recently put together a free repository of ecommerce product datasets—it’s publicly available at https://github.com/octaprice/ecommerce-product-dataset.

Currently, there are only two datasets (both from Amazon’s bird food category, each with around 1,800 products), which include attributes like product categories, images, prices, brand names, reviews, and even product image URLs.

The information available in the dataset can be especially useful for anyone doing machine learning or data science stuff — price prediction, product categorization, or image analysis.

The plan is to add more datasets on a regular basis.

I’d love to hear your thoughts on which websites or product categories you’d find interesting for the next releases.

I can pretty much collect data from any site (within reason!), so feel free to drop some ideas. Also, let me know if there are any additional fields/attributes you think would be valuable to include for research or analysis.

Thanks in advance for any feedback, and I look forward to hearing your suggestions!

Hi everyone,

I’m working on a project to create a comprehensive database of tourist attractions across India—everything from iconic landmarks to hidden gems. My goal is to make travel easier and more personalized for travelers. I’ll not resell it, but still going to use in planning software for commercial purposes.

I need data columns like Location details (city, state), coords, images.

My Challenges:

Scraping data: I’ve considered scraping websites, but I’m not sure of the legality or technical challenges. Using APIs: Google Maps API is great but expensive for the scale I need. Are there any free or low-cost alternatives? Collaborative sources: Is there any open-source or community-driven data for Indian attractions?

I’ve tried scraping OSM but didn’t got appropriate results. A lot of the data needs extensive verification to be useful.

Hello everyone,

I’m currently working on my bachelor’s thesis., which focuses on the non-invasive diagnosis of endometriosis using biomarkers like microRNAs and machine learning. My goal is to reproduce existing studies and analyze their methodologies.

For this, I am looking for datasets from endometriosis patients (e.g., miRNA sequencing data from blood, saliva, or tissue samples) that are either publicly available or can be accessed upon request. Does anyone have experience with this or know where I could find such datasets? Ive checked GEO and reached out to authors of a relevant paper (still waiting for a response).

If anyone has tips on where to find such datasets or has experience with similar projects, I’d be incredibly grateful for your guidance!

Thank you so much in advance!

New to ML. Trying to be able to detect fonts on images with computer text (like text added to an image in PhotoShop)

What do the numbers mean here: https://github.com/google/fonts/blob/main/tags/all/families.csv

I have a clean dataset with the last 20+ years of ncaa tournament games (round, seeds, result, score) along with ~100 traditional and advanced team stats from multiple public sources as they were pre-tournament. I’ve done a lot of feature engineering and can add those metrics in too (ex: 3-pt % to opponents 3-pt defense, raw and normalized by diff SOS type approaches).

It’s nothing crazy extensive (no player stats, injuries, trends) but it’s cleaner and more comprehensive than anything I’ve found available for free download / scraping.

I put the scripts together a few years ago with non-trivial code effort and manual QC (name formatting etc). It wouldn’t be particularly difficult to reproduce for a decent programmer. I’m sure AI has made that type of process more accessible but it’d still take some time for most.

Having never sold a dataset is there any value here? I’m not expecting much but the work is already done.

I’ve started the process of including regular season games (stats at game time) if that would help but probably won’t finish without understanding value. Same for game lines / betting info but only if the dataset is useless without them. They’re messier to pull.

Does anyone know of a dataset (free or paid) which contains the sitemaps of all the websites on the web?

Yes I know that tens of millions of websites update their sitemaps daily. I know that not every website has a sitemap. I know that a decent chunk (10-20% by volume will be for p*rn). I know that this data takes up a lot of space (250-350tb based on my calculations).

The closest dataset I’m familiar with is common crawl, but they only capture 10% of the web at best and they focus more on full pages and less on sitemaps.

I know the odds of this being available is pretty slim, but I wanted to see if anyone has come across a huge sitemap list like this before.

P.S. I have a 1.5PB homelab and have the means to store all this data as well as process it. So it might be a non-standard request, but i’m asking for real reasons, not a hypothetical.

I’m excited to share Content Extractor with Vision LLM, an open-source Python tool that extracts content from documents (PDF, DOCX, PPTX), describes embedded images using Vision Language ModelsI’m excited to share Content Extractor with Vision LLM, an open-source Python tool that extracts content from documents (PDF, DOCX, PPTX), describes embedded images using Vision Language Models, and saves the results in clean Markdown files.

This is an evolving project, and I’d love your feedback, suggestions, and contributions to make it even better!

✨ Key Features

Multi-format support: Extract text and images from PDF, DOCX, PPTX. Advanced image description: Choose from local models (Ollama’s llama3.2-vision) or cloud models (OpenAI GPT-4 Vision). Two PDF processing modes: Text + Images: Extract text and embedded images. Page as Image: Preserve complex layouts with high-resolution page images. Markdown outputs: Text and image descriptions are neatly formatted. CLI interface: Simple command-line interface for specifying input/output folders and file types. Modular & extensible: Built with SOLID principles for easy customization. Detailed logging: Logs all operations with timestamps.

🛠️ Tech Stack

Programming: Python 3.12 Document processing: PyMuPDF, python-docx, python-pptx Vision Language Models: Ollama llama3.2-vision, OpenAI GPT-4 Vision

📦 Installation

Clone the repo and install dependencies using Poetry. System dependencies like LibreOffice and poppler are required for processing specific file types.

Detailed setup instructions: GitHub Repo

🚀 How to Use

Clone the repo and install dependencies. Start the Ollama server: ollama serve. Pull the llama3.2-vision model: ollama pull llama3.2-vision. Run the tool:bashCopy codepoetry run python main.py –source /path/to/source –output /path/to/output –type pdf Review results in clean Markdown format, including extracted text and image descriptions.

💡 Why Share?

This is a work in progress, and I’d love your input to:

Improve features and functionality Test with different use cases Compare image descriptions from models Suggest new ideas or report bugs

📂 Repo & Contribution

GitHub: Content Extractor with Vision LLM

Feel free to open issues, create pull requests, or fork the repo for your own projects.

🤝 Let’s Collaborate!

This tool has a lot of potential, and with your help, it can become a robust library for document content extraction and image analysis. Let me know your thoughts, ideas, or any issues you encounter!

Looking forward to your feedback, contributions, and testing results.

, and saves the results in clean Markdown files.

This is an evolving project, and I’d love your feedback, suggestions, and contributions to make it even better!

✨ Key Features

Multi-format support: Extract text and images from PDF, DOCX, PPTX. Advanced image description: Choose from local models (Ollama’s llama3.2-vision) or cloud models (OpenAI GPT-4 Vision). Two PDF processing modes: Text + Images: Extract text and embedded images. Page as Image: Preserve complex layouts with high-resolution page images. Markdown outputs: Text and image descriptions are neatly formatted. CLI interface: Simple command-line interface for specifying input/output folders and file types. Modular & extensible: Built with SOLID principles for easy customization. Detailed logging: Logs all operations with timestamps.

🛠️ Tech Stack

Programming: Python 3.12 Document processing: PyMuPDF, python-docx, python-pptx Vision Language Models: Ollama llama3.2-vision, OpenAI GPT-4 Vision

📦 Installation

Clone the repo and install dependencies using Poetry. System dependencies like LibreOffice and poppler are required for processing specific file types.

Detailed setup instructions: GitHub Repo

🚀 How to Use

Clone the repo and install dependencies. Start the Ollama server: ollama serve. Pull the llama3.2-vision model: ollama pull llama3.2-vision. Run the tool:bashCopy codepoetry run python main.py –source /path/to/source –output /path/to/output –type pdf Review results in clean Markdown format, including extracted text and image descriptions.

💡 Why Share?

This is a work in progress, and I’d love your input to:

Improve features and functionality Test with different use cases Compare image descriptions from models Suggest new ideas or report bugs

📂 Repo & Contribution

GitHub: Content Extractor with Vision LLM

Feel free to open issues, create pull requests, or fork the repo for your own projects.

🤝 Let’s Collaborate!

This tool has a lot of potential, and with your help, it can become a robust library for document content extraction and image analysis. Let me know your thoughts, ideas, or any issues you encounter!

Looking forward to your feedback, contributions, and testing results.

Greetings.

I’ve been searching for days, seeking high and low, for a dataset matching what I described in the title.

From what I’ve found, there is a wealth of information for counts pertaining to number of children with 1 or more allegations, but not much for counts and/or totals for allegations themselves.

The best resource seems to be the California Child Welfare Indicators Project. In the report index I linked, you’ll see two reports that I found (at first) to be the most promising. Under the Fundamentals heading, there’s Allegations: Child Maltreatment Allegations – Child Count. It’s close, but because they’re again counting children and not allegations, I can’t use it. The other report, under CWS Rates, is Allegation Rates: Child Maltreatment Allegation Rates. It seems so close, but when I look at the options under Report Output, they list the rates (obviously), the total child population, and children with allegations. Looking at the descriptions for the data, it appears I can’t even infer the totals using the incidence rates, but I may be wrong.

Lastly, the report I was most excited about is found under Process Measures; the one labeled 2B. It’s titled “Referrals by Time to Investigation” and I thought that, since every report to CPS requires a response, that this was what I was looking for. Alas, this report only totals allegations that are deemed worthy of an in-person investigation.

So, here I am seeking the help of the Dataset community. Does anyone have any recommendations where I might look to find total reports made to CPS? Have I already found it among the reports listed at the CCWIP and just don’t realize it?

Should I reach out to them and just ask for the data?

I appreciate any help the community can provide.

Many thanks.

Hi, I am searching for open data for which I can analyze what kind of jobs are more prevalent in each city worldwide? (ex. more software engineer jobs in London than Paris, more cleaner jobs in Seoul than London, etc). Does anyone have idea where I can get these types of data? I found some 1.3m job openings data in Linkedin from kaggle, but this seems to contain the information only from Canada, united states and united kingdom.

Hey Reddit! 👋

Happy New Year! To kick off 2025, we’re giving away 90 days of free access to our NCAA Basketball API to the first 20 people who sign up by Friday, January 10. This isn’t a sales pitch—there’s no commitment, no credit card required—just an opportunity for those of you who love building, experimenting, and exploring with sports data.

Here’s what you’ll get for all conferences:

Real-time game stats Post-game stats Season aggregates

Curious about the API? You can check out the full documentation here: API Documentation.

We know there are tons of creative developers, analysts, and data enthusiasts here on Reddit who can do amazing things with access to this kind of data, and we’d love to see what you come up with. Whether you’re building an app, testing a project, or just curious to explore, this is for you.

If you’re interested, join our discord to signup. Spots are limited to the first 20, so don’t wait too long!

We’re really excited to see how you’ll use this. If you have any questions, feel free to ask in the comments or DM us.

Hello everyone… May I ask if anyone have any real-world datasets about photovoltaic? I am gonna use this for a school research project about the effectiveness of machine-learning based photovoltaic system for predictive maintenance. I currently use synthetic data, however I am not that confident in its validity, and it might be the reason for us to be cooked in our defense…

import pandas as pd import numpy as np from sklearn.model_selection import train_test_split from sklearn.preprocessing import MinMaxScaler from sklearn.metrics import classification_report, accuracy_score, confusion_matrix, ConfusionMatrixDisplay, precision_recall_curve import tensorflow as tf from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense, Dropout from tensorflow.keras.callbacks import EarlyStopping import matplotlib.pyplot as plt # Generate synthetic data np.random.seed(42) data = { “Temperature (°C)”: np.random.uniform(15, 45, 1000), # Ambient temperature “Irradiance (W/m²)”: np.random.uniform(100, 1200, 1000), # Solar irradiance “Voltage (V)”: np.random.uniform(280, 400, 1000), # Voltage output “Current (A)”: np.random.uniform(4, 12, 1000), # Current output } # Create DataFrame df = pd.DataFrame(data) df[“Power (W)”] = df[“Voltage (V)”] * df[“Current (A)”] df[“Fault”] = np.where((df[“Power (W)”] < 2000) | (df[“Voltage (V)”] < 320), 1, 0) # Fault criteria # Preprocess data features = [“Temperature (°C)”, “Irradiance (W/m²)”, “Voltage (V)”, “Current (A)”] target = “Fault” X = df[features] y = df[target] # Train-test split X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) # Scale features scaler = MinMaxScaler() X_train_scaled = scaler.fit_transform(X_train) X_test_scaled = scaler.transform(X_test) # Build ANN model model = Sequential([ Dense(128, input_dim=X_train_scaled.shape[1], activation=’relu’), Dropout(0.3), Dense(64, activation=’relu’), Dense(1, activation=’sigmoid’) # Sigmoid for binary classification ]) model.compile(optimizer=’adam’, loss=’binary_crossentropy’, metrics=[‘accuracy’]) # Early stopping early_stopping = EarlyStopping(monitor=’val_loss’, patience=5, restore_best_weights=True) # Train ANN model history = model.fit( X_train_scaled, y_train, epochs=50, batch_size=32, validation_split=0.2, verbose=1, callbacks=[early_stopping] ) # Evaluate model y_pred = (model.predict(X_test_scaled) > 0.5).astype(“int32”) print(“ANN Accuracy:”, accuracy_score(y_test, y_pred)) print(“Classification Report:n”, classification_report(y_test, y_pred)) # Confusion Matrix cm = confusion_matrix(y_test, y_pred) disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=[0, 1]) disp.plot(cmap=”Blues”) plt.title(“Confusion Matrix (ANN)”) plt.show() # Precision-Recall Curve y_scores = model.predict(X_test_scaled).ravel() precision, recall, _ = precision_recall_curve(y_test, y_scores) plt.plot(recall, precision, marker=’.’, label=”ANN”) plt.title(“Precision-Recall Curve”) plt.xlabel(“Recall”) plt.ylabel(“Precision”) plt.legend() plt.show() # Plot training history plt.plot(history.history[‘accuracy’], label=’Train Accuracy’) plt.plot(history.history[‘val_accuracy’], label=’Validation Accuracy’) plt.title(“Training and Validation Accuracy (ANN)”) plt.xlabel(“Epoch”) plt.ylabel(“Accuracy”) plt.legend() plt.show() Does the synthetic data generated in this code, particularly the ranges for temperature, irradiance, voltage, and current, as well as the fault definition criteria, realistically reflect the operational parameters and fault conditions of photovoltaic systems? Could someone with expertise in photovoltaic system analysis validate whether this data and fault classification logic are appropriate and credible for use in a school research project? (Our research is about studying the effectiveness of machine learning-based photovoltaic systems for predictive maintenance). I tried implementing real-world data with this research, however with limited time and resources, I think using synthetic data would be the best option for this research.

import pandas as pd import numpy as np from sklearn.model_selection import train_test_split from sklearn.preprocessing import MinMaxScaler from sklearn.metrics import classification_report, accuracy_score, confusion_matrix, ConfusionMatrixDisplay, precision_recall_curve import tensorflow as tf from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense, Dropout from tensorflow.keras.callbacks import EarlyStopping import matplotlib.pyplot as plt # Generate synthetic data np.random.seed(42) data = { “Temperature (°C)”: np.random.uniform(15, 45, 1000), # Ambient temperature “Irradiance (W/m²)”: np.random.uniform(100, 1200, 1000), # Solar irradiance “Voltage (V)”: np.random.uniform(280, 400, 1000), # Voltage output “Current (A)”: np.random.uniform(4, 12, 1000), # Current output } # Create DataFrame df = pd.DataFrame(data) df[“Power (W)”] = df[“Voltage (V)”] * df[“Current (A)”] df[“Fault”] = np.where((df[“Power (W)”] < 2000) | (df[“Voltage (V)”] < 320), 1, 0) # Fault criteria # Preprocess data features = [“Temperature (°C)”, “Irradiance (W/m²)”, “Voltage (V)”, “Current (A)”] target = “Fault” X = df[features] y = df[target] # Train-test split X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) # Scale features scaler = MinMaxScaler() X_train_scaled = scaler.fit_transform(X_train) X_test_scaled = scaler.transform(X_test) # Build ANN model model = Sequential([ Dense(128, input_dim=X_train_scaled.shape[1], activation=’relu’), Dropout(0.3), Dense(64, activation=’relu’), Dense(1, activation=’sigmoid’) # Sigmoid for binary classification ]) model.compile(optimizer=’adam’, loss=’binary_crossentropy’, metrics=[‘accuracy’]) # Early stopping early_stopping = EarlyStopping(monitor=’val_loss’, patience=5, restore_best_weights=True) # Train ANN model history = model.fit( X_train_scaled, y_train, epochs=50, batch_size=32, validation_split=0.2, verbose=1, callbacks=[early_stopping] ) # Evaluate model y_pred = (model.predict(X_test_scaled) > 0.5).astype(“int32”) print(“ANN Accuracy:”, accuracy_score(y_test, y_pred)) print(“Classification Report:n”, classification_report(y_test, y_pred)) # Confusion Matrix cm = confusion_matrix(y_test, y_pred) disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=[0, 1]) disp.plot(cmap=”Blues”) plt.title(“Confusion Matrix (ANN)”) plt.show() # Precision-Recall Curve y_scores = model.predict(X_test_scaled).ravel() precision, recall, _ = precision_recall_curve(y_test, y_scores) plt.plot(recall, precision, marker=’.’, label=”ANN”) plt.title(“Precision-Recall Curve”) plt.xlabel(“Recall”) plt.ylabel(“Precision”) plt.legend() plt.show() # Plot training history plt.plot(history.history[‘accuracy’], label=’Train Accuracy’) plt.plot(history.history[‘val_accuracy’], label=’Validation Accuracy’) plt.title(“Training and Validation Accuracy (ANN)”) plt.xlabel(“Epoch”) plt.ylabel(“Accuracy”) plt.legend() plt.show() Does the synthetic data generated in this code, particularly the ranges for temperature, irradiance, voltage, and current, as well as the fault definition criteria, realistically reflect the operational parameters and fault conditions of photovoltaic systems? Could someone with expertise in photovoltaic system analysis validate whether this data and fault classification logic are appropriate and credible for use in a school research project? (Our research is about studying the effectiveness of machine learning-based photovoltaic systems for predictive maintenance). I tried implementing real-world data with this research, however with limited time and resources, I think using synthetic data would be the best option for this research.

I have tried many times on websites,but haven’t reply any response until now.

May I ask if anyone have any real-world datasets about photovoltaic? I am goint to use it for a school research project. Which is about the effectiveness of machine-learning based photovoltaic system for predictive maintenance. I currently use synthetic data, however I am not that confident in its validity. Any reccomendations, suggestions, and opinions are highly encouraged.

’m on the lookout for a dataset that contains individual-level data with measurements taken both before and after an event, intervention, or change. It doesn’t have to be from a specific field—I’m open to anything in areas like healthcare, economics, education, or social studies.

Ideally, the dataset would include a variety of individual characteristics, such as age, income, education, or health status, along with outcome variables measured at both time points so I can analyze changes over time.

It would be great if the dataset is publicly available or easy to access, and it should preferably have enough data points to support statistical analysis. If you know of any databases, repositories, or specific studies that match this description, I’d really appreciate it if you could share them or point me in the right direction.

Thanks so much in advance for your help! 😊

I’m looking specifically for granularity in terms of wealth percentage. There’s tons of datasets that go something like top .1%/1%/10%/50%/90% or so, but I’d really need something that goes AT LEAST by individual percent (as in top 1%, 2%, 3%, 4%, all the way down to the bottom 99%), if not fractions of a percent as well. Or any dataset where I’d be able to calculate those statistics from.

Thank you in advance! Any leads towards such a data set would be greatly appreciated!

I have a list of Fortune 1000 firms and want to filter them on NAICS, since I only need a particular industry. The NAICS is not included. Does anyone know whether there is an easy way to do this, instead of looking it up for each company individually? Thank you!

How do you calculate sample size based on odds ratios and confidence intervals?

Using SPSS, you can do sample size based on what test you are using so I am using one way ANOVA and that wanted Standard deviation and mean but all previous articles have odds ratios and CIs so how do I calculate sample size?

Hello,

I’m looking for a couple of hours, i can’t find a dataset that will provide me like 5k+ dishes/recipes that will include the name, the ingredients, the description and the precise region like: Pizza Margarita will be Napoli.

I’m not sure i found all the datasets website yet, if you have any info or any advices to find something similar or a way to scrape a website that includes those informations i’m up for it.

Thanks

I have two datasets that relate to each other. The first dataset consists of images on one column and the time stamp and voltage level at that time. the second dataset is the weather forecast, solar irradiance, and other features ( 10+). the data provided is for each 30 mins of each day for 3 years, while the images are pictures of the sky for each minute of the day. I need help to direct me to the way that I should combine these datasets into one and then later train it with a machine/deep learning-based model analysis where the output is the forecast of the voltage level based on the features.

In my previous experiences, I never dealt with Time Series datasets so I am asking about the correct way to do this, any recommendations are appreciated.

Hi all:

We’re building an exploratory data tool, and we’re hoping to simulate a data warehouse that has data from common tools, like Stripe and Hubspot. The data would be “fake” but simulate the real world.

Does anyone have any clever ideas on how to acquire data sets which are “real world” like this?

The closest thing I can think of is someone using a data synthesizer like gretel.ai or a competitor on a real world data set and being willing to share it.

Thanks,

Hi,

I’m working on developing a free tool to help homeowners and buyers better navigate the real estate market. To make this tool effective, I need access to the following data:

Dates homes were listed and sold Home features (e.g., square footage, lot size, number of bedrooms/bathrooms, etc.) Information about homes currently on the market

I initially hoped to use the Zillow API, but unfortunately, they’re not granting access. Are there any other free or low-cost data sources or APIs that you’d recommend for accessing this type of information?

Your insights and suggestions would mean a lot. Thanks in advance for your help!

I will use this to create a receipt scanner that logs all the items a user purchases. Ideally, an item should have the receipt abbrevation (like MISF TORTILLA),the corresponding actual item name (like Mission Flour Tortilla Wraps), and the UPC/SKU with the store name.

I will use this to create a receipt scanner that logs all the items a user purchases. Ideally, an item should have the receipt abbrevation (like MISF TORTILLA),the corresponding actual item name (like Mission Flour Tortilla Wraps), and the UPC/SKU with the store name.