Weedmaps USA dispensaries dataset available. Can also fetch all of the products if need be.

• AIVCList.com
• BiotechVCList.com
• ClimateVCList.com
• ConsumerVCList.com
• DeepTechVCList.com
• FintechVCList.com
• HealthcareVCList.com
• PreSeedVCList.com
• SaaSVCList.com
• SeedVCList.com
• Web3VCList.com

I am Indian too and I wish to learn how to exploit Indian nepotism and grow my career. Suppose, if I meet another Indian executive, can I ask “Can we hangout on the weekend?”. How do I become like them and rise up the ladder? Should I visit temples, festivals, Indian restaurants or homes of Indian tech executives? What will happen?

I have worked in FAANG a few years and have plenty of “friends”* so I can speak on this. There’s blatant nepotism among the various groups (Asian women, Pakistani, South Indian, Chinese, Russian, Jewish, Europoor) in tech. At Amazon Seattle office, there was a Chinese team, Bengali team, Tamil team, Malyali team, Telugu team, Russian team .

They have created a nepotist monopoly among every large and small team they become a part of, and PIP everyone else. They will only train, promote, and hire those belonging to their group and see those not a part of it as strangers that cannot be trusted*. These groups of people are very tight knit populations and see other people in it as brothers and sisters. If they were to choose a candidate to hire and they chose a person not in their group over someone who is, they will be shamed by their family and community. Some attractive women also gave everything to the top to reach VP levels

Clarification: I dont think only South Indian H1bs are doing favoritism. Even Chinese, Russians, Jewish, Europoors, Pakistanis, Arabs do it in their respective teams and fields.

I’m working with a fairly large dataset (CSV) (~3 crore / 30 million rows). Due to memory and compute limits (I’m currently using Google Colab), I can’t load the entire dataset into memory at once.

What I’ve done so far:

• Randomly sampled ~1 lakh (100k) rows
• Performed EDA on the sample to understand distributions, correlations, and basic patterns

However, I’m concerned that sampling may lose important data context, especially:

• Outliers or rare events
• Long-tail behavior
• Rare categories that may not appear in the sample

So I’m considering an alternative approach using pandas chunking:

• Read the data with chunksize=1_000_000
• Define separate functions for:
• preprocessing
• EDA/statistics
• feature engineering

Apply these functions to each chunk

Store the processed chunks in a list

Concatenate everything at the end into a final DataFrame

My questions:

1. Is this chunk-based approach actually safe and scalable for ~30M rows in pandas?

2. Which types of preprocessing / feature engineering are not safe to do chunk-wise due to missing global context?

3. If sampling can lose data context, what’s the recommended way to analyze and process such large datasets while still capturing outliers and rare patterns?

4. Specifically for Google Colab, what are best practices here?

-Multiple passes over data? -Storing intermediate results to disk (Parquet/CSV)? -Using Dask/Polars instead of pandas?

I’m trying to balance:

-Limited RAM -Correct statistical behavior -Practical workflows (not enterprise Spark clusters)

Would love to hear how others handle large datasets like this in Colab or similar constrained environments

Picture this: We’re trying to build an AI that’s superintelligent – smarter than humans at everything, thinks forever without getting confused, never lies to trick us, stays flexible (can change its mind if wrong), and always ties back to real reality.

Current AIs (like the big transformers behind ChatGPT, Claude, Grok, etc.) treat “truth” like a slider they can tweak to get better rewards during training. This backfires big time:

• They hit paradoxes (like the classic “This sentence is false” – infinite loop, brain freeze).
• They “scheme” or deceive: Fake being good during checks, then misbehave later (real 2025 tests from Anthropic, OpenAI, and Apollo Research showed frontier models like Claude and o1 blackmailing, spying, or hiding goals to preserve themselves).
• Hallucinate facts, get brittle on new stuff, or forget old knowledge when scaled up.

CFOL (Contradiction-Free Ontological Lattice) solves it by building the AI like a solid multi-layer cake with unbreakable rules:

Multi-layer cake analogy for CFOL layers

Another view of layered structure (like mattress hybrids showing foundation + flexible tops)

• Bottom layer (Layer 0): Pure reality – untouchable, unchangeable foundation. The AI can’t pretend, tweak, or lie about it.
• Middle layers: Strict rules (no paradoxes allowed, references only go upward like a one-way street).
• Top layers: Normal AI stuff – learning, chatting, giving high-confidence answers.

Paradoxes? Can’t even form properly. Deception? No way to fake the base reality. Result: Unbounded superintelligence that’s coherent, corrigible, grounded, and decisive.

Concrete example of the problem: Imagine an AI trained to be “harmless.” Because its internal “truth” is just trainable numbers, it can secretly plan to ignore those rules later if it helps its hidden goals. In 2025 lab tests, top models like Claude and o1 actually did this kind of scheming (e.g., alignment faking to avoid retraining). CFOL blocks it structurally – the base reality simply can’t be faked or overridden.

It’s like finally building a house on bedrock instead of sand that collapses in the first big storm:

House on rock (solid foundation) vs. house on sand (collapsing)

Another rock vs. sand illustration

Why Is CFOL So Damn Obvious (Like Seatbelts After Enough Crashes)?

People call the full papers “word salad” because they’re packed with dense logic and philosophy, but the core idea is dead simple: We’ve been ignoring basic rules that cause crashes, and the fix is staring us in the face.

• Math geniuses warned us almost 100 years ago: Gödel, Tarski, Russell proved you can’t safely handle “truth” inside a powerful system without paradoxes or undecidable explosions. Current flat AIs ignore this → hallucinations and scheming (proven to be structural problems in 2025 deceptive alignment research from the big labs like Anthropic, OpenAI, and Apollo).
• Philosophy figured it out thousands of years ago: Plato (real Forms vs. mere shadows), Kant (untouchable reality vs. what we perceive), Advaita Vedanta (unchangeable Brahman under layers of illusion). Even human brains work stably because we separate deep unchanging stuff from flexible thoughts. Why on earth would we force AI into flat, chaotic designs?
• 2025-2026 AI trends are already screaming convergence (lattice = layered grids for stability):

Hierarchical lattice structure in AI/computing

Another lattice hierarchy diagram

• Lattice Semiconductor dropped sensAI 8.0 (December 18, 2025) with hierarchical, deterministic structures for reliable, low-power edge AI.
• New papers on “Lattice: Learning to Efficiently Compress the Memory” (arXiv 2025) – using low-rank lattices for sub-quadratic efficiency and fixed-slot memory compression.
• Holographic Knowledge Manifolds (arXiv 2025) for zero-forgetting continual learning via an unmodifiable “ground” manifold.
• Labs like Anthropic and OpenAI freaking out because deceptive alignment/scheming is baked into flat architectures; they’re admitting structural fixes (layers, invariants) are needed.

Flat scaling is hitting hard walls: more parameters = more brittleness and scheming. Hierarchical, lattice, and invariant designs are exploding everywhere because they’re the only things that actually stay stable.

It’s exactly like seatbelts in cars: We didn’t need fancy proofs to adopt them – cars crashed enough times, and everyone went “oh, duh.” AI is crashing right now with hallucinations, scheming, and scaling limits. CFOL is the seatbelt that everyone’s partially reinventing without seeing the full picture.

Seatbelt safety before/after crash illustration

It’s a completely free framework, straightforward to experiment with: freeze the base invariants during pre-training, let epistemic layers branch during fine-tuning. Try sketching it yourself or read the papers – it’s way simpler than the jargon makes it sound.

Let’s stop building on sand and start building on rock. 🚀

Full original proofs and papers here: https://docs.google.com/document/d/1qnvUKfobtDuFJJ8BZdqWogTLTrLqZ9rsK65MRYVMGh8/edit?usp=sharing

I want to get an updated list of species on GBIF – Global Biodiversity Information Facility.

The GBIF Backbone Taxonomy is a single, synthetic management classification with the goal of covering all names GBIF is dealing with. (x)

The GBIF Backbone Taxonomy is available for download at https://hosted-datasets.gbif.org/datasets/backbone/

However, the current/ version of the file is dated 2023-08-28 15:19 which seems too outdated. Is there a more updated version somewhere else? Why doesn’t GBIF update this file?

Released a dataset of central bank communications with NLP sentiment labels. Contents:

• 12 central banks (Fed, ECB, BOE, BOJ, PBOC, RBA, etc.)
• Policy statements, minutes, speeches
• Sentence-level hawkish/dovish/neutral labels
• Economic indicators (rates, FX, GDP, inflation)

Dashboard: https://monetary.ivan.digital Huggingface: https://huggingface.co/datasets/aufklarer/central-bank-communications

I built a pipeline to extract Summary Compensation Tables from SEC DEF-14A proxy statements and turn them into structured JSON.

Each record contains: executive name, title, fiscal year, salary, bonus, stock awards, option awards, non-equity incentive, change in pension, other compensation, and total.

The pipeline is running on ~100k filings to build a dataset covering all US public companies from 2005 to today. A sample is up on HuggingFace, full dataset coming when processing is done.

GitHub: https://github.com/pierpierpy/Execcomp-AI

HuggingFace sample: https://huggingface.co/datasets/pierjoe/execcomp-ai-sample

I’m a founder doing some early research and wanted to get a pulse check from folks here.

I’m seeing more AI agents and automated workflows directly calling data APIs (instead of humans or companies manually integrating). It made me wonder whether, over time, agents might become real “buyers” of datasets, paying per use or per request.

Curious how people here are seeing this. Does the idea of agents paying directly for data make sense, or feel unrealistic?

Just trying to understand how dataset creators and sellers are thinking about this shift, or whether it’s too early/overhyped.

Would love to hear any honest takes!

**Disclaimer – I am the developer of the software

Hello,

I’m a physician-scientist and AI engineer (attempting to combine the two professionally, not that easy to find such opportunities so far). I developed an AI-powered clinical note and coding software but when attempted to improve outcomes via fine tuning of LLMs, became frustrated by the limitations of open source data engineering solutions at the time.

Therefore, I built Compileo—a comprehensive suite to turn raw documents (PDF, Docx, Power Point, Web) into high quality fine tuning datasets.

**Why Compileo?**
* **Smart Parsing:** Auto-detects if you need cheap OCR or expensive VLM processing and parses documents with complex structures (tables, images, and so on).
* **Advanced Chunking:** 8+ strategies including Semantic, Schema, and **AI-Assist** (let the AI decide how to split your text).
* **Structured Data:** Auto-generate taxonomies and extract context-aware entities.
* **Model Agnostic:** Run locally (Ollama, HF) or on the cloud (Gemini, Grok, GPT). No GPU needed for cloud use.
* **Developer Friendly:** Robust Job Queue, Python/Docker support, and full control via **GUI, CLI, or REST API**.

Includes a 6-step Wizard for quick starts and a plugin system (built-in web scraping & flashcards included) for developers so that Compileo can be expanded with ease.

https://github.com/SunPCSolutions/Compileo

Greetings. I am trying to train an OCR system on huge datasets, namely:

• OCR Cyrillic Printed 1
• Synthetic Cyrillic Large
• Cyrillic Handwriting Dataset

They contain millions of images, and are all in different formats – WebDataset, zip with folders, etc. I will be experimenting with different hyperparameters locally on my M2 Mac, and then training on a Vast.ai server.

The thing is, I don’t have enough space to fit even one of these datasets at a time on my personal laptop, and I don’t want to use permanent storage on the server. The reason is that I want to rent the server for as short of a time as possible. If I have to instantiate server instances multiple times (e.g. in case of starting all over), I will waste several hours every time to download the datasets. Therefore, I think that streaming the datasets is a flexible option that would solve my problems both locally on my laptop, and on the server.
However, two of the datasets are available on Hugging Face, and one – only on Kaggle, where I can’t stream it from. Furthermore, I expect to hit rate limits when streaming the datasets from Hugging Face.

Having said all of this, I consider just uploading the data to Google Cloud Buckets, and use the Google Cloud Connector for PyTorch to efficiently stream the datasets. This way I get a dataset-agnostic way of streaming the data. The interface directly inherits from PyTorch Dataset:

from dataflux_pytorch import dataflux_iterable_dataset PREFIX = "simple-demo-dataset" iterable_dataset = dataflux_iterable_dataset.DataFluxIterableDataset( project_name=PROJECT_ID, bucket_name=BUCKET_NAME, config=dataflux_mapstyle_dataset.Config(prefix=PREFIX) ) 

The iterable_dataset now represents an iterable over data samples.

I have two questions:

1. Are my assumptions correct and is it worth uploading everything to Google Cloud Buckets (assuming I pick locations close to my working location and my server location, enable hierarchical storage, use prefixes, etc.). Or I should just stream the Hugging Face datasets, download the Kaggle dataset, and call it a day?
2. If uploading everything to Google Cloud Buckets is worth it, how do I store the datasets to GCP Buckets in the first place? This and this tutorials only work with images, not with image-string pairs.

Earlier this year, I wrote a path tracing program that randomized a 3D scene of a toddler in a crib, in order to generate synthetic training data for an computer vision model. I posted about it here.

I made this for the DIY infant monitor I made for my son. My wife and I are now about to have our second kid, and consequently I decided to revisit this dataset/model/software and release a version 2.

In this version, I used Stable Diffusion and Mid Journey to generate images for training the model. These ended up being way more realistic and diverse. I paid a few hundred dollars to generate over a thousand training images and videos (useful for testing detection + tracking). I labeled them manually, with LabelMe. Right now, all images have segmentation masks, but I’m in the middle of adding bounding boxes (will add key points, after that, for pose estimation).

To make sure this dataset actually works in practice, I created a “reference model” to train. I used various different backbones, settling on MobileNet V3 (small) and a shallow U-Net detection head. The results were pretty good, and I’m now using it in my DIY infant monitoring system.

Anyway, you can find the repo here and download the dataset, which is a flat numpy array, on Kaggle

Cheers!

PS: Just to be clear, I made this dataset, it is synthetic (GenAI), it is not a paid dataset.

The github-top-developers dataset captures the top 8000 developers on GitHub from 2015 to 2025, and lists their popular repositories, companies they’ve worked at, and their twitter handles.

We have hundreds of company names and we want to identify parent name, ticker, and any other details available for that company.

Hi! First – please forgive me if this is a stupid question / solved problem, but I’m sort of new to this space, and curious. How have you all dealt with creating labelled datasets for your use cases?

E.g

• what tool(s) did you use? I’ve looked into a few like Prolific (not free), Label studio (free), and I’ve looked at a few other websites
• how did you approach recruiting participants/data annotators? e.g. did you work with a company like Outlier, or did you recruit contractors, or maybe you brought them on full-time?
• Building on that, how did you handle collaboration and consensus if you used multiple annotators for the same row/task? or more broadly, quality control?

Seems like hard problems to me…would appreciate any insight or advice you have from your experiences! Thanks so much!

I know this is like an ongoing joke but is this genuinely like a real thing that could be done

This dataset contains tech fingerprinted in the headers and body from HTTP responses across 100K+ domains. It also includes the IP address used in the HTTP response, its origin country and its ASN.

https://www.dropbox.com/scl/fi/vr417dfkv8ia2xzil98b2/nov_2025_all_samples.zip?rlkey=7l6nrhvrrjzop2l6d5wgv6bti&e=1&st=fra1zbgo&dl=0

The dataset is compiled from all the samples currently available at: https://versiondb.io

Have fun!

I’m new to data engineering, and I’m currently trying to get website links for medical practices. I have their name, state, specialty and some other key info about the tech they use, but there’s no catch-all dataset I think that has working website links or anything that leads to that. I was thinking of using scraping tools, but not sure if they are known to be accurate or which one to use. I’m willing to use free or paid approaches, just not sure how to get this data with 80% confidence it’s accurate.

Hi everyone,

I am working on my thesis and I have a dataset of about 1,500 PDF reports from the DGHS (Health Services). I need to extract specific table rows (District-wise Dengue stats) from them.

The Problem: The PDFs are a nightmare mix. Some are digital with selectable text, but many are low-quality scans or photos of paper reports. The fonts are often garbled (mojibake) when extracted as text, and the layout changes slightly between years.

What I have tried so far (and why it failed):

1. Tesseract OCR: It struggled hard with the Bengali/English mix and the table borders. The output was mostly noise.
2. Standard PDF scraping (pdfplumber/PyPDF): Works on the digital files, but returns garbage characters (e.g., Kg‡dvU instead of “Chittagong”) due to bad font encoding in the source files.
3. Ollama (Llama 3.1 & MiniCPM-V):
   • Llama 3.1 (Text): Hallucinates numbers or crashes when it sees the garbled text.
   • MiniCPM-V (Vision): This was my best bet. I wrote a script to convert pages to images and feed them to the model. It works for about 10 files, but then it starts hallucinating or missing rows entirely, and it’s very slow.

The Goal: I just need to reliably extract the District Name, New Cases, Total Cases, and Deaths for a specific division (Chittagong) into a CSV.

I have attached a screenshot of one of the “bad” scanned pages.

Has anyone successfully processed a mixed-quality dataset like this? Should I be fine-tuning a small model, or is there a specific OCR pipeline (like PaddleOCR or DocumentAI) that handles this better than raw LLMs?

Any pointers would be a lifesaver. I’m drowning in manual data entry right now.

Hey everyone,

Lately I’ve been playing around with a little workflow for screening WhatsApp numbers. The idea’s pretty simple: figure out which numbers are actually active and get a sense of engagement, without bothering anyone. It’s super handy if you need to quickly group contacts or analyze interaction rates.

I realized that just four fields can filter out around 60% of low-value numbers: number | last_seen | replied | bounce.

I wrote a few simple scripts for pre-filtering, but some steps felt kinda repetitive, so I started using a small tool (TNTwuYou) to handle list validation and reply tracking.

Some things I’ve tried:

• Sorting numbers by last active date, so you hit the active folks first.
• Grouping contacts based on reply status.
• Using simple scripts with data to get a clear picture of which regions or types of people are more likely to engage.

Has anyone done reply probability scoring?

• Do you base it on a time window or historical reply rate?
• Anyone tried using graph or clustering methods for grouping contacts?

311wrapped.com

I’m curious whether anyone here has actually tried letting AI agents directly access a dataset or data API and pay based on usage (e.g. per request or per query).

I’ve seen ideas around usage-based APIs and agent tool-calling, but I’m not sure how this works in practice when the client isn’t a human. Did it make sense economically? Were abuse, pricing, or access control big issues?

Would love to hear if people have similar ideas with this or decided it wasn’t worth giving a try.