Recently I built a dataset of hundreds of millions of tables, crawled from the Internet and open data providers, to train an AI tabular foundation model. Searching through the datasets is super difficult, b\/c off-the-shelf tech just doesn’t exist for searching through messy tables at that scale.<\/p>\n
So I’ve been working on this side project, Gini. It has subsets of FRED and data.gov–I’m trying to keep the data manageably small so I can iterate faster, while still being interesting. I picked a random time slice from data.gov<\/a> so there’s some bias towards Pennsylvania and Virginia. But if it looks worthwhile, I can easily backfill a lot more datasets. <\/p>\n Currently it does a table-level hybrid search, and each result has customizable visualizations of the dataset (this is hit-or-miss, it’s just a proof-of-concept).<\/p>\n I’ve also built column-level vector indexes with some custom embedding models I’ve made. It’s not surfaced in the UI yet–the UX is difficult. But it lets me rank results by “joinability”–I’ll add it to the UI this week. Then you could start from one table (your own or a dataset you found via search) and find tables to join with it. This could be like “enrichment” data, joining together different years of the same dataset, etc.<\/p>\n Eventually I’d like to be able to find, clean & prep & join, and build up nice visualizations by just clicking around in the UI.<\/p>\n Anyway, if this looks promising, let me know and I’ll keep building. Or tell me why I should give up!<\/p>\n https:\/\/app.ginidata.com\/<\/a><\/p>\n Fun tech details: I run a data pipeline that crawls and extracts tables from lots of formats (CSVs, HTML, LaTeX, PDFs, digs inside zip\/tar\/gzip files, etc.) into a standard format, post-processes the tables to clean them up and classify them and extract metadata, then generate embeddings and index them. I have lots of other data sources already implemented, like I’ve already extracted tables from all research papers in arXiv so that you can search research tables from papers.<\/p>\n (I don’t make any money from this and I’m paying for this myself. I’d like to find a sustainable business model, but “charging for search” is not something I’m interested in…)<\/p>\n<\/div>\n submitted by \/u\/9us <\/a>
[link]<\/a><\/span> [comments]<\/a><\/span><\/p>
