{"id":41945,"date":"2026-08-03T21:27:11","date_gmt":"2026-08-03T19:27:11","guid":{"rendered":"https:\/\/www.graviton.at\/letterswaplibrary\/i-built-an-open-source-pipeline-to-make-the-entire-gdelt-2-0-events-archive-easier-to-use-self-promotion\/"},"modified":"2026-08-03T21:27:11","modified_gmt":"2026-08-03T19:27:11","slug":"i-built-an-open-source-pipeline-to-make-the-entire-gdelt-2-0-events-archive-easier-to-use-self-promotion","status":"publish","type":"post","link":"https:\/\/www.graviton.at\/letterswaplibrary\/i-built-an-open-source-pipeline-to-make-the-entire-gdelt-2-0-events-archive-easier-to-use-self-promotion\/","title":{"rendered":"I Built An Open-source Pipeline To Make The Entire GDELT 2.0 Events Archive Easier To Use [self-promotion]"},"content":{"rendered":"

<\/p>\n

\n

Hi everyone,<\/p>\n

I’m a PhD candidate working in AI, and my research relies heavily on the GDELT 2.0 Events<\/strong> dataset.<\/p>\n

While GDELT is an incredible resource, I found that the first hurdle was simply getting the data into a usable format<\/em>. The historical archive consists of thousands of compressed files spanning more than 45 years, and I kept repeating the same preprocessing steps every time I started a new experiment.<\/p>\n

So I built GdeltForge<\/a>, an open-source pipeline that automates that process.<\/p>\n

It downloads the complete GDELT Events archive, verifies file integrity, converts everything to Apache Parquet, filters invalid records, and can generate reproducible samples for machine learning and data analysis.<\/p>\n

Originally this was just a tool for my own research, but I thought it might be useful for others working with GDELT, or anyone interested in building datasets from large public archives.<\/p>\n

The project is still in beta, and I’d love feedback from people who regularly work with large datasets:<\/p>\n