Hi everyone,
I’m a PhD candidate working in AI, and my research relies heavily on the GDELT 2.0 Events dataset.
While GDELT is an incredible resource, I found that the first hurdle was simply getting the data into a usable format. The historical archive consists of thousands of compressed files spanning more than 45 years, and I kept repeating the same preprocessing steps every time I started a new experiment.
So I built GdeltForge, an open-source pipeline that automates that process.
It downloads the complete GDELT Events archive, verifies file integrity, converts everything to Apache Parquet, filters invalid records, and can generate reproducible samples for machine learning and data analysis.
Originally this was just a tool for my own research, but I thought it might be useful for others working with GDELT, or anyone interested in building datasets from large public archives.
The project is still in beta, and I’d love feedback from people who regularly work with large datasets:
- Is there functionality you’d expect from a tool like this?
- Are there preprocessing steps you usually perform before analysis that could be automated?
- What would make it more useful for your own research or projects?
My goal is to eventually publish it on PyPI once it’s been tested by a broader community, so any feedback or contributions would be greatly appreciated.
If you’re someone who has worked with GDELT before, I’d be especially interested in hearing about your workflow and the pain points you’ve encountered.
submitted by /u/Hendrik-Lorentz
[link] [comments]
