Something I keep running into when observing and working with long-term climate datasets: the data exists, it’s publicly available and can be easily accessed, reasonably well-documented, and yet asking a simple cross-decade question still takes an unreasonable amount of work to actually answer.
For example if I want to ask something like “how has the distribution of extreme precipitation events shifted since 1980 across the Gulf Coast?”, it will take forever to actually find an answer. Starting off with finding the right NOAA dataset, then figuring out which version covers your full time range, dealing with the fact that station coverage changed significantly in the 90s, reconciling pre and post-satellite era measurement methodologies, writing custom scripts to normalize format differences across dataset versions, then finally doing the actual analysis.
Do you think with the AI wave we are seeing, this entire process or any part of it will be improved? NOAA NCEI alone archives over 229 terabytes of data every month across 130+ observing platforms, but the part of analyzing and understanding those data, I feel like, still lags behind.
Are you also mostly doing Python + manual joins and custom preprocessing per dataset? Has anything changed in how you approach cross-decade multi-source queries, or is it still bespoke work every time?
submitted by /u/_tnhii
[link] [comments]
