{"id":34278,"date":"2025-06-10T07:27:43","date_gmt":"2025-06-10T05:27:43","guid":{"rendered":"https:\/\/www.graviton.at\/letterswaplibrary\/self-promotion-i-processed-and-standardized-16-7tb-of-sec-filings\/"},"modified":"2025-06-10T07:27:43","modified_gmt":"2025-06-10T05:27:43","slug":"self-promotion-i-processed-and-standardized-16-7tb-of-sec-filings","status":"publish","type":"post","link":"https:\/\/www.graviton.at\/letterswaplibrary\/self-promotion-i-processed-and-standardized-16-7tb-of-sec-filings\/","title":{"rendered":"[self-promotion] I Processed And Standardized 16.7TB Of SEC Filings"},"content":{"rendered":"

<\/p>\n

\n

SEC data is submitted in a format called Standardized Generalized Markup Language. A SGML Submission may contain many different files. For example, this Form 4<\/a> contains xml and txt files. This isn’t really important unless you want to work with a lot of data, e.g. the entire SEC corpus.<\/p>\n

If you do want to work with a lot of SEC data, your choice is either to buy the parsed SGML data or get it from the SEC’s website.<\/p>\n

Scraping the data is slow. The SEC rate limits you to 5 request per second for extended durations. There are about 16,000,000 submissions so this takes awhile. A much faster approach is to download the bulk data files here<\/a>. However, these files are in SGML form.<\/p>\n

I’ve written a fast SGML parser <\/a>here under the MIT License. The parser has been tested on the entire corpus, with > 99.99% correctness. This is about as good as it gets, as the remaining errors are mostly due to issues on the SEC’s side. For example, some files have errors, especially in the pre 2001 years.<\/p>\n

Some stats about the corpus:<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
File Type<\/th>\nTotal Size (Bytes)<\/th>\nFile Count<\/th>\nAverage Size (Bytes)<\/th>\n<\/tr>\n<\/thead>\n
htm<\/td>\n7,556,829,704,482<\/td>\n39,626,124<\/td>\n190,703.23<\/td>\n<\/tr>\n
xml<\/td>\n5,487,580,734,754<\/td>\n12,126,942<\/td>\n452,511.5<\/td>\n<\/tr>\n
jpg<\/td>\n1,760,575,964,313<\/td>\n17,496,975<\/td>\n100,621.73<\/td>\n<\/tr>\n
pdf<\/td>\n731,400,163,395<\/td>\n279,577<\/td>\n2,616,095.61<\/td>\n<\/tr>\n
xls<\/td>\n254,063,664,863<\/td>\n152,410<\/td>\n1,666,975.03<\/td>\n<\/tr>\n
txt<\/td>\n248,068,859,593<\/td>\n4,049,227<\/td>\n61,263.26<\/td>\n<\/tr>\n
zip<\/td>\n205,181,878,026<\/td>\n863,723<\/td>\n237,555.19<\/td>\n<\/tr>\n
gif<\/td>\n142,562,657,617<\/td>\n2,620,069<\/td>\n54,411.8<\/td>\n<\/tr>\n
json<\/td>\n129,268,309,455<\/td>\n550,551<\/td>\n234,798.06<\/td>\n<\/tr>\n
xlsx<\/td>\n41,434,461,258<\/td>\n721,292<\/td>\n57,444.78<\/td>\n<\/tr>\n
xsd<\/td>\n35,743,957,057<\/td>\n832,307<\/td>\n42,945.64<\/td>\n<\/tr>\n
fil<\/td>\n2,740,603,155<\/td>\n109,453<\/td>\n25,039.09<\/td>\n<\/tr>\n
png<\/td>\n2,528,666,373<\/td>\n119,723<\/td>\n21,120.97<\/td>\n<\/tr>\n
css<\/td>\n2,290,066,926<\/td>\n855,781<\/td>\n2,676.0<\/td>\n<\/tr>\n
js<\/td>\n1,277,196,859<\/td>\n855,781<\/td>\n1,492.43<\/td>\n<\/tr>\n
html<\/td>\n36,972,177<\/td>\n584<\/td>\n63,308.52<\/td>\n<\/tr>\n
xfd<\/td>\n9,600,700<\/td>\n2,878<\/td>\n3,335.89<\/td>\n<\/tr>\n
paper<\/td>\n2,195,962<\/td>\n14,738<\/td>\n149.0<\/td>\n<\/tr>\n
frm<\/td>\n1,316,451<\/td>\n417<\/td>\n3,156.96<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

The SGML parsing package<\/a>, Stats on processing the corpus,<\/a> convenience package for SEC data<\/a>. <\/p>\n<\/div>\n

submitted by \/u\/status-code-200 <\/a>
[link]<\/a><\/span> [comments]<\/a><\/span><\/p>