Made A Fast Go Downloader For Massive Files (beats Aria2 By 1.4x)

Hey guys, we’re a couple of CS students who got annoyed with slow single-connection downloads, so we built Surge. Figured the datasets crowd might find it handy for scraping huge CSVs or image directories.

It’s a TUI download manager, but it also has a headless server mode which is perfect if you just want to leave it running on a VPS to pull data overnight.

• It splits files and maximizes bandwidth by using parallel chunk downloading.
• It is much more stable and fast than using a browser like Chrome or Firefox!
• You can use it remotely (over LAN for something like a home lab)
• You can deploy it easily via Docker compose.
• We benched it against standard tools and it beat aria2c by about 1.38x, and was over 2x faster than wget.

Check it out if you want to speed up your data scraping pipelines.

GH: github.com/surge-downloader/surge