I\u2019ve been collaborating with a colleague on a project that involves some pretty hefty datasets, and moving them back and forth has been a headache. Some of the files are 50\u2013100GB each, and in total we\u2019re looking at hundreds of gigabytes. Standard cloud storage options don\u2019t seem built for this either they throttle speeds, enforce strict limits, or require subscriptions that don\u2019t make sense for one off transfers.<\/p>\n
We\u2019ve tried compressing and splitting files, but that just adds more time and confusion when the recipient has to reassemble everything. Mailing drives might be reliable, but it feels outdated and isn\u2019t practical when you need results quickly. Ideally, I\u2019d like something that\u2019s both fast and secure, since we\u2019re dealing with research data.<\/p>\n
For those of you who routinely share large datasets across universities, labs, or organizations what\u2019s worked best in your experience? Do you stick with institutional servers and FTP setups, or is there a practical modern tool for big dataset transfers?<\/p>\n<\/div>\n
submitted by \/u\/d4rk_diamond <\/a>
[link]<\/a><\/span> [comments]<\/a><\/span><\/p>
