I am trying to load an OpenNeuro dataset into Google Colab to train a model. Based on the website, the dataset size is said to be 13.46 GB, which can definitely be accommodated by the free version of Colab which usually has around 50 GB of free disk space. I first attempted to download it using AWS CLI by running
!pip install awscli !aws s3 sync –no-sign-request s3://openneuro.org/ds003949 ds003949-download/
But the process terminated as Colab ran out of disk space.
I then attempted to download with openneuro-py, and shrink my download range to just the derivatives folder.
!pip install openneuro-py !openneuro-py download –dataset=ds003949 –include=derivatives/*
Again, I ran out of disk space before the download finished.
I am new to OpenNeuro so I don’t know how their datasets work exactly, or how to get the “true” dataset size. I tried loading a smaller 6 GB dataset into Colab with the above methods, and the dataset size did match what was stated on the website). I have minimal storage on my local hardware so I would like to try getting it loaded into Colab first before I attempt that route.
Would appreciate some help or advice on what I did wrong from anyone with experience working with OpenNeuro or neuroimaging data. Thanks!
submitted by /u/botsunny
[link] [comments]