\u200b<\/p>\n
I have a ~59 GB Parquet dataset stored in Google Drive, and I want to use it for ML training on Kaggle GPUs.<\/p>\n
My folder structure is:<\/p>\n
My Drive\/<\/p>\n
\u2514\u2500\u2500 DATASET\/<\/p>\n
\u2514\u2500\u2500 data\/<\/p>\n
\u2514\u2500\u2500 processed\/<\/p>\n
\u2514\u2500\u2500 era5_land_preprocessed\/<\/p>\n
\u251c\u2500\u2500 train\/ \u2192 60 Parquet files \u2192 44.34 GB<\/p>\n
\u251c\u2500\u2500 validation\/ \u2192 12 Parquet files \u2192 8.83 GB<\/p>\n
\u2514\u2500\u2500 test\/ \u2192 8 Parquet files \u2192 5.82 GB<\/p>\n
Total: 80 Parquet files, ~58.99 GB<\/p>\n
I don’t want to download the dataset to my local computer. I was planning to use Google Colab + Kaggle API to transfer the files directly from Google Drive to a Kaggle Dataset.<\/p>\n
My questions are:<\/p>\n
- \n
- \n
Is uploading a ~59 GB dataset to Kaggle this way reliable?<\/p>\n<\/li>\n
- \n
Is Google Drive \u2192 Colab \u2192 Kaggle API the best approach?<\/p>\n<\/li>\n
- \n
Should I create one Kaggle dataset containing “train\/”, “validation\/”, and “test\/”, or split them into multiple Kaggle datasets?<\/p>\n<\/li>\n
- \n
Are there any problems with uploading large Parquet files to Kaggle?<\/p>\n<\/li>\n
- \n
Is there a better\/faster method for transferring this amount of data to Kaggle?<\/p>\n<\/li>\n<\/ol>\n
The dataset will be used for climate\/weather ML forecasting, and I need to train models using Kaggle’s GPUs.<\/p>\n
Any advice from people who have handled 50+ GB datasets on Kaggle would be appreciated.<\/p>\n<\/div>\n
submitted by \/u\/Aesthete_fellow <\/a>
[link]<\/a><\/span> [comments]<\/a><\/span><\/p>
