Greetings. I am trying to train an OCR system on huge datasets, namely:<\/p>\n
- \n
- OCR Cyrillic Printed 1<\/a><\/li>\n
- Synthetic Cyrillic Large<\/a><\/li>\n
- Cyrillic Handwriting Dataset<\/a><\/li>\n<\/ul>\n
They contain millions of images, and are all in different formats –
WebDataset<\/code>,zip<\/code> with folders, etc. I will be experimenting with different hyperparameters locally on my M2 Mac, and then training on a Vast.ai<\/a> server.<\/p>\nThe thing is, I don’t have enough space to fit even one of these datasets at a time on my personal laptop, and I don’t want to use permanent storage on the server. The reason is that I want to rent the server for as short of a time as possible. If I have to instantiate server instances multiple times (e.g. in case of starting all over), I will waste several hours every time to download the datasets. Therefore, I think that streaming the datasets is a flexible option that would solve my problems both locally on my laptop, and on the server.
However, two of the datasets are available on Hugging Face, and one – only on Kaggle, where I can’t stream it from. Furthermore, I expect to hit rate limits when streaming the datasets from Hugging Face.<\/p>\nHaving said all of this, I consider just uploading the data to Google Cloud Buckets, and use the
Google Cloud Connector for PyTorch<\/code> to efficiently stream the datasets. This way I get a dataset-agnostic way of streaming the data. The interface directly inherits from PyTorchDataset<\/code>:<\/p>\nfrom dataflux_pytorch import dataflux_iterable_dataset PREFIX = \"simple-demo-dataset\" iterable_dataset = dataflux_iterable_dataset.DataFluxIterableDataset( project_name=PROJECT_ID, bucket_name=BUCKET_NAME, config=dataflux_mapstyle_dataset.Config(prefix=PREFIX) ) <\/code><\/pre>\n\n
The
iterable_dataset<\/code> now represents an iterable over data samples.<\/p>\n<\/blockquote>\nI have two questions:<\/p>\n
- \n
- Are my assumptions correct and is it worth uploading everything to Google Cloud Buckets (assuming I pick locations close to my working location and my server location, enable hierarchical storage, use prefixes, etc.). Or I should just stream the Hugging Face datasets, download the Kaggle dataset, and call it a day?<\/li>\n
- If uploading everything to Google Cloud Buckets is worth it, how do I store the datasets to GCP Buckets in the first place? This<\/a> and this<\/a> tutorials only work with images, not with image-string pairs.<\/li>\n<\/ol><\/div>\n
submitted by \/u\/Suspicious-Pick-7961 <\/a>
[link]<\/a><\/span> [comments]<\/a><\/span><\/p>
- Synthetic Cyrillic Large<\/a><\/li>\n
