“Streaming datasets: 100x More Efficient” is a new blog post sharing improvements on dataset streaming to train AI models<\/p>\n
link: https:\/\/huggingface.co\/blog\/streaming-datasets<\/a><\/p>\n Summary of the blog post:<\/p>\n We boosted there is also a 1min video explaining the impact of this: https:\/\/x.com\/andimarafioti\/status\/1982829207471419879<\/a><\/p>\n<\/div>\n submitted by \/u\/qlhoest <\/a> \n
load_dataset('dataset', streaming=True)<\/code>, streaming datasets without downloading them with one line of code! Start training on multi-TB datasets immediately, without complex setups, downloading, no “disk out of space”, or 429 \u201cstop requesting!\u201d errors.
It’s super fast! Outrunning our local SSDs when training on 64xH100 with 256 workers downloading data. We’ve improved streaming to have 100x fewer requests, \u2192 10\u00d7 faster data resolution \u2192 2x sample\/sec, \u2192 0 worker crashes at 256 concurrent workers.<\/p>\n<\/blockquote>\n
[link]<\/a><\/span> [comments]<\/a><\/span><\/p>
