Hey amazing people! First post here! Today, I’m excited to announce that you can now train your own reasoning model with just 5GB VRAM for Qwen2.5 (1.5B) using our open-source project Unsloth: https:\/\/github.com\/unslothai\/unsloth<\/a> <\/p>\n GRPO is the algorithm behind DeepSeek-R1 and how it was trained. You need a dataset with about 500 rows in question, answer pairs and a reward function and you can then start the whole process!<\/p>\n This allows any open LLM like Llama, Mistral, Phi etc. to be converted into a reasoning model with chain-of-thought process. The best part about GRPO is it doesn’t matter if you train a small model compared to a larger model as you can fit in more faster training time compared to a larger model so the end result will be very similar! You can also leave GRPO training running in the background of your PC while you do other things!<\/p>\n Due to our newly added Efficient GRPO algorithm, this enables 10x longer context<\/strong> lengths while using 90% less VRAM<\/strong> vs. every other GRPO LoRA\/QLoRA (fine-tuning) implementations with 0 loss in accuracy. With a standard GRPO setup, Llama 3.1 (8B) training at 20K context length demands 510.8GB of VRAM. However, Unsloth\u2019s 90% VRAM reduction brings the requirement down to just 54.3GB in the same setup. We leverage our gradient checkpointing<\/a> algorithm which we released a while ago. It smartly offloads intermediate activations to system RAM asynchronously whilst being only 1% slower. This shaves a whopping 372GB VRAM<\/strong> since we need num_generations = 8. We can reduce this memory usage even further through intermediate gradient accumulation. Use our GRPO notebook with 10x longer context using Google’s free GPUs: Llama 3.1 (8B) on Colab<\/a>-GRPO.ipynb) <\/p>\n Blog for more details on the algorithm, the Maths behind GRPO, issues we found and more: https:\/\/unsloth.ai\/blog\/grpo<\/a>)<\/p>\n GRPO VRAM Breakdown:<\/p>\n Metric Unsloth TRL + FA2 Training Memory Cost (GB) 42GB 414GB GRPO Memory Cost (GB) 9.8GB 78.3GB Inference Cost (GB) 0GB 16GB Inference KV Cache for 20K context (GB) 2.5GB 2.5GB Total Memory Usage 54.3GB (90% less)<\/strong> 510.8GB<\/strong> <\/p>\n Also we spent a lot of time on our Guide (with pics) for everything on GRPO + reward functions\/verifiers so would highly recommend you guys to read it: docs.unsloth.ai\/basics\/reasoning<\/a><\/p>\n Thank you so so much for reading! \ud83d\ude00<\/p>\n<\/div>\n submitted by \/u\/yoracale <\/a>
[link]<\/a><\/span> [comments]<\/a><\/span><\/p>
