Fine tune LLAMA 3.2 1B on Python dataset

[dtdo90]

After reading Sebastian's book, I’m more excited than ever to explore real-world applications of LLMs! I am the type who needs to understand the model's structure before I could confidently dive into any type of project, and this book made everything crystal clear!

My first project is to fine-tune LLAMA 3.2 on a Python dataset. The code is available in my repo for anyone interested https://github.com/dtdo90/Llama3.2_python_dataset.

[rasbt]

Nice, thanks for sharing @dtdo90 . I noticed you were using HF transformers. This is totally reasonable for real-world applications. But in case you are curious how this model looks like when implemented from scratch, I do have some bonus material on it here :) https://github.com/rasbt/LLMs-from-scratch/tree/main/ch05/07_gpt_to_llama

[dtdo90]

Thank you @rasbt for the suggestion! Your bonus materials is gold and I've learnt a lot!

I actually re-coded your repo before having the confidence to do anything with Llama. It would be super to see how to incorporate quantization in QLoRA into the model coded from the scratch. I am not sure how this could be done without the help of HF transformers.

[rasbt]

Good point, I haven't had a chance to add quantization here yet. The most popular library for quantization and QLoRA is bitsandbytes (I think transformers may use it under the hood): https://github.com/bitsandbytes-foundation/bitsandbytes