nileshkendre/llama2-7b-qlora-sft

Llama-2-7B QLoRA Supervised Fine-Tuning (SFT)

Overview

This repository contains a Parameter-Efficient Fine-Tuned (PEFT) version of the Llama-2-7B Chat model using the QLoRA approach. The model was instruction-tuned using the Guanaco LLaMA2 1K dataset with the Hugging Face TRL SFTTrainer.

The objective of this project was to understand and implement an end-to-end Large Language Model (LLM) fine-tuning workflow using modern open-source tools, including Transformers, PEFT, BitsAndBytes, and TRL.

---

Model Details

Item	Value
Base Model	NousResearch/Llama-2-7b-chat-hf
Fine-Tuning Method	PEFT
Technique	QLoRA
Training Method	Supervised Fine-Tuning (SFT)
Framework	Hugging Face Transformers
Trainer	TRL SFTTrainer
Quantization	4-bit
Dataset	mlabonne/guanaco-llama2-1k

---

Dataset

This model was fine-tuned using the mlabonne/guanaco-llama2-1k instruction dataset available on Hugging Face.

The dataset contains approximately 1,000 instruction-response pairs covering a wide range of tasks, including:

• Question Answering
• General Knowledge
• Reasoning
• Coding
• Writing Assistance
• Conversational AI
• Instruction Following

The dataset is designed for supervised instruction tuning of chat-based Large Language Models.

---

Training Pipeline

The complete training workflow consisted of the following stages:

1. Load the Llama-2-7B Chat base model.
2. Load the tokenizer.
3. Quantize the base model to 4-bit using BitsAndBytes.
4. Configure PEFT using LoRA adapters.
5. Prepare the Guanaco instruction dataset.
6. Perform Supervised Fine-Tuning (SFT) using TRL's SFTTrainer.
7. Save the trained LoRA adapter.
8. Push the trained adapter to the Hugging Face Hub.

---

Training Configuration

Parameter	Value
Epochs	1
Batch Size	1
Gradient Accumulation Steps	2
Learning Rate	2e-4
Optimizer	paged_adamw_32bit
Weight Decay	0.001
Max Gradient Norm	0.3
Warmup Ratio	0.03
Quantization	4-bit
LoRA Rank (r)	64
LoRA Dropout	0.1

---

Why PEFT?

Parameter-Efficient Fine-Tuning (PEFT) enables efficient adaptation of Large Language Models by training only a small set of additional parameters instead of updating the entire model. This significantly reduces GPU memory consumption, training time, and storage requirements while maintaining competitive performance.

---

Why QLoRA?

QLoRA combines Low-Rank Adaptation (LoRA) with 4-bit quantization using the BitsAndBytes library. By loading the frozen base model in 4-bit precision and training only the LoRA adapter weights, QLoRA enables fine-tuning of billion-parameter language models on limited GPU resources.

---

Repository Structure

adapter_config.json
adapter_model.safetensors
README.md

---

How to Load the Model

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    "NousResearch/Llama-2-7b-chat-hf"
)

tokenizer = AutoTokenizer.from_pretrained(
    "NousResearch/Llama-2-7b-chat-hf"
)

model = PeftModel.from_pretrained(
    base_model,
    "nileshkendre/llama2-7b-qlora-sft"
)

---

Skills Demonstrated

• Large Language Models (LLMs)
• Hugging Face Transformers
• PEFT (Parameter-Efficient Fine-Tuning)
• LoRA
• QLoRA
• Supervised Fine-Tuning (SFT)
• TRL SFTTrainer
• BitsAndBytes Quantization
• Hugging Face Hub
• Model Publishing

---

Limitations

This repository contains only the LoRA adapter weights and not the original Llama-2-7B base model.

The model was fine-tuned on the public mlabonne/guanaco-llama2-1k instruction dataset containing approximately 1,000 instruction-response pairs. It is intended to demonstrate the end-to-end PEFT/QLoRA fine-tuning workflow and should be considered an educational portfolio project rather than a production-ready conversational assistant.

Additional fine-tuning, evaluation, and safety testing would be required before deploying it in production.

---

Future Work

Future enhancements include:

• Fine-tuning on larger instruction datasets
• Hyperparameter optimization
• Multi-epoch training
• Domain-specific instruction tuning
• Retrieval-Augmented Generation (RAG) integration
• Evaluation using standard LLM benchmarks
• Merging LoRA adapters with the base model for deployment

---

Acknowledgements

• Hugging Face Transformers
• Hugging Face TRL
• Hugging Face PEFT
• BitsAndBytes
• NousResearch Llama-2-7B Chat
• mlabonne Guanaco Dataset

---

Author

Nilesh Kendre

AI Engineer | Generative AI | LLMs | NLP | RAG | Agentic AI

This repository showcases hands-on implementation of PEFT, QLoRA, and Supervised Fine-Tuning (SFT) using the Hugging Face ecosystem.