The Promise vs. Reality of QLoRA
Everyone loves QLoRA until their model starts speaking nonsense after 500 steps. We tried fine-tuning a 70B parameter model for a highly specific legal text summarization task using the Hugging Face PEFT (Parameter-Efficient Fine-Tuning) library.
We set the rank ($r$) to 64 and alpha to 16. The training loss curves on Weights & Biases looked absolutely beautiful, steadily decreasing with no spikes. But when we deployed the adapter to our staging environment, the qualitative evaluation was a disaster. The model had catastrophically forgotten how to format basic JSON objects and was hallucinating citations.
The Architectural Flaw
The issue stems from the extreme quantization (4-bit NormalFloat) combined with a rank that was too low to capture the highly structured domain knowledge required for legal texts. The low-rank updates were essentially overwriting the model's fundamental instruction-following pathways.
from peft import LoraConfig, get_peft_model
# WHAT FAILED:
# config = LoraConfig(r=64, lora_alpha=16, target_modules=["q_proj", "v_proj"])
# WHAT WORKED (DoRA + Full target modules):
config = LoraConfig(
r=128,
lora_alpha=64,
target_modules="all-linear",
use_dora=True # Weight-Decomposed Low-Rank Adaptation
)The Fix
We had to switch to DoRA (Weight-Decomposed Low-Rank Adaptation), which decouples the magnitude and direction of the pre-trained weights. Furthermore, we had to carefully curate our instruction mix to include 20% general domain instruction data (from OpenHermes-2.5) to prevent catastrophic forgetting. The lesson? Don't blindly trust loss curves.