AI Terms Dictionary

Definition

LoRA freezes pre-trained model weights and inserts trainable decomposition matrices into each layer of the Transformer architecture. By optimizing only these low-rank matrices, LoRA significantly reduces the number of trainable parameters, memory footprint, and computational cost during fine-tuning. This technique allows for rapid adaptation to specific downstream tasks while maintaining the general knowledge of the base model, making it highly popular for efficient custom model training.

Summary

Low-Rank Adaptation is a parameter-efficient fine-tuning method that injects trainable rank decomposition matrices into existing model weights.

Key Concepts

• Parameter-Efficient Fine-Tuning
• Rank Decomposition
• Freezing Weights
• Adapter Modules

Use Cases

• Customizing LLMs
• Domain-Specific Adaptation
• Resource-Constrained Training

Code Example

1
2
3

from peft import LoraConfig, get_peft_model
config = LoraConfig(r=8, lora_alpha=32)
model = get_peft_model(base_model, config)

Related Terms

• PEFT
• Fine-Tuning
• Quantization