LoRA: Low-Rank Adaptation of Large Language Models
Freeze the model and train a thin correction beside each weight, because the change fine-tuning makes turns out to need only a few directions.
Read the original ↗A Microsoft paper from 2021 about fine-tuning GPT-3 without making a 350 GB copy of it for every task. It proposes one small change to how a weight update is stored, and then spends its most interesting section asking how big that update ever needed to be.
Included with a plan
This discussion is included with a plan. The first three lessons of every track are free, with their boards and videos. A plan unlocks every lesson, board, video, lab and paper for one price.
Related on chalk
The same idea as a board, lab, video or lesson.
- BoardFine-tuning and AlignmentW plus B times A drawn out, with the 256-times count for one 4096 by 4096 matrix at rank 8
- LabThe LoRA rankthe rank dial on query and value of a 7B, the paper's own choice of matrices, with the frozen base's memory the adapter cannot remove
- VideoWhat fine-tuning actually is: prompting, retrieval, LoRA and the order to try them inthe tracing-paper chapter: frozen base, two thin matrices, a small swappable file, and QLoRA after it
- LessonFine-tuning & AlignmentUses the board Fine-tuning and Alignment
- LessonA LoRA RunUses the board Fine-tuning and Alignment
Further reading
Read the original for the full argument and its evidence.
- The paper (arXiv) ↗Version 2, published at ICLR 2022. Read section 4 for the method and all of section 7; the rank table and its footnote are the part people cite without reading.
- QLoRA: Efficient Finetuning of Quantized LLMs ↗Dettmers and others, 2023. The frozen base stored in 4 bits, and the finding that adapting query and value alone was not enough to match full fine-tuning.
- LoRA Learns Less and Forgets Less ↗Biderman and others, 2024. The careful comparison on code and mathematics, and the measured rank of what full fine-tuning actually learns.