Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support finetuning base model weights in QAT + LoRA flow #2089

Open
Tracked by #2008
ebsmothers opened this issue Nov 27, 2024 · 0 comments
Open
Tracked by #2008

Support finetuning base model weights in QAT + LoRA flow #2089

ebsmothers opened this issue Nov 27, 2024 · 0 comments
Assignees

Comments

@ebsmothers
Copy link
Contributor

Now that the QAT + LoRA recipe has landed in #1931, we can support a finetuning flow like the one used to generate the quantized Llama 3.2 1B and 3B checkpoints (see e.g. the 1B checkpoint here). Unlike traditional LoRA, one path for finetuning with QAT + LoRA involves updating both the LoRA weights and the base model weights (with the fake quantization operation), as referenced in this blog. We should add the option in our QAT + LoRA recipe to make all params trainable, not just the LoRA ones. This can be done by modifying the call to set_trainable_params here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants