-
Notifications
You must be signed in to change notification settings - Fork 256
PEFT integration #27
Copy link
Copy link
Closed
Description
Compatibility with https://docs.nvidia.com/nemo-framework/user-guide/latest/nemo-2.0/features/peft.html
- Port over walk utilities to run functions over modules: [peft] Port walking utils for peft integration #43
- Port over AdapterWrapper and LoRA linear layers and utilities: [peft] Port base adapter wrapper and lora utils #44
- Port over ModuleMatcher: [peft] Port module matcher over from NeMo #68
- Define PeFT base class + LoRA to apply peft over the whole model: [peft] Define PEFT base class and LoRA transform #71
- Integrate into config container / model initialization: load from pretrained, apply peft transformation: Integrate peft into training loop #94
- Optimize checkpointing flow: saving adapters params, loading adapters + optimizer states only: Integrate peft into training loop #94
- Add DoRA: [peft] Port DoRA support #82
- Add Canonical LoRA: [peft] Port canonical LoRA #83
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels