github
/
minimind
mirror of https://github.com/jingyaogong/minimind.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Modify LoRA application conditions in model_lora.py 

原本的lora只作用于方阵上，也就是q_proj和o_proj。修改完会同时作用于qkvo
pull/570/head
Ltimbe 2025-12-05 21:13:42 +08:00 committed by GitHub
parent 5e1447b913
commit be979ec9e7
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
model/model_lora.py
View File

@ -20,8 +20,8 @@ class LoRA(nn.Module):
def apply_lora(model, rank=8): def apply_lora(model, rank=8):
for name, module in model.named_modules(): for name, module in model.named_modules():
if isinstance(module, nn.Linear) and module.weight.shape[0] == module.weight.shape[1]: if 'lora' not in name and ('q_proj' in name or 'k_proj' in name or 'v_proj' in name or 'o_proj' in name):
lora = LoRA(module.weight.shape[0], module.weight.shape[1], rank=rank).to(model.device) lora = LoRA(module.weight.shape[1], module.weight.shape[0], rank=rank).to(model.device)
setattr(module, "lora", lora) setattr(module, "lora", lora)
original_forward = module.forward original_forward = module.forward