lora_scale #3
Description
Great work!
I noticed in your lora_switching_module.py line 39, you have lora_module.set_scale(active_adapter,self.active_adapter_scales[i][active_adapter]).
I think this should be lora_module.scaling[active_adapter] = self.active_adapter_scales[i][active_adapter].
Otherwise, the scaling continously changes if rank is not equal to alpha.