Fix: Ensure main learning rate is used in LoRA training #3264
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The GUI logic was preventing the main learning rate from being passed to the training script if text_encoder_lr or unet_lr was set. This caused issues with optimizers like Prodigy, which might default to a very small LR if the main LR isn't provided. This commit modifies kohya_gui/lora_gui.py to ensure the main learning_rate is always included in the parameters passed to the training script, allowing optimizers to use your specified main LR, TE LR, and UNet LR correctly.
This commit refactors the learning rate (LR) handling in `kohya_gui/lora_gui.py` for LoRA training. The previous fix for LR misinterpretation involved commenting out a line. This commit completes the cleanup by: - Removing the `do_not_set_learning_rate` variable and its associated conditional logic, which became redundant. - Renaming the float-converted `learning_rate` to `learning_rate_float` for clarity. - Ensuring that `learning_rate_float` and the float-converted `unet_lr_float` are consistently used when preparing the `config_toml_data` for the training script. This makes the code cleaner and the intent of always passing the main learning rate (along with specific TE/UNet LRs) more direct. The functional behavior of the LR fix remains the same.
This commit introduces a helper function, `get_effective_lr_messages`, into `kohya_gui/lora_gui.py` and integrates it into the `train_model` function. The purpose is to provide you with clearer information about how the learning rates set in the GUI (Main LR, Text Encoder LR, U-Net LR, T5XXL LR) will be interpreted and effectively applied by the underlying `sd-scripts` training engine. Before training commences, the GUI will now log: - The Main LR. - The effective LR for the primary Text Encoder (CLIP), indicating if it's a specific value or a fallback to the Main LR. - The effective LR for the T5XXL Text Encoder (if applicable), indicating its source (specific, inherited from primary TE, or fallback to Main LR). - The effective LR for the U-Net, indicating if it's a specific value or a fallback to the Main LR. This enhances transparency by helping you understand how your LR settings interact, without modifying the `sd-scripts` submodule.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The GUI logic was preventing the main learning rate from being passed to the training script if text_encoder_lr or unet_lr was set. This caused issues with optimizers like Prodigy, which might default to a very small LR if the main LR isn't provided.
This commit modifies kohya_gui/lora_gui.py to ensure the main learning_rate is always included in the parameters passed to the training script, allowing optimizers to use your specified main LR, TE LR, and UNet LR correctly.