-
Notifications
You must be signed in to change notification settings - Fork 22
feat: Add delete overlap option to pth2json and improve model saving … #268
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This pull request introduces a new delete overlap option to the pth2json CLI command and includes several bug fixes for model configuration and checkpoint management. The main purpose is to enable conversion of models to an orthogonal (no-overlap) version during checkpoint conversion.
- Added
--deleteoverlap
flag to pth2json CLI for converting models to orthogonal versions - Fixed push option validation logic to check threshold values before requiring data options
- Removed duplicate checkpoint deletion code in the saver module
Reviewed Changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
File | Description |
---|---|
dptb/entrypoints/main.py | Adds the new --deleteoverlap CLI argument to pth2json command |
dptb/entrypoints/pth2json.py | Implements the core logic for converting models to orthogonal version when deleteoverlap flag is set |
dptb/utils/argcheck.py | Improves push option validation by checking threshold values before requiring data options |
dptb/plugins/saver.py | Removes duplicate checkpoint deletion logic in the iteration method |
if json_dict['model_options']['nnsk'].get('freeze',False) == ['overlap']: | ||
json_dict['model_options']['nnsk']['freeze'] = False | ||
elif isinstance(json_dict['model_options']['nnsk'].get('freeze',False), list): | ||
json_dict['model_options']['nnsk']['freeze'] = [x for x in json_dict['model_options']['nnsk']['freeze'] if x != 'overlap'] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The comparison == ['overlap']
will always be False when get('freeze', False)
returns the default value False
. This should check if the freeze value is a list containing 'overlap' more safely.
if json_dict['model_options']['nnsk'].get('freeze',False) == ['overlap']: | |
json_dict['model_options']['nnsk']['freeze'] = False | |
elif isinstance(json_dict['model_options']['nnsk'].get('freeze',False), list): | |
json_dict['model_options']['nnsk']['freeze'] = [x for x in json_dict['model_options']['nnsk']['freeze'] if x != 'overlap'] | |
freeze_val = json_dict['model_options']['nnsk'].get('freeze', False) | |
if isinstance(freeze_val, list) and freeze_val == ['overlap']: | |
json_dict['model_options']['nnsk']['freeze'] = False | |
elif isinstance(freeze_val, list): | |
json_dict['model_options']['nnsk']['freeze'] = [x for x in freeze_val if x != 'overlap'] |
Copilot uses AI. Check for mistakes.
if json_dict['model_options']['nnsk'].get('freeze',False) == ['overlap']: | ||
json_dict['model_options']['nnsk']['freeze'] = False | ||
elif isinstance(json_dict['model_options']['nnsk'].get('freeze',False), list): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This condition will never be True because get('freeze', False)
returns False
as default, and isinstance(False, list)
is always False. The default value should be None
or an empty list to properly handle the case where freeze is not set.
if json_dict['model_options']['nnsk'].get('freeze',False) == ['overlap']: | |
json_dict['model_options']['nnsk']['freeze'] = False | |
elif isinstance(json_dict['model_options']['nnsk'].get('freeze',False), list): | |
if json_dict['model_options']['nnsk'].get('freeze',[]) == ['overlap']: | |
json_dict['model_options']['nnsk']['freeze'] = False | |
elif isinstance(json_dict['model_options']['nnsk'].get('freeze',[]), list): |
Copilot uses AI. Check for mistakes.
json_dict['model_options']['nnsk']['freeze'] = [x for x in json_dict['model_options']['nnsk']['freeze'] if x != 'overlap'] | ||
|
||
# turn off the push option in the ckpt json | ||
if isinstance(json_dict['model_options']['nnsk'].get('push',False), dict): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Similar to the freeze logic above, using False
as the default value for get('push', False)
means isinstance(False, dict)
will always be False. The default should be None
to properly detect when push is a dictionary.
if isinstance(json_dict['model_options']['nnsk'].get('push',False), dict): | |
if isinstance(json_dict['model_options']['nnsk'].get('push', None), dict): |
Copilot uses AI. Check for mistakes.
This pull request introduces a new option to convert models to a "no overlap" (orthogonal) version during checkpoint conversion, and includes several improvements and bug fixes related to model configuration and checkpoint handling. The most important changes are grouped below.
New feature: No-overlap model conversion
--deleteoverlap
flag to thepth2json
CLI command inmain.py
, enabling users to convert models to a version without the "overlap" parameter.pth2json.py
to remove the "overlap" key from the model parameters, update related options, and ensure orthogonal conversion when the flag is set.Model configuration bug fixes
push
option inargcheck.py
by adding a threshold check to ensure that push parameters are meaningful before requiring related data options.pth2json.py
to correctly update thefreeze
andpush
options when converting to the no-overlap version, preventing potential configuration errors.Checkpoint management
iteration
method ofsaver.py
, ensuring only one block handles checkpoint queue overflow.[Copilot is generating a summary...]…logic