Skip to content

feat: Add delete overlap option to pth2json and improve model saving … #268

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Aug 11, 2025

Conversation

QG-phy
Copy link
Collaborator

@QG-phy QG-phy commented Aug 11, 2025

This pull request introduces a new option to convert models to a "no overlap" (orthogonal) version during checkpoint conversion, and includes several improvements and bug fixes related to model configuration and checkpoint handling. The most important changes are grouped below.

New feature: No-overlap model conversion

  • Added a --deleteoverlap flag to the pth2json CLI command in main.py, enabling users to convert models to a version without the "overlap" parameter.
  • Implemented logic in pth2json.py to remove the "overlap" key from the model parameters, update related options, and ensure orthogonal conversion when the flag is set.

Model configuration bug fixes

  • Improved handling of the push option in argcheck.py by adding a threshold check to ensure that push parameters are meaningful before requiring related data options.
  • Fixed logic in pth2json.py to correctly update the freeze and push options when converting to the no-overlap version, preventing potential configuration errors.

Checkpoint management

  • Cleaned up redundant code in the checkpoint deletion logic within the iteration method of saver.py, ensuring only one block handles checkpoint queue overflow.[Copilot is generating a summary...]…logic

@QG-phy QG-phy requested a review from Copilot August 11, 2025 15:48
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This pull request introduces a new delete overlap option to the pth2json CLI command and includes several bug fixes for model configuration and checkpoint management. The main purpose is to enable conversion of models to an orthogonal (no-overlap) version during checkpoint conversion.

  • Added --deleteoverlap flag to pth2json CLI for converting models to orthogonal versions
  • Fixed push option validation logic to check threshold values before requiring data options
  • Removed duplicate checkpoint deletion code in the saver module

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
dptb/entrypoints/main.py Adds the new --deleteoverlap CLI argument to pth2json command
dptb/entrypoints/pth2json.py Implements the core logic for converting models to orthogonal version when deleteoverlap flag is set
dptb/utils/argcheck.py Improves push option validation by checking threshold values before requiring data options
dptb/plugins/saver.py Removes duplicate checkpoint deletion logic in the iteration method

Comment on lines +50 to +53
if json_dict['model_options']['nnsk'].get('freeze',False) == ['overlap']:
json_dict['model_options']['nnsk']['freeze'] = False
elif isinstance(json_dict['model_options']['nnsk'].get('freeze',False), list):
json_dict['model_options']['nnsk']['freeze'] = [x for x in json_dict['model_options']['nnsk']['freeze'] if x != 'overlap']
Copy link
Preview

Copilot AI Aug 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comparison == ['overlap'] will always be False when get('freeze', False) returns the default value False. This should check if the freeze value is a list containing 'overlap' more safely.

Suggested change
if json_dict['model_options']['nnsk'].get('freeze',False) == ['overlap']:
json_dict['model_options']['nnsk']['freeze'] = False
elif isinstance(json_dict['model_options']['nnsk'].get('freeze',False), list):
json_dict['model_options']['nnsk']['freeze'] = [x for x in json_dict['model_options']['nnsk']['freeze'] if x != 'overlap']
freeze_val = json_dict['model_options']['nnsk'].get('freeze', False)
if isinstance(freeze_val, list) and freeze_val == ['overlap']:
json_dict['model_options']['nnsk']['freeze'] = False
elif isinstance(freeze_val, list):
json_dict['model_options']['nnsk']['freeze'] = [x for x in freeze_val if x != 'overlap']

Copilot uses AI. Check for mistakes.

Comment on lines +50 to +52
if json_dict['model_options']['nnsk'].get('freeze',False) == ['overlap']:
json_dict['model_options']['nnsk']['freeze'] = False
elif isinstance(json_dict['model_options']['nnsk'].get('freeze',False), list):
Copy link
Preview

Copilot AI Aug 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This condition will never be True because get('freeze', False) returns False as default, and isinstance(False, list) is always False. The default value should be None or an empty list to properly handle the case where freeze is not set.

Suggested change
if json_dict['model_options']['nnsk'].get('freeze',False) == ['overlap']:
json_dict['model_options']['nnsk']['freeze'] = False
elif isinstance(json_dict['model_options']['nnsk'].get('freeze',False), list):
if json_dict['model_options']['nnsk'].get('freeze',[]) == ['overlap']:
json_dict['model_options']['nnsk']['freeze'] = False
elif isinstance(json_dict['model_options']['nnsk'].get('freeze',[]), list):

Copilot uses AI. Check for mistakes.

json_dict['model_options']['nnsk']['freeze'] = [x for x in json_dict['model_options']['nnsk']['freeze'] if x != 'overlap']

# turn off the push option in the ckpt json
if isinstance(json_dict['model_options']['nnsk'].get('push',False), dict):
Copy link
Preview

Copilot AI Aug 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar to the freeze logic above, using False as the default value for get('push', False) means isinstance(False, dict) will always be False. The default should be None to properly detect when push is a dictionary.

Suggested change
if isinstance(json_dict['model_options']['nnsk'].get('push',False), dict):
if isinstance(json_dict['model_options']['nnsk'].get('push', None), dict):

Copilot uses AI. Check for mistakes.

@QG-phy QG-phy merged commit 40f71f6 into deepmodeling:main Aug 11, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant