Support pinning adapter via server args.

[lifuhuang]

Follow-up PR to add server args for loading pinned adapter.

[gemini-code-assist]

Summary of Changes

Hello @lifuhuang, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces the capability to specify 'pinned' LoRA adapters directly through server command-line arguments. This allows certain LoRA models to be kept in GPU memory from the server's inception, optimizing performance for recurrent use cases by minimizing loading overhead. The changes involve a significant refactoring of how LoRA paths are parsed and managed internally, supporting new argument formats that include a 'pinned' flag. The update also includes corresponding documentation and expanded test coverage to validate the new functionality.

Highlights

• Enhanced LoRA Path Argument Parsing: The --lora-paths server argument now supports a new, more flexible format. Users can specify LoRA adapters as a list of strings (e.g., lora1 or lora_name=lora_path) or as JSON dictionaries (e.g., {"lora_name":"lora1", "lora_path":"path/to/lora1", "pinned":true}), allowing the pinned status to be set at server startup.
• Server-Side LoRA Adapter Pinning: LoRA adapters can now be pre-loaded and 'pinned' to GPU memory directly when the SGLang server starts. This feature is designed to improve performance for frequently accessed adapters by reducing memory transfer and reinitialization overhead.
• Internal LoRA Path Data Structure Unification: The internal representation and handling of lora_paths within the LoRAManager and LoRARegistry have been refactored to consistently use a List[LoRARef], streamlining the management of LoRA configurations.
• Updated Documentation and Test Coverage: The lora.ipynb advanced features notebook has been updated with new examples demonstrating how to launch a server with initially pinned and unpinned adapters using the new argument formats. Comprehensive unit tests have also been added/modified to ensure the correct parsing and functionality of the new pinning feature.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request adds support for pinning LoRA adapters via server arguments, allowing for more flexible and efficient LoRA serving. The changes include updating the argument parsing to handle JSON-formatted LoRA path specifications with a pinned flag, modifying the LoRA manager and registry to use a list of LoRARef objects instead of a dictionary, and updating documentation and tests to reflect these new capabilities.

My review found one area for improvement in the argument parsing logic for robustness. The rest of the changes appear correct and well-implemented.

[lifuhuang]

@zhyncs can I get a merge? Thanks!