add custom chat template #3251
Merged
add custom chat template #3251
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your contribution! |
LiqinruiG
reviewed
Aug 7, 2025
| @@ -161,6 +161,9 @@ The following extra parameters are supported: | |||
| chat_template_kwargs: Optional[dict] = None | |||
| # Additional parameters passed to the chat template, used for customizing dialogue formats (default None). | |||
|
|
|||
| chat_template: Optional[str] = None | |||
There was a problem hiding this comment.
docs/zh/parameters.md 和 docs/parameters.md 中补充启动参数说明。 中补充启动参数说明。
fastdeploy/entrypoints/llm.py
Outdated
|
|
||
| messages_len = len(messages) | ||
| for i in range(messages_len): | ||
| messages[i] = {"messages": messages[i]} | ||
| messages[i] = {"messages": messages[i], "chat_template": chat_template} |
There was a problem hiding this comment.
这个参数，不建议放在 messages 中， 放在 chat_template_kwargs 中吧？ 后面的透传行为可以继续传下去。
| @@ -104,6 +105,7 @@ async def lifespan(app: FastAPI): | |||
| pid = os.getppid() | |||
| else: | |||
| pid = os.getpid() | |||
| chat_template = load_chat_template(args.chat_template) | |||
There was a problem hiding this comment.
这个适合放在外面进行 load， 这里面会启动多进程。
| @@ -84,6 +85,8 @@ async def create_chat_completion(self, request: ChatCompletionRequest): | |||
| api_server_logger.info(f"create chat completion request: {request_id}") | |||
|
|
|||
| try: | |||
| if request.chat_template is None: | |||
| request.chat_template = self.chat_template | |||
There was a problem hiding this comment.
要放在 to_dict_for_infer 之后， 因为chat_template 不是 openai 标准参数， 不能直接放在 request 传递。
| @@ -0,0 +1,22 @@ | |||
| import os | |||
There was a problem hiding this comment.
单测除了验证模板打开， 还需要验证请求时增加 自定义模板。
LiqinruiG
approved these changes
Aug 18, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
增加自定义chat template
1.启动服务时通过--chat-template参数可以指定加载的chat-template，会覆盖模型原有的模板
2.请求时通过chat_template参数可以传递chat-template模板，传递的模板会覆盖模型原有的模板，以及启动服务时通过--chat-template指定的模板