[Bug] Missing tool name in answer

[rstanislav]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 5. Please use English, otherwise it will be closed.

Describe the bug

Hello, i have found a strange BUG, i'm trying to use sglang with n8n and noticed that function calls not working, i have written a simple proxy app and noticed that in sglang answer with function call at final chunk function name in missing, and that is strange because in first chunks its present, and final answer becomes invalid as result, here is example:
Request:

2025-03-23 18:15:35,962 - DEBUG - Remote request body for 2ad4de47 to /v1/chat/completions:
{
"model": "casperhansen/llama-3.3-70b-instruct-awq",
"temperature": 1,
"top_p": 1,
"frequency_penalty": 0,
"presence_penalty": 0,
"n": 1,
"stream": true,
"tools": [
{
"type": "function",
"function": {
"name": "vector_store",
"description": "Tool retrives user names that are already in database",
"parameters": {
"type": "object",
"properties": {
"input": {
"type": "string"
}
},
"required": [
"input"
],
"additionalProperties": false,
"$schema": "http://json-schema.org/draft-07/schema#"
}
}
}
],
"stream_options": {
"include_usage": true
},
"messages": [
{
"role": "system",
"content": "You are a helpful assistant, extract user name from chat message, tell if its found if its already preset in vector store\ntools & when to use them:\n\n1 - Vector store\ncontains db of user name already in system\n\nExample workflow:\nWhen chat message arrives\nMessage: hi, my name is boss\nActions taken:\n1) username 'boss' extracted and checked in vector store\n2) if user with extracted username ('boss') is not found, answer user that its not found\n3) if username found: answer user that user with name "boss" is found."
},
{
"role": "user",
"content": "hi my name is stanislav"
}
]
}

Response:

INFO: 192.168.100.29:57726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2025-03-23 18:15:36,126 - DEBUG - Stream chunk for request 2ad4de47:
14c
data: {"id":"7874fe5978be41a1ad2daf0e67658490","object":"chat.completion.chunk","created":17427
2025-03-23 18:15:36,127 - DEBUG - Stream chunk for request 2ad4de47:
42934,"model":"casperhansen/llama-3.3-70b-instruct-awq","choices":[{"index":0,"delta":{"role":"assis
2025-03-23 18:15:36,127 - DEBUG - Stream chunk for request 2ad4de47:
tant","content":"","reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":null,
2025-03-23 18:15:36,226 - DEBUG - Stream chunk for request 2ad4de47:
"matched_stop":null}],"usage":null}

1a1
data: {"id":"7874fe5978be41a1ad2daf0e67658490","object":
2025-03-23 18:15:36,227 - DEBUG - Stream chunk for request 2ad4de47:
"chat.completion.chunk","created":1742742934,"model":"casperhansen/llama-3.3-70b-instruct-awq","choi
2025-03-23 18:15:36,227 - DEBUG - Stream chunk for request 2ad4de47:
ces":[{"index":0,"delta":{"role":"assistant","content":null,"reasoning_content":null,"tool_calls":[{
2025-03-23 18:15:36,227 - DEBUG - Stream chunk for request 2ad4de47:
"id":"0","type":"function","function":{"name":"vector_store","arguments":""}}]},"logprobs":null,"fin
2025-03-23 18:15:36,383 - DEBUG - Stream chunk for request 2ad4de47:
ish_reason":"tool_call","matched_stop":null}],"usage":null}

1a3
data: {"id":"7874fe5978be41a1ad2
2025-03-23 18:15:36,383 - DEBUG - Stream chunk for request 2ad4de47:
daf0e67658490","object":"chat.completion.chunk","created":1742742934,"model":"casperhansen/llama-3.3
2025-03-23 18:15:36,384 - DEBUG - Stream chunk for request 2ad4de47:
-70b-instruct-awq","choices":[{"index":0,"delta":{"role":"assistant","content":null,"reasoning_conte
2025-03-23 18:15:36,384 - DEBUG - Stream chunk for request 2ad4de47:
nt":null,"tool_calls":[{"id":"0","type":"function","function":{"name":"","arguments":"{"input": "
2025-03-23 18:15:36,403 - DEBUG - Stream chunk for request 2ad4de47:
"}}]},"logprobs":null,"finish_reason":"tool_call","matched_stop":null}],"usage":null}

199
data:
2025-03-23 18:15:36,403 - DEBUG - Stream chunk for request 2ad4de47:
{"id":"7874fe5978be41a1ad2daf0e67658490","object":"chat.completion.chunk","created":1742742935,"mode
2025-03-23 18:15:36,404 - DEBUG - Stream chunk for request 2ad4de47:
l":"casperhansen/llama-3.3-70b-instruct-awq","choices":[{"index":0,"delta":{"role":"assistant","cont
2025-03-23 18:15:36,404 - DEBUG - Stream chunk for request 2ad4de47:
ent":null,"reasoning_content":null,"tool_calls":[{"id":"0","type":"function","function":{"name":"","
2025-03-23 18:15:36,405 - DEBUG - Stream chunk for request 2ad4de47:
arguments":"stan"}}]},"logprobs":null,"finish_reason":"tool_call","matched_stop":null}],"usage":null
2025-03-23 18:15:36,423 - DEBUG - Stream chunk for request 2ad4de47:
}

19d
data: {"id":"7874fe5978be41a1ad2daf0e67658490","object":"chat.completion.chunk","created":
2025-03-23 18:15:36,424 - DEBUG - Stream chunk for request 2ad4de47:
1742742935,"model":"casperhansen/llama-3.3-70b-instruct-awq","choices":[{"index":0,"delta":{"role":"
2025-03-23 18:15:36,424 - DEBUG - Stream chunk for request 2ad4de47:
assistant","content":null,"reasoning_content":null,"tool_calls":[{"id":"0","type":"function","functi
2025-03-23 18:15:36,425 - DEBUG - Stream chunk for request 2ad4de47:
on":{"name":"","arguments":"islav"}"}}]},"logprobs":null,"finish_reason":"tool_call","matched_stop"
2025-03-23 18:15:36,461 - DEBUG - Stream chunk for request 2ad4de47:
:null}],"usage":null}

ee
data: {"id":"7874fe5978be41a1ad2daf0e67658490","object":"chat.completio
2025-03-23 18:15:36,462 - DEBUG - Stream chunk for request 2ad4de47:
n.chunk","created":1742742935,"model":"casperhansen/llama-3.3-70b-instruct-awq","choices":[],"usage"
2025-03-23 18:15:36,463 - DEBUG - Final stream chunk for request 2ad4de47:
:{"prompt_tokens":311,"total_tokens":329,"completion_tokens":18}}

e
data: [DONE]

As you can see in 1a1 chunk there is tool name, and starting from next chunk - 1a3 its missing

Reproduction

--

Environment

Latest version.

[Fridge003]

cc @shuaills Please have a look at this issue

[kebe7jun]

This seems to be normal, the function name only appears in the first chunk.

[rstanislav]

This seems to be normal, the function name only appears in the first chunk.

Here is example of VLLM answer(working without problems):

2025-03-24 05:41:44,387 - DEBUG - Stream chunk for request 88922764:
109
data: {"id":"chatcmpl-30ebe6e948f546faae0c8937da0c50aa","object":"chat.completion.chunk","creat
2025-03-24 05:41:44,388 - DEBUG - Stream chunk for request 88922764:
ed":1742784103,"model":"casperhansen/llama-3.3-70b-instruct-awq","choices":[{"index":0,"delta":{"rol
2025-03-24 05:41:44,648 - DEBUG - Stream chunk for request 88922764:
e":"assistant","content":""},"logprobs":null,"finish_reason":null}]}


16f
data: {"id":"chatcmpl-3
2025-03-24 05:41:44,649 - DEBUG - Stream chunk for request 88922764:
0ebe6e948f546faae0c8937da0c50aa","object":"chat.completion.chunk","created":1742784103,"model":"casp
2025-03-24 05:41:44,649 - DEBUG - Stream chunk for request 88922764:
erhansen/llama-3.3-70b-instruct-awq","choices":[{"index":0,"delta":{"tool_calls":[{"id":"chatcmpl-to
2025-03-24 05:41:44,649 - DEBUG - Stream chunk for request 88922764:
ol-72dc072e706945b6b1bcd82bb722081c","type":"function","index":0,"function":{"name":"vector_store"}}
2025-03-24 05:41:44,699 - INFO - Queue stats: 0/20 in queue, 1/1 active
2025-03-24 05:41:44,813 - DEBUG - Stream chunk for request 88922764:
]},"logprobs":null,"finish_reason":null}]}


12e
data: {"id":"chatcmpl-30ebe6e948f546faae0c8937da0
2025-03-24 05:41:44,813 - DEBUG - Stream chunk for request 88922764:
c50aa","object":"chat.completion.chunk","created":1742784103,"model":"casperhansen/llama-3.3-70b-ins
2025-03-24 05:41:44,814 - DEBUG - Stream chunk for request 88922764:
truct-awq","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"input
2025-03-24 05:41:44,833 - DEBUG - Stream chunk for request 88922764:
\": \""}}]},"logprobs":null,"finish_reason":null}]}


124
data: {"id":"chatcmpl-30ebe6e948f546faae
2025-03-24 05:41:44,834 - DEBUG - Stream chunk for request 88922764:
0c8937da0c50aa","object":"chat.completion.chunk","created":1742784103,"model":"casperhansen/llama-3.
2025-03-24 05:41:44,835 - DEBUG - Stream chunk for request 88922764:
3-70b-instruct-awq","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":
2025-03-24 05:41:44,854 - DEBUG - Stream chunk for request 88922764:
"stan"}}]},"logprobs":null,"finish_reason":null}]}


128
data: {"id":"chatcmpl-30ebe6e948f546faae0
2025-03-24 05:41:44,855 - DEBUG - Stream chunk for request 88922764:
c8937da0c50aa","object":"chat.completion.chunk","created":1742784103,"model":"casperhansen/llama-3.3
2025-03-24 05:41:44,856 - DEBUG - Stream chunk for request 88922764:
-70b-instruct-awq","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"
2025-03-24 05:41:44,876 - DEBUG - Stream chunk for request 88922764:
islav\"}"}}]},"logprobs":null,"finish_reason":null}]}


13d
data: {"id":"chatcmpl-30ebe6e948f546fa
2025-03-24 05:41:44,877 - DEBUG - Stream chunk for request 88922764:
ae0c8937da0c50aa","object":"chat.completion.chunk","created":1742784103,"model":"casperhansen/llama-
2025-03-24 05:41:44,878 - DEBUG - Stream chunk for request 88922764:
3.3-70b-instruct-awq","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments
2025-03-24 05:41:44,879 - DEBUG - Stream chunk for request 88922764:
":""}}]},"logprobs":null,"finish_reason":"tool_calls","stop_reason":128008}]}


f7
data: {"id":"ch
2025-03-24 05:41:44,879 - DEBUG - Stream chunk for request 88922764:
atcmpl-30ebe6e948f546faae0c8937da0c50aa","object":"chat.completion.chunk","created":1742784103,"mode
2025-03-24 05:41:44,880 - DEBUG - Stream chunk for request 88922764:
l":"casperhansen/llama-3.3-70b-instruct-awq","choices":[],"usage":{"prompt_tokens":325,"total_tokens
2025-03-24 05:41:44,881 - DEBUG - Final stream chunk for request 88922764:
":349,"completion_tokens":24}}


e
data: [DONE]


0

Looks like delta mechanics is a bit broken in sglang or i don't understand what is happening, even if its supposed to remove tool name after first chunk, it should be removed with "name" tag itself

[kebe7jun]

Are you meaning there shouldn't have "name":""?

[rstanislav]

Thanks for answer! i think i found the problem - in openai docs there is section about streaming tool calls:
https://platform.openai.com/docs/guides/function-calling?api-mode=chat#streaming

Answer should be in format:

[{"index": 0, "id": "call_DdmO9pD3xa9XTPNJ32zg2hcA", "function": {"arguments": "", "name": "get_weather"}, "type": "function"}]
[{"index": 0, "id": null, "function": {"arguments": "{\"", "name": null}, "type": null}]
[{"index": 0, "id": null, "function": {"arguments": "location", "name": null}, "type": null}]
[{"index": 0, "id": null, "function": {"arguments": "\":\"", "name": null}, "type": null}]
[{"index": 0, "id": null, "function": {"arguments": "Paris", "name": null}, "type": null}]
[{"index": 0, "id": null, "function": {"arguments": ",", "name": null}, "type": null}]
[{"index": 0, "id": null, "function": {"arguments": " France", "name": null}, "type": null}]
[{"index": 0, "id": null, "function": {"arguments": "\"}", "name": null}, "type": null}]
null

Currently sglang has name is not null(its "")(but it should be preset with null value) and also, maybe more important - delta value is in wrong position:
here is tcpdump capture of sglang response:

HTTP/1.1 200 OK
date: Fri, 21 Feb 2025 22:35:02 GMT
server: uvicorn
content-type: text/event-stream; charset=utf-8
transfer-encoding: chunked

data: {"id":"e6ddb972ba4e46b990af453ef46fbb8a","object":"chat.completion.chunk","created":1740177302,"model":"casperhansen/llama-3.3-70b-instruct-awq","choices":[{"index":0,"delta":{"role":"assistant","content":"","tool_calls":null},"logprobs":null,"finish_reason":"","matched_stop":null}],"usage":null}

data: {"id":"e6ddb972ba4e46b990af453ef46fbb8a","object":"chat.completion.chunk","created":1740177302,"model":"casperhansen/llama-3.3-70b-instruct-awq","choices":[{"index":0,"delta":{"role":"assistant","content":null,"tool_calls":[{"id":"0","type":"function","function":{"name":"vector_store","arguments":""}}]},"logprobs":null,"finish_reason":"tool_call","matched_stop":null}],"usage":null}

data: {"id":"e6ddb972ba4e46b990af453ef46fbb8a","object":"chat.completion.chunk","created":1740177302,"model":"casperhansen/llama-3.3-70b-instruct-awq","choices":[{"index":0,"delta":{"role":"assistant","content":null,"tool_calls":[{"id":"0","type":"function","function":{"name":"","arguments":"{\"input\": \""}}]},"logprobs":null,"finish_reason":"tool_call","matched_stop":null}],"usage":null}

data: {"id":"e6ddb972ba4e46b990af453ef46fbb8a","object":"chat.completion.chunk","created":1740177302,"model":"casperhansen/llama-3.3-70b-instruct-awq","choices":[{"index":0,"delta":{"role":"assistant","content":null,"tool_calls":[{"id":"0","type":"function","function":{"name":"","arguments":"user\"}"}}]},"logprobs":null,"finish_reason":"tool_call","matched_stop":null}],"usage":null}

data: {"id":"07c6965432864ed69c709de5668ef540","model":"casperhansen/llama-3.3-70b-instruct-awq","choices":[],"usage":{"prompt_tokens":272,"total_tokens":289,"completion_tokens":17}}

data: [DONE]

And here is example of VLLM (working):

2025-03-24 15:15:03,895 - DEBUG - Stream chunk for request 9705e802:
109
data: {"id":"chatcmpl-0d23b730b45243a48c4a0ec70963befd","object":"chat.completion.chunk","created":1742818502,"model":"casperhansen/llama-3.3-70b-instruct-awq","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}]}



2025-03-24 15:15:04,153 - DEBUG - Stream chunk for request 9705e802:
16f
data: {"id":"chatcmpl-0d23b730b45243a48c4a0ec70963befd","object":"chat.completion.chunk","created":1742818502,"model":"casperhansen/llama-3.3-70b-instruct-awq","choices":[{"index":0,"delta":{"tool_calls":[{"id":"chatcmpl-tool-05afb42657de44f2969f9dacdad15530","type":"function","index":0,"function":{"name":"vector_store"}}]},"logprobs":null,"finish_reason":null}]}



2025-03-24 15:15:04,319 - DEBUG - Stream chunk for request 9705e802:
12e
data: {"id":"chatcmpl-0d23b730b45243a48c4a0ec70963befd","object":"chat.completion.chunk","created":1742818502,"model":"casperhansen/llama-3.3-70b-instruct-awq","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"input\": \""}}]},"logprobs":null,"finish_reason":null}]}



2025-03-24 15:15:04,340 - DEBUG - Stream chunk for request 9705e802:
124
data: {"id":"chatcmpl-0d23b730b45243a48c4a0ec70963befd","object":"chat.completion.chunk","created":1742818502,"model":"casperhansen/llama-3.3-70b-instruct-awq","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"stan"}}]},"logprobs":null,"finish_reason":null}]}



2025-03-24 15:15:04,361 - INFO - Queue stats: 0/20 in queue, 1/1 active
2025-03-24 15:15:04,361 - DEBUG - Stream chunk for request 9705e802:
128
data: {"id":"chatcmpl-0d23b730b45243a48c4a0ec70963befd","object":"chat.completion.chunk","created":1742818502,"model":"casperhansen/llama-3.3-70b-instruct-awq","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"islav\"}"}}]},"logprobs":null,"finish_reason":null}]}



2025-03-24 15:15:04,382 - DEBUG - Stream chunk for request 9705e802:
13d
data: {"id":"chatcmpl-0d23b730b45243a48c4a0ec70963befd","object":"chat.completion.chunk","created":1742818502,"model":"casperhansen/llama-3.3-70b-instruct-awq","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":""}}]},"logprobs":null,"finish_reason":"tool_calls","stop_reason":128008}]}



2025-03-24 15:15:04,382 - DEBUG - Stream chunk for request 9705e802:
f7
data: {"id":"chatcmpl-0d23b730b45243a48c4a0ec70963befd","object":"chat.completion.chunk","created":1742818502,"model":"casperhansen/llama-3.3-70b-instruct-awq","choices":[],"usage":{"prompt_tokens":325,"total_tokens":349,"completion_tokens":24}}


e
data: [DONE]

Looks like after first chunk "role":"assistant","content":"" part should not be repeated in next chunks(currently its always
**delta":{"role":"assistant","content":null**,"tool_calls":[..

and most important(VLLM example):
delta":{"tool_calls":[{"id":"chatcmpl-tool-05afb42657de44f2969f9dacdad15530","type":"function","index":0,"function":{"name":"vector_store"}}]}

Delta should be at tool_calls (just like in openai example), also - ID of tool calls in sglang is a bit 'simple' i think, that can affect answers if multiple same tool calls made during session, because after second tool call LLM will not be able to understand context as all tool calls will have tool name and almost same ID (maybe its possible to make it in long format as id of main answer ('e6ddb972ba4e46b990af453ef46fbb8a') (so it will be close to openai api examples)?

[rstanislav]

@kebe7jun 😔 is there any data or debug steps that i can test/provide to help to fix this issue ?

[kebe7jun]

I submited a pr to fix this issue: #4704

[rstanislav]

I submited a pr to fix this issue: #4704

But i tested more and found the problem, missing tool name is not main reason(its should not be removed, it has to have null value instead of '' + its the wrong structure of streaming tool output, please take a look at my message #4700 (comment)