[Bug] common_ops.abi3.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKSsb

[shahizat]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 5. Please use English, otherwise it will be closed.

Describe the bug

Hello,

I successfully built the sgl-kernel with sm_120 (NVIDIA RTX 50 series) and CUDA 12.8, but encountered the following issue when running sglang.launch_server command. Please help.

INFO 04-06 14:51:04 [__init__.py:256] Automatically detected platform cuda.
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/admin2/Projects/sglang/python/sglang/launch_server.py", line 6, in <module>
    from sglang.srt.entrypoints.http_server import launch_server
  File "/home/admin2/Projects/sglang/python/sglang/srt/entrypoints/http_server.py", line 45, in <module>
    from sglang.srt.entrypoints.engine import _launch_subprocesses
  File "/home/admin2/Projects/sglang/python/sglang/srt/entrypoints/engine.py", line 40, in <module>
    from sglang.srt.managers.data_parallel_controller import (
  File "/home/admin2/Projects/sglang/python/sglang/srt/managers/data_parallel_controller.py", line 27, in <module>
    from sglang.srt.managers.io_struct import (
  File "/home/admin2/Projects/sglang/python/sglang/srt/managers/io_struct.py", line 25, in <module>
    from sglang.srt.managers.schedule_batch import BaseFinishReason
  File "/home/admin2/Projects/sglang/python/sglang/srt/managers/schedule_batch.py", line 45, in <module>
    from sglang.srt.configs.model_config import ModelConfig
  File "/home/admin2/Projects/sglang/python/sglang/srt/configs/model_config.py", line 25, in <module>
    from sglang.srt.layers.quantization import QUANTIZATION_METHODS
  File "/home/admin2/Projects/sglang/python/sglang/srt/layers/quantization/__init__.py", line 54, in <module>
    from sglang.srt.layers.quantization.awq import AWQConfig
  File "/home/admin2/Projects/sglang/python/sglang/srt/layers/quantization/awq.py", line 6, in <module>
    from sgl_kernel import awq_dequantize
  File "/home/admin2/.virtualenvs/sglang/lib/python3.10/site-packages/sgl_kernel/__init__.py", line 12, in <module>
    from sgl_kernel import common_ops
ImportError: /home/admin2/.virtualenvs/sglang/lib/python3.10/site-packages/sgl_kernel/common_ops.abi3.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKSsb

Reproduction

  
  python3 -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-8B-Instruct \
    --dtype bfloat16 \
    --attention-backend flashinfer \
    --enable-torch-compile 

Environment

Python: 3.10.12 (main, Feb 4 2025, 14:57:36) [GCC 11.4.0]
CUDA available: True
GPU 0: NVIDIA GeForce RTX 5090
GPU 0 Compute Capability: 12.0
CUDA_HOME: /usr/local/cuda-12.8
NVCC: Cuda compilation tools, release 12.8, V12.8.93
CUDA Driver Version: 570.124.06
PyTorch: 2.8.0.dev20250405+cu128
sglang: 0.4.4.post4
sgl_kernel: 0.0.8
flashinfer: Module Not Found
triton: 3.3.0
transformers: 4.51.0
torchao: 0.9.0
numpy: 1.26.4
aiohttp: 3.11.14
fastapi: 0.115.11
hf_transfer: 0.1.9
huggingface_hub: 0.30.1
interegular: 0.3.3
modelscope: 1.24.0
orjson: 3.10.15
outlines: 0.1.11
packaging: 24.2
psutil: 7.0.0
pydantic: 2.10.6
multipart: Module Not Found
zmq: Module Not Found
uvicorn: 0.34.0
uvloop: 0.21.0
vllm: 0.7.4.dev254+ged6ea065.d20250319.cu128
xgrammar: 0.1.11
openai: 1.67.0
tiktoken: 0.9.0
anthropic: 0.49.0
litellm: 1.63.12
decord: 0.6.0
NVIDIA Topology:
GPU0 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X 0-63 0 N/A

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

ulimit soft: 1024

[FlamingoPg]

@shahizat Hi, maybe you need to try use torch 2.5.1 for prebuild whl, or try make build to build a whl for yourself. sgl-kernel prebuild whl need specific torch version.

[shahizat]

Hi @yinfan98 , Thanks for your reply. I finally managed to get it running, but I'm encountering an issue when using --enable-torch-compile.

File "/home/admin2/Projects/sglang/python/sglang/srt/managers/scheduler.py", line 1999, in run_scheduler_process
   scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank, dp_rank)
 File "/home/admin2/Projects/sglang/python/sglang/srt/managers/scheduler.py", line 249, in __init__
   self.tp_worker = TpWorkerClass(
 File "/home/admin2/Projects/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 63, in __init__
   self.worker = TpModelWorker(server_args, gpu_id, tp_rank, dp_rank, nccl_port)
 File "/home/admin2/Projects/sglang/python/sglang/srt/managers/tp_worker.py", line 74, in __init__
   self.model_runner = ModelRunner(
 File "/home/admin2/Projects/sglang/python/sglang/srt/model_executor/model_runner.py", line 177, in __init__
   self.initialize(min_per_gpu_memory)
 File "/home/admin2/Projects/sglang/python/sglang/srt/model_executor/model_runner.py", line 215, in initialize
   self.init_cuda_graphs()
 File "/home/admin2/Projects/sglang/python/sglang/srt/model_executor/model_runner.py", line 933, in init_cuda_graphs
   self.cuda_graph_runner = CudaGraphRunner(self)
 File "/home/admin2/Projects/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 267, in __init__
   self.capture()
 File "/home/admin2/Projects/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 351, in capture
   ) = self.capture_one_batch_size(bs, forward)
 File "/home/admin2/Projects/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 443, in capture_one_batch_size
   run_once()
 File "/home/admin2/Projects/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 436, in run_once
   logits_output = forward(input_ids, forward_batch.positions, forward_batch)
 File "/home/admin2/.virtualenvs/compile/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 658, in _fn
   return fn(*args, **kwargs)
 File "/home/admin2/.virtualenvs/compile/lib/python3.10/site-packages/torch/_dynamo/external_utils.py", line 70, in inner
   return fn(*args, **kwargs)
 File "/home/admin2/.virtualenvs/compile/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
   return func(*args, **kwargs)
 File "/home/admin2/.virtualenvs/compile/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
   return func(*args, **kwargs)
 File "/home/admin2/Projects/sglang/python/sglang/srt/models/llama.py", line 420, in forward
   hidden_states = self.model(
 File "/home/admin2/.virtualenvs/compile/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
   return self._call_impl(*args, **kwargs)
 File "/home/admin2/.virtualenvs/compile/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
   return forward_call(*args, **kwargs)
 File "/home/admin2/Projects/sglang/python/sglang/srt/models/llama.py", line 309, in forward
   hidden_states, residual = layer(
 File "/home/admin2/.virtualenvs/compile/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
   return self._call_impl(*args, **kwargs)
 File "/home/admin2/.virtualenvs/compile/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
   return forward_call(*args, **kwargs)
 File "/home/admin2/Projects/sglang/python/sglang/srt/models/llama.py", line 252, in forward
   hidden_states = self.self_attn(
 File "/home/admin2/.virtualenvs/compile/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
   return self._call_impl(*args, **kwargs)
 File "/home/admin2/.virtualenvs/compile/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
   return forward_call(*args, **kwargs)
 File "/home/admin2/Projects/sglang/python/sglang/srt/models/llama.py", line 183, in forward
   attn_output = self.attn(q, k, v, forward_batch)
 File "/home/admin2/.virtualenvs/compile/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
   return self._call_impl(*args, **kwargs)
 File "/home/admin2/.virtualenvs/compile/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
   return forward_call(*args, **kwargs)
 File "/home/admin2/Projects/sglang/python/sglang/srt/layers/radix_attention.py", line 68, in forward
   return forward_batch.attn_backend.forward(
 File "/home/admin2/Projects/sglang/python/sglang/srt/layers/attention/base_attn_backend.py", line 68, in forward
   return self.forward_decode(
 File "/home/admin2/Projects/sglang/python/sglang/srt/layers/attention/flashinfer_backend.py", line 453, in forward_decode
   def forward_decode(
 File "/home/admin2/.virtualenvs/compile/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 850, in _fn
   return fn(*args, **kwargs)
 File "/home/admin2/.virtualenvs/compile/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1207, in forward
   return compiled_fn(full_args)
 File "/home/admin2/.virtualenvs/compile/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 331, in runtime_wrapper
   all_outs = call_func_at_runtime_with_args(
 File "/home/admin2/.virtualenvs/compile/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 126, in call_func_at_runtime_with_args
   out = normalize_as_list(f(args))
 File "/home/admin2/.virtualenvs/compile/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 692, in inner_fn
   outs = compiled_fn(args)
 File "/home/admin2/.virtualenvs/compile/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 498, in wrapper
   return compiled_fn(runtime_args)
 File "/home/admin2/.virtualenvs/compile/lib/python3.10/site-packages/torch/_inductor/output_code.py", line 558, in __call__
   return self.current_callable(inputs)
 File "/home/admin2/.virtualenvs/compile/lib/python3.10/site-packages/torch/_inductor/utils.py", line 2441, in run
   return model(new_inputs)
 File "/tmp/torchinductor_admin2/ep/cepb3c6zwmkvrmrp3vmmrhbtxj64yszv7k2ozv7ufstfpok7gauf.py", line 93, in call
   triton_poi_fused_index_put_0.run(arg0_1, arg1_1, arg3_1, 1024, stream=stream0)
 File "/home/admin2/.virtualenvs/compile/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 1014, in run
   if launcher.store_cubin and (not benchmark_run or not self.cuda_kernel_saved):
AttributeError: 'NoneType' object has no attribute 'store_cubin'

[2025-04-09 17:45:13] Received sigquit from a child process. It usually means the child failed.
Killed

[FlamingoPg]

I believe this is a bug in torch/triton, not caused by SGLang. It appears during torch.compile when inductor processes the generated triton kernel. This might be an issue with the torch/triton version.

[shahizat]

@yinfan98, you're right, man,it works using the following command with flashinfer as the backend:

 python3 -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --port 30000 \
  --attention-backend flashinfer \
  --enable-torch-compile

But it doesn't work when using triton as the backend. Could it be a Triton issue?

[FlamingoPg]

@yinfan98, you're right, man,it works using the following command with flashinfer as the backend:

 python3 -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --port 30000 \
  --attention-backend flashinfer \
  --enable-torch-compile

But it doesn't work when using triton as the backend. Could it be a Triton issue?

I think so~

[dyyoungg]

@shahizat Hi, maybe you need to try use torch 2.5.1 for prebuild whl, or try make build to build a whl for yourself. sgl-kernel prebuild whl need specific torch version.

The environment I'm using is torch 2.5.1 + cu118. After running make build, I still encountered the same issue.

ImportError: env/lib/python3.10/site-packages/sgl_kernel/common_ops.abi3.so: undefined symbol: _Z24ApplyTokenBitmaskInplaceN2at6TensorES0_St8optionalIS0_E

How to solve this?

[FlamingoPg]

_Z24ApplyTokenBitmaskInplaceN2at6TensorES0_St8optionalIS0_E

I see, i will work on this

[pyh-129]

@shahizat Hi, maybe you need to try use torch 2.5.1 for prebuild whl, or try make build to build a whl for yourself. sgl-kernel prebuild whl need specific torch version.

The environment I'm using is torch 2.5.1 + cu118. After running make build, I still encountered the same issue.

ImportError: env/lib/python3.10/site-packages/sgl_kernel/common_ops.abi3.so: undefined symbol: _Z24ApplyTokenBitmaskInplaceN2at6TensorES0_St8optionalIS0_E

How to solve this?

Same issue here. How to solve?

[FinnGu]

@shahizat Hi, maybe you need to try use torch 2.5.1 for prebuild whl, or try make build to build a whl for yourself. sgl-kernel prebuild whl need specific torch version.

The environment I'm using is torch 2.5.1 + cu118. After running make build, I still encountered the same issue.

ImportError: env/lib/python3.10/site-packages/sgl_kernel/common_ops.abi3.so: undefined symbol: _Z24ApplyTokenBitmaskInplaceN2at6TensorES0_St8optionalIS0_E

How to solve this?

+1

[JOY-SWang]

it seems to be a problem of sql_kernel, after reading modal-labs/modal-examples#1149 (comment)
I bisected to sgl-kernel 0.1.0 to 0.1.1 upgrade. Seems to work now by pinning to old 0.1.0. 😮‍💨

Not sure what changed exactly.

[FinnGu]

I was able to work around this issue by completely reinstalling flashinfer and downgrading sglang to 0.4.5.post3, not sure which dependencies changed in that process, it was a lot.

[pprp]

I was able to work around this issue by completely reinstalling flashinfer and downgrading sglang to 0.4.5.post3, not sure which dependencies changed in that process, it was a lot.

+1

[pprp]

after upgrade torch from 2.5 to 2.6 and then upgrade sgl-kernel from 0.1.0 to 0.1.1, it worked. BTW, cuda is 12.8