deps: lazy import optional dependencies gguf and torchvision

[vhain]

Motivation

Current main branch has broken dependency. Following packages are not defined as dependency (in pyproject.toml) but is required to run srt (via sglang.launch_server):

• torchvision - This PR
• gguf - This PR
• compressed_tensors - ref: [Fix] Add compressed_tensors as deps #4819
• openai - introduced in refactor: bug fixes and refactor for vlm #4661
• partial_json_parser - introduced in Feature/function calling update #2700
• einops - introduced in:
  • [Feature] Support minicpmv v2.6 #2785
  • [Feature] Support Deepseek-VL2 #2798
  • [Fix] Address remaining issues of supporting MiniCPMV #2977
  • model: Support Janus-pro #3203
  • feat: Support Qwen 2.5 vl #3258
  • model: support gemma-3-it #4424

Although this PR does not tackle all the broken dependencies, it fixes two easy ones.

Modifications

This PR updates codes that use torchvision, and gguf to lazy load them.

Next Steps

Something need to be done with remainders:

• compressed_tensors - I saw on-going discussion in [Fix] Add compressed_tensors as deps #4819 (comment)
• openai - I think we could define our own BadRequestError class or equivalent and refrain from using this type since we have only single usage of openai from srt:
  

  sglang/python/sglang/srt/managers/multimodal_processors/base_processor.py

  Line 13 in 0bc0bf5

  from openai import BadRequestError

• partial_json_parser - I think we could include this as srt's dependency as it's being used in the base detector class for tool (function) calling.
• einops - I think this could be included as srt's dependency as it's being widely used. or we could make it optional (lazy load) since it's seems to be only required for vision models?

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

[zhyncs]

partial_json_parser - I think we could include this as srt's dependency as it's being used in the base detector class for tool (function) calling.

einops - I think this could be included as srt's dependency as it's being widely used. or we could make it optional (lazy load) since it's seems to be only required for vision models?

agree @vhain

[zhyncs]

@vhain please merge the latest main

[zhyncs]

srt git:(ryan/optional-deps/gguf_torchvision) python3 -m unittest test_vision_openai_server.TestGemma3itServer
command=python3 -m sglang.launch_server --model-path google/gemma-3-4b-it --trust-remote-code --chat-template gemma-it --host 127.0.0.1 --port 2157
[2025-03-27 21:33:10] server_args=ServerArgs(model_path='google/gemma-3-4b-it', tokenizer_path='google/gemma-3-4b-it', tokenizer_mode='auto', skip_tokenizer_init=False, load_format='auto', trust_remote_code=True, dtype='auto', kv_cache_dtype='auto', quantization=None, quantization_param_path=None, context_length=None, device='cuda', served_model_name='google/gemma-3-4b-it', chat_template='gemma-it', completion_template=None, is_embedding=False, revision=None, host='127.0.0.1', port=2157, mem_fraction_static=0.88, max_running_requests=None, max_total_tokens=None, chunked_prefill_size=8192, max_prefill_tokens=16384, schedule_policy='fcfs', schedule_conservativeness=1.0, cpu_offload_gb=0, page_size=1, tp_size=1, stream_interval=1, stream_output=False, random_seed=781419702, constrained_json_whitespace_pattern=None, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, log_level='info', log_level_http=None, log_requests=False, log_requests_level=0, show_time_cost=False, enable_metrics=False, decode_log_interval=40, api_key=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, dp_size=1, load_balance_method='round_robin', ep_size=1, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', lora_paths=None, max_loras_per_batch=8, lora_backend='triton', attention_backend='flashinfer', sampling_backend='flashinfer', grammar_backend='xgrammar', speculative_algorithm=None, speculative_draft_model_path=None, speculative_num_steps=5, speculative_eagle_topk=4, speculative_num_draft_tokens=8, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, disable_radix_cache=False, disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_nccl_nvls=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, disable_mla=False, disable_overlap_schedule=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_ep_moe=False, enable_deepep_moe=False, enable_torch_compile=False, torch_compile_max_bs=32, cuda_graph_max_bs=160, cuda_graph_bs=None, torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, allow_auto_truncate=False, enable_custom_logit_processor=False, tool_call_parser=None, enable_hierarchical_cache=False, hicache_ratio=2.0, enable_flashinfer_mla=False, enable_flashmla=False, flashinfer_mla_disable_ragged=False, warmups=None, debug_tensor_dump_output_folder=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_bootstrap_port=8998)
[2025-03-27 21:33:10] For Gemma 3, we downcast float32 to bfloat16 instead of float16 by default. Please specify `dtype` if you want to use float16.
[2025-03-27 21:33:10] Downcasting torch.float32 to torch.bfloat16.
[2025-03-27 21:33:10] The following error message 'operation scheduled before its operands' can be ignored.
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.50, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
[2025-03-27 21:33:18] Use chat template for the OpenAI-compatible API server: gemma-it
[2025-03-27 21:33:20 TP0] For Gemma 3, we downcast float32 to bfloat16 instead of float16 by default. Please specify `dtype` if you want to use float16.
[2025-03-27 21:33:20 TP0] Downcasting torch.float32 to torch.bfloat16.
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.50, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
[2025-03-27 21:33:23 TP0] Overlap scheduler is disabled for multimodal models.
[2025-03-27 21:33:23 TP0] For Gemma 3, we downcast float32 to bfloat16 instead of float16 by default. Please specify `dtype` if you want to use float16.
[2025-03-27 21:33:23 TP0] Downcasting torch.float32 to torch.bfloat16.
[2025-03-27 21:33:23 TP0] Automatically reduce --mem-fraction-static to 0.836 because this is a multimodal model.
[2025-03-27 21:33:23 TP0] Init torch distributed begin.
[2025-03-27 21:33:26 TP0] Failed to import pynvml with ModuleNotFoundError("No module named 'pynvml'")
[2025-03-27 21:33:26 TP0] Init torch distributed ends. mem usage=0.00 GB
[2025-03-27 21:33:26 TP0] Load weight begin. avail mem=139.21 GB
[2025-03-27 21:33:26 TP0] The following error message 'operation scheduled before its operands' can be ignored.
[2025-03-27 21:33:28 TP0] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:01<00:01,  1.25s/it]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:02<00:00,  1.20s/it]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:02<00:00,  1.21s/it]

[2025-03-27 21:33:30 TP0] Load weight end. type=Gemma3ForConditionalGeneration, dtype=torch.bfloat16, avail mem=131.08 GB, mem usage=8.13 GB.
[2025-03-27 21:33:31 TP0] KV Cache is allocated. #tokens: 834615, K size: 54.12 GB, V size: 54.12 GB
[2025-03-27 21:33:31 TP0] Memory pool end. avail mem=14.61 GB
2025-03-27 21:33:31,320 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
[2025-03-27 21:33:31 TP0] Capture cuda graph begin. This can take up to several minutes. avail mem=13.95 GB
Capturing batches (avail_mem=12.55 GB):   0%|                                                                                                                       | 0/23 [00:00<?, ?it/s]2025-03-27 21:33:31,890 - INFO - flashinfer.jit: Loading JIT ops: batch_decode_with_kv_cache_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_qk_256_head_dim_vo_256_posenc_0_use_swa_False_use_logits_cap_False
2025-03-27 21:33:31,913 - INFO - flashinfer.jit: Finished loading JIT ops: batch_decode_with_kv_cache_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_qk_256_head_dim_vo_256_posenc_0_use_swa_False_use_logits_cap_False
Capturing batches (avail_mem=9.72 GB): 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 23/23 [00:08<00:00,  2.59it/s]
[2025-03-27 21:33:40 TP0] Capture cuda graph end. Time elapsed: 8.91 s. avail mem=9.66 GB. mem usage=4.29 GB.
/sgl-workspace/sglang/python/sglang/srt/utils.py:823: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:206.)
  tensor_data = torch.ByteTensor(
[2025-03-27 21:33:44 TP0] max_total_num_tokens=834615, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=2049, context_len=1048576
[2025-03-27 21:33:45] INFO:     Started server process [43750]
[2025-03-27 21:33:45] INFO:     Waiting for application startup.
[2025-03-27 21:33:45] INFO:     Application startup complete.
[2025-03-27 21:33:45] INFO:     Uvicorn running on http://127.0.0.1:2157 (Press CTRL+C to quit)
[2025-03-27 21:33:46] INFO:     127.0.0.1:46746 - "GET /get_model_info HTTP/1.1" 200 OK
[2025-03-27 21:33:46 TP0] Prefill batch. #new-seq: 1, #new-token: 7, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0,
2025-03-27 21:33:46,416 - INFO - flashinfer.jit: Loading JIT ops: batch_prefill_with_kv_cache_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_qk_256_head_dim_vo_256_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False
2025-03-27 21:33:46,442 - INFO - flashinfer.jit: Finished loading JIT ops: batch_prefill_with_kv_cache_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_qk_256_head_dim_vo_256_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False
[2025-03-27 21:33:46] INFO:     127.0.0.1:46748 - "POST /generate HTTP/1.1" 200 OK
[2025-03-27 21:33:46] The server is fired up and ready to roll!
[2025-03-27 21:33:53 TP0] Prefill batch. #new-seq: 1, #new-token: 1, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-03-27 21:33:54] INFO:     127.0.0.1:47404 - "GET /health_generate HTTP/1.1" 200 OK
.[2025-03-27 21:33:54 TP0] Prefill batch. #new-seq: 1, #new-token: 23, #cached-token: 1, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-03-27 21:33:54 TP0] Prefill batch. #new-seq: 1, #new-token: 283, #cached-token: 1, token usage: 0.00, #running-req: 1, #queue-req: 0,
[2025-03-27 21:33:55 TP0] Prefill batch. #new-seq: 2, #new-token: 273, #cached-token: 295, token usage: 0.00, #running-req: 2, #queue-req: 0,
[2025-03-27 21:33:55 TP0] Number of images does not match number of special image tokens in the input text. Got 256 image tokens in the text but 512 tokens from image embeddings.
[2025-03-27 21:33:55] INFO:     127.0.0.1:47450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-03-27 21:33:55] INFO:     127.0.0.1:47416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-03-27 21:33:55 TP0] Decode batch. #running-req: 2, #token: 345, token usage: 0.00, gen throughput (token/s): 9.21, #queue-req: 0,
[2025-03-27 21:33:55 TP0] Prefill batch. #new-seq: 2, #new-token: 2, #cached-token: 306, token usage: 0.00, #running-req: 2, #queue-req: 0,
[2025-03-27 21:33:55] INFO:     127.0.0.1:47420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-03-27 21:33:55] INFO:     127.0.0.1:47434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-03-27 21:33:55 TP0] Prefill batch. #new-seq: 1, #new-token: 1, #cached-token: 283, token usage: 0.00, #running-req: 1, #queue-req: 0,
[2025-03-27 21:33:55] INFO:     127.0.0.1:47452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-03-27 21:33:55] INFO:     127.0.0.1:47458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-03-27 21:33:55 TP0] Prefill batch. #new-seq: 1, #new-token: 1, #cached-token: 283, token usage: 0.00, #running-req: 1, #queue-req: 0,
[2025-03-27 21:33:55 TP0] Prefill batch. #new-seq: 2, #new-token: 2, #cached-token: 306, token usage: 0.00, #running-req: 2, #queue-req: 0,
[2025-03-27 21:33:56] INFO:     127.0.0.1:47480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-03-27 21:33:56 TP0] Decode batch. #running-req: 3, #token: 346, token usage: 0.00, gen throughput (token/s): 179.04, #queue-req: 0,
[2025-03-27 21:33:56] INFO:     127.0.0.1:47482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-03-27 21:33:56] INFO:     127.0.0.1:47472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-03-27 21:33:56] INFO:     127.0.0.1:47496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-03-27 21:33:56 TP0] Prefill batch. #new-seq: 1, #new-token: 1, #cached-token: 283, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-03-27 21:33:56 TP0] Prefill batch. #new-seq: 1, #new-token: 1, #cached-token: 23, token usage: 0.00, #running-req: 1, #queue-req: 0,
[2025-03-27 21:33:56 TP0] Decode batch. #running-req: 2, #token: 328, token usage: 0.00, gen throughput (token/s): 129.39, #queue-req: 0,
[2025-03-27 21:33:56] INFO:     127.0.0.1:47508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-03-27 21:33:56] INFO:     127.0.0.1:47512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
.[2025-03-27 21:33:57 TP0] Prefill batch. #new-seq: 1, #new-token: 555, #cached-token: 12, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-03-27 21:33:57 TP0] Decode batch. #running-req: 1, #token: 596, token usage: 0.00, gen throughput (token/s): 53.99, #queue-req: 0,
[2025-03-27 21:33:57 TP0] Decode batch. #running-req: 1, #token: 636, token usage: 0.00, gen throughput (token/s): 100.52, #queue-req: 0,
[2025-03-27 21:33:57] INFO:     127.0.0.1:47526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
------------------------------
Multi images response:
Here's a description of each image:

**Image 1:** A man is ironing clothes on a portable ironing board while standing on the roof of a yellow taxi in a busy New York City street.

**Image 2:** The logo for "SGL" features a stylized branching tree icon connected to a small square containing code symbols.
------------------------------
.[2025-03-27 21:33:58 TP0] Prefill batch. #new-seq: 1, #new-token: 30, #cached-token: 284, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-03-27 21:33:58] INFO:     127.0.0.1:47532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
..[2025-03-27 21:33:58 TP0] Prefill batch. #new-seq: 1, #new-token: 1, #cached-token: 283, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-03-27 21:33:58 TP0] Decode batch. #running-req: 1, #token: 308, token usage: 0.00, gen throughput (token/s): 58.87, #queue-req: 0,
[2025-03-27 21:33:58] INFO:     127.0.0.1:47540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
..
----------------------------------------------------------------------
Ran 7 tests in 55.491s

OK

It works well for me locally.