[Fix] Reduce busy polling when scheduler is idle

[p12tic]

Motivation

In setups which have long inactivity periods it is desirable to reduce system power consumption when sglang does nothing. This would lead not only to power savings, but also to more CPU thermal headroom when a request eventually comes. This is important in cases when multiple GPUs are connected as each GPU would otherwise pin one thread at 100% CPU usage.

Primary use case is residential or small commercial users serving LLMs to a very small number of users.

FIXES #1730.

This PR includes a simplier alternative to #1731. There's less risk of adverse performance impact because the affected code runs when server is idle.

The reason for #1731 closure was specified as in-progress refactoring that would resolve the performance problem by itself. Because half a year passed since that time, I'm bringing up a fix again with the hope that maybe it will be deemed acceptable this time.

Modifications

The proposed solution is to use zmq.Poller on all sockets that may receive data that needs handling immediately. Doing this on idle should have few risks with little latency impact.

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

[xiezhq-hermann]

I think this is neat, can you make it an sys arg options instead of default behavior?
@zhyncs @merrymercy @ByronHsu what do you think?

[zacksiri]

Looking forward to having this merged.

[p12tic]

@xiezhq-hermann thanks for review. I added --sleep-on-idle to "Other runtime options" server options.

[p12tic]

@xiezhq-hermann Just a friendly ping :-)

[xiezhq-hermann]

The change looks good to me, and I just fixed a minor lint problem.
Thanks a lot for the contribution and sorry for the delay : )

[p12tic]

@merrymercy Just a friendly ping.. This PR has been in ready-to-merge status for 2 weeks.

[p12tic]

Rebased.

[p12tic]

@merrymercy @xiezhq-hermann Just a friendly ping again :) Anything I could do to make this PR to land? It has been in "approved, ready to merge" status for almost a month now.

[xiezhq-hermann]

@p12tic sorry for the delay due to some coordination problem on our side and thanks again for your work and patience : )