Skip to content

Remove mrope position sync #9460

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Remove mrope position sync #9460

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
fb31562
remove mrope position sync
timmy-feng Aug 19, 2025
d63e5c3
Merge branch 'main' into syncless-mrope-positions
timmy-feng Aug 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 16 additions & 17 deletions python/sglang/srt/model_executor/forward_batch_info.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -513,24 +513,23 @@ def _compute_mrope_positions(
for batch_idx in range(batch_size):
mm_input = batch.multimodal_inputs[batch_idx]
if self.forward_mode.is_decode():
mrope_position_deltas = (
[0]
if mm_input is None
else flatten_nested_list(mm_input.mrope_position_delta.tolist())
)
next_input_positions = []
for mrope_position_delta in mrope_position_deltas:
# batched deltas needs to be processed separately
# Convert list of lists to tensor with shape [3, seq_len]
next_input_positions += [
MRotaryEmbedding.get_next_input_positions(
mrope_position_delta,
int(self.seq_lens[batch_idx]) - 1,
int(self.seq_lens[batch_idx]),
)
]
# 3 * N
mrope_positions_list[batch_idx] = torch.cat(next_input_positions, dim=1)
if mm_input is None:
mrope_positions_list[batch_idx] = torch.full(
(3, 1),
self.seq_lens[batch_idx] - 1,
dtype=torch.int64,
device=model_runner.device,
)
else:
mrope_position_deltas = (
mm_input.mrope_position_delta
.flatten()
.to(model_runner.device, non_blocking=True)
)
mrope_positions_list[batch_idx] = (
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we still need MRotaryEmbedding::get_next_input_positions after this, or we can remove it?

mrope_position_deltas + self.seq_lens[batch_idx] - 1
).unsqueeze(0).repeat(3, 1)
elif self.forward_mode.is_extend():
extend_seq_len, extend_prefix_len = (
batch.extend_seq_lens[batch_idx],
Expand Down