[Bug] fix gemma-2-2b-it-FP8 accuracy

[zhyncs]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 5. Please use English, otherwise it will be closed.

Describe the bug

The accuracy of neuralmagic/gemma-2-2b-it-FP8 drops from 0.62 to 0.52 in the main branch. It was detected by our nightly CI run. We need to fix this.

neuralmagic/gemma-2-2b-it-FP8 | 0.512 | 0.6

https://github.com/sgl-project/sglang/actions/runs/13800885290

Reproduction

N/A

Environment

N/A

[lengrongfu]

/assign

[lengrongfu]

@zhyncs I would like to try to deal with this issue.

I found this message Model neuralmagic/gemma-2-2b-it-FP8 score (0.5120) is below threshold (0.6000), How can we fix this problem?

[merrymercy]

The accuracy is good two months ago. https://github.com/sgl-project/sglang/actions/runs/12987515144
Can you bisect which commits introduced the problem and try to fix it?

[zhyncs]

@laixinn

[laixinn]

@zhyncs Noted, I will check this issue.

[WineChord]

@zhyncs @merrymercy Here I'd like to share some of my findings.

The accuracy of neuralmagic/gemma-2-2b-it-FP8 dropped from ~0.612 to ~0.500 during the last two months. Actually, there are two critical points causing the performance drop:

1. feat: enable ragged fa3 by default on hopper 12.4+ #3442 defaults use_ragged = True, causing the performance drop from ~0.612 to ~0.520 (consistent drop of >= 0.08)
2. feat: support flashinfer mla attention for deepseek v3 #3550 upgrades flashinfer from 0.2.0.post2 to 0.2.1.post1, causing the performance drop from ~0.520 to ~0.50 (consistent drop of >= 0.02). The latest tag (v0.2.3) of flashinfer still suffers from the same performance issue.

I am still investigating the detailed reasons. I hope the findings shed some light on this issue.

[merrymercy]

@WineChord Thanks for the investigation. Can you submit a fix?

@yzh119 Please also take a look

[WineChord]

Glad to, I am working on it.