Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseAPI / Product · ActionableX: @lmsysorg→ original

SGLang Speeds Up Choosing Among 16 Options on Qwen3-8B: 95% of Responses in 20.6 ms vs 54.1 ms

When an LLM needs to pick the best option rather than write text, ordinary generation does unnecessary work. SGLang has added the /v1/score endpoint and a MIS mode: the shared instructions are read once, and the 16 candidates are scored individually. On Qwen3-8B, 95% of requests (the p95 metric) finish in 20.6 ms instead of 54.1 ms with generate. For ranking and choosing agent actions, scoring beats generation.

AI-processed from X: @lmsysorg; edited by Hamidun News
SGLang Speeds Up Choosing Among 16 Options on Qwen3-8B: 95% of Responses in 20.6 ms vs 54.1 ms
Source: X: @lmsysorg. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceX: @lmsysorg ↗
Editor-in-Chief
Key metric #1/v1/score returns scores only for the labels you need: Yes/No or A/B/C
Key metric #2On Qwen3-0.6B, p95 stays below about 100 ms as load grows
Key metric #3Generate and one-at-a-time scoring (SIS) take seconds under the same load

In production, many of us default to a chat model for tasks that don't need text. Classifying support tickets, ranking search results, and choosing an agent's next step all boil down to one question: which option wins? The LMSYS team, which develops the SGLang engine, has published a blog post on serving these "decision models" faster and more accurately. The headline number: on Qwen3-8B with 16 candidates, 95% of requests finish in 20.6 ms, compared with 54.1 ms under ordinary generation.

A word on the metric. p95 is the latency that 95% of requests do not exceed. It reflects not the average but the near-worst case, and it is the figure usually used to decide whether a model can go into an interactive scenario.

Where ordinary generation wastes work

As an example, the authors use Open-Jev. This model scores each candidate with a separate request: it receives a general task description and a single candidate, and it answers "Yes" or "No." The probability of that answer becomes the score. Suppose you are choosing which of 16 actions an agent should take: each one needs its own score.

Serving this scheme through ordinary generation creates two problems.

First, to get the probabilities, you have to ask generate for the top-k log-probabilities, meaning the several most likely tokens. If the label you need (say, "Yes") doesn't make it into that list, you lose it.

Second, the instructions, examples, and dialogue history are identical for every candidate, but in a naive setup the model re-reads them for each of the 16. The more options there are, the more redundant work it does.

What SGLang added

First, the /v1/score endpoint. It returns scores for exactly the labels you request: Yes/No, A/B/C, or any other set. The token you need cannot drop out of the output, because you aren't guessing whether it will land in the top list.

Second, multi-item scoring (MIS) mode. The shared context is processed once, and each candidate is scored in isolation: the options cannot see each other and do not affect each other's scores. This matters: if candidates could "peek" at one another, their order in the list could skew the results.

An analogy: you read the competition rules to the jury once, then show the entries one by one. You don't repeat the rules before every entry, and the entrants don't interfere with each other.

The benchmark numbers

According to LMSYS, MIS latency barely grows as the number of candidates rises from 2 to 16. On Qwen3-8B, p95 for 16 candidates fell from 54.1 ms with Generate to 20.6 ms with MIS, roughly 2.6 times faster.

Under load, the gap is wider. On Qwen3-0.6B, p95 in MIS mode stays below roughly 100 ms as the number of requests grows. Generate and SIS (where each candidate is scored in a separate request) climb to seconds. For an agent choosing an action inside a loop, the difference between 20 ms and several seconds is the difference between a responsive product and a sluggish one.

What to do in practice

If you build ranking, filtering, query routing, or agent action selection on top of an LLM, check whether you are running all of it through generate. When the decision reduces to choosing among labels, scoring gives a more reliable result and more predictable latency.

There are caveats too. The figures come from the engine's own authors, on two Qwen3 models (0.6B and 8B) and their own workload. On your hardware, with your context length and candidate count, results may differ. The gains are most expected where the shared context is long and there are many candidates, which is exactly the recomputation MIS eliminates. If the context is short and there are only two options, the effect will be more modest.

The benchmarks and launch commands are published in the LMSYS blog. Teams already running SGLang should take one current ranking task, switch it to /v1/score, and compare p95 against the previous approach.

Story

Key Questions Answered

Where ordinary generation wastes work?

As an example, the authors use Open-Jev. This model scores each candidate with a separate request: it receives a general task description and a single candidate, and it answers "Yes" or "No." The probability of that answer becomes the score. Suppose you are choosing which of 16 actions an agent should take: each one needs its own score. Serving this scheme through ordinary generation creates two problems.

What SGLang added?

First, the /v1/score endpoint. It returns scores for exactly the labels you request: Yes/No, A/B/C, or any other set. The token you need cannot drop out of the output, because you aren't guessing whether it will land in the top list. Second, multi-item scoring (MIS) mode. The shared context is processed once, and each candidate is scored in isolation: the options cannot see each other and do not affect each other's scores.

What are the key details on The benchmark numbers?

According to LMSYS, MIS latency barely grows as the number of candidates rises from 2 to 16. On Qwen3-8B, p95 for 16 candidates fell from 54.1 ms with Generate to 20.6 ms with MIS, roughly 2.6 times faster. Under load, the gap is wider. On Qwen3-0.6B, p95 in MIS mode stays below roughly 100 ms as the number of requests grows. Generate and SIS (where each candidate is scored in a separate request) climb to seconds.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…