Mellum2.1's draft head pays on a shared GPU: 1.45x with eight requests, 1.11x with one
Mellum2.1's MTP head made eight parallel requests 1.45x faster on an RTX A6000, one request only 1.11x. It also runs on a 16 GB RTX 5080.
Should you switch on the draft head JetBrains just shipped for Mellum2.1, its open coding model? On a GPU that serves several requests at once, yes. With eight requests at once on a 48 GB RTX A6000 on our cloud, total output went from 418 to 607 tokens (pieces of words) per second: 1.45x, measured directly on the vLLM server. With one request and JetBrains' recommended setting, the gain was 1.11x, from 107 to 119 tokens per second. JetBrains' model page promises 1.4 to 1.8x. For anyone who deploys models, the switch costs one flag and under a gigabyte of extra weights. What you can do with it today: turn it on for a shared GPU, try two guesses per step for a single user, and know that the model also runs in 8-bit on a real 16 GB RTX 5080.
A small guesser that the full model checks
Mellum2.1 Thinking is JetBrains' open model for code and reasoning. It has 12B parameters, but only 2.5B of them work on each word, which makes it light and fast. It came out in September with the draft head marked "coming soon". The head arrived on 30 October as a separate 0.86 GB file, and JetBrains says the model itself is unchanged. We spotted it in a r/LocalLLaMA post by pmttyji.
The head guesses the word after the one just written. The full model then checks the guess. If it agrees, you get two words for roughly the price of one step. If not, the guess is thrown away. It works like a drummer counting in the next song: nearly free when the band agrees, wasted when it does not.
JetBrains' page says the model "verifies every drafted token, so the output is unchanged", with about 90% of the guesses accepted. It does not say on which GPU, or with how many requests, it measured its 1.4 to 1.8x range. So we measured it, changing only that one flag between runs, with vLLM 0.33, the serving software JetBrains recommends.
One request: the guesses land, the check eats the gain
With JetBrains' setting of one guess per step, one request went from 106.8 to 118.9 tokens per second. vLLM's own counters say 87.2% of the guesses were accepted, close to JetBrains' 90%. The model guesses well; checking the guess costs almost as much as it saves.
Our best explanation, not measured: vLLM runs Mellum through generic code, with no version tuned for this model yet. And a model this light per word is so fast that the check is not cheap by comparison.
In everyday terms, a coding answer of about 600 tokens took 5.9 seconds without the head and 4.9 seconds with it, through our endpoint. You save a second on a long answer.
Two guesses per step did better for a single request. Only two in three landed, but the speed rose to 140.2 tokens per second, 1.31x. Three guesses fell back to 124.8.
Eight requests at once: inside JetBrains' range
| RTX A6000, full size, tokens per second | Plain | 1 guess | 2 guesses | 3 guesses |
|---|---|---|---|---|
| One request | 106.8 | 118.9 (1.11x) | 140.2 (1.31x) | 124.8 (1.17x) |
| Eight at once | 418.4 | 607.3 (1.45x) | 581.8 (1.39x) | 663.5 (1.59x) |
With eight requests at once, the same work finished in about two thirds of the time, and every number of guesses paid. That runs against the usual pattern, where this kind of guessing gains less as requests pile up. We measured it once and cannot explain it.
The head's cost is memory. vLLM keeps a notepad of every conversation in progress, and the head takes room from it. On the A6000 the notepad shrank from 1,122,580 tokens to 1,027,634, about 8.5% less. On a 48 GB GPU, that is close to free.
16 GB was enough, with 8-bit weights
The smallest GPU that worked was a real RTX 5080 with 16 GB. vLLM loaded a compressed 8-bit copy, half the published size, with conversations of up to 16,384 tokens. It wrote 217.2 tokens per second plain and 288.7 with one guess, 1.33x, at 97% of its memory. At full size, a 24 GB cap did not start.
On a small GPU the notepad matters more. On the RTX 5080 it went from 52,400 tokens to 29,019 with the head on: room for three full conversations dropped to one.
Quality held everywhere. The model found the bug in JetBrains' own example every time, its interval-merging function passed our four checks in 13 of 13 outputs, and it answered 400 to the math problem in every run. "Output is unchanged" did not hold byte for byte, though, and it did not without the head either. Compare results in your tests, not bytes.
On InferenceKey, one setting in the SDK example switches the head on or off, and a cold A6000 was ready in about 8 minutes. One catch: our platform passes one request at a time to each model, so through our endpoint today you get the one-request 1.11x. That is open on our side.
Who should care
- A team or several agents sharing one GPU: turn the head on; the parallel gain is where JetBrains' promise holds.
- One developer, one request at a time: set two guesses per step and measure on your own traffic.
- A 16 GB GPU: use 8-bit weights and expect less room for parallel conversations.
- Tests that compare model output byte by byte: they will break with or without the head.
The honest limit: each setting ran once, on five short prompts of ours, so treat small gaps as noise.
Read every run, the memory accounting and the GPU ladder in the full Mellum2.1 MTP test.