Mellum2.1's MTP head: 1.11x faster on one request, 1.45x on eight, and the model runs in 8-bit on a 16 GB RTX 5080

The guesses were right 87 times in 100, and the model still wrote only 11% faster. That is what we measured when we switched on the small guesser that JetBrains ships with Mellum2.1, its open coding model. JetBrains' model page promises 1.4 to 1.8 times faster writing with the same output. With JetBrains' recommended setting, one request on a 48 GB RTX A6000 on our cloud went from 107 to 119 tokens (pieces of words) per second. With eight requests at once, measured directly on the vLLM server, the gain was 1.45x, inside JetBrains' range; through our endpoint today you get the one-request 1.11x. If you run your own GPUs, the guesser paid far more with eight requests at once than with one. In a compressed 8-bit copy at half its published size, the model also runs on a real 16 GB RTX 5080.

One extra block guesses the next word, and the model checks every guess

The guesser is a small extra part of the model that guesses the word after the one just written. The full model then checks the guess. If it agrees, it keeps two words for roughly the price of one step. If not, it throws the guess away and carries on. JetBrains calls this part the head.

Think of a bandmate calling the next chord just before the change. Most of the time the bassist only nods and plays it, instead of working it out. The call is nearly free. The nod still takes a beat, and that beat decides how much time the band saves.

That is the promise on JetBrains' model page. The model "verifies every drafted token, so the output is unchanged", while writing runs 1.4 to 1.8 times faster, with about 90% of the guesses accepted. The checking is the catch. It costs time too, and JetBrains does not say on which GPU or with how many requests it measured that range.

Mellum2.1 Thinking is JetBrains' open model for code and reasoning. It is a small model where only 2.5B of its 12B parameters work on each word. JetBrains released it in September and wrote that the draft head was "coming soon". The head arrived on 30 October as a separate download. JetBrains says the model itself is unchanged. The head adds one 0.86 GB file and one line of configuration.

One flag was the only change, on two real GPUs and five prompts

Turning the head on is one extra flag, and it was the only thing we changed between runs. We served the model with vLLM 0.33, the open-source serving software JetBrains recommends. Without the flag, the server ignores the head and runs the plain model.

The main GPU was an RTX A6000 with 48 GB on our cloud, running the model at full size, as published. We sent the same five prompts with and without the head. They were JetBrains' own "find the bug" question, a coding task, a math problem, an explanation and a tool call. Then we bypassed our platform and measured on the server itself, one request and eight at once, with one, two and three guesses per step. A second machine, a real 16 GB RTX 5080, showed how small a GPU will do.

The limits, once: our prompts, not the authors' tests, and no document long enough to fill the model's full conversation length of 131,072 tokens. Each setting ran once, on short answers, so treat small gaps as noise. We tested up to eight requests at once, not more. vLLM runs Mellum through generic code, not code written for this model, and a native version could change these numbers. The 32 GB and 24 GB results come from capping the A6000's memory, not from real GPUs of that size.

One request: 1.11x with JetBrains' setting, 1.31x with two guesses

With JetBrains' setting of one guess per step, a single request went from 106.8 to 118.9 tokens per second on the A6000. That is 1.11x, well under the promised 1.4x floor.

In everyday terms, a coding answer of about 600 tokens took 5.9 seconds without the head and 4.9 seconds with it, through our endpoint. You save a second on a long answer. You will not feel it in a chat window, and an agent that makes fifty such calls saves under a minute.

The guesses were not the problem. vLLM's own counters say 87.2% of them were accepted, close to JetBrains' 90%. The model guesses well, and the check eats most of the gain. Our best explanation, not measured: vLLM has no tuned code for Mellum yet. And a model this light per word is so fast that checking costs about as much as it saves.

Two guesses per step did better: 140.2 tokens per second, 1.31x. Only two in three landed (66.6%), but each hit saved two steps. Three guesses: about half landed (48.6%), 124.8 tokens per second. Two guesses won for one request. The vLLM documentation suggests starting at one guess. Try one and two on your own traffic.

Eight requests at once: 1.45x, inside JetBrains' range

With eight requests at once on the A6000, the same work finished in about two-thirds of the time: 607.3 tokens per second against 418.4, 1.45x, measured directly on the vLLM server; through our endpoint today you get the one-request 1.11x. That runs against the usual pattern, where speculative decoding gains less as requests pile up; we measured it once and cannot explain it.

Across one, two and three guesses, eight at once landed between 1.39x and 1.66x, so any of them pays. There was no consistent winner between two and three guesses.

The head costs under a gigabyte of weights, paid in conversation room

The head adds under a gigabyte (0.80 GiB) on the A6000. vLLM keeps the GPU's total memory use fixed, so the cost shows up somewhere else.

The server keeps a notepad of every conversation in progress, so it does not reread it at each word. Whatever memory the model does not use goes to that notepad. With the head on, the A6000's notepad shrank from 1,122,580 tokens to 1,027,634, about 8.5% less. That still holds almost eight conversations at the model's full length of 131,072 tokens. On a 48 GB GPU the head is close to free.

On a 16 GB GPU it is not. On the RTX 5080 of the next section, the notepad went from 52,400 tokens to 29,019: room for three full conversations of 16,384 tokens dropped to one. That is the number to weigh before you switch the head on with a small GPU.

Which GPU: 16 GB was enough, on a real RTX 5080 with 8-bit weights

The smallest GPU that worked was a real RTX 5080 with 16 GB. We had vLLM load a compressed copy, half the published size (8-bit), with conversations of up to 16,384 tokens. It wrote 217.2 tokens per second plain, 288.7 with one guess (1.33x) and 291.2 with two. It used 97% of its memory at peak and got every answer right. That beats the A6000 at full size. Likely reasons, not measured: it is a newer GPU, and smaller weights mean less to read per word.

You do not need 48 GB; that was just our main test GPU. At full size, the 24 GB cap did not start: the weights alone take 23.47 GiB and leave no room for the notepad.

Same answers in quality, not byte for byte

Every run gave right answers, with the head and without it. The model found the bug in JetBrains' example every time. Its interval-merging function passed our four checks in all 13 outputs we tested. It answered 400 to the math problem in every run. Its tool calls came back well formed.

"Output is unchanged" did not hold byte for byte. With the model told to always pick its most likely word, two runs of the same prompt on the same server gave answers of 369 and 810 tokens. That happened without the head too (380 and 829). The vLLM documentation says so: the GPU's rounding, and how many requests run together, can change the exact words. If your tests compare model output byte for byte, they will break with or without the head. Compare the result instead.

On InferenceKey: one setting flips the head, and a cold GPU was ready in 8 minutes

One setting in the SDK example switches the head on or off, and the server restarts in 101 seconds. A cold machine took 488.9 seconds, about 8 minutes, from our request until it was ready to answer. About four of those minutes went to installing vLLM 0.33 and three to vLLM warming up its engine.

We picked vLLM 0.33.0 in our standard setup and ran JetBrains' command as in our test. Nothing had to be built or patched. The RTX 5080 took 20 minutes, most of it slow package downloads on that machine.

The problem you will hit: our platform passes one request at a time to each model. Eight requests through our endpoint wait in line, so the 1.45x gain above was measured on the server itself, not through the platform. Through our endpoint today, you get the one-request gain. Status: open on our side.

This run used a local copy of the platform with the same worker, not the hosted service.

Should you turn it on? Yes on a shared GPU, and test two guesses

A GPU shared by a team or by several agents: turn it on; through InferenceKey you will see the gain once the worker sends parallel requests (open on our side).
One developer, one request at a time: set two guesses per step and measure.
A 16 GB GPU: use 8-bit weights and expect less room for parallel conversations. Proven on a newer RTX 5080; older 16 GB GPUs may fail at load.
A 24 GB GPU: 8-bit weights worked under an emulated 24 GB cap.
Tests that compare output byte for byte: compare results, not bytes.

To switch the head on and off on your own GPU, run python main.py and then MELLUM_MTP=0 python main.py from the Mellum2.1 MTP SDK example. It needs a private NVIDIA worker with 48 GB (32 GB passed only under an emulated cap), or 16 GB with MELLUM_FP8=1 MELLUM_CONTEXT=16384, plus an SDK token and an API key. Through our endpoint it shows the one-request gain, and it deletes the deployment when it finishes.

For engineers

Credits

JetBrains: the model, trained with its MTP head, under Apache-2.0; the MTP repository with the head in model-mtp.safetensors, the base Thinking release, the GGUF repository with MTP draft files, the serving recipes, the Quickstart prompt we ran and the benchmark table.
The vLLM project: MTP speculative decoding (--speculative-config '{"method": "mtp", ...}'), the spec-decode metrics we read acceptance from, online FP8 quantization, and the Transformers modeling backend that serves Mellum and its draft in 0.33.0 (TransformersMoEForCausalLM, TransformersMoEForMTP).
Hugging Face Transformers: the modeling code the backend runs; 5.20 is the first version that loads the head instead of skipping it.
pmttyji on r/LocalLLaMA: where we spotted it (post of 2026-11-02).
Our part: the deployment through the platform, the plain against MTP runs on the same GPU, the 1/2/3-token and concurrency sweep, the memory accounting of the head, the VRAM ladder on two real GPUs plus emulated caps, the SDK example and the demo video.

Model and files

JetBrains/Mellum2.1-12B-A2.5B-Thinking-MTP: sha d55846102dd1c047e95d4841070f88290227c3b7, created 2026-10-30T17:20:32Z, last modified 2026-10-30T18:12:02Z, not gated, Apache-2.0, 17 likes and 1,302 downloads on 2026-11-13. Five BF16 shards of 24.30 GB plus model-mtp.safetensors of 856,767,160 bytes, 25.17 GB in all.
Base JetBrains/Mellum2.1-12B-A2.5B-Thinking: sha 92ddae9f, created 2026-09-20, last modified 2026-10-07. Its card listed GGUF builds and the MTP head as "coming soon".
GGUF repo: sha 4159d684, last modified 2026-10-22, BF16, MXFP4_MOE, Q4_K_M, Q6_K and Q8_0 plus mtp/ draft GGUFs in the same quants. Not tested.
Architecture: 12B total, 2.5B active, 28 layers (21 sliding-window 1,024 and 7 full attention), hidden 2304, 64 experts with 8 active, GQA 32 Q / 4 KV heads, head_dim 128, vocab 98,304, context 131,072 (YaRN factor 16 on the full-attention layers), num_nextn_predict_layers: 1. Base model JetBrains/Mellum2-12B-A2.5B-Base.
Card requirements: vLLM 0.33 or newer with transformers 5.20 or newer. "vLLM releases up to 0.32 refuse the checkpoint"; transformers before 5.20 skips the head tensors with a warning. Other paths on the card, not tested: use_mtp=True in transformers generate, and llama.cpp with --spec-type draft-mtp --spec-draft-n-max 1.
Card Quickstart sampling: temperature 0.6, top_p 0.95, top_k 20, max_tokens 81920.
The card does not state the GPU, batch size or precision behind its 1.4 to 1.8x range.

Authors' benchmarks (not reproduced)

Benchmark	Mellum2.1	Mellum2 Thinking	Gemma 4 E4B	Qwen3.5 9B
LiveCodeBench v6	82.0	69.4	69.4	75.4
SWE-bench Verified	47.0	2.0	23.0	50.0
SWE-bench Pro	28.0	0.0	4.0	38.0
Terminal-Bench 2.1	17.4	0.6	3.4	21.7
AIME 25/26	83.3	60.1	45.0	86.7
GPQA Diamond	64.6	51.0	53.1	77.8
BFCL v4	62.3	49.6	52.5	58.5

Also on the card for Mellum2.1: HumanEval+ 91.5, MBPP+ 79.4, IFEval 90.6, HarmBench 8.5 (lower is better). Agentic runs used the Pi v0.73.1 harness.

Setup

GPUs on InferenceKey Cloud: RTX A6000 48 GB (46,068 MiB, compute capability 8.6, driver 595.91.07) for the main runs, plain and MTP, BF16; RTX 5080 16 GB (16,303 MiB, compute capability 12.0, driver 580.178.04) for the real small-card rung, FP8. An RTX A4000 (Ampere, 16 GB) was not available, so FP8 on a real 16 GB Ampere card is untested.
Backend: the built-in vllm backend with vllm_version="0.33.0" on the workload spec, which installs torch 2.14.0 and transformers 5.20.x. vLLM 0.33.0 maps MellumForCausalLM to the Transformers modeling backend; there is no native vLLM Mellum implementation yet. Why vLLM: it is the card's primary recipe.
A local InferenceKey manager, reached by the workers through tunnels. Production never touched. Weights pre-downloaded to the container disk on each worker (36 s on the A6000).
Measurements: one sweep per setting, no repeats; five prompts x 2 through the endpoint (greedy, streaming); concurrency and draft-depth sweeps direct to vLLM on the worker after warm-up; acceptance from vllm:spec_decode_* counters; VRAM from nvidia-smi sampled every second.

Step by step with the SDK example

Clone inferencekey-sdk and open examples/mellum21-12b-a2-5b-mtp-vllm-private-nvidia (commit 0a5188e70d0c32c95c9e36a17dcb75078102e48f).
cp .env.example .env and fill in the SDK token (ik_sdk_, workload read and write), an ik_live_ API key that allows the workload, the project and IK_WORKER_ID: a private NVIDIA worker with a CUDA 13 driver (580 or newer), 32 GB for BF16 or 16 GB with FP8.
pip install -r requirements.txt plus the local SDK build with maturin (the SDK is not on PyPI).
python main.py. It ensure()s a fixed vllm workload (slug mellum21-thinking, vllm_version="0.33.0") on your worker, waits up to 2,400 s, asks the card's Quickstart question with generate_text(), streams a merge_intervals task with time to first token and tokens per second, runs a get_weather agent loop of up to 4 turns, writes outputs/answers-<mtp|plain>.json, and calls delete() unless IK_KEEP_ALIVE=1.
Variants: MELLUM_MTP=0 python main.py for the plain model; MELLUM_FP8=1 MELLUM_CONTEXT=16384 MELLUM_SPEC_TOKENS=2 python main.py for a 16 GB card. Other settings: MELLUM_SPEC_TOKENS (default 1, the card's), MELLUM_CONTEXT (131072), MELLUM_GPU_UTIL (0.90).

The command the backend runs (MTP on, card setting):

vllm serve JetBrains/Mellum2.1-12B-A2.5B-Thinking-MTP \
  --max-model-len 131072 --gpu-memory-utilization 0.90 --max-num-seqs 16 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser hermes \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'

For the plain model drop --speculative-config. For the RTX 5080 rung add --quantization fp8, set --max-model-len 16384 and num_speculative_tokens 1 or 2. The example sends only temperature and top_p; a code comment says the card's top_k=20 cannot travel through the endpoint (not checked against the manager code).

Timings

Step	Time
A6000 cold, `ensure()` to ready	488.9 s
of which vLLM 0.33.0 install	about 4 min
weights pre-download	36 s
weights load (23.47 GiB)	12.3 s
torch.compile	35.5 s
engine init (profile, KV cache, warm-up)	179.65 s
A6000 restart with MTP off	101.2 s (engine init 47.2 s)
Direct rungs, warm caches	66 to 126 s
RTX 5080, `ensure()` to ready	1,207.1 s (about 14 min of pip downloads at a few hundred kB/s on that host; load 19.7 s; engine init 260.0 s)

Time to first token through the endpoint: 0.23 to 0.29 s in every case.

Speed through the endpoint (A6000, BF16, greedy, streaming)

Prompt	Plain: tokens / s / tok/s	MTP 1 draft: tokens / s / tok/s
Card Quickstart bug (x2)	381 / 3.69 / 110.2	372 / 3.24 / 123.5
`merge_intervals` (x2)	619 / 5.91 / 109.1	603 / 4.90 / 129.6
n^2+1 divisible by 5, n <= 1000	380 and 829 / 3.74 and 7.90 / 110.3 and 108.2	369 and 810 / 3.16 and 6.77 / 127.0 and 124.3
Hash map explanation	792 and 881 / 7.55 and 8.45 / 108.3 and 107.6	940 and 718 / 8.06 and 6.19 / 120.4 and 120.7
Lisbon or Oslo (tools)	280 / 2.78 / 109.7, two parallel calls	244 / 2.16 / 126.7, Lisbon only

Acceptance during this bench on the MTP workload: 3365/3767 = 89.3%.

Speed direct to vLLM (warm)

A6000, BF16, 131,072 ctx, util 0.90	1 req greedy	1 req card sampling	8 at once greedy	8 at once card sampling	Accepted
Plain	106.8	105.7	418.4	316.4
MTP, 1 draft (card)	118.9 (1.11x)	118.6 (1.12x)	607.3 (1.45x)	462.1 (1.46x)	87.2% (8396/9623)
MTP, 2 drafts	140.2 (1.31x)	138.9 (1.31x)	581.8 (1.39x)	525.9 (1.66x)	66.6% (9931/14906)
MTP, 3 drafts	124.8 (1.17x)	125.5 (1.19x)	663.5 (1.59x)	478.5 (1.51x)	48.6% (10070/20709)

RTX 5080, FP8, 16,384 ctx, util 0.90	1 req greedy	1 req card sampling	8 at once greedy	8 at once card sampling	Accepted
Plain	217.2	212.4	896.7	468.1 (not comparable: 3 of 8 ran 64 s, 30,205 tokens, on a 52,400-token KV)
MTP, 1 draft	288.7 (1.33x)	279.7 (1.32x)	1,276.5 (1.42x)	966.1	87.7% (10443/11913)
MTP, 2 drafts	291.2 (1.34x)	283.9	1,297.4 (1.45x)	1,212.0	68.6% (14209/20726, all traffic)

Through the endpoint on the RTX 5080 with MTP 2: 298 to 344 tok/s.

At eight at once, three drafts led with greedy decoding and two with the card's sampling; one run each.

What the MTP head costs in memory

Card, precision	Weights plain / MTP	KV cache plain	KV cache MTP 1	Peak plain / MTP
RTX A6000, BF16, util 0.90	22.67 / 23.47 GiB (+0.80)	16.75 GiB = 1,122,580 tokens	15.87 GiB = 1,027,634 tokens (-8.5%); 16.0 GiB = 1,036,154 in the direct rung	41,017 / 41,147 MiB
RTX 5080, FP8, util 0.90, 16,384 ctx	11.74 / 12.14 GiB (+0.40)	1.36 GiB = 52,400 tokens	0.87 GiB = 29,019 tokens (-45%); 25,967 with MTP 2	14,364 / 15,832 MiB

The peak is set by --gpu-memory-utilization, so the head's cost shows up as lost KV, not as a higher peak on the A6000. The highest single A6000 sample was 41,280 MiB, during the MTP workload. vLLM also warned Add 6 padding layers, may waste at most 27.27% KV cache memory.

VRAM ladder

GPU or cap	Config	Peak VRAM	Speed (1 request)	Output OK	Real or emulated
RTX A6000 48 GB	BF16, 131,072, plain	41,017 MiB	106.8 tok/s	yes	real
RTX A6000 48 GB	BF16, 131,072, MTP 1 (card)	41,147 MiB	118.9 tok/s	yes	real
RTX A6000 48 GB	BF16, 131,072, MTP 2	41,033 MiB	140.2 tok/s	yes	real
RTX A6000 48 GB	BF16, 131,072, MTP 3	41,049 MiB	124.8 tok/s	yes	real
32 GB cap (util 0.71)	BF16, 32,768, MTP 2	32,391 MiB (KV 7.34 GiB, 337,838 tokens)	140.6 tok/s	yes	emulated
24 GB cap (util 0.52)	BF16, 16,384, MTP 2	none	none	no: `ValueError: No available memory for the cache blocks` (BF16 weights 23.47 GiB)	emulated
24 GB cap (util 0.52)	FP8 online, 32,768, MTP 2	23,779 MiB (weights 12.18 GiB, KV 10.08 GiB, 463,944 tokens)	211.4 tok/s	yes	emulated
16 GB cap (util 0.345)	FP8 online, 16,384, MTP 2	15,819 MiB serving, 21,917 MiB load transient (Marlin weight-only FP8 on Ampere)	209.3 tok/s	yes	emulated
RTX 5080 16 GB	FP8, 16,384, plain	14,364 MiB	217.2 tok/s	yes	real
RTX 5080 16 GB	FP8, 16,384, MTP 1	15,832 MiB of 16,303	288.7 tok/s	yes	real
RTX 5080 16 GB	FP8, 16,384, MTP 2 (SDK example through the endpoint)	15,832 MiB of 16,303	291.2 tok/s direct; 303.3 tok/s on the example's code task	yes	real

Not tried: 12 GB (FP8 weights alone are 12.18 GiB), the GGUF and llama.cpp path, FP8 on a real 24 GB card (3090/4090), BF16 on a real 32 GB card, more than 16K context at 16 GB, a long prompt using the full 131,072 window, transformers use_mtp=True, and the authors' benchmarks. FP8 quality was checked on our five prompts only.

Quality checks

Card Quickstart (def mean(xs): return sum(xs) / len(xs) - 1): bug found on every run and rung.
merge_intervals: 13 of 13 outputs checked passed 4 asserts.
Math (n^2+1 divisible by 5, n <= 1000): 400 on every run.
Tools: valid hermes-format calls on every run.
SDK example runs: A6000 MTP on, code task 510 tokens in 4.36 s, 126.1 tok/s, TTFT 0.32 s; A6000 plain, 474 tokens, 109.4 tok/s; RTX 5080 FP8 MTP 2, 700 tokens in 2.47 s, 303.3 tok/s.

Remaining problems

Single-request speed-up below the card's range on the A6000 in BF16. 1.11x with one draft token although 87.2% are accepted. Hypothesis, not proven: the generic Transformers modeling backend plus vLLM's Using default MoE config. Performance might be sub-optimal!, and a verify step that costs almost as much as it saves on a 2.5B-active MoE. Two draft tokens give 1.31x. Open upstream: no native vLLM Mellum implementation yet.
One job at a time per workload on the worker (max_in_flight slots=1). Parallel requests through the endpoint queue, so the concurrency figures were measured direct to vLLM. Open on our side.
Greedy outputs not byte-identical between runs, with and without MTP (MTP: math 369 vs 810 tokens, hash map 940 vs 718; plain: 380 vs 829, 792 vs 881). Answers stayed right. Expected per the vLLM docs: speculative decoding is lossless "up to the precision limits of hardware numerics", and batch size can shift logprobs and output probabilities.
Tool calling varied between runs. On the A6000 in BF16 with MTP the model called tools one city per turn (Lisbon, then Oslo); without MTP it made two parallel get_weather calls, and so did run 3 on the RTX 5080 (FP8, MTP 2). Clients need an agent loop. The first version of our example allowed one round and returned an empty final answer. Fixed: the example loops up to 4 turns.
First sampled request after a restart is slow: Triton JIT-compiles _gumbel_sample_kernel and others, 52.4 tok/s then 105.8. Warm up before measuring.
Online FP8 on Ampere loads through a peak: 21,917 MiB for about 3 s under the 16 GB cap. A real 16 GB Ampere card may fail at load. The RTX 5080 (native FP8) did not show it.
vLLM warnings: KV padding (may waste at most 27.27% KV cache memory), default MoE config, and max_num_scheduled_tokens is set to 2048 based on the speculative decoding settings.
Slow package downloads on the RTX 5080 host: about 14 of its 20 minutes to ready.

Demo video

The 165-second narrated demo: what Mellum2.1 is (the model card, the MTP paragraph, the authors' benchmarks, and a Terminal-Bench 2.1 comparison with GPT-5.6 Sol and Claude Mythos 5 taken from BenchLM's mirror of the Vals leaderboard (fetched 2026-11-08, not re-checked), with a different harness from the authors'), the deployment in the panel on the RTX A6000 with the head on (a coding task at 128 tokens per second, 640 tokens in 5.1 s, and the tool loop, Lisbon then Oslo, in 4.9 s), the key comparison, plain against MTP, and the GPU ladder. English subtitles included.

Where MTP pays: on the RTX A6000 in BF16, one request goes from 107 to 119 tokens per second, eight at once from 418 to 607 (1.45x), with 87% of drafted tokens accepted

Sources

JetBrains/Mellum2.1-12B-A2.5B-Thinking-MTP (model card, config, MTP head).
JetBrains/Mellum2.1-12B-A2.5B-Thinking (base release).
JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF (not tested).
vLLM speculative decoding docs, fetched 2026-11-13 (redirects to the speculative decoding and MTP pages).
r/LocalLLaMA post by pmttyji, 2026-11-02 (where we spotted it).
SDK example, commit 0a5188e70d0c32c95c9e36a17dcb75078102e48f.