Clover-35B-A3B on one RTX 4090: 208 tokens/s, but not as a diffusion model

"Add support for DiffuSpace's Clover-35B-A3B-Preview, in AR mode only." That line was merged this week into llama.cpp, the common free program for running models yourself. That limit sets your speed. AR means plain one-token-at-a-time writing. Clover is an open diffusion language model, built to write blocks of text in parallel, not word by word. We put the official compressed file on a single RTX 4090 with stock llama.cpp. After a 17-second fix to the file, it wrote about 208 tokens (word pieces) per second, a page of text in about three seconds. It also read prompts of up to 128K tokens (a short novel) using 21.4 of the card's 24 GB. The authors' 515 and 680 tokens per second came from bigger cards and the full model, and the parallel trick is not in this code. Clover is worth a try on one 24 GB card today. Do not buy hardware for the diffusion speed yet.

Three tiles: 208 tokens per second on one RTX 4090; the full 128K context in 24 GB (21,924 MiB peak, needle found), and a 16 GB memory budget works at half speed, about 102 tokens per second with 10 layers of experts in system RAM, emulated on the 4090 at 15,456 MiB

A model built to fill in blocks of text, not to type word by word

Clover's promise is speed, and it comes from how it writes. An ordinary language model works like a typist. It produces one token, then the next, and each one waits for the one before. Clover was trained to work more like someone filling in a crossword. It lays out a block of 32 empty slots. In each pass it fills every slot it is at least 95% sure of, all at once, and comes back for the rest. Fewer passes per block means more text per second, but only if the software running it does the crossword.

The model itself is DiffuSpace's first release. Team members come from the University of Hong Kong group behind Dream 7B, an earlier diffusion model. Clover is derived from Qwen3.6-35B-A3B, an open model from Alibaba's Qwen team. It has 35 billion parameters in total, but only about 3 billion work on each token. Think of a team of 256 specialists: each word consults 8 of them plus one generalist, and the rest wait in the back room. That is why a model this big can be quick. It reads up to 131,072 tokens of prompt and answer together (the "128K"). The license is Apache-2.0, so commercial use is fine.

DiffuSpace's blog makes strong speed claims. On one NVIDIA H200, a data-center card, it reports 515 tokens per second on agent tasks, against 365 for a sped-up Qwen model of similar size. On a newer GB300 data-center card it reports 680. Both are for one request at a time, with the full uncompressed model. The blog does not say what software ran them, or in which mode. It also reports strong scores on coding tests.

Their most vivid demo is a drone dispatch: one compound message with five tasks. In their demo Clover gets all five right in 0.97 seconds, while a Qwen model takes 3.78 seconds and mistakes one drone's mapping mission for a medical one. We ran the same idea: ours took 2.1 to 2.3 seconds against their 0.97, on a 4090 with a compressed file in one-token mode, so it is not a like-for-like race.

The engine that runs it on a 24 GB card types one token at a time

On a 24 GB card with llama.cpp, Clover runs as an ordinary model. That is not a bug in our setup. It is what the merged code says it does.

Of the three ways DiffuSpace lists, only llama.cpp has finished support that fits 24 GB. That support, pull request #31306 by Maphist0, was merged on 2 November and shipped in the 4 November release, v0.9.0. Its notes list "Clover-35B-A3B-Preview diffusion model, AR mode": the typist. A later pull request is meant to add a faster mode that drafts blocks in parallel. We found no such pull request open in llama.cpp.

The model page's setup steps use that missing speed-up, a self-speculative option v0.9.0 does not include. We did not run them.

The faster modes exist in two other places, neither easy on one gaming card:

DiffuSpace's own Python code does the full block-by-block filling. It needs about 69 GB for the weights alone, at full precision. That means an 80 GB data-center card.
A pull request to SGLang, another free serving program (#46927, by Zirui Wu), adds the guess-then-check mode. The model drafts a whole block in one parallel pass, then checks the draft the ordinary way. It is still open, waiting for review, with its test suite not yet run. Its own validation reports 453.59 tokens per second for one request on one H200.

So we measured what you actually get on one card from code you can install now.

The official file needed a 17-second patch before it answered "Canberra"

After a small patch, the official file loads in stock llama.cpp and answers right. Without it, you will hit a wall in the first minute.

As published, llama.cpp v0.9.0 refuses to load the file. It stops with key not found in model: clover_moe.expert_shared_count. We added just that missing label and tried again. The model loaded, and to "What is the capital of Australia? One word." it answered with 600 copies of the letter G, one note held like guitar feedback.

The cause: DiffuSpace wrote its files with its own build of llama.cpp, the one with the speed-up. The file differs from what llama.cpp expects in three small ways. It lacks that one label. It carries 120 extra pieces that only the diffusion mode uses. And in each of 30 layers it stores one small set of numbers in a different mathematical form.

Our script, patch_gguf.py, fixes all three. It copies every other byte as is, so the model is not re-compressed. It took 17 seconds on the 19.6 GB file. After it, the answer was Canberra.

Status: fixed in our SDK example, still open in DiffuSpace's files and in llama.cpp. We tested the patched file only with llama.cpp v0.9.0.

About 208 tokens per second on one RTX 4090, and the whole 128K window fits

With the patched file, Clover wrote 202 to 211 tokens per second for one request, and the first token came back in 0.13 to 0.31 seconds.

In this mode, 208 tokens per second is typist speed, not crossword speed. We did not run the Qwen3.6-35B-A3B base on this card, so we do not know whether Clover in this mode is faster or better than its base.

What we asked	What came back	Time
The compound drone dispatch message, five orders	5 of 5 right, as clean JSON, in every run	2.1 to 2.3 s
"Is it warmer in Lisbon or in Oslo?" with a weather tool	Two tool calls at once, both valid	1.29 s
A whole desktop in one HTML file (the authors' showcase idea)	A working page: dock, draggable windows, clock, notes, a calculator that says 7 x 6 = 42	20.3 s
A code hidden in an 89,367-token prompt, full 128K window	Found	13.45 s

Two rough edges a reader will notice:

It always thinks first. For "Canberra" that is a short note to itself. A 400-word essay took 22.7 seconds and 4,665 tokens, most of them spent thinking. The switch to turn thinking off is not passed through by our endpoint yet. That is a known gap on our side, still open.
The generated calculator has a clipped "=" key. The window is too short, so half the key is hidden. The page came out byte-identical twice, flaw included.

Which GPU: 24 GB runs it all, a 16 GB budget worked at half speed

Every memory budget answered right, and a 16 GB budget kept about half the speed.

The trick for smaller cards is the back room from the specialists picture. llama.cpp can keep some layers of experts in the computer's main memory (RAM) instead of on the card. Each word still visits them, so it gets slower, but the card needs less memory.

Card or memory budget	How it ran	Peak memory on the card	Speed	Real card?
RTX 4090 24 GB	everything on the card	see above	see above	yes
16 GB budget	experts of 10 layers in RAM	15,456 MiB	about 102 tokens/s	no
12 GB budget	experts of 22 layers in RAM	10,270 MiB	about 65 tokens/s	no
8 GB budget	all experts in RAM	2,494 MiB	about 43 tokens/s	no

The 16 GB row is the decision line. It fits in 15,456 MiB and still writes about 102 tokens per second, so the drone dispatch took 4.4 seconds instead of 2.2. The 12 GB and 8 GB budgets also answered every question right, just slower.

The rows marked "no" ran on the same 4090 with its memory capped, so treat them as a memory measurement, not a speed promise. On a real 16 GB card in a smaller desktop, speed will depend on that machine's processor and memory. You also need enough RAM to hold the experts you move off the card. We have no result from a real 16 GB or 12 GB card.

DiffuSpace reports 101.8 tokens per second on a 16 GB RTX 5070 Ti, next to our 102. The match is a coincidence: a different card, a different compression, an unnamed engine.

On InferenceKey: a version pin, a file patch, a five-minute first build

Our standard llama.cpp setup ran a model whose support was three days old. We did not change the platform code. We set a version, patched the file, and gave the worker three CUDA build settings it does not yet set on its own (open on our side, listed in the SDK example).

On InferenceKey, a worker is the small program on your GPU machine that takes orders from the platform. Our worker builds llama.cpp itself. Its default version is older and does not know Clover. So we told it to use version v0.9.0, and the worker built that exact release for the GPU the first time. That build took 4 min 50 s. After that, asking the platform for the model got a ready endpoint in 11.7 seconds.

From there it is the same OpenAI-style API as every other model on the platform. The drone orders, the tool calls and the desktop page above all went through it. This run used our own copy of the InferenceKey platform, with the same code as the hosted service. We have not called the hosted endpoint, so treat that path as untested. Our sample program sends the model to one machine you name, so inference runs on your own hardware. Requests still pass through the platform, hosted or your own copy.

Should you use it? Yes as a 24 GB agent model, not yet for the diffusion speed

You want an open agent model on one 24 GB card: worth a try. Apply the patch first, and compare it with its Qwen base on your own tasks.
You have a 16 GB card: probably, with experts in RAM, at about half speed.
You want the diffusion speed the blog shows: not on llama.cpp today. Watch the SGLang pull request, or use DiffuSpace's own Python code on an 80 GB card.
You care about the benchmark scores: they are the vendor's, and may not hold in one-token-at-a-time mode. We tried a few prompts of our own and all came back right: a start, not proof.

To run the same deployment, drone dispatch and desktop page on your own card, run python main.py from the Clover-35B-A3B SDK example. It needs an InferenceKey project with an SDK token and an API key. You also need a private NVIDIA worker with 24 GB, with the file patched once by its patch_gguf.py.

For engineers

Credits

DiffuSpace: the model, the official GGUF files with checksums, the blog, the UAV dispatch and GenUI desktop demos. Team members come from the HKU NLP group behind Dream 7B (April 2025; the Dream family passed 2.7 million Hugging Face downloads).
Alibaba's Qwen team: Qwen3.6-35B-A3B, the base Clover is derived from.
Maphist0: llama.cpp PR #31306, the AR-mode Clover support, validated on BF16 CPU and H200 CUDA. The llama.cpp maintainers merged it and shipped v0.9.0.
Zirui Wu (WilliamZR): SGLang PR #46927, native serving and self-speculative decoding (open).
AI Search: where we spotted the model (2026-11-01 news video, chapter at 49:23), which noted the full model weighs 69 GB.
Our part: the deployment through the platform, diagnosing the GGUF incompatibility and writing patch_gguf.py, the prompts and tests, the measurements, the VRAM ladder, the SDK example and the demo video.

Model and files

model_type clover_moe, 40 layers: full attention every 4th, 30 linear-attention (Gated DeltaNet) layers. Hidden 2048, 16 heads / 2 KV heads, head_dim 256, 256 routed experts top-8 plus 1 shared, vocab 248,320, max_position_embeddings 131072.
Diffusion settings in config.json: block_size 32, LowConfidence denoising with threshold 0.95.
Generation defaults: greedy, presence_penalty 0.9, repetition_penalty 1.05, plus diffusion-only dllm_length_penalty 10.0 (start 8192, window 8192) and dllm_think_end_bias 10.0 (start 4096, window 4096), which have no llama.cpp equivalent. max_new_tokens 24576.
GGUF repo Diffuspace/Clover-35B-A3B-Preview-GGUF, not gated, apache-2.0, snapshot 32e22a02b885e173177b7ce7661ad13448b4f89e (last modified 2026-10-29). Q4_0 19,586,909,600 B (18.24 GiB), Q4_K_M 22,998,009,248 B, Q8_0 37,863,343,264 B.
Q4_0 sha256 3e2cf14ca47f6e41c251f08237ac79cd37ad81a2ea2a220fff731e34c66930d5, matched.
BF16 repo: about 69 GB of weights, custom code, transformers==5.19.0, PyTorch 2.10+.
The GitHub repo's backend table: Transformers = block diffusion denoising; SGLang (#46927) = block diffusion denoising and self-speculative; llama.cpp (#31306) = self-speculative. The repo currently holds only README.md, benchmark.png and sft/; backend code "will be added".

Runtime status on 2026-11-05

PR #31306 merged 2026-11-02T17:37:28Z, merge commit e4b6c8ca6a6e8da70248da30709e897e7cc8615e. Its description: "in AR mode only. Self-speculative generation for Clover is split into a separate PR."
First build with Clover: b12052 (2026-11-02T17:37:44Z). Release v0.9.0 (2026-11-04T13:08:06Z), notes: "Four new architectures are supported (Clover, Limite, Metis-1.6 and Mistral Large 4)" and "Clover-35B-A3B-Preview diffusion model, AR mode (#31306)".
common/arg.cpp at v0.9.0: 0 occurrences of draft-self. No open Clover self-speculative PR found in llama.cpp (Maphist0's other PRs are #31706 and #28919).
The GGUF card's quick start fetches pull/31306/head and runs llama-speculative-simple --spec-type draft-self. Unchanged as of 2026-11-05. We did not run it.
SGLang #46927: open, created 2026-10-28, updated 2026-11-04, review required, CI not run (no run-ci label), 42 files. Its validation on one H200: MATH500 96.6% at 453.59 per-request output tokens/s (block 16, DP=1, concurrency 1); DP=4 at concurrency 8: 95.2%, 4,268.73 aggregate tokens/s, 192.13 per request.

Authors' claims (not reproduced)

From the blog (the page defaults to Chinese; numbers match our English capture):

Benchmark	Clover-35B-A3B-Preview
SWE-bench Verified	73.6
Terminal-Bench 2.1	51.7
tau3 Overall	65.3
GPQA Diamond	83.8
ZebraLogic	95.8
AA-LCR	72.0
BFCL-v4	67.9
MMLU-Pro	84.8

SWE-bench Verified is a test of real coding tasks.
Compared in the blog against LLaDA2.2 flash, Mercury 2.5, Qwen3.6-35B-A3B and Qwen3.5-35B-A3B.
Speed: 680 tokens/s on one GB300 on internal agent workloads; 515 tokens/s on one H200 against 365 for a similar-sized Qwen with DFlash2. BF16, batch 1. Engine and decoding mode not named.
On-device: INT4 on an RTX 5070 Ti 16 GB, 101.8 tokens/s on MATH-500 against 48.5 for similar-sized MiniCPM and Qwen models. Engine not named.
UAV dispatch demo: five correct results in 0.97 s; Qwen at 3.78 s misreads UAV5's mapping mission as medical.

Setup

One RTX 4090 24 GB (24,564 MiB, 450 W, compute capability 8.9) on InferenceKey Cloud, host with 32 vCPU and 124 GB RAM, 100 GB container disk.
A local InferenceKey manager (never production) bootstrapped the worker. Every main-run call went through /v1/chat/completions of that manager.
Backend: the built-in llamacpp. The worker's default llama.cpp tag is b9748, which has no Clover, so the per-workload pin is required. Per-workload pins override worker-wide environment. On NVIDIA Linux the worker builds llama-server with -DGGML_CUDA=ON at runtime.
Why llama.cpp: of the three backends DiffuSpace lists, it is the only one with merged support, and the Q4_0 fits 24 GB. The other paths, DiffuSpace's Hugging Face Transformers custom code and the open SGLang branch, need about 69 GB of BF16 weights.
The main calls went through the manager's endpoint. The 128K run and the smaller memory budgets were direct llama-server runs on the same worker.

Step by step with the SDK example

On the worker host, once:

pip install -U "huggingface_hub[hf_xet]" gguf numpy
hf download Diffuspace/Clover-35B-A3B-Preview-GGUF Clover-35B-A3B-Preview-Q4_0.gguf --local-dir /root/models
curl -LO https://raw.githubusercontent.com/inferencekey/inferencekey-sdk/main/examples/clover-35b-a3b-llamacpp-private-nvidia/patch_gguf.py
python ./patch_gguf.py /root/models/Clover-35B-A3B-Preview-Q4_0.gguf /root/models/Clover-35B-A3B-Preview-Q4_0-ar.gguf

On PyTorch CUDA 13 images, add to the worker's environment before the first build (see Remaining problems, 4):

CUDACXX=/usr/local/cuda/bin/nvcc
PATH=/usr/local/cuda/bin:$PATH
CMAKE_BUILD_PARALLEL_LEVEL=<cores>

Keep the file on local disk, not a network mount. What patch_gguf.py does:

adds clover_moe.expert_shared_count = 1;
leaves out the 120 diffusion-only tensors: ssm_bd_history.weight, ssm_bd_history.bias, ssm_bd_logit_scale.weight and ssm_conv1d.weight_right, 4 per GDN layer (upstream's conversion/clover.py filters them too);
rewrites the 30 blk.N.ssm_a vectors as -exp(A_log), as upstream's conversion/qwen.py writes them. The official file stores raw A_log (layer 0 ssm_a[0] = -3.3125 = A_log[0] of the BF16 checkpoint);
byte-copies every other tensor, no re-quantization, following what upstream's own conversion code does. 17 s.

On your machine:

git clone https://github.com/inferencekey/inferencekey-sdk
cd inferencekey-sdk/examples/clover-35b-a3b-llamacpp-private-nvidia
cp .env.example .env      # INFERENCEKEY_SDK_TOKEN, INFERENCEKEY_API_KEY, INFERENCEKEY_PROJECT, IK_WORKER_ID
pip install -r requirements.txt   # plus the local SDK build with maturin, see ../CONTRIBUTING.md
python main.py            # IK_KEEP_ALIVE=1 keeps the workload afterwards

Optional: CLOVER_GGUF (default /root/models/Clover-35B-A3B-Preview-Q4_0-ar.gguf) and CLOVER_CONTEXT (default 16384; 131072 fits 24 GB).

main.py calls ensure() on a fixed llamacpp workload pinned to IK_WORKER_ID with this command:

INFERENCEKEY_LLAMACPP_TAG=v0.9.0 llama-server -m /root/models/Clover-35B-A3B-Preview-Q4_0-ar.gguf \
  --alias clover-35b-a3b -ngl 99 -fa on -c 16384 --jinja --reasoning-format deepseek \
  --presence-penalty 0.9 --repeat-penalty 1.05

Then it waits until ready, asks a short question with generate_text(), sends the dispatch message and expects a JSON array, asks for the desktop page and saves outputs/desktop.html and outputs/answers.json. It calls delete() at the end unless IK_KEEP_ALIVE=1.

Timings

Step	Time
Worker online after machine creation	about 47 s
llama.cpp v0.9.0 CUDA build on the worker (`-j30`, arch 89)	4 min 50 s
GGUF download (in parallel with the build)	35 s, about 560 MB/s
`patch_gguf.py`	17 s
`ensure()` to ready, built and patched file on local disk	11.7 s
Direct `llama-server` load of the patched file	about 6 s cold, 1.6 to 3.6 s warm
First platform run with the unpatched file	failed at 292 s, `bootstrap step failed`

Speed and outputs (temperature 0, streaming through the endpoint unless noted)

Test	Completion tokens	Time	First token	Decode tokens/s	Result
"Capital of Australia, one word" x3	37	0.31 to 0.46 s	0.13 to 0.28 s	204 to 211	Canberra
Same with `enable_thinking: false`	37	0.46 s		211	still thinks: the gateway drops `chat_template_kwargs`
"What is diffusion?"	1,402	6.8 s	0.13 s	210	correct physics answer
400-word essay	4,665	22.7 s	0.20 s	207	done
Tools (Lisbon vs Oslo)	206	1.29 s	0.31 s	210	two parallel `get_weather` calls, valid JSON, finish `tool_calls`
Dispatch (3 tests, example, recorded terminal)	393 to 403	2.10 to 2.28 s	0.21 to 0.31 s	208 to 209	5 of 5 orders right
GenUI desktop (example)	4,163	20.29 s		208.5	11,880-char HTML, works, `=` clipped
Eclipse explainer page	11,526	56.94 s	0.14 s	202.9	HTML produced, not inspected in depth
Needle, 10,788-token prompt, 16K context	56	1.77 s	1.48 s	194.5	7319-CLOVER
Needle, 89,367-token prompt, 128K context (direct `llama-server`, same worker)		13.45 s		143.4	7319-CLOVER; prefill 7,363 tokens/s

VRAM ladder

All rows: patched Q4_0, llama.cpp v0.9.0, -ngl 99 -fa on. Only the first row went through the platform; the others were direct llama-server runs on the same worker after the workload was deleted. Peak VRAM from nvidia-smi every second.

Target	Config	Peak VRAM	Speed; dispatch time	Output OK	Real card
RTX 4090 24 GB	`-c 16384` (SDK example, via platform)	19,572 MiB	202 to 211 tokens/s; 2.1 to 2.3 s	yes	yes
RTX 4090 24 GB	`-c 131072`, 89,367-token needle (direct `llama-server`)	21,924 MiB	202 short, 143.4 at 89K; 2.09 s	yes, needle found	yes
16 GB cap	`-c 16384 --n-cpu-moe 10`	15,456 MiB	101.9 to 102.5; 4.41 s	yes	no, emulated
16 GB cap	`-c 16384 --n-cpu-moe 14`	13,728 MiB	82.7 to 82.9; 5.46 s	yes	no, emulated
12 GB cap	`-c 16384 --n-cpu-moe 22`	10,270 MiB	64.9 to 65.1; 7.03 s	yes	no, emulated
8 GB cap	`-c 16384 --cpu-moe`	2,494 MiB	43.1; 10.74 s	yes	no, emulated
RTX 5080 16 GB	`-c 16384 --n-cpu-moe 10` and `8`, two machines on the same host	n/a	n/a	discarded	real card, broken host

"Output OK" means Canberra and 5 of 5 dispatch orders. With experts on the CPU the dispatch wording differs from the all-GPU run, but all five orders are right.
Emulated rows ran on the 4090 host with 16 CPU threads for llama.cpp and the experts of N layers in system RAM. They measure the GPU memory a smaller card needs, not its speed.
RTX 5080 16 GB, two machines on the same host: ggml_cuda_init: failed to initialize CUDA: unknown error, then no usable GPU found, --gpu-layers option will be ignored; torch also raised RuntimeError: CUDA unknown error. llama-server ran on the CPU (about 22 tokens/s, 1 MiB on the GPU). Discarded: there is no real 16 GB or 12 GB number.
Not tried: a working real 16 or 12 GB card, this mode on an H200, the diffusion mode at all, the Qwen3.6-35B-A3B base on the 4090, Q4_K_M, Q8_0, the BF16 Transformers diffusion path, the SGLang branch.

Remaining problems

Official GGUF incompatible with upstream llama.cpp v0.9.0. E llama_model_load: error loading model: error loading model hyperparameters: key not found in model: clover_moe.expert_shared_count; with only the key added, 600 x G. Worked around by patch_gguf.py in the SDK example. Open upstream: the files need regenerating with upstream's converter, or llama.cpp needs to accept the fork layout. We have not filed an upstream issue.
GGUF card instructions use an option upstream lacks. --spec-type draft-self does not exist in v0.9.0. Not run by us. Open.
No real 16 GB result. Broken host, see the ladder. Open.
Worker CUDA build needs environment the bootstrap does not set on images with nvcc off PATH (the PyTorch CUDA 13 images), otherwise cmake configure failed. Workaround in the worker's environment: CUDACXX=/usr/local/cuda/bin/nvcc, PATH=/usr/local/cuda/bin:$PATH, CMAKE_BUILD_PARALLEL_LEVEL=<cores>, plus INFERENCEKEY_LLAMACPP_READY_TIMEOUT_SECS for the first build. Still open in the worker.
The gateway drops chat_template_kwargs, so thinking cannot be switched off per request. Known, open.
Generated calculator window too short, the = key half clipped. A model output flaw, reproducible at temperature 0.

Demo video

The 173-second narrated demo: the authors' benchmarks and demos first (credited to DiffuSpace), then our deployment in the panel, the dispatch and desktop page on the RTX 4090, and the memory ladder.

Sources

Diffuspace/Clover-35B-A3B-Preview (model card, config, generation config).
Diffuspace/Clover-35B-A3B-Preview-GGUF.
DiffuSpace/Clover-35B-A3B-Preview on GitHub.
DiffuSpace blog post.
llama.cpp PR #31306 and the v0.9.0 release notes.
SGLang PR #46927.
AI Search news video, 2026-11-01 (where we spotted it).
SDK example, commit f10c3749717c0d7fcfd58fa8c3be5c1d7dc2ef35.