SenseNova-Loop-9B: a looped 9B fits a 20 GB card, and pays in speed

A looped 9B that runs its middle layers twice: right answers on a real 20 GB card for short prompts, at 2 to 3 tokens/s, 9.2 with an SGLang PR.

Published · By InferenceKey · 10 min read + technical appendix

A model's size usually predicts its GPU and its speed. SenseNova-Loop-9B-Preview, an open model from SenseTime, stores 9 billion learned numbers (parameters) but runs many of them twice per word. So it fits like a 9B, and its authors count the work per word like a 14.6B. We ran it uncompressed on a real 20 GB RTX 4000 Ada and on a 48 GB A40 on our cloud. Both read a chart, solved a word problem and called a tool correctly. On the authors' code it pays in speed: 2 to 3 words a second. Switching the loop off doubles that and breaks the answers.

With that code a chart answer took about three minutes. An unreleased SGLang build solved the word problem in 34 seconds instead of 101. Stock llama.cpp and vLLM cannot load it yet. A 20 GB card gets you in the door for short prompts, and the example at the end runs it.

Same SenseNova-Loop-9B weights, same word problem: three painters paint 2 fences in 5 hours, how many hours do 6 painters need for 8 fences? Left, the trained setting, 2 passes through the looped block, 56 layer runs per word: 277 tokens, 2.34 tokens per second, 118 seconds, a green box on the step that gives 8 divided by four fifths equals 10 hours and a big green Answer: 10. Right, the loop switched off in our loading code, 1 pass, 32 layer runs per word: 1,024 tokens (the limit), 4.7 tokens per second, 218 seconds, a red box where it drifts into Chinese and a red box where it starts repeating one word, followed by a symbol, and a big red no answer

Same file, same question. On the left the model runs its middle layers twice, as it was trained. On the right we made it run them once.

MemorySpeedAccuracy
Fits a real 20 GB card for prompts up to about 3K tokensA chart answer takes about 3 minutes (a word problem drops from 101 s to 34 s with the SGLang pull request)Right answers on all three requests on both cards

A 9B that computes like a 14.6B: 32 layers stored, 56 run

The model is a stack of 32 processing steps (layers), and every word makes 56 layer passes through them. Layers 4 to 27 run once, then again, with the very same parameters. SenseTime's model page calls it "~9B total parameters; ~14.6B inference parameters". That is their count of the compute per word, not one we checked, and not a claim about quality.

It works like a band chart with a repeat sign over the bridge, where the band plays the bridge twice but the sheet music gains no pages. Standard serving software has no code for that repeat yet.

It reads images as well as text, and was trained on texts of up to 262,144 tokens (the "256K"). Its Apache 2.0 license allows commercial use, no form needed. The model page's news section dates the release to 30 October, and the page was last updated on 2 November.

On Terminal-Bench 2.1, a 0-to-100 test of agents working in a terminal, the authors report 40.45, against 29.20 for Qwen3.5 9B. On the same board, the three-times-larger Qwen3.5 27B scores 47.19 and Nanbeige's 3B model scores 49.44. We did not reproduce those numbers. We spotted the model in AI Search's news video of 1 November and a r/LocalLLaMA thread the day it came out. Nothing we had published before covered a model that reuses its own layers, so we tested it.

Stock llama.cpp, Ollama, LM Studio and vLLM cannot load it yet

The first question an engineer asks is "does it run in my usual tool?" This week the answer is no for the usual tools that run models on your own machine or server.

  • llama.cpp, Ollama, LM Studio. The official llama.cpp cannot load it. A feature request opened on 3 November is still open. AtomicChat's converted files need a modified llama.cpp and read text only. Their model page says LM Studio and Ollama do not support it yet either.
  • vLLM and SGLang. These are the serving programs that put a model behind an API for many users. Support exists only as proposed code changes (pull requests), not releases. The vLLM one, by sufubao, is text-only for now. We built the SGLang one, by WANDY666, and measured it below.

So today, on one card, only the authors' own code runs it on both text and images. It runs in Python, with the model code shipped alongside the weights. That is what we deployed.

Our test: the authors' code on two real cards, one request at a time

We loaded the model uncompressed, in the format the authors list. The main run used a 48 GB A40 on our cloud. The confirmation run used a real 20 GB RTX 4000 Ada, a workstation card.

The same three requests went to both: a word problem with the model thinking first, a question about a chart image, and a tool call. A tool call is the model asking your program to run a function, here an exchange-rate lookup. We added long prompts with a hidden code to find on both cards, and the loop experiment from the picture above on the A40.

We sent one request at a time, and ran each request once: it shows the model works end to end, not how good it is. The longest prompt was 12,046 tokens, nowhere near 256K. We did not try a real 24 GB card, compressed versions, several users at once, or a normal 9B without a loop on the same card. So we cannot yet say how much of the slowness is the loop and how much is the software.

Right answers on both cards, but a chart answer takes about three minutes

Every main request came back right on both cards, with the same answers (the chart text was identical). The chart was a made-up bakery electricity bill. The model found August as the lowest month at 2,140 kWh, worked out 3,110 kWh saved between two quarters, and priced it at 653.10 euros. The tool call came back as a clean get_exchange_rate with USD and EUR.

RequestA40 48 GBRTX 4000 Ada 20 GBAnswer
Word problem, thinking on101 s, 3.0 tokens/s138 s, 2.25 tokens/s10, right on both
Chart image, thinking on (455 tokens)157 s, 2.9 tokens/s204 s, 2.25 tokens/s2,140 kWh, 3,110 kWh, 653.10 EUR, right on both
Tool call29 s34 sget_exchange_rate(USD, EUR), right on both

A chart answer with thinking takes minutes, which is fine for a batch job overnight and painful for a chat window.

What holds it back is probably not the GPU itself. On this path the CPU hands the GPU its work in many small pieces per word. On the A40, one CPU core sat at 100 % while the GPU was busy only 38 to 53 % of the time (mostly 33 to 47 % on the RTX 4000 Ada). Every word runs 56 layers, so every word pays for 56 layers' worth of handing out work. We did not measure why the A40 was about a third faster.

It looks like a software problem: the SGLang pull request ran on the same A40 at 9.1 to 9.2 tokens per second. The word problem that took 101 seconds through Transformers took 33.6 seconds there, with identical text. The chart answer was identical too.

One wrong answer worth knowing: asked a trick question about the price of a pen without thinking first, it said 0.65 on both cards. The right answer is 0.15.

Switch the loop off and it writes twice as fast, and stops making sense

Running layers 4 to 27 once instead of twice took the model from about 2.3 to 4.7 tokens per second, and broke it. The parameters were the same. Only the number of passes changed, through a switch we added to our loading code.

On the word problem it went wrong in its first sentence, drifted into Chinese, repeated one word, then a symbol, until it hit our 1,024-token limit. On the chart it read 3,350 as 3,530, wrote "kg" for kWh, switched language and answered 1,414 kWh and 293.34 euros, both wrong. A short story repeated its own opening.

Trimming the loop does buy speed, just not speed you can use: the model was trained to expect the second pass. We also tried three passes, which the model was never trained on. It still wrote fluent text, about a third slower than with two passes.

The built-in speed-up (MTP) did not run on any engine we tried

The model page lists MTP, a small part that guesses the next few words for the big model to check in one go, as "Available". On one GPU we could not run it.

The Transformers code skips those weights on load. The vLLM pull request lists MTP as unsupported. The SGLang pull request loaded a second full copy of the model and failed. With our one-line fix, the guessing part loaded, then crashed. One other serving program documents MTP, but only on 2 to 4 GPUs. Whether MTP speeds this model up, we cannot tell you yet.

Which GPU: a real 20 GB card works, for prompts up to about 3K tokens

The smallest card that worked is a real RTX 4000 Ada with 20 GB, uncompressed, through our platform (its results are in the table above). It handled a 3,046-token prompt. A prompt of about 6,000 tokens ran out of memory. The server answered that request with an error and kept serving the next one.

The model takes 17.5 GiB once loaded, and a 12,046-token prompt pushes it to 19.9 GiB. A 20 GB card really holds 19.55 GiB, so that does not fit. The parameters are shared between the two passes, but each pass keeps its own notes on the prompt, and those notes grow with its length. That is why long prompts, not the weights, decide between 20 and 24 GB.

The 48 GB A40 is where we started, not what you need:

Card or memory budgetReal or emulatedResult
A40 48 GBrealall passed, including a 12K-token prompt (reference)
24 GB budget (22.5 GiB cap on the A40)emulatedall passed, including the 12K-token prompt, same answers
RTX 4000 Ada 20 GBrealmain requests and a 3K-token prompt passed, about 6K tokens ran out of memory
20 GB budget (18.5 GiB cap on the A40)emulatedshort prompts passed, the 12K-token prompt ran out of memory
16 GB budget (14.5 GiB cap on the A40)emulateddoes not load

An emulated row means we capped how much of the A40 the model could use. It shows whether the model fits, not how a smaller card performs.

On InferenceKey: ready in 57 seconds, with the authors' code behind a chat API

The model needs its authors' own code, and our platform can run code like that. The A40 answered 57.4 seconds after we asked for the model, with the weights already on disk. About 36 of those seconds went to a one-time warm-up before the first request. A fresh 20 GB worker took 106.7 seconds, about 30 of them setting up its Python packages first.

The worker ran a short Python file of ours, a few hundred lines around the authors' Transformers code. It takes ordinary chat requests with images and tools and returns the answer, the model's thinking, any tool calls and timings.

This run used a local copy of the platform, not the hosted service, with the same worker code. The SGLang numbers come from a separate build outside the platform.

Should you use it? Yes for patient, short-prompt jobs on a 20 to 24 GB card

  • You want an open model that reads images and calls tools on one small card: worth a try. A real 20 GB card ran it uncompressed, with right answers. On that card it is a short-prompt tool, not the long-context agent the model page describes.
  • You need long prompts: plan on 24 GB or more, per the table above.
  • You need speed: not yet. On the authors' code, a thinking answer takes minutes. The SGLang pull request cuts a 101-second answer to 34 today, but it is unmerged. A patched llama.cpp fork reportedly reaches about 31 tokens per second on a compressed, text-only build (llama.cpp issue #31571); we have not tested it.
  • You hoped to cut the loop for speed: don't. The second pass is what keeps the answers coherent.

To get the same word problem, chart answer and tool call on your own NVIDIA worker, run python main.py from the SenseNova-Loop-9B SDK example. It needs about 19 GB of free GPU memory and 40 GB of disk, and deletes the deployment when done.

For engineers

Credits

  • SenseTime (OpenSenseNova): the model, the looped decoder and their use of two-stream mHC in the looped block, the Transformers modeling code shipped in the Hugging Face repo, the model card, benchmarks and showcase examples. Apache-2.0. The GitHub repo holds the README, assets and examples only (checked through the GitHub API; the page fetch timed out).
  • The Qwen team: the Qwen3.5 text backbone (qwen3_5_text) and vision tower (qwen3_5_vision) the model is built on.
  • WANDY666: SGLang PR #47281, "[Feat] sensenova_loop: Add multimodal Loop and mHC inference", open, head 336c54ac, last updated 2026-10-30. Our 9.2 tokens/s comes from it.
  • sufubao: vLLM PR #64832, "[Model] Add SenseNova Loop support", open, head 86014ad4, updated 2026-11-04.
  • ModelTC (LightLLM): PR #1665 merged 2026-10-29 with a deployment guide and an "Enable MTP" section (--mtp_mode eagle_with_att --mtp_step 3 --tp 4); PR #1695 merged 2026-11-09 with AWQ checkpoints ("AWQ TP2 passed text, image, MTP, and tool-call smoke tests"). Latest tagged release is v1.2.0 (2026-08-10), so this is main only.
  • AtomicChat: the GGUF files, made with a patched llama.cpp on top of commit 5047748f11. GuideboardLabs: an out-of-tree llama.cpp fork (GuideboardLabs/llama.cpp-sensenova-loop, branch sensenova-loop) and a second GGUF repo, both linked from llama.cpp issue #31571.
  • flash-linear-attention: the Gated DeltaNet kernels the Transformers path uses (Triton JIT at warm-up).
  • AI Search (video, 1 Nov 2026, at 51:58) and the r/LocalLLaMA thread of 30 Oct, where we spotted the model.
  • Our part: the custom backend and SDK example, the deploy on two cards, the loop ablation switch, the SGLang build and MTP attempts, the measurements.

Model and files

  • Weights loaded in bf16 (the authors' listed format) from shards stored mostly in FP32; the mHC routers stay FP32.

  • Each loop visit keeps its own attention/recurrent cache (vLLM PR doc); only the full-attention layers, one in four, keep a KV cache that grows with the prompt.

  • SenseNovaLoopForConditionalGeneration, model_type qwen3_5_text; 32 decoder layers; hidden 4096, FFN 12,288, 16 query heads / 4 KV heads, head_dim 256, vocab 248,320; each group of 4 layers is 3 Gated DeltaNet (linear attention) + 1 full attention; vision tower qwen3_5_vision, 27 blocks; max_position_embeddings 262,144; mamba_ssm_dtype float32; mtp_num_hidden_layers 1.

  • loop_settings {start: 4, end: 28, eval_count: 2, counts: [2]}: layers 0 to 3 once, block [4, 28) twice with shared weights, layers 28 to 31 and the head once, 56 layer executions per forward pass.

  • Two-stream mHC (manifold-constrained hyper-connections): inside the looped block the model carries two copies of its running state and mixes them with small learned routing tables, kept in FP32 while the backbone is bf16. This is the custom part that ordinary engines need new code for.

SenseTime's figure: 32 layers stored, 56 layers run. A pre-loop block of the first layers, a looped block run N times with the same middle layers, a post-loop block and the output head. Below, the two-stream mHC router: two copies of the model's running state mixed by a small learned table, values like 0.6 and 0.4

  • mhc {streams: 2, iterations: 20, start: 4, end: 28}: Sinkhorn normalization with 20 iterations; card: "BF16 backbone; FP32 mHC routing parameters" (export precision; the published shards are mostly FP32 and our backend casts them to bf16 on load); the modeling code forces FP32 on mhc_attn / mhc_ffn through _keep_in_fp32_modules.
  • The modeling code ignores the MTP weights: _keys_to_ignore_on_load_unexpected = [r"^mtp\."].
  • Checkpoint at revision 9937e91c7301b7dcbb5bf17e3e4e95c499a11e99 (lastModified 2026-11-02): 8 safetensors shards of about 38.65 GB (mostly FP32), mtp.safetensors 486,582,880 B, tokenizer.json about 20 MB. Loaded in bf16 it is 17,960 MiB. Some catalogues (ours included) estimated "about 23 GB loaded in bf16"; the measured figure is 17.5 GiB. The AtomicChat BF16 GGUF is listed at 16.7 GB.
  • Card acceleration table: MTP "Available"; Dspark and Dflash2 "Training in progress".
  • Authors' checked runtime: Transformers 5.18.0, PyTorch 2.11.0 with CUDA 13.0, trust_remote_code=True.

Authors' benchmarks (not reproduced)

BenchmarkSenseNova-Loop-9B
Terminal-Bench 2.140.45 (Qwen3.5 9B 29.20, Qwen3.5/3.6 35B-A3B 40.45, Qwen3.5 27B 47.19, Nanbeige 4.2-3B 49.44, IQuest-Coder 40B-Loop 17.98)
BIRD-Dev71.38
WorkBuddy66.89
AIDABench58.87
TeamBench48.89
T3-Banking18.56
AutomationBench13.72
RaccoonPPTBench (internal)70.70

Where it runs today (checked 2026-11-09)

EngineStatusNotes
Transformers 5.18 + trust_remote_codeworks (what we deployed)text and images, no MTP, 2.2 to 3.0 tokens/s, launch-bound
SGLang PR #47281open PR, built and tested by ustext and images, 9.1 to 9.2 tokens/s on an A40, MTP not wired
vLLM PR #64832open PR, not triedtext only, "Image inputs and MTP remain unsupported", unquantized weights only, no CPU offload with shared loop weights; its doc notes each loop visit keeps its own attention/recurrent cache, so weight sharing does not shrink the cache
LightLLM mainmerged, not in a release, not triedcard example ghcr.io/modeltc/lightllm:main with --tp 4; MTP documented
llama.cpp upstreamnot supportedissue #31571 open; only the unrelated SenseNova U1 is supported
AtomicChat GGUF on patched llama.cppnot triedtext only, no mmproj, MTP block excluded; BF16 16.7 GB, Q8_0 8.9 GB, Q6_K 6.9 GB, Q5_K_M 6.0 GB, Q4_K_M 5.2 GB (file table; the sidebar lists different sizes)
Ollama, LM Studionot supported per the GGUF card and issue #31571not tried by us

Setup

  • GPU: one A40 48 GB on our cloud (cc 8.6, driver 595.91.07 / CUDA 13.2, 9 vCPU). Confirmation: one real RTX 4000 Ada Generation 20 GB (20,475 MiB, cc 8.9, 9 vCPU).
  • Platform: a local InferenceKey manager built from main (3666da3b), worker connected through a tunnel; production not touched.
  • Backend: custom Python backend sensenova-loop-9b 0.1.1, because no built-in engine can load the architecture and Transformers is the only path with images that the authors checked.
  • Weights pre-downloaded with hf_xet: 38.2 GB in 21 s (A40) and 28 s (RTX 4000 Ada).

Step by step with the SDK example

  1. Clone inferencekey-sdk and open examples/sensenova-loop-9b-vision-chat-custom-backend-private-nvidia (commit 6c585a0462e33351a17ba5050ec8ee8cc8c01be5). Files: backend.py, publish.py, main.py, requirements.txt, README.md, .env.example, samples/energy-chart.png.
  2. cp .env.example .env and fill in INFERENCEKEY_SDK_TOKEN, INFERENCEKEY_API_KEY, INFERENCEKEY_PROJECT and IK_WORKER_ID: a private NVIDIA worker, bf16-capable (Ampere or newer), about 19 GB of free GPU memory (21 GB for a 12K-token prompt), about 40 GB of free disk and a driver for CUDA 12.8 or newer.
  3. Install the client: pip install requests and the local SDK build (pip install maturin && (cd ../../bindings/python && maturin develop --release)). requirements.txt is for the worker: inferencekey, transformers==5.18.0, torch==2.11.0, torchvision==0.26.0, accelerate, safetensors, pillow, numpy, flash-linear-attention, huggingface_hub[hf_xet].
  4. python publish.py once, with INFERENCEKEY_TENANT_ID and INFERENCEKEY_USER_TOKEN (or upload dist/*.tar.gz in Dashboard, Custom backends).
  5. python main.py. It ensure()s a workload with backend sensenova-loop-9b, task text2text, policy FIXED, ready_timeout_secs 3600 and request_timeout_secs 1200, waits, sends the three requests to /v1/chat/completions, writes ./outputs/*.json and calls delete() unless IK_KEEP_ALIVE=1.

What the backend does:

  • snapshot_download pinned to revision 9937e91c... with ignore_patterns ['assets/*', 'examples/*', '*.md', 'mtp.safetensors'].
  • AutoModelForImageTextToText in bf16 with trust_remote_code, AutoProcessor, then a warm-up.
  • LOOP_VRAM_LIMIT_GIB caps PyTorch with torch.cuda.set_per_process_memory_fraction (how we emulated the smaller cards).
  • Per request (in extra_body): enable_thinking (default true), max_tokens (capped at 8192), temperature / top_p (greedy by default), tools, and loop_passes 1 to 4 (default 2, the config's eval_count), which rebuilds the text model's _loop_plan for that request. Images as data: URLs, longest side capped at 1536.
  • Returns content, reasoning, tool_calls (OpenAI shape) and timings (tokens, tokens/s, first token, peak VRAM, layer runs per token).

The request that produced the ablation, written with the SDK example's ask() helper:

ask(url, key, [{"role": "user", "content":
    "Three painters paint 2 fences in 5 hours. Working at the same rate, how many hours "
    "do 6 painters need for 8 fences? End with 'Answer: <number>'."}],
    max_tokens=1024, enable_thinking=False, loop_passes=1)  # 2 = trained setting

Timings

PhaseA40 (run 1)RTX 4000 Ada (fresh worker)
Ready after ensure()57.4 s106.7 s
Python venvreusedabout 30 s
Load16.3 s22.8 s
Warm-up36.2 s (32.8 s to first token, Triton JIT of the fla kernels)40.4 s

In-process ladder runs with a warm Triton cache: setup 20.6 to 22.6 s (load 11.3 to 12.8 s, warm-up 4.7 to 4.8 s).

Requests through the platform (greedy, 2 passes)

CaseA40RTX 4000 Ada 20 GB
Word problem, thinking on306 tokens, 101.4 s, 3.02 tok/s, first token 0.45 s, peak 18,088 MiB, "Answer: 10"310 tokens, 137.9 s, 2.25 tok/s, "Answer: 10"
Chart, thinking on (964 prompt tokens)455 tokens, 156.6 s, 2.93 tok/s, first token 1.55 s, peak 18,295 MiB455 tokens, 203.8 s, 2.25 tok/s, identical text
Tool call, thinking off73 tokens, 29.4 s, 2.49 tok/s73 tokens, 33.6 s, 2.17 tok/s
Chart, thinking off161 tokens, 69.2 s, 2.33 tok/s, right
Pen price (true 0.15), thinking off"0.65", wrong"0.65", wrong
second_largest() coderight, 35 tokens, 2.58 tok/s
Story, 160 tokensfluent, 2.47 tok/s
Hidden code in a long prompt12,046 tokens: "7319-LOOP" right, first token 14.6 s (5.3 s warm), peak 20,383 MiB3,046 tokens right, first token 11.1 s, peak 18,649 MiB; about 6K tokens: 503 "CUDA out of memory. Tried to allocate 142.00 MiB ... total capacity of 19.55 GiB", next request 200

Chart arithmetic: 9,940 - 6,830 = 3,110 kWh, at 0.21 EUR/kWh = 653.10 EUR.

Loop ablation (loop_passes, A40, thinking off, greedy)

Case2 passes (56 layer runs, trained)1 pass (32)3 passes (84, untrained)
Fencesright, 277 tokens, 2.34 tok/s, 118 sno answer: "( 6 个围栏) 27 次", then "并排" repeated, then "∀" to 1,024 tokens, 4.70 tok/s, 218 s
Chartrightreads 3,350 as 3,530, writes "kg", switches to Chinese, 1,414 kWh / 293.34 EUR (wrong), 4.43 tok/s
Pen price"0.65" (wrong)rambles, "欧元", cut at 64 tokens
Train timetablecoherent"Zarhanda", "Start: 11:47 (09:47)"
Storyfluent, 2.47 tok/srepeats its opening, "seerrag" / "searg", 4.46 tok/sfluent, 1.68 tok/s

Why it is slow on Transformers

Decode is launch-bound: the Python process at 100 % of one CPU core, GPU utilization 38 to 53 % (nvidia-smi every second). causal_conv1d is not installed and falls back to the PyTorch reference path. Every token runs 56 layer executions, so the kernel launch count roughly scales with the loop; the 1-pass ablation's 2x speed-up is consistent with that.

SGLang PR #47281 (outside the platform, same A40)

git clone --filter=blob:none https://github.com/sgl-project/sglang.git && cd sglang
git fetch origin pull/47281/head && git checkout 336c54ac1e721197806d94b6929f567446a59693
uv venv -p 3.12 ../sgvenv && source ../sgvenv/bin/activate   # venv python and ninja on PATH for every step below
SGLANG_BUILD_RUST_EXTS=none uv pip install -e ./python
uv pip install ninja   # CUDA graph capture JIT-compiles kernels
python -m sglang.launch_server --model-path <snapshot of 9937e91c> --served-model-name sensenova-loop \
  --context-length 32768 --mem-fraction-static 0.88 --max-running-requests 4
  • Build 73 s: sglang 0.5.24.dev152+g336c54ac1, torch 2.14.1, sglang-kernel 0.4.10, flashinfer 0.7.2, transformers 5.19.0.
  • --mem-fraction-static applies to free memory: at 0.5 beside our backend it failed with "Not enough GPU memory for hybrid (mamba/linear-attention) state cache ... max_mamba_cache_size=-29".
  • Running: CUDA graphs for batch 1/2/4 in 62 s, KV 49,618 tokens, mamba cache 26 slots. Story 400 tokens 9.2 tok/s (twice), fence 306 tokens in 33.6 s (identical text to Transformers), chart 455 tokens 9.1 tok/s (identical answer), tool call 67 tokens 9.2 tok/s (raw <tool_call>, no parser set). TTFT 0.37 to 0.81 s.

MTP attempts

  • Transformers: mtp.* ignored on load. vLLM PR: documented as unsupported.
  • SGLang PR, speculative decoding NEXTN, 3 steps, top-k 1, 4 draft tokens. Attempt 1 beside our backend: OOM loading the draft. Attempt 2 (GPU free): the draft was the full SenseNovaLoopForConditionalGeneration again (17.57 GB), then failed on hybrid state cache memory; the PR does not map sensenova_loop to the Qwen3.5 MTP draft class.
  • Attempt 3 with our one-line patch (adds the architecture to the Qwen3_5ForCausalLMMTP rewrite list in model_config.py): MTP head loads (4.26 GB, 1.85 s), target verify graphs captured (29 s), draft decode CUDA graph capture crashes with TypeError: AttentionBackend.forward() missing 3 required positional arguments: 'q', 'k', and 'v' (radix_linear_attention.py:151 from qwen3_5.py:1087). Stopped there.
  • LightLLM main documents MTP (--mtp_mode eagle_with_att --mtp_step 3 --tp 4, test/start_scripts/sensenova_loop/sensenova_loop_mtp.sh); untested by us. No speed-up from MTP has been measured by us on any path.

GPU memory and VRAM ladder

  • After load: 17,960 MiB allocated. nvidia-smi 18,599 to 18,783 MiB for the process with short prompts.
  • PyTorch peak: 18,037 to 18,295 MiB (short, up to 966 tokens and one image), 18,649 MiB (3,046 tokens), 20,374 to 20,383 MiB (12,046 tokens).
GPU or capConfigPeak VRAMTime per outputOutput OK?Real or emulated
A40 48 GBbf16, custom Transformers backend, through the platform18,295 MiB (short), 20,383 MiB (12K)chart 69.2 s (2.33 tok/s), fence 118 syes (reference)real
22.5 GiB cap (24 GB card)same backend in-process, set_per_process_memory_fraction; chart + fence + 12K needle, thinking off20,374 MiB PyTorch, 22,278 MiB smichart 68.9 s (2.35 tok/s), fence 120 s, needle first token 5.3 syes, identical answersemulated
RTX 4000 Ada 20 GBfresh worker through the platform, main.py + long prompts18,295 MiB (chart), 18,649 MiB (3K)chart with thinking 204 s (2.25 tok/s), tool 33.6 syes up to about 3K tokens; about 6K tokens 503 OOM ("Tried to allocate 142.00 MiB ... total capacity of 19.55 GiB")real
18.5 GiB cap (20 GB card)same in-process18,291 MiB PyTorch, 19,174 MiB smichart 68.2 s (2.38 tok/s), fence 115 s (2.40)short yes; 12K OOM ("Tried to allocate 48.00 MiB ... 18.50 GiB allowed")emulated
14.5 GiB cap (16 GB card)sameno: load fails ("Tried to allocate 17.54 GiB ... 14.50 GiB allowed")emulated

Emulated rungs ran on the A40, so their speeds are A40 speeds. Not tried: a real 24 GB card, the GGUF files, int8 / fp8 / 4-bit loads, CPU offload, concurrency, the 256K context, LightLLM.

Remaining problems

  1. No released engine serves the model. SGLang and vLLM support are open PRs (vLLM text only); LightLLM is on main only. Open upstream.
  2. Slow decode on the reference path: 2.2 to 3.0 tok/s, launch-bound. Open upstream; the SGLang PR gives 9.2.
  3. MTP advertised, not runnable on any one-GPU path we tried. Open upstream.
  4. PyPI SDK runtime behind the docs: backend 0.1.0 failed with "cannot import name 'chat_completion' from 'inferencekey.backend'" because the worker installs PyPI inferencekey 0.260725.739. Fixed in backend 0.1.1 (chat routed to process()); the publishing gap is open on our side.
  5. Tokenizer warning at every load ("incorrect regex pattern ... fix_mistral_regex", Transformers 5.18.0). Outputs were fine. Not investigated.
  6. 20 GB limit: about 6K-token prompts run out of memory; the backend stays up.
  7. Model: the pen price question is wrong without thinking on both cards.

Demo video

SenseNova-Loop-9B deploy test

161 s, narrated: the authors' showcase examples first, the architecture and their benchmark chart, the chart answer through the endpoint on the A40, the loop comparison, speeds and memory, and the real 20 GB worker. One correction: the speed insert says no public engine runs MTP yet. LightLLM main documents MTP on multi-GPU setups (see above); we did not test it.

Sources