Groot-19B-A3B fits its full 256K context on one 32 GB RTX 5090, and a 24 GB memory cap is not enough

Groot-19B-A3B's authors say its full 256K context fits on one 32 GB RTX 5090. We checked on our cloud, and it does: all 262,144 tokens (the "256K") started, and a 205,525-token prompt came back right. Capped at 24 GB, the same file left room for about 9,600 tokens, too little for agent work. So plan on 32 GB for this build. We used the 20.9 GB compressed build (FP8), vLLM, the open-source server our platform ships, and the authors' settings. For a team that owns its hardware, that is a 256K-token window on one gaming-class card.

What you want to know	What we measured
Smallest GPU that ran it	one real RTX 5090, 32 GB, full 262,144-token window
Writing speed, one request	262 to 304 tokens/s, 170 to 211 without its speed-up layer
Tool calls	valid JSON, two calls in one turn plus the answer in 1.8 s
24 GB of memory (simulated on the same card)	will not start with 64K tokens, 9,600 at most

A 19B model that its authors put ahead of its own 35B parent

Groot-19B-A3B is built from a bigger model and retrained for agents. Its authors report that it beats the original. GrootTeam started from Qwen3.6-35B-A3B, the Qwen team's open 35B model, and cut it to 19B. The authors' page does not say how. They retrained it on a small amount of open data, fewer than 0.1 billion tokens, says the model page. They published it on 23 October in four builds, and we spotted it in AI Search's news video of 1 November.

It works like a large team where only a few specialists handle each word. About 3 billion of its 19 billion parameters work on each word. So it writes fast, but the GPU must hold all 19 billion.

On the authors' table, it beats both Qwen3.6-35B-A3B and Gemma-4-26B-A4B on 12 of 16 benchmarks. Two of them matter most for agent work:

Terminal-Bench 2.1, where an agent works in a command line: 52.1 against 43.8 for Qwen3.6 and 26.5 for Gemma-4.
NL2Repo, where an agent builds a whole code repository from a description: 35.7 against 28.8 and 16.6.

The authors also say its answers were the shortest of 12 models: 23.1% shorter than Qwen3.6, 65.6% shorter than Gemma-4. Their page asks you to read that together with accuracy, which is fair. They compare it only with open models of its size, not with the current closed flagships.

It also reads images: asked to read the authors' chart, it gave about 42, against 41.44 on their page. The license tag on all four repos says Apache-2.0, which allows commercial use, though the LICENSE file the page links is missing. Besides the full 38.9 GB file there are compressed builds of 20.9 GB (the one we tested), 13.5 GB and 11.7 GB.

"256K on one card" is a promise about memory, and Groot's smaller weights make the room

The promise is about memory. Think of it as a notepad. A model on a GPU needs memory for two things: its own weights, and a notepad that holds the conversation so far. The weights are a fixed cost. The notepad grows with every token of context you allow. Groot keeps the layout of Qwen3.6-35B-A3B: a quarter of the model takes full notes on every word, the rest keeps a short summary. So the notepad is as small as Qwen3.6's. What frees the room is the weights, 19B instead of 35B.

The authors' page says the 20.9 GB and 13.5 GB builds "run the full 256K context on one RTX 5090", and adds that this is an estimate from memory sizes. So we went straight to that card.

The limits: one GPU, one request at a time, our own prompts, a few runs each. This checks whether the model fits and how fast it answers. Its skill as an agent comes from the authors' benchmarks: beyond a two-call tool test, we did not test it, and no multi-step agent session ran.

The full window fits: a 205,525-token prompt answered right, first word after 35 seconds

The full 262,144-token window started on the 32 GB card. After loading the weights, vLLM had 6.8 GiB left for the notepad, enough for 307,024 tokens: the full window plus about 45,000 tokens to spare. So one agent gets the full window. Several agents at once must each take a shorter one.

We hid one sentence 60% of the way into a 205,525-token prompt: the access code for a warehouse. The model found it and gave the code back. The first word of the answer came after 34.6 seconds, and the full answer after 40.3. One right answer shows the window works. It does not test reasoning across long documents.

The GPU was almost full while it did that. Serving normally, it used about 93% of the card's memory. The long prompt left under 2% free. It is a tour van packed so tight the amp is wedged against the door: it rolls, but nothing else gets in.

About 270 tokens per second, and the speed-up layer adds about a quarter

A 4,457-token answer took about 17 seconds through our API, 262 to 304 tokens per second. Short answers were at the top of that range, and replies of 4,500 to 5,200 tokens at the bottom. The reply started in under a second (0.26 to 0.61 s).

Part of that speed comes from a small extra layer the authors ship with the model. It guesses the next two tokens, and the main model checks the guess in the same step it would have spent on one. When the guesses are right, several tokens come out for the price of one. With that layer switched off, vLLM wrote 170 to 211 tokens per second. On long answers the layer added about 25%, roughly: one run used our API, the other went straight to vLLM.

Tool calls came back clean, and every answer starts with a long think

Tool calling worked the first time. We tried the authors' own example, the weather in Paris. The model asked our code for get_weather with the city in valid JSON in 0.9 seconds. It answered with the result in under one more second.

Then we gave it two cities: is it warmer now in Lisbon or in Oslo? In one turn it asked for both cities at once, two calls side by side. After our fake weather service answered, it wrote "Lisbon is warmer right now with 19°C compared to Oslo's 4°C". The whole exchange took 1.81 seconds. An agent framework expects exactly this shape.

Plan for long thinking even on short questions: Groot thinks before every reply, and the thinking counts as output. "The capital of Australia, in one word" took 111 to 131 tokens, almost all of it thinking, before "Canberra". A request for a 300-word essay produced 2,383 to 7,156 tokens across four runs. In the two runs we counted, 2,043 and 6,785 of those tokens were thinking. We did not run Qwen3.6 on the same prompts, so we cannot confirm or refute the authors' 23.1%. Set a generous token limit.

Which GPU: a real 32 GB RTX 5090 at full context, nothing smaller started

Only the real 32 GB card started. Below 32 GB, nothing we tried started. We did not have smaller cards for this test, so we told vLLM on the same 5090 to use only part of its memory. That emulates a smaller card's budget, but it is not a real smaller card.

24 GB budget, FP8, 65,536 tokens asked: did not start. The weights alone take 86% of that budget, which left 0.48 GiB for the notepad. vLLM computed a maximum of 9,648 tokens, too short for an agent that reads files and tool results. The authors' page recommends keeping at least 128K.
16 GB budget, the 13.5 GB build, 32,768 tokens asked: did not start. It crashed at startup, a software fault rather than lack of memory, so we could not check the authors' claim for this build.

The appendix lists every rung we skipped.

On InferenceKey: the stock vLLM backend and the authors' own recipe, ready in about 10 minutes

A new model should not need a custom engine, and this one did not. Our platform runs a model on a worker, a GPU machine you own or rent and connect to it. We pointed the worker at the FP8 build with vLLM and the authors' settings, and it served. From the deploy call to a ready server took 611.5 seconds on a fresh machine. About 3 minutes went to installing vLLM, 28 seconds to downloading 20.9 GB, and 5 minutes to vLLM warming up the engine and its speed-up layer. On a restart, with that work saved on disk, the warm-up took 87 seconds instead of five minutes. Part of that slow start is ours: our worker leaves out a ready-built part, so vLLM builds it itself. Status: open.

The problem you will hit first is the thinking switch. The authors' page turns thinking off with a per-request setting. Our API does not pass that setting through to the model yet, so the request still thinks. Asked with thinking off, it still reasoned before it said "The Eiffel Tower is located in Paris, France." Status: open on our side. The workaround is to start the server with thinking off by default, and our example's README shows the one-line flag. The rest of what we found, including two small errors on the authors' page, is in the appendix.

Should you use it? Yes if you need a 256K window on one 32 GB card

If you want a long context on hardware you own, this is a strong candidate on fit and speed.
If your card has 24 GB, skip the 20.9 GB build. The authors suggest their 11.7 GB one, which we did not test.
If you need quick chat replies, budget for the thinking that comes before every answer.
If you need the smaller 13.5 GB build, test it yourself on another server version: it did not start in our setup.

To run the same deployment, a tool call and the authors' chart on your own GPU, run python main.py from the Groot-19B-A3B SDK example. It needs an InferenceKey project with an SDK token and an API key, and a private NVIDIA worker with 32 GB or more. It deletes the deployment when it finishes.

For engineers

Credits

Groot-19B-A3B-261023: GrootTeam. They post-trained the model, published the BF16, FP8, NVFP4 and GGUF builds (created 2026-10-23, not gated, license:apache-2.0 tag), the MTP layer, the vLLM and SGLang recipes, the benchmark tables and the charts we used as vision tests. Repos: BF16, FP8, NVFP4, GGUF.
Base model: Qwen3.6-35B-A3B from the Qwen team.
Serving: the vLLM project (0.32.0) and FlashInfer.
Where we spotted it: AI Search, "AI finally sleeps, Gemini 4 Carbon, ETH cracked, energy-based models, Anthropic IPO: AI NEWS", uploaded 2026-11-01, Groot chapter at 47:33. The video covers four Groot variants; this post is only about the 19B.
Our part: the deployment through the platform, the prompts and tests, the measurements, the VRAM ladder, the SDK example and the demo video.

Model and files

A mixture-of-experts (MoE) model, the design we walked through in our Sage-A1.5 post. Its hybrid layout comes from Qwen3.6-35B-A3B.
Architecture Qwen3_5MoeForConditionalGeneration: about 19B total parameters plus a 0.4B MTP layer, about 3B active per token. 40 layers, 10 full attention (full_attention_interval 4) and 30 linear attention. Hidden 2048, 16 attention heads, 2 KV heads, head_dim 256, vocab 248,320. 128 routed experts plus 1 shared, 8 routed active per token. max_position_embeddings 262,144, extensible to about 1M with YaRN (factor 4.0), per the authors' page.
FP8: block 128x128, E4M3, dynamic activations.
Rough KV estimate for the full-attention layers: 10 layers x 2 KV heads x 256 x 2 (K and V) x 2 B = 20 KiB per token, so about 5 GiB for 256K, plus the MTP layer and the linear-attention states.

Build	Size on the page	HF `usedStorage`	Repo sha
BF16	38.9 GB	38,883,686,769 B (no `mtp.safetensors` listed)	`d089dadf`
FP8 (E4M3, dynamic)	20.9 GB	20,946,032,529 B (6 shards + `mtp.safetensors` 450,538,880 B)	`49137722`
NVFP4 (NVFP4 experts, FP8 attention)	13.5 GB	13,546,543,381 B (4 shards + `mtp.safetensors` 883,452,968 B)	`8ff73aca`
GGUF Q4_K_M	11.7 GB	11,657,618,560 B + mmproj BF16 902,822,016 B	`2fa1ec95`

Authors' claims (not reproduced)

Benchmark	Groot-19B-A3B	Qwen3.6-35B-A3B	Gemma-4-26B-A4B
Terminal-Bench 2.1	52.10	43.82	26.52
NL2Repo-Bench	35.65	28.77	16.55
LongBench v2	62.03	48.12	47.99
16-benchmark unweighted average	41.44	38.08	30.05

Ahead of both on 12 of 16 benchmarks. 45.7% fewer total parameters than Qwen3.6-35B-A3B, 26.9% fewer than Gemma-4-26B-A4B.
Token budget: shortest average output of the 12 measured models across the 16 runs, 23.1% fewer output tokens than Qwen3.6-35B-A3B and 65.6% fewer than Gemma-4-26B-A4B.
Eval settings: thinking mode at default effort, temperature 1.0, top_p 0.95, top_k 20, 256K context. Terminal-Bench 2.1 with the Claude Code agent (max_output_tokens 128K, max_turns 500, 12 h timeout). NL2Repo-Bench with the Claude Code agent, max_turns 1000, 80K context. HLE judged by gpt-5.5.
"Where it runs" (stated as estimated from memory capacity): BF16 on one 48 or 80 GB GPU; FP8 and NVFP4 at 256K on one RTX 5090; GGUF at 128K on a 16 GB RTX 4060 Ti and 256K on a 24 GB RTX 4090/3090; 256K on 128 GB unified-memory machines. Against Qwen3.6-35B-A3B: on 16 GB Groot runs 128K while Qwen3.6 does not load; on an RTX 5090, 256K against 64K.
Runtime requirements: Transformers >= 5.8.1, vLLM >= 0.19.1, SGLang >= 0.5.10. Thinking on by default, off with chat_template_kwargs={"enable_thinking": False}, no /think or /nothink switches. Recommended sampling for thinking: temperature 1.0, top_p 0.95, top_k 20, min_p 0, presence_penalty 1.5. Coding: temperature 0.6, presence 0. Non-thinking: temperature 0.7, top_p 0.8.

Setup

One RTX 5090, 32,607 MiB, 575 W, driver 595.91.07 (CUDA 13.2), compute capability 12.0, 100 GB disk, on our cloud (InferenceKey Cloud).
Why this card: the test plan named an L40S for the main run, but the authors' headline claim names the RTX 5090, so the main run went straight to it. The squeeze rungs ran on the same machine under emulated memory caps.
Requests went through the platform's OpenAI-compatible endpoint (/v1/chat/completions, streaming); the MTP-off speed runs went to vLLM directly.
Stock vllm backend, vllm_version="0.32.0" (installs Transformers 5.19 and torch 2.13 for CUDA 13). SGLang not used.
A local InferenceKey manager built from origin/main ce3af2dc bootstrapped the worker. Production was not involved.
Sampling for the speed runs: temperature 1.0, top_p 0.95, presence_penalty 1.5, streaming. top_k=20 comes from generation_config.json.

Step by step with the SDK example

Clone inferencekey-sdk and open examples/groot-19b-a3b-fp8-agent-vllm-private-nvidia/ (commit 1c254204).
Build the Python SDK locally with maturin (see examples/CONTRIBUTING.md, section 4) and pip install -r requirements.txt (requests>=2.31).
cp .env.example .env and set INFERENCEKEY_SDK_TOKEN (ik_sdk_, scopes workload:read and workload:write), INFERENCEKEY_API_KEY (ik_live_, with the workload in allowed_endpoints), INFERENCEKEY_PROJECT and IK_WORKER_ID (a private NVIDIA worker with 32 GB or more and a CUDA 13 driver, 580 or newer). Optional: GROOT_CONTEXT (default 262144), GROOT_MTP (default 1), IK_KEEP_ALIVE=1.
python main.py. The flow: ensure() a fixed vllm workload pinned to your worker (ExecutionPolicy.FIXED), wait_until_ready(timeout=2400), a plain question with generate_text(), the authors' streamed "I love Groot backwards" prompt, a get_weather tool call with the tool result sent back, the authors' chart as an image, answers to outputs/answers.json, then delete() unless IK_KEEP_ALIVE=1.

The command the example deploys:

vllm serve GrootTeam/Groot-19B-A3B-261023-FP8 --max-model-len 262144 \
  --gpu-memory-utilization 0.92 --max-num-seqs 8 --enable-prefix-caching \
  --limit-mm-per-prompt '{"image": 2, "video": 0}' \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --speculative-config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}'

To serve without thinking, add:

--default-chat-template-kwargs '{"enable_thinking": false}'

Main example output: the backwards prompt, 460 tokens in 2.12 s at 307.1 tok/s, answered "toorG evol I"; the MoE explanation question in 3.8 s.

Timings

ensure() to ready on a cold worker: 611.5 s. vLLM 0.32.0 install about 170 s, weight download 27.8 s, load 51.7 s (19.67 GiB), engine warm-up 312.6 s (compile 37.5 s, two CUDA-graph and speculator captures of about 2 min each, FlashInfer JIT).
Warm restart (MTP off, caches on disk): engine warm-up 86.8 s (compile 37.1 s), weights 19.24 GiB in 23.4 s.

Memory

--gpu-memory-utilization 0.92, 262,144 context, MTP on: available KV 6.77 GiB = 307,024 tokens, maximum concurrency 1.17x. CUDA graph pool 0.19 GiB.
Same with MTP off: 7.37 GiB = 376,253 tokens, 1.44x.
nvidia-smi every second: 30,226 MiB while serving, 32,016 of 32,607 MiB peak during the 205K prompt (530 to 566 W).

Speed (one request, streaming)

Path	Prompt	Completion tokens	First token	Decode
Endpoint, MTP on	"capital of Australia, one word"	111 / 115	0.26 / 0.60 s	284.4 / 304.1 tok/s
Endpoint, MTP on	300-word essay	5,159 / 4,457	0.61 / 0.61 s	265.9 / 261.9 tok/s
Direct vLLM, MTP off	short	125 / 131	0.59 / 0.54 s	170.5 / 187.1 tok/s
Direct vLLM, MTP off	essay	2,383 / 7,156	0.79 / 0.68 s	210.4 / 210.6 tok/s
Endpoint, MTP on	205,525-token needle prompt	119	34.62 s	prefill about 5,900 tok/s incl. transfer, total 40.27 s, "4471-HERON" right

The MTP comparison mixes two paths (MTP on through the gateway, MTP off direct). Gateway overhead was not measurable in our earlier Sage-A1.5 test and was not re-measured here.
Reasoning tokens (usage.completion_tokens_details.reasoning_tokens, direct runs): 120 of 125 on the short question; 2,043 of 2,383 and 6,785 of 7,156 on the essays. reasoning_content 9.6K to 23.7K characters on the essays.

Tools, vision, thinking

The authors' get_weather example: {"city": "Paris"}, valid JSON, 0.9 s; final answer 0.96 s.
Two cities: one turn with two parallel get_weather calls (Lisbon, Oslo; 101 completion tokens, 0.93 s), then the answer (50 tokens, 0.88 s), 1.81 s total. Recorded take: 2.9 s (turn 1 454 tokens in 1.99 s, turn 2 65 tokens in 0.91 s). Reasoning length on tool turns was not reliably captured in the main test (0 chars recorded, while the recorded take shows 1,287 and 147 chars).
Scatter chart (9,279 prompt tokens): "Groot-19B-A3B ... approximately 42" in 4.12 s. Bars chart (16,329 prompt tokens): "52.10 ... Gemma-4 (26.52) and Qwen3.6 (43.82)" in 9.21 s, first token 7.25 s.
chat_template_kwargs={"enable_thinking": false} through the endpoint: still thought (571 chars of reasoning, 157 completion tokens).

VRAM ladder

GPU or cap	Config	Peak VRAM	Speed	Output OK	Real or emulated
RTX 5090 32 GB	FP8, 262,144 ctx, util 0.92, MTP on (the example, through the endpoint)	30,226 MiB serving, 32,016 MiB during the 205K prompt	262 to 304 tok/s	yes	real
RTX 5090 32 GB	FP8, 262,144 ctx, util 0.92, MTP off (direct vLLM)	not sampled separately	170 to 211 tok/s	yes	real
24 GB cap	FP8, 65,536 ctx, util 0.72, MTP on	did not start	n/a	no	emulated
16 GB cap	NVFP4, 32,768 ctx, util 0.48, MTP on	did not start (weights 12.66 GiB loaded in 35.6 s)	n/a	no	emulated

24 GB cap error:

ValueError: To serve at least one request with the model's max seq len (65536), (1.66 GiB KV cache is needed, which is larger than the available KV cache memory (0.48 GiB). Based on the available memory, the estimated maximum model length is 9648.

16 GB cap, NVFP4: vLLM logs that the GPU has no native FP4 support and falls back to weight-only FP4 compression with the Marlin kernel, then the FlashInfer 0.7.0.post1 JIT build of gen_gemm_sm120_group_gemm_fp8_groupwise_e4m3_bf16_majorfalse_sm120 and its siblings fails (nvcc 13.0.88), ending in RuntimeError: Engine core initialization failed.

Not tried: a 24 GB FP8 rung at 9.6K or less (vLLM's own number says it would start, but the authors recommend at least 128K), the GGUF build on llama.cpp, NVFP4 on another vLLM or FlashInfer version or on SGLang, real 24 and 16 GB cards, the RTX PRO 4500 32 GB, the L40S, BF16. The SDK README's compatibility line lists RTX PRO 4500/5000, L40S and A6000-class cards as well; only the RTX 5090 was measured.

Remaining problems

Endpoint drops chat_template_kwargs, top_k and min_p. The authors' page turns thinking off per request with chat_template_kwargs={"enable_thinking": False}. The manager's ChatCompletionRequest has no such fields, so they are dropped. Thinking cannot be turned off per request. Workaround: --default-chat-template-kwargs on the server. Status: open, feature gap.
Slow first start. 611.5 s cold, mostly engine warm-up. The worker does not install flashinfer-cubin with vLLM 0.32 (warning "FlashInfer's precompiled kernels are missing"). Status: open.
Full window at the edge of the card. Less than 600 MiB free during the 205K prompt. Keep other processes off the GPU, or set GROOT_CONTEXT=131072.
NVFP4 does not start on the RTX 5090 with vLLM 0.32.0 (see the ladder): no native FP4 path on sm_120, so vLLM uses the Marlin weight-only fallback, then the FlashInfer JIT build fails. This may be our worker image, which lacks flashinfer-cubin (problem 2). Options: add the precompiled kernels, try another vLLM or FlashInfer version, or try SGLang (none tested). For 16 and 24 GB cards the authors point to the GGUF build on llama.cpp (not tested). Status: open, possibly our worker image.
Model page errors. The serving commands use Groot/Groot-19B-A3B-261023, which does not resolve (HF API 401); the repo is GrootTeam/Groot-19B-A3B-261023. The front matter links a LICENSE file that returns 404 in all four repos; the license is only the apache-2.0 tag and metadata. Status: reported here, still on the page.
Deprecation. vLLM 0.32 logs qwen3_next_mtp as a deprecated alias of mtp. It works.
Scheduler limit. The speculative settings set max_num_scheduled_tokens to 2048.

Summary image

Three numbers from our test of Groot-19B-A3B FP8 with vLLM on one RTX 5090: the full 256K context fits, with room for 307,024 tokens and a 205,525-token prompt answered right; 262 to 304 tokens per second for one request with the model's draft layer and 170 to 211 without; 29.5 GiB on the GPU while serving, 32,016 of 32,607 MiB at the peak of the long prompt

Demo video

The 143-second narrated demo: the deployment in the panel, the two-city tool call on the RTX 5090, the authors' chart and the memory ladder. The poster shows our demo tenant.

Sources

Model page and repos: BF16, FP8, NVFP4, GGUF (README identical in all four), and the LICENSE link (404 on 2026-11-03).
Hugging Face API listings of the four repos, fetched 2026-11-03.
SDK example, commit 1c25420460d2277219490af8bcf0ec7c173aa423.
AI Search video, IGICYFyP_EI, where we spotted the model.
Our own logs: the vLLM and worker logs, nvidia-smi samples every second, and the JSONL outputs of the speed, needle, tool, chart and thinking tests.