A model that would need 1.5 TB of VRAM in FP16, running in just 192 GB

On Tuesday, October 6, we tried to load DeepSeek-V4.1-Flash, a 763B parameter model, on a machine with two 96 GB graphics cards. The first two attempts ran out of memory. The third one started and began answering.

Models this size usually run on data center machines. We wanted to know whether two cards were enough and what they could deliver. We measured all of it and wrote it up in detail in a paper (PDF). Here is the summary:

And the model is worth it. On LiveBench, an independent leaderboard that rotates its questions so nobody can memorize them, it ranks first of 69 in agentic coding:

These are scores of the full models, served by their own providers.

Why it did not fit

With 763B parameters at two bytes each, the model would take about 1.5 TB in FP16. DeepSeek publishes it already compressed, at about 475 GB, and the community brings it down to about 310. Split across the two cards, each one gets 102.1 GB, and each card has 95.6. That is six gigabytes short per card, before counting the memory the conversation itself needs. The first two attempts failed for exactly that reason.

How it got in: part of the model in the computer's memory

DeepSeek-V4.1-Flash works like a team of specialists. Each of its 40 layers has 384 of them, and for each token (a piece of a word) only 6 do any work. The rest sit idle.

If most of them are idle, they do not need to live on the card. The third attempt used a file that sends a third of those specialists, 4,608 of 15,360, to the computer's regular memory. When a token needs one of them, the card reads it from there directly. That leaves each card with about 72 GB, and the model fits with room to spare.

Who made it possible

That trick is not ours, and the result belongs to many people. DeepSeek released the model's weights so anyone could use them. coolbho3k compressed it to 3 bits. turboderp created the compression format and the software that makes it fast. Victor Cruz connected it to vLLM, an open source model server. Diffbot had the idea of sending specialists to the computer's memory, built it and published the recipe.

We put it all together, made it work end to end and measured it: what fits in memory, how fast it answers, how many people it serves, what it costs and whether it still answers as well. Before starting we checked who had written each piece, to credit everyone for their part. We also confirmed that, with only what vLLM ships by default, the model does not run on these cards.

Huge contexts without losing pace

A coding agent works with a lot of context: the project, the history, the open files. So we tested it with huge texts.

With a short question, the server wrote each token in 7.6 thousandths of a second. With a context sixty-four times longer, about 256,000 tokens, it took 8.3, almost the same.

Reading a huge text for the first time does take a while: 18 seconds for 128,000 tokens. But an agent always works on the same project, and what has already been read does not need reading again. With four people working on the same giant context, the answer started arriving in under two seconds.

Four agents at full speed

Programming with an agent beside you needs it to answer smoothly. We set the bar at 40 tokens per second for each person.

With one agent working alone, the server gave it about 186 tokens per second. With four at once, 59 each, well above the bar. We tested six situations, from short sessions to 256,000-token contexts, and in every one that got that far it held four agents at once without dropping below 40.

19 developers on a single server

Four agents at once may sound like few, but nobody asks the AI to write all the time.

We measured a real week of one developer using Claude Code. Of the whole week, the AI was writing for them for about 2.8 hours, 1.7%. The rest of the time they were the one reading, thinking and typing.

Imagine a bar with four stools at the counter. People come in, order, go to their table and come back a while later. With four stools you serve far more than four people. With 19 developers like this one, fewer than one time in a hundred will five of them overlap. Even then nobody goes without an answer: everything just runs a bit slower for that stretch.

And it pays off

The server costs 3.73 euros an hour, about 2,723 euros a month running day and night. That equals 15 subscriptions at 180 euros. With 19 people, the server is already cheaper than paying for their subscriptions.

With bigger teams the numbers improve, because the rush hours of many people spread out better. 100 developers fit on 4 servers, 39% cheaper than paying for their subscriptions. 250 fit on 7, 58% cheaper.

These are machine rental figures, without counting whoever sets it up and maintains it, and a commercial subscription includes a whole product. On the other hand, with the model on your machines you decide which model you use and when you change it. And the code and secrets the agent reads, which are a lot (this developer's agent reread 234 tokens of context for every one it wrote), stay in house. On machines you own, that advantage is complete.

Compressed, but just as smart

Compressing a model to 3 bits sounds like losing quality. We checked by giving our version and DeepSeek's official service the same three exams:

• Math: both got 96% right and missed exactly the same questions.
• Following instructions: ours 87%, the official one 85%.
• Writing code: ours 96%, the official one 98%.

On exams of about 150 questions, one or two points are statistical noise. In what we measured, the version that runs on two cards answers like the official service.

Every figure in this article comes from a data file, and the paper does not even build if one is missing.

What comes next

The model runs on two cards, serves a team of 19 people and answers like the original. One question is left: how much faster would it be on a data center machine with enough memory for the whole model.

The experiment that will tell us is already written and ready. It is called E9.

For engineers

The exact figures, for anyone who wants to check them. The full method, tables and limits are in the paper (PDF).

• Memory. In BF16 the weights would take about 1,526.7 GB (763.2B parameters, llmrun.dev estimate). DeepSeek's original checkpoint (FP4 and FP8) 475.2 GiB and the 3-bit EXL3 pack 309.3 GiB. Without offload it needs 102.1 GiB per GPU against 95.6 available, and with the gen30 plan, 72.4 GiB. The host holds 4,608 of 15,360 routed experts, both Engram tables in int4 and the embedding, read in place over PCIe through UVA. 2x RTX PRO 6000 Blackwell.
• Authorship. Weights by DeepSeek. EXL3 pack by coolbho3k. EXL3 and ExLlamaV3 by turboderp. vllm-exl3 by Victor Cruz. Per-expert plan, split launch and kernels by Diffbot, on top of vLLM. vLLM's own UVA offload, with the same bytes on the host, fails at init with KV block 64 and dies on the first request with 128.
• Reading and writing. Median TPOT of 7.6 ms after 4K tokens and 8.3 ms after 256K (pod 2). TTFT of 18.33 s for 128K and 45.07 s for 260,000. With a cached 256K prefix and 4 users: TTFT p95 of 1.66 s and median TPOT of 16.6 ms.
• Agents (E13, pod 3). Agent turns, tok/s per stream (median / p5): 1: 186/134, 2: 95/68, 3: 70/59, 4: 59/52, 6: 40/34, 8: 31/24. Capacity of 4 streams with a 10% margin. Five 128K contexts used at most 48% of the KV cache.
• Seats. q = 7.1% in an 8 h window. P(Binomial(19, q) > 4) ≤ 1%, and at 5%, 28 fit. Break-even at 15.1 subscriptions of 180 euros.
• Quality. Fixed subsets, no thinking, against DeepSeek's API. GSM8K 0.960 against 0.960 (150 items). IFEval +0.013 [-0.013, +0.047]. HumanEval -0.018 [-0.049, +0.012].
• Topology. A pod with its GPUs on different NUMA nodes was between 1.17 and 1.51 times slower. Check nvidia-smi topo -m before measuring.
• Limits.
  • Each experiment comes from a single boot.
  • There is no all-in-VRAM baseline yet (E9).
  • The agent turns are synthetic and the profile is one developer's.
  • The binomial model assumes independent people, so it is optimistic for a team on the same schedule.
  • Quality covers three subsets without thinking. The LiveBench figure is for the unquantized model at max effort.