Sage-A1.5 on one 24 GB card: 136 tokens/s, and a check mode that doubles the work on easy questions

Thirty-five billion parameters on disk, about three billion at work for each word. That gap is why Sage-A1.5, an open research agent from Shanghai AI Laboratory, fits on one gaming card. We ran the authors' own compressed file on one RTX 4090, a 24 GB gaming card, on our cloud (InferenceKey Cloud), with standard open-source software. Its agent machinery worked there: it handed out tasks, asked our code to search, and answered. It wrote about 136 tokens per second, a long reply in 11 seconds, and got our one three-part research question right in all three test runs. Its new Verification Mode, which makes it check its own work, gave the same right answers on easy questions, but took 2.2 times the time and twice the tokens. The model runs on a GPU worker you control, and the tools run in your own code. Leave the check mode off for lookups, and test it on your hard cases.

The image shows one question asked twice on the same card: in which city was hafnium discovered, and by whom. Both runs got it right, and the one with the mode on kept checking facts it had already read.

35 billion parameters, 3 billion awake for each word

Sage-A1.5 is big on disk and small in motion, and that is why one gaming card can serve it. It is a mixture-of-experts model. Picture a team of 256 specialists, plus one generalist who is always in the room. For every word it writes, the model consults 8 specialists. The rest wait. So of 35 billion parameters, only about 3 billion do work at each step, according to the project page. The speed follows the 3 billion, and the memory bill follows the 35.

The model is retrained from two others: the Qwen team's open 35B model and Agents-A1, the lab's own earlier agent, released in June. The paper, by Wenjie Lou and 98 co-authors, went up on 20 October. The full model and the compressed file, about 22 GB, are both Apache-2.0: free for commercial use, and open to download without a form. A small extra file lets the model read images, and two modes you switch on yourself turn it into an agent:

• Agentic Orchestration. You give the model one tool that hands out tasks. The model then splits a question into tasks and hands each one to a sub-agent, which searches and reads on its own and reports back. It is a lead researcher sending three assistants to the library at once.
• Verification Mode. One line added to the model's instructions tells it to check its own work before it answers. The authors say it needs nothing else installed.

The authors' scores come from the uncompressed model and were not reproduced here

On BrowseComp, which asks for hard-to-find facts on the web, the authors report 81.7 out of 100, against 91.2 for Kimi-K3, the best model on their table. In a separate on/off test over all 1,266 BrowseComp questions, Verification Mode raised the score from 72.5% to 82.9%. We cannot tell whether the 81.7 already includes that gain. Their other scores are in the appendix.

Our test: one RTX 4090, the authors' own file, our own questions

The limit first: a handful of easy, Wikipedia-backed questions on one card, one agent at a time. That checks the machinery (tool calls, handing out tasks, answers coming back), not the breadth of its research.

We served the authors' compressed file with llama.cpp, the open software that runs models on a single machine, on one RTX 4090 on our cloud. We used the authors' recommended settings. All requests went through our platform, the way an application would call it, and we checked speed against llama.cpp directly as well.

The questions were ours. Each needed a few facts from Wikipedia: who discovered rhenium and where, which city hafnium is named after, when technetium was found. The model searched and read live pages through two tools we wrote. For orchestration, we asked one question, run three times: how many years passed between the discoveries of three elements, so each sub-agent took one.

About 136 tokens per second, and one research question right in all three runs

On one RTX 4090 the model wrote about 136 tokens per second for one request, and the first token arrived in about a quarter of a second. Going through our platform added no measurable delay: llama.cpp alone gave the same speed. A 1,489-token reply took about 11 seconds.

It reads long prompts fast too. We hid a made-up tag, 7342-ORCHID, inside a 92,662-token prompt, about the length of a novel. The model read the whole thing in 31 seconds and found the tag.

The research question came back right every time we asked it. In each of three test runs, the model handed out three tasks at once, one per element. The three sub-agents searched Wikipedia and sent back their findings. The lead model then wrote the answer: hafnium in 1923, rhenium in 1925, technetium in 1937, 14 years from first to last. Every tool call came out in the exact format our code needed to run it. A simpler test with one search call came back clean 3 times out of 3, in under a second each.

Time depends on whether the sub-agents run side by side, and our platform's default makes them queue. After a deploy, it sends the model one request at a time, so the three sub-agents waited in line and the run took 72.0 seconds. One API call raised the limit to two, and runs then took 50 and 56 seconds. Status: the default is still one, and the call is in the appendix.

The vision part works as well. We showed it the authors' own benchmark chart and asked for a value. It read 80.0 off the right bar, and 50.0 as the best of the larger models, both correct.

The catch is how long it thinks. This model reasons at length before it answers. A two-sentence definition took 14.4 seconds, because it reasoned through 1,905 tokens first. Even 17 x 23 took 336 tokens of reasoning. At 136 tokens per second that is fine for a research task, and slow for a chat box.

Verification Mode doubled the work on easy questions and changed no answer

On our easy questions, Verification Mode bought nothing we could measure. All 12 answers were right with the mode off and with it on.

The extra work was mostly repetition. With the mode on, the reasoning filled up with "Let me verify" turns after the facts were already confirmed. The gap varied: in the other hafnium run, the mode on made one tool call fewer than the mode off. One run took 52.7 seconds because a Wikipedia search timed out and the model tried again.

That does not contradict the authors. Their gain came on BrowseComp, where facts are hard to find and a wrong first lead is common.

Which GPU: a real 24 GB card at full speed, about a fifth of the speed or less below it

The smallest card that ran Sage-A1.5 at full speed was a real 24 GB RTX 4090. With our example's settings, room for 32,768 tokens of conversation and two requests at once, it used 21.3 of the card's 23.5 GiB. The model card's own setup, 64K tokens with vision, peaked at 23.1 GiB. 128K tokens, text only, peaked at 23.4 GiB, leaving the card almost full. Every one of those fit, with no loss of speed.

Below 24 GB, the model only runs if part of it moves to the computer's main memory. Llama.cpp can park some specialists there and let the processor run them, which is much slower. We simulated smaller cards on the same 4090, by moving specialists out until the card's memory stayed under each limit.

• Simulated 16 GB: 14.4 GiB on the card, about 5.6 times slower, and the research question still came back right in 98 seconds.
• Simulated 12 GB: 10.8 GiB on the card, and the question came back right in 93 seconds.
• Simulated 8 GB: 3.0 GiB with every specialist in main memory. One of two research runs used up its 12-step limit after 270 seconds without an answer. The retry was right.
• Not tried: real 16 and 12 GB cards, the RTX 3090, 32 GB cards, the uncompressed model.

The simulated speeds depend on the computer's processor and memory, so treat them as a direction, not a promise. The smaller settings also need free main memory for the parked specialists, up to about 17 GB when all of them move there.

On InferenceKey: ready in 104 seconds, after one fix to the model card's command

A 35B agent model should not need a custom engine, and this one did not. From the deploy call to serving took 104.1 seconds, on the llama.cpp version our platform already ships: download the 21.7 GB file, load it, answer. Our platform runs a model on a worker, a GPU machine you own or rent and connect to it, and its default llama.cpp loaded Sage-A1.5 without changes. The first time on a fresh machine, llama.cpp also has to be built for the card, which took 10 and a half minutes. After that, a restart takes 4 to 7 seconds.

The one problem you will hit is in the model card itself. Its llama.cpp command uses a reasoning setting called deepthink, which llama.cpp does not have. The server quits on start with "Unknown reasoning format: deepthink". We checked a much newer llama.cpp release too, and it still does not exist there. The fix is to write deepseek instead. Status: fixed in our example, still in the authors' card.

A second gap is ours. The authors switch Verification Mode on with a setting our platform does not pass to the model yet. Our example adds the same switch line to the model's instructions instead. We checked that the model sees exactly the same prompt both ways. Status: open on our side, worked around in the example.

Should you try it? Yes on one 24 GB card: the agent machinery worked

• If you want an agent that searches, reads and reports on your own hardware, try it.
• If you care about accuracy on hard questions, test Verification Mode on your own hard cases.
• If you need quick chat replies, look elsewhere. It thinks through hundreds or thousands of tokens before short answers.
• If your card has 16 GB or less, expect a fraction of the speed, and only with plenty of main memory.

To ask Sage-A1.5 the same research question on your own card, with sub-agents and with Verification Mode off and on, run python main.py from the Sage-A1.5 SDK example. It needs an InferenceKey project with an SDK token and an API key, and a private NVIDIA worker with 24 GB or more. It deletes the deployment when it finishes.

llama-server -hf InternScience/Sage-A1.5-Q4_K_M-GGUF --hf-file Sage-A1.5-Q4_K_M.gguf \
  --no-mmproj --alias sage-a1.5 -ngl 99 -fa on -c 32768 -np 2 --kv-unified \
  --jinja --reasoning-format deepseek \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0 --presence-penalty 1.5

system = "You are a careful research assistant." + "\n\nVerification protocol: enabled"