Nanbeige4.3-33B: a dense 33B at 42 tokens/s on 24 GB, 3-bit for 16 GB
A dense 33B agent model on one 24 GB card: 42 tokens/s, clean tool calls, 64K context. A 3-bit file runs it on a real 16 GB card at 14 tokens/s.
Which card do you need for a 33-billion-parameter coding model that uses all of them on every word? We ran Nanbeige4.3-33B, an Apache-2.0 model, on a single 24 GB RTX 4090 rented on our cloud. With a 4-bit compressed file it wrote about 42 tokens per second. Its tool calls came back valid in all our tries. The 4090 also held a 64K-token prompt, roughly 50,000 words. A real 16 GB workstation card passed our checks with a 3-bit file, at about 14 tokens per second. Its authors put it ahead of Qwen3.8-27B on two of three agent coding tests. The whole run is packaged as an SDK example for your own card. One serious problem: when the model reasons before replying (the default), its final agent answer came back empty in 2 of 3 runs.
That picture is the decision for anyone with less than 24 GB.
A dense 33B pays for every parameter on every word
To write each word, the card has to read the whole model file from its own memory, and the file is 20.8 GB. That read sets the speed ceiling. A smaller file means less to read and more words per second. Anything kept outside the card's memory is read far more slowly, and speed collapses.
That is the price of being dense. The agent models we tested before it, such as Groot-19B and Clover-35B-A3B, work differently. Think of a big band where, for each bar, only a few soloists play and the rest sit out. A model with 35 billion parameters in that style may use only 3 billion per word, so it is quick. Nanbeige4.3-33B is dense: the whole band plays every bar. All 33 billion parameters work on every token.
The model went up on Hugging Face on 24 October. Its authors call it the "teacher" of the series: smaller models will learn from it later. They claim "state-of-the-art agentic performance among open models in the ~30B parameter class". Its model page lists prompts of up to 512K tokens. We ran up to 64K. It thinks before it answers by default, and anyone can download it without asking.
The authors compare it with three open models of similar size. These are their numbers, run the same way for every model, at maximum thinking. We did not reproduce them.
| Test (authors' numbers) | Nanbeige4.3-33B | Qwen3.8-27B |
|---|---|---|
| Terminal-Bench 2.1, agent work in a command line | 80.9 | 73.0 |
| DeepSWE, agentic software engineering | 65.5 | 42.5 |
| SWE-Bench Pro, fixing real issues in real code | 60.3 | 61.7 |
The third row is a loss, and the authors print it, along with two more losses on other tests. Claude, GPT and other closed models are not in the table.
Size is the hard part. Uncompressed, the model is 66 GB. The authors' own serving recipes spread the model over four GPUs and say nothing about running it on one. bartowski published compressed single-file versions for llama.cpp, a free program for running models yourself. Those files have 123,000 downloads already.
Our test: one 4-bit file, one 24 GB card, our own prompts
We used bartowski's recommended 4-bit file (20.8 GB). It ran on one RTX 4090, the whole model on the card, through our stock llama.cpp setup. Most tests went through the same OpenAI-style API a product would call.
Our checks were small. We asked for a one-word answer, the 9.11 or 9.8 question from the model's own page, and a coding task with its own tests. We also tried two tool calls and a secret word hidden in long prompts. A tool call is the model asking your code to run a function: here a stock price, and a weather lookup for two cities at once.
The limits: one request at a time, our own prompts rather than the authors' benchmarks, two real cards, nothing beyond 64K, and not the authors' own four-GPU server setup. We tested single tool calls and one agent turn, not an agent working through a codebase. We parsed the tool calls with llama.cpp, not with the qwen3_coder parser the model's page recommends for vLLM and SGLang. That route is untested.
About 42 tokens per second on one RTX 4090, with valid tool calls
On the 4090 the model wrote 42 to 44 tokens per second for one request. The first token came back in a quarter to half a second on short prompts. In reading terms, a long paragraph appears in about four seconds.
| What we asked | What came back | Time |
|---|---|---|
| "Capital of Australia, one word" | Canberra, after a short thought | about 1.2 s |
| "Which number is bigger, 9.11 or 9.8?" | 9.8, with the decimal place explained, in both runs | 3.6 to 4.6 s |
| A coding task with its own tests | Code that runs, tests pass | 24.6 s |
| A stock price tool | get_stock_price({"symbol":"AAPL"}), valid | about 2.2 s |
| "Is it warmer in Lisbon or in Oslo?" | Two weather calls at once, both valid | 2.8 to 3.0 s |
| The same, after the tool results came back | "Lisbon is warmer ... 19°C ... 4°C" (see below) | 3.1 s |
Each tool call came back in the standard OpenAI format, valid in both tries per call on the 4090.
For one developer driving an agent, that speed is comfortable.
64K of context fits in 24 GB, and writing slows as it fills
The 4090 held a 64K-token prompt and found a secret word hidden at token 61,428. To fit, the memory it keeps of the prompt had to be squeezed to a quarter of its size, and writing fell to 16 tokens per second.
Besides the model, the card keeps notes on everything the model has read so far, and they grow with every token. Full-size notes stop at 16K. Halving them reaches 32K, quartering them 64K, with under 1 GB of the card left.
| Context | Notes compressed to | Peak memory (MiB) | Secret word found at | Answer took | Writing speed at that depth |
|---|---|---|---|---|---|
| 16K | no (16-bit) | 23,186 | 14,909 tokens | 7.9 s | 37 tokens/s |
| 32K | 8-bit | 23,434 | 30,734 tokens | 18.8 s | 25 tokens/s |
| 64K | 4-bit | 23,802 | 61,428 tokens | 49.1 s | 16 tokens/s |
All three found the word. At 64K, writing runs at under half the short-prompt speed, because each new word also reads 61,000 tokens of notes.
Which GPU: a real 16 GB card works with the 3-bit file, at about 14 tokens per second
The smallest card that passed is a real one: an RTX 2000 Ada, a 16 GB, 70-watt workstation card, running bartowski's 3-bit file. It passed the same checks at about a third of the 4090's speed, so a 400-token answer takes about 30 seconds instead of 10.
| Card or memory budget | File and setup | Peak memory (MiB) | Speed | Checks | Real card? |
|---|---|---|---|---|---|
| RTX 2000 Ada 16 GB | 3-bit, 8K, 8-bit notes | 14,392 | 13.6 to 14.4 tokens/s | all right | yes |
| 16 GB budget | 3-bit, all on the card | 14,730 | 56.5 tokens/s | all right | no, run on the 4090 |
| 16 GB budget | 4-bit, a third of it off the card | 14,413 | 1.6 tokens/s | right, unusable | no, run on the 4090 |
| 12 GB budget | 2-bit, 8K | 10,670 | 74.9 tokens/s | one wrong explanation | no, run on the 4090 |
Read the "no" rows as memory measurements, not speed promises. Each ran on the full 24 GB 4090, with no memory limit, and peaked under the budget. There the 3-bit file ran at 56.5 tokens per second, because there is less to read. The real 16 GB card ran it at 14, because it reads its memory far more slowly.
The 2-bit file, which peaked under 12 GB on the 4090, got the numbers right but said the 8 in 9.8 sits "in the hundredths place". It is in the tenths. One slip in four checks is enough to keep 2-bit away from code.
The problem you will hit: answers land in the thinking channel and come back empty
With thinking on, the default, some answers arrive with an empty reply. The whole answer sits in the model's thinking field instead. Status: open.
It happened on every 400-word essay we asked for, three of three. It also hit the agent's short final answer after the weather results in two of three runs, once on each card. Short one-off answers and the tool calls themselves were never affected.
The model's prompt wrapper opens a thinking block, the model sometimes answers without closing it, and llama.cpp files the whole answer as thinking. When we sent the request straight to llama.cpp with thinking switched off, the essay arrived in the normal reply, all 3,151 characters of it.
Through InferenceKey, that per-request switch does not get through yet: our endpoint drops it. That gap is on our side and still open. Until it is fixed, one server setting in the launch command (in the appendix) should turn thinking off for every request. We have not run it, so its effect on the agent's final answer is untested. The authors measured their scores at maximum thinking and recommend keeping it for agents. The SDK example also flags an empty reply and shows the text from the thinking field.
One smaller thing: if you give the model no standing instructions, its template adds its own, in Chinese. They say the model is Nanbeige, made by BOSS Zhipin. They did no harm in our tests, but send your own.
On InferenceKey: ready in five minutes, almost all of it compiling
The card is yours by the hour, and our standard llama.cpp setup ran this model without changes to the platform. We set one version number and wrote one launch command.
On InferenceKey, a worker is the small program on your GPU machine that takes orders from the platform. Ours builds llama.cpp for the exact GPU the first time it is asked. On the 4090 the deployment was ready 303.7 seconds after we asked for it. Almost all of that was that build, about five minutes. The 20.8 GB file had been fetched in parallel in 15 seconds, and loading it took about 4. Once built, relaunching llama.cpp with the file cached was ready in 3 to 5 seconds.
On the RTX 2000 Ada machine the same build took 19 minutes 50 seconds, because that host's processors were throttled. It was ready after about 20 minutes. Plan for a slow first start on a small host: a ready-built llama.cpp for NVIDIA is still missing on our side.
This run used our own local copy of the InferenceKey platform, with the same code as the hosted service. The SDK example sends the model to one machine you name, so the weights and the inference stay on your hardware.
Should you use it? Worth a try on 24 GB, if you can live with empty agent answers for now
- You have a 24 GB card and want an open agent model for coding: worth a try. Check the authors' scores against your own tasks.
- You need long prompts: up to 64K on one 24 GB card, slower as it fills.
- You have a 16 GB card: use the 3-bit file entirely on the card, never the 4-bit file with a third off it. Speed depends on your card's memory.
- You run it as an agent today: expect empty final answers with thinking on. Turning thinking off on the server should fix it.
To run the same tests on your own 24 GB NVIDIA worker, run python main.py from the Nanbeige4.3-33B SDK example. It needs a project, an SDK token and an API key, and deletes the deployment when done.
For engineers
Credits
- The Nanbeige team (BOSS Zhipin): the model, the base model, the model cards, the benchmark table and the chat template with Qwen3-Coder style XML tool calls. Released 2026-10-24 under Apache-2.0.
- bartowski: the imatrix GGUF quants (llama.cpp b11845) we used: Q4_K_M, IQ3_XXS and IQ2_XXS.
- axteam: an NVFP4A16 build for vLLM (not tested by us).
- The llama.cpp maintainers: the server, the
--jinjatemplate support that parsed the XML tool calls, and release v0.9.0. - AI Search: where we spotted the model (2026-11-01 news video, chapter at 43:03), which flagged the 66.3 GB size and "quite some high-end hardware".
- Our part: the deployment through the platform, the tests and prompts, the memory ladder on two real cards plus three 4090 runs sized to fit 16 and 12 GB budgets, diagnosing the empty-reply problem, the SDK example and the demo video.
Model and files
Nanbeige/Nanbeige4.3-33B: shadf919e1e, last modified 2026-10-28, 33,118,977,024 BF16 parameters, about 66.3 GB in 14 shards, 19,178 downloads and 1,052 likes on 2026-11-06. Base modelNanbeige4.3-33B-Base: shae061adde.- Architecture: dense
LlamaForCausalLM, 52 layers, 56 attention heads, 8 KV heads, head_dim 128, hidden 7168, FFN 22016, vocab 166,912,max_position_embeddings524,288,rope_theta1.2e8. bartowski/Nanbeige_Nanbeige4.3-33B-GGUF: sha9beb6842, created 2026-10-25, 123,000 downloads. Q4_K_M 20.79 GB, Q4_K_S 19.74 GB, IQ4_XS 18.64 GB, IQ3_XXS 14.49 GB, IQ2_XXS 10.07 GB, Q5_K_M 25.07 GB, Q8_0 35.20 GB, bf16 66.24 GB. Q4_K_M used: 20,788,825,216 B, sha2562cef7985...matching the Hugging Face LFS hash.axteam/Nanbeige4.3-33B-NVFP4A16: W4A16 FP4 for vLLM, about 22.1 GB, tested by its author on vLLM 0.32.0 on B200 only, its own lm-eval average 98.9% of BF16. Not tested by us.- KV cache at f16: 52 x 8 x 128 x 2 x 2 B = 208 KiB per token (8K is 1.6 GiB, 32K is 6.5 GiB).
q8_0about half,q4_0about a quarter. - Rough sizing from the source video's notes (an estimate, not measured): about 80 GB to serve BF16, about 40 GB at 8-bit, about 20 GB at 4-bit.
Authors' claims (not reproduced)
Nanbeige4.3-33B / Qwen3.8-27B / Muse-Glimmer-30B / Gemma4-31B, highest thinking effort, same harness, preserve_thinking=true:
| Benchmark | Nanbeige4.3-33B | Qwen3.8-27B | Muse-Glimmer-30B | Gemma4-31B |
|---|---|---|---|---|
| Terminal-Bench 2.1 (Terminus 2) | 80.9 | 73.0 | 61.8 | 47.2 |
| Terminal-Bench 4.0 | 23.8 | 6.3 | 1.6 | 0.0 |
| DeepSWE | 65.5 | 42.5 | 4.4 | 2.7 |
| SWE-Bench Pro | 60.3 | 61.7 | 51.2 | 36.9 |
| JobBench | 44.8 | 34.5 | 16.6 | 9.0 |
| GDPval | 84.8 | 86.4 | 75.8 | 71.4 |
| GPQA Diamond | 89.5 | 89.2 | 83.5 | 91.3 |
| AIME 2026 | 95.2 | 94.5 | 94.7 | 89.2 |
| AA-LCR | 84.0 | 82.0 | 83.3 | 72.0 |
Also Nanbeige vs Qwen: SWE-Atlas-QnA 50.8 vs 41.9, NL2Repo 53.0 vs 48.3, SkillsBench 61.1 vs 56.4, Automation Bench 54.2 vs 38.3, Toolathlon-Verified 70.4 vs 68.5, Tau3-Bench 48.8 vs 48.5. Base model vs Gemma4-31B-Base: MMLU-Pro 66.9 vs 59.3, GPQA 62.0 vs 41.5, SuperGPQA 45.6 vs 33.8, GSM8K 92.9 vs 88.9, Math 57.3 vs 43.6, MBPP 74.6 vs 72.5, MMLU 85.5 vs 84.2, CMMLU 86.8 vs 76.9.
One inconsistency in the card: JobBench for Muse-Glimmer-30B is 16.6 in the table and the occupation heatmap, 18.3 in the bar figure. The authors compare only with open models of about 30B; third-party Terminal-Bench 2.1 boards use other harnesses, so we draw no frontier comparison.
Card usage notes: thinking on by default (enable_thinking); preserve_thinking recommended for multi-turn tool use; tool format xml (default) or json; sampling 1.0 / 0.95 (up to 65,536 new tokens) for agentic and tool use, 0.6 / 0.95 (up to 262,144) for reasoning and chat. SGLang and vLLM recipes: tensor parallel 4, --reasoning-parser qwen3 --tool-call-parser qwen3_coder (vLLM also --enable-auto-tool-choice, --gpu-memory-utilization 0.8).
Setup
- GPUs on InferenceKey Cloud: RTX 4090 24 GB (450 W, compute capability 8.9) for the main run; RTX 2000 Ada 16 GB (70 W) for the real 16 GB rung.
- Backend: the built-in
llamacppbackend,INFERENCEKEY_LLAMACPP_TAG=v0.9.0(the GGUF was made with b11845, the worker default is older). Llama architecture, so any recent tag loads it; v0.9.0 is newer than the quantizer's build. - Why llama.cpp and not vLLM: the card's recipes assume four GPUs at BF16, and the GGUF route is what fits 24 GB. The
qwen3_coderparser was not tested; llama.cpp--jinjaparsed the XML tool calls into OpenAItool_calls. - Routing: the short tests, the SDK example and the RTX 2000 Ada checks went through the OpenAI-style endpoint; the long-context and budget rungs went straight to llama-server on the pod.
- A local InferenceKey manager (same code as the hosted service) with two workers. Production was never touched.
- Sampling: the card's values, 0.6 / 0.95 for chat and 1.0 / 0.95 for tools. Ladder rungs with seed 42.
Step by step with the SDK example
- Clone inferencekey-sdk and open
examples/nanbeige43-33b-agent-llamacpp-private-nvidia(commit10209c3596fe1fd28dd63a30442eb543e6a456cc). cp .env.example .envand fill inINFERENCEKEY_SDK_TOKEN,INFERENCEKEY_API_KEY,INFERENCEKEY_PROJECTandIK_WORKER_ID(a private NVIDIA worker with 24 GB). Optional:NANBEIGE_CONTEXT(default 8192),NANBEIGE_KV_TYPE(defaultf16),NANBEIGE_GGUF_FILE,IK_KEEP_ALIVE=1.pip install -r requirements.txtplus the local SDK build with maturin (seeexamples/CONTRIBUTING.mdin the SDK repo).- If the worker image has
nvccoffPATH(the PyTorch CUDA 13 images), set this in the worker's environment and restart it:
CUDACXX=/usr/local/cuda/bin/nvcc
PATH=/usr/local/cuda/bin:$PATH
CMAKE_BUILD_PARALLEL_LEVEL=<number of CPU cores>
python main.py. Itensure()s a fixedllamacppworkload (task_typetext2text,ExecutionPolicy.FIXED) pinned to your worker, waits up to 3,600 s, asks the card's question withgenerate_text(), makes theget_stock_pricecall, runs the two-call weather agent turn, writesoutputs/answers.jsonand callsdelete()unlessIK_KEEP_ALIVE=1.
The launch command the backend runs:
INFERENCEKEY_LLAMACPP_TAG=v0.9.0 llama-server \
-hf bartowski/Nanbeige_Nanbeige4.3-33B-GGUF \
--hf-file Nanbeige_Nanbeige4.3-33B-Q4_K_M.gguf \
--alias nanbeige4.3-33b -ngl 99 -fa on \
-c 8192 -ctk f16 -ctv f16 \
--jinja --reasoning-format deepseek
For 32K use -c 32768 -ctk q8_0 -ctv q8_0; for 64K -c 65536 -ctk q4_0 -ctv q4_0. For a 16 GB card set NANBEIGE_GGUF_FILE=Nanbeige_Nanbeige4.3-33B-IQ3_XXS.gguf and NANBEIGE_KV_TYPE=q8_0. Expected to avoid empty replies: append --chat-template-kwargs '{"enable_thinking":false}' (thinking off for every request). Untested: we never launched this flag, and the agent final turn was never re-run with thinking off; agent quality with thinking off not measured.
Timings
| Step | RTX 4090 | RTX 2000 Ada |
|---|---|---|
ensure() to ready, fresh worker | 303.7 s | 1,203.6 s |
| llama.cpp v0.9.0 CUDA build | about 5 min (32 parallel jobs) | 19 min 50 s (CPU-throttled host, 48 vCPU visible, load about 7) |
| GGUF download (pre-fetched in parallel, not on the ready path) | 15 s (Q4_K_M) | 20 s (IQ3_XXS) |
| Model load | about 4 s | |
| llama-server relaunch on the pod (binary built, file cached; not a platform deployment) | 2.7 to 4.5 s |
Speed and outputs (RTX 4090, Q4_K_M, 8K, f16 cache, streaming through the endpoint)
| Test | Completion tokens | Seconds | First token s | Decode tokens/s | Result |
|---|---|---|---|---|---|
| Capital of Australia x2 | 40 / 40 | 1.19 / 1.27 | 0.28 / 0.37 | 43.7 / 44.4 | Canberra |
| 9.11 or 9.8 | 183 | 4.55 | 0.28 | 42.9 | right, decimal place explained |
merge_intervals + 3 asserts | 1,026 | 24.62 | 0.43 | 42.4 | asserts pass |
| 400-word essay x3 | 641 / 552 / 591 | 15.4 / 13.2 / 14.2 | 0.26 to 0.37 | 42.6 | content empty, all in reasoning_content |
get_stock_price | 72 | 2.17 | 0.50 | 43.2 | {"symbol":"AAPL"}, finish tool_calls |
| Lisbon or Oslo | 101 | 2.82 | 0.47 | 43.0 | two parallel get_weather calls |
| Needle, 6,142-token prompt | 81 | 4.65 | 2.65 | 40.4 | found |
chat_template_kwargs enable_thinking=false via the endpoint | 40 | 2.16 | 1.24 | 43.5 | dropped by the gateway, still thinks |
SDK example run on the 4090: card question 3.6 s; stock call 2.27 s; agent step 1 (two calls) 2.97 s; step 2 3.12 s at 44.3 tokens/s, right answer. Power peak 448.6 W.
Long context on the 4090: prefill 2,514 / 2,038 / 1,544 tokens/s at 14,909 / 30,734 / 61,428 tokens; short answers about 41 to 42 tokens/s in all three configurations.
RTX 2000 Ada through the endpoint: Canberra 2.9 to 3.1 s, card question 8.65 s (14.0 s in the example run) right, stock call 4.64 to 5.26 s, two parallel weather calls 8.09 to 8.23 s, merge_intervals 1,398 tokens in 102.7 s with asserts passing, agent final answer empty (problem 1). Peak power 69.8 W, at its 70 W limit.
VRAM ladder
Peak VRAM from nvidia-smi. Emulated rungs ran on the full 24 GB 4090 with no memory limit set; the configuration peaked under the budget. Checks: Canberra, 9.11 vs 9.8, the get_stock_price call, and merge_intervals with its own asserts.
| GPU or budget | Config | Peak VRAM | Speed | Output OK | Real or emulated |
|---|---|---|---|---|---|
| RTX 4090 24 GB | Q4_K_M, 8K, f16 (the example, through the endpoint) | 21,907 MiB | 42 to 44 tokens/s; Canberra 1.2 s | yes | real |
| RTX 4090 24 GB | Q4_K_M, 16K, f16 | 23,186 MiB | 42.1; needle 14,909 tokens 7.9 s, 37.3 at depth | yes | real |
| RTX 4090 24 GB | Q4_K_M, 32K, q8_0 | 23,434 MiB | 41.3; needle 30,734 tokens 18.8 s, 24.6 at depth | yes | real |
| RTX 4090 24 GB | Q4_K_M, 64K, q4_0 | 23,802 MiB | 41.2; needle 61,428 tokens 49.1 s, 16.4 at depth | yes | real |
| 16 GB budget | IQ3_XXS, 8K, q8_0, all on GPU | 14,730 MiB | 56.5; Canberra 0.82 s, card 2.65 s, code 2,537 tokens 46.0 s | yes | emulated on the 4090 |
| 16 GB budget | Q4_K_M, -ngl 36 (16 of 52 layers on CPU), 8K, q8_0 | 14,413 MiB | 1.6; Canberra 24.6 s, card 96.1 s, code 815 s | yes, unusable | emulated on the 4090 |
| 12 GB budget | IQ2_XXS, 8K, q8_0 | 10,670 MiB | 74.9; code 1,395 tokens 18.7 s | answers right, one explanation wrong | emulated on the 4090 |
| RTX 2000 Ada 16 GB | IQ3_XXS, 8K, q8_0, through the endpoint | 14,392 MiB | 13.6 to 14.4 | yes; agent final answer empty | real |
Smallest that passed: the real RTX 2000 Ada 16 GB with IQ3_XXS, about a third of the 4090's speed. The example README lists 21,508 MiB for its configuration (a reference rung); 21,907 MiB is the peak during the main tests, including the recorded take.
Not tried: a multi-step coding agent on a real repository (only single tool calls and one agent turn); the hosted endpoint (a local manager only); concurrent requests; 128K or more on 24 GB; another 24 GB card (A5000, RTX 3090); a real 12 GB card; an 8 GB budget; NVFP4A16 on vLLM; vLLM or SGLang with the qwen3_coder parser; quality of the q4_0 cache beyond the needle test; the authors' benchmarks.
Remaining problems
- Answers in the thinking channel. With thinking on, the template ends the prompt with
<|im_start|>assistant\n<think>\n. On some answers the model writes the answer directly and never emits</think>;--reasoning-format deepseekthen returns it all asreasoning_contentwithcontent= "". Seen on the essay (3 of 3) and the agent final turn (2 of 3 runs: the recorded take on the 4090 and the RTX 2000 Ada run; the first example run had content). Withenable_thinking=falsethe template inserts<think>\n\n</think>\n\nand the essay came back incontent(3,151 chars). Proposed workaround through InferenceKey, not run:--chat-template-kwargs '{"enable_thinking":false}'.main.pyflags empty replies withanswer_text(). Not tested: other reasoning formats,--reasoning-budget. Open. - The endpoint drops per-request
chat_template_kwargs, so thinking cannot be switched off per request through InferenceKey. Known gap on our side, open. - Partial offload is not a way to fit 16 GB with a dense model: 1.6 tokens/s with 16 of 52 layers on the CPU. Use the smaller quant fully on the GPU.
- Slow first start: the worker builds llama.cpp from source on NVIDIA (5 min on the 4090 host, 19 min 50 s on a throttled host). No prebuilt CUDA binary yet. Open.
- The worker build needs
nvcconPATHon some images; workaround in the step-by-step above. Open in the worker. - Default Chinese system prompt: with no system message the template injects "你是南北阁，一款由BOSS直聘自主研发并训练的专业大语言模型。" ("You are Nanbeige, a professional LLM developed and trained by BOSS Zhipin"); with tools and no system message, a different Chinese tool-expert prompt. Send your own.
- Model card: the JobBench inconsistency above, and no single-GPU or GGUF guidance.
- 2-bit quality: IQ2_XXS explained 9.8 as "8 in the hundredths place".
Demo video
The 154-second narrated demo: the authors' benchmarks (credited to the Nanbeige team), the deployment in the panel on the RTX 4090, the card question and the agent turn (with the empty reply narrated as the catch), the long-context results and the memory ladder.
Sources
- Nanbeige/Nanbeige4.3-33B (model card, config, chat template).
- Nanbeige/Nanbeige4.3-33B-Base.
- bartowski/Nanbeige_Nanbeige4.3-33B-GGUF.
- axteam/Nanbeige4.3-33B-NVFP4A16.
- AI Search news video, 2026-11-01 (where we spotted it).
- SDK example, commit
10209c3596fe1fd28dd63a30442eb543e6a456cc.