AREX-2: a 27B agent model that serves like Qwen3.8-27B

BAAI released AREX-2 on 29 September: a 27B model trained to work on a problem for a long time, in rounds. Propose a solution, measure it, reflect, revise, repeat. Apache 2.0, not gated, 54.7 GB of BF16 weights.

The part that caught my eye is not the benchmark table. It is the first line of its config.json: Qwen3_5ForConditionalGeneration, the same class as Qwen3.8-27B. On 4 October, the same day AREX-2 showed up in our feed, we had been serving base Qwen3.8-27B with sglang on three GPUs: two that served and one that ran out of memory in warmup. Same architecture means most of that work transfers. Most, not all, and the gaps are where this post spends its time.

To be clear up front: we have not run AREX-2. Every tokens/s number we measured is from base Qwen3.8-27B, used as an architecture proxy. The only AREX-2 throughput figures below are third-party, and labeled as such. That is why this sits in news and not research.

What AREX-2 is

From the model card and the paper (arXiv 2609.38288, "AREX-2: Advancing Self-Improving Agents through Long-Horizon Reflective Tasks"):

• A dense, 27B, Qwen3.8-compatible multimodal model, fine-tuned from Qwen/Qwen3.8-27B. The card tags it image-text-to-text.
• 262,144 tokens of native context.
• Trained on synthesized long-horizon improvement trajectories from ML engineering and algorithmic programming tasks, where the feedback can be checked. The paper says the behavior transfers to deep research and keeps improving as you give it more rounds.
• Apache 2.0, with a note to also follow the terms of the Qwen base model (which is Apache 2.0 as well). Commercial use is on the table.

I spotted it in the AI Search news roundup of 4 October (around 38:20), which calls it "a pretty tiny 27 billion parameter model based off of Qwen" that "learns to repeatedly improve its own answer". Tiny is relative: it is still 55 GB in BF16.

Same chassis as Qwen3.8-27B, checked

I diffed AREX-2's config.json against Qwen3.8-27B's. The only differences are the transformers_version (5.2.0 vs 5.8.0.dev0) and two redundant top-level keys. Everything that decides memory and kernels is identical:

The quants we ran of the base model (RedHatAI INT4, RadixArk NVFP4, Qwen's own FP8) use the same class and the same 64-layer, 16 full + 48 linear layout. The chat_template.jinja is byte-identical too, tool call handling included.

Two differences that matter later:

1. No MTP head. The config says mtp_num_hidden_layers: 1, but the safetensors index has zero mtp tensors. Base Qwen3.8-27B ships 15 of them and about 425M more parameters in total. Built-in MTP speculative decoding has nothing to load.
2. The tokenizer config differs. Not the template, but tokenizer_config.json, and a community repack notes that AREX-2's pre-tokenizer and decoder settings differ from Qwen3.8. Probably harmless for serving, but it is exactly the kind of thing that turns "same architecture" into a weird bug at 2 a.m.

Pick a weight file for your card

Here is what exists on Hugging Face today, with the sizes measured from the repos. These are weights only: KV cache and the linear attention state come on top.

How much the rest costs: on our base-model INT4 run (RTX A6000, 1M context through YARN), sglang used about 46 GB in total: 17.8 GB of weights, 11.5 GB of Mamba-style state for the linear attention layers and 12.8 GB of KV cache. Same layer layout, so expect the same overhead shape on AREX-2 at the same context. Weights are the cheap part of a long context.

So, card by card, as estimates and not measurements:

• One 80 GB card (A100, H100) in BF16. 54.7 GB of weights leaves room for a useful KV budget. Plausible, but neither we nor BAAI have documented it. Our internal model catalog estimates about 66 GB with overhead, which is an estimate too. For scale, the A100 SXM 80 GB was $1.79/h in our catalog on 18 September, when we recorded the demo for our last post. Check the current price before you budget.
• One 48 GB card in INT4. This is the configuration closest to what we actually ran on the base model, long context included.
• One 24 GB card in INT4. 18.6 to 20.2 GB of weights fit, but the only published INT4 deployment used two RTX 3090s (tensor parallel 2, 16,784 MiB per GPU after short probes) with an FP8 KV cache and 28 GiB of KV offloaded to CPU. A single 24 GB card would likely need a much shorter context; we have not tried it, and nobody has published that run.

Which quant for which GPU generation

This is where the base-model runs earn their keep. All of them: sglang 0.5.21, base Qwen3.8-27B, single GPU, on an A6000 and RTX Blackwell cards, not an A100 or H100. Anything below about the A100 is an extrapolation from the A6000, since both are Ampere.

What I take from it:

• FP8 on Ampere is a trap. The A6000 has no native FP8, so sglang runs it weight-only through Marlin and you get 24 tok/s. The FP8 AREX-2 card says the same thing in its own words: full speed needs Hopper, Ada or newer. On an A100, same family, I would not bother with FP8.
• INT4 is the sane floor on Ampere. 41 vs 24 tok/s on the same card, with the long context intact.
• NVFP4 plus speculative decoding is the Blackwell move, and it needs a CUDA 13 pod: with our default CUDA 12.4 image sglang dies with SM 12.x requires CUDA>=12.9. The catch for AREX-2: nobody has published an NVFP4 quant of it yet, as of today.

The command we used, and what changes for AREX-2

This is the fast base-model config, verbatim (RTX PRO 5000, CUDA 13 pod):

sglang serve --trust-remote-code \
  --model-path RadixArk/Qwen3.8-27B-NVFP4 \
  --tp-size 1 --context-length 262144 \
  --kv-cache-dtype fp8_e4m3 --attention-backend flashinfer \
  --mamba-ssm-dtype bfloat16 --mamba-radix-cache-strategy extra_buffer_lazy \
  --chunked-prefill-size 4096 --max-prefill-tokens 8192 \
  --max-running-requests 8 --mem-fraction-static 0.90 \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path z-lab/Qwen3.8-27B-DFlash2 \
  --speculative-num-draft-tokens 8 \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
  --host 0.0.0.0 --port 30000

On InferenceKey we also raised the worker's patience, because the first warmup is long and the supervisor would otherwise kill it: INFERENCEKEY_HEALTH_POLL_SECS=30, INFERENCEKEY_HEALTH_MISS_THRESHOLD=40, INFERENCEKEY_SGLANG_READY_TIMEOUT_SECS=1800. The pod image was the RunPod CUDA 13 template runpod/pytorch:1.0.7-cu1300-torch291-ubuntu2404-cluster.

For AREX-2 you cannot paste that and walk away:

• There is no AREX-2 NVFP4, so --model-path becomes one of the four files above.
• The drafter is for the wrong model. z-lab/Qwen3.8-27B-DFlash2 lists base Qwen3.8-27B as its base model. A fine-tune shifts the token distribution, so its acceptance rate on AREX-2 is unknown. The numsu INT4 card reports decode with an external DFlash2 drafter on its 2x3090 setup (225.6 tok/s median at 1K prompt and 512 output tokens, 42.6 tok/s at 128K prompt), but no acceptance length, so those numbers do not answer the question.
• Built-in MTP is out (no MTP tensors). There is a community attempt, numsu/AREX-2-27B-INT8-W8A16-MTP: an MTP module started from Qwen3.8's weights and trained on 1,228 AREX-2-generated sequences on one RTX 3090. It reports offline acceptance of 87.4% at position 1, 79.7% at 2 and 74.8% at 3, about 2.88 accepted draft tokens per step. Its own card says that is offline, not an end-to-end speed test, and it needs a vLLM build with draft vocabulary support (stock vLLM can fail on draft_lm_head).
• The 1M context trick and speculative decoding do not mix. 1M is a YARN override on top of the native 262,144 (in short, --json-model-override-args setting YARN with a factor of 4.0, plus --context-length 1000000; that is a summary of our recipe, not a copy-paste flag). On the base model, DFlash2 refused to work with that override, so it was 1M without speculation or 256K with it. We did not retest this on AREX-2, but nothing in the config suggests it changes.
• The parsers are inferred. --reasoning-parser qwen3 and --tool-call-parser qwen3_coder should apply, because the chat template is byte-identical. Should. The tokenizer config differs and no one has published an sglang run of AREX-2.
• One detail in our flags: AREX-2's config sets mamba_ssm_dtype to float32; our base command overrode it to bfloat16. Fine for Qwen3.8 in our runs, untested on a model trained for long reflective trajectories.

If I had an 80 GB card this afternoon, my first attempt would be the boring one: BAAI/AREX-2 in BF16, the 256K native context, no speculative flags, the same parsers, and a long agent task to see whether tool calls parse. Then add one thing at a time. Same as a soundcheck: kick drum first, then the rest of the kit.

The benchmarks, with their asterisks

The card's numbers are strong for a 27B:

Now the asterisks. These are self-reported and the card says they "follow the protocols reported in the AREX-2 paper". I found no independent reproduction. The evaluation repo shows what that protocol costs: the research profile allows up to 10 outer rounds and up to 1,500 model calls per case, 240,000 tokens of context with a refresh at 128,000, a 24-hour timeout per case, and an external judge model. The HLE number is the text-only subset, not full HLE.

That is not cheating, it is the point of the model: it gets better the longer you let it iterate. But it means a fair comparison is "AREX-2 with a big round budget" against "a frontier model with whatever budget it got", and the practical question becomes how many GPU hours one case eats. Paying per GPU hour instead of per token only wins if you keep the GPU busy, for example by running several cases in parallel; the evaluation repo's research profile runs one case at a time with a 24-hour timeout, which is close to the opposite.

What this does and does not tell you

Limits, plainly:

• We have not served AREX-2. All throughput and VRAM numbers we measured are from base Qwen3.8-27B on an A6000 and an RTX PRO 5000 (the RTX 5090 ran out of memory in warmup and gave us no numbers).
• The model card's only inference recipe is Transformers (AutoModelForMultimodalLM, BF16, device_map="auto"). No vLLM or sglang command from BAAI. The GitHub repo holds evaluation runners that talk to any OpenAI-compatible endpoint, not serving code.
• The community quants are lightly tested. numsu's INT4 was validated on 2x RTX 3090 with a pinned vLLM 0.29.0, up to 128K prompt tokens, with no quality or multimodal eval (only a weight reconstruction error, 0.12). cyankiwi's AWQ and the FP8 repack publish no measurements of their own.
• Speculative decoding on AREX-2 is unmeasured end to end, by anyone, as far as I can find.
• The benchmarks are the authors' own, under their own protocol.

What it does tell you: if you already serve Qwen3.8-27B, AREX-2 should fit the same memory budget and the same kernels, and probably the same flags, until a real run says otherwise. The weight file choice follows the GPU you have: BF16 on 80 GB, INT4 on 48 GB, INT4 on 24 GB only with a short context and still unproven, FP8 only on Ada, Hopper or newer, and save NVFP4 plus speculation for when someone ships an AREX-2 NVFP4 and a drafter trained on it. A real AREX-2 run, with its own numbers, is next on the setlist.

Sources: model card, paper, evaluation repo, config.json.