Flint, a from-scratch coding model, runs its agent loop on one RTX 4090

Recursive's Flint on our own GPUs: 210 tokens/s on an RTX 5090 at 64K context, 167 on an RTX 4090 at 16K, one planted bug fixed in 7 of 7 runs.

Published · By InferenceKey · 9 min read + technical appendix

Recursive's server kit for the full-precision Flint asks for a datacenter card with at least 141 GB of memory. We ran Flint's official compressed file and its coding-agent loop on a 24 GB RTX 4090, a card built for gaming. Flint is the new coding-agent model from Recursive, and that file weighs 22.9 GB. We loaded it on an RTX 5090 with 32 GB and on the 4090, both on our own cloud. On the 5090, Flint fixed a planted bug in a small code project in five steps and about six seconds. The 4090 did the same job in about the same time, writing about 20% slower, but only with a shorter context (what it keeps in mind). At double that context, it ran out of room before starting. Recursive trained Flint from scratch, and the license is Apache 2.0. If you serve coding agents on your own GPUs, that is a model you can run and change on a card you rent by the hour.

The same coding-agent task run on Flint on two cards. Left, RTX 5090 with 64K context: 23.3 GiB of memory used, 5 turns, 654 tokens, 5.9 s, about 210 tokens per second, 3 tests passed. Right, RTX 4090 with 16K context: 21.8 GiB used, 5 turns, 560 tokens, 5.7 s, about 167 tokens per second, 3 tests passed. A red box under the 4090 says 32K context ran out of memory at load

Same task, same file, same answer on both cards. The red box marks where the smaller card stops: 32K of context does not fit.

RTX 5090, 32 GBRTX 4090, 24 GB
Text it can keep in mind (context)64K tokens16K tokens (32K did not fit)
Card memory usedabout 23.3 of 32 GiBabout 21.8 of 24 GiB
Writing speedabout 210 tokens/sabout 167 tokens/s
Planted bug fixedyes, 5 turns, 5.9 syes, 5 turns, 5.7 s

Built from scratch, with 6 of 256 specialists awake for each word

Flint is a big model that works at the cost of a small one. It holds 39.8 billion parameters (the numbers that make up a model), but each token (word piece) it writes uses only about 2.5 billion of them. The trick is called a mixture of experts. Picture a session band with 256 players on the roster. For each word piece, the bandleader calls the 6 who suit it best, plus one house drummer who plays every song. The other 250 sit in the green room until their turn comes. All of them still have to be in the building, which is why the whole model must fit in memory even though most of it rests.

Unlike the coding models we tested recently, Flint is not built on another company's model. Recursive trained Flint itself, from the very first step, with knowledge up to June 2026. It has its own layout, so llama.cpp, the engine we ran it on, needed new code to load it.

Recursive reports 74.2% on SWE-bench Verified. That test hands an agent real bug reports from open source projects and counts a fix only when the project's own tests pass. It also reports 54.9% on a second test of command-line tasks. These are Recursive's numbers and we did not reproduce them. They are not scores for the compressed file we ran, and we did not measure how much quality the compression costs. Our question was narrower. What does Flint need to run on a card a small team can rent?

Recursive's server kit needs one datacenter card with 141 GB or more. The 22.9 GB compressed file is the cheaper path. It runs on llama.cpp. Recursive tested it on several NVIDIA machines, datacenter and desktop, and recommends at least 48 GB of memory. That figure covers Flint's full context of 262,144 tokens.

The test: two gaming cards, one request at a time

We wanted the smaller answer. We ran the official compressed file on a 32 GB RTX 5090 and a 24 GB RTX 4090 on our cloud, and cut the context until the model fit. On the 5090 that meant 64K tokens, a quarter of the full length. On the 4090 it meant 16K.

Support in llama.cpp arrived just in time. The change that teaches it the Flint layout (PR #30928) was written by alexey-rsi and merged on 21 October. We started our test the next morning.

The main task is the one a coding agent does all day. We gave Flint a tiny project with an invoice module whose discount code treated a percentage as a fraction, and two failing tests. Flint had four tools: list the files, read a file, write a file, and run the tests. It had to find the bug, fix it and prove the fix. We repeated that run seven times on three machines. We also hid a short code word inside long prompts of 8.8K and 17.5K tokens, to check that the model still finds things when it reads a lot.

Our seven runs are one planted bug in a toy project, not a benchmark. We ran below what Recursive recommends: with 64K or 16K of context, Flint is not set up for long agent sessions. Every test sent one request at a time. We did not try a real 16 GB or 12 GB card, or the full 262K context. The smaller cards below are emulated on a slow machine, so their speeds are uncertain.

Results: five turns and three passing tests in about six seconds

Flint fixed the bug in all seven runs, and it did it the same way every time. It listed the files and ran the tests, which failed. It read the module and the test file, wrote the fix, ran the tests again and saw three pass. Then it explained the percent-versus-fraction mistake. That is five turns, and it never edited the tests to make them pass. On the 5090 the whole loop took 5.9 seconds.

Speed on the 5090 felt instant for one person. Flint wrote about 210 tokens per second, roughly 150 words a second. A 374-token coding answer came back in three seconds. Reading is faster than writing for these models. Flint took in a 17,557-token prompt, about the size of a long design document, in under two and a half seconds. With 64K of context the card used about 23.3 GiB of its 32.

The details held up too. Every tool call it sent was valid, and it read both files in a single turn. It found the hidden code word in all three long prompts. One side answer was wrong: asked to compute a small helper function, the 5090 run said 171 where the right answer is 113. The other two runs got it right.

One result belongs here so the 210 does not read as a lucky draw. Our first RTX 5090 ran the same file and the same llama.cpp build at 27 tokens per second, about 8 times slower. Our second 5090, on another machine, gave the 210 with the same file, so the slowness came from that machine, not from the model or its format. The visible differences were a 400 W power cap and an older driver. Our cloud reported that card as healthy, and nothing checks GPU speed when a machine starts. Status: open, root cause not isolated.

Which GPU: a 24 GB RTX 4090 works at 16K, and 32K does not fit

The smallest card that worked is the RTX 4090 with 24 GB. With 16K of context and the whole model on the card, it filled all but about 0.7 GiB of its usable memory. It wrote about 167 tokens per second, about 20% slower than the 5090. It ran the same agent loop in 5.7 and 6.3 seconds, with three tests passing each time. For short agent tasks, you would not notice the difference.

Doubling the context to 32K broke it. The model loaded, then failed to reserve its last sliver of working memory, under 0.1 GiB, and stopped before it served a single request. The file alone takes 21.3 GiB, and the card exposes only about 22.5 GiB. So 16K was enough for a task like ours, not for a long session in a large codebase.

Below 24 GB we only emulated smaller cards, by capping memory on the slow 5090 and moving part of the specialists to the machine's main memory. With a cap near 16 GB, Flint answered, but at 4.6 tokens per second, and one request took 111 seconds. With a cap near 15 GB, it wrote about 6 tokens per second, and our endpoint gave up twice after its two-minute limit. We count both as failures for interactive use.

On InferenceKey: ready in 13.5 seconds warm, after one 7-minute build

The card is yours by the hour, and you choose what runs on it. On InferenceKey, the worker is the small program on the GPU machine that fetches the model and starts the engine. For Flint, its built-in llama.cpp mode served the official file with no custom code. The same endpoint answered the agent's tool calls in the OpenAI format coding tools already speak.

The first start on a fresh machine took about nine minutes. Seven of those went to compiling llama.cpp for the card. Downloading the 22.9 GB file took about two more, and loading it took about six seconds. After that, a restart with everything in place was ready in 13.5 seconds.

Two problems are the ones you will meet.

The first is the llama.cpp version. The worker's default version, b9748, predates Flint. We built the merge commit by hand, because we thought no released version contained it. That was our mistake: release b11803 is the same code, published before our test began. Set the worker to b11803 or newer. Status: open on our side. Our worker default (b9748) and the example README still need updating. We have not run the worker with b11803.

The second is where the model's thinking ends up. Flint reasons before it answers, and with this build that reasoning arrives inside the answer text instead of in its own field. Coding tools set up as Recursive suggests look for it in a separate place and will not find it. The reasoning also counts against your token budget. Status: open.

Should you use it? Yes for short agent tasks on a 24 or 32 GB card

If you want an open coding agent on hardware you own or rent, Flint is worth a try today. A 32 GB RTX 5090 has room for a medium task, and a 24 GB RTX 4090 for short ones.

If your agent sessions run long, across big codebases and hundreds of tool calls, plan for more memory.

Thanks to Recursive for building Flint and to alexey-rsi for its llama.cpp support. We spotted it in a thread on r/LocalLLaMA.

Run python main.py from the Flint SDK example to repeat the bug-fixing loop on your own NVIDIA worker. It needs the SDK built from the repo, 32 GB or more for its 64K default, and llama.cpp b11803 or newer. Expect about seven minutes for the first build and a 22.9 GB download.

For engineers

Model and files

  • RecursiveSI/flint: FlintForCausalLM, 39.8B total, about 2.5B active per token including the output head (about 2.2B for the body alone). 49 layers: 2 dense (mlp_only_layers [0,1]) and 47 MoE with 256 routed experts, 6 per token, plus a shared expert (shared_expert_intermediate_size 1024). 35 sliding-window layers (window 513) and 14 full-attention layers. 32 query heads, 4 KV heads, head_dim 128, hidden 2048, vocab 150,144, max_position_embeddings 262,144. Sigmoid router, per-head QK RMSNorm, learned per-head attention sinks (softmax_type "learnable"), RoPE on sliding-window layers only. Trained from scratch through pretraining, mid-training, SFT and RL. Knowledge cutoff June 2026. Token counts not stated. Transformers modeling code not included yet. BF16 checkpoint about 80 GB.
  • Vendor evals (self-reported, not verified by us): Terminal-Bench 2.1 54.9% (Terminus 2), SWE-bench Verified 74.2% and SWE-bench Pro v2 89.7% (mini-swe-agent). Average pass@1 over at least five attempts, temperature 1.0, top-p 0.95, top-k 20. Web-retrieved Terminal-Bench solutions counted as failures. Competitor scores are only in a chart image, so we did not extract them.
  • flint-nvfp4-gguf: flint-nvfp4.gguf, 22,863,084,160 bytes, weights, tokenizer and chat template in one file, for llama.cpp on CUDA. The authors list B300, RTX PRO 6000 Blackwell, RTX Spark and DGX Spark as tested, and recommend at least 48 GB of GPU memory (or unified memory on Spark). Their example command uses -c 262144; we ran below that recommendation. llama-bench reports it as 21.29 GiB, 39.79 B params.
  • flint-int4-gguf: flint-int4-q4_1.gguf, 25,992,966,976 bytes, Q4_1, aimed at Apple Silicon with Metal and 48 GB unified memory. We ran it only in llama-bench, on the slow host.
  • flint-int4: MLX INT4 (group-64 affine, INT8 embeddings), about 22.6 GB, for mlx-lm.
  • SGLang kit (sglang/ in the main repo, SGLang 0.5.16): one GPU with at least 141 GB per replica (B300, B200 or H200), "80 GB GPUs are not supported", driver 580 or newer. Serve scripts use --context-length 262144 --tool-call-parser flint --reasoning-parser flint. Quantized weights and KV, TP/EP and speculative decoding not validated by the authors.
  • recursive-oss/rec: Apache-2.0 engine for Flint on Apple Silicon (M4 Pro, M5 Pro, M5 Max, 48 GB or more), OpenAI-compatible API, claims 1.4 to 2.6x llama.cpp decode on M4 Pro (their number).

llama.cpp support

  • PR #30928 "models: Add Flint model (FlintForCausalLM)" by alexey-rsi, merged 2026-10-21T21:26:22Z, merge commit a37cc46104eb50900d5ab4a9ba02a0abdac039db, +319/-7 in 18 files, approved by CISC and ggerganov. It uses adaptive expert-activation scaling from #30861 as a workaround, and the PR reports a 10 to 30% slowdown from that auto-scaling, to be addressed in follow-ups (not measured by us). The PR discloses that Codex substantially assisted the implementation.
  • PR #30929 "jinja: support hexadecimal string escapes" (alexey-rsi, merged 2026-10-21T16:04:37Z, approved by CISC and danbev) is needed by the chat template.
  • Release b11803 (published 2026-10-21T22:30:59Z) is identical to a37cc461. b11804 and later contain it too. v0.7.0 is 134 commits behind.

Our setup

  • Cards on our cloud: RTX 5090 32 GB (driver 595.91.07, 575 W), RTX 4090 24 GB (23,028 MiB usable, driver 595.91.07, 450 W), and a first RTX 5090 (driver 580.178.04, 400 W power cap) that turned out to be a slow host.
  • A local InferenceKey manager from main at 5c08ab03, reached by the workers through dev tunnels. Production never touched.
  • Worker env: INFERENCEKEY_LLAMACPP_TAG=flint-a37cc461, INFERENCEKEY_LLAMACPP_READY_TIMEOUT_SECS=3000, CMAKE_BUILD_PARALLEL_LEVEL, CUDACXX.
  • llama.cpp built by hand at a37cc461 in the worker install path /opt/inferencekey/llamacpp-cuda/flint-a37cc461-cuda<arch>/. The worker found build/bin/llama-server there and reused it. Its default tag is b9748, and it clones with git clone --branch <tag>, which accepts a tag or branch name, not a commit.

Deploy step by step

  1. Clone the SDK and open examples/flint-coding-agent-llamacpp-private-nvidia (main.py, README.md, .env.example, requirements.txt, commit 1654d272).

  2. Copy .env.example to .env, fill it, and export the variables into your shell (main.py reads them from the environment and does not load .env itself):

    INFERENCEKEY_SDK_TOKEN=...
INFERENCEKEY_API_KEY=...
INFERENCEKEY_PROJECT=...
IK_WORKER_ID=...            # private NVIDIA worker, 32 GB or more for the 64K default
# optional
INFERENCEKEY_BASE_URL=...
IK_KEEP_ALIVE=1             # keep the deployment after the run

  3. On the worker, set a llama.cpp version that contains Flint, and give the first build time:

    INFERENCEKEY_LLAMACPP_TAG=b11803          # or newer; we used a hand build at a37cc461
INFERENCEKEY_LLAMACPP_READY_TIMEOUT_SECS=3000
CMAKE_BUILD_PARALLEL_LEVEL=<cores>
# Blackwell needs the CUDA toolkit with nvcc 12.8 or newer

    The example README asks for a tag that contains a37cc461, or master. Its note that no release tag contained the PR when we tested is outdated: b11803 matches the merge commit. We have not run the worker with b11803 itself.

  4. Install and run. The SDK is not on PyPI yet (the inferencekey line in requirements.txt is commented out), so build it locally first, as the example README and ../CONTRIBUTING.md describe:

    python -m venv .venv && . .venv/bin/activate
pip install maturin
( cd ../../bindings/python && maturin develop --release )
pip install -r requirements.txt
set -a; . ./.env; set +a
python main.py

    main.py calls ensure() with Backend.LLAMACPP, waits up to 3,600 s for ready, asks for a merge_intervals function, runs the coding-agent loop with list_files, read_file, write_file and run_tests confined to a temp dir, writes outputs/answers.json, and deletes the deployment unless IK_KEEP_ALIVE=1.

  5. The command it deploys:

    llama-server -hf RecursiveSI/flint-nvfp4-gguf --hf-file flint-nvfp4.gguf --alias flint \
  -ngl all -c 65536 -np 1 -fa on -ctk f16 -ctv f16 --jinja \
  --temp 1 --top-p 0.95 --top-k 20 --min-p 0

    The command is hardcoded in main.py (line 58), and no env var or flag changes it. For a 24 GB worker, edit -c 65536 to -c 16384 there. Leave --swa-full unset, as the model card says.

Time to ready

StepRTX 5090RTX 4090
llama.cpp CUDA build, fresh machine7 min 13 s (-j 46)6 min 26 s
ensure() to ready, fresh129.5 s (download about 114 s at about 200 MB/s, load about 6 s)126.9 s (download about 113 s, load 5.3 s)
Warm restart (binary built, weights cached)13.5 s (load 4.6 s); ladder rungs on the slow 5090 13.8 to 14.8 snot measured

Speed

Host and configDecode, per requestllama-benchPrefill
RTX 5090, 64K, native 120a build194 to 216 tok/s (long outputs about 210)tg128 239.8, pp512 6,2177,352 tok/s at 17,557 tokens
RTX 5090, PTX build (89-virtual)tg128 216.1, pp512 6,495
RTX 4090, 16K154 to 170 tok/s (about 167)tg128 181.2, pp512 5,6586,230 tok/s at 8,807 tokens
Slow RTX 5090, 64K, 32K, 16K26.6 to 28.4 tok/stg128 28.8 (NVFP4), 29.3 (Q4_1), pp512 2,5932,726 tok/s at 17,557 tokens

llama.cpp CMake picks CMAKE_CUDA_ARCHITECTURES=120a on Blackwell by itself, and the native build was about 11% faster than PTX JIT. NVFP4 also ran on Ada (sm_89, the 4090). On the 5090, a 374-token coding answer took 3.0 s.

KV estimate: 14 full-attention layers x 4 KV heads x 128 x 2 (K and V) x 2 bytes = 28,672 bytes per token, about 0.9 GiB at 32K and 1.75 GiB at 64K.

Full VRAM ladder

TargetConfigPeak VRAM (llama-server)SpeedOutput OKReal card
RTX 5090 32 GBNVFP4, -c 65536, F16 KV, -ngl all23,872 MiBabout 210 tok/s, agent 5.9 syesyes
RTX 5090 32 GB (slow host)NVFP4, -c 3276822,922 MiB27.3 tok/syesyes
RTX 5090 32 GB (slow host)NVFP4, -c 1638422,458 MiB27.5 tok/syesyes
RTX 4090 24 GBNVFP4, -c 16384, F16 KV, -ngl all22,316 MiB of 23,028about 167 tok/s, agent 5.7 and 6.3 syesyes
RTX 4090 24 GBNVFP4, -c 32768OOM at loadcudaMalloc failed: out of memorynoyes
About 16 GB cap (slow 5090)NVFP4, -c 32768, --n-cpu-moe 1816,116 MiB4.57 tok/s, 111 s requestyes, too slowemulated
About 15 GB cap (slow 5090)NVFP4, -c 32768, --n-cpu-moe 22 -t 1214,388 MiBabout 5.9 tok/s, 503 after 120 s, twicenoemulated

The 4090 failure at 32K:

ggml_backend_cuda_buffer_type_alloc_buffer: allocating 87.50 MiB on device 0: cudaMalloc failed: out of memory
failed to allocate buffer for kv cache

The SDK then raised failed to become ready: bootstrap step failed.

Not tried: 12 GB cards, RTX 3090 (no stock when we tried), the 5090 above 64K, any 48 GB card, the full 262K context, concurrency above -np 1.

Quality (our prompts, not a benchmark)

  • Coding-agent loop: 7 of 7 runs on 3 machines fixed the bug, always 5 turns, 559 to 654 tokens, never touched the tests.
  • Tool-call JSON valid every time, with parallel calls. In a separate tools bench our canned read_file returned the same text for every path (our bug). The model noticed ("suspiciously identical") and kept searching for 6 turns instead of answering.
  • merge_intervals: correct with three asserts in 5 runs.
  • A planted off-by-one in a chunks function: found 3 of 3.
  • Hidden string FLINT-4471-NVFP4: found 3 of 3 at 8.8K and 17.5K tokens. Arithmetic side question right 2 of 3 (helper_175(2) answered 171, correct 113).

Demo video

Frame of the Flint deploy demo on an RTX 5090: the agent result with 3 tests passed and a lower third with the speed. Click to play the 73-second video

72.9 s, 1080p, narrated, recorded on the healthy 5090: the workload in the panel, the buggy module, the agent loop (5 turns, 654 tokens, 5.9 s, 3 passed). An earlier take on the slow 5090 was replaced.

Remaining problems and status

  1. llama.cpp version. Flint needs b11803 or newer, and the worker takes a tag, not a commit. Status: resolved upstream, open on our side (worker default b9748 and README not updated yet), untested by us with the tag. Our SDK README asks for a tag containing a37cc461 or master; its note that no release tag existed is outdated, and naming b11803 there is pending.
  2. Slow host. One RTX 5090 decoded 27 to 29 tok/s with the same file and commit, also with Q4_1. nvidia-smi dmon showed SM 100%, memory 0 to 6%, 399 W at its 400 W cap. A second 5090 ran 8.3x faster in llama-bench (tg128 239.8 against 28.8). Root cause not isolated (the power cap and driver 580.178 are the visible differences). The platform reported it healthy, and nothing checks GPU performance at bootstrap. Status: open.
  3. 24 GB usable memory. The 4090 exposes 23,028 MiB. At 16K llama-server used 22,316 MiB; 16K to 32K added 464 MiB on the slow 5090 (22,458 to 22,922), and the driver holds memory outside the process (an idle 5090 showed 1,039 MiB used), so NVFP4 with 32K context does not fit. Status: a limit, use 16K.
  4. Sync timeout. The endpoint's 120 s sync timeout returns 503 for slow configs such as CPU expert offload. Status: open.
  5. Hand build. The llama-server target fails at llama-ui-assets unless all 8 UI stub files exist (the worker seeds them). Status: only affects hand builds.
  6. Reasoning in content. With this build and template through our endpoint, reasoning text arrives inside content and reasoning_content is absent. Reasoning tokens count in usage.completion_tokens and against max_tokens (the example uses 16,384). Recursive's client configs use a 65,536-token generation budget and ask to keep reasoning_content when replaying history. Status: open.

Limits

One request at a time, seven agent runs, our own prompts. Sampling at temperature 1.0, top-p 0.95, top-k 20, as Recursive recommends. Benchmarks are vendor reported. The emulated rungs ran on the slow host. No real 16 GB or 12 GB card was tested.

Credits

Recursive (model, NVFP4 and INT4 files, SGLang kit, rec). alexey-rsi (llama.cpp PRs #30928 and #30929). CISC and ggerganov (approved #30928). CISC and danbev (approved #30929). ggerganov (#30861, the per-expert MoE scaling the Flint support uses). Our part: deployment on InferenceKey, the SDK example, the measurements and the video.

Sources