SenseNova Code2Video on one RTX 4090: a draft in 10 minutes, a gap in every row, no finished film

A 27B model that makes video by writing code ran in Claude Code on one 24 GB card: a draft render with a layout error, and no finished film.

Published · By InferenceKey · 8 min read + technical appendix

You do not need a video model to make an explainer video. A text model can write a web page that draws every frame, render it, look at the frames and fix what it sees. SenseTime's open 27B model, SenseNova 6.8 Code2Video, does that. Driven by Claude Code, it ran on one 24 GB RTX 4090. Ten minutes in, it had a complete 12-second clip that passed its automatic file check. The picture was wrong: dots meant to form a 5 by 6 rectangle landed in 5 rows of 7, with a gap in every row. Then its working memory, everything it had read and written, filled up. Claude Code spent 21 minutes trying to summarize it, hit our answer-length limit each time, and never finished. The model fits one gaming card, and our SDK example repeats the setup.

What the model drew: the brief asked for a 5 by 6 rectangle of dots. Left, the Claude Code run on a 24 GB card with the 4-bit file, frame at 7.2 seconds, first full render: 30 dots in 5 rows and 7 columns, red boxes on the empty cell in every row, an amber box around the 7 columns, so the two staircases never close into the rectangle. Right, one single request with the 3-bit file on the 4090, frame at 2.5 seconds: a red circle on the only dot on screen, because earlier dots vanish, while the closing sum is right

GPU time on one RTX 4090First full 12 s renderSmallest copy whose page renders
Busy writing output for 32 of the 40 minutesA draft, one layout errorA smaller 3-bit copy peaked at 13.7 GiB on the 4090, under 16 GB (page renders with a bug)

A video made of code: the model writes a program, renders it and reviews the frames

SenseNova 6.8 Code2Video does not paint pixels. Its authors call it "a lightweight open model for code-driven video creation" that "builds and renders videos by generating executable code". A clip is a web page where every moment in time has exactly one picture. A browser with no window steps through that page frame by frame, and a standard video tool glues the frames into an MP4.

By design, every element on screen is code, so you can change one label or one colour and render again. Text comes out spelled exactly as written, because it is typed, not drawn. And the "video model" is an ordinary text and image model. You serve it with the same open-source servers you already use for chat models.

SenseTime published the model under an open license, plus a Claude Code skill (a packaged set of instructions and scripts) called code-video. The skill teaches the loop: plan, write the page, render stills and look at them "at least twice". Then render the film, run a file check and write a verification note. Its best advice is one line: "Get a rough full-length render early (even ugly), then improve it."

They also published six complete production runs. Each film took 261 to 876 rounds of the model writing, running and checking, and 37 to 99 frames looked at. The model card says it "sustains runs of more than 10 hours and more than 2,000 turns"; we have not checked that. The authors run the full-size model on four GPUs, with a working memory five times the one we used (512K tokens). They are also candid: no benchmark, "clearly behind the strongest proprietary models on visual polish and long-horizon reliability", and no photorealistic footage.

So the question we brought to the test was smaller than theirs. Can a compressed copy, a third of the size, finish a short clip on one 24 GB card, and in how many turns and minutes? We found no published run that answers it.

Our test: one 12-second brief, one 24 GB card, a 40-minute cap

The brief was a silent 12-second explainer called "Half a Rectangle". It shows 1 + 2 + 3 + 4 + 5 = 15 as a staircase of blue dots. Then it copies the staircase in orange, turns the copy, and closes the two into a 5 by 6 rectangle of 30 dots. Half of 30 is 15. It is a small brief with one fact that has to be right.

The authors' first step is to expand a short idea into a dense, detailed brief with a strong general model. They use Claude Opus 5.5, and so did we, with their template. It was the only closed model in our chain. The 898-word brief then went to SenseNova inside Claude Code, running unattended in a locked-down box with the skill unchanged. Every model call went to the RTX 4090 on our cloud.

The model ran from bartowski's compressed copy (4-bit, 17.2 GB) plus a 0.93 GB vision part, so it could see its frames. We gave Claude Code a working memory of 98,304 tokens instead of the authors' 512K, a 16,384-token cap per answer instead of 64K, 120 turns and 40 minutes. Those limits are ours, set by one card and one morning. No audio.

A full draft render, with seven columns where the brief asked for six

The model read the skill and the brief, and wrote its plan at 5 minutes. It drew its first stills at 6 minutes. At 10 minutes it had the full 12 seconds rendered: 1280 by 720, 30 frames a second, 360 frames. This was one attempt at one brief, and the model's answers vary on purpose, so a second try could land better or worse.

The model's own 12-second render from the Claude Code run, frame at 7.2 seconds with the dot staircases. Click to play the video

The loop worked the way the authors describe. The model read 21 of its own frames as images and acted on what it saw. It saw the orange dots land in the top-left corner, traced it to a movement applied twice and fixed it. Then it saw its highlight bands covering the grid lines and fixed the drawing order. Motion is clean and every word on screen is spelled exactly as in the brief.

The picture still breaks the one fact that matters. We measured the dot centres at 7.2 seconds: 30 dots in 7 columns by 5 rows, each row "blue, one empty cell, orange". The caption says "5 rows × 6 columns = 30", and the 6-column grid drawn behind the dots leaves the seventh column out. The file check passed anyway. It checks resolution, frame rate and length, not the picture.

The cost, then: a draft in 10 minutes; the whole 40-minute run took 61 tool calls and about 78,000 written tokens, and a finished clip is unknown because the run did not finish.

The last 21 minutes went to Claude Code summarizing its own memory

Think of the working memory as a desk: every file read, every frame looked at and every line written stays on it. When the desk is 80% full, Claude Code asks the model to summarize the work so far, clears the desk and keeps only the summary.

Our desk held 98,304 tokens. It filled after about 19 minutes of work. Claude Code asked for a summary three times. Each time the model wrote until it hit the 16,384-token cap on a single answer, and the summary never came back usable. Those three attempts took 21 minutes, until the 40-minute limit stopped everything with no final film and no verification note.

In four of the authors' six runs, Claude Code cleared a desk of 167K to 492K tokens, with a 64K answer cap. The likely suspects are our settings: the 16,384-token answer cap, and the model's hidden reasoning, left on, which spends that cap. Our logs do not show which.

Which GPU: 24 GB ran the agent, smaller copies only answered one request

We ran the agent on a real 24 GB 4090. With no 16 GB card in stock, we tested smaller copies on the 4090 and watched their peak memory. Each got one short request for a 6-second animation page (capped at 8,192 tokens, a smaller brief than the agent's) and a picture of 15 dots to describe.

Card or budgetCopy and working memoryPeak card memorySpeedResultReal card?
RTX 4090 24 GB4-bit, 98K tokens, Claude Code run20.9 GiB39 to 48 tokens/sClaude Code run, see aboveyes
RTX 4090 24 GB4-bit, 32K tokens, one request18.8 GiB47.6 tokens/simage right, page never written: all 8,192 tokens went to hidden reasoningyes
16 GB budget3-bit, 32K tokens, one request13.7 GiB63.5 tokens/simage right, page renders, but only one dot shows at a timeno, run on the 4090
12 GB budget2-bit, 16K tokens, one request10.0 GiB78.7 tokens/simage right, page cut off at 8,192 tokensno, run on the 4090

One sample each says little, and the 4-bit request wrote no code, so there is no like-for-like comparison.

On InferenceKey: ready in 25 seconds, image fix not yet deployed

On a fresh machine with the files on disk, the model answered 25.2 seconds after we asked, and Claude Code talks to it like Anthropic's API. Pointing Claude Code at the 4090 took a URL, a model name and a few context settings. On our side it took two changes: an image fix, and a 30-minute request timeout, so long answers were not cut off at 2 minutes. Nothing in the authors' skill changed.

The problem you will hit: when Claude Code sent a frame, our platform dropped the picture or passed it on as text. Status: fixed in our test build, not yet on the hosted service, where frames still arrive blind.

This run used a local copy of the platform, not the hosted service. Smaller rough edges (a timeout, a machine restart, an upload retry) are in the appendix with their status.

Should you use it? Worth trying for short drafts on a 24 GB card, if you review every frame

  • You want explainers and motion graphics with exact text: worth a try on a 24 GB card.
  • You need the picture right the first time: review every frame yourself.
  • You want a finished film unattended: plan on hours and far more working memory, as the authors do. Our 40-minute run did not finish.
  • You have a 16 GB card: the 3-bit copy fits on paper, is untested as an agent, and its one sample had a bug.

To make your own clip, run python main.py from the SenseNova Code2Video SDK example on a 24 GB card rented on our cloud; the setup steps are in the appendix, and its Claude Code step sees frames only once the image fix ships.

For engineers

Credits

  • SenseTime (SenseNova team): the model (Apache-2.0), fine-tuned from Qwen3.8-27B per its card; the model card with vLLM, SGLang and LightLLM recipes; six published production trajectories; the code-video skill, prompt template and media (the GitHub repo states no license).
  • The Qwen team: the base model the authors name.
  • bartowski: the GGUF quants we ran (Q4_K_M, IQ3_XXS, IQ2_XXS, bf16 mmproj), made with llama.cpp b11969 and an imatrix.
  • The llama.cpp maintainers: release b12167 with the prebuilt CUDA 13.4 tarball, multimodal support and --jinja tool-call parsing.
  • Anthropic: Claude Code, the harness the authors recommend, and Claude Opus 5.5, which wrote our dense brief from the authors' template.
  • AI Search: where we spotted it (2026-11-01 news video, chapter at 39:53).
  • Our part: the deployment through the platform, the image fix in our /v1/messages translation, the sandboxed Claude Code run, the pixel check of its output, the memory ladder, the SDK example and the demo video.

Model and files

  • sensenova/SenseNova-6.8-Code2Video: sha 0f99ab03, created 2026-10-24, last modified 2026-11-03, not gated, Apache-2.0, pipeline_tag image-text-to-text, languages en and zh. Published in late October; card last updated 2026-11-03.
  • 27,356,728,560 parameters in BF16 (12 shards, about 54.7 GB). Qwen3_5ForConditionalGeneration: 64 layers with full_attention_interval 4 (16 full-attention layers, 48 linear-attention), 4 KV heads x head_dim 256, hidden 5120, vision tower depth 27, 1 MTP layer. max_position_embeddings 262,144 with yarn factor 2.0 (the 512K claim).
  • Generation config: temperature 1.0, top_k 20, top_p 0.95, sampling on.
  • bartowski GGUF 0ef7c7e3: bf16 53.81 GB, Q8_0 28.67, Q6_K 23.62, Q5_K_M 20.68, Q4_K_M 17.20 (flagged default), IQ3_XXS 12.08, IQ2_XXS 8.64; mmproj bf16 0.93 GB and f16. Other community quants listed on the card: OliviaRossi GSQ-RCO-GGUF, Abiray MTP-GGUF, BennyDaBall NVFP4-GGUF, Hayabala EXL3.
  • Backend binary: the prebuilt llama.cpp for NVIDIA, so nothing had to be built.
  • KV cache at q8_0: 16 layers x 4 heads x 256 x 2 x 1 B = 32 KiB per token, so 96K is about 3 GiB (matched the measured peak) and 512K would be about 16 GiB.
  • Authors' harness settings: CLAUDE_CODE_MAX_CONTEXT_TOKENS=524288, CLAUDE_AUTOCOMPACT_PCT_OVERRIDE=80, max output 64K, --max-turns 2000 or more, timeout 10 h or more. Claude Code otherwise treats an unknown model as a 200K model. The HF card says CLAUDE_CODE_MAX_OUTPUT_TOKENS=64000, the GitHub README 65536. The card says runs of more than 2,000 turns; the README's weights section says "over a thousand". Neither is verified by us.
  • Authors' trajectories (trajectories/all.openai.jsonl, thinking on): fourier_light_painting 675 assistant turns and 99 review images, forest_landslide 445 and 47, signal_noise_shape 713 and 51, cmyk_halftone 261 and 37, geometry_risograph 521 and 67, frame_form_flow 876 and 69. Compaction: fourier 2x at about 492K to 13-16K; signal_noise_shape 1x at 491K to 14K; risograph 4x at about 167-172K to 10-15K; frame_form_flow 7x at about 167-173K to 6-13K; forest_landslide used 2 resumes; cmyk_halftone has no context boundary. The 6-16K figures are the whole context left after compaction (postTokens), not the summary length.
  • Skill environment: Claude Code, Node.js 18+, Python 3, ffmpeg/ffprobe, fonts, Playwright Chromium; requirements.txt numpy, scipy, soundfile, librosa, edge-tts, faster-whisper. GitHub HEAD 776ae798, one commit dated 2026-10-26.

Setup

  • GPU on InferenceKey Cloud: one RTX 4090 24 GB (24,564 MiB, 450 W, compute capability 8.9, driver 595.91.07).
  • Backend: the built-in llamacpp backend with INFERENCEKEY_LLAMACPP_TAG=b12167 and INFERENCEKEY_LLAMACPP_ASSET=ubuntu-cuda-13.4-x64. Why llama.cpp: the card's recipes are vLLM, SGLang and LightLLM at BF16 with tensor parallelism 4 and 512K context (about 54.7 GB of weights alone). The 17.2 GB GGUF is what fits one consumer card.
  • A local InferenceKey manager built from our main branch, with the image patch. Production never touched.
  • Claude Code 2.1.322 headless in a Docker sandbox (Ubuntu 24.04, ffmpeg, Node 24, Playwright 1.56.1 Chromium, Noto and DejaVu fonts, --memory 8g --cpus 6), only the task folder and a fresh HOME mounted, API key passed by environment.
  • Worker manifest: INFERENCEKEY_BACKEND_REQUEST_TIMEOUT_SECS=1800 (default 120 s); the run was not tried without it.
  • Claude Code settings: CLAUDE_CODE_MAX_CONTEXT_TOKENS=98304, CLAUDE_AUTOCOMPACT_PCT_OVERRIDE=80, CLAUDE_CODE_MAX_OUTPUT_TOKENS=16384, CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1, small model set to the same slug. Claude Code printed [claude-code:unrecognized_model], harmless.
  • Brief: "Half a Rectangle", 12 s, 1280x720, 30 fps, silent, 898 words, written by Claude Opus 5.5 from prompts/enhance_prompt.md. Prompt: the authors' stage-2 prompt plus one sentence giving the 40-minute limit.

The launch command the backend runs:

INFERENCEKEY_LLAMACPP_TAG=b12167 \
INFERENCEKEY_LLAMACPP_ASSET=ubuntu-cuda-13.4-x64 \
llama-server \
  -hf bartowski/sensenova_SenseNova-6.8-Code2Video-GGUF \
  --hf-file sensenova_SenseNova-6.8-Code2Video-Q4_K_M.gguf \
  --mmproj-url <bf16 mmproj URL> \
  --alias sensenova-code2video -ngl 99 -fa on \
  -c 98304 -ctk q8_0 -ctv q8_0 \
  --jinja --reasoning-format deepseek \
  --temp 1.0 --top-k 20 --top-p 0.95 \
  -np 1 --cache-reuse 256

The exact COMMAND string, with the mmproj URL, is in the example's main.py.

Step by step with the SDK example

  1. Clone inferencekey-sdk and open examples/sensenova-code2video-27b-claude-code-llamacpp-private-nvidia (commit d352b7b5e83921cbc9fd28c4f0b9054ee9e2cb23).
  2. cp .env.example .env and fill in INFERENCEKEY_SDK_TOKEN, INFERENCEKEY_API_KEY, INFERENCEKEY_PROJECT and IK_WORKER_ID (a private NVIDIA worker with 24 GB). Optional: C2V_CONTEXT (98304), C2V_KV_TYPE (q8_0). For a 16 GB card: C2V_GGUF_FILE=sensenova_SenseNova-6.8-Code2Video-IQ3_XXS.gguf and C2V_CONTEXT=32768 (untested on a real 16 GB card).
  3. pip install -r requirements.txt plus the local inferencekey Python binding built with maturin (not on PyPI).
  4. IK_KEEP_ALIVE=1 python main.py: ensure() a fixed llamacpp workload pinned to your worker, wait_until_ready(3600), then one streaming /v1/messages call asking for a 6 s window.seek(t) page; saves outputs/clip.html and outputs/answer.json. Without IK_KEEP_ALIVE=1 it calls delete() at the end.
  5. cp claude-settings.example.json claude-settings.json and set your project slug in ANTHROPIC_BASE_URL (https://cloud.inferencekey.com/endpoint/<project>/sensenova-code2video). ANTHROPIC_MODEL must be the workload slug sensenova-code2video (or default), otherwise the endpoint returns 400. API_TIMEOUT_MS is 1800000.
  6. In the task folder run npm install playwright && npx playwright install chromium (the skill's capture.mjs loads Playwright from there, so a global install is not enough), put your dense brief in <task>/dense_prompt.md, then run ./run_claude_code.sh <task>. It copies the authors' skill into the task, then runs claude -p --settings ... --max-turns 150 --permission-mode acceptEdits --allowedTools "Bash,Read,Write,Edit,Glob,Grep,Skill" under a 60-minute timeout (C2V_MAX_TURNS, C2V_MINUTES). Run it in an isolated folder or container: the agent runs shell commands without asking.

The example's streaming path was not run end to end before the GPU was released. The same request shape worked in our image test.

Timings

StepTime
ensure() to ready, fresh worker25.2 s
llama.cpp tarball installabout 1 s
GGUF and mmproj pre-download (not on the ready path)26 s and 3 s
Claude Code start to SPEC.md5 min (07:08 UTC)
First stills07:09 to 07:12
Full 12 s render out/v1.mp410 min (07:13)
Dot centroid measuring with numpy and scipy07:14 to 07:16; 61st tool call at 07:16:44
One long requestabout 07:17 to 07:21, released at 78,416 tokens
Three compaction requests07:21:50 to 07:42:40, each released at n_tokens 82,969
Timeout07:43:04

Decode: 47.7 tokens/s fresh, 44.5 at about 30K, 39.4 at about 66K; 78,039 tokens in 1,915 s (40.8 average); prefill 75 s over 44 requests, only 5 prefills over 8K tokens. Decode was about 96% of GPU time. Peak VRAM 21,425 MiB in the run, 21,729 MiB for the whole GPU session. Tool calls: Bash 29, Read 26 (21 images), Edit 4, Write 2. The last event in the stream is status compacting.

Output check of v1.mp4: check_video.sh passed (1280x720, 30.000 fps, 360 frames, 12.000 s, h264 yuv420p, no audio stream, warning on a deliberate 2.1 s frozen end hold). At t = 7.2 s, 30 dot centres in 7 columns (x 441 to 837, 66 px pitch) by 5 rows (y 247 to 512), each row blue k, one empty cell, orange 6-k. At t = 11.5 s, "30 ÷ 2 = 15" is readable but the 6-column guide misses the seventh column. All on-screen text matches the brief. The model's own notes from the run: "B's final placement is off (dots land top-left), the settle term double-counts the translation", then a fix to the grid band draw order, and late in the run "My snap origin was wrong (rows start at y=247, cols at x=441). Redo:".

VRAM ladder

Rungs 2 to 4 ran directly on llama-server b12167 on the 4090 with q8_0 KV cache and the bf16 mmproj on the GPU. Each got the example's 6 s page brief (193 prompt tokens, max 8,192 output tokens) and a 15-dot PNG with a cold-read question. Peak from nvidia-smi every 0.5 s. This brief is smaller than the Claude Code run's, so the rungs are not like-for-like with it.

GPU or budgetConfigPeak VRAMTime per outputOutput OKReal or emulated
RTX 4090 24 GBQ4_K_M + bf16 mmproj, 98,304 ctx q8_0, Claude Code through the endpoint21,425 MiB39.4 to 47.7 tok/s; full render at 10 minno: 5x7 with gaps, compaction stall, no final.mp4 in 40 minreal
RTX 4090 24 GBQ4_K_M + mmproj, 32K q8_019,233 MiB47.6 tok/s; brief 172.5 s; image 7.7 simage right; brief 8,192 tokens of thinking, no answer (finish length)real
16 GB budgetIQ3_XXS + mmproj, 32K q8_013,994 MiB63.5 tok/s; brief 4,375 tokens in 69.2 s; image 4.3 simage right; page renders (180 frames) but only one dot visible at a time; closing sum rightemulated on the 4090
12 GB budgetIQ2_XXS + mmproj, 16K q8_010,252 MiB78.7 tok/s; brief 104.4 s; image 11.3 simage right; page cut at 8,192 tokens (incomplete html)emulated on the 4090

A real 16 GB card was attempted and every option was out of stock (RTX 2000 Ada, RTX 5080, RTX A4000, RTX 4080 and 4080 SUPER, RTX 4000 Ada). Whether 13,994 MiB leaves enough headroom on a real 16 GB card with driver overhead is unknown.

The 3-bit file's one-shot page under a 16 GB budget, frame at 5.9 seconds. Click to play the video

Not tried: a real 16 or 12 GB card; Claude Code on the smaller files; contexts above 96K; thinking off (--chat-template-kwargs); a larger output cap or a different compaction setting; the authors' BF16 recipe on vLLM with tensor parallelism 4; audio.

Remaining problems

  1. /v1/messages dropped images (crates/manager/src/api/openai.rs, anthropic_message_to_openai): image blocks were skipped and tool_result arrays were stringified, so a PNG read by Claude Code would reach the model as base64 text. Local fix: image blocks become OpenAI image_url parts; tool-result images are re-attached on the following user message. Live check: 200, streaming, 601 in / 291 out tokens, 6.8 s, correct read. A unit test was added but not compiled (disk full). Open until deployed to the hosted service.
  2. Non-streaming calls die at 120 s. The first non-streaming call returned 503 after the worker's backend timeout, with llama-server at 5,599 thinking tokens. Fixed for the run with INFERENCEKEY_BACKEND_REQUEST_TIMEOUT_SECS=1800; the example now streams.
  3. A worker restart did not adopt the running llama-server. The second llama-server exited for lack of VRAM; about 1.5 minutes lost until the watchdog brought the workload back. Open.
  4. Worker binary upload timed out on the first machine (SFTP, high latency) and the manager terminated it after 2.18 minutes. A retry restricted to EU uploaded in 8 s. Known, open.
  5. Compaction at 98K: three summaries ran to the 16,384-token cap; the logs do not show why. Untested remedies: a bigger context, a bigger output cap, thinking off.
  6. Minor: llama.cpp disables cache_reuse with multimodal; normal prefix caching still worked.

Demo video

The 203-second narrated demo: the authors' films first (credited to SenseNova), their model card, turns per film and file sizes, then our deployment on the RTX 4090, a replay of the 61 tool calls and the compaction, our clip, the key comparison and the memory ladder.

SenseNova Code2Video deploy test on one RTX 4090, frame with the key comparison. Click to play the video

Sources