RelayVSR upscales a 3-second 480p clip 4x in 10 seconds of GPU time on one 24 GB card

Credits

• Xijun Wang, Xin Li, Zirui Lang, Suhang Yao, Haoran Li and Zhibo Chen (University of Science and Technology of China): RelayVSR, the paper "RelayVSR: Large-Small Model Collaboration for Efficient Real-World Video Super-Resolution" (arXiv 2609.37850, submitted 29 Sep 2026), the weights (Apache-2.0), the code with inference.py and refvsr_inference, and the project page.
• Wan-AI: Wan2.2-TI2V-5B (Apache-2.0), the base of the reference generator.
• FlashVSR (OpenImagingLab, Zhuang et al., 2026): a baseline in the paper and acknowledged in the authors' README. The paper names the small network a Dual-Memory Video Transformer of the authors' design; in code its class is FlashVSRConditionNet, described as "FlashDecoder-style streaming VSR with configurable persistent conditions" (checkpoint flash.safetensors).
• Hugging Face Diffusers: diffusers (pinned at 0.37.1), acknowledged by the authors.
• Blender Foundation: Tears of Steel (CC) Blender Foundation | mango.blender.org, CC BY 3.0, our test clip.
• AI Search: where we spotted it (AI NEWS, 2026-11-08, chapter at 02:32).
• Our part: the custom backend, the deployment through the platform, the timings, the memory ladder, the visual checks, the SDK example and the demo video.

Model and method

• kopper/RelayVSR: sha 751954dc8adf34aa1d636a39fc548b9089703c85, lastModified 2026-11-13T05:04:59Z, not gated, apache-2.0, base model Wan-AI/Wan2.2-TI2V-5B. Files: wan.safetensors 10,066,399,024 B (reference generator adapted from Wan2.2-TI2V-5B, prompt embeddings included) and flash.safetensors 127,759,784 B (the lightweight VSR network; the authors' default is "S, step 60000, EMA").
• Mechanism, "Sparse Generative Relay": the large model produces reference latents only for sparse keyframes; a lightweight "Dual-Memory Video Transformer" super-resolves every frame from those references and the low-resolution video.
• In code, CollaborativeFrameStream (refvsr_inference/runtime/streaming.py) runs the Wan generator once per endpoint (one step at timestep 1000, guidance 1.0) on the first and last frame of each window, with overlapping endpoints. The small network decodes each frame of the window conditioned on both endpoint latents (dual-endpoint), with a two-frame video attention window (current frame and one predecessor). A 72-frame clip gets 6 references with window 16 and 4 with window 32.
• Paper Table 5 (the "32 million parameters" of the narrative are trainable parameters): Wan2.2-TI2V-5B with a rank-512 LoRA (1.32B trainable parameters, LoRA plus LR projection). Small network: 10 transformer blocks of width 512, video attention window of 2 frames (24 x 24 spatially), 31.9M trainable parameters.
• Training of the large model ends with Video-Aware Reference Optimization (VARO), a reinforcement-learning step with two rewards: a system-level reward on the video produced by the fixed small network, and a reference-level reward on decoded keyframe quality.
• Wan2.2-TI2V-5B card: 16 x 16 x 4 compression VAE, 720p at 24 fps, single consumer GPU such as a 4090 (at least 24 GB with offload flags).
• Authors' flags: --window-size default 16 ("Reference interval + 1"; it counts both ends, so keyframes fall every 15 frames, and window 32 gives one every 31; the README says to raise it "to let the small model process more intermediate frames"), --dtype bf16|fp16|fp32 (default bf16), --max-frames, --fps, --seed 42, --start-frame, --cpu-threads 8. Output is exactly 4x the input, FPS preserved, audio not copied. Pins: torch 2.10.0 (cu128 extra), diffusers 0.37.1, accelerate 1.15.0, triton 3.6, av. Python 3.11 to 3.13, Linux, NVIDIA GPU. No offload or low-VRAM switch.

Authors' numbers (paper Table 8, 201 frames, 4x, one A100 80 GB, interval 15; not reproduced by us except where noted):

Baselines at 1080p in the same table (fps / GB): SwiftVR 23.48 / 31.36, FlashVSR 7.80 / 24.45, Stream-DiffVSR 0.85 / 25.92, SeedVR2-3B 0.79 / 73.89, DOVE 0.56 / 41.08, SparkVSR 0.56 / 41.30, RealBasicVSR 18.50 / 24.81, RealViformer 14.87 / 12.36.

Table 9 (A100, dual-endpoint) by keyframe interval 5 / 10 / 15 / 20 / 25: 1080p 18.10 / 25.60 / 29.29 / 32.20 / 34.13 fps; 2160p 4.91 / 7.12 / 8.23 / 9.17 / 9.69 fps.

The two heavier upscalers named in the narrative are SeedVR2-3B and DOVE.

Table 11 (good-same-bad user study, positive favors RelayVSR), overall quality: +26.7 vs SwiftVR, -3.3 vs FlashVSR (near-neutral), +46.8 vs SeedVR2, +53.6 vs DOVE. Against DOVE, content fidelity -8.9 and temporal stability -8.7: DOVE is the one rated better on those two.

Setup and why

• GPUs: one RTX 4090 24 GB on our cloud (InferenceKey Cloud) for the main run and the memory ladder (24,564 MiB, 450 W). Then a real RTX 3090 24 GB on our cloud (24,576 MiB, 350 W) as a second consumer card. Both are the standard 24 GB variants. A 16 GB card was the goal for a third rung, but every 16 GB option we tried returned "no stock" (RTX 4080 SUPER, RTX 4080, RTX 5080, RTX 2000 Ada, RTX A4000, V100 16 GB), and the 20 GB RTX 4000 Ada and RTX A4500 returned the same.
• Backend: a new custom Python backend, relayvsr-video-upscale 0.1.0 (RelayVSRBackend), CPython 3.12. setup() fetches the authors' source tarball at commit e856773350f53036d825404c9ea9228e6dffadca (only refvsr_inference/, the demo clip and LICENSE), downloads both checkpoints at the pinned revision, builds load_checkpoints + WanVSRPipeline exactly as inference.py, and runs a 17-frame 128 x 224 warm-up. process() decodes the clip, streams it through CollaborativeFrameStream, sums the authors' DeviceSpan timings per stage, encodes H.264 at CRF 18 and raises the CRF (18, 22, 26, 30, 34) until the base64 fits 7.5 MB. Why custom: no built-in backend serves a video-in, video-out model. Why the authors' code: their numbers are measured on it, and it is the only path that runs the two models together.
• Transport: the SDK has no video task type for custom backends, so the package and workload are tagged text2image and the video travels as base64 JSON on the chat route (/v1/chat/completions).
• Stack: a local InferenceKey manager, nats-proxy and worker built from main. Production was never touched.
• Weights on the worker: pre-downloaded to the worker's local disk (8.6 s on the 4090 machine, 83.5 s on the 3090 machine) while the worker built the venv.

Step by step with the SDK example

The example is commit c06208ce on inferencekey-sdk main: backend.py, publish.py, main.py, requirements.txt, README.md, .env.example.

You need an SDK token (ik_sdk_), an API key (ik_live_), the project slug, the id of a private NVIDIA worker assigned to the project (IK_WORKER_ID), and your tenant id plus a dashboard session token for publish.py and to turn on the chat route (or do both in the dashboard). The worker needs about 25 GB of disk (venv about 7 GB, weights 10.2 GB) and gcc (Triton and FlexAttention compile kernels on first use).

The SDK is not on PyPI yet, so the client uses the local build in the repo (examples/CONTRIBUTING.md, section 4). In .env.example, INFERENCEKEY_TENANT_ID and INFERENCEKEY_USER_TOKEN are commented out: uncomment and fill them for publish.py and the chat route, or do both steps in the dashboard.

git clone https://github.com/inferencekey/inferencekey-sdk.git
cd inferencekey-sdk/examples/relayvsr-video-upscale-custom-backend-private-nvidia
python -m venv .venv && . .venv/bin/activate
pip install maturin requests
( cd ../../bindings/python && maturin develop --release )   # local SDK build into this venv
cp .env.example .env         # INFERENCEKEY_SDK_TOKEN, INFERENCEKEY_API_KEY,
                             # INFERENCEKEY_PROJECT, IK_WORKER_ID, and uncomment
                             # INFERENCEKEY_TENANT_ID, INFERENCEKEY_USER_TOKEN
python publish.py            # once: packages backend.py and uploads it to your tenant
python main.py my-clip.mp4   # or no argument: the authors' 360p demo clip

main.py calls ensure() for a fixed workload on IK_WORKER_ID (ready_timeout_secs 2400, request_timeout_secs 900), turns on the chat endpoint, waits with wait_until_ready(2400), upscales the clip with window 16 and window 32 and writes outputs/x4-w16.mp4 and outputs/x4-w32.mp4. It deletes the workload on exit unless IK_KEEP_ALIVE=1. RVSR_CLIP_FRAMES (default 72) sets the frames per call.

A request is JSON in the last user message:

{"video_b64": "<base64 MP4, up to 5 MB>", "window_size": 16, "max_frames": 72, "seed": 42}

video_url (https) or "demo": true work instead of video_b64. window_size must be 2 to 64. Input frames up to 960 x 544 (RVSR_MAX_PIXELS), up to 241 frames (RVSR_MAX_FRAMES). The answer's message content is JSON: video_b64 (H.264, no audio), width, height, frames, fps, keyframes, crf and timings (model_fps, wan_ms, flash_ms, first_frame_ms, wall_s, peak_vram_mib). The endpoint is POST /endpoint/<project>/relayvsr/v1/chat/completions.

Backend settings, passed through the workload env: RVSR_VRAM_LIMIT_GIB (caps the backend's GPU memory with torch.cuda.set_per_process_memory_fraction), RVSR_DTYPE, RVSR_MAX_FRAMES, RVSR_MAX_PIXELS, RVSR_OUTPUT_DIR (keeps a CRF 18 copy on the worker) and RVSR_WARMUP.

Full measurements

Time to ready:

• RTX 4090: "ready after 71.2 s" from the SDK. Backend log: download 20.1s, load 8.4s, warmup 4.2s, weights in VRAM 9.62 GiB.
• RTX 3090: "ready after 165.8 s" (venv about 2 min on that host).

Through the endpoint, warm calls, seed 42. Model fps = frames / model time (GPU time of both models, the authors' measure). Round trip from the HTTP client to the local manager.

• Peak reserved memory: 21,408 MiB (1920p), 16,112 MiB (1440p), 13,520 MiB (1080p). Card peak in nvidia-smi (1 s sampling): 21,900 MiB on the 4090, 21,750 MiB on the 3090.
• First call at a new frame size (Triton and FlexAttention compile for the geometry), 4090: 4.71 vs 7.33 fps at 1920p, 8.88 vs 23.85 fps at 1080p. 3090: 2.59 fps at 1920p on the first call.
• Where the 1920p round trip goes (4090, window 16): decode 2.1 s, streaming loop with CPU H.264 CRF 18 encode 15.0 s (model 9.8 s of it), CRF 22 re-encode 22.0 s because the CRF 18 file was above the 7.5 MB base64 limit; backend wall 40.1 s. The 1920p clips came back at CRF 22, about 4.9 MB.
• 1080p against the authors: 23.85 fps (window 16) on the 4090 against their 29.29 fps on an A100 80 GB; window 32 gives 29.1 on the 4090. Their 1080p peak is 13.82 GB, ours 13,364 MiB (14.0 GB).

VRAM ladder. Emulated rungs ran on the 4090 after deleting the workload, in the backend venv, one process per rung, RVSR_VRAM_LIMIT_GIB mapped to torch.cuda.set_per_process_memory_fraction, same clips, seed 42.

Not tried: a real 16 GB or 12 GB card (no stock), 12 GB with any offload (the code has none), fp8 or int8, the first-keyframe mode, clips longer than 72 frames. A real 16 GB card adds driver and context overhead that the cap does not model, so the 16 GB rung is a budget, not a card.

Test clips and quality notes (visual inspection, one seed, not a benchmark)

• Every narrative number is a warm call (after one warm-up clip at that frame size) with seed 42.
• Clips: (a) 3 s of Tears of Steel from 00:02:09, cropped to 950 x 534, scaled to 854 x 480, x264 CRF 28, 72 frames at 24 fps; (b) the same clip at 480 x 270; (c) the authors' demo assets/demo/input.mp4 (624 x 360, 16 fps), first 48 frames.
• 480p to 1920p, window 16: face, beard, hair strands and the plaid jacket clearly sharper than bicubic 4x; the stone balustrade gains texture. No obvious artifacts in the frames we checked (36, 38, 47).
• Window 32 vs 16 at frame 47 (window 16 reference at 45, window 32 references at 31 and 62): window 32 visibly soft on the face during the head turn. Frame 38, 7 frames from a reference in both: alike.
• No quantitative quality metric of our own, no audio. We checked still frames, not flicker across keyframe boundaries in motion, and bicubic was the only baseline (no Real-ESRGAN or RealBasicVSR on the same card).
• Units: memory figures are MiB or GiB (1 GiB is about 1.07 GB); the 20.4 GiB peak is 20,880 MiB, about 21.9 GB.

Full RelayVSR output, 3416 x 1920, window 16, 72 frames, CRF 22 as returned by the endpoint:

Demo video

A narrated 3-minute walk-through (188.5 s): what RelayVSR is, with the authors' examples and figures credited; the panel serving it on the RTX 4090; our speed numbers, the key comparison and the memory ladder.

Remaining problems

1. Round trip dominated by CPU encoding: 26 to 41 s for a 72-frame 1920p clip against 8 to 10 s of GPU time, including a second encode at CRF 22 to fit the 8 MB message. Status: open, a design limit of returning video inline; an object-store output URL would avoid the re-encode. Send 2 to 3 second chunks for long videos.
2. No video task type for custom backends: the backend is tagged text2image and the video rides the chat route as base64 JSON. Status: open in the SDK, documented in the example.
3. Shared local object store full: publishing the backend to our dev VM's shared MinIO returned put custom backend to S3 ... (XMinioStorageFull) because the VM's root disk was at 100 %. Status: worked around for the test with a private MinIO on /dev/shm; the dev VM disk still needs attention. It does not affect the hosted service.
4. Demo recorder: the panel recorder's Page.goto to /login timed out three times (Timeout 30000ms exceeded). Status: fixed with a storyboard start_url of about:blank.
5. Minor: the worker logged NATS connection established twice at start, harmless.

Sources

• Model card
• Code (commit e856773350f53036d825404c9ea9228e6dffadca)
• Paper, arXiv 2609.37850
• Project page
• Base model, Wan2.2-TI2V-5B
• FlashVSR, a baseline in the paper, acknowledged by the authors
• Where we spotted it, AI Search
• SDK example