Carbon-X sorts DNA variants on one gaming card, but check its reasons
Carbon-X called 75% of 40 ClinVar variants right on a 24 GB RTX 4090 with 8-bit weights, at 2.5 s per answer. Its explanations often missed.
A DNA model that gets the verdict right and the reasoning wrong is still useful, as long as you know which half to trust. That is where our test left Carbon-X, Hugging Face's new open model that reads raw DNA inside an ordinary chat and answers in English. On a 24 GB RTX 4090 on our cloud, with its weights shrunk to 8 bits, it called 75% of 40 known variants from ClinVar harmful or harmless correctly. The authors report 74.9%. A question on two 4,096-letter sequences came back in about 2.5 seconds. If you deploy models for a lab or a genomics team, this is a DNA specialist that runs on stock vLLM with one command line, on one card you control, so unpublished sequences never leave your machine. Use its labels as a first pass over a long list. Do not trust the explanations it writes around them.
Same question, two cards, one run each. Both said pathogenic. Only one pointed at the right letter, and neither named the gene.
A 14B chat model that learned to read DNA
Carbon-X is Qwen3-14B, the open chat model from the Qwen team, retrained on DNA and biology text by a group from Hugging Face, Zhongguancun Academy and TIGEM / University of Naples Federico II. They published the weights under Apache-2.0, not gated, together with their training and eval recipes. The credit for the model is theirs. Our part was serving it through our platform, building a public test set and measuring it.
The one new piece is how it cuts DNA into tokens, the chunks a language model reads. Carbon-X adds 5,515 DNA tokens to Qwen's vocabulary, and most of them are six-letter blocks of A, C, G and T. One token covers six bases, the way a guitarist reads a six-note run as one shape. That is why long sequences fit: our two 4,096-letter sequences plus the question came to 1,445 tokens. Paste a sequence into a sentence and the model marks it as DNA on its own.
The authors' headline is an average of 66.7% across five DNA benchmarks, against 67.5% for GPT-6.1 Sol and 60.9% for Opus 5.5, at a small fraction of the cost by their own pricing. We did not reproduce that average: their evaluation data is not public, so we built our own 120-question set from public sources.
What we found: the labels match the authors
Forty of our questions were real ClinVar variants in protein-coding genes, 20 harmless and 20 harmful, from the same test set the authors use. The model saw only the two sequences and our question, never the gene name.
| ClinVar variants (40) | Speed, one request | |
|---|---|---|
| RTX A6000 48 GB, 16-bit weights | 80.0% | 19.5 tokens/s |
| RTX 4090 24 GB, 8-bit weights | 75.0% | 50.5 tokens/s |
| Authors (all 1,018 variants) | 74.9% |
Both rows are with the model's randomness switched off. The gap between the cards is two variants out of 40, inside the noise, and the two agreed on 36 of 40. The smaller card was also faster, because the 8-bit weights do most of the work: a full variant question took 2.4 to 2.6 seconds on the 4090, against 7.1 to 7.5 on the 48 GB card in 16-bit.
The errors lean one way. On the 48 GB card it caught 19 of the 20 harmful variants but flagged 8 of the 20 harmless ones as harmful. On a long real list, that is a lot of harmless cases landing on a human's desk.
The reasons are where it breaks
The explanations around the labels are often wrong, even when the label is right. On one real variant where a T became a C, it gave the right label with the rationale "The variant changes C to C". In the authors' own sickle-cell example, the 48 GB card named the wrong base in 12 of 20 tries. Over 20 tries on each card it said pathogenic 16 times and benign 4 times, and with randomness off the 48 GB card said benign. The authors warn that the model "can miscount bases, confuse coordinates". It is not a clinical tool, and they say so.
Who should care
- Teams sorting many variants by likely harm. It is a usable first pass, but sample more than once and keep a human on the flagged list.
- Labs whose sequences cannot leave the building. Apache-2.0 weights on one 24 GB card you run yourself.
- Anyone who needs the why. Compute positions and gene names in code; do not take the model's word for them.
- Teams on 16 GB cards. Skip it for now: none of the vLLM setups we tried at 16 GB worked.
On InferenceKey, a cold 48 GB worker was serving Carbon-X in 356 seconds, and the 4090 in 262.5 seconds. Most of that was installing vLLM and warming up the engine, not the 30 GB download. One rough edge: answers with DNA come back wrapped in literal <dna> tags, so strip them in your parser. Status: open.
The honest limit: 40 variants is a small sample, so read our numbers as "matches the authors or not", not as a ranking.
Read the full test
The Research post has the GenomeQA and next-bases results, the GPU memory ladder, the exact vLLM command and the SDK example that asks the authors' questions on your own card. Read the full Carbon-X test in the Research post.