AstaBrief-8B: a cited research report from your own sources, on one 24 GB GPU
Ai2's AstaBrief-8B turns a question and your own excerpts into a cited report. We ran it on one 24 GB GPU and counted every citation.
"Note that the current year is 2025." That line sits in the middle of a one-page text file that the Allen Institute for AI (Ai2) published next to its new model. The file is the exact set of instructions AstaBrief-8B was trained on, and the model card is blunt about it: use a different prompt and you may get "degraded or inconsistent behavior". So you keep the file as it is, the 2025 included, and you fill in two blanks: your question, and the passages from papers or documents that your own search found. Out comes a report in sections, with a short summary per section and a reference after each claim. Will it think it is still 2025? In our reports it never claimed the wrong year, though we did not test whether it treats 2025 papers as the newest work there is.
The size is not new, since an 8-billion-parameter model has fit on a 24 GB card for a while; the training is. AstaBrief is an open model of that size trained for one job, the writing step of "deep research" (the AI tools that read dozens of sources and hand you a report), with one fixed prompt, and Ai2 published how it scores against its own slower multi-step system. We ran it on a single 24 GB NVIDIA card and counted every reference it wrote. Here is what you take away:
| What | Result |
|---|---|
| Model | AstaBrief-8B by Ai2: Alibaba's open Qwen3-8B model (8 billion parameters), retrained to write cited reports. The weights can be downloaded freely and used commercially (Apache 2.0 licence). Ai2's recommended prompt is for non-commercial use only (see below). A 16.4 GB download |
| Ai2's grade on 100 real research questions | 87.0 out of 100, against 77.3 for the model it started from |
| GPU we used | one NVIDIA RTX A5000, a 24 GB card |
| Memory used | about 22 GB of 24 GB at peak |
| Speed for one report | about a minute (40 to 65 s), writing about 39 tokens a second (a token is roughly three quarters of a word) |
| Four reports at once | 85 s for all four, against 209 s one after another |
| References checked | 363 of 380 point to a passage we supplied (95.5%) |
| The catch | 8 of 11 reports also cite 1 to 3 sources that were not in the prompt |
An open-book exam with page numbers
The easiest way to picture what AstaBrief does is an open-book exam where every sentence must carry a page number. You hand the model the book: 15 short excerpts, each with a label made of an ID, the first author, the year and the citation count (how many other papers cite that paper). The model writes its answer and puts the label of the excerpt it used in square brackets after each claim. When it says something that is not in the book, the prompt tells it to mark it as "(LLM Memory)", the model's own recollection.
This split is what makes the output checkable. A label in brackets either matches an excerpt you gave it, or it does not. You do not need to understand the paper to run that check, you need a list of the labels you sent.
What Ai2 released
On 2 October Ai2 open-sourced AstaBrief, the model behind the "Fast mode" of the report feature in Asta, its research assistant. Next to it, Asta keeps a "Thinking mode" powered by Claude. The older pipeline works in stages: it summarizes the retrieved snippets, groups them, then writes. AstaBrief skips the middle and writes the whole report in one pass. Across the full Asta pipeline, Ai2 measured an average of 51.1 s per report in Fast mode against 178.5 s in Thinking mode.
Ai2 started from Qwen3-8B and showed it about 47,000 example reports written by its existing tools running on Claude and OpenAI models, for questions taken from real queries in Asta's logs. Then it showed it about 6,000 pairs of reports, one better and one worse, picked by two AI graders, and kept a pair only when both graders agreed. Ai2 says this was more stable and cheaper than the alternatives, and that the biggest gain came from throwing away example reports with few citations.
There is a licence split to know about before you build anything. The model can be downloaded without asking and used commercially (Apache 2.0), but the instructions file and the better/worse report pairs are for non-commercial use only (CC-BY-NC-4.0), and Ai2 describes the model as meant for research and education. Because the instructions file is non-commercial, we do not ship a copy of it in our code: our example script downloads it from Ai2 each time it runs. That does not change its terms. Whoever runs the script uses the prompt under Ai2's non-commercial licence.
Ai2 also makes the case for running it in house. Its blog says open weights let institutions run AstaBrief on their own infrastructure, "which is necessary when research questions reveal sensitive or unpublished work", and it released an example workflow that researchers can adapt to make reports from their own PDFs.
How good it is, by Ai2's numbers
Ai2 graded it on 100 computer science research questions written by real users. AstaBrief-8B gets 87.0 out of 100, against 77.3 for the plain Qwen3-8B, and the gain is mostly in the references. Ai2 checks whether each claim is fully supported by the sources the report gives for it: that holds for about 78 claims in 100 with AstaBrief, against about 65 for the plain model. And of the sources it gives, about 9 in 10 really support the claim, up from about 3 in 4. The one score that went down, very slightly, is Ai2's measure of how much of each answer stays on point: 90.6 before, 89.0 after.
In Ai2's side-by-side comparison with the report from its slower multi-step system, AstaBrief's report was preferred on about 7 of 10 test questions (and on just over half of a separate set of questions Ai2 used during development). That comparison was judged by an AI model, and the table in the model card does not say which one. Ai2's own word for the result is "competitive with". Another Ai2 model of the same size, DR-Tulu-8B, scores slightly higher on the same test, and on a second test AstaBrief comes last of the three, about 54 against 56 and 60. In a small study with three researchers, DR-Tulu was preferred overall while two of the three preferred AstaBrief on citation accuracy.
Ai2 also names a limit its scores do not catch: a model can cite the right study and still overstate what it found.
Running it on InferenceKey
The card and the software
To the software that runs it, AstaBrief-8B looks like any other Qwen3 model, so it runs on vLLM, the open-source serving program built into InferenceKey, with no custom code. We rented an NVIDIA RTX A5000 on RunPod, a 24 GB workstation card in the same memory class as a gaming RTX 4090. Our software needs a recent NVIDIA driver, so we only rented machines that had one. That is our requirement, not the model's.
The model takes 16.4 GB of the card's 24 GB. We let each request be up to about 16,000 tokens long (question, excerpts and report together), the length Ai2 used in its last round of training. That leaves room for two requests of that full length at once, or about four of the size we actually sent (roughly 7,000 tokens each, question plus report).
The steps
Our SDK example (a ready-made Python script using InferenceKey's library) does the whole thing in one script: it tells InferenceKey to run the model on your GPU machine, waits until the model is ready, builds Ai2's prompt from your question and excerpts, asks for one report, and shuts the model down at the end. The heart of it:
MODEL = "allenai/AstaBrief_8B"
COMMAND = f"vllm serve {MODEL} --max-model-len 16384 --dtype bfloat16"
spec = WorkloadSpec(
name="AstaBrief-8B cited reports (vLLM)",
slug="astabrief-8b",
model=MODEL,
backend=Backend.VLLM,
command=COMMAND,
task_type="text2text",
worker_id=os.environ["IK_WORKER_ID"],
execution_policy=ExecutionPolicy.FIXED,
execution_policy_config={"replicas": 1},
)
ref = mgmt.ensure(spec)
mgmt.wait_until_ready(ref.workload_slug, timeout=1800)
out = DataClient.from_env().endpoint(ref.workload_slug).generate_text(
messages=[{"role": "user", "content": build_prompt(query, references)}],
temperature=0.7, max_tokens=4096,
)
The script fetches Ai2's instructions and gives each of your excerpts the same kind of label the model saw in training: an ID, the author, the year and the citation count. Run python main.py for the six sample excerpts that come with it, or python main.py my_references.json for your own. The model answers in the same format as OpenAI's service, so code your team already wrote for OpenAI can talk to it by changing one address.
We ran the test from a private copy of InferenceKey, never the live service. To use real material, we asked 4 questions (three about making AI models faster and more reliable, one about microplastics and health) and gave each one the summaries of 15 papers from OpenAlex, a free catalogue of research papers.
What we measured
A report takes about a minute (41.5 to 64.7 s alone on the card). Starting the model on a freshly rented machine is a one-time wait of about four minutes (245.7 s), more than half of it (137 s) installing the serving software. Downloading the model took 18 s, and starting it took about another minute and a quarter. At its peak the card used about 22 of its 24 GB.
A report starts appearing a second and a half after you ask, and a typical report runs 1,600 to 2,400 tokens, roughly a few pages.
Can one card write several reports at once? Yes. Four reports sent together finished in 85 s, against 209 s one after another. Each report took a little longer on its own (52 to 85 s instead of 42 to 65 s), but all four were done in under a minute and a half. The reports were not identical from run to run, so read this as a fair picture rather than an exact timing.
Getting there needed one setting: how many requests the model may take at once (the chart calls this "concurrency"). Ours started at one, and on the first try the four reports queued: 198 s, and one report lost (see below).
Checking the page numbers
Back to the open-book exam. Across 11 reports (one of them cut at the 4,096-token cap), the model wrote 380 references, and 363 of them, 95.5%, point to an excerpt we supplied. It also used the book well: each report drew on 11 to 15 of its 15 sources.
The other 17 are the part to plan for. In 8 of the 11 reports the model cited 1 to 3 labels that were not in the prompt. Some are a supplied paper with a mangled label: one Thapliyal 2025 paper came back without the W at the start of its ID, with the last digit changed and a citation count of 111 instead of 45 (the number of papers that cite it). Others are papers from the model's memory, such as a Derraik 2002 paper with 2,594 citations, written in exactly the same label format as the real excerpts and not tagged "(LLM Memory)". The prompt's own mechanism for that tag was used once in 11 reports.
We also saw drift in details. One report got a paper's citation count wrong (how many other papers cite it): it wrote 25 instead of 27, seven times over. In the short sample run, the report turned "one eighth" of a cost into "1/4 cost", and added a section with no reference at all. That is exactly the limit Ai2 warned about: the right source next to a claim that is not quite what the source says.
The practical answer is cheap. Since every label you sent is known, a few lines of code can flag any bracket in the report that does not match one of them before a person reads it. Numbers that come from the excerpts deserve a second look by a human either way: checking the references stays your job.
The demo video
The video shows the model running on the RTX A5000 in the InferenceKey dashboard, Ai2's prompt filled with the six sample excerpts in the built-in test screen, and the report appearing section by section. With only the six short sample excerpts to read, instead of fifteen, the first words appeared in under a quarter of a second (230 ms), and the whole report took about a minute (62.46 s).
What went wrong
Two problems were ours, in the platform. The first has a workaround today; the second has no fix yet.
- A model tied to one GPU takes one request at a time. When our SDK pins a model to one specific GPU machine, the platform lets that model take only one request at a time. Added any other way, it takes up to 20 by default. That is why the four reports above queued. It is our bug, and one settings call fixes it today (details in "For engineers").
- A long wait ends in an empty answer. A request that waits more than 3 minutes for its first word is closed with an empty answer and no error, while the GPU goes on writing the report for nobody (107 s more in our case). It should return a clear error and cancel the job. We have no fix for this yet.
The rest belongs to the model or the rental market. The output needs light cleaning before anyone reads it. Every report starts with an empty "thinking" tag, as Qwen3 models write when thinking is on (we did not test switching it off), and the section summaries are marked slightly differently from what the prompt asks. Both are easy to strip with a few lines of code. Switching our source labels to plain numbers stopped the model dropping a letter from them, but it still wrote wrong labels for two supplied papers, so the label check is not optional. Our run notes also record a scrap of the training data's own markup in the middle of one report; we did not recheck it, but a simple clean-up filter is worth adding.
Who did what
Ai2's Asta team built and released AstaBrief: the training data from Asta's own logs and pipelines, the two training rounds with their own open-source training code (open-instruct) on 8 of NVIDIA's large data-centre GPUs (H100), the evaluation, and the prompt. Their training reports came from pipelines running on models by Anthropic and OpenAI, the competing reports of the second round also came from DeepSeek models, and GPT-4.1 and DeepSeek-R1 served as graders. Ai2 also published the example workflow for reports from your own PDFs, and made the argument for running the model on your own infrastructure. The Qwen team trained Qwen3-8B, the base everything sits on. The vLLM project serves it with no changes. Hugging Face hosts the weights and the prompt, OpenAlex gave us the abstracts for real excerpts, and RunPod rented us the card. We spotted the release in an AI Search video.
Our part was putting it together on InferenceKey, writing the SDK example, running it end to end on one 24 GB card, measuring speed and memory, and counting every reference in the reports.
What to take away
If you already have a search step over your own documents, AstaBrief-8B is a report writer you can run next to it on one 24 GB card, about a minute per report and several at once. Our EmbeddingGemma 2 post covers that search side. As Ai2 itself argues, the reason to run this yourself is control: the question, the excerpts and the report stay on a machine you choose. For a company there is one condition attached. Ai2's recommended prompt is licensed for non-commercial use only, so a commercial product needs either a check of that licence or its own prompt in the same format, and the model card warns that a different prompt may make the model behave worse.
The limits, plainly. Our test was small: 4 questions, 15 abstracts each, 11 reports (one cut at the 4,096-token cap). We did not grade the reports or compare them with another model on our questions. The quality numbers above are Ai2's, and they come only from computer science questions. Three of our four questions were about AI and computing too, and only one (microplastics) was outside that field, so we can say nothing about other subjects. The inputs were short abstracts of public papers, not the internal company documents this is pitched for. Our 15 excerpts per question filled less than a third of the room we allowed (about 5,000 of 16,000 tokens), so we do not know how it behaves with much longer inputs.
We checked that each reference points to an excerpt we sent. We did not check that each sentence says what its excerpt says. So 95.5% means the references are real, not that the claims are right. The drift above is what we noticed along the way, not a line-by-line review.
And the model writes confident references that you have to check. One of the labels it wrote reads [4411306415 | Thapliyal et al. | 2025 | Citations: 111]. The excerpt we sent was [W4411306417 | Thapliyal et al. | 2025 | Citations: 45]. Would your readers notice?
For engineers
Model (model card, SFT checkpoint, Ai2 blog, config.json, DPO data, base Qwen3-8B). HF repo allenai/AstaBrief_8B, sha 6a34f54d..., last modified 2026-10-03, license:apache-2.0, not gated, 1,979 downloads last month and 66 likes when we fetched it. Lineage: Qwen3-8B-Base, Qwen3-8B, AstaBrief_8B_SFT (SFT on AstaBrief_SFT_Mix), AstaBrief_8B (offline DPO on AstaBrief_DPO_Mix), both stages in open-instruct on 8x H100. Qwen3ForCausalLM, 36 layers, hidden 4096, intermediate 12288, 32 query heads and 8 KV heads, head_dim 128, max_position_embeddings 40960, rope_theta 1,000,000, no rope_scaling, vocab 151936, bfloat16. generation_config.json: temperature 0.6, top_k 20, top_p 0.95. The card's example uses temperature 0.7, top_p 0.95, max_tokens 4096, stop_token_ids=[eos]. Four safetensors shards, 16,381,516,824 bytes, plus four duplicate pytorch_model-*.bin shards and only a pytorch_model.bin.index.json.
Training. SFT: 5 epochs, LR 5e-06, warmup 0.03, grad accumulation 4, max sequence length 32,768, BF16. DPO: 7 epochs, LR 5e-06 linear, warmup 0.1, DPO Norm loss, beta 10, per-device batch 1, grad accumulation 8, max sequence length 16,000, BF16. So the 16K figure is the DPO stage, not the whole training. Blog: about 90K queries from filtered Asta logs, 47K SFT examples from ScholarQA pipelines backed by Claude 3.5 Sonnet, Claude 3.7 Sonnet, o3, o4-mini and GPT-4.1, about 6K DPO pairs (one report from Claude-backed ScholarQA, the competing one from o3, o4-mini, DeepSeek-V3 or DeepSeek-R1, winner picked by the two judges, GPT-4.1 and DeepSeek-R1, kept only on agreement), 95% agreement with humans. The DPO dataset card says 6.6k instances, queries from OpenScholar and Asta ScholarQA until June 2025, CC BY-NC 4.0, with third-party model outputs subject to provider terms.
Ai2 evaluation. ScholarQA-CS2 test (avg / ingredient recall / answer precision / citation precision / citation recall): Qwen3-8B 77.3 / 77.8 / 90.6 / 76.2 / 64.6, AstaBrief-8B-SFT 83.7 / 85.2 / 90.4 / 87.7 / 71.3, AstaBrief-8B 87.0 / 90.2 / 89.0 / 90.5 / 78.2. Comparison table (SQA-CS2 dev / test / DeepScholarBench): Asta ScholarQA 87.6 / 86.2 / 60.25, DR-Tulu-8B 86.5 / 88.8 / 56.26 (win rate vs Asta SQA 36% dev, 54% test), AstaBrief-8B 86.3 / 87.0 / 53.50 (55% dev, 72% test). Blog: evaluations ran in 2025 against frontier models of that time, and the metrics do not measure whether a report keeps the scope and strength of its sources. Early usage among 374 Asta users: 29.1% used Fast mode on two or more days, 3.67 report threads on average, 23% never switched back to Thinking mode, positive feedback 84.2% Fast vs 85.2% Thinking (sparse).
Prompt. sft_prompt.txt, 3,154 bytes, dataset allenai/AstaBrief_prompts revision 355b7a260fce95820a2bdf359ca1ddec920ee974, CC-BY-NC-4.0 (it also holds pairwise_judge_prompt.txt and query_filter_prompt.txt). Placeholders [QUERY] and [SECTION_REFERENCES] inside a <section_references> block of "[ID]": "quote" pairs. Cite inline as [ID], own knowledge as (LLM Memory). Each section starts SECTION; plus a name, then TLDR; and a two-sentence summary without citations. Only LIST or SYNTHESIS formats. A citation count of 100 or more allows "notable". Training keys look like [230644543 | Miguel et al. | 2020 | Citations: 5].
Our setup. RunPod NVIDIA RTX A5000 24 GB, SECURE cloud (rents for about 0.27 USD an hour), image runpod/pytorch:1.0.7-cu1300-torch291-ubuntu2404-cluster. RTX 4090, RTX A5000 and RTX 3090 COMMUNITY answered 409 (out of stock). Driver 580.159.04 (CUDA 13.0), 230 W power limit, compute capability 8.6. Pods filtered with allowed_cuda_versions 13.0 to 13.4. Worker bootstrap 76 s after the pod POST. Local InferenceKey manager reached through HTTP and TCP tunnels, production never touched. Backend: built-in vllm, vLLM 0.20.0 (worker default), torch 2.11.0+cu130. Command vllm serve allenai/AstaBrief_8B --max-model-len 16384 --dtype bfloat16 (gpu-memory-utilization 0.92 is the effective value applied by the worker, not part of the command). The SDK script deletes the workload at the end unless IK_KEEP_ALIVE=1. SDK example commit 728d890 on inferencekey-sdk main (main.py, README.md, .env.example, requirements.txt, sample_references.json, .gitignore).
Time to ready. 245.7 s from ensure() to ready on a cold pod: vLLM install 137 s (venv 8.1 GB), weight download 17.94 s (safetensors only), weight load 2.82 s, model loading 15.27 GiB in 21.90 s, engine init 40.49 s (compilation 23.42 s), vLLM ready 91 s after start.
Memory and power. KV cache 4.6 GiB = 33,472 tokens, max concurrency 2.04x at 16,384 tokens per request. vLLM process 21,850 MiB, peak GPU total 22,069 MiB of 24,564. Peak power 230.07 W.
Per request (official prompt, 15 OpenAlex abstracts per question, temperature 0.7, max_tokens 4096, streaming through the local gateway, tokens counted with the model's tokenizer because the stream carried no usage even with stream_options.include_usage):
| Run | Prompt tok | Report tok | First token s | Total s | Decode tok/s |
|---|---|---|---|---|---|
| Sequential, 4 questions | 4,374 to 5,068 | 1,570 to 2,444 | 1.37 to 1.58 | 41.5 to 64.7 | 38.7 to 39.1 |
| 4 at once, concurrency 1 | 4,359 to 5,053 | 1,472, 1,926, 4,096 (hit max_tokens), 0 | 1.22, then 40.4 and 91.5 (queued) | 38.9 to 197.9, one empty at 180.0 | 38.5 to 39.0 |
| 4 at once, concurrency 8 | 4,359 to 5,053 | 1,707 to 2,874 | 0.37 to 0.74 (prefix caching) | 51.5 to 84.6 | 33.4 to 34.2 |
Our prompts (4.4K to 5.1K tokens) stayed well under the 16,384-token limit, and nothing was measured near it. Wall time for four reports: 209 s sequential, 198 s at concurrency 1 (3 of 4 delivered), 85 s at concurrency 8. vLLM log during the concurrency 8 run: Avg generation throughput 132.0 to 136.0 tokens/s with Running: 4 reqs. main.py with the 6 sample excerpts (non-streaming): 40.9 s, 1,171 prompt tokens, 1,563 report tokens, 8 sections, 6 of 6 references cited. Panel tester: first token 230 ms, total 62.46 s (its 63.6 tokens/s and 3,970 total tokens include the estimated prompt).
Citations. 11 reports (one, concurrency 1, cut at max_tokens 4096), 380 inline citations, 363 (95.5%) to a supplied excerpt. Per report, supplied/not supplied: 22/1, 14/1, 23/3, 41/5, 75/2, 50/2, 27/1, 31/0, 39/0, 19/2, 22/0. 8 of 11 reports cite keys not in the prompt: 17 marks, 11 distinct keys. Mangled copies of supplied papers: [4411306415 | Thapliyal et al. | 2025 | Citations: 111] for [W4411306417 | Thapliyal et al. | 2025 | Citations: 45], [4414281756 | Kadac-Czapska et al. | 2025 | Citations: 13] for [W4414281754 | Kadac-Czapska et al. | 2025 | Citations: 9] (both in the run with W-prefixed OpenAlex IDs: W dropped, last digit changed), and, abbreviated, Sheng et al. 2024 with 160 citations (key 269117055) and Jiang et al. 2023 with 20 citations (key 3664640) for the supplied FlexGen and LLMLingua papers. From memory, in the same key format and not tagged LLM Memory: Santhanam 2023 (163), Sharma 2023 (505), Jiang 2020 (1666), Daza 2021 (87), Sala-Newhausen 2020 (352), Derraik 2002 (2594), Antonini 2021 (102). Supplied sources used per report: 11 to 15 of 15. One [LLM Memory] citation in 11 reports.
Format. With OpenAlex IDs like W4402683901 the model often dropped the W when citing, so the later runs used numeric IDs. That stopped the dropped W, not the wrong keys: the numeric-ID runs still produced the Sheng 2024 and Jiang 2023 keys above for supplied papers. Every report starts with an empty <think>\n\n</think> block (Qwen3-8B documents that the block is always present, possibly empty, when thinking is enabled, and we did not test switching it off). 4 to 8 SECTION; blocks per report. 0 of 12 saved files contain TLDR; (one tester take showed it). Metadata drift: Nouri 2025 cited with 25 citations instead of 27 (seven times in one report), Chen et al./80 for a supplied Zixuan/25 key. The 6-excerpt sample turned "one eighth" into "1/4 cost" and "Up to 4x cheaper", and added an uncited "Operational considerations" section. Our run notes record a fragment [4]<Paper corpusId="2024" paperTitle="(2024)" isShortName></Paper> in the report of the final demo take. We did not recheck it frame by frame. Prompt line "Note that the current year is 2025": grep over the saved report files (12 bench reports plus 2 sample runs) found no sentence stating the current year or doing year arithmetic; 2025 and 2026 appear only as publication years of cited papers (in keys and in author-year mentions such as "Wang et al. (2025)"). Not checked for softer recency framing.
Platform issues. (1) ensure() with a worker_id creates the assignment with a literal max_in_flight of 1 (insert_assignment_with_version(..., wid, 1) in crates/manager/src/services/workloads.rs), while the assignments API default DEFAULT_MAX_IN_FLIGHT is 20. Workaround: PATCH /api/projects/<pid>/assignments/<workload>/<worker> with {"max_in_flight":8}, answered 204. (2) No workaround yet. STREAM_FIRST_TOKEN_TIMEOUT_MS = 180_000 in crates/manager/src/api/openai.rs: a queued stream ended at 180.02 s with HTTP 200, no content and no finish_reason, and the worker still completed that job in 106,767 ms. (3) Streams through the gateway carried no usage.
Test data. Semantic Scholar's API answered 429, so we used OpenAlex: works with abstracts, from 2019, more than 8 citations, 15 per question. Prompt sizes 4,359 to 5,068 tokens with the model's own tokenizer.
Video and cleanup. Demo video: 1920x1080, 131.4 s. Cleanup: workload and pod deleted (RunPod returns 404), tunnels, NATS, database and the empty MinIO bucket removed.
Sources: model card, SFT model card, Ai2 blog, prompt, DPO data, Qwen3-8B, SDK example, demo video.