MemT-1.0-Nano keeps a chat's memory in a 41.9 MB file, and it answered 9 of 9 questions after a reload

Model and files

• MemTensor/MemT-1.0-Nano, sha 30fe0b22422c4cc4c87475f98db5c8baa5c1a902, created 2026-10-15, last modified 2026-10-20T05:49:52Z, not gated, Apache-2.0. 26 safetensors shards plus memt_block.safetensors, configuration_memt.py, modeling_memt.py and tech_report.pdf. 70.3 GB downloaded. 18,765 downloads and 123 likes on 2026-10-26.
• Card: 36B total, 3B active, 451M trainable memory parameters, 10M dynamic memory state. library: transformers, tag custom_code.
• config.json: model_type memt_qwen3_5_moe, MemTQwen3_5MoeForCausalLM. Backbone qwen3_5_moe: 40 layers with full_attention_interval 4 (10 full attention, 30 linear attention, GatedDeltaNet), hidden 2048, 256 experts with 8 per token, 16 attention heads, 2 KV heads, head_dim 256, vocab 248,320, max_position_embeddings 262,144. memt block: gdn_version stable_gdn2, memory_update_mode token, num_memory_heads 16, chunk_tokens 2048.
• Release: paper on 19 October 2026, weights and code by 20 October.
• Sibling MemTensor/MemT-1.0-Flash: DeepSeek-V4-Flash backbone, 288B total, 17B active, 3.61B trainable memory parameters, 336M dynamic memory state, HF license tag mit.
• Licensing: weights Apache-2.0 (HF card). All original code in MemTensor/MemT-1.0 on GitHub, including memt-sglang, is PolyForm Noncommercial 1.0.0 ("Commercial use of the repository software is not permitted"). The paper is CC BY-NC-SA 4.0.

Session API (modeling_memt.py at the pinned sha)

• chat(query, *, memory=None, enable_memory=False, enable_thinking=None, return_dict=False). With enable_memory=True the turn's plaintext history is committed into the memory state and dropped. Other turns stay as plaintext history in the session.
• save_state(path, *, overwrite=False) refuses to overwrite an existing file by default, flushes plaintext turns, and writes FP32 tensors as safetensors with metadata format memt-memory-v1 plus an identity hash (config, chat template, vocab).
• Sessions are guarded by a threading.RLock.
• The state is a dict of 10 FP32 tensors [1, 16, 256, 256], one per full-attention layer: 10 x 16 x 256 x 256 x 4 B = 41,943,040 B. Measured file: 41,944,024 B, so about 984 B of header and metadata.

Condensed from the card's Quick Start (Python 3.11+, PyTorch 2.11.0 built for CUDA 13, pip install transformers==5.12.1 accelerate==1.15.0 safetensors==0.8.0):

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "MemTensor/MemT-1.0-Nano"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="auto",
    attn_implementation="sdpa",
)

session = model.create_session(tokenizer)
session.chat("<facts to remember>", enable_memory=True, enable_thinking=False)
session.save_state("memmy_memory.safetensors")
session.close()

restored = model.create_session(tokenizer, state="memmy_memory.safetensors")
print(len(restored.messages))  # 0

Storage arithmetic (ours, not measured)

Plain KV cache of this backbone: 10 full-attention layers x 2 KV heads x 256 x 2 (K and V) x 2 B (bf16) = 20,480 B per token. The 41.9 MB state equals the KV of about 2,048 tokens. At 131,072 tokens the KV is 2.68 GB, 64x the state (the card and paper say about 66x, as a logical per-trajectory size that excludes shared weights). Below about 2,000 tokens of history the file is larger than the KV it replaces. The linear-attention (GatedDeltaNet) states exist in both cases and are not counted. The paper also notes that DenseRAG storage is cheaper than Nano's fixed state at short histories and only overtakes it as records accumulate.

Our setup

• One NVIDIA A100-SXM4-80GB on our cloud, driver 595.91.07, CUDA 13.2, 170 GB container disk (two other A100 configurations had no stock at the time). Weights on the container disk, never a network volume.
• A local InferenceKey manager built from main, connected to the worker over tunnels. Production was never touched.
• Custom Python backend memt-1-0-nano 0.1.1: Transformers 5.12.1 with trust_remote_code, PyTorch 2.11 (CUDA 13). setup() downloads the pinned revision with snapshot_download, then loads by repo id plus revision with dtype=torch.bfloat16, device_map={"": 0} (whole model on GPU 0, no silent CPU spill) and attn_implementation="sdpa". Greedy decoding (do_sample=False), max_new_tokens 256 by default.
• task_type text2text, process() only. Operations travel in extra_body.memt = {"op": "remember" | "ask" | "save" | "close" | "load" | "status", "session": ..., "text" | "name": ...} on /v1/chat/completions. Plain chat text also works (remember: ..., save: <name>, forget, load: <name>, anything else is an ask). Each answer returns JSON with the reply, token counts, seconds, tokens/s, memory_updated, plaintext_messages_kept, file size and sha256, and VRAM.

Deploy step by step

From the SDK example (commit ef656124 on main):

git clone https://github.com/inferencekey/inferencekey-sdk
cd inferencekey-sdk/examples/memt-nano-memory-state-custom-backend-private-nvidia
cp .env.example .env          # INFERENCEKEY_SDK_TOKEN, INFERENCEKEY_API_KEY,
                              # INFERENCEKEY_PROJECT, IK_WORKER_ID, and for the
                              # upload INFERENCEKEY_TENANT_ID, INFERENCEKEY_USER_TOKEN
python -m venv .venv && . .venv/bin/activate
pip install maturin requests
( cd ../../bindings/python && maturin develop --release )
python publish.py             # once: packages backend.py; uploads only if TENANT_ID and USER_TOKEN are set
python main.py                # deploy, remember x2, save, close, load, ask x4, control ask

• Needs an NVIDIA driver for CUDA 13 (580+), an 80 GB card, about 90 GB of local disk, Python 3.11 to 3.13 on the worker.
• Optional env: MEMT_MAX_NEW_TOKENS (default 256), IK_KEEP_ALIVE=1 to keep the workload after the run, MEMT_LOAD_IN_8BIT=1 (bitsandbytes; see the ladder, it fails), MEMT_GPU_CAP_GIB (torch.cuda.set_per_process_memory_fraction, to emulate a smaller card), MEMT_REVISION.
• main.py writes every request, answer, token count and timing to outputs/memory-test.json.

The 65-second demo: remember, save, close, load and ask, from a terminal client against the worker.

Time to ready

Measurements

Memory test, run 2 (bf16; seconds are GPU time in the backend, the round trip adds 0.3 to 1.1 s):

Input tokens grow from 27 to 210 in the restored session because ask turns stay as plaintext history. main.py's first keyword check scored the school date as a miss because it wanted "9 september"; by reading it is right, and the committed check is looser.

Long-input test (bf16, MEMT_MAX_NEW_TOKENS=96): one remember of 1,467 input tokens, reply "Noted.", 4.02 s, plaintext kept 0. Save 0.66 s, 41,944,024 B. Load 1.35 s. Five of five recalled: fridge code 5832 (1.55 s), landlord Rui Matos with rent on the 3rd (2.01 s), policy PT-77-0912 (1.76 s), Silva cake on 14 November, three-tier lemon (2.45 s), backup technician Marta with a phone ending in 4410 (2.21 s).

Demo client: remember 58 tokens, "Noted." in 1.4 s; save 0.68 s; load 1.41 s; "BX-4471" in 2.0 s (21-token prompt); "Forno Azul bakery in Porto" in 1.7 s (58-token prompt).

Speed: about 9 to 10 tokens/s decode (128 tokens in 12.72 s) with PyTorch reference GatedDeltaNet kernels (flash-linear-attention and causal-conv1d not installed) and HF generate. Not tuned.

GPU memory, bf16: 65.4 GiB allocated after load, 65.45 to 65.55 GiB peak allocated during requests, 67,605 MiB on the card idle after load, 68,117 MiB card peak (nvidia-smi, 1 s samples). The narrative gives these as GiB (66.5 GiB peak; the 8-bit load asked for 63.67 GiB).

VRAM ladder

Why 8-bit barely shrinks: the 256 fused experts per layer are 3D parameters, not nn.Linear, so bitsandbytes leaves them in bf16 (about 2 GiB saved). Not tried: a real 48 GB card, H100, CPU offload, fp8, community quantizations (none from MemTensor at test time). Smallest that passed: one 80 GB A100 in bf16.

The authors' SGLang path (not tested)

• memt-sglang (import memt_sglang) in MemTensor/MemT-1.0: native session memory on SGLang 0.5.16. Pins PyTorch 2.11.0, Transformers 5.12.1, flashinfer-python[cu13] 0.6.14, sglang-kernel 0.4.5. PolyForm Noncommercial 1.0.0.
• inference/start_memt_nano.sh: CUDA_VISIBLE_DEVICES=0,1, --tp-size 2 --ep 1 --dp-size 1, --max-prefill-tokens 16384, --max-sessions-per-rank 6, --max-inflight-commits-per-rank 2, --min-kv-tokens 2048, --max-total-tokens 65536, --max-running-requests 8, --mem-fraction-static 0.60, --dtype bfloat16, triton attention, linear-attention and MoE backends, snapshots in ./memt-state, port 30000. The guide says these settings have "not been deeply optimized".
• Constraints: single-node eager execution, unquantized BF16 safetensors only; chunked prefill, radix cache, overlap scheduling and CUDA graphs must stay disabled; the launcher rejects trust_remote_code.
• Snapshots are server-side .safetensors under --state-directory. Routes: GET /memt/identity, GET /memt/capabilities, POST /memt/control, POST /generate with a MemT RID. Leases (TTL 1 to 3600 s, default 300), transactions and epochs; a server restart needs a new session plus a snapshot import. HF snapshots and server exports are interchangeable for the same model and tokenizer.
• Whether it fits Nano on one 80 GB card, and its throughput, are not documented and were not tested.
• We found no vLLM support for MemT. The memt-sglang launcher is the authors' own integration, not stock SGLang.

Paper figures (not reproduced)

arXiv 2610.26825, "MemT-1.0: Towards Memory-Native Evolution", MemTensor Team, v1 19 October 2026, v2 20 October. 34B training tokens, continued pre-training plus post-training, Gated DeltaNet-2 with a stable token-level memory update, Mixture-of-Storage.

• Table 2, 11 memory benchmarks, average: MemT-1.0-Flash 50.6, MemT-1.0-Nano 47.0, Temp-LoRA 38.1 (best other parametric method). Textual memory on the Qwen3.6-35B-A3B backbone: MemOS 68.5, Full Context 66.3, Mem0 66.0, OpenViking 65.4, DenseRAG 56.5, Graphiti 45.7. Nano on LongMemEval-S 32.6 against Full Context 76.8 and MemOS 80.8; on BEAM-1M 35.8 against MemOS 71.3.
• Table 3, general capabilities in the noise setting, delta against Qwen3.6-35B-A3B: MMLU-Pro -1.6, GPQA Diamond -1.5, IFEval +3.3, LongBench v2 -2.0, BigCodeBench-Hard +2.0, LiveCodeBench v6 -5.8, average -0.9.
• Section 6.5, HF inference on two GPUs, concurrency 1, BEAM histories: slower than full context at short histories, about equal at 32K, end-to-end speedups of about 1.20x at 64K, 1.86x at 128K and 3.99x at 256K. Query TTFT (time to the first word of the answer) about 104 to 112 ms against about 257 ms (128K) and 445 ms (256K) for full context.
• Sections 4.3 and 6.6: per-trajectory state about 33x smaller than full context at 64K and 66x at 128K.

Remaining problems and status

1. Loading from the HF cache snapshot path fails with trust_remote_code. from_pretrained(<snapshot dir>) raises FileNotFoundError: ... models--MemTensor--MemT-1.0-Nano/blobs/configuration_memt.py, because Transformers follows the modeling_memt.py symlink into blobs/ and resolves the relative import there. Status: fixed in backend 0.1.1 by loading by repo id plus revision.
2. 8-bit does not fit a smaller card (ladder above). Status: no workaround; needs an official quantization or expert-aware quantization.
3. Changing config.env on a running custom-backend workload does nothing. The manager sends reload_backend and the worker logs reload_backend handle_load_model failed: unknown backend memt-1-0-nano; the old backend keeps serving. Workaround: delete the workload and run main.py again. Status: open.
4. Manual GPU machine creation ignores the disk setting. The handler hard-codes DEFAULT_POD_DISK_GB = 100 and no longer reads RUNPOD_CONTAINER_DISK_GB. 70.3 GB of weights plus a torch venv is tight on 100 GB. Status: patched locally for this test only, open.
5. The dashboard chat tester cannot drive a process()-only backend (it streams). The demo used a terminal client. Status: known, open.
6. The state file cannot travel through the API (41.9 MB against an 8 MB result limit). Status: by design for now; it stays on the worker disk and the API returns size and sha256.
7. The tunnels between our test manager and the worker dropped once; a supervisor loop reconnected them without restarting the worker.

Limits

Nine recall questions plus one control on our own made-up facts (two more in the demo client), greedy decoding, requests one at a time. The HF session uses a per-session lock and the backend serializes requests, so multi-user throughput is unknown. One card type. No benchmark from the paper was reproduced, and our 64x is arithmetic, not a measurement. We did not measure reread time or per-session card memory against a full-context baseline. The hosted InferenceKey endpoint is untested: this ran on a local test manager.

Credits

MemTensor built MemT-1.0, trained the memory block, wrote the session API, the SGLang integration and the paper, and released the Nano weights under Apache-2.0. The Qwen team made Qwen3.6-35B-A3B, Nano's backbone. Hugging Face Transformers runs the authors' custom code. We spotted the model in the AI Search news video of 25 October, and r/LocalLLaMA had a thread on it on 22 October. Our part: the custom backend, the deployment on InferenceKey, the load fix, the memory tests and the measurements.

Sources

• MemTensor/MemT-1.0-Nano model card, config.json, modeling_memt.py at 30fe0b22
• MemT-1 collection
• MemTensor/MemT-1.0 on GitHub, start_memt_nano.sh
• Paper, arXiv 2610.26825
• AI Search news video, 2026-10-25, MemT 1.0 chapter from 38:33
• SDK example, commit ef656124