Pocket TTS in Castilian Spanish: a voice that runs on CPU

We needed a voice for narrations, demo videos and product walkthroughs, in English and in Spanish. We did not want to rent a GPU for it: a voice-over is a few minutes of audio, and a GPU sitting idle between two narrations is exactly the kind of waste we talk about elsewhere on this site. So we picked Pocket TTS from Kyutai, a text-to-speech model built to run on a CPU, and fine-tuned its Spanish model so it speaks the Spanish of Spain.

The weights are on Hugging Face: inferencekey/pocket-tts-spanish-24l-castilian.

Why Pocket TTS

• It runs on a CPU. No CUDA, no ROCm, no GPU at all. It installs with pip and a CPU build of PyTorch.
• It is fast enough. The English model reads text about three times faster than real time on an eight-core desktop CPU (numbers below).
• It speaks several languages. Kyutai publishes models for English, Spanish, French, German, Italian, Portuguese and Dutch, each in a small 6-layer version and a larger 24-layer one.
• It clones a voice from a short clip. We use the same voice (alba) in every language.
• It has a simple server. pocket-tts serve keeps the model in memory and answers HTTP requests, so we run a small pool of servers and send one sentence to each.

The problem: Spanish with a foreign accent

English sounds natural out of the box. Spanish was understandable, but it did not sound like Spain: it reads with a foreign accent, and it pronounces the Castilian "z" and soft "c" as an "s" (the seseo of Latin American Spanish). For an audience in Spain that is the first thing you notice.

So we fine-tuned the 24-layer Spanish model (spanish_24l, about 316 million parameters in the part that generates speech) on about 2,000 clips of Castilian Spanish speech, on one AMD GPU with ROCm.

What we learned from the run:

• Fewer steps were better. Validation loss reached its lowest point at step 600 (0.4951) and then went up again, to 0.5095 at step 1500. At step 1500 the model had seen the data about 47 times and started to drop or invent words: the ASR found complete sentences in only 3 of 6 test clips. Step 600 got all 6 right, like the base model.
• We publish the EMA weights of step 600. EMA is a moving average of the weights during training. It changes the model less than the raw weights, and that is what we wanted: keep everything the base model does well and fix the accent.
• An ASR is not enough to judge the accent. Qwen3-ASR writes "caza" and "casa" the same way, so it tells you whether words are missing, not how they sound. We checked every sample with the ASR for missing words and by ear for the accent.

Listen

Every sample uses the same voice (alba), unedited, generated on CPU. Every clip passed an ASR check for missing words.

Spanish: base model vs our fine-tune

"¿Sabías que el tren de las siete y cuarto sale hoy con veinte minutos de retraso?"

• Base model spanish_24l
• Our fine-tune

"La panadería de la esquina abre a las ocho y hace el mejor pan de centeno del barrio."

• Base model spanish_24l
• Our fine-tune

English

• "Did you know the seven fifteen train is running twenty minutes late today?"
• "This voice runs on a regular CPU, with no graphics card at all, faster than real time."

French

• "Tu savais que le train de sept heures et quart a vingt minutes de retard aujourd'hui ?"
• "La boulangerie du coin ouvre à huit heures et fait le meilleur pain de seigle du quartier."

German

• "Wusstest du, dass der Zug um Viertel nach sieben heute zwanzig Minuten Verspätung hat?"
• "Die Bäckerei an der Ecke öffnet um acht und backt das beste Roggenbrot im Viertel."

English, French and German use Kyutai's own 6-layer models. Spanish uses our fine-tune. With those four, we cover what we need: English and Spanish for our own videos, and French and German for the other languages of this site.

One tip from the French run: the model sometimes misreads times. "Sept heures et quart" came out wrong in two of three tries, and the ASR caught it. If a sentence matters, check it and generate it again.

How fast on CPU

Measured on our development machine, a shared VM on a Ryzen 7 5700X (8 cores), with 3 server instances of 2 threads each and six everyday sentences per model:

The fine-tune costs the same as the base 24-layer model: same architecture, different weights. The 24-layer models are about three times slower than the 6-layer ones, so on a CPU they are fine for narrations, not for long audiobooks. Each instance of the 24-layer model takes about 2.1 GB of RAM.

Use it

Install Pocket TTS and point it at our config on Hugging Face:

pip install pocket-tts
pocket-tts generate \
  --config hf://inferencekey/pocket-tts-spanish-24l-castilian/config.yaml \
  --text "¿Sabías que el tren de las siete y cuarto sale hoy con veinte minutos de retraso?" \
  --output-path tren.wav

Without --voice it uses alba. To keep it in memory and call it over HTTP:

pocket-tts serve --config hf://inferencekey/pocket-tts-spanish-24l-castilian/config.yaml

Two things that made our Spanish narrations better:

• Send whole sentences, or whole short paragraphs. Every chunk you send ends with its own closing intonation, so cutting a paragraph into many small pieces makes it sound choppy.
• Write numbers as words. "Siete y cuarto" is read reliably; "7:15" is not always.

The model card on Hugging Face has the Python example and more samples (base vs fine-tune).