PocketTTS CoreML

CoreML conversion of kyutai/pocket-tts for on-device inference on Apple platforms.

Models

Model	Description	Size
cond_step	KV cache prefill (voice + text conditioning)	~200MB
flowlm_step	Autoregressive generation (transformer_out + EOS)	~200MB
flow_decoder	Flow matching denoiser (8 Euler steps per frame)	~190MB
mimi_decoder	Streaming audio codec (1920 samples per frame)	~11MB

Voices

4 pre-encoded voices in constants_bin/:

alba (default), azelma, cosette, javert

Voice cloning is supported: the Mimi encoders are included (mimi_encoder*.mlmodelc at the root for English, v2.1/<lang>/mimi_encoderv3.mlmodelc per language pack). In FluidAudio, use PocketTtsManager.cloneVoice() or fluidaudiocli tts "…" --backend pocket --clone-voice speaker.wav.

Small models (pocket-tts 4.0.0): `v4/<model>/`

Core ML packs for Kyutai's six small English models released 2026-10-20: english_18m_2026-10, english_7m_2026-10, english_2m_2026-10 and their reward-fine-tuned *_refl_2026-10 twins (a different architecture from the 100M: grouped-query attention, z_proj, low-rank text table, one-step flow heads, separate 128-d Mimi decoder). Each pack holds:

File	Precision / device	Role
`pocket_state.mlmodelc`	fp16 MLState multifunction, Neural Engine, iOS 18 / macOS 15	`prefill` (text block into the FlowLM KV state), `generate` (FlowLM step + flow head per frame), `generate_decode` (step + head + fp16 Mimi in one dispatch), plus `write_state` / `decode` where the ANE planner accepts them
`mimi_decoder.mlmodelc`	fp32, CPU, iOS 17	streaming Mimi decoder, 1920 samples per frame; use this one
`flowlm_step`, `cond_prefill`, `flow_decoder_fused` `.mlmodelc`	fp32 I/O, CPU/GPU, iOS 17	exact reference path (~1e-6 vs PyTorch), no MLState needed
`constants_bin/`		tokenizer, folded text table, BOS / speaker constants, 27 voice KV snapshots (`<voice>.safetensors`)
`arch.json`, `pocket_state.json`, `manifest.json`		every dimension, threshold and state shape a host needs

Recommended host path: generate on the Neural Engine + mimi_decoder (fp32) on the CPU — this is the FluidAudio default (PocketTtsSmallManager, --backend pocket-small). The all-ANE generate_decode is faster (18m: 2.9 vs 4.4 ms per 80 ms frame) but its fp16 SEANet adds a white noise floor at about −72 dBFS, audible as hiss in pauses; fp16 rounding, not an ANE fault, and mixed precision is rejected inside MLState programs. Chunk text at sentence boundaries to ≤ 50 tokens (upstream MAX_TOKEN_PER_CHUNK): on a whole paragraph several voices never emit EOS, in PyTorch as well. The phase input of the Mimi functions is a one-hot of frame < n ? frame : n + frame % n with n = capacity / 16 (see pocket_state.json).

M5 Pro, FluidAudio CLI, 15 s passage: 18m 24x real time (109 MB RSS), 7m 30x (86 MB), 2m 51x (58 MB); fused fp16 path 34x / 42x / 52x. Whisper transcripts match the PyTorch output. Voice cloning for the small models is not wired yet (the Mimi encoder is the 100M English one; only speaker_proj_weight differs and ships in the pack). Conversion code and the full trial log live in the FluidAudio PR #1051 thread.

Phonon 31m (Gradium): `phonon/phonon-31m-en/`

Core ML pack for gradium/phonon-31m-en, Gradium's 31M-parameter English TTS released 2026-10-20. It is the Pocket TTS recipe trained by the Kyutai founders' spin-off. This folder is Apache-2.0, the license of the original weights: see phonon/phonon-31m-en/LICENSE and its model card.

Same layout and host path as the v4/ packs: pocket_state.mlmodelc (generate on the Neural Engine) plus the fp32 mimi_decoder.mlmodelc on the CPU.
103 voices (25 English, 12 characters, French / German / Spanish / Portuguese catalogue voices, plus the checkpoint default) as conditioning rows in voices/. The model speaks English only.
Phonon's differences are baked into the graphs or listed in arch.json: erf GELU, the summed-conditioner input bias, a 251-wide Mimi window, EOS threshold +4.0, and a 25 ms onset fade.
Matches Gradium's own ptts runtime: same frame counts, and identical Whisper transcripts on the Neural Engine path. About 20 ms to first audio and ~20x real time on an M5 Pro.

let tts = PocketTtsSmallManager(model: .phonon31m)
try await tts.initialize()
let wav = try await tts.synthesize(text: "Hello from Phonon.", voice: "en_us_Easton")

Usage

import FluidAudioTTS

let manager = PocketTtsManager()
try await manager.initialize()
let audio = try await manager.synthesize(text: "Hello, world!")

See https://github.com/FluidInference/FluidAudio for the full Swift framework.

License

CC-BY-4.0, inherited from https://huggingface.co/kyutai/pocket-tts. Attribution to Kyutai is required.

References

- https://huggingface.co/kyutai/pocket-tts
- https://arxiv.org/abs/2410.00037
- https://github.com/FluidInference/FluidAudio