Safetensors
English

duplex-0

A small speech-to-speech model trained from random weights on about 750 hours of English speech. It is a research checkpoint and is far from a usable assistant: its speech is about half as intelligible as real speech on our benchmark.

This page is the training report for the runs of 17 November 2026. The training scripts are in scripts/.

Training report, 17 November 2026

Summary. Three training options were compared on the same data, seed and settings: audio only, one text track and two text tracks. All three produce speech of similar quality: 60 to 68 confirmed words per minute after the conversational stage, against 132.8 for real speech. A text track lowers first-codebook bits per second by 12% to 14%. One text track scores highest on intelligibility after the conversational stage and audio only scores highest on turn-taking, both by margins close to the range over seeds. The text the models write is related to the speech but is far from a transcript.

Model

  • 201 million parameters, or 225 million with two text tracks, which adds a second text output head. A temporal transformer of 12 layers Γ— 768 predicts one step per 80 ms frame, and a depth transformer of 2 layers Γ— 256 predicts that frame's audio tokens.
  • Audio is Mimi tokens: 8 codebooks at 12.5 frames per second from 24 kHz audio. The model has two audio tracks, one per speaker, so a frame holds 16 codebooks.
  • Text, when used, is the words a speaker says, written as word-piece tokens at the frame where each word starts. The number of text tokens per frame varies, and an "advance" token ends each frame's text.
  • Audio only: no text.
  • One text track: the model writes the words of the track it speaks as, just before speaking them.
  • Two text tracks: it also writes the other speaker's words, 320 ms (4 frames) after each word starts. Each text track has its own output head.

Data

  • Pre-training: 749 hours of English speech, 408 hours from YODAS3 and 341 hours from public podcasts.
  • Conversational: 156 hours of two-speaker podcast conversation with frequent turn-taking and a separate audio track per speaker. This is a subset of the pre-training data.
  • Whole recordings and podcast feeds are held out for validation and testing, about 4 hours each per source.

Schedule

One H100 per option, batch 8, bf16, compiled, seed 20261117 for every run.

Stage Length Learning rate
Pre-training Stopped when the first codebook on held-out audio had not improved for two evaluations, one evaluation per epoch Warmup 100 steps, then constant 6e-4
Conversational One epoch, from the kept pre-training checkpoint Peak 2e-4, warmup 60 steps, annealed over the last 20%
Option Kept epoch (stopped at) Steps Steps per second Pre-training time
Audio only 10 (12) 13,392 7.29 37 min
One text track 13 (15) 20,595 6.25 63 min
Two text tracks 10 (12) 17,064 4.57 73 min

The kept pre-training checkpoints are taken at a constant learning rate, so they are not annealed. The conversational epoch took 2 to 5 minutes.

First codebook against the bigram and trigram floors

The first codebook of each track carries most of the speech content. The floors are what a model scores if it only counts the previous one or two first-codebook tokens.

First codebook bits per second during pre-training

Audio bits per second during pre-training

Held-out pre-training audio, in bits per second (lower is better):

Option Checkpoint Audio First codebook Text, speaking track Text, listening track
Audio only Pre-training, epoch 10 817.5 70.7 none none
One text track Pre-training, epoch 13 804.1 62.4 13.0 (5.87 bits per word token) none
Two text tracks Pre-training, epoch 10 807.7 60.9 12.3 (5.29 bits per word token) 4.6 (1.74 bits per word token)
Audio only After the conversational stage 812.5 70.2 none none
One text track After the conversational stage 799.5 61.9 13.1 (5.96 bits per word token) none
Two text tracks After the conversational stage 801.4 60.2 12.2 (5.31 bits per word token) 4.4 (1.75 bits per word token)

The bigram floor is 82.0 and the trigram floor is 80.5 on the pre-training validation set.

Benchmarks

  • Intelligibility: the model continues a 10-second prompt for 20 seconds. The score is the number of words per minute in its speech that two transcribers agree on. Real speech scores 132.8.
  • Turn-taking: the model hears a real caller and replies. The score rewards making sound in the caller's pauses and staying quiet while the caller speaks. Real speech scores 0.47.

The intelligibility prompt gives the model the audio of both speakers. A model with text tracks also gets the true words of each of its text tracks during the prompt, as in training, and writes its own text after it. Turn-taking has no prompt. Sampling is at temperature 1 with top-k 250. Figures are the mean of three seeds over 24 held-out spans, with the range over seeds in brackets.

Benchmarks by stage

Option Stage Confirmed words per minute Turn-taking score Sound over the caller
Audio only Pre-training 69.5 (65.6 to 74.6) 0.37 (0.34 to 0.41) 31%
Audio only Conversational 60.4 (58.6 to 63.5) 0.35 (0.34 to 0.38) 17%
One text track Pre-training 66.8 (61.4 to 70.1) 0.31 (0.29 to 0.32) 28%
One text track Conversational 67.5 (61.6 to 74.4) 0.31 (0.20 to 0.37) 32%
Two text tracks Pre-training 62.9 (61.1 to 64.8) 0.23 (0.21 to 0.25) 53%
Two text tracks Conversational 62.4 (57.6 to 66.6) 0.33 (0.31 to 0.37) 33%

Real speech makes sound over the caller 7% of the time. The conversational epoch raises the turn-taking score of two text tracks from 0.23 to 0.33 and leaves its confirmed words per minute unchanged. For audio only it lowers confirmed words per minute and halves the sound over the caller. With the true prompt text, one text track is within the seed range of audio only after pre-training and above it after the conversational epoch.

What the text tracks write

After the conversational stage, the text track of the speaker the model speaks as writes these word tokens per second during the turn-taking benchmark:

Option While the caller speaks While the caller is quiet
Real speech 0.41 3.05
One text track 2.35 5.33
Two text tracks 3.53 7.36

Both models write more while the caller is quiet, so they follow turns. Both still write more words than a real speaker does.

How well the written text matches the speech was measured on the same runs (60 seconds generated per clip, 72 clips pooled). The model's own text is compared with a transcript of the audio it generated on that track. The caller's text, which only the two-text-track model writes, is compared with the caller's true words, so it measures speech recognition. Word error rate is substitutions, deletions and insertions divided by reference words; lower is better and 0 is a perfect match.

Text Option Pre-training Conversational Unrelated text
Own speech One text track 0.90 0.78 1.34 to 1.43
Own speech Two text tracks 1.75 1.29 1.72 to 2.00
Caller's speech Two text tracks 0.84 0.71 1.18 to 1.23

"Unrelated text" pairs each clip's written text with the reference of a different span. The transcriber used for the own-speech reference scores 0.11 to 0.18 against the true words on real audio. The written text is related to the audio and improves with the conversational stage, but it is far from a transcript: after the conversational stage the caller's text gets 45% of the caller's words right, and most errors on the model's own text are extra words.

Samples

Each clip is 30 seconds: 10 seconds of a real held-out conversation as the prompt, then 20 seconds that the model generates for both speakers (seed 1, temperature 1, top-k 250, from the checkpoint after the conversational stage). The text-track models are given the true prompt text. The "real" clip is the true continuation. Both tracks are mixed into one file, and all audio is decoded from Mimi tokens.

Prompt Real Audio only One text track Two text tracks
1
2
3
4

The files are in report/samples/.

Checkpoints of this report

Folders resplit-20261117/<stage>-<option> hold the checkpoint at the end of a stage and resplit-20261117/<stage>-<option>-best the best one by held-out first codebook, where <stage> is pretraining or conversational and <option> is none, one or two. Each folder holds model.safetensors, config.json, checkpoint.json and train.json (the full run report). The benchmark rows above use pretraining-<option>-best and conversational-<option>.

Licence

The model weights and the training scripts in scripts/ are released under the Apache License 2.0 (LICENSE).

The model reads and writes audio as tokens of the Mimi codec and uses the text tokenizer of Moshi, both by Kyutai. Part of the training data comes from YODAS3.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Datasets used to train Trelis/duplex-0

Space using Trelis/duplex-0 1

Collection including Trelis/duplex-0