ASLP-lab/CN-MultiDialect-ASR

On-Policy Self-Distillation for Multi-Dialect ASR: Mastering Dialects, Retaining Mandarin

Shuiyuan Wang¹ · Bingshen Mu¹ · Pengshen Zhang² · Chengyou Wang¹ · Yujie Liao¹ · Chengdong Liang² · Binbin Zhang² · Qiangze Feng³ · Lei Xie¹

¹ Audio, Speech and Language Processing Group (ASLP@NPU), School of Computer Science, Northwestern Polytechnical University, Xi'an, China
² WeNet Community
³ NEXDATA TECHNOLOGY INC.

This repository hosts the released CN-MultiDialect-ASR checkpoint, adapted from Qwen3-ASR-1.7B with a three-stage pipeline: continual pre-training (CPT), dialect supervised fine-tuning (SFT), and On-Policy Self-Distillation (OPSD). The goal is to improve Chinese dialect recognition without raising Mandarin CER.

• Paper: arXiv:2608.11898
• Code, demo, and training scripts: ASLP-lab/CN-MultiDialect-ASR

Overview of the staged adaptation pipeline. Top: base model, CPT, SFT, and OPSD. Bottom: OPSD with student on-policy prefixes, a frozen teacher conditioned on the reference transcript as privileged context, soft targets q_t, and token-level KL.

Demo

Video demo with live waveforms and model transcriptions for Mandarin, English, four core dialects, and 15 ChinaVoices dialects.

Key Features

• Mandarin–dialect balanced adaptation: improves Chinese dialect ASR while retaining Mandarin recognition.
• Three-stage pipeline: CPT strengthens the Chinese ASR foundation, dialect SFT specializes for dialects, and OPSD refines the final checkpoint.
• On-Policy Self-Distillation: trains on student-decoded prefixes with soft teacher targets, reducing the train–test mismatch of teacher-forced ASR training.
• Drop-in inference: compatible with the official qwen-asr package.

Quickstart

Inference is compatible with Qwen3-ASR. We recommend installing the official qwen-asr package in a clean environment.

Environment Setup

conda create -n qwen3-asr python=3.12 -y
conda activate qwen3-asr
pip install -U qwen-asr

For faster inference with the vLLM backend:

pip install -U qwen-asr[vllm]

Model Download

You can load the model directly from Hugging Face, or download it locally first:

# Hugging Face
pip install -U "huggingface_hub[cli]"
hf download ASLP-lab/CN-MultiDialect-ASR --local-dir ./CN-MultiDialect-ASR

# ModelScope (recommended for users in Mainland China)
pip install -U modelscope
modelscope download --model ASLP-lab/CN-MultiDialect-ASR --local_dir ./CN-MultiDialect-ASR

Python Inference

Load the model with Qwen3ASRModel.from_pretrained and call transcribe:

import torch
from qwen_asr import Qwen3ASRModel

model = Qwen3ASRModel.from_pretrained(
    "ASLP-lab/CN-MultiDialect-ASR",  # or "./CN-MultiDialect-ASR" for a local path
    dtype=torch.bfloat16,
    device_map="cuda:0",
    # attn_implementation="flash_attention_2",
    max_inference_batch_size=32,
    max_new_tokens=256,
)

results = model.transcribe(
    audio="path/to/audio.wav",
    language="Chinese",  # or None for automatic language detection
)

print(results[0].language)
print(results[0].text)

Batch inference is also supported:

results = model.transcribe(
    audio=[
        "path/to/mandarin.wav",
        "path/to/dialect.wav",
    ],
    language=["Chinese", "Chinese"],
)

for r in results:
    print(r.language, r.text)

For vLLM backend, streaming inference, and forced alignment, see the Qwen3-ASR repository.

Method Overview

Stage	Training data	Goal	Objective
CPT	Full Mandarin-dialect collection (~100k hours)	Build a stronger Chinese ASR foundation	Cross-entropy
SFT	Same sources with higher dialect sampling weight and a small Mandarin anchor	Lower dialect CER	Cross-entropy
OPSD	Dialect refinement subset (~5k hours)	Improve dialect recognition without hurting Mandarin	Token-level KL

At inference time, only the student pathway is used.

Performances

Dialect Overview

Higher is better. Left: 5 public dialect sets; right: 18 internal dialects. Both panels use the same radial scale (0.2–1.0). The figure compares the Qwen3-ASR baseline with the released CN-MultiDialect-ASR (OPSD) checkpoint.

Public Dialect CER (%)

Evaluation set	Dialect	Qwen3-ASR	CN-MultiDialect-ASR
WenetSpeech-Yue Long	Cantonese	9.99	8.80
WenetSpeech-Yue Short	Cantonese	6.93	5.31
WenetSpeech-Chuan Easy	Sichuan	12.38	11.86
WenetSpeech-Chuan Hard	Sichuan	21.79	21.74
WenetSpeech-Wu	Wu	25.74	16.26
Dialect Avg.		15.37	12.79

Internal Dialect CER (%)

Dialect	Qwen3-ASR	CN-MultiDialect-ASR
Anhui	18.95	13.08
Cantonese	10.06	7.74
Changsha	14.79	10.23
Chaoshan	45.59	25.21
Dongbei	6.45	5.80
Henan	8.46	5.99
Kejia	60.47	28.60
Minnan	30.03	18.59
Nanchang	33.41	15.58
Nanjing	13.37	9.33
Shanxi	28.53	18.69
Shaanxi	9.68	6.28
Shandong	8.78	7.64
Shanghai	15.78	12.07
Sichuan	5.99	5.38
Suzhou	50.35	20.73
Wuhan	11.30	7.59
Xuzhou	6.12	5.04
Internal Avg.	21.01	12.42

Mandarin CER (%)

Evaluation set	Qwen3-ASR	CN-MultiDialect-ASR
AISHELL-1	1.57	1.38
AISHELL-2	2.79	2.52
KeSpeech	5.11	4.56
SpeechIO-1	0.75	0.86
SpeechIO-2	3.83	3.39
SpeechIO-3	1.39	1.27
Test_Meeting	6.74	6.85
Test_Net	5.46	5.30
Mandarin Avg.	3.46	3.27

Citation

If you use this model, please cite:

@misc{wang2026onpolicyselfdistillationmultidialectasr,
  title={On-Policy Self-Distillation for Multi-Dialect ASR: Mastering Dialects, Retaining Mandarin},
  author={Shuiyuan Wang and Bingshen Mu and Pengshen Zhang and Chengyou Wang and Yujie Liao and Chengdong Liang and Binbin Zhang and Qiangze Feng and Lei Xie},
  year={2026},
  eprint={2608.11898},
  archivePrefix={arXiv},
  primaryClass={eess.AS},
  url={https://arxiv.org/abs/2608.11898}
}

License

The released model is licensed under Apache 2.0.

Contact

For questions or collaborations, please contact wangshuiyuan@mail.nwpu.edu.cn.

You are also welcome to join our WeChat group for technical discussions and updates.

Scan to join our WeChat discussion group