CoTTAGE

CoTTAGE is an audio judge based on Qwen2.5-Omni-7B. It explains its judgment, then predicts a mean opinion score (MOS) from 1.00 to 5.00.

It supports two tasks:

• REL: how well the audio matches a text description.
• OVL: overall audio quality, without a text description.

This repository contains the full model weights. No separate LoRA adapter is needed.

Usage with Transformers

Use Python 3.10 and a CUDA GPU. This example does not need ms-swift, vLLM, or qwen-omni-utils.

pip install "torch>=2.6" "transformers>=4.57,<5" accelerate librosa soundfile

Replace clip.wav and description with your audio file and its text prompt. The example scores the clip on REL and OVL separately. Audio is converted to 16 kHz mono. Only text is generated; speech output is disabled.

import librosa
import torch
from transformers import Qwen2_5OmniForConditionalGeneration, Qwen2_5OmniProcessor

model_id = "AutoAudioReasoningMOS/CoTTAGE"
audio_path = "clip.wav"
description = "A dog barks twice, then a car drives past."

model = Qwen2_5OmniForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
    enable_audio_output=False,
).eval()
processor = Qwen2_5OmniProcessor.from_pretrained(model_id)

rel_prompt = (
    "Rate the semantic alignment between this audio and the following text "
    "description on a scale from 1.00 to 5.00, where 1.00 means no match and "
    "5.00 means a perfect match. Think step by step, then give the final score. "
    "Output the score as a number rounded to two decimal places. Put your "
    "reasoning inside <think></think> and the final score inside <score></score>."
    f"\n\nText: {description}\n"
)
ovl_prompt = (
    "Rate the overall audio quality of this audio clip generated by a "
    "text-to-audio model on a scale from 1.00 to 5.00, where 1.00 means very "
    "poor quality and 5.00 means excellent quality. Think step by step, then "
    "give the final score. Output the score as a number rounded to two decimal "
    "places. Put your reasoning inside <think></think> and the final score "
    "inside <score></score>.\n"
)

# The processor accepts a waveform directly; no temporary WAV is needed.
audio, _ = librosa.load(audio_path, sr=16000, mono=True)

for task, prompt in [("REL", rel_prompt), ("OVL", ovl_prompt)]:
    messages = [{
        "role": "user",
        "content": [
            {"type": "audio", "audio": audio_path},
            {"type": "text", "text": prompt},
        ],
    }]
    chat = processor.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True,
    )
    inputs = processor(
        text=chat,
        audio=[audio],
        sampling_rate=16000,
        padding=True,
        return_tensors="pt",
    ).to(model.device).to(model.dtype)

    with torch.inference_mode():
        output_ids = model.generate(
            **inputs,
            return_audio=False,
            thinker_do_sample=False,
            thinker_max_new_tokens=1024,
        )

    # Decode only the generated answer, excluding the input prompt.
    answer_ids = output_ids[:, inputs["input_ids"].shape[1]:]
    text = processor.batch_decode(
        answer_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False,
    )[0]
    print(f"{task}:\n{text}\n")

The loading and generation APIs follow the Qwen2.5-Omni Transformers documentation.

Usage with ms-swift and vLLM

You can also use ms-swift's vLLM engine to speed up the inference process:

pip install "ms-swift>=3.12" "vllm>=0.10.2" "transformers>=4.57" \
    qwen-omni-utils librosa soundfile

import os
import tempfile
from pathlib import Path

os.environ["ENABLE_AUDIO_OUTPUT"] = "0"

import librosa
import soundfile as sf
from swift.llm import InferRequest, RequestConfig, VllmEngine

model = "AutoAudioReasoningMOS/CoTTAGE"
audio_path = "clip.wav"
description = "A dog barks twice, then a car drives past."

rel_prompt = (
    "Rate the semantic alignment between this audio and the following text "
    "description on a scale from 1.00 to 5.00, where 1.00 means no match and "
    "5.00 means a perfect match. Think step by step, then give the final score. "
    "Output the score as a number rounded to two decimal places. Put your "
    "reasoning inside <think></think> and the final score inside <score></score>."
    f"\n\nText: {description}\n"
)
ovl_prompt = (
    "Rate the overall audio quality of this audio clip generated by a "
    "text-to-audio model on a scale from 1.00 to 5.00, where 1.00 means very "
    "poor quality and 5.00 means excellent quality. Think step by step, then "
    "give the final score. Output the score as a number rounded to two decimal "
    "places. Put your reasoning inside <think></think> and the final score "
    "inside <score></score>.\n"
)

engine = VllmEngine(model)
config = RequestConfig(max_tokens=1024, temperature=0.0)

with tempfile.TemporaryDirectory() as tmp:
    wav, _ = librosa.load(audio_path, sr=16000, mono=True)
    prepared_audio = str(Path(tmp) / "clip.wav")
    sf.write(prepared_audio, wav, 16000)

    for task, prompt in [("REL", rel_prompt), ("OVL", ovl_prompt)]:
        request = InferRequest(messages=[{
            "role": "user",
            "content": [
                {"type": "audio", "audio": prepared_audio},
                {"type": "text", "text": prompt},
            ],
        }])
        response = engine.infer([request], config)
        text = response[0].choices[0].message.content
        print(f"{task}:\n{text}\n")

For REL, supply the text used to generate the audio. For OVL, only the audio is used. To run just one task, keep only its entry in the loop.

Output

Each response contains an explanation in <think> tags and a score in <score> tags. For example:

<think>
The motor stays at a steady pitch and volume. It does not speed up or stop
as described in the text.
</think>
<score>1.80</score>

Both examples store the latest response in text. Extract its score:

import re

match = re.search(r"<score>\s*([0-9]+(?:\.[0-9]+)?)\s*</score>", text)
score = float(match.group(1)) if match else None

REL ranges from no match (1.00) to a perfect match (5.00). OVL ranges from very poor quality (1.00) to excellent quality (5.00). Run the tasks separately. A response gives a score for the requested task, not both.