darioooooo0o/drama-3-w4a16-32gb

Drama-3 W4A16 · 32 GB edition

Built with Fish Audio. This is an unofficial quantization of fishaudio/drama-3, Fish Audio's 35B (3B active) multi-speaker TTS model, sized for 32 GB GPUs such as the RTX 5090.

• Routed experts are int4.
• The backbone's attention, linear-attention and shared-expert layers are int8.
• Peak GPU memory measured with the launch settings below: 30.7 GiB.

Quantized and tested by @imdariotoo. It is not affiliated with or endorsed by Fish Audio. For 40 GB+ cards, drama-3-w4a16 keeps the backbone in bf16.

Listen first

All clips were generated with this checkpoint, using the 32 GB launch settings below. Transcripts are from Whisper large-v3.

prompt	audio	Whisper transcript
Hello, how are you? The weather is lovely today, so let's take a long walk by the river.		exact
Voice clone of the cookbook reference clip: Get the trust fund to the bank early.		exact
Voice clone with tags: [sighs] Fine. If we miss it, it's on you. [laughs] Well, maybe that's fair.		exact
The quarterly report is due on Thursday, so please finish it by noon. After that, we will review the numbers together and decide whether the new pricing plan should launch next month or wait until spring.		exact
今天天气很好，我们一起去公园散步吧。		exact (Whisper wrote it in traditional characters)

At a glance

	fishaudio/drama-3	drama-3-w4a16	this repo
download	41.5 GB	25.9 GB	24.5 GB
routed experts	FP8	int4	int4
attention / linear attention / shared experts (250 matrices)	bf16	bf16	int8 (W8A16, group 128)
embeddings, lm_head, audio decoder	bf16	bf16	bf16
GPU memory, as tested	> 48 GB	37.5 GB	30.7 GiB peak

Requirements

• GPU: an NVIDIA GPU with ≥ 32 GB and compute capability ≥ 8.0, e.g. RTX 5090, or any 40/48/80 GB card.
• 30.7 GiB of a 5090's 31.8 GiB is used, so run it headless or with nothing else on the GPU; a desktop session on the same card may not leave enough room.
• Linux, CUDA 12.8+, Python 3.12, SGLang-Omni. About 26 GB of disk.

Quick start

git clone https://github.com/sgl-project/sglang-omni.git && cd sglang-omni
uv venv -p 3.12 .venv && . .venv/bin/activate
uv pip install --prerelease=allow -e ".[fishaudio]"

hf download darioooooo0o/drama-3-w4a16-32gb --local-dir drama-3-32gb
python drama-3-32gb/apply_patch.py .       # idempotent; adds the int4 + int8 quantization configs

sgl-omni serve --model-path drama-3-32gb --port 8000 \
  --tts_engine.gpu_memory_fraction 0.50 --vocoder.gpu_memory_fraction 0.12 \
  --tts_engine.factory.max_new_tokens 1024 \
  --preprocessing.total_reserve_bytes 3GiB --preprocessing.factory.max_reference_audio_seconds 20

What the flags do, and what they cost:

• The engine gets 50% of the GPU (the weights plus a small KV cache) and the vocoder 12%.
• max_new_tokens 1024: each generated chunk can be up to about 47 s of audio. Long text is still chunked automatically.
• Voice-cloning references are limited to 20 s.
• The preprocessing reserve is 3 GiB.
• The fractions are of the whole card. On a 48 GB card, the same flags give a bigger budget than on a 5090; scale them so the total stays under your VRAM.

Requests are the same as for the original:

curl -X POST http://localhost:8000/v1/audio/speech -H "Content-Type: application/json" \
  -d '{"input": "Hello, how are you?"}' --output out.wav

Voice cloning, tags, multi-speaker and streaming are covered in the Drama 3 cookbook.

How it was tested (and what wasn't)

• Memory: measured on an RTX A6000 (48 GB) with exactly the flags above. The peak of 30.7 GiB was sampled every 0.5 s while generating the five clips. The memory budget is the same on any CUDA GPU, but this has not yet run on an actual RTX 5090: both rented 5090 hosts failed to start. If you run it on a 5090, a report on X is appreciated.
• Quality: the five clips above, transcribed with Whisper, all exact. These are spot checks, not a benchmark.
• Kernels: the int8 layers use SGLang's GPTQ-Marlin W8A16 kernel and the int4 experts its moe_wna16 kernel, so both need compute capability 8.0 or newer.

How it was made

1. quantize.py dequantizes the FP8 experts with their 128×128 block scales and re-quantizes them to int4: round-to-nearest, symmetric, group 128, with a 20-point clipping search per group. Mean relative error per expert matrix is 10.3%.
2. quantize_l8.py turns 250 backbone linears into int8 GPTQ format (symmetric, group 128). These are linear_attn.in_proj_qkv/in_proj_z/out_proj, self_attn.q/k/v/o_proj and shared_expert.gate/up/down_proj. Mean relative error is 0.77%.
3. apply_patch.py adds two quantization configs to SGLang-Omni's Drama 3 loader: bsgemm-moe-w4a16 and bsgemm-moe-w4a16-l8, the latter for this repo. They route experts to moe_wna16 and the listed linears to GPTQ-Marlin, and leave everything else bf16.

No calibration data was used. Stats are in quant_stats.json.

Limitations

• Coverage: spot-checked only (5 prompts). Int4 experts can cost some prosody or voice similarity compared with FP8, so listen first.
• Feature caps: the 32 GB launch settings limit reference clips to 20 s and generated chunks to about 47 s.
• Hardware: not yet run on an actual RTX 5090 (see above).
• Runtime version: tested with SGLang-Omni main on 2026-10-13. The patch may need updating if its loader changes.

License and attribution

This is a Derivative Work of Fish Audio Materials under the Fish Audio Community License, Version 1.0 (LICENSE), the same license as the original. NOTICE lists the attribution and the changes.

In short (the LICENSE is what counts):

• Research and non-commercial use are free.
• Commercial use is allowed only under USD 500k annual revenue.
• No hosted TTS or voice-cloning service without a license from Fish Audio.
• No cloning of real people without their consent.
• Don't remove watermarks.
• Don't use the outputs to train other speech or foundation models.

These Fish Audio Materials are licensed under the Fish Audio Community License, Version 1.0. Copyright © 39 AI, INC. All Rights Reserved.

Credits

• Model: Fish Audio.
• Runtime: SGLang-Omni and SGLang kernels.
• Quantization, patch and tests: @imdariotoo. Questions: X or the repo discussions.