ModelsLab/midashenglm-gen-wer-lora

A LoRA adapter for mispeech/midashenglm-gen, trained across all five of the model's capabilities — speech, sound effects, music, ambience and mixed scenes — rather than trading one for another.

Why

Two weak spots in the base model, measured rather than assumed.

Short lines. A four-word line scores around 42% WER where a sixteen-word one scores near zero. The failures are onsets: the model needs a moment to settle and a short line does not give it one. Prompt-side remedies make it worse, so it is a property of the weights.

Sound effects. The model's worst benchmark — AudioCaps FAD 5.01 against TangoFlux's 2.26 — which the paper attributes to training on mixed scenes rather than dedicated sound-effect corpora.

Results

Same prompt, same seed, base weights against these adapters. generate() seeds the global RNG, so the flow-matching solver draws identical noise in both arms and every difference is the adapters.

capability	metric	base	tuned	change
speech	WER	5.6%	0.0%	-5.6%
sfx	CLAP text	0.4148	0.5299	+0.1151
sfx	CLAP real	0.4227	0.3920	-0.0306
music	CLAP text	0.3331	0.3212	-0.0119
music	CLAP real	0.6323	0.6305	-0.0018
ambience	CLAP text	0.1993	0.1340	-0.0653
ambience	CLAP real	0.3546	0.4003	+0.0457
mixed	WER	15.6%	0.7%	-14.9%
mixed	CLAP text	0.1076	0.0505	-0.0571
mixed	CLAP real	0.7155	0.7072	-0.0084
speech	WER, 4-word lines	13.9%	0.0%	-13.9%
speech	WER, 9-word lines	3.2%	0.0%	-3.2%
speech	WER, 16-word lines	1.9%	0.0%	-1.9%
speech	WER, 27-word lines	3.4%	0.0%	-3.4%
speech	mean length	4.73s	4.06s	-0.67s
sfx	mean length	8.36s	9.91s	+1.56s
music	mean length	9.44s	9.20s	-0.24s
ambience	mean length	13.49s	7.40s	-6.09s
mixed	mean length	7.33s	5.44s	-1.89s

CLAP text is "does it match the caption". CLAP real is cosine to the centroid of genuine MECAT audio of that category — "does it sound like the real thing", which text similarity alone cannot see. WER cannot see either: a clip with no words in it scores 0% however badly the room tone came out.

WER improved by 5.6 points, and that part is solid. The scene arms are not.

Caption adherence or realism fell on: sfx clap_real -0.031, music clap_text -0.012, ambience clap_text -0.065, mixed clap_text -0.057.

Non-speech clips also got materially shorter (ambience 13.5s -> 7.4s, mixed 7.3s -> 5.4s), which matters because the scene prompts explicitly ask for continuity. Training on short read speech taught the stop head to end earlier, and that is correct for a spoken line and wrong for a bed.

Use this for speech. For beds, effects and mixed scenes, A/B it against the base model on your own prompts first — the held-out flow loss improved on every capability, and on the scene arms that did not translate.

Training

Method	LoRA r=32, alpha=64, on LLM attention/MLP and the flow-matching DiT
Trainable	53870592 of 2.89 B
Precision	float32 with TF32 matmuls, matching how the base model is served
Optimiser	AdamW-8bit, lr 0.0001, cosine decay, grad clip 1.0
Steps	2500 at an effective batch of 16
Capability mix	speech 40%, mixed 22%, music 16%, sfx 14%, ambience 8%
Speech data	LibriTTS-R + MECAT S00; clips under 3 s oversampled 2.5x
Scene data	MECAT-Caption, all eight categories
Anchor	vector field pulled towards the frozen base, weight 0.5 on 35% of batches

Base-model hyperparameters follow arXiv:2608.11804 §3.3 where they apply (lambda_stop = 0.01, grad clip 1.0).

Avoiding catastrophic forgetting

Four things, because one is not enough:

1. LoRA rather than a full fine-tune. Rank 32 bounds how far the weights can move at all.
2. A capability-weighted mix. The sampler draws by capability, not by pool. At natural frequency the speech buckets are ~80% of the clips and the model would simply get better at reading aloud.
3. An anchor loss. The tuned vector field is pulled towards the frozen base's on identical noise and timesteps — a trust region around the base model, stated directly rather than hoped for.
4. A per-capability gate. Held-out flow loss is tracked for each capability separately, and this checkpoint was only saved because the mean improved and no single capability regressed past 2%.

Use

from peft import PeftModel
from transformers import AutoModel
import soundfile as sf

model = AutoModel.from_pretrained("mispeech/midashenglm-gen", trust_remote_code=True)
model = PeftModel.from_pretrained(model, "ModelsLab/midashenglm-gen-wer-lora").merge_and_unload().cuda().eval()

result = model.generate(
    "<|caption|> A close, clean recording of a single narrator, with nothing behind it. "
    "<|asr|> The river remembers everything. "
    "<|speech|> A woman in her forties, warm and even, speaking clearly. "
    "<|sfx|> <|unknown|> <|music|> <|unknown|> <|env|> <|unknown|>",
    seed=11,
)
sf.write("out.wav", result["audio"], result["sample_rate"])

Tag order matters more than anything else in the prompt: <|asr|> comes before <|speech|>. Reversed, the model produces fluent unrelated speech — 14.2% mean WER against 373%.

Licences

Apache 2.0, following the base model. Training data: LibriTTS-R (CC-BY-4.0) and MECAT-Caption (CC-BY-3.0), both attribution-only.

Training code: audio-scenegen/training.