Bagpiper-Base

Bagpiper-Base is the pretrained 8B checkpoint from Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions. It learns a bidirectional mapping between physical audio and detailed natural-language descriptions spanning speech, music, environmental sound, and mixtures of them.

This is the Base checkpoint, not the instruction-tuned assistant. Use espnet/bagpiper-sft for the caption-then-process model trained to answer open-ended audio requests. Bagpiper-Base is intended for audio-to-rich-caption probing, rich-caption-conditioned audio generation, and downstream initialization.

Repository contents

File	Description
base.pt	Native ESPnet weight-only checkpoint
train_stage2_qwen3_base.yaml	Reconstructed checkpoint-compatible model configuration

base.pt is a PyTorch mapping with model state under the top-level module key. It contains 1,382 tensors: 1,381 BF16 tensors from training and one deterministic FP32 vocab_weight buffer required by the current strict ESPnet loader. It contains no optimizer, scheduler, or exact-resume state. This is not a Transformers from_pretrained directory, and no vLLM compatibility is claimed.

The YAML reconstructs the architecture and vocabulary of the released checkpoint. The original stage-2 launch referenced this filename but did not commit it, so its trainer fields should not be treated as an archival record of every training setting.

Download and integrity

hf download espnet/bagpiper \
  --revision d7b91ab261116973fb65cf44ff9590f6f242323b \
  --local-dir bagpiper-base

echo "c9917c1237bd44ea0ea8b41746e5ec5aecae92273538c9b6e4040d26f69a6521  bagpiper-base/base.pt" \
  | sha256sum -c -

Architecture

Component	Released configuration
Text backbone and tokenizer	Qwen/Qwen3-8B-Base
Audio input encoder	Qwen/Qwen3-Omni-30B-A3B-Instruct
Audio output tokenizer	Eight-stream, delay-interleaved hf-audio/xcodec-hubert-general
Interface	Interleaved text and audio tokens in ESPnet SpeechLM
Checkpoint dtype	BF16 model tensors; FP32 deterministic vocabulary-weight buffer

The audio encoder converts input waveforms into continuous representations. X-Codec provides discrete output tokens that can be decoded back into audio. Expanded text and audio embeddings allow the Qwen3 backbone to model both directions through one shared sequence interface.

ESPnet runtime

The native consumer is espnet2/speechlm/bin/inference.py. Reconstruct the model with the supplied training YAML, pair it with the task-appropriate inference YAML from the Bagpiper recipe, and pass base.pt through --model-checkpoint.

The released state dict was loaded through ESPnet's strict inference path on an NVIDIA H100 using ESPnet PR #6503 at commit 1992821bfb6d68c6636d1b75340b8b2b591bc015 and Transformers 5.5.4. A complete clean-room installation and end-to-end quickstart have not yet been validated; the Hub Inference API is therefore disabled. The paper estimates roughly 24 GB of GPU memory for BF16 inference, with longer inputs, decoding, or additional workers requiring more.

Pretraining

Bagpiper-Base is the result of the first two stages of the paper's curriculum: connector and embedding warmup followed by full pretraining. The audio encoder and codec remain frozen during full pretraining while the backbone, connectors, and expanded vocabulary are optimized. The paper reports training on 80 NVIDIA GH200 GPUs.

The 600B figure in the paper is a token budget, not a row count: 300B text-to-audio, 150B audio-to-text, and 150B text-only tokens. The audio pipeline began from roughly 422M raw audio-caption pairs, capped clips at 30 seconds, and used machine-generated rich captions plus quality, alignment, and deduplication filters.

The associated public Bagpiper_PreTrain_Data snapshot is a large artifact from this pipeline, not a claim of a complete one-to-one dump of every raw pair or text-only source used in training.

Paper-reported Base probes

These results evaluate the pretrained audio-rich-caption mapping. They were not rerun as part of this Hub card update and must not be interpreted as end-to-end task-solving scores.

Direction / probe	Result
Audio -> rich caption -> external-LLM ASR, LibriSpeech test-clean WER	5.0
Audio -> rich caption -> external-LLM QA, MMAU-Mini accuracy	69.0
Rich caption -> speech, LibriSpeech test-clean WER	1.8
Rich caption -> general audio, MMAU-Mini FAD	2.98
Rich caption -> general audio, MMAU-Mini audio-audio CLAP	0.55

For the understanding probes, an external text LLM answers using only Bagpiper-Base's generated caption; the model does not solve the final task itself. The speech-generation WER uses only the roughly 60% of test-clean examples whose conditioning rich captions preserve the full transcription. Generation baselines receive transcripts or short captions, whereas Bagpiper-Base receives full rich captions. See the paper for the complete protocol, comparisons, cycle-consistency results, and text-capability probes.

Intended use

• Research on bidirectional audio-rich-caption representations.
• Rich-caption generation for analysis or as input to a separately evaluated downstream system.
• Rich-caption-conditioned generation of speech, music, sound, and mixtures.
• Initialization for downstream ESPnet SpeechLM fine-tuning.

For general instruction following and open-ended audio question answering or generation, start from espnet/bagpiper-sft. For speech-specialized synthesis, see espnet/bagpiper-tts-sft.

Limitations and responsible use

Rich captions are an explicit information bottleneck: omissions, hallucinations, or incorrect acoustic details can propagate into downstream answers or generated audio. Generated audio may contain artifacts, wrong words, unsafe content, or unintended resemblance to people or styles. Most headline probes are English-focused, some depend on model-based evaluation, and performance outside the evaluated distributions is unknown.

Training sources and machine-generated captions may contain copyrighted, identifiable, sensitive, biased, or offensive material. Do not use this checkpoint for impersonation, deceptive media, surveillance, biometric authentication, or high-stakes automated decisions. Obtain appropriate rights and consent, disclose synthetic media where relevant, and apply human review and domain-specific safeguards.

No blanket model-weight license is declared in this card. The checkpoint does not override the terms of its upstream components or training sources; review those terms for the intended use and jurisdiction.

Provenance and validation

• Selected checkpoint: step_260000/global_step259988.
• Raw source: JinchuanTian/Bagpiper_checkpoints at revision f226d242bb0d595b02e2289f8784693e88470c18.
• Raw model-state SHA-256: 58c64ff430e2d54170403b77471c28e49b432669a9b264f41efaf0bedd52fd7a.
• Released base.pt: 18,188,844,317 bytes; SHA-256 c9917c1237bd44ea0ea8b41746e5ec5aecae92273538c9b6e4040d26f69a6521.
• Hub revision before this card update: d7b91ab261116973fb65cf44ff9590f6f242323b.
• The converter preserves all 1,381 source tensors and adds only the deterministic vocab_weight buffer; the resulting state dict passes the native strict ESPnet loader.

Related resources

• Project page and audio gallery
• Accepted COLM 2026 paper
• Bagpiper ESPnet recipe
• Bagpiper SFT model
• Bagpiper-TTS SFT model
• Bagpiper pretraining data
• Bagpiper SFT data

Citation

@inproceedings{tian2026bagpiper,
  title     = {Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions},
  author    = {Jinchuan Tian and Haoran Wang and Bo-Hao Su and Chien-yu Huang and
               Qingzheng Wang and Jiatong Shi and William Chen and Xun Gong and
               Siddhant Arora and Chin-Jou Li and Masao Someki and Takashi Maekaku and
               Keita Goto and Yusuke Shinohara and Jin Sakuma and
               Chao-Han Huck Yang and Shinji Watanabe},
  booktitle = {Third Conference on Language Modeling},
  year      = {2026},
  url       = {https://openreview.net/forum?id=FuHs64E3X6}
}