parakeet-tdt-0.6b-v3 — half-precision and 4-bit checkpoints

Reduced-precision builds of NVIDIA's nvidia/parakeet-tdt-0.6b-v3, packaged for direct GPU loading. Three artifacts, same model:

Artifact	Format	Size	For
parakeet-tdt-0.6b-v3-bf16.nemo	bfloat16	1.25 GB	Ampere and newer (compute capability 8.0+)
parakeet-tdt-0.6b-v3-fp16.nemo	float16	1.25 GB	anything with CUDA, including pre-Ampere (sm_75: GTX 16-series, RTX 20-series, Tesla T4)
parakeet-tdt-0.6b-v3-nvfp4/	NVFP4 weights, bf16 activations	0.49 GB	Ampere and newer, when VRAM is the constraint

The two .nemo files are pure casts of the fp32 weights — identical to upstream apart from the conversion. Restoring from either loads straight onto a CUDA GPU without ever materialising an fp32 copy, which is faster to start (~13 s vs ~22 s) and roughly halves peak load-time VRAM.

The NVFP4 directory is not a cast: 4-bit scales are chosen by calibrating on real audio, so it ships pre-quantized. It halves VRAM again at no accuracy cost this evaluation can detect, and costs 2.4× the decode latency.

Which one?

• bf16 — the default. Fastest, and what the accuracy of the others is measured against.
• fp16 — if your GPU predates Ampere, where bf16 is not merely slower but unsupported. On a card that does both, the two are statistically indistinguishable and fp16 is marginally closer to fp32 numerically.
• nvfp4 — only when you are short of VRAM. It is the smallest by a wide margin and the slowest of the three; it buys memory, not throughput.

This repository redistributes the model under its original CC BY 4.0 license. See the upstream model card for architecture, training data, supported languages, evaluation, and intended use.

Usage

bf16 / fp16 (NeMo)

import nemo.collections.asr as nemo_asr
from huggingface_hub import hf_hub_download

ckpt = hf_hub_download("lightware-dev/parakeet-tdt-0.6b-v3",
                       "parakeet-tdt-0.6b-v3-bf16.nemo")   # or -fp16.nemo
model = nemo_asr.models.ASRModel.restore_from(ckpt, map_location="cuda")
model.eval()
print(model.transcribe(["audio.wav"]))

NVFP4

Needs nvidia-modelopt alongside NeMo. The loader lives in lightware-dev/blurt as server/nvfp4.py:

import os
from huggingface_hub import snapshot_download
from server import nvfp4          # from github.com/lightware-dev/blurt

root = snapshot_download("lightware-dev/parakeet-tdt-0.6b-v3",
                         allow_patterns="parakeet-tdt-0.6b-v3-nvfp4/*")
model = nvfp4.load_snapshot(os.path.join(root, "parakeet-tdt-0.6b-v3-nvfp4"),
                            device="cuda")
print(model.transcribe(["audio.wav"]))

In blurtd itself this is just PARAKEET_DTYPE=nvfp4.

The load path builds the module graph on the CPU, replays the quantization recipe onto it, then reads the packed tensors straight onto the GPU — so the device never holds a bf16 copy. That is the entire point: quantizing at startup instead would require the GPU to hold the full bf16 model first (peaking at 2.53 GB), which a card small enough to need 4-bit cannot do.

What NVFP4 is here

NVFP4 is E2M1 — four bits per weight (1 sign, 2 exponent, 1 mantissa, so eight representable magnitudes) — grouped in blocks of 16, each block carrying its own E4M3 fp8 scale. The block scale is what makes four bits survivable: precision is allocated locally, so a block of small weights is not crushed by a large outlier elsewhere in the row. It is also why the packed model is 0.49 GB rather than the 0.27 GB a naive 4-bit count would suggest.

Only the 217 encoder Linear layers are quantized — 533M of 627M parameters (86%). The convolutions, LSTM decoder and joint network stay bf16, and so do all activations (W4A16). The joint network's vocabulary projection is deliberately excluded: it runs inside the TDT decode loop, where an extra unpack per step would cost latency and feed the argmax directly.

Contents of the directory:

File	Size	Holds
weights.safetensors	488.6 MB	217 packed uint8 weights (two 4-bit values per byte), 217 float8_e4m3fn block scales, 701 bfloat16 tensors
recipe.json	1.8 MB	quantizer layout and calibrated amax values
model_config.yaml	0.2 MB	NeMo config, tokenizer paths localized
tokenizer files	0.5 MB	SentencePiece model and vocabs

No pickles. torch.load reconstructs arbitrary Python objects, so loading one is equivalent to running its author's code — a poor property for a file fetched at service startup. Weights use safetensors (a length-prefixed JSON header over raw tensor bytes, structurally unable to carry code), the recipe is JSON, the config is YAML. The single class the recipe names is resolved by attribute lookup against a modelopt.* allowlist, never called.

Evaluation

All figures below come from one corpus of 208 clips / 25.8 minutes: 100 LibriSpeech test-clean utterances, the same utterances degraded with white noise at 10 dB and 5 dB SNR, babble at 5 dB and a near-clipping loud variant, plus 8 synthetic clips. Greedy TDT decoding, WER after text normalisation, on an RTX 5090. Differences against bf16 are paired bootstrap estimates — paired because both models decode identical audio, so per-utterance noise cancels instead of being counted twice.

NVFP4 was calibrated on 62 clips drawn from different LibriSpeech rows than the 208 scored ones, asserted disjoint at build time. Calibrating on the evaluation set would tune the quantizer to its own test.

NVFP4 vs bf16

	bf16	NVFP4
WER, all 208 clips	2.99 %	2.86 %
WER, clean real speech	2.40 %	2.40 %
Difference vs bf16	—	−0.128 pp, 95 % CI [−0.693, +0.295], p = 0.69
Identical transcripts	—	179 / 208 (86.1 %)
Encoder output vs fp32, mean relative L2	0.0436	0.2257
Encoder output vs fp32, worst cosine	0.8351	0.8261
Non-finite activations	0	0
Empty transcripts	3 / 208	1 / 208
Weights	1.31 GB	0.51 GB
Real-time factor	0.0134	0.0316

NVFP4 is not better than bf16. Its point estimate is lower and its babble-noise slice looks notably better, but the confidence interval spans −0.69 to +0.30 pp — that is noise landing favourably. At 208 clips and a ~3 % base rate the corpus cannot resolve a difference below roughly 0.3 pp, so the honest claim is "no cost we can measure", not "identical".

What is visible: four bits changes the exact text of 14 % of clips while leaving the aggregate untouched, and pushes the encoder output about 5× further from fp32 than bf16 does. The greedy decoder absorbs most of that drift.

Per slice:

Slice	n	bf16	NVFP4
Clean real speech	100	2.40 %	2.40 %
White noise, 10 dB SNR	25	1.90 %	2.61 %
White noise, 5 dB SNR	25	3.32 %	4.50 %
Babble, 5 dB SNR	25	8.29 %	6.16 %
Near-clipping loud	25	1.42 %	1.42 %
Synthetic	8	2.66 %	0.53 %
All	208	2.99 %	2.86 %

The babble and synthetic slices move in NVFP4's favour and the white-noise slices against it, on 25 and 8 clips respectively. Do not read a denoising story into that — these are small slices and the aggregate CI already says the corpus cannot separate the two models.

Memory

Measured per precision in a fresh process, loading the published artifact the way a service would:

	bf16 .nemo	NVFP4 snapshot
File on disk	1255 MB	491 MB
Load time	20.3 s	20.9 s
Peak CUDA while loading	2.532 GB	0.776 GB
Resident after load	1.276 GB	0.776 GB
Peak during decode	1.430 GB	0.931 GB
nvidia-smi after load	3032 MB	1370 MB

The load-time peak is the figure that decides whether a card can run the model at all, and it is where shipping pre-quantized weights pays: 0.78 GB against 2.53 GB. Quantizing a bf16 checkpoint at startup would produce the same 0.51 GB of weights but still need 2.53 GB to get there.

Load time is a wash. Of the ~21 s, only 0.3 s is device work — 0.2 s to read 489 MB of packed tensors onto CUDA and 0.1 s to bind them. The rest is CPU overhead that should not need to exist: NeMo cannot construct this model on the meta device (ConformerEncoder.__init__ calls .item()), so a full module graph is built and zero-filled, and ModelOpt's restore replays the compression on those throwaway weights rather than installing the packed layout directly.

Speed

NVFP4 decodes 2.4× slower than bf16 (RTF 0.0316 vs 0.0134). Encoder forward time is 95.6 ms against 27.0 ms.

The reason is that Parakeet's encoder is bound by kernel launches, not arithmetic. Encoder forward time barely responds to how much audio it is given:

Clip length	bf16 encoder forward
2.2 s	27.2 ms
5.4 s	27.6 ms
35.6 s	28.9 ms

Sixteen times the audio for 6 % more time means the GPU spends its life launching kernels across 24 conformer layers and 220 Linears. Quantization makes the multiplies cheaper — the part that was never the bottleneck — while adding an unpack per layer. It lands net negative. At RTF 0.032 the model is still ~30× faster than real time, which is imperceptible for dictation, but NVFP4 should not be chosen expecting throughput.

fp16 vs bf16 vs fp32

	fp32	bf16	fp16
WER, all clips	2.96 %	2.99 %	2.96 %
WER, clean real speech	2.40 %	2.40 %	2.40 %
Encoder output vs fp32, mean relative L2	—	0.0436	0.0052
Encoder output vs fp32, worst cosine similarity	—	0.8351	0.9948
Weights on disk / in VRAM	2.5 GB	1.31 GB	1.31 GB
Real-time factor	0.0117	0.0134	0.0118

fp16 and bf16 produced identical transcripts on 207 of 208 clips. The WER difference between them is +0.026 pp (95 % bootstrap CI [0.000, +0.077], p = 0.74) — statistically indistinguishable, and exactly zero on real speech.

fp16 tracks fp32 roughly 8× more closely than bf16 does at the activation level, which is what you would expect from its 10 mantissa bits against bf16's 7. No overflow was observed: zero non-finite activations anywhere, including on the deliberately near-clipping loud slice. The narrower fp16 exponent range is not a practical problem for this model at inference.

Other quantizations evaluated, and why they are not here

The same corpus and method were used to test five other post-training schemes. None is published — this table is here so the choice of NVFP4 is checkable rather than asserted.

Scheme	WER, all	vs bf16	Weights	RTF	Identical transcripts
bf16 (reference)	2.99 %	—	1.31 GB	0.0134	—
NVFP4 W4A16 (published)	2.86 %	−0.128 pp [−0.69, +0.30], p = 0.69	0.51 GB	0.0316	179 / 208
INT4 AWQ, weight-only	3.19 %	+0.204 pp [−0.08, +0.52], p = 0.19	0.49 GB	0.0328	176 / 208
NVFP4 W4A4 (4-bit activations too)	3.50 %	+0.510 pp [+0.14, +0.96], p = 0.006	0.51 GB	0.0384	169 / 208
INT8 weight-only	2.96 %	−0.026 pp [−0.18, +0.12], p = 0.88	0.74 GB	0.0161	203 / 208
INT8 dynamic activations	2.99 %	+0.000 pp [−0.21, +0.22]	0.74 GB	0.0608	189 / 208
FP8 dynamic activations	3.04 %	+0.051 pp [−0.08, +0.20], p = 0.61	0.74 GB	0.0312	200 / 208

Two results shaped the choice:

• Weights tolerate four bits; activations do not. NVFP4 W4A4 is the only variant with a statistically significant regression (CI excludes zero, p = 0.006), and it is worst on babble noise (10.43 % against bf16's 8.29 %). Quantizing weights only is what keeps 4-bit free.
• Four bits is a better deal than eight here. NVFP4 saves 38 % more memory than INT8 weight-only for the same undetectable accuracy cost, which inverts the usual expectation that fewer bits means more risk.

Nothing was faster than bf16 at any width, for the launch-bound reason above.

Caveats

• English-only corpus. These figures say nothing directly about the other 24 languages the base model supports. For the pure casts (bf16/fp16) there is a strong prior that a weight cast behaves the same everywhere; for NVFP4 that prior is weaker, because the scales were calibrated on English speech. If you use it on another language, measure.
• 208 clips bounds the resolution. Differences below roughly 0.3 pp are beyond what this corpus can see.
• NVFP4 is tied to its tooling. Built with ModelOpt 0.45.0 / PyTorch 2.12.1 / NeMo 2.7.3; recipe.json records the ModelOpt version. Cross-version restore is untested — treat the snapshot as rebuildable from source rather than archival.
• Speed figures are RTX 5090 (sm_120). The quality figures carry across hardware; the timings do not.
• Both .nemo files were cast from the upstream fp32 weights — the fp16 one is not a bf16 → fp16 round trip, which would inherit bf16's coarser mantissa for no benefit. NVFP4 was quantized from the bf16 checkpoint.

Provenance

• Base model: nvidia/parakeet-tdt-0.6b-v3
• bf16 / fp16: fp32 weights cast, re-saved as .nemo (701 float tensors, 627,090,582 parameters, unchanged key set)
• NVFP4: 217 encoder Linear layers quantized to NVFP4 W4A16 with NVIDIA TensorRT Model Optimizer 0.45.0, calibrated on 62 LibriSpeech clips held out from the evaluation set, compressed to packed 4-bit and serialized as safetensors + JSON
• License: CC BY 4.0 (inherited from the base model)
• Built and evaluated with: lightware-dev/blurt — scripts/build_bf16_ckpt.py, scripts/build_nvfp4_snapshot.py, scripts/make_eval_corpus.py and scripts/compare_precision.py

Each snapshot build records the transcripts it produced in memory; the loader is verified by reproducing them exactly from disk (20 / 20), since a mis-bound scale buffer still yields fluent English.

Previously published as lightware-dev/parakeet-tdt-0.6b-v3-bf16 and lightware-dev/parakeet-tdt-0.6b-v3-fp16; both are now consolidated here.