clapkit-coreml — fp16-safe CoreML conversions of LAION CLAP (both encoders)

CoreML conversions of laion/clap-htsat-unfused — both towers of the CLAP audio↔text model — converted from the PyTorch source (transformers ClapModel) with the campaign's fp16-survivable numerical guard discipline, so they are correct on the Apple GPU and Neural Engine, not just CPU.

Consumed by the clapkit crate of the coremlit workspace (Rust, sync, sans-I/O). Each graph emits the final 512-d joint-space embedding pre-L2-norm; the caller performs the L2 normalization (this keeps the fp16 rsqrt guard class out of the graph entirely).

Contents

artifact	form	what it is
clap_audio.{mlpackage,mlmodelc}	fp16	HTSAT audio tower + audio_projection (spectrogram-input)
clap_text.{mlpackage,mlmodelc}	fp16	RoBERTa text tower + text_projection
clap_audio_int8.{mlpackage,mlmodelc}	int8	audio tower — 8-bit k-means-palettized weights (weight-only; activations fp16)
clap_text_int8.{mlpackage,mlmodelc}	int8	text tower — 8-bit k-means-palettized weights (weight-only; activations fp16)

.mlpackage is the canonical distributable; .mlmodelc is the compiled form the test suite loads directly. CHECKSUMS.sha256 covers every file. The _int8 siblings are an optional ~2× smaller tier (see "int8 weight-only tier" below); the fp16 graphs remain the default.

I/O contract (pinned from the artifact metadata)

encoder	input(s)	output
audio	input_features fp32 [1, 1, 1001, 64] (log-mel)	audio_embeds fp32 [1, 512] (pre-norm)
text	input_ids int32 [1, 512], attention_mask int32 [1, 512]	text_embeds fp32 [1, 512] (pre-norm)

• Audio is 48 kHz mono, a documented deviation from the workspace's 16 kHz convention (CLAP's native rate). One inference = one fixed 480,000-sample (10 s) window → the [1, 1, 1001, 64] mel.
• Text length is fixed at 512 (the model's max). Padding a shorter prompt to 512 with the attention mask reproduces the natural-length embedding exactly (verified cosine 1.0), because RoBERTa derives positions from input_ids and the mask zeroes the padding.

The mel frontend lives in Rust (spectrogram-input), by measurement

The audio graph takes the log-mel spectrogram, not raw audio. Riding the STFT + Slaney-mel + power_to_dB frontend inside the graph was attempted and rejected on measurement:

• a faithful in-graph STFT reproduces HF's ClapFeatureExtractor mel only in float64 (the reference promotes to f64; an in-graph f32 STFT lands 0.55–0.90 cosine away from the correct path end-to-end) — f64 is hostile to an fp16 ANE graph;
• the power_to_dB floor amin = 1e-10 is 1680× below fp16's smallest subnormal (2^-24) — exactly the vanishing-guard class this campaign guards against.

So the mel is a Rust port validated bit-for-bit against textclap's mel.rs (the frontend oracle). Its parameters: n_fft = 1024, hop = 480, n_mels = 64, fmin = 50, fmax = 14000, periodic Hann, Slaney scale + Slaney norm, center=True reflect padding, 10·log10(max(·, 1e-10)), HTSAT input-norm none, time-major [1001, 64] → reshaped to [1, 1, 1001, 64].

Verification (measured, this conversion)

• MIL-level fp16-guard audit CLEAN — 55 guard sites total (audio 30, text 25), every effective floor ≥ 2^-24; no decomposed softmax→log; no unresolved guards. The text tower's LayerNorm eps = 1e-12 (below fp16 subnormal) is raised to 2^-24 by the fp16 conversion; audio LayerNorm/BatchNorm eps ≈ 1e-5 survive as-is. Normalization is out of the graph, so there is no rsqrt/real_div guard.

• PyTorch fp32 vs CoreML fp32 (CPU) on real inputs: worst cosine 1.00000000 (audio, 10 real clips: music / speech / SFX / ambient) and 1.00000000 (text, 12 varied prompts). The graph conversion is faithful; the HTSAT reshape_mel2img bicubic resize is reproduced by an exact baked-constant matmul.

• CoreML fp16 vs fp32, worst cosine per compute unit:

  encoder	ALL	CPU+GPU	CPU
  audio	0.99999573	0.99999390	0.99996626
  text	0.99994979	0.99999733	0.99987553

  fp16 is clean on every placement — fp16 is shipped (no fp32 fallback needed).

int8 weight-only tier (optional, ~2× smaller)

Alongside the fp16 graphs this repo ships an 8-bit weight-only tier of both towers — clap_audio_int8 / clap_text_int8. Weights are stored 8-bit and dequantized to fp16 at runtime; activations stay fp16 (identical graph maths, only the constant weights are compressed). Produced from the shipped fp16 .mlpackages by coremltools post-training compression — no reconversion, same source weights.

• Method: 8-bit k-means palettization, per-tensor (coremltools.optimize.coreml.palettize_weights, nbits=8, mode="kmeans", weight_threshold=2048). Chosen over linear-symmetric int8 (linear_quantize_weights) by measurement: on the text tower linear int8 costs up to 0.43% cosine (worst 0.99571 vs fp16) while k-means costs 0.13% (worst 0.99873); on audio the two are close (0.99965 vs 0.99971). k-means wins on both towers and is decisive on text, so it is the shipped _int8 tier. Tiny bias / LayerNorm-gain tensors (< 2048 elements) stay fp16; one RoBERTa attention-mask fill constant (±inf) is likewise left uncompressed.

• int8-vs-fp16 embedding cosine (identical inputs, CPU; the conversion verification set — 10 real clips + 12 varied prompts including CJK — plus the committed golden_mel fixture):

  tower	worst	mean
  audio	0.99970584	0.99982787
  text	0.99873250	0.99965990

  Well inside a 0.5% cosine budget on both towers. (int8-vs-fp32-source worst: audio 0.99967, text 0.99897 — barely further from the PyTorch source than fp16 itself, which is 0.99997 / 0.99988.)

• Zero-shot ranking unchanged. The end-to-end gate (a ~192 s speech clip → 20 windows → aggregate → 4-anchor zero-shot score) returns the identical ranking on int8 as on fp16: top label "This is a sound of a person speaking" at logit 8.88 (fp16 8.82), same order for music / dog / rain, same margins.

• Placement unchanged. Cross-compute-unit agreement holds (audio 0.99998, text 0.99994 — ≥ the 0.9999 band on All / CPU+NE / CPU+GPU / CPU). Audio still fails ANECCompile() and falls back to GPU/CPU; text still compiles for the ANE — the palettized weights change neither.

• Size (weight.bin bytes, fp16 → int8):

  tower	fp16	int8	ratio
  audio	60,562,432	30,409,920	1.99×
  text	251,665,792	125,746,688	2.00×

  Compiled .mlmodelc: audio 58 MB → 29 MB, text 240 MB → 120 MB.

Same toolchain pin as the fp16 conversion (coremltools 9.0; scikit-learn provides the k-means step). The int8 tier is a pure recompression of the pinned fp16 artifacts, so its provenance and licensing are exactly those of the fp16 graphs below.

Placement guidance (measured, never marketed)

• text: compiles for and runs on the ANE/GPU/CPU; fp16-clean on all.
• audio: the HTSAT graph currently fails ANE compilation (ANECCompile()) and falls back to GPU/CPU — still fp16-clean there. Consumers should not assume ANE placement for the audio tower; clapkit selects the compute unit and does not assert ANE.

Toolchain (pinned)

coremltools 9.0 · torch 2.5.1 · transformers 5.14.0 · numpy 1.26.4 · python 3.11.15. Source revision: laion/clap-htsat-unfused@8fa0f1c6d0433df6e97c127f64b2a1d6c0dcda8a.

Upstream provenance & licensing

These are derivative conversions: the same LAION CLAP weights, re-emitted as CoreML graphs (projection heads in-graph, L2-norm out, mel frontend externalized).

component	upstream	license
both encoders	laion/clap-htsat-unfused	see note
tokenizer (used by clapkit, not redistributed here)	Xenova/clap-htsat-unfused @c28f2883… (tokenizer.json sha dc239041…)	derived from the RoBERTa tokenizer

License note. Attribution to laion/clap-htsat-unfused is provided as required. The LAION CLAP checkpoints are commonly attributed as CC-BY-4.0 (the position taken by the consuming project and reflected in the front-matter), while the current upstream HF model card declares apache-2.0. Both licenses require attribution, which this repository provides; downstream users should honor whichever the upstream author intends. If you are the upstream author and want changes to this redistribution, please open a discussion.