AbstractPhil/aleph-splat-0

aleph-splat — splat attention research line

Softmax-free attention through tiny signed codebooks. Every head is a small frozen "aleph": K unit anchors read by a closed-form signed address w_k = sinh(u_k)/Σ_j cosh(u_j) — reconstructive, never comparative (no argmax, no top-k, no softmax anywhere, including across heads). Attention is a write/read through the codebook cells; token-to-token affinity is address agreement through a K-cell bottleneck — linear in sequence length.

Play with it

splat_attention.py — one file, torch-only:

from splat_attention import SplatAttention
attn = SplatAttention(d_model=512, M=64, K=8, rotary=True)
y = attn(x)   # (B, L, d) -> (B, L, d)

python splat_attention.py runs a demo + speed bench. Knobs for the gradient-heuristic variants are constructor args: head_gates (learnable per-head attenuation), train_centers (moving windows), train_codebooks (unfrozen frames), rotary/global_frac (transport geometry).

What has been measured (battery/splat_battery.json)

• Product code: M differentiated small codebooks beat one monolith at equal half-axis budget — joint readback .859 (1×64) → .955 (16×4), monotone; identical copies collapse. Random independent frames match constructed ones.
• Differentiation is an attractor: near-copies re-differentiate under a shared objective with no diversity pressure.
• Comparative composition loses: softmax-over-heads costs ~.10 reconstruction vs budget-weighted blending.
• Storage is partition-blind: associative capacity scales with total cells regardless of head partition — addressing capacity and memory capacity are different resources.
• Retrieval at scale: 2048 splat-sharded heads: top-1 .9995 @ 2k context / .934 @ 8k where an equal-cell monolith reads .042 / .0015.
• Speed: flat µs/token vs MHA's quadratic — slower below ~2k tokens, ~2× faster at 8k; torch.compile (inductor) gives a further 3–4×.
• The failure that shaped the design: local-only positional windows at short L degenerate into a ±3-token blur — cross-position transport dies and a cls-pooled encoder collapses (measured, then repaired). The repair is rotary=True: RoPE applied to the address query, so relative position enters every affinity while heads stay global (cross-position recall .548 where the broken form gave ~0).

Status

Research prototype under active iteration. Trained-at-scale encoder results for the splat arm are pending; the sibling trained arms (anchored-FFN trunks with the same address mechanism in the MLP) live at alephlm-0. Numbers above are measurements on the stated probes — nothing more is claimed.