gszauer/Gab100M

Gab 100M

• DEMO at giftofgab.chat
• Pre-trained on: https://huggingface.co/datasets/gszauer/Gab100MPretrain
• Fine-tuned on: https://huggingface.co/datasets/gszauer/Gab100MFinetune

Gab 100M is a small full-parameter causal language model trained locally with MLX and exported for Hugging Face Transformers using custom model code. Load it with trust_remote_code=True.

from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained(".", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(".", trust_remote_code=True)

prompt = "<|user|>Explain photosynthesis in simple terms.<|end|><|assistant|>"
inputs = tok(prompt, return_tensors="pt")
out = model.generate(
    **inputs,
    max_new_tokens=300,
    eos_token_id=tok.convert_tokens_to_ids("<|end|>"),
    pad_token_id=tok.convert_tokens_to_ids("<|pad|>"),
    use_cache=False,
)
print(tok.decode(out[0], skip_special_tokens=False))

Architecture

This model is a decoder-only causal transformer. It is not a stock Llama model, even though several parameter names follow Llama-style naming. In particular, the MLP is exact GeLU with up_proj and down_proj; there is no SwiGLU gate projection.

Configuration:

• Vocabulary size: 10,000 total token ids.
• Context length: 4,096 tokens.
• Layers: 12 transformer blocks.
• Hidden size: 768.
• Attention heads: 12.
• Head dimension: 64.
• Attention projection size: 12 * 64 = 768.
• MLP intermediate size: 3,456.
• Positional encoding: RoPE, base/theta 100,000.
• Normalization: RMSNorm with epsilon 1e-5.
• Activation: exact GeLU.
• Dropout: 0.0.
• Biases: no attention or MLP biases.
• Embeddings: input embeddings are tied to the output projection.
• Weight dtype in this export: fp32.

Forward Pass

Given integer token ids input_ids with shape (batch, sequence), the model performs:

1. Token embedding lookup:

   h = embed_tokens[input_ids]
   

2. For each transformer block:

   h = h + SelfAttention(RMSNorm(h))
   h = h + MLP(RMSNorm(h))
   

3. Final RMSNorm:

   h = RMSNorm(h)
   

4. Tied output projection:

   logits = h @ embed_tokens.weight.T
   

RMSNorm

For a hidden vector x:

rms = rsqrt(mean(x^2) + 1e-5)
RMSNorm(x) = weight * x * rms

The normalization math is done in float32 for numerical stability.

Attention

Each block uses standard multi-head causal self-attention:

q = q_proj(x)
k = k_proj(x)
v = v_proj(x)
q, k, v -> reshape to (batch, heads, sequence, head_dim)
q, k = RoPE(q, k)
attention = softmax((q @ k.T) / sqrt(head_dim) + causal_mask)
out = attention @ v
out = o_proj(out)

All heads are query/key/value heads; there is no grouped-query attention.

RoPE

RoPE is applied to all 64 dimensions of each head before attention. The inverse frequency vector is:

inv_freq[i] = 1 / (100000 ** (i / 64)), for i = 0, 2, 4, ..., 62

For a token position p, compute:

freqs = p * inv_freq
emb = concat(freqs, freqs)
q_rot = q * cos(emb) + rotate_half(q) * sin(emb)
k_rot = k * cos(emb) + rotate_half(k) * sin(emb)

Where:

rotate_half([x1, x2]) = [-x2, x1]

with x1 and x2 being the first and second halves of the head dimension.

MLP

The feed-forward network is:

MLP(x) = down_proj(gelu(up_proj(x), exact=True))

There is no gate_proj.

Weight Layout

The exported model.safetensors uses these parameter names:

model.embed_tokens.weight
model.layers.N.input_layernorm.weight
model.layers.N.self_attn.q_proj.weight
model.layers.N.self_attn.k_proj.weight
model.layers.N.self_attn.v_proj.weight
model.layers.N.self_attn.o_proj.weight
model.layers.N.post_attention_layernorm.weight
model.layers.N.mlp.up_proj.weight
model.layers.N.mlp.down_proj.weight
model.norm.weight

There is no separate lm_head.weight; the output projection is tied to model.embed_tokens.weight.

Tokenizer

The tokenizer is a byte-level BPE tokenizer with a 10,000-token vocabulary. It uses special tokens plus 256 byte tokens and learned BPE merges.

Important special tokens:

Token	Meaning
`<	end
`<	user
`<	assistant
<think>	Start visible thinking trace
</think>	End visible thinking trace
`<	pad

Chat Format

This model supports a simple two-role chat format. It does not require or use a system role.

Single-turn prompt:

<|user|>QUESTION<|end|><|assistant|>

The model should generate:

ANSWER<|end|>

Multi-turn prompt:

<|user|>QUESTION 1<|end|><|assistant|>ANSWER 1<|end|><|user|>QUESTION 2<|end|><|assistant|>

Thinking can be forced by opening a thinking tag after the assistant marker:

<|user|>QUESTION<|end|><|assistant|><think>

The expected completion format is:

reasoning...</think>final answer<|end|>

For normal non-thinking responses, omit <think>:

<|user|>QUESTION<|end|><|assistant|>

Notes

• Generation should use <|end|> as the EOS token.
• This export disables KV caching in generation_config.json because the included custom model implementation favors correctness and simplicity.
• The model was trained as a learning project