facebook/MobileMoE-L-Base

MobileMoE-L (Base) Model Card

MobileMoE is a family of on-device Mixture-of-Experts (MoE) language models with sub-billion active parameters, designed to push the quality–efficiency Pareto frontier for on-device LLMs, including three model scales (S/M/L): 0.3B/0.5B/0.9B active parameters (1.3B/2.8B/5.3B total), with <3 GB INT4 weight footprints to fit in mobile DRAM. Each scale is released in three variants: a Base model (pre-training + mid-training), an SFT model (supervised fine-tuning), and a QAT model (quantization-aware training). You are currently in the MobileMoE-L-Base repository — the pre-trained 0.9B-active base model.

	S	M	L
Active / total params	272M / 1.3B	528M / 2.8B	922M / 5.3B
Layers	20	26	32
Model dimension	768	1024	1280
Heads (Q / KV)	12 / 4	16 / 4	20 / 4
Routed experts	60	60	60
Top-k	4	4	4
INT4 weight memory	0.68 GB	1.48 GB	2.75 GB
Base	MobileMoE-S-Base	MobileMoE-M-Base	MobileMoE-L-Base
SFT	MobileMoE-S-SFT	MobileMoE-M-SFT	MobileMoE-L-SFT
QAT (INT4)	MobileMoE-S-QAT	MobileMoE-M-QAT	MobileMoE-L-QAT

For the detailed technical report: 📝 MobileMoE: Scaling On-Device Mixture of Experts

For more versions, check out the 🤗 MobileMoE Collection

MobileMoE establishes a new Pareto frontier for on-device LLMs. Average benchmark accuracy, computed over 14 benchmarks spanning commonsense, knowledge, science, comprehension, and reasoning, is plotted against (a) per-token inference compute F_inf = 2N_act (GFLOPs) and (b) total parameters N_total (B); in (b), x-axis tick labels show total params (B) | projected INT4 memory (GB).

Key Features

• A new Pareto frontier for on-device LLMs. Across 14 foundational benchmarks, MobileMoE matches or exceeds leading on-device dense LLMs at 2–4× fewer inference FLOPs, and matches or surpasses the state-of-the-art MoE OLMoE-1B-7B with up to 60% fewer parameters.
• Scaling-law-derived architecture. The architecture is derived from an on-device MoE scaling law that jointly optimizes under mobile memory and compute constraints, identifying an on-device sweet spot: moderate sparsity, with fine-grained experts and shared expert.
• Four-stage recipe. Pre-training → mid-training → instruction fine-tuning → INT4 quantization-aware training, all on open-source datasets.

Model Information

Model: MobileMoE-L-Base (pre-trained + mid-trained)
Active Parameters: 922M
Total Parameters: 5.3B
Layers: 32
Model Dimension: 1280
Attention Heads: 20
KV Heads: 4 (GQA)
Head Dimension: 64
Routed Experts: 60 (fine-grained, FFN hidden dim 640 each)
Active Experts per Token: 4 (top-k sigmoid routing, with normalization)
Shared Expert: 1, always on (FFN hidden dim 2560)
Vocabulary Size: 128,256
Other Features: QK-Norm, tied input/output embeddings, RoPE (θ = 500,000)
Input Modality: Text
Output Modality: Text
Languages: English
Training Stages: Pre-training → mid-training
Context Length: 8,192 tokens
Precision: BF16
Model Developer: Meta
Model Release Date: Aug 2026
License: MobileMoE is FAIR NC licensed

Results

All numbers below are for the base (pre-trained) models, re-evaluated under identical settings with greedy decoding using lm-eval; few-shot counts are given in parentheses after the benchmark name, and benchmarks shown without one are evaluated 0-shot.

Foundational benchmarks

Capability	Benchmark	Gemma 3 1B	MobileLLM-Pro	Llama 3.2 1B	OLMo 2 1B	SmolLM2 1.7B	Qwen3.5 2B	OLMoE-1B-7B	MobileMoE-L
Active / total params		1.0B	1.1B	1.2B	1.5B	1.7B	1.9B	1.3B / 6.9B	922M / 5.3B
Commonsense Reasoning	HellaSwag	62.2	66.2	64.2	68.4	71.4	65.9	77.0	74.6
	PIQA	74.9	76.6	75.1	75.9	77.6	74.7	80.5	80.0
	SIQA	42.9	48.4	42.8	44.0	44.2	43.5	43.9	54.3
	WinoGrande	58.8	63.2	61.3	65.0	66.1	64.6	69.1	68.2
Knowledge	MMLU (5-shot)	26.2	32.3	31.4	42.4	50.2	54.1	52.6	59.6
	NaturalQuestions (5-shot)	10.7	15.6	12.0	14.1	15.4	10.9	20.6	26.7
	TriviaQA (5-shot)	35.7	43.2	40.7	47.1	49.6	32.6	62.3	58.1
Science	ARC-Challenge (25-shot)	39.3	52.5	40.3	45.2	53.3	54.3	55.0	57.0
	ARC-Easy	72.2	76.6	61.8	73.4	73.4	71.4	76.6	81.7
	OpenBookQA	36.8	43.2	36.6	39.8	43.8	37.8	45.0	42.8
Reading	BoolQ	66.6	77.5	63.6	62.9	72.4	69.4	74.8	75.7
	DROP (3-shot)	23.4	22.5	27.9	34.6	27.3	53.6	29.8	64.7
Reasoning	BIG-Bench Hard (3-shot)	30.5	33.0	29.8	33.3	34.0	48.2	33.5	37.8
	GSM8K (8-shot)	2.3	6.6	7.3	38.6	31.0	65.3	12.3	55.7
Average		41.6	47.0	42.5	48.9	50.7	53.3	52.4	59.8

Training

MobileMoE uses a four-stage recipe. This checkpoint is the output of stage 2 (mid-training).

	Pre-training	Mid-training	SFT	QAT
Context length	2,048	8,192	8,192	8,192
Total tokens	~6T	~500B	~126B	~21B
Peak learning rate	4×10-4	4×10-5	4×10-6	4×10-6
LR schedule	Cosine	Linear	Cosine	Cosine
Token dispatch	drop-and-pad	drop-and-pad	dropless	dropless

How to use

MobileMoE uses a custom architecture (model_type: mobilemoe) that is not yet part of upstream transformers, so trust_remote_code=True is required. The modeling code ships in this repo (configuration_mobilemoe.py, modeling_mobilemoe.py).

Requirements

pip install "torch>=2.1" "transformers>=4.57" "safetensors>=0.4" "accelerate>=1.0"

Verified with the following versions:

Package	Version
torch	2.8.0 (cu128)
transformers	4.57.6
tokenizers	0.22.2
safetensors	0.7.0
accelerate	1.13.0

For batch evaluation we recommend vLLM (≥ 0.10.2) with enforce_eager=True.

Text generation

This is a base model — it has no chat template and is not instruction-tuned. Prompt it with plain text continuation:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL_ID = "facebook/MobileMoE-L-Base"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    trust_remote_code=True,
    dtype=torch.bfloat16,
)
model.to("cuda" if torch.cuda.is_available() else "cpu")
model.eval()

prompt = "Why are open-source on-device language models great?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    input_ids=inputs["input_ids"],
    attention_mask=inputs["attention_mask"],
    max_new_tokens=64,
    do_sample=False,
    temperature=None,
    top_p=None,
    pad_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Known issues. Loading the tokenizer on transformers 4.57.6 prints a fix_mistral_regex=True warning. Please ignore it and do not set the flag, as MobileMoE uses the Llama-3 tokenizer whose default tokenization is already correct.

Citation

@article{chen2026mobilemoe,
  title={MobileMoE: Scaling On-Device Mixture of Experts},
  author={Chen, Yanbei and Huang, Hanxian and Chang, Ernie and Szwejbka, Jacob and Desai, Digant and Liu, Zechun and Chandra, Vikas and Krishnamoorthi, Raghuraman},
  journal={arXiv preprint arXiv:2605.27358},
  year={2026}
}

License

MobileMoE is distributed under the FAIR Noncommercial Research License.