ARK-65M

A bilingual Italian/English language model with 65 million parameters, built from scratch by ModotAI. Trained entirely on a single NVIDIA RTX 3070.

Try it live: chat.things-ai.org

---

Overview

ARK-65M is a compact transformer language model designed as a fast, efficient assistant that delegates factual knowledge to web search rather than memorizing it in its weights. At 65M parameters, the model excels at conversation, math reasoning, code generation, and tool use — while relying on a web_search tool for factual questions it cannot reliably store.

Architecture

Component	Detail
Parameters	68,244,768 (65M effective)
Layers	14 (7 GQA + 7 MSA, alternating)
Hidden dim	576
Attention heads	8 (query), 2 (KV, grouped)
Head dim	72
FFN	SwiGLU, 1536 intermediate
Context length	2048 tokens
Vocabulary	32,768 (shared tokenizer)
Positional encoding	RoPE (θ=10000)
Normalization	RMSNorm

The architecture alternates between Grouped Query Attention (GQA) and Multi-Head Self-Attention (MSA) layers, combining the efficiency of GQA with the expressiveness of full attention.

Training

Pretraining

• Tokens: ~6.5 billion (100:1 token-to-parameter ratio)
• Hardware: Single NVIDIA RTX 3070 (8GB VRAM)
• Duration: ~5 days continuous
• Throughput: ~32k tokens/second

Data mix:

Source	Share
FineWeb-Edu (English)	30%
CulturaX IT (Italian)	22%
Italian Common Corpus	13%
Reasoning (Soreqen)	8%
StackExchange	8%
Python/Bash code	~7%
Math CoT (synthetic)	5%
Other	~7%

Supervised Fine-Tuning (SFT)

SFT was performed from the base pretrained model with ~86k examples across 3 epochs:

• UltraChat 200k — multi-turn English conversations (15k)
• Alpaca Cleaned — instruction following (8k)
• Camoscio — Italian instruction following (6k)
• Glaive Function Calling — tool use patterns (15k)
• Pandora Tool Calling — tool use with refusals (5k)
• TriviaQA / NQ Open — factual QA (16k)
• Web Search Tool Call — custom dataset teaching web search delegation (~4.8k)
• Math CoT — synthetic arithmetic with chain-of-thought (~14k)
• Identity + Conversation — custom ARK-65M identity data (~2k)

Capabilities

What it does well

• Identity: Correctly identifies itself as ARK-65M by ModotAI (100% accuracy)
• Math: Arithmetic with step-by-step reasoning via <|thinking|> tags (~83%)
• Tool calling: Generates structured JSON tool calls when appropriate (100%)
• Web search delegation: Routes factual questions to web_search tool instead of hallucinating
• Bilingual: Responds naturally in both Italian and English
• Conversation: Handles greetings, thanks, and basic multi-turn dialogue

What it doesn't do

• Factual recall: With 65M parameters, the model cannot reliably memorize encyclopedic knowledge. It will hallucinate if asked factual questions without the web search tool enabled.
• Long-form generation: Context window is 2048 tokens.
• Complex reasoning: Word problems and multi-step logic are limited.

Usage

Inference (CLI)

python inference.py \
  --checkpoint model.pt \
  --prompt "Chi è Elon Musk?" \
  --system "Sei ARK-65M, un modello linguistico italiano creato da ModotAI. Rispondi in modo utile e preciso. Hai accesso al tool web_search per cercare informazioni fattuali."

Chat format (ChatML)

ARK-65M uses a ChatML-style format with special role tokens:

<|im_start|><|system|>Sei ARK-65M, un modello linguistico italiano creato da ModotAI. Rispondi in modo utile e preciso. Hai accesso al tool web_search per cercare informazioni fattuali.<|im_end|>
<|im_start|><|user|>Chi è Elon Musk?<|im_end|>
<|im_start|><|assistant|><|tool_calls|>[{"name": "web_search", "arguments": {"query": "Elon Musk chi è"}}]<|/tool_calls|><|im_end|>
<|im_start|><|tool|>web_search: Elon Musk è un imprenditore sudafricano naturalizzato statunitense, CEO di Tesla e SpaceX.<|im_end|>
<|im_start|><|assistant|>Elon Musk è un imprenditore sudafricano naturalizzato statunitense. È il CEO di Tesla e SpaceX.<|im_end|>

Special tokens

Token	Purpose
<|im_start|> / <|im_end|>	Message boundaries
<|system|> <|user|> <|assistant|> <|tool|>	Role markers
<|tool_calls|> / <|/tool_calls|>	Tool call JSON wrapper
<|thinking|> / <|/thinking|>	Chain-of-thought (math)

Server deployment

ARK-65M runs as part of the Quark Multi-Model Inference Engine (app.py v14.0) with automatic server-side web search execution:

1. User sends a question
2. Model generates a web_search tool call (if factual)
3. Server executes the search via Serper API
4. Model reformulates the search results into a response
5. User receives the final answer — tool call is transparent

Benchmark (SFT v3, epoch 2)

Category	Score
Identity	100.0%
Tool call	100.0%
Math	83.3%
Code	66.7%
Conversation	66.7%
Reasoning	33.3%
Facts (without web search)	38.9%
Overall	70.5%

With web search enabled, factual accuracy improves dramatically as the model no longer halluccinates — it retrieves real information from the web.

Tokenizer

Uses ThingAI/msqark-tokenizer (32,768 vocab), shared with the Quark-270M model family. The embedding layer accounts for ~27.7% of total parameters — a deliberate tradeoff to reuse an existing tokenizer rather than training a smaller one.

Files

File	Description
model.pt	SFT checkpoint (epoch 2)
ark65m.py	Model architecture
config.json	Model configuration

Hardware requirements

• Inference: Any GPU with ≥2GB VRAM (or CPU)
• SFT: NVIDIA GPU with ≥6GB VRAM
• Pretraining: NVIDIA GPU with ≥8GB VRAM

License

This model is released for research and personal use.

Author

Built by ModotAI — an independent Italian AI research project focused on efficient language models.