PRISM-CTE

PRISM + Adaptive Continuous Thought Engine

Extension of PRISM with a learnable continuous thought loop, trained on thefinalboss/fractus-datasets.

This is not a pure transformer. It keeps the Multi-Rate Bus + polymorphic experts of PRISM and adds adaptive internal thinking, holographic fact injection, and progressive capacity stacking — aligned with the Fractus continuous-thought line of work.

---

What this is

Piece	Role
PRISM core	Multi-Rate Bus (O(n)), polymorphic router (neural / memory / symbolic experts), shared memory tape
CTE	Continuous Thought Engine — recurrent internal steps, accumulation updates, adaptive depth
ThinkController	Learns how much to think (budget + continue logits) during training
Holo bind	VSA-style circular convolution: inject facts into memory without gradient
PCS	Progressive Capacity Stacking — grow width (e.g. 256 to 384) with weight transfer
Thought loss	Aux term: keep effective thought steps above a floor, limit collapse

Training philosophy: not mass CE grind only. Curriculum mix, external knowledge (Holo), capacity growth (PCS), and metrics that match continuous dynamics (effective steps, wall-clock tok/s, generation coherence).

---

Architecture (short)

tokens -> embed -> [ PrismBlock + CTE ] x L -> norm -> lm_head
                      |
                      +-- Multi-Rate Bus
                      +-- Router -> neural | memory | symbolic experts
                      +-- Shared memory tape (read/write)
                      +-- CTE loop (adaptive steps, accumulation)

• Memory tape is working memory (shaped by writes), optionally blended with a holographic store of bound facts.
• CTE is differentiable; the controller is trained jointly with next-token CE.
• Aux loss (bounded): load-balance + soft entropy + thought coherence — total loss tracks CE (no dive to -50).

---

Repos

Repo	Content
GitHub (code)	https://github.com/AFKmoney/prism-cte
HF code	https://huggingface.co/thefinalboss/prism-cte
HF checkpoints	https://huggingface.co/thefinalboss/prism-cte-checkpoints
Upstream PRISM	https://github.com/AFKmoney/prism
Data	https://huggingface.co/datasets/thefinalboss/fractus-datasets

Hourly cron on the training machine uploads live weights + log tails to the checkpoints repo.

---

What we built and ran

1. CTE integration

• prism/cte.py — ThinkController, ContinuousThoughtEngine, PrismBlockWithCTE
• prism/model_cte.py — PrismCTE, continuous trainer hooks, bounded aux + thought loss
• Unit tests: forward shapes, gradients into controller, continuous loop

2. Four training arms

1. Holo (prism/holo.py) — FFT bind/unbind, inject_text_facts, blend into MemoryState (zero-grad on fact content)
2. PCS (prism/pcs.py) — grow state dict 256 to 384 (and beyond) with pad/interp
3. 2x GPU — independent processes per GPU (stable); manual replica path also exists in run_arms.py
4. Thought loss — inside model aux (think floor ~1.5 + mild variance)

3. Data pipeline

• Full Fractus index on disk (memmap, not full RAM load):
  • phase2 npy shards: ~3.44B tokens
  • + pre-tokenized .pt corpora: ~1.5B
  • Total available ~ 4.96B tokens
• Curriculum sampling across phase2 / neuro / paradigms / generated / other

4. Training phases (history)

Phase	What happened
Smoke + real slice	cognitive_skills slice; CE 10.8 to 0.16; real words in generation
Full stream CE	sequential stream; CE to ~0.01 in ~6M tokens (saturated for 18M) — stopped pure grind
Curriculum opt	harder mix; CE stays meaningful
4-arms	Holo + PCS384 + dual GPU + thought loss
Max util (live)	B=96, T=128, 2x RTX 5060 Ti, ~15.1 GB VRAM, ~44k tok/s per GPU, util up to 98%

5. Live metrics (max-util run, indicative)

B=96  T=128  x 2 independent GPUs
~44k tok/s / GPU
CE ~1.2-1.6 (descending)
think ~2.6
VRAM ~15118 / 16311 MiB
tokens seen (per process) ~20M+ and climbing

1 full epoch over ~5B tokens ~ 1.3 day / GPU at this rate. For an ~18M model, useful training is often much less than a full multi-epoch 5B pass (see Chinchilla-scale token budgets).

---

Key scripts

Script	Role
run_maxutil.py	High-throughput training (current default: B=96 T=128, one process per GPU)
run_arms.py	4-arms loop (Holo + PCS + replica GPU + thought)
run_full_corpus.py	Full memmap stream over all shards
run_optimized.py	Curriculum weights + cosine LR + progressive context
run_continue.py / run_real.py	Earlier Fractus real-data loops
train_fractus_style.py	Fractus-style metrics (wall-clock, word ratio, DDP path)
test_cte.py	CTE / controller / continuous trainer tests
hourly_hf_push.sh	Hourly push of hot ckpts + tails (token via env/secret file, not in git)

Core package

prism/
  cte.py          # Adaptive CTE + ThinkController
  model_cte.py    # PrismCTE + aux / thought loss
  holo.py         # Holographic VSA tape
  pcs.py          # Progressive capacity grow
  memory.py       # Shared memory bus
  mrb.py          # Multi-Rate Bus
  router.py       # Polymorphic experts
  block.py model.py config.py ...

---

Quick start

git clone https://github.com/AFKmoney/prism-cte.git
cd prism-cte
pip install torch transformers datasets huggingface_hub

# tests
python test_cte.py

# high-throughput train (adapt data paths to your Fractus shards)
CUDA_VISIBLE_DEVICES=0 python run_maxutil.py
CUDA_VISIBLE_DEVICES=1 python run_maxutil.py   # second GPU

Checkpoints:

from huggingface_hub import hf_hub_download
import torch
path = hf_hub_download("thefinalboss/prism-cte-checkpoints", "prism_cte_hot_g0.pt")
ckpt = torch.load(path, map_location="cpu", weights_only=False)
# ckpt["model"], ckpt.get("steps"), ckpt.get("tokens_seen")

---

Design notes

• Aux scaling (definitive): load-balance x0.02, entropy x0.001 clamped, + thought term — CE and total loss stay aligned.
• PCS grow to 384 worked for capacity experiments; a NaN event corrupted one 384 ckpt — max-util continued from clean 256 fractus_full weights. Re-grow when stable.
• Tied embeddings + DataParallel was unreliable; prefer one process per GPU or proper DDP.
• Holo facts are knowledge side-channels: the model still learns to use the tape; bound content itself is not backpropped.

---

Status

• Adaptive CTE trainable end-to-end
• Bounded aux + thought loss
• Full ~4.96B token pipeline (memmap)
• Holo inject + PCS utilities
• Dual GPU max-util (~98% util, ~15GB VRAM)
• HF + GitHub code; hourly checkpoint uploads
• Longer multi-epoch sweep / larger PCS once stable
• Cleaner torchrun DDP
• Richer Holo fact packs from Fractus cognitive skills

---

License / context

MIT (same spirit as upstream PRISM). Built for the Fractus continuous-thought stack (AFKmoney / thefinalboss).