Release: SpikeWhale slider panel (HF dataset picker, stop/back), DaisyChain-Web (P2P WebRTC training, DaisyAdam, checkpoints, room host approval, verified-units-only)
Browse files- README.md +67 -39
- daisychain/spikewhale_panel.py +185 -0
- daisychain/spikewhale_task.py +132 -0
- export_luts_web.py +44 -0
- web/.gitignore +3 -0
- web/LICENSE +21 -0
- web/README.md +92 -0
- web/package-lock.json +37 -0
- web/package.json +14 -0
- web/public/app.js +334 -0
- web/public/index.html +123 -0
- web/public/luts_meta.json +1 -0
- web/public/mul_lut.bin +3 -0
- web/public/relu_lut.bin +3 -0
- web/public/requant_lut.bin +3 -0
- web/public/traincore.js +91 -0
- web/public/verified_core.js +84 -0
- web/public/webgpu.js +91 -0
- web/server.js +111 -0
- web/test_core.js +54 -0
- web/test_optimizer.js +63 -0
- web/test_verified.js +48 -0
README.md
CHANGED
|
@@ -3,22 +3,23 @@ license: mit
|
|
| 3 |
tags:
|
| 4 |
- distributed-training
|
| 5 |
- old-hardware
|
| 6 |
-
-
|
| 7 |
-
-
|
| 8 |
-
-
|
| 9 |
---
|
| 10 |
|
| 11 |
-
# 🌼 DaisyChain — Old Hardware Training Pipeline
|
| 12 |
-
|
| 13 |
-
> **Whats available is currently the template. The Main Core Update will be uploaded later on today 7/12**
|
| 14 |
|
|
|
|
|
|
|
| 15 |
|
|
|
|
| 16 |
|
| 17 |
-
> **In plain terms:** DaisyChain lets you use **old / spare machines** to
|
| 18 |
-
> neural networks. The training runs through **emulated GPU logic** —
|
| 19 |
-
> INT8 units (GUDA-style) that stand in for a GPU's math — so machines
|
| 20 |
-
> a modern GPU can still do the work. Chain several together and they
|
| 21 |
-
> shared model as a cluster.
|
| 22 |
> Before you rely on it, see what it **can't** do → [Limitations](docs/LIMITS.md).
|
| 23 |
|
| 24 |
**Use the hardware you already have to train.** Each machine runs the emulated
|
|
@@ -27,17 +28,12 @@ model, and DaisyChain pools the machines data-parallel: device selection,
|
|
| 27 |
capacity-weighted sharding, gradient sync, a P2P setup, and a live dashboard.
|
| 28 |
Two ways to run — **Docker** or **Python**.
|
| 29 |
|
| 30 |
-
> Built by **DaisyChainAI**. Point it at your model + data and it trains across
|
| 31 |
-
> whatever old machines you have, through the emulated GPU logic.
|
| 32 |
-
|
| 33 |
-
**Repositories:** [GitHub](https://github.com/quzi93/DaisyChain-Train) · [🤗 HuggingFace](https://huggingface.co/DaisyChainAI/DaisyChain-Train)
|
| 34 |
-
|
| 35 |
---
|
| 36 |
|
| 37 |
## ⚠️ Read this first
|
| 38 |
-
DaisyChain is for **small models on spare hardware**. It **pools compute,
|
| 39 |
-
memory** (the model must fit on one
|
| 40 |
-
**not** a substitute for a real GPU on
|
| 41 |
**[docs/LIMITS.md](docs/LIMITS.md)** — please read it before relying on it.
|
| 42 |
|
| 43 |
---
|
|
@@ -52,7 +48,7 @@ docker compose -f docker/docker-compose.yml up --build
|
|
| 52 |
Brings up a 3-node demo cluster + dashboard on one machine.
|
| 53 |
|
| 54 |
### Python (real machines)
|
| 55 |
-
On every machine (`pip install
|
| 56 |
```bash
|
| 57 |
export MASTER_ADDR=100.101.102.10 # coordinator IP (Tailscale 100.x recommended)
|
| 58 |
export MASTER_PORT=29560
|
|
@@ -70,6 +66,33 @@ An interactive menu: Docker, Python node, or just install deps.
|
|
| 70 |
|
| 71 |
Full walkthrough: **[docs/QUICKSTART.md](docs/QUICKSTART.md)**.
|
| 72 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 73 |
---
|
| 74 |
|
| 75 |
## How it works
|
|
@@ -79,8 +102,8 @@ model. Two things happen:
|
|
| 79 |
|
| 80 |
1. **The compute runs through the emulated GPU logic.** By default the model is
|
| 81 |
built from `VerifiedLinear` layers, so every forward multiply / requantize /
|
| 82 |
-
ReLU is done by the **bundled verified INT8 units** (`daisychain/verified/`)
|
| 83 |
-
|
| 84 |
so you can see the emulated logic doing the work.
|
| 85 |
2. **The machines are pooled data-parallel.** Each node trains on its own shard;
|
| 86 |
gradients are capacity-weighted and combined into the exact full-batch
|
|
@@ -94,42 +117,45 @@ model. Two things happen:
|
|
| 94 |
```
|
| 95 |
|
| 96 |
## Bring your own model
|
| 97 |
-
DaisyChain trains any **Task** (`build_model` / `sample` / `loss`). Copy
|
| 98 |
`examples/my_task_template.py`, set `DAISY_TASK=your_module:YourTask`. Use
|
| 99 |
`VerifiedLinear` (see `daisychain/verified_task.py`) to run your model's compute
|
| 100 |
through the emulated units. See **[docs/CUSTOM_TASK.md](docs/CUSTOM_TASK.md)**.
|
| 101 |
|
| 102 |
## Plain-float alternative
|
| 103 |
-
|
| 104 |
-
|
| 105 |
-
|
| 106 |
-
verified units.
|
| 107 |
|
| 108 |
## The dashboard
|
| 109 |
-
`daisychain-dashboard` (or the Docker service) serves a Tailwind page at
|
| 110 |
-
|
| 111 |
-
|
| 112 |
|
| 113 |
## Networking
|
| 114 |
-
Use **Tailscale** for a P2P mesh so machines on different networks get stable
|
| 115 |
-
|
| 116 |
|
| 117 |
---
|
| 118 |
|
| 119 |
## Layout
|
| 120 |
```
|
| 121 |
daisychain/cluster.py capacity-weighted CPU/GPU data-parallel trainer
|
| 122 |
-
daisychain/task.py the Task interface + loader
|
| 123 |
daisychain/train.py entry point (daisychain-train)
|
| 124 |
-
daisychain/example_task.py default runnable task (plain float)
|
| 125 |
daisychain/verified/ bundled trained N/N units + VerifiedLinear (train through them)
|
| 126 |
-
daisychain/verified_task.py
|
|
|
|
|
|
|
| 127 |
daisychain/dashboard/ agent + P2P scanner + Tailwind server
|
| 128 |
docker/ Dockerfile, dashboard image, compose (demo cluster)
|
| 129 |
scripts/setup.bat / setup.sh interactive setup helpers
|
| 130 |
config/ nodes + cluster env examples
|
| 131 |
examples/my_task_template.py starting point for your own model
|
| 132 |
docs/ QUICKSTART, LIMITS, CUSTOM_TASK, TAILSCALE
|
|
|
|
|
|
|
|
|
|
|
|
|
| 133 |
```
|
| 134 |
|
| 135 |
## Install
|
|
@@ -137,19 +163,21 @@ docs/ QUICKSTART, LIMITS, CUSTOM_TASK, TAILSCALE
|
|
| 137 |
pip install torch numpy psutil
|
| 138 |
pip install -e . # exposes: daisychain-train, daisychain-agent, daisychain-dashboard
|
| 139 |
```
|
| 140 |
-
|
| 141 |
Requires Python ≥ 3.9, PyTorch ≥ 2.0. Multi-node is reliable on **Linux/macOS**;
|
| 142 |
-
on **Windows use Docker/WSL** (see LIMITS).
|
| 143 |
|
| 144 |
---
|
| 145 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 146 |
**License:** MIT · **Author:** Dean Byrne (Quazim0t0) · **Org:** DaisyChainAI
|
| 147 |
|
| 148 |
## Citation
|
| 149 |
-
|
| 150 |
```bibtex
|
| 151 |
@misc{byrne2026daisychain,
|
| 152 |
-
title = {DaisyChain: An Old Hardware Training Pipeline},
|
| 153 |
author = {Byrne, Dean (Quazim0t0)},
|
| 154 |
year = {2026},
|
| 155 |
howpublished = {\url{https://huggingface.co/DaisyChainAI/DaisyChain-Train}},
|
|
|
|
| 3 |
tags:
|
| 4 |
- distributed-training
|
| 5 |
- old-hardware
|
| 6 |
+
- int8
|
| 7 |
+
- webgpu
|
| 8 |
+
- webrtc
|
| 9 |
---
|
| 10 |
|
| 11 |
+
# 🌼 DaisyChain-Train — Old Hardware Training Pipeline
|
|
|
|
|
|
|
| 12 |
|
| 13 |
+
**Part of DaisyChain on 🤗 Hugging Face → https://huggingface.co/DaisyChainAI**
|
| 14 |
+
Model page (weights + card): https://huggingface.co/DaisyChainAI/DaisyChain-Train
|
| 15 |
|
| 16 |
+
---
|
| 17 |
|
| 18 |
+
> **In plain terms:** DaisyChain-Train lets you use **old / spare machines** to
|
| 19 |
+
> train neural networks. The training runs through **emulated GPU logic** —
|
| 20 |
+
> verified INT8 units (GUDA-style) that stand in for a GPU's math — so machines
|
| 21 |
+
> *without* a modern GPU can still do the work. Chain several together and they
|
| 22 |
+
> train one shared model as a cluster.
|
| 23 |
> Before you rely on it, see what it **can't** do → [Limitations](docs/LIMITS.md).
|
| 24 |
|
| 25 |
**Use the hardware you already have to train.** Each machine runs the emulated
|
|
|
|
| 28 |
capacity-weighted sharding, gradient sync, a P2P setup, and a live dashboard.
|
| 29 |
Two ways to run — **Docker** or **Python**.
|
| 30 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 31 |
---
|
| 32 |
|
| 33 |
## ⚠️ Read this first
|
| 34 |
+
DaisyChain-Train is for **small models on spare hardware**. It **pools compute,
|
| 35 |
+
not memory** (the model must fit on one machine), scaling is **sublinear**, and
|
| 36 |
+
it is **not** a substitute for a real GPU on large models. Full envelope in
|
| 37 |
**[docs/LIMITS.md](docs/LIMITS.md)** — please read it before relying on it.
|
| 38 |
|
| 39 |
---
|
|
|
|
| 48 |
Brings up a 3-node demo cluster + dashboard on one machine.
|
| 49 |
|
| 50 |
### Python (real machines)
|
| 51 |
+
On every machine (`pip install -e .`):
|
| 52 |
```bash
|
| 53 |
export MASTER_ADDR=100.101.102.10 # coordinator IP (Tailscale 100.x recommended)
|
| 54 |
export MASTER_PORT=29560
|
|
|
|
| 66 |
|
| 67 |
Full walkthrough: **[docs/QUICKSTART.md](docs/QUICKSTART.md)**.
|
| 68 |
|
| 69 |
+
### 🐋 SpikeWhale control panel (sliders → real training)
|
| 70 |
+
```bash
|
| 71 |
+
python -m daisychain.spikewhale_panel
|
| 72 |
+
# open http://localhost:8899
|
| 73 |
+
```
|
| 74 |
+
A web control panel: pick model size / training settings with sliders, choose any
|
| 75 |
+
HuggingFace dataset you have access to (default: streamed FineWeb-Edu), hit
|
| 76 |
+
Start, and watch the live loss. Stop and re-adjust any time with
|
| 77 |
+
**← Back to settings**. Launches the real DaisyChain training underneath.
|
| 78 |
+
|
| 79 |
+
### 🌐 DaisyChain-Web (train by opening a browser tab)
|
| 80 |
+
```bash
|
| 81 |
+
cd web && npm install && node server.js
|
| 82 |
+
# open http://localhost:8787 on every device
|
| 83 |
+
```
|
| 84 |
+
Zero-install browser training: devices on the same network auto-group
|
| 85 |
+
(Snapdrop-style) and train a shared model **peer-to-peer over WebRTC**, computing
|
| 86 |
+
through the same verified INT8 units (WebGPU, with the identical units on CPU
|
| 87 |
+
for machines without it — there is no plain-float path). Private cross-network
|
| 88 |
+
rooms via `?room=CODE` with **host approval** — the room creator accepts each
|
| 89 |
+
device before it can join. Includes gradient averaging with a deterministic
|
| 90 |
+
Adam optimizer (identical state on every peer, nothing extra over the wire),
|
| 91 |
+
checkpoint **download** (.pt) and **upload → broadcast** so one device can
|
| 92 |
+
restore the whole group after a failure.
|
| 93 |
+
|
| 94 |
+
**Live demo:** https://huggingface.co/spaces/DaisyChainAI/DaisyChain-Web
|
| 95 |
+
|
| 96 |
---
|
| 97 |
|
| 98 |
## How it works
|
|
|
|
| 102 |
|
| 103 |
1. **The compute runs through the emulated GPU logic.** By default the model is
|
| 104 |
built from `VerifiedLinear` layers, so every forward multiply / requantize /
|
| 105 |
+
ReLU is done by the **bundled verified INT8 units** (`daisychain/verified/`) —
|
| 106 |
+
the emulated GPU math. Rank 0 prints **cluster-wide unit-invocation counts**
|
| 107 |
so you can see the emulated logic doing the work.
|
| 108 |
2. **The machines are pooled data-parallel.** Each node trains on its own shard;
|
| 109 |
gradients are capacity-weighted and combined into the exact full-batch
|
|
|
|
| 117 |
```
|
| 118 |
|
| 119 |
## Bring your own model
|
| 120 |
+
DaisyChain-Train trains any **Task** (`build_model` / `sample` / `loss`). Copy
|
| 121 |
`examples/my_task_template.py`, set `DAISY_TASK=your_module:YourTask`. Use
|
| 122 |
`VerifiedLinear` (see `daisychain/verified_task.py`) to run your model's compute
|
| 123 |
through the emulated units. See **[docs/CUSTOM_TASK.md](docs/CUSTOM_TASK.md)**.
|
| 124 |
|
| 125 |
## Plain-float alternative
|
| 126 |
+
To skip the emulated units and train with normal float math on each machine, set
|
| 127 |
+
`DAISY_TASK=daisychain.example_task:ExampleTask`. Same cluster, same pooling —
|
| 128 |
+
the model math just runs as ordinary float instead of through the verified units.
|
|
|
|
| 129 |
|
| 130 |
## The dashboard
|
| 131 |
+
`daisychain-dashboard` (or the Docker service) serves a Tailwind page at `:8080`
|
| 132 |
+
— readiness banner, P2P connectivity scan, pooled cores/RAM + capacity plan
|
| 133 |
+
(per-node device, weight, batch), and live training loss.
|
| 134 |
|
| 135 |
## Networking
|
| 136 |
+
Use **Tailscale** for a P2P mesh so machines on different networks get stable IPs
|
| 137 |
+
on one interface — **[docs/TAILSCALE.md](docs/TAILSCALE.md)**.
|
| 138 |
|
| 139 |
---
|
| 140 |
|
| 141 |
## Layout
|
| 142 |
```
|
| 143 |
daisychain/cluster.py capacity-weighted CPU/GPU data-parallel trainer
|
|
|
|
| 144 |
daisychain/train.py entry point (daisychain-train)
|
|
|
|
| 145 |
daisychain/verified/ bundled trained N/N units + VerifiedLinear (train through them)
|
| 146 |
+
daisychain/verified_task.py default task: forward runs on the verified units
|
| 147 |
+
daisychain/example_task.py plain-float alternative task
|
| 148 |
+
daisychain/task.py the Task interface + loader
|
| 149 |
daisychain/dashboard/ agent + P2P scanner + Tailwind server
|
| 150 |
docker/ Dockerfile, dashboard image, compose (demo cluster)
|
| 151 |
scripts/setup.bat / setup.sh interactive setup helpers
|
| 152 |
config/ nodes + cluster env examples
|
| 153 |
examples/my_task_template.py starting point for your own model
|
| 154 |
docs/ QUICKSTART, LIMITS, CUSTOM_TASK, TAILSCALE
|
| 155 |
+
daisychain/spikewhale_task.py trains the real SpikeWhale on streamed HF datasets
|
| 156 |
+
daisychain/spikewhale_panel.py slider control panel (localhost:8899)
|
| 157 |
+
web/ DaisyChain-Web: P2P browser training (WebRTC + WebGPU)
|
| 158 |
+
export_luts_web.py regenerates web/public LUTs from the trained units
|
| 159 |
```
|
| 160 |
|
| 161 |
## Install
|
|
|
|
| 163 |
pip install torch numpy psutil
|
| 164 |
pip install -e . # exposes: daisychain-train, daisychain-agent, daisychain-dashboard
|
| 165 |
```
|
|
|
|
| 166 |
Requires Python ≥ 3.9, PyTorch ≥ 2.0. Multi-node is reliable on **Linux/macOS**;
|
| 167 |
+
on **Windows use Docker/WSL** (see [Limitations](docs/LIMITS.md)).
|
| 168 |
|
| 169 |
---
|
| 170 |
|
| 171 |
+
## Links
|
| 172 |
+
- **DaisyChain on Hugging Face:** https://huggingface.co/DaisyChainAI
|
| 173 |
+
- **This model:** https://huggingface.co/DaisyChainAI/DaisyChain-Train
|
| 174 |
+
|
| 175 |
**License:** MIT · **Author:** Dean Byrne (Quazim0t0) · **Org:** DaisyChainAI
|
| 176 |
|
| 177 |
## Citation
|
|
|
|
| 178 |
```bibtex
|
| 179 |
@misc{byrne2026daisychain,
|
| 180 |
+
title = {DaisyChain-Train: An Old Hardware Training Pipeline},
|
| 181 |
author = {Byrne, Dean (Quazim0t0)},
|
| 182 |
year = {2026},
|
| 183 |
howpublished = {\url{https://huggingface.co/DaisyChainAI/DaisyChain-Train}},
|
daisychain/spikewhale_panel.py
ADDED
|
@@ -0,0 +1,185 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""SpikeWhale training control panel (CLI: python -m daisychain.spikewhale_panel).
|
| 2 |
+
|
| 3 |
+
A web page with sliders for the SpikeWhale config. Pick a size your hardware can
|
| 4 |
+
handle, hit Start, and it launches the real DaisyChain training (SpikeWhale +
|
| 5 |
+
FineWeb-Edu) and streams the live loss. The exact env is shown so you can run the
|
| 6 |
+
same command on other machines to train distributed.
|
| 7 |
+
"""
|
| 8 |
+
import json
|
| 9 |
+
import os
|
| 10 |
+
import subprocess
|
| 11 |
+
import sys
|
| 12 |
+
import threading
|
| 13 |
+
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
| 14 |
+
|
| 15 |
+
PORT = int(os.environ.get("SW_PANEL_PORT", "8899"))
|
| 16 |
+
_proc = None
|
| 17 |
+
_log = [] # rolling training log
|
| 18 |
+
_lock = threading.Lock()
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
def _pump(proc):
|
| 22 |
+
for line in iter(proc.stdout.readline, ""):
|
| 23 |
+
with _lock:
|
| 24 |
+
_log.append(line.rstrip("\n"))
|
| 25 |
+
if len(_log) > 400:
|
| 26 |
+
del _log[:200]
|
| 27 |
+
proc.stdout.close()
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
def start_training(cfg):
|
| 31 |
+
global _proc
|
| 32 |
+
if _proc and _proc.poll() is None:
|
| 33 |
+
return False, "already running"
|
| 34 |
+
env = dict(os.environ)
|
| 35 |
+
env.update({
|
| 36 |
+
"MASTER_ADDR": "127.0.0.1", "MASTER_PORT": "29610", "WORLD_SIZE": "1",
|
| 37 |
+
"RANK": "0", "USE_LIBUV": "0", "PYTHONUNBUFFERED": "1",
|
| 38 |
+
"DAISY_TASK": "daisychain.spikewhale_task:SpikeWhaleTask",
|
| 39 |
+
"DAISY_SW_HIDDEN": str(cfg["hidden"]), "DAISY_SW_LAYERS": str(cfg["layers"]),
|
| 40 |
+
"DAISY_SW_HEADS": str(cfg["heads"]), "DAISY_SW_EXPERTS": str(cfg["experts"]),
|
| 41 |
+
"DAISY_SW_SEQLEN": str(cfg["seqlen"]),
|
| 42 |
+
"DAISY_STEPS": str(cfg["steps"]), "DAISY_LR": str(cfg["lr"]),
|
| 43 |
+
"DAISY_OPTIMIZER": "adam", "DAISY_BASE_BATCH": str(cfg["batch"]),
|
| 44 |
+
})
|
| 45 |
+
if cfg.get("dataset"):
|
| 46 |
+
env["DAISY_SW_DATASET"] = cfg["dataset"]
|
| 47 |
+
# blank subset means the dataset's default config; unset any inherited one
|
| 48 |
+
if cfg.get("subset"):
|
| 49 |
+
env["DAISY_SW_SUBSET"] = cfg["subset"]
|
| 50 |
+
else:
|
| 51 |
+
env.pop("DAISY_SW_SUBSET", None)
|
| 52 |
+
with _lock:
|
| 53 |
+
_log.clear()
|
| 54 |
+
_log.append(f"launching training: hidden={cfg['hidden']} layers={cfg['layers']} "
|
| 55 |
+
f"experts={cfg['experts']} seqlen={cfg['seqlen']} steps={cfg['steps']} "
|
| 56 |
+
f"dataset={env.get('DAISY_SW_DATASET', 'HuggingFaceFW/fineweb-edu')}"
|
| 57 |
+
+ (f":{env['DAISY_SW_SUBSET']}" if env.get("DAISY_SW_SUBSET") else ""))
|
| 58 |
+
_proc = subprocess.Popen([sys.executable, "-u", "-m", "daisychain.train"],
|
| 59 |
+
env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
|
| 60 |
+
text=True, bufsize=1)
|
| 61 |
+
threading.Thread(target=_pump, args=(_proc,), daemon=True).start()
|
| 62 |
+
return True, "started"
|
| 63 |
+
|
| 64 |
+
|
| 65 |
+
PAGE = """<!doctype html><html><head><meta charset="utf-8">
|
| 66 |
+
<meta name="viewport" content="width=device-width, initial-scale=1">
|
| 67 |
+
<title>SpikeWhale · DaisyChain trainer</title>
|
| 68 |
+
<style>
|
| 69 |
+
body{font-family:system-ui,-apple-system,Segoe UI,Roboto,sans-serif;max-width:720px;margin:0 auto;
|
| 70 |
+
padding:22px;background:#efe4c9;color:#2a1d0a}
|
| 71 |
+
@media(prefers-color-scheme:dark){body{background:#14100a;color:#ede1c3}}
|
| 72 |
+
h1{margin:0 0 2px}.sub{color:#6b4423;margin:0 0 16px}
|
| 73 |
+
@media(prefers-color-scheme:dark){.sub{color:#c9b072}}
|
| 74 |
+
.card{background:#fbf6e8;border:1px solid rgba(139,111,71,.3);border-radius:10px;padding:16px;margin:12px 0}
|
| 75 |
+
@media(prefers-color-scheme:dark){.card{background:#1f1a12;border-color:rgba(201,176,114,.35)}}
|
| 76 |
+
label{display:flex;justify-content:space-between;font-size:.92rem;margin:10px 0 4px;font-weight:600}
|
| 77 |
+
input[type=range]{width:100%;accent-color:#4a7c2e}
|
| 78 |
+
input[type=text]{width:100%;padding:8px 10px;border-radius:8px;border:1px solid rgba(139,111,71,.4);
|
| 79 |
+
background:rgba(255,255,255,.5);color:inherit;font-family:'Courier New',monospace;font-size:.9rem}
|
| 80 |
+
@media(prefers-color-scheme:dark){input[type=text]{background:rgba(0,0,0,.25);border-color:rgba(201,176,114,.35)}}
|
| 81 |
+
.val{font-family:'Courier New',monospace;color:#4a7c2e;font-weight:700}
|
| 82 |
+
@media(prefers-color-scheme:dark){.val{color:#9bc466}}
|
| 83 |
+
button{background:linear-gradient(135deg,#4a7c2e,#2d5016);color:#f5ecd9;border:0;border-radius:8px;
|
| 84 |
+
padding:12px 26px;font-weight:800;font-size:1rem;cursor:pointer}
|
| 85 |
+
.num{font-family:'Courier New',monospace;font-size:2rem;font-weight:700;text-align:center;
|
| 86 |
+
color:#f5ecd9;background:linear-gradient(135deg,#2d5016,#1f3a0f);border-radius:10px;padding:14px}
|
| 87 |
+
pre{background:rgba(0,0,0,.06);border-radius:8px;padding:10px;max-height:220px;overflow:auto;
|
| 88 |
+
font-size:.78rem;white-space:pre-wrap;font-family:'Courier New',monospace}
|
| 89 |
+
@media(prefers-color-scheme:dark){pre{background:rgba(0,0,0,.3)}}
|
| 90 |
+
.lbl{font-size:11px;font-weight:800;letter-spacing:1.5px;text-transform:uppercase;color:#6b4423;margin-bottom:8px}
|
| 91 |
+
@media(prefers-color-scheme:dark){.lbl{color:#c9b072}}
|
| 92 |
+
</style></head><body>
|
| 93 |
+
<h1>🐋 SpikeWhale · DaisyChain</h1>
|
| 94 |
+
<p class="sub">Pick a size your hardware can train, then start. Trains the real SpikeWhale on streamed FineWeb-Edu, distributed by DaisyChain. Smaller = faster on old hardware.</p>
|
| 95 |
+
<div id="settings">
|
| 96 |
+
<div class="card"><div class="lbl">Model size</div>
|
| 97 |
+
<label>Hidden size <span class="val" id="vhidden">256</span></label><input type="range" id="hidden" min="64" max="768" step="64" value="256">
|
| 98 |
+
<label>Layers <span class="val" id="vlayers">4</span></label><input type="range" id="layers" min="1" max="12" step="1" value="4">
|
| 99 |
+
<label>Attention heads <span class="val" id="vheads">4</span></label><input type="range" id="heads" min="1" max="8" step="1" value="4">
|
| 100 |
+
<label>MoE experts <span class="val" id="vexperts">4</span></label><input type="range" id="experts" min="1" max="8" step="1" value="4">
|
| 101 |
+
<label>Sequence length <span class="val" id="vseqlen">128</span></label><input type="range" id="seqlen" min="32" max="512" step="32" value="128">
|
| 102 |
+
</div>
|
| 103 |
+
<div class="card"><div class="lbl">Training</div>
|
| 104 |
+
<label>Learning rate ×1e-4 <span class="val" id="vlr">30</span></label><input type="range" id="lr" min="1" max="100" step="1" value="30">
|
| 105 |
+
<label>Batch (per step) <span class="val" id="vbatch">4</span></label><input type="range" id="batch" min="1" max="16" step="1" value="4">
|
| 106 |
+
<label>Steps <span class="val" id="vsteps">200</span></label><input type="range" id="steps" min="20" max="2000" step="20" value="200">
|
| 107 |
+
</div>
|
| 108 |
+
<div class="card"><div class="lbl">Data</div>
|
| 109 |
+
<label for="dataset">HuggingFace dataset</label>
|
| 110 |
+
<input type="text" id="dataset" value="HuggingFaceFW/fineweb-edu" spellcheck="false">
|
| 111 |
+
<label for="subset">Config / subset <span style="font-weight:400">(blank = default)</span></label>
|
| 112 |
+
<input type="text" id="subset" value="sample-10BT" spellcheck="false">
|
| 113 |
+
<p class="sub" style="margin:.6rem 0 0;font-size:.82rem">Any streamable text dataset with a <code>text</code> column works. For gated/private datasets, log in first with <code>huggingface-cli login</code> on this machine — the trainer inherits your token.</p>
|
| 114 |
+
</div>
|
| 115 |
+
</div>
|
| 116 |
+
<div class="card" style="text-align:center">
|
| 117 |
+
<button id="startbtn" onclick="start()">Start training</button>
|
| 118 |
+
<button id="backbtn" onclick="goBack()" style="display:none;background:linear-gradient(135deg,#6b4423,#4a2f18)">← Back to settings</button>
|
| 119 |
+
<p class="sub" style="margin:.6rem 0 0" id="status">idle</p></div>
|
| 120 |
+
<div class="card"><div class="lbl">Live loss</div><div class="num" id="loss">—</div></div>
|
| 121 |
+
<div class="card"><div class="lbl">Log</div><pre id="log"></pre></div>
|
| 122 |
+
<script>
|
| 123 |
+
const ids=["hidden","layers","heads","experts","seqlen","lr","batch","steps"];
|
| 124 |
+
ids.forEach(k=>{const el=document.getElementById(k);const v=document.getElementById("v"+k);
|
| 125 |
+
el.oninput=()=>v.textContent=el.value;});
|
| 126 |
+
function cfg(){const c={};ids.forEach(k=>c[k]=+document.getElementById(k).value);c.lr=c.lr/1e4;
|
| 127 |
+
c.dataset=document.getElementById("dataset").value.trim();
|
| 128 |
+
c.subset=document.getElementById("subset").value.trim();return c;}
|
| 129 |
+
function showSettings(on){document.getElementById("settings").style.display=on?"":"none";
|
| 130 |
+
document.getElementById("startbtn").style.display=on?"":"none";
|
| 131 |
+
document.getElementById("backbtn").style.display=on?"none":"";}
|
| 132 |
+
async function start(){document.getElementById("status").textContent="starting…";
|
| 133 |
+
showSettings(false);
|
| 134 |
+
await fetch("/start",{method:"POST",body:JSON.stringify(cfg())});}
|
| 135 |
+
async function goBack(){await fetch("/stop",{method:"POST"});
|
| 136 |
+
showSettings(true);document.getElementById("status").textContent="stopped — adjust and start again";}
|
| 137 |
+
async function poll(){try{const r=await fetch("/log");const d=await r.json();
|
| 138 |
+
document.getElementById("log").textContent=d.log.slice().reverse().join("\\n");
|
| 139 |
+
let last="—";for(const l of d.log){const m=l.match(/cluster-avg loss ([0-9.]+)/);if(m)last=m[1];}
|
| 140 |
+
document.getElementById("loss").textContent=last;
|
| 141 |
+
if(document.getElementById("backbtn").style.display!=="none")
|
| 142 |
+
document.getElementById("status").textContent=d.running?"training…":"idle / done";}catch(e){}}
|
| 143 |
+
setInterval(poll,1000);poll();
|
| 144 |
+
</script></body></html>"""
|
| 145 |
+
|
| 146 |
+
|
| 147 |
+
class H(BaseHTTPRequestHandler):
|
| 148 |
+
def _send(self, body, ctype="text/html; charset=utf-8", code=200):
|
| 149 |
+
b = body.encode() if isinstance(body, str) else body
|
| 150 |
+
self.send_response(code); self.send_header("Content-Type", ctype)
|
| 151 |
+
self.send_header("Content-Length", str(len(b))); self.end_headers(); self.wfile.write(b)
|
| 152 |
+
|
| 153 |
+
def do_GET(self):
|
| 154 |
+
if self.path.startswith("/log"):
|
| 155 |
+
with _lock:
|
| 156 |
+
running = _proc is not None and _proc.poll() is None
|
| 157 |
+
self._send(json.dumps({"log": list(_log), "running": running}), "application/json")
|
| 158 |
+
else:
|
| 159 |
+
self._send(PAGE)
|
| 160 |
+
|
| 161 |
+
def do_POST(self):
|
| 162 |
+
if self.path.startswith("/start"):
|
| 163 |
+
n = int(self.headers.get("Content-Length", 0))
|
| 164 |
+
cfg = json.loads(self.rfile.read(n) or "{}")
|
| 165 |
+
ok, msg = start_training(cfg)
|
| 166 |
+
self._send(json.dumps({"ok": ok, "msg": msg}), "application/json")
|
| 167 |
+
elif self.path.startswith("/stop"):
|
| 168 |
+
global _proc
|
| 169 |
+
if _proc and _proc.poll() is None:
|
| 170 |
+
_proc.terminate()
|
| 171 |
+
with _lock:
|
| 172 |
+
_log.append("training stopped from the panel")
|
| 173 |
+
self._send(json.dumps({"ok": True}), "application/json")
|
| 174 |
+
|
| 175 |
+
def log_message(self, *a):
|
| 176 |
+
pass
|
| 177 |
+
|
| 178 |
+
|
| 179 |
+
def main():
|
| 180 |
+
print(f"[spikewhale-panel] http://localhost:{PORT}", flush=True)
|
| 181 |
+
ThreadingHTTPServer(("0.0.0.0", PORT), H).serve_forever()
|
| 182 |
+
|
| 183 |
+
|
| 184 |
+
if __name__ == "__main__":
|
| 185 |
+
main()
|
daisychain/spikewhale_task.py
ADDED
|
@@ -0,0 +1,132 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""DaisyChain Task that trains the REAL SpikeWhale model on streamed FineWeb-Edu.
|
| 2 |
+
|
| 3 |
+
No reimplementation: it imports your actual `model_v2.SpikeWhaleLM` + `config.
|
| 4 |
+
SpikeWhaleConfig` + `spike_tokenizer.SpikeTokenizer`, builds a size chosen by the
|
| 5 |
+
sliders (env vars), and streams FineWeb-Edu for data. DaisyChain then distributes
|
| 6 |
+
it across machines; matmuls can be routed through the verified/GUDA units.
|
| 7 |
+
|
| 8 |
+
Point DaisyChain at it:
|
| 9 |
+
DAISY_TASK=daisychain.spikewhale_task:SpikeWhaleTask daisychain-train
|
| 10 |
+
|
| 11 |
+
Sliders (env):
|
| 12 |
+
DAISY_SW_PATH folder holding model_v2.py/config.py/tokenizer.json
|
| 13 |
+
DAISY_SW_HIDDEN hidden_size (default 256)
|
| 14 |
+
DAISY_SW_LAYERS num_hidden_layers (default 4)
|
| 15 |
+
DAISY_SW_HEADS num_attention_heads (default 4)
|
| 16 |
+
DAISY_SW_EXPERTS n_routed_experts (default 4)
|
| 17 |
+
DAISY_SW_SEQLEN sequence length (default 128)
|
| 18 |
+
DAISY_SW_MTP MTP heads (0=off) (default 0)
|
| 19 |
+
DAISY_SW_DATASET HF dataset (default HuggingFaceFW/fineweb-edu)
|
| 20 |
+
DAISY_SW_SUBSET dataset config name (default sample-10BT)
|
| 21 |
+
"""
|
| 22 |
+
import os
|
| 23 |
+
import sys
|
| 24 |
+
|
| 25 |
+
import torch
|
| 26 |
+
|
| 27 |
+
_DEF_PATH = os.environ.get("DAISY_SW_PATH", r"C:\Users\quaz\Desktop\Spikewhale")
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
def _import_spikewhale():
|
| 31 |
+
if _DEF_PATH not in sys.path:
|
| 32 |
+
sys.path.insert(0, _DEF_PATH)
|
| 33 |
+
from config import SpikeWhaleConfig
|
| 34 |
+
from model_v2 import SpikeWhaleLM
|
| 35 |
+
from spike_tokenizer import SpikeTokenizer
|
| 36 |
+
return SpikeWhaleConfig, SpikeWhaleLM, SpikeTokenizer
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
def _envi(k, d):
|
| 40 |
+
return int(os.environ.get(k, d))
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
def build_config():
|
| 44 |
+
SpikeWhaleConfig, _, _ = _import_spikewhale()
|
| 45 |
+
hidden = _envi("DAISY_SW_HIDDEN", 256)
|
| 46 |
+
return SpikeWhaleConfig(
|
| 47 |
+
hidden_size=hidden,
|
| 48 |
+
num_hidden_layers=_envi("DAISY_SW_LAYERS", 4),
|
| 49 |
+
num_attention_heads=_envi("DAISY_SW_HEADS", 4),
|
| 50 |
+
head_dim=32, qk_rope_head_dim=16,
|
| 51 |
+
q_lora_rank=max(32, hidden // 4), o_lora_rank=max(32, hidden // 8),
|
| 52 |
+
num_key_value_heads=1,
|
| 53 |
+
max_position_embeddings=max(256, _envi("DAISY_SW_SEQLEN", 128)),
|
| 54 |
+
moe_intermediate_size=hidden,
|
| 55 |
+
n_routed_experts=_envi("DAISY_SW_EXPERTS", 4), n_shared_experts=1,
|
| 56 |
+
num_experts_per_tok=min(2, _envi("DAISY_SW_EXPERTS", 4)),
|
| 57 |
+
num_hash_layers=1, hc_mult=2,
|
| 58 |
+
num_nextn_predict_layers=_envi("DAISY_SW_MTP", 0),
|
| 59 |
+
engram_table_size=4096, engram_compress_dim=48, engram_num_heads=2,
|
| 60 |
+
)
|
| 61 |
+
|
| 62 |
+
|
| 63 |
+
class _FineWebStream:
|
| 64 |
+
"""Streams FineWeb-Edu, tokenizes, yields fixed-length token windows."""
|
| 65 |
+
|
| 66 |
+
def __init__(self, tokenizer, seqlen, rank=0, world=1):
|
| 67 |
+
self.tok, self.seqlen = tokenizer, seqlen
|
| 68 |
+
self.buf = []
|
| 69 |
+
ds_path = os.environ.get("DAISY_SW_DATASET", "HuggingFaceFW/fineweb-edu")
|
| 70 |
+
# empty string = "use the dataset's default config" (custom datasets
|
| 71 |
+
# usually have no named config; the sample-10BT default only fits fineweb-edu)
|
| 72 |
+
default_subset = "sample-10BT" if ds_path == "HuggingFaceFW/fineweb-edu" else ""
|
| 73 |
+
subset = os.environ.get("DAISY_SW_SUBSET", default_subset)
|
| 74 |
+
self.eos = getattr(tokenizer, "eos_token_id", 1) or 1
|
| 75 |
+
try:
|
| 76 |
+
from datasets import load_dataset
|
| 77 |
+
ds = load_dataset(ds_path, name=subset or None, split="train", streaming=True)
|
| 78 |
+
ds = ds.shard(num_shards=world, index=rank) if world > 1 else ds
|
| 79 |
+
self.it = iter(ds)
|
| 80 |
+
self.source = f"{ds_path}:{subset or 'default'}"
|
| 81 |
+
except Exception as e:
|
| 82 |
+
print(f"[spikewhale] FineWeb stream unavailable ({e}); using local fallback text", flush=True)
|
| 83 |
+
self.it = None
|
| 84 |
+
self.source = "local-fallback"
|
| 85 |
+
|
| 86 |
+
def _more_tokens(self):
|
| 87 |
+
if self.it is not None:
|
| 88 |
+
row = next(self.it)
|
| 89 |
+
text = row.get("text", "") or ""
|
| 90 |
+
else:
|
| 91 |
+
text = ("Education is the process of learning and acquiring knowledge. "
|
| 92 |
+
"Small models can still learn useful patterns from good data. ")
|
| 93 |
+
ids = self.tok.encode(text, add_special_tokens=False)
|
| 94 |
+
self.buf.extend(ids + [self.eos])
|
| 95 |
+
|
| 96 |
+
def next_window(self):
|
| 97 |
+
while len(self.buf) < self.seqlen + 1:
|
| 98 |
+
self._more_tokens()
|
| 99 |
+
w = self.buf[: self.seqlen + 1]
|
| 100 |
+
self.buf = self.buf[self.seqlen:]
|
| 101 |
+
return w
|
| 102 |
+
|
| 103 |
+
|
| 104 |
+
class SpikeWhaleTask:
|
| 105 |
+
def __init__(self):
|
| 106 |
+
SpikeWhaleConfig, SpikeWhaleLM, SpikeTokenizer = _import_spikewhale()
|
| 107 |
+
self.cfg = build_config()
|
| 108 |
+
tok_file = os.path.join(_DEF_PATH, "tokenizer.json")
|
| 109 |
+
self.tok = SpikeTokenizer(vocab_file=tok_file)
|
| 110 |
+
self.seqlen = _envi("DAISY_SW_SEQLEN", 128)
|
| 111 |
+
rank = _envi("RANK", 0); world = _envi("WORLD_SIZE", 1)
|
| 112 |
+
self.stream = _FineWebStream(self.tok, self.seqlen, rank, world)
|
| 113 |
+
self._SpikeWhaleLM = SpikeWhaleLM
|
| 114 |
+
n = None
|
| 115 |
+
print(f"[spikewhale] data source: {self.stream.source}", flush=True)
|
| 116 |
+
|
| 117 |
+
def build_model(self):
|
| 118 |
+
torch.manual_seed(0) # identical init on every node
|
| 119 |
+
m = self._SpikeWhaleLM(self.cfg)
|
| 120 |
+
n = sum(p.numel() for p in m.parameters())
|
| 121 |
+
print(f"[spikewhale] model built: {n:,} params "
|
| 122 |
+
f"(hidden={self.cfg.hidden_size}, layers={self.cfg.num_hidden_layers}, "
|
| 123 |
+
f"experts={self.cfg.n_routed_experts}, seqlen={self.seqlen})", flush=True)
|
| 124 |
+
return m
|
| 125 |
+
|
| 126 |
+
def sample(self, n):
|
| 127 |
+
rows = [self.stream.next_window()[:self.seqlen] for _ in range(n)]
|
| 128 |
+
t = torch.tensor(rows, dtype=torch.long) # (n, seqlen)
|
| 129 |
+
return t, t # labels == input_ids; model shifts internally
|
| 130 |
+
|
| 131 |
+
def loss(self, model, X, y):
|
| 132 |
+
return model(input_ids=X, labels=y).loss
|
export_luts_web.py
ADDED
|
@@ -0,0 +1,44 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Export the bundled verified units to lookup tables for the browser
|
| 2 |
+
(DaisyChain-Web). These are the emulated GPU logic, materialized: the browser
|
| 3 |
+
computes through THESE, not plain float.
|
| 4 |
+
|
| 5 |
+
Writes three little binaries into daisychain-web/public/:
|
| 6 |
+
mul_lut.bin int16[65536] signed 8x8 product, indexed [au*256 + bu]
|
| 7 |
+
requant_lut.bin int8 [65536] int16->int8 requant, indexed [acc & 0xFFFF]
|
| 8 |
+
relu_lut.bin int8 [256] int8 ReLU, indexed [byte]
|
| 9 |
+
luts_meta.json dims + requant shift (for dequant)
|
| 10 |
+
"""
|
| 11 |
+
import json
|
| 12 |
+
import os
|
| 13 |
+
import numpy as np
|
| 14 |
+
|
| 15 |
+
from daisychain.verified.qat import load_units
|
| 16 |
+
from daisychain.verified.lut import build_mul8_lut, build_requant16_lut, build_relu8_lut
|
| 17 |
+
|
| 18 |
+
OUT = os.path.join(os.path.dirname(__file__), "..", "daisychain-web", "public")
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
def main():
|
| 22 |
+
mul, rq, relu = load_units()
|
| 23 |
+
mul_lut = build_mul8_lut(mul).astype(np.int16) # (256,256) -> flat 65536
|
| 24 |
+
req_lut = build_requant16_lut(rq).astype(np.int8) # 65536
|
| 25 |
+
relu_lut = build_relu8_lut(relu).astype(np.int8) # 256
|
| 26 |
+
|
| 27 |
+
os.makedirs(OUT, exist_ok=True)
|
| 28 |
+
mul_lut.reshape(-1).tofile(os.path.join(OUT, "mul_lut.bin"))
|
| 29 |
+
req_lut.tofile(os.path.join(OUT, "requant_lut.bin"))
|
| 30 |
+
relu_lut.tofile(os.path.join(OUT, "relu_lut.bin"))
|
| 31 |
+
meta = {"mul": [256, 256], "requant": 65536, "relu": 256, "shift": rq.shift}
|
| 32 |
+
with open(os.path.join(OUT, "luts_meta.json"), "w") as f:
|
| 33 |
+
json.dump(meta, f)
|
| 34 |
+
|
| 35 |
+
# sanity: LUT must equal the true signed product (verified units are exact)
|
| 36 |
+
a, b = 37, -19
|
| 37 |
+
au, bu = a & 0xFF, b & 0xFF
|
| 38 |
+
assert int(mul_lut[au, bu]) == a * b, "mul LUT mismatch"
|
| 39 |
+
print("exported mul_lut(int16 65536), requant_lut(int8 65536), relu_lut(int8 256)")
|
| 40 |
+
print("shift =", rq.shift, "| sanity 37*-19 =", int(mul_lut[au, bu]))
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
if __name__ == "__main__":
|
| 44 |
+
main()
|
web/.gitignore
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
node_modules/
|
| 2 |
+
*.log
|
| 3 |
+
.DS_Store
|
web/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
MIT License
|
| 2 |
+
|
| 3 |
+
Copyright (c) 2026 Dean Byrne (Quazim0t0) / DaisyChainAI
|
| 4 |
+
|
| 5 |
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
| 6 |
+
of this software and associated documentation files (the "Software"), to deal
|
| 7 |
+
in the Software without restriction, including without limitation the rights
|
| 8 |
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
| 9 |
+
copies of the Software, and to permit persons to whom the Software is
|
| 10 |
+
furnished to do so, subject to the following conditions:
|
| 11 |
+
|
| 12 |
+
The above copyright notice and this permission notice shall be included in all
|
| 13 |
+
copies or substantial portions of the Software.
|
| 14 |
+
|
| 15 |
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
| 16 |
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
| 17 |
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
| 18 |
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
| 19 |
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
| 20 |
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
| 21 |
+
SOFTWARE.
|
web/README.md
ADDED
|
@@ -0,0 +1,92 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 🌼 DaisyChain-Web — train by opening a page
|
| 2 |
+
|
| 3 |
+
Part of **DaisyChain** → https://huggingface.co/DaisyChainAI
|
| 4 |
+
|
| 5 |
+
Open a link on two or more devices and they **train a shared model together,
|
| 6 |
+
peer-to-peer, right in the browser** — no install, no accounts. Devices connect
|
| 7 |
+
over **WebRTC** (like Snapdrop); each one computes **through the verified INT8
|
| 8 |
+
units** — the same emulated GPU logic as the rest of DaisyChain, run as a
|
| 9 |
+
**WebGPU** lookup-table matmul (with a CPU fallback for old machines) — and they
|
| 10 |
+
average gradients directly between peers.
|
| 11 |
+
|
| 12 |
+
> This is the browser-native version of DaisyChain: instead of setting up nodes,
|
| 13 |
+
> people join a training run by opening a URL.
|
| 14 |
+
|
| 15 |
+
## Run it
|
| 16 |
+
|
| 17 |
+
```bash
|
| 18 |
+
npm install
|
| 19 |
+
npm start # serves on http://localhost:8787
|
| 20 |
+
```
|
| 21 |
+
|
| 22 |
+
Open `http://localhost:8787` in two tabs (or two devices — see HTTPS note). They
|
| 23 |
+
find each other, connect P2P, and you click **Start training** on each. Watch the
|
| 24 |
+
shared loss fall on both.
|
| 25 |
+
|
| 26 |
+
Quick self-check of the training math (no browser):
|
| 27 |
+
```bash
|
| 28 |
+
npm test # 2-peer gradient averaging: converges, replicas bit-identical
|
| 29 |
+
```
|
| 30 |
+
|
| 31 |
+
## How it works
|
| 32 |
+
|
| 33 |
+
| Piece | Role |
|
| 34 |
+
|-------|------|
|
| 35 |
+
| `server.js` | tiny **WebSocket signaling** server (introduces peers, relays WebRTC offers/ICE) + static host. It never sees the compute. |
|
| 36 |
+
| WebRTC data channels | **P2P** gradient exchange between browsers (STUN for NAT). |
|
| 37 |
+
| `public/verified_core.js` | the **verified INT8 units** in the browser — quantize → LUT multiply → dequant, STE backward. The emulated GPU logic doing the training compute. |
|
| 38 |
+
| `public/webgpu.js` | runs the verified multiply as a **WebGPU** LUT-matmul compute shader, with an automatic **CPU/JS fallback**. |
|
| 39 |
+
| `public/*.bin` | the units as lookup tables (`mul_lut`, `requant_lut`, `relu_lut`), exported from the trained weights by `daisychain/export_luts_web.py`. |
|
| 40 |
+
| `public/app.js` | the WebRTC mesh + the training loop + gradient averaging. |
|
| 41 |
+
|
| 42 |
+
Each peer starts from the **same** deterministically-seeded weights (no weight
|
| 43 |
+
broadcast needed), trains on its own data shard **through the verified units**,
|
| 44 |
+
and every step broadcasts its gradient and averages everyone's — so all peers
|
| 45 |
+
converge to the **same** model.
|
| 46 |
+
|
| 47 |
+
## What's verified
|
| 48 |
+
- **Through the units** (`node test_verified.js`): 2 peers training through the
|
| 49 |
+
verified INT8 multiply converge and stay **bit-identical** (0.0 param diff).
|
| 50 |
+
- **Signaling** (`node -e ...`): peer discovery + relay works.
|
| 51 |
+
- **End-to-end in-browser**: two tabs connected over **real WebRTC**, both on
|
| 52 |
+
**WebGPU running the verified INT8 units**, trained a shared model together —
|
| 53 |
+
loss fell steadily, peers in sync. (`node test_core.js` also checks the plain
|
| 54 |
+
float loop.)
|
| 55 |
+
|
| 56 |
+
## Regenerate the unit LUTs
|
| 57 |
+
The `.bin` tables are exported from the trained DaisyChain units:
|
| 58 |
+
```bash
|
| 59 |
+
cd ../daisychain && python export_luts_web.py # writes mul/requant/relu LUTs into ../daisychain-web/public
|
| 60 |
+
```
|
| 61 |
+
|
| 62 |
+
## Who connects to whom (Snapdrop-style)
|
| 63 |
+
|
| 64 |
+
Peers are grouped by their **public IP**, so **only devices on the same network
|
| 65 |
+
auto-connect** — open the page on your phone and laptop at home and they find
|
| 66 |
+
each other, but a stranger viewing the same URL from another network does **not**
|
| 67 |
+
join your group. To connect across networks with people you invite, everyone
|
| 68 |
+
opens `?room=YOUR-CODE` (a shared private room). The server only relays WebRTC
|
| 69 |
+
handshakes; it never sees the training.
|
| 70 |
+
|
| 71 |
+
**Safety note:** WebRTC peers connect directly, so devices in your group can see
|
| 72 |
+
each other's IP address, and there's no gradient authentication — a malicious
|
| 73 |
+
peer could poison the shared model. Train only with devices/people you trust.
|
| 74 |
+
This is a proof of concept, not a hardened public service.
|
| 75 |
+
|
| 76 |
+
## Honest limits
|
| 77 |
+
- **Secure context required.** WebGPU and cross-device WebRTC need **localhost or
|
| 78 |
+
HTTPS**. For real multi-device use, serve over HTTPS (a tunnel, a host, or a HF
|
| 79 |
+
Space) — plain `http://192.168.x.x` won't get WebGPU.
|
| 80 |
+
- **No WebGPU? It falls back to CPU** — slower, but old machines (e.g. Windows XP/7
|
| 81 |
+
via **Supermium**, old Macs via a compatibility browser) can still join at the
|
| 82 |
+
CPU tier. WebGPU itself needs a GPU/driver with a modern backend, which very old
|
| 83 |
+
hardware usually lacks.
|
| 84 |
+
- **Synchronous barrier**: the slowest peer paces each step. Peer-dropout handling
|
| 85 |
+
is minimal (a timeout, then it proceeds) — this is a proof of concept, not
|
| 86 |
+
hardened.
|
| 87 |
+
- **Small models only** — WebRTC bandwidth and browser compute cap the size. Same
|
| 88 |
+
envelope as the rest of DaisyChain: pools compute, not memory.
|
| 89 |
+
|
| 90 |
+
---
|
| 91 |
+
|
| 92 |
+
**License:** MIT · **Author:** Dean Byrne (Quazim0t0) · **Org:** DaisyChainAI
|
web/package-lock.json
ADDED
|
@@ -0,0 +1,37 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"name": "daisychain-web",
|
| 3 |
+
"version": "0.1.0",
|
| 4 |
+
"lockfileVersion": 3,
|
| 5 |
+
"requires": true,
|
| 6 |
+
"packages": {
|
| 7 |
+
"": {
|
| 8 |
+
"name": "daisychain-web",
|
| 9 |
+
"version": "0.1.0",
|
| 10 |
+
"license": "MIT",
|
| 11 |
+
"dependencies": {
|
| 12 |
+
"ws": "^8.21.0"
|
| 13 |
+
}
|
| 14 |
+
},
|
| 15 |
+
"node_modules/ws": {
|
| 16 |
+
"version": "8.21.0",
|
| 17 |
+
"resolved": "https://registry.npmjs.org/ws/-/ws-8.21.0.tgz",
|
| 18 |
+
"integrity": "sha512-Vsp28b7DRcimFQvrqu2Wek3z1iYxDCWqHYB8Qsnk/S4RfaCQzPGPyBNuVjJV3cd6UiKtUtp6sNM77gWvzcCH+g==",
|
| 19 |
+
"license": "MIT",
|
| 20 |
+
"engines": {
|
| 21 |
+
"node": ">=10.0.0"
|
| 22 |
+
},
|
| 23 |
+
"peerDependencies": {
|
| 24 |
+
"bufferutil": "^4.0.1",
|
| 25 |
+
"utf-8-validate": ">=5.0.2"
|
| 26 |
+
},
|
| 27 |
+
"peerDependenciesMeta": {
|
| 28 |
+
"bufferutil": {
|
| 29 |
+
"optional": true
|
| 30 |
+
},
|
| 31 |
+
"utf-8-validate": {
|
| 32 |
+
"optional": true
|
| 33 |
+
}
|
| 34 |
+
}
|
| 35 |
+
}
|
| 36 |
+
}
|
| 37 |
+
}
|
web/package.json
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"name": "daisychain-web",
|
| 3 |
+
"version": "0.1.0",
|
| 4 |
+
"description": "Train a shared model P2P in the browser — WebRTC mesh + WebGPU compute. Open a page, become a node.",
|
| 5 |
+
"license": "MIT",
|
| 6 |
+
"author": "Dean Byrne (Quazim0t0) / DaisyChainAI",
|
| 7 |
+
"scripts": {
|
| 8 |
+
"start": "node server.js",
|
| 9 |
+
"test": "node test_core.js"
|
| 10 |
+
},
|
| 11 |
+
"dependencies": {
|
| 12 |
+
"ws": "^8.21.0"
|
| 13 |
+
}
|
| 14 |
+
}
|
web/public/app.js
ADDED
|
@@ -0,0 +1,334 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// DaisyChain-Web client: connect P2P (WebRTC), compute (WebGPU or CPU), and
|
| 2 |
+
// train a shared model together — averaging gradients over the data channels.
|
| 3 |
+
"use strict";
|
| 4 |
+
|
| 5 |
+
const DIN = 16, H = 16, DOUT = 4, NPER = 128, LR = 0.03, DEFAULT_STEPS = 300;
|
| 6 |
+
const D = { n: NPER, din: DIN, h: H, dout: DOUT };
|
| 7 |
+
const STUN = [{ urls: "stun:stun.l.google.com:19302" }];
|
| 8 |
+
|
| 9 |
+
const ui = {
|
| 10 |
+
status: document.getElementById("status"),
|
| 11 |
+
backend: document.getElementById("backend"),
|
| 12 |
+
me: document.getElementById("me"),
|
| 13 |
+
peers: document.getElementById("peers"),
|
| 14 |
+
loss: document.getElementById("loss"),
|
| 15 |
+
step: document.getElementById("step"),
|
| 16 |
+
bar: document.getElementById("bar"),
|
| 17 |
+
diff: document.getElementById("diff"),
|
| 18 |
+
log: document.getElementById("log"),
|
| 19 |
+
start: document.getElementById("start"),
|
| 20 |
+
save: document.getElementById("save"),
|
| 21 |
+
load: document.getElementById("load"),
|
| 22 |
+
loadBtn: document.getElementById("loadBtn"),
|
| 23 |
+
requests: document.getElementById("requests"),
|
| 24 |
+
};
|
| 25 |
+
function log(m) { ui.log.textContent = `${new Date().toLocaleTimeString()} ${m}\n` + ui.log.textContent; }
|
| 26 |
+
function setStatus(s) { ui.status.textContent = s; }
|
| 27 |
+
|
| 28 |
+
// ---- deterministic RNG so every peer agrees on W_true and W0 (no broadcast) --
|
| 29 |
+
function mulberry32(a) { return function () { a |= 0; a = a + 0x6D2B79F5 | 0; let t = Math.imul(a ^ a >>> 15, 1 | a); t = t + Math.imul(t ^ t >>> 7, 61 | t) ^ t; return ((t ^ t >>> 14) >>> 0) / 4294967296; }; }
|
| 30 |
+
function randn(n, rng) { const r = rng || Math.random; const o = new Float32Array(n); for (let i = 0; i < n; i += 2) { let u = 0, v = 0; while (u === 0) u = r(); while (v === 0) v = r(); const m = Math.sqrt(-2 * Math.log(u)); o[i] = m * Math.cos(2 * Math.PI * v); if (i + 1 < n) o[i + 1] = m * Math.sin(2 * Math.PI * v); } return o; }
|
| 31 |
+
|
| 32 |
+
// ---- state -----------------------------------------------------------------
|
| 33 |
+
// friendly device name (cottagecore, Snapdrop-style)
|
| 34 |
+
const ADJ = ["Mossy", "Golden", "Amber", "Fern", "Hazel", "Cozy", "Wandering", "Little",
|
| 35 |
+
"Sunny", "Misty", "Wild", "Quiet", "Brave", "Dusty", "Merry"];
|
| 36 |
+
const NOUN = ["Fox", "Hare", "Owl", "Badger", "Toad", "Sparrow", "Otter", "Deer",
|
| 37 |
+
"Hedgehog", "Mushroom", "Acorn", "Willow", "Robin", "Fawn", "Moth"];
|
| 38 |
+
const deviceName = ADJ[Math.floor(Math.random() * ADJ.length)] + " " +
|
| 39 |
+
NOUN[Math.floor(Math.random() * NOUN.length)];
|
| 40 |
+
|
| 41 |
+
let myId = null, compute = null, ws = null, L = null, wasDenied = false;
|
| 42 |
+
const pcs = new Map(), chans = new Map(); // peerId -> RTCPeerConnection / DataChannel
|
| 43 |
+
const names = new Map(); // peerId -> device name
|
| 44 |
+
const incoming = new Map(); // step -> Map(peerId -> Float32Array)
|
| 45 |
+
let W1, W2, Xdata, Ydata, training = false; // 2-layer verified model
|
| 46 |
+
let trainedSteps = 0; // steps baked into the current weights
|
| 47 |
+
function nmeOf(id) { return names.get(id) || id; }
|
| 48 |
+
|
| 49 |
+
function room() { return new URLSearchParams(location.search).get("room"); } // null -> group by network
|
| 50 |
+
function updatePeers() { ui.peers.textContent = chans.size ? [...chans.keys()].map(nmeOf).join(", ") : "(none yet)"; }
|
| 51 |
+
|
| 52 |
+
// ---- signaling + WebRTC ----------------------------------------------------
|
| 53 |
+
function connectSignaling() {
|
| 54 |
+
const proto = location.protocol === "https:" ? "wss" : "ws";
|
| 55 |
+
const params = new URLSearchParams();
|
| 56 |
+
params.set("name", deviceName);
|
| 57 |
+
if (room()) params.set("room", room()); // no code -> group by network
|
| 58 |
+
ws = new WebSocket(`${proto}://${location.host}/?${params}`);
|
| 59 |
+
ws.onopen = () => setStatus(room() ? `connected — private room "${room()}"` : "connected — grouping with devices on your network");
|
| 60 |
+
ws.onclose = () => { if (!wasDenied) setStatus("signaling disconnected"); };
|
| 61 |
+
ws.onmessage = async (ev) => {
|
| 62 |
+
const msg = JSON.parse(ev.data);
|
| 63 |
+
if (msg.type === "welcome") {
|
| 64 |
+
myId = msg.id;
|
| 65 |
+
if (msg.room) log(`group: ${msg.room.startsWith("net:") ? "your network" : "private room " + msg.room.replace("room:", "")}`);
|
| 66 |
+
if (msg.host) { log("you host this room — joiners wait for your approval"); setStatus(`hosting private room "${room()}"`); }
|
| 67 |
+
else if (room()) setStatus(`accepted into private room "${room()}"`);
|
| 68 |
+
// I'm newest: initiate to everyone already here
|
| 69 |
+
for (const p of msg.peers) { names.set(p.id, p.name); initiatePeer(p.id); }
|
| 70 |
+
} else if (msg.type === "waiting") {
|
| 71 |
+
setStatus("knocking — waiting for the room's host to let you in…");
|
| 72 |
+
} else if (msg.type === "denied") {
|
| 73 |
+
wasDenied = true;
|
| 74 |
+
setStatus("the host declined your request to join");
|
| 75 |
+
log("join request declined by the host");
|
| 76 |
+
} else if (msg.type === "host") {
|
| 77 |
+
log("the host left — you are now the host of this room");
|
| 78 |
+
setStatus(`hosting private room "${room()}"`);
|
| 79 |
+
} else if (msg.type === "join-request") {
|
| 80 |
+
addJoinRequest(msg.id, msg.name);
|
| 81 |
+
} else if (msg.type === "peer-joined") {
|
| 82 |
+
names.set(msg.id, msg.name);
|
| 83 |
+
log(`${msg.name} joined (they will connect to me)`);
|
| 84 |
+
} else if (msg.type === "peer-left") {
|
| 85 |
+
log(`${nmeOf(msg.id)} left`); cleanupPeer(msg.id); names.delete(msg.id); updatePeers();
|
| 86 |
+
} else if (msg.type === "signal") {
|
| 87 |
+
await onSignal(msg.from, msg.data);
|
| 88 |
+
}
|
| 89 |
+
};
|
| 90 |
+
}
|
| 91 |
+
function signal(to, data) { ws.send(JSON.stringify({ type: "signal", to, data })); }
|
| 92 |
+
|
| 93 |
+
// host-side Accept/Deny row for a knocking device (textContent only — the name
|
| 94 |
+
// comes off the wire and must never be parsed as HTML)
|
| 95 |
+
function addJoinRequest(id, name) {
|
| 96 |
+
const row = document.createElement("div");
|
| 97 |
+
row.style.cssText = "display:flex;gap:8px;align-items:center;justify-content:space-between;margin-top:10px;flex-wrap:wrap";
|
| 98 |
+
const who = document.createElement("span");
|
| 99 |
+
who.textContent = `🚪 ${name} wants to join`;
|
| 100 |
+
const btn = (label, allow) => {
|
| 101 |
+
const b = document.createElement("button");
|
| 102 |
+
b.textContent = label;
|
| 103 |
+
b.style.cssText = "padding:6px 14px;font-size:.85rem" + (allow ? "" : ";background:#8b2e25");
|
| 104 |
+
b.onclick = () => {
|
| 105 |
+
ws.send(JSON.stringify({ type: "admit", id, allow }));
|
| 106 |
+
row.remove();
|
| 107 |
+
log(allow ? `you let ${name} in` : `you declined ${name}`);
|
| 108 |
+
};
|
| 109 |
+
return b;
|
| 110 |
+
};
|
| 111 |
+
row.append(who, btn("Accept", true), btn("Deny", false));
|
| 112 |
+
ui.requests.appendChild(row);
|
| 113 |
+
}
|
| 114 |
+
|
| 115 |
+
function newPC(peerId) {
|
| 116 |
+
const pc = new RTCPeerConnection({ iceServers: STUN });
|
| 117 |
+
pc.onicecandidate = (e) => { if (e.candidate) signal(peerId, { candidate: e.candidate }); };
|
| 118 |
+
pc.onconnectionstatechange = () => { if (pc.connectionState === "failed") cleanupPeer(peerId); };
|
| 119 |
+
pcs.set(peerId, pc);
|
| 120 |
+
return pc;
|
| 121 |
+
}
|
| 122 |
+
function initiatePeer(peerId) {
|
| 123 |
+
const pc = newPC(peerId);
|
| 124 |
+
const dc = pc.createDataChannel("daisy");
|
| 125 |
+
setupChannel(peerId, dc);
|
| 126 |
+
pc.createOffer().then(o => pc.setLocalDescription(o)).then(() => signal(peerId, { sdp: pc.localDescription }));
|
| 127 |
+
}
|
| 128 |
+
async function onSignal(from, data) {
|
| 129 |
+
let pc = pcs.get(from);
|
| 130 |
+
if (data.sdp) {
|
| 131 |
+
if (!pc) { pc = newPC(from); pc.ondatachannel = (e) => setupChannel(from, e.channel); }
|
| 132 |
+
await pc.setRemoteDescription(data.sdp);
|
| 133 |
+
if (data.sdp.type === "offer") {
|
| 134 |
+
const ans = await pc.createAnswer(); await pc.setLocalDescription(ans);
|
| 135 |
+
signal(from, { sdp: pc.localDescription });
|
| 136 |
+
}
|
| 137 |
+
} else if (data.candidate && pc) {
|
| 138 |
+
try { await pc.addIceCandidate(data.candidate); } catch (e) {}
|
| 139 |
+
}
|
| 140 |
+
}
|
| 141 |
+
function setupChannel(peerId, dc) {
|
| 142 |
+
dc.binaryType = "arraybuffer";
|
| 143 |
+
dc.onopen = () => { chans.set(peerId, dc); updatePeers(); log(`connected to ${nmeOf(peerId)}`); ui.start.disabled = false; };
|
| 144 |
+
dc.onclose = () => { chans.delete(peerId); updatePeers(); wake(); };
|
| 145 |
+
dc.onmessage = (e) => onGrad(peerId, e.data);
|
| 146 |
+
}
|
| 147 |
+
function cleanupPeer(id) { const pc = pcs.get(id); if (pc) pc.close(); pcs.delete(id); chans.delete(id); wake(); }
|
| 148 |
+
|
| 149 |
+
// ---- checkpoints ------------------------------------------------------------
|
| 150 |
+
// File layout (also the broadcast payload after the sentinel):
|
| 151 |
+
// 8 bytes magic "DAISYPT1" | int32 din,h,dout,steps | f32 W1 | f32 W2
|
| 152 |
+
// DaisyChain's own format (not torch-pickle) — .pt extension for familiarity.
|
| 153 |
+
const CKPT_MAGIC = "DAISYPT1";
|
| 154 |
+
const CKPT_SENTINEL = -2; // wire: [int32 -2][checkpoint bytes]
|
| 155 |
+
|
| 156 |
+
function packCheckpoint() {
|
| 157 |
+
const buf = new ArrayBuffer(8 + 16 + (W1.length + W2.length) * 4);
|
| 158 |
+
new Uint8Array(buf, 0, 8).set([...CKPT_MAGIC].map(c => c.charCodeAt(0)));
|
| 159 |
+
new Int32Array(buf, 8, 4).set([DIN, H, DOUT, trainedSteps]);
|
| 160 |
+
new Float32Array(buf, 24, W1.length).set(W1);
|
| 161 |
+
new Float32Array(buf, 24 + W1.length * 4, W2.length).set(W2);
|
| 162 |
+
return buf;
|
| 163 |
+
}
|
| 164 |
+
function parseCheckpoint(buf) {
|
| 165 |
+
const magic = String.fromCharCode(...new Uint8Array(buf, 0, 8));
|
| 166 |
+
if (magic !== CKPT_MAGIC) throw new Error("not a DaisyChain checkpoint");
|
| 167 |
+
const [din, h, dout, steps] = new Int32Array(buf, 8, 4);
|
| 168 |
+
if (din !== DIN || h !== H || dout !== DOUT)
|
| 169 |
+
throw new Error(`shape mismatch: file is ${din}×${h}×${dout}, this build is ${DIN}×${H}×${DOUT}`);
|
| 170 |
+
if (buf.byteLength !== 24 + (din * h + h * dout) * 4) throw new Error("truncated checkpoint");
|
| 171 |
+
return { steps, w1: new Float32Array(buf.slice(24, 24 + din * h * 4)),
|
| 172 |
+
w2: new Float32Array(buf.slice(24 + din * h * 4)) };
|
| 173 |
+
}
|
| 174 |
+
function applyCheckpoint(ck, from) {
|
| 175 |
+
W1.set(ck.w1); W2.set(ck.w2); trainedSteps = ck.steps;
|
| 176 |
+
ui.save.disabled = false;
|
| 177 |
+
ui.step.textContent = `${ck.steps} baked in`;
|
| 178 |
+
log(`checkpoint loaded (${ck.steps} steps) ${from ? "from " + from : "from file"} — all set to resume`);
|
| 179 |
+
}
|
| 180 |
+
function broadcastCheckpoint() {
|
| 181 |
+
const ck = packCheckpoint();
|
| 182 |
+
const msg = new ArrayBuffer(4 + ck.byteLength);
|
| 183 |
+
new Int32Array(msg, 0, 1)[0] = CKPT_SENTINEL;
|
| 184 |
+
new Uint8Array(msg, 4).set(new Uint8Array(ck));
|
| 185 |
+
let n = 0;
|
| 186 |
+
for (const dc of chans.values()) if (dc.readyState === "open") { dc.send(msg); n++; }
|
| 187 |
+
log(`checkpoint pushed to ${n} device(s)`);
|
| 188 |
+
}
|
| 189 |
+
function saveCheckpoint() {
|
| 190 |
+
const blob = new Blob([packCheckpoint()], { type: "application/octet-stream" });
|
| 191 |
+
const a = document.createElement("a");
|
| 192 |
+
a.href = URL.createObjectURL(blob);
|
| 193 |
+
a.download = `daisychain-${DIN}x${H}x${DOUT}-step${trainedSteps}.pt`;
|
| 194 |
+
a.click();
|
| 195 |
+
URL.revokeObjectURL(a.href);
|
| 196 |
+
}
|
| 197 |
+
|
| 198 |
+
// ---- gradient wire format: [int32 step][float32 grad...] -----------------
|
| 199 |
+
function packGrad(step, grad) {
|
| 200 |
+
const buf = new ArrayBuffer(4 + grad.byteLength);
|
| 201 |
+
new Int32Array(buf, 0, 1)[0] = step;
|
| 202 |
+
new Float32Array(buf, 4).set(grad);
|
| 203 |
+
return buf;
|
| 204 |
+
}
|
| 205 |
+
const waiters = new Set(); // pending waitForGrads checkers
|
| 206 |
+
function wake() { for (const w of waiters) w(); }
|
| 207 |
+
function onGrad(peerId, buf) {
|
| 208 |
+
const step = new Int32Array(buf, 0, 1)[0];
|
| 209 |
+
if (step === CKPT_SENTINEL) { // a peer pushed a checkpoint
|
| 210 |
+
if (training) { log(`ignored checkpoint from ${nmeOf(peerId)} (training in progress)`); return; }
|
| 211 |
+
try { applyCheckpoint(parseCheckpoint(buf.slice(4)), nmeOf(peerId)); }
|
| 212 |
+
catch (e) { log(`bad checkpoint from ${nmeOf(peerId)}: ${e.message}`); }
|
| 213 |
+
return;
|
| 214 |
+
}
|
| 215 |
+
const grad = new Float32Array(buf.slice(4));
|
| 216 |
+
if (!incoming.has(step)) incoming.set(step, new Map());
|
| 217 |
+
incoming.get(step).set(peerId, grad);
|
| 218 |
+
wake(); // resolve waits immediately (no polling)
|
| 219 |
+
}
|
| 220 |
+
function broadcastGrad(step, grad) { const b = packGrad(step, grad); for (const dc of chans.values()) if (dc.readyState === "open") dc.send(b); }
|
| 221 |
+
// Event-driven: re-checked on every gradient arrival and peer departure, plus a
|
| 222 |
+
// coarse fallback timer (background tabs throttle timers to ~1s, so the old
|
| 223 |
+
// 15ms poll was the bottleneck there). Peers that left are dropped from the
|
| 224 |
+
// wait — a device dying no longer costs the full timeout every step.
|
| 225 |
+
function waitForGrads(step, cohort, timeoutMs = 8000) {
|
| 226 |
+
return new Promise((resolve) => {
|
| 227 |
+
const t0 = Date.now();
|
| 228 |
+
let timer = null;
|
| 229 |
+
const check = () => {
|
| 230 |
+
const live = cohort.filter(id => chans.has(id)); // prune departed peers
|
| 231 |
+
const got = incoming.get(step) || new Map();
|
| 232 |
+
const have = live.filter(id => got.has(id));
|
| 233 |
+
if (have.length === live.length || Date.now() - t0 > timeoutMs) {
|
| 234 |
+
waiters.delete(check); clearInterval(timer);
|
| 235 |
+
resolve(have.map(id => got.get(id)));
|
| 236 |
+
}
|
| 237 |
+
};
|
| 238 |
+
waiters.add(check);
|
| 239 |
+
timer = setInterval(check, 500); // safety net only
|
| 240 |
+
check();
|
| 241 |
+
});
|
| 242 |
+
}
|
| 243 |
+
|
| 244 |
+
// ---- compute: one async training step THROUGH the verified units -----------
|
| 245 |
+
async function localStep() {
|
| 246 |
+
// forward runs through the verified INT8 multiply (WebGPU or CPU); STE backward
|
| 247 |
+
const fwd = await Verified.forward(Xdata, Ydata, W1, W2, D, L, compute.matmulInt8);
|
| 248 |
+
const grad = Verified.backward(Xdata, W1, W2, fwd, D); // flat [gW1, gW2]
|
| 249 |
+
return { loss: fwd.loss, grad };
|
| 250 |
+
}
|
| 251 |
+
|
| 252 |
+
// ---- the training loop -----------------------------------------------------
|
| 253 |
+
async function train() {
|
| 254 |
+
if (training) return; training = true; ui.start.disabled = true;
|
| 255 |
+
const cohort = [...chans.keys()]; // lock the cohort (departed peers are pruned per-step)
|
| 256 |
+
const steps = DEFAULT_STEPS;
|
| 257 |
+
const opt = TrainCore.makeAdam(W1.length + W2.length, { lr: 0.2 }); // swept: best on the verified-unit STE grads
|
| 258 |
+
log(`training started — cohort ${cohort.length} peer(s), world ${cohort.length + 1}, optimizer ${opt.name}`);
|
| 259 |
+
for (let s = 0; s < steps; s++) {
|
| 260 |
+
const { loss, grad } = await localStep();
|
| 261 |
+
broadcastGrad(s, grad);
|
| 262 |
+
const remote = cohort.length ? await waitForGrads(s, cohort) : [];
|
| 263 |
+
const all = [grad, ...remote];
|
| 264 |
+
const avg = TrainCore.averageGrads(all);
|
| 265 |
+
const upd = opt.step(avg); // DaisyAdam on the cluster-avg grad
|
| 266 |
+
Verified.splitApply(W1, W2, upd, 1); // W -= 1 * upd (lr folded into upd)
|
| 267 |
+
incoming.delete(s);
|
| 268 |
+
trainedSteps++;
|
| 269 |
+
if (s % 10 === 0 || s === steps - 1) {
|
| 270 |
+
ui.loss.textContent = loss.toFixed(5);
|
| 271 |
+
ui.step.textContent = `${s + 1} / ${steps}`;
|
| 272 |
+
ui.bar.style.width = `${Math.round(100 * (s + 1) / steps)}%`;
|
| 273 |
+
await new Promise(r => setTimeout(r, 0)); // yield to UI
|
| 274 |
+
}
|
| 275 |
+
}
|
| 276 |
+
ui.diff.textContent = `done — trained through the verified units; all peers share one model.`;
|
| 277 |
+
log(`training done — final loss ${ui.loss.textContent}`);
|
| 278 |
+
training = false;
|
| 279 |
+
ui.save.disabled = false;
|
| 280 |
+
ui.start.disabled = false;
|
| 281 |
+
}
|
| 282 |
+
|
| 283 |
+
// 2-layer float target (matches the model shape) for a learnable task
|
| 284 |
+
function target(X) {
|
| 285 |
+
const Wt1 = randn(DIN * H, mulberry32(42)), Wt2 = randn(H * DOUT, mulberry32(43));
|
| 286 |
+
const hpre = TrainCore.matmul(X, Wt1, NPER, DIN, H);
|
| 287 |
+
for (let i = 0; i < hpre.length; i++) hpre[i] = Math.max(0, hpre[i]);
|
| 288 |
+
return TrainCore.matmul(hpre, Wt2, NPER, H, DOUT);
|
| 289 |
+
}
|
| 290 |
+
|
| 291 |
+
// ---- boot ------------------------------------------------------------------
|
| 292 |
+
(async function () {
|
| 293 |
+
// Neural Units are mandatory: no LUTs -> no training, period. There is no
|
| 294 |
+
// float fallback path anywhere in this app; both backends (WebGPU shader and
|
| 295 |
+
// CPU JS) compute every product through the verified mul8 LUT.
|
| 296 |
+
try {
|
| 297 |
+
L = await Compute.loadLUTs(); // the verified units, as tables
|
| 298 |
+
if (!(L.mul instanceof Int16Array) || L.mul.length !== 65536)
|
| 299 |
+
throw new Error("mul8 LUT malformed");
|
| 300 |
+
// self-test: the unit must reproduce a known product before we trust it
|
| 301 |
+
if (L.mul[((7 & 0xFF) * 256) + (-3 & 0xFF)] !== -21)
|
| 302 |
+
throw new Error("mul8 LUT self-test failed (7 × -3 ≠ -21)");
|
| 303 |
+
compute = await Compute.initCompute(L);
|
| 304 |
+
} catch (e) {
|
| 305 |
+
setStatus("NEURAL UNITS UNAVAILABLE — training disabled");
|
| 306 |
+
ui.backend.textContent = "unavailable";
|
| 307 |
+
log(`FATAL: verified neural units failed to load (${e.message}). ` +
|
| 308 |
+
`This build only trains through the units — there is no fallback.`);
|
| 309 |
+
ui.start.disabled = true;
|
| 310 |
+
return; // no signaling, no training
|
| 311 |
+
}
|
| 312 |
+
ui.backend.textContent = `${compute.backend.toUpperCase()} — ${compute.label} · through verified INT8 units`;
|
| 313 |
+
// deterministic shared init (no weight broadcast); per-peer random data shard
|
| 314 |
+
W1 = randn(DIN * H, mulberry32(7));
|
| 315 |
+
W2 = randn(H * DOUT, mulberry32(8));
|
| 316 |
+
Xdata = randn(NPER * DIN);
|
| 317 |
+
Ydata = target(Xdata);
|
| 318 |
+
ui.me.textContent = deviceName;
|
| 319 |
+
updatePeers();
|
| 320 |
+
connectSignaling();
|
| 321 |
+
ui.start.onclick = train;
|
| 322 |
+
ui.save.onclick = saveCheckpoint;
|
| 323 |
+
ui.loadBtn.onclick = () => { if (training) { log("can't load a checkpoint mid-training"); return; } ui.load.click(); };
|
| 324 |
+
ui.load.onchange = async () => {
|
| 325 |
+
const f = ui.load.files[0]; ui.load.value = "";
|
| 326 |
+
if (!f) return;
|
| 327 |
+
try {
|
| 328 |
+
const ck = parseCheckpoint(await f.arrayBuffer());
|
| 329 |
+
applyCheckpoint(ck, null);
|
| 330 |
+
broadcastCheckpoint(); // every device resumes from this
|
| 331 |
+
} catch (e) { log(`checkpoint rejected: ${e.message}`); }
|
| 332 |
+
};
|
| 333 |
+
log(`ready — this device is "${deviceName}", computing through verified units on ${compute.backend.toUpperCase()}`);
|
| 334 |
+
})();
|
web/public/index.html
ADDED
|
@@ -0,0 +1,123 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<!doctype html>
|
| 2 |
+
<html lang="en">
|
| 3 |
+
<head>
|
| 4 |
+
<meta charset="utf-8">
|
| 5 |
+
<meta name="viewport" content="width=device-width, initial-scale=1">
|
| 6 |
+
<title>DaisyChain-Web — train by opening a page</title>
|
| 7 |
+
<style>
|
| 8 |
+
:root {
|
| 9 |
+
--page-bg: #efe4c9;
|
| 10 |
+
--card-bg: #fbf6e8;
|
| 11 |
+
--card-border: rgba(139, 111, 71, 0.30);
|
| 12 |
+
--text: #2a1d0a;
|
| 13 |
+
--text-soft: #6b4423;
|
| 14 |
+
--accent: #4a7c2e;
|
| 15 |
+
--accent-deep: #2d5016;
|
| 16 |
+
--counter-bg: linear-gradient(135deg, #2d5016 0%, #1f3a0f 100%);
|
| 17 |
+
--counter-num: #f5ecd9;
|
| 18 |
+
--counter-label: #c9b072;
|
| 19 |
+
--btn: linear-gradient(135deg, #4a7c2e 0%, #2d5016 100%);
|
| 20 |
+
--btn-hover: linear-gradient(135deg, #5a8c3e 0%, #3d6020 100%);
|
| 21 |
+
--warn: #8b2e25;
|
| 22 |
+
--link: #4a7c2e;
|
| 23 |
+
--track: rgba(139, 111, 71, 0.22);
|
| 24 |
+
}
|
| 25 |
+
@media (prefers-color-scheme: dark) {
|
| 26 |
+
:root {
|
| 27 |
+
--page-bg: #14100a; --card-bg: #1f1a12; --card-border: rgba(201, 176, 114, 0.35);
|
| 28 |
+
--text: #ede1c3; --text-soft: #c9b072; --accent: #9bc466; --accent-deep: #6b9039;
|
| 29 |
+
--counter-bg: linear-gradient(135deg, #1a2e0d 0%, #0c1606 100%);
|
| 30 |
+
--counter-label: #c9b072; --warn: #ff9b8e; --link: #9bc466; --track: rgba(201,176,114,0.20);
|
| 31 |
+
}
|
| 32 |
+
}
|
| 33 |
+
* { box-sizing: border-box; }
|
| 34 |
+
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
|
| 35 |
+
max-width: 720px; margin: 0 auto; padding: 24px 18px 40px; line-height: 1.5;
|
| 36 |
+
background: var(--page-bg); color: var(--text); }
|
| 37 |
+
h1 { margin: 0 0 2px; font-size: 1.7rem; letter-spacing: .3px; }
|
| 38 |
+
.sub { color: var(--text-soft); margin: 0 0 18px; font-size: .92rem; }
|
| 39 |
+
.sub code { background: rgba(74,124,46,.12); padding: 1px 5px; border-radius: 4px; }
|
| 40 |
+
.card { background: var(--card-bg); border: 1px solid var(--card-border);
|
| 41 |
+
border-radius: 10px; padding: 14px 16px; margin: 12px 0;
|
| 42 |
+
box-shadow: 0 2px 10px rgba(0,0,0,0.06); }
|
| 43 |
+
.lbl { color: var(--text-soft); font-size: 11px; font-weight: 800; letter-spacing: 1.5px;
|
| 44 |
+
text-transform: uppercase; margin-bottom: 8px; }
|
| 45 |
+
.row { display: flex; justify-content: space-between; gap: 12px; padding: 3px 0; font-size: .95rem; }
|
| 46 |
+
.k { color: var(--text-soft); } .v { font-weight: 700; text-align: right; }
|
| 47 |
+
.device { font-family: 'Courier New', monospace; font-size: 1.5rem; font-weight: 700;
|
| 48 |
+
color: var(--accent-deep); }
|
| 49 |
+
@media (prefers-color-scheme: dark) { .device { color: var(--accent); } }
|
| 50 |
+
.counter { background: var(--counter-bg); border-radius: 10px; padding: 14px 16px; text-align: center; }
|
| 51 |
+
.counter .num { font-family: 'Courier New', monospace; font-size: 2.2rem; font-weight: 700; color: var(--counter-num); }
|
| 52 |
+
.counter .cl { color: var(--counter-label); font-size: 11px; text-transform: uppercase; letter-spacing: 1.5px; }
|
| 53 |
+
.track { width: 100%; height: 10px; border-radius: 6px; background: var(--track); overflow: hidden; margin-top: 12px; }
|
| 54 |
+
#bar { height: 10px; width: 0; background: linear-gradient(90deg, #6b9039, #9bc466); transition: width .25s; }
|
| 55 |
+
button { background: var(--btn); color: #f5ecd9; border: 0; border-radius: 8px;
|
| 56 |
+
padding: 12px 26px; font-weight: 800; font-size: 1rem; letter-spacing: .5px; cursor: pointer;
|
| 57 |
+
box-shadow: 0 3px 10px rgba(74,49,16,0.18); transition: .15s; }
|
| 58 |
+
button:hover:not(:disabled) { background: var(--btn-hover); box-shadow: 0 5px 14px rgba(74,49,16,0.28); }
|
| 59 |
+
button:disabled { background: #8b7d5e; opacity: .55; cursor: not-allowed; box-shadow: none; }
|
| 60 |
+
pre { background: rgba(0,0,0,0.05); border-radius: 8px; padding: 10px; max-height: 150px;
|
| 61 |
+
overflow: auto; font-size: .78rem; color: var(--text-soft); white-space: pre-wrap; margin: 0;
|
| 62 |
+
font-family: 'Courier New', monospace; }
|
| 63 |
+
@media (prefers-color-scheme: dark) { pre { background: rgba(0,0,0,0.25); } }
|
| 64 |
+
.note { color: var(--text-soft); font-size: .82rem; }
|
| 65 |
+
.note b { color: var(--warn); }
|
| 66 |
+
.diff { color: var(--accent-deep); font-weight: 600; font-size: .88rem; }
|
| 67 |
+
@media (prefers-color-scheme: dark) { .diff { color: var(--accent); } }
|
| 68 |
+
</style>
|
| 69 |
+
</head>
|
| 70 |
+
<body>
|
| 71 |
+
<h1>🌼 DaisyChain-Web</h1>
|
| 72 |
+
<p class="sub">Open this on your other devices <b>on the same network</b> and they train a shared model together — peer-to-peer, right in the browser, through the emulated GPU logic. Only devices on your network are grouped (like Snapdrop). To invite people across networks, everyone opens <code>?room=YOUR-CODE</code> — the person who created the room approves each device before it can join.</p>
|
| 73 |
+
|
| 74 |
+
<div class="card">
|
| 75 |
+
<div class="lbl">🌲 This device</div>
|
| 76 |
+
<div class="device" id="me">—</div>
|
| 77 |
+
<div class="row" style="margin-top:8px"><span class="k">Status</span><span class="v" id="status">starting…</span></div>
|
| 78 |
+
<div class="row"><span class="k">Compute</span><span class="v" id="backend">detecting…</span></div>
|
| 79 |
+
</div>
|
| 80 |
+
|
| 81 |
+
<div class="card">
|
| 82 |
+
<div class="lbl">🍄 Devices in your group</div>
|
| 83 |
+
<div class="row"><span class="v" id="peers" style="text-align:left">(none yet)</span></div>
|
| 84 |
+
<div id="requests"></div>
|
| 85 |
+
</div>
|
| 86 |
+
|
| 87 |
+
<div class="card" style="text-align:center">
|
| 88 |
+
<button id="start" disabled>Start training</button>
|
| 89 |
+
<p class="note" style="margin:.6rem 0 0">Enabled once another device joins. (Or open a second tab to try it.)</p>
|
| 90 |
+
</div>
|
| 91 |
+
|
| 92 |
+
<div class="card">
|
| 93 |
+
<div class="lbl">✦ Training</div>
|
| 94 |
+
<div class="row"><span class="k">Step</span><span class="v" id="step">— / —</span></div>
|
| 95 |
+
<div class="counter" style="margin:10px 0"><div class="num" id="loss">—</div><div class="cl">cluster-avg loss · lower is better</div></div>
|
| 96 |
+
<div class="track"><div id="bar"></div></div>
|
| 97 |
+
<div class="row" style="margin-top:10px"><span class="diff" id="diff"></span></div>
|
| 98 |
+
</div>
|
| 99 |
+
|
| 100 |
+
<div class="card">
|
| 101 |
+
<div class="lbl">💾 Model checkpoint</div>
|
| 102 |
+
<div style="display:flex;gap:10px;flex-wrap:wrap;justify-content:center">
|
| 103 |
+
<button id="save" disabled>Download model (.pt)</button>
|
| 104 |
+
<button id="loadBtn">Load checkpoint…</button>
|
| 105 |
+
<input type="file" id="load" accept=".pt" style="display:none">
|
| 106 |
+
</div>
|
| 107 |
+
<p class="note" style="margin:.6rem 0 0;text-align:center">Loading a checkpoint applies it here <b>and</b> pushes it to every connected device — use it to recover the group after a failure.</p>
|
| 108 |
+
</div>
|
| 109 |
+
|
| 110 |
+
<div class="card">
|
| 111 |
+
<div class="lbl">❋ Log</div>
|
| 112 |
+
<pre id="log"></pre>
|
| 113 |
+
</div>
|
| 114 |
+
|
| 115 |
+
<p class="note">Needs a secure context (localhost or HTTPS) for WebGPU + cross-device WebRTC. No WebGPU? The same verified INT8 units run on CPU — old machines (e.g. via Supermium) still join, just slower. Every training step goes through the Neural Units; there is no plain-float path, and if the units fail to load, training is disabled.</p>
|
| 116 |
+
<p class="note"><b>Heads up:</b> peers connect directly (WebRTC), so devices in your group can see each other's IP address, and there's no gradient authentication — only train with devices/people you trust. Proof of concept.</p>
|
| 117 |
+
|
| 118 |
+
<script src="traincore.js"></script>
|
| 119 |
+
<script src="verified_core.js"></script>
|
| 120 |
+
<script src="webgpu.js"></script>
|
| 121 |
+
<script src="app.js"></script>
|
| 122 |
+
</body>
|
| 123 |
+
</html>
|
web/public/luts_meta.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"mul": [256, 256], "requant": 65536, "relu": 256, "shift": 8}
|
web/public/mul_lut.bin
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5cecff7e22049d0083ad9ee36dcf0695222c61621bacfd5a401c7b133abe892d
|
| 3 |
+
size 131072
|
web/public/relu_lut.bin
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2acb03ba7520467636273208563f8e733494748f4aa5ac2dba89d9560050da79
|
| 3 |
+
size 256
|
web/public/requant_lut.bin
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:173444ecfa293433329a333289983a665c481d913e9fd1c2778b55380ca4dd31
|
| 3 |
+
size 65536
|
web/public/traincore.js
ADDED
|
@@ -0,0 +1,91 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// Shared training math — pure JS. Used as the WebGPU fallback in the browser,
|
| 2 |
+
// and unit-tested directly in Node. Model: linear regression Y = X @ W (MSE).
|
| 3 |
+
// The GEMMs here are exactly what the WebGPU compute shader replaces.
|
| 4 |
+
(function (root) {
|
| 5 |
+
"use strict";
|
| 6 |
+
|
| 7 |
+
// C(m×n) = A(m×k) @ B(k×n), all Float32Array row-major
|
| 8 |
+
function matmul(A, B, m, k, n) {
|
| 9 |
+
const C = new Float32Array(m * n);
|
| 10 |
+
for (let i = 0; i < m; i++) {
|
| 11 |
+
for (let p = 0; p < k; p++) {
|
| 12 |
+
const a = A[i * k + p];
|
| 13 |
+
if (a === 0) continue;
|
| 14 |
+
const bo = p * n, co = i * n;
|
| 15 |
+
for (let j = 0; j < n; j++) C[co + j] += a * B[bo + j];
|
| 16 |
+
}
|
| 17 |
+
}
|
| 18 |
+
return C;
|
| 19 |
+
}
|
| 20 |
+
|
| 21 |
+
function transpose(A, rows, cols) {
|
| 22 |
+
const T = new Float32Array(rows * cols);
|
| 23 |
+
for (let i = 0; i < rows; i++)
|
| 24 |
+
for (let j = 0; j < cols; j++) T[j * rows + i] = A[i * cols + j];
|
| 25 |
+
return T;
|
| 26 |
+
}
|
| 27 |
+
|
| 28 |
+
// Forward + loss + gradient for one shard.
|
| 29 |
+
// X: n×din, W: din×dout, y: n×dout
|
| 30 |
+
// gradW = (2/n) * Xᵀ @ (X@W - y) (din×dout)
|
| 31 |
+
// matmulFn lets the browser swap in the WebGPU GEMM (same signature as matmul).
|
| 32 |
+
function forwardLossGrad(X, y, W, n, din, dout, matmulFn) {
|
| 33 |
+
const mm = matmulFn || matmul;
|
| 34 |
+
const pred = mm(X, W, n, din, dout); // n×dout (GEMM)
|
| 35 |
+
const resid = new Float32Array(n * dout);
|
| 36 |
+
let loss = 0;
|
| 37 |
+
for (let i = 0; i < n * dout; i++) {
|
| 38 |
+
const r = pred[i] - y[i];
|
| 39 |
+
resid[i] = r; loss += r * r;
|
| 40 |
+
}
|
| 41 |
+
loss /= (n * dout);
|
| 42 |
+
const Xt = transpose(X, n, din); // din×n
|
| 43 |
+
const g = mm(Xt, resid, din, n, dout); // din×dout (GEMM)
|
| 44 |
+
const scale = 2 / n;
|
| 45 |
+
for (let i = 0; i < g.length; i++) g[i] *= scale;
|
| 46 |
+
return { pred, loss, gradW: g };
|
| 47 |
+
}
|
| 48 |
+
|
| 49 |
+
function applyGrad(W, gradAvg, lr) {
|
| 50 |
+
for (let i = 0; i < W.length; i++) W[i] -= lr * gradAvg[i];
|
| 51 |
+
}
|
| 52 |
+
|
| 53 |
+
// average a list of gradient Float32Arrays (equal weight)
|
| 54 |
+
function averageGrads(grads) {
|
| 55 |
+
const out = new Float32Array(grads[0].length);
|
| 56 |
+
for (const g of grads) for (let i = 0; i < g.length; i++) out[i] += g[i];
|
| 57 |
+
for (let i = 0; i < out.length; i++) out[i] /= grads.length;
|
| 58 |
+
return out;
|
| 59 |
+
}
|
| 60 |
+
|
| 61 |
+
// DaisyAdam — Adam with bias correction, applied to the cluster-averaged
|
| 62 |
+
// gradient. State is a pure function of the gradient sequence, so every peer
|
| 63 |
+
// that averages the same gradients keeps bit-identical moments: no optimizer
|
| 64 |
+
// state ever crosses the wire. Momentum also smooths the noisy STE gradients
|
| 65 |
+
// coming out of the verified INT8 units.
|
| 66 |
+
function makeAdam(dim, opts) {
|
| 67 |
+
const o = opts || {};
|
| 68 |
+
const lr = o.lr ?? 0.02, b1 = o.beta1 ?? 0.9, b2 = o.beta2 ?? 0.999, eps = o.eps ?? 1e-8;
|
| 69 |
+
const m = new Float32Array(dim), v = new Float32Array(dim);
|
| 70 |
+
let t = 0;
|
| 71 |
+
return {
|
| 72 |
+
name: `adam(lr=${lr})`,
|
| 73 |
+
// returns the update u; caller does W[i] -= u[i]
|
| 74 |
+
step(g) {
|
| 75 |
+
t++;
|
| 76 |
+
const c1 = 1 - Math.pow(b1, t), c2 = 1 - Math.pow(b2, t);
|
| 77 |
+
const u = new Float32Array(dim);
|
| 78 |
+
for (let i = 0; i < dim; i++) {
|
| 79 |
+
m[i] = b1 * m[i] + (1 - b1) * g[i];
|
| 80 |
+
v[i] = b2 * v[i] + (1 - b2) * g[i] * g[i];
|
| 81 |
+
u[i] = lr * (m[i] / c1) / (Math.sqrt(v[i] / c2) + eps);
|
| 82 |
+
}
|
| 83 |
+
return u;
|
| 84 |
+
},
|
| 85 |
+
};
|
| 86 |
+
}
|
| 87 |
+
|
| 88 |
+
const api = { matmul, transpose, forwardLossGrad, applyGrad, averageGrads, makeAdam };
|
| 89 |
+
if (typeof module !== "undefined" && module.exports) module.exports = api;
|
| 90 |
+
else root.TrainCore = api;
|
| 91 |
+
})(typeof self !== "undefined" ? self : this);
|
web/public/verified_core.js
ADDED
|
@@ -0,0 +1,84 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// Verified INT8 compute — the emulated GPU logic, in the browser.
|
| 2 |
+
// A layer's forward runs THROUGH the units: quantize -> LUT multiply -> requant
|
| 3 |
+
// -> optional ReLU -> dequant. Backward is a straight-through estimator (the
|
| 4 |
+
// integer path has no gradient), so ordinary float weights still learn.
|
| 5 |
+
// Same units as the Python/Docker DaisyChain; here they're lookup tables.
|
| 6 |
+
(function (root) {
|
| 7 |
+
"use strict";
|
| 8 |
+
|
| 9 |
+
let TC; // TrainCore (matmul/transpose) — resolved per environment at the end
|
| 10 |
+
|
| 11 |
+
function quantize(X) {
|
| 12 |
+
let mx = 0; for (let i = 0; i < X.length; i++) { const a = Math.abs(X[i]); if (a > mx) mx = a; }
|
| 13 |
+
const scale = Math.max(mx / 127, 1e-8);
|
| 14 |
+
const q = new Int8Array(X.length);
|
| 15 |
+
for (let i = 0; i < X.length; i++) { let v = Math.round(X[i] / scale); q[i] = v < -128 ? -128 : v > 127 ? 127 : v; }
|
| 16 |
+
return { q, scale };
|
| 17 |
+
}
|
| 18 |
+
|
| 19 |
+
// int8 matmul via the verified multiply LUT: acc(m×n) = sum_k mulLUT[Xq,Wq]
|
| 20 |
+
function lutMatmulJS(Xq, Wq, m, k, n, L) {
|
| 21 |
+
const C = new Int32Array(m * n), mul = L.mul;
|
| 22 |
+
for (let i = 0; i < m; i++) {
|
| 23 |
+
for (let p = 0; p < k; p++) {
|
| 24 |
+
const au = (Xq[i * k + p] & 0xFF) * 256, wo = p * n, co = i * n;
|
| 25 |
+
for (let j = 0; j < n; j++) C[co + j] += mul[au + (Wq[wo + j] & 0xFF)];
|
| 26 |
+
}
|
| 27 |
+
}
|
| 28 |
+
return C;
|
| 29 |
+
}
|
| 30 |
+
|
| 31 |
+
// one verified layer forward; returns float out (+ cache for STE backward).
|
| 32 |
+
// Every product goes through the verified INT8 multiply (mul8 LUT) with exact
|
| 33 |
+
// int32 accumulation — i.e. an emulated INT8 tensor-core GEMM — then dequant.
|
| 34 |
+
async function linearFwd(X, W, m, k, n, L, useRelu, matmulInt8) {
|
| 35 |
+
const xq = quantize(X), wq = quantize(W);
|
| 36 |
+
const acc = await (matmulInt8 || lutMatmulJS)(xq.q, wq.q, m, k, n, L); // verified multiply
|
| 37 |
+
const dq = xq.scale * wq.scale;
|
| 38 |
+
const out = new Float32Array(m * n);
|
| 39 |
+
const mask = useRelu ? new Uint8Array(m * n) : null;
|
| 40 |
+
for (let i = 0; i < m * n; i++) {
|
| 41 |
+
let v = acc[i] * dq;
|
| 42 |
+
if (useRelu) { if (v > 0) mask[i] = 1; else v = 0; }
|
| 43 |
+
out[i] = v;
|
| 44 |
+
}
|
| 45 |
+
return { out, mask };
|
| 46 |
+
}
|
| 47 |
+
|
| 48 |
+
// 2-layer MLP: X→H (relu) →dout. Forward through verified units, MSE loss.
|
| 49 |
+
async function forward(X, y, W1, W2, D, L, matmulInt8) {
|
| 50 |
+
const { n, din, h, dout } = D;
|
| 51 |
+
const l1 = await linearFwd(X, W1, n, din, h, L, true, matmulInt8);
|
| 52 |
+
const l2 = await linearFwd(l1.out, W2, n, h, dout, L, false, matmulInt8);
|
| 53 |
+
const resid = new Float32Array(n * dout); let loss = 0;
|
| 54 |
+
for (let i = 0; i < resid.length; i++) { const r = l2.out[i] - y[i]; resid[i] = r; loss += r * r; }
|
| 55 |
+
loss /= resid.length;
|
| 56 |
+
return { loss, resid, z1: l1.out, mask1: l1.mask };
|
| 57 |
+
}
|
| 58 |
+
|
| 59 |
+
// STE backward (verified matmul treated as float X@W). Returns flat [gW1, gW2].
|
| 60 |
+
function backward(X, W1, W2, fwd, D) {
|
| 61 |
+
const { n, din, h, dout } = D;
|
| 62 |
+
const { resid, z1, mask1 } = fwd;
|
| 63 |
+
const s = 2 / n;
|
| 64 |
+
const dout_ = new Float32Array(resid.length);
|
| 65 |
+
for (let i = 0; i < resid.length; i++) dout_[i] = resid[i] * s;
|
| 66 |
+
const mm = TC.matmul, tr = TC.transpose;
|
| 67 |
+
const gW2 = mm(tr(z1, n, h), dout_, h, n, dout); // z1ᵀ @ dout
|
| 68 |
+
const dz1 = mm(dout_, tr(W2, h, dout), n, dout, h); // dout @ W2ᵀ
|
| 69 |
+
for (let i = 0; i < dz1.length; i++) if (!mask1[i]) dz1[i] = 0; // relu grad
|
| 70 |
+
const gW1 = mm(tr(X, n, din), dz1, din, n, h); // Xᵀ @ dz1
|
| 71 |
+
const g = new Float32Array(gW1.length + gW2.length);
|
| 72 |
+
g.set(gW1, 0); g.set(gW2, gW1.length);
|
| 73 |
+
return g;
|
| 74 |
+
}
|
| 75 |
+
|
| 76 |
+
function splitApply(W1, W2, gAvg, lr) {
|
| 77 |
+
for (let i = 0; i < W1.length; i++) W1[i] -= lr * gAvg[i];
|
| 78 |
+
for (let j = 0; j < W2.length; j++) W2[j] -= lr * gAvg[W1.length + j];
|
| 79 |
+
}
|
| 80 |
+
|
| 81 |
+
const api = { quantize, lutMatmulJS, linearFwd, forward, backward, splitApply };
|
| 82 |
+
if (typeof module !== "undefined" && module.exports) { TC = require("./traincore.js"); module.exports = api; }
|
| 83 |
+
else { TC = root.TrainCore; root.Verified = api; }
|
| 84 |
+
})(typeof self !== "undefined" ? self : this);
|
web/public/webgpu.js
ADDED
|
@@ -0,0 +1,91 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// WebGPU INT8 matmul via the verified multiply LUT — the emulated GPU logic
|
| 2 |
+
// running on the browser's GPU. Automatic CPU fallback (same LUT) for machines
|
| 3 |
+
// without WebGPU (e.g. old PCs via Supermium). initCompute() returns
|
| 4 |
+
// { backend, label, matmulInt8(Xq, Wq, m, k, n, L) -> Int32Array }
|
| 5 |
+
// matching Verified.lutMatmulJS, so the trainer is device-blind.
|
| 6 |
+
(function (root) {
|
| 7 |
+
"use strict";
|
| 8 |
+
|
| 9 |
+
const WGSL = `
|
| 10 |
+
@group(0) @binding(0) var<storage, read> Xq : array<i32>; // int8 byte per elem
|
| 11 |
+
@group(0) @binding(1) var<storage, read> Wq : array<i32>;
|
| 12 |
+
@group(0) @binding(2) var<storage, read> lut : array<i32>; // 65536 signed products
|
| 13 |
+
@group(0) @binding(3) var<storage, read_write> C : array<i32>;
|
| 14 |
+
@group(0) @binding(4) var<uniform> dims : vec3<u32>; // m, k, n
|
| 15 |
+
@compute @workgroup_size(8, 8)
|
| 16 |
+
fn main(@builtin(global_invocation_id) gid : vec3<u32>) {
|
| 17 |
+
let m = dims.x; let k = dims.y; let n = dims.z;
|
| 18 |
+
let row = gid.x; let col = gid.y;
|
| 19 |
+
if (row >= m || col >= n) { return; }
|
| 20 |
+
var s : i32 = 0;
|
| 21 |
+
for (var p = 0u; p < k; p = p + 1u) {
|
| 22 |
+
let au = u32(Xq[row * k + p] & 255);
|
| 23 |
+
let bu = u32(Wq[p * n + col] & 255);
|
| 24 |
+
s = s + lut[au * 256u + bu];
|
| 25 |
+
}
|
| 26 |
+
C[row * n + col] = s;
|
| 27 |
+
}`;
|
| 28 |
+
|
| 29 |
+
async function loadLUTs(base) {
|
| 30 |
+
base = base || "";
|
| 31 |
+
const [mulB, reqB, reluB, meta] = await Promise.all([
|
| 32 |
+
fetch(base + "mul_lut.bin").then(r => r.arrayBuffer()),
|
| 33 |
+
fetch(base + "requant_lut.bin").then(r => r.arrayBuffer()),
|
| 34 |
+
fetch(base + "relu_lut.bin").then(r => r.arrayBuffer()),
|
| 35 |
+
fetch(base + "luts_meta.json").then(r => r.json()),
|
| 36 |
+
]);
|
| 37 |
+
return { mul: new Int16Array(mulB), requant: new Int8Array(reqB),
|
| 38 |
+
relu: new Int8Array(reluB), shift: meta.shift };
|
| 39 |
+
}
|
| 40 |
+
|
| 41 |
+
async function initCompute(L) {
|
| 42 |
+
const cpu = { backend: "cpu", label: "CPU (JS)",
|
| 43 |
+
matmulInt8: (Xq, Wq, m, k, n, LL) => root.Verified.lutMatmulJS(Xq, Wq, m, k, n, LL) };
|
| 44 |
+
if (!(root.navigator && navigator.gpu)) return cpu;
|
| 45 |
+
try {
|
| 46 |
+
const adapter = await navigator.gpu.requestAdapter();
|
| 47 |
+
if (!adapter) return cpu;
|
| 48 |
+
const device = await adapter.requestDevice();
|
| 49 |
+
const module = device.createShaderModule({ code: WGSL });
|
| 50 |
+
const pipeline = device.createComputePipeline({ layout: "auto", compute: { module, entryPoint: "main" } });
|
| 51 |
+
// upload the multiply LUT once (as i32)
|
| 52 |
+
const lut32 = new Int32Array(L.mul); // widen int16 -> int32
|
| 53 |
+
const lutBuf = device.createBuffer({ size: lut32.byteLength, usage: GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_DST });
|
| 54 |
+
device.queue.writeBuffer(lutBuf, 0, lut32);
|
| 55 |
+
const info = adapter.info || {};
|
| 56 |
+
return { backend: "webgpu", label: info.description || info.vendor || "WebGPU",
|
| 57 |
+
matmulInt8: (Xq, Wq, m, k, n) => gpuMatmul(device, pipeline, lutBuf, Xq, Wq, m, k, n) };
|
| 58 |
+
} catch (e) { console.warn("WebGPU init failed, CPU fallback:", e); return cpu; }
|
| 59 |
+
}
|
| 60 |
+
|
| 61 |
+
async function gpuMatmul(device, pipeline, lutBuf, Xq, Wq, m, k, n) {
|
| 62 |
+
const X32 = Int32Array.from(Xq), W32 = Int32Array.from(Wq); // byte -> i32
|
| 63 |
+
const bufX = mk(device, X32.byteLength, GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_DST);
|
| 64 |
+
const bufW = mk(device, W32.byteLength, GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_DST);
|
| 65 |
+
const bytesC = m * n * 4;
|
| 66 |
+
const bufC = mk(device, bytesC, GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_SRC);
|
| 67 |
+
const bufD = mk(device, 16, GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST);
|
| 68 |
+
device.queue.writeBuffer(bufX, 0, X32);
|
| 69 |
+
device.queue.writeBuffer(bufW, 0, W32);
|
| 70 |
+
device.queue.writeBuffer(bufD, 0, new Uint32Array([m, k, n]));
|
| 71 |
+
const bind = device.createBindGroup({ layout: pipeline.getBindGroupLayout(0), entries: [
|
| 72 |
+
{ binding: 0, resource: { buffer: bufX } }, { binding: 1, resource: { buffer: bufW } },
|
| 73 |
+
{ binding: 2, resource: { buffer: lutBuf } }, { binding: 3, resource: { buffer: bufC } },
|
| 74 |
+
{ binding: 4, resource: { buffer: bufD } } ] });
|
| 75 |
+
const enc = device.createCommandEncoder();
|
| 76 |
+
const pass = enc.beginComputePass();
|
| 77 |
+
pass.setPipeline(pipeline); pass.setBindGroup(0, bind);
|
| 78 |
+
pass.dispatchWorkgroups(Math.ceil(m / 8), Math.ceil(n / 8)); pass.end();
|
| 79 |
+
const read = mk(device, bytesC, GPUBufferUsage.COPY_DST | GPUBufferUsage.MAP_READ);
|
| 80 |
+
enc.copyBufferToBuffer(bufC, 0, read, 0, bytesC);
|
| 81 |
+
device.queue.submit([enc.finish()]);
|
| 82 |
+
await read.mapAsync(GPUMapMode.READ);
|
| 83 |
+
const out = new Int32Array(read.getMappedRange().slice(0));
|
| 84 |
+
read.unmap();
|
| 85 |
+
[bufX, bufW, bufC, bufD, read].forEach(b => b.destroy());
|
| 86 |
+
return out;
|
| 87 |
+
}
|
| 88 |
+
function mk(device, size, usage) { return device.createBuffer({ size, usage }); }
|
| 89 |
+
|
| 90 |
+
root.Compute = { initCompute, loadLUTs };
|
| 91 |
+
})(self);
|
web/server.js
ADDED
|
@@ -0,0 +1,111 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// DaisyChain-Web signaling + static host.
|
| 2 |
+
// - Serves public/ (the page users open).
|
| 3 |
+
// - WebSocket signaling: introduces peers in a room and relays WebRTC
|
| 4 |
+
// offers/answers/ICE. It never sees the compute — that's P2P over WebRTC.
|
| 5 |
+
// Only dependency: `ws`. Run: npm install && node server.js
|
| 6 |
+
const http = require("http");
|
| 7 |
+
const fs = require("fs");
|
| 8 |
+
const path = require("path");
|
| 9 |
+
const crypto = require("crypto");
|
| 10 |
+
const { WebSocketServer } = require("ws");
|
| 11 |
+
|
| 12 |
+
// Snapdrop-style: peers are grouped by their PUBLIC IP, so only devices on the
|
| 13 |
+
// same network auto-discover each other. An explicit ?room=CODE overrides this
|
| 14 |
+
// to connect across networks (share the code with people you invite).
|
| 15 |
+
function clientIP(req) {
|
| 16 |
+
const xff = req.headers["x-forwarded-for"];
|
| 17 |
+
if (xff) return xff.split(",")[0].trim();
|
| 18 |
+
return (req.socket.remoteAddress || "unknown").replace(/^::ffff:/, "");
|
| 19 |
+
}
|
| 20 |
+
function roomFor(req) {
|
| 21 |
+
const u = new URL(req.url, "http://x");
|
| 22 |
+
const code = u.searchParams.get("room");
|
| 23 |
+
if (code) return "room:" + code;
|
| 24 |
+
const h = crypto.createHash("sha256").update(clientIP(req)).digest("hex").slice(0, 10);
|
| 25 |
+
return "net:" + h; // same network -> same room (IP not exposed in the id)
|
| 26 |
+
}
|
| 27 |
+
|
| 28 |
+
const PORT = process.env.PORT || 8787;
|
| 29 |
+
const PUB = path.join(__dirname, "public");
|
| 30 |
+
const TYPES = { ".html": "text/html", ".js": "text/javascript",
|
| 31 |
+
".css": "text/css", ".json": "application/json" };
|
| 32 |
+
|
| 33 |
+
const server = http.createServer((req, res) => {
|
| 34 |
+
let p = decodeURIComponent(req.url.split("?")[0]);
|
| 35 |
+
if (p === "/") p = "/index.html";
|
| 36 |
+
const file = path.join(PUB, path.normalize(p));
|
| 37 |
+
if (!file.startsWith(PUB)) { res.writeHead(403); return res.end(); }
|
| 38 |
+
fs.readFile(file, (err, data) => {
|
| 39 |
+
if (err) { res.writeHead(404); return res.end("not found"); }
|
| 40 |
+
res.writeHead(200, { "Content-Type": TYPES[path.extname(file)] || "application/octet-stream" });
|
| 41 |
+
res.end(data);
|
| 42 |
+
});
|
| 43 |
+
});
|
| 44 |
+
|
| 45 |
+
const wss = new WebSocketServer({ server });
|
| 46 |
+
// roomId -> { peers: Map(id -> {ws,name}), host: id|null, pending: Map(id -> {ws,name}) }
|
| 47 |
+
// Private rooms (room:CODE) are gated: the creator is the host, and everyone
|
| 48 |
+
// arriving later waits until the host accepts them — knowing the code is not
|
| 49 |
+
// enough. Network rooms (net:) keep auto-join (same LAN, Snapdrop-style).
|
| 50 |
+
const rooms = new Map();
|
| 51 |
+
let nextId = 1;
|
| 52 |
+
|
| 53 |
+
function send(ws, obj) { if (ws.readyState === 1) ws.send(JSON.stringify(obj)); }
|
| 54 |
+
|
| 55 |
+
wss.on("connection", (ws, req) => {
|
| 56 |
+
const roomId = roomFor(req);
|
| 57 |
+
const name = (new URL(req.url, "http://x").searchParams.get("name") || "").slice(0, 40) || ("p" + nextId);
|
| 58 |
+
const id = "p" + (nextId++);
|
| 59 |
+
ws.peerId = id; ws.roomId = roomId;
|
| 60 |
+
if (!rooms.has(roomId)) rooms.set(roomId, { peers: new Map(), host: null, pending: new Map() });
|
| 61 |
+
const room = rooms.get(roomId);
|
| 62 |
+
const isPrivate = roomId.startsWith("room:");
|
| 63 |
+
|
| 64 |
+
function admit(pid, peer) {
|
| 65 |
+
const roster = [...room.peers.entries()].map(([qid, v]) => ({ id: qid, name: v.name }));
|
| 66 |
+
send(peer.ws, { type: "welcome", id: pid, room: roomId, peers: roster, host: room.host === pid });
|
| 67 |
+
for (const [, v] of room.peers) send(v.ws, { type: "peer-joined", id: pid, name: peer.name });
|
| 68 |
+
room.peers.set(pid, peer);
|
| 69 |
+
}
|
| 70 |
+
|
| 71 |
+
if (isPrivate && room.peers.size === 0) room.host = id; // creator hosts
|
| 72 |
+
if (isPrivate && room.host !== id) {
|
| 73 |
+
room.pending.set(id, { ws, name });
|
| 74 |
+
send(ws, { type: "waiting" });
|
| 75 |
+
const h = room.peers.get(room.host);
|
| 76 |
+
if (h) send(h.ws, { type: "join-request", id, name });
|
| 77 |
+
} else {
|
| 78 |
+
admit(id, { ws, name });
|
| 79 |
+
}
|
| 80 |
+
|
| 81 |
+
ws.on("message", (buf) => {
|
| 82 |
+
let msg; try { msg = JSON.parse(buf); } catch { return; }
|
| 83 |
+
if (msg.type === "signal" && msg.to && room.peers.has(id)) { // relay WebRTC signaling
|
| 84 |
+
const target = room.peers.get(msg.to);
|
| 85 |
+
if (target) send(target.ws, { type: "signal", from: id, data: msg.data });
|
| 86 |
+
} else if (msg.type === "admit" && id === room.host) { // host verdict on a joiner
|
| 87 |
+
const p = room.pending.get(msg.id);
|
| 88 |
+
if (!p) return;
|
| 89 |
+
room.pending.delete(msg.id);
|
| 90 |
+
if (msg.allow) admit(msg.id, p);
|
| 91 |
+
else { send(p.ws, { type: "denied" }); p.ws.close(); }
|
| 92 |
+
}
|
| 93 |
+
});
|
| 94 |
+
|
| 95 |
+
ws.on("close", () => {
|
| 96 |
+
room.pending.delete(id);
|
| 97 |
+
if (room.peers.delete(id))
|
| 98 |
+
for (const [, v] of room.peers) send(v.ws, { type: "peer-left", id });
|
| 99 |
+
if (room.host === id) { // host left: promote oldest
|
| 100 |
+
room.host = room.peers.keys().next().value ?? null;
|
| 101 |
+
const h = room.peers.get(room.host);
|
| 102 |
+
if (h) {
|
| 103 |
+
send(h.ws, { type: "host" });
|
| 104 |
+
for (const [pid, p] of room.pending) send(h.ws, { type: "join-request", id: pid, name: p.name });
|
| 105 |
+
}
|
| 106 |
+
}
|
| 107 |
+
if (room.peers.size === 0 && room.pending.size === 0) rooms.delete(roomId);
|
| 108 |
+
});
|
| 109 |
+
});
|
| 110 |
+
|
| 111 |
+
server.listen(PORT, () => console.log(`DaisyChain-Web on http://localhost:${PORT}`));
|
web/test_core.js
ADDED
|
@@ -0,0 +1,54 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// Verifies the training loop + 2-peer gradient averaging in pure Node (no
|
| 2 |
+
// browser). Simulates two peers each holding half the data; they average
|
| 3 |
+
// gradients every step. Proves: (a) it converges, (b) both replicas stay
|
| 4 |
+
// identical — the same guarantees the browser P2P version needs.
|
| 5 |
+
const T = require("./public/traincore.js");
|
| 6 |
+
|
| 7 |
+
function randn(n) {
|
| 8 |
+
const a = new Float32Array(n);
|
| 9 |
+
for (let i = 0; i < n; i++) {
|
| 10 |
+
let u = 0, v = 0;
|
| 11 |
+
while (u === 0) u = Math.random();
|
| 12 |
+
while (v === 0) v = Math.random();
|
| 13 |
+
a[i] = Math.sqrt(-2 * Math.log(u)) * Math.cos(2 * Math.PI * v);
|
| 14 |
+
}
|
| 15 |
+
return a;
|
| 16 |
+
}
|
| 17 |
+
|
| 18 |
+
const din = 16, dout = 4, nPer = 128, steps = 400, lr = 0.05;
|
| 19 |
+
|
| 20 |
+
// ground-truth weights
|
| 21 |
+
const Wtrue = randn(din * dout);
|
| 22 |
+
function makeShard() {
|
| 23 |
+
const X = randn(nPer * din);
|
| 24 |
+
const y = T.matmul(X, Wtrue, nPer, din, dout); // clean targets
|
| 25 |
+
return { X, y };
|
| 26 |
+
}
|
| 27 |
+
const A = makeShard(), B = makeShard();
|
| 28 |
+
|
| 29 |
+
// both peers start from the SAME W0 (initiator broadcasts it)
|
| 30 |
+
const W0 = randn(din * dout);
|
| 31 |
+
const Wa = Float32Array.from(W0), Wb = Float32Array.from(W0);
|
| 32 |
+
|
| 33 |
+
let loss = 0;
|
| 34 |
+
for (let s = 0; s < steps; s++) {
|
| 35 |
+
const ra = T.forwardLossGrad(A.X, A.y, Wa, nPer, din, dout);
|
| 36 |
+
const rb = T.forwardLossGrad(B.X, B.y, Wb, nPer, din, dout);
|
| 37 |
+
const avg = T.averageGrads([ra.gradW, rb.gradW]); // <-- exchanged P2P
|
| 38 |
+
T.applyGrad(Wa, avg, lr);
|
| 39 |
+
T.applyGrad(Wb, avg, lr);
|
| 40 |
+
loss = (ra.loss + rb.loss) / 2;
|
| 41 |
+
if (s % 80 === 0 || s === steps - 1)
|
| 42 |
+
console.log(` step ${s} cluster-avg loss ${loss.toFixed(6)}`);
|
| 43 |
+
}
|
| 44 |
+
|
| 45 |
+
let maxDiff = 0;
|
| 46 |
+
for (let i = 0; i < Wa.length; i++) maxDiff = Math.max(maxDiff, Math.abs(Wa[i] - Wb[i]));
|
| 47 |
+
let recovery = 0;
|
| 48 |
+
for (let i = 0; i < Wtrue.length; i++) recovery = Math.max(recovery, Math.abs(Wa[i] - Wtrue[i]));
|
| 49 |
+
|
| 50 |
+
console.log(`\nreplica max param diff: ${maxDiff.toExponential(3)}`);
|
| 51 |
+
console.log(`max |W - W_true|: ${recovery.toExponential(3)}`);
|
| 52 |
+
const ok = loss < 1e-3 && maxDiff < 1e-9 && recovery < 0.05;
|
| 53 |
+
console.log(ok ? "\nCORE TEST PASSED — converged, replicas in sync." : "\nCORE TEST FAILED");
|
| 54 |
+
process.exit(ok ? 0 : 1);
|
web/test_optimizer.js
ADDED
|
@@ -0,0 +1,63 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// Verifies DaisyAdam (TrainCore.makeAdam) on the path that matters: training
|
| 2 |
+
// THROUGH the verified INT8 units, where STE gradients are noisy and plain SGD
|
| 3 |
+
// plateaus. Checks (a) two replicas fed the same averaged gradients stay
|
| 4 |
+
// bit-identical, (b) Adam reaches a lower loss than SGD in the same steps.
|
| 5 |
+
const fs = require("fs");
|
| 6 |
+
const path = require("path");
|
| 7 |
+
const T = require("./public/traincore.js");
|
| 8 |
+
const V = require("./public/verified_core.js");
|
| 9 |
+
|
| 10 |
+
function loadLUTs() {
|
| 11 |
+
const p = (f) => path.join(__dirname, "public", f);
|
| 12 |
+
return {
|
| 13 |
+
mul: new Int16Array(fs.readFileSync(p("mul_lut.bin")).buffer.slice(0)),
|
| 14 |
+
requant: new Int8Array(fs.readFileSync(p("requant_lut.bin")).buffer.slice(0)),
|
| 15 |
+
relu: new Int8Array(fs.readFileSync(p("relu_lut.bin")).buffer.slice(0)),
|
| 16 |
+
};
|
| 17 |
+
}
|
| 18 |
+
function randn(n, rng) { const r = rng || Math.random; const o = new Float32Array(n); for (let i = 0; i < n; i += 2) { let u = 0, v = 0; while (u === 0) u = r(); while (v === 0) v = r(); const m = Math.sqrt(-2 * Math.log(u)); o[i] = m * Math.cos(2 * Math.PI * v); if (i + 1 < n) o[i + 1] = m * Math.sin(2 * Math.PI * v); } return o; }
|
| 19 |
+
function mulberry32(a) { return function () { a |= 0; a = a + 0x6D2B79F5 | 0; let t = Math.imul(a ^ a >>> 15, 1 | a); t = t + Math.imul(t ^ t >>> 7, 61 | t) ^ t; return ((t ^ t >>> 14) >>> 0) / 4294967296; }; }
|
| 20 |
+
|
| 21 |
+
const L = loadLUTs();
|
| 22 |
+
const D = { n: 128, din: 16, h: 16, dout: 4 }, steps = 300;
|
| 23 |
+
|
| 24 |
+
const Wtrue1 = randn(D.din * D.h, mulberry32(42)), Wtrue2 = randn(D.h * D.dout, mulberry32(43));
|
| 25 |
+
function target(X) { const hpre = T.matmul(X, Wtrue1, D.n, D.din, D.h); for (let i = 0; i < hpre.length; i++) hpre[i] = Math.max(0, hpre[i]); return T.matmul(hpre, Wtrue2, D.n, D.h, D.dout); }
|
| 26 |
+
const XA = randn(D.n * D.din, mulberry32(11)), yA = target(XA);
|
| 27 |
+
const XB = randn(D.n * D.din, mulberry32(12)), yB = target(XB);
|
| 28 |
+
|
| 29 |
+
async function run(useAdam) {
|
| 30 |
+
const W1a = randn(D.din * D.h, mulberry32(7)), W2a = randn(D.h * D.dout, mulberry32(8));
|
| 31 |
+
const W1b = Float32Array.from(W1a), W2b = Float32Array.from(W2a);
|
| 32 |
+
const dim = W1a.length + W2a.length;
|
| 33 |
+
const oa = T.makeAdam(dim, { lr: 0.2 }), ob = T.makeAdam(dim, { lr: 0.2 });
|
| 34 |
+
let loss = 0;
|
| 35 |
+
for (let s = 0; s < steps; s++) {
|
| 36 |
+
const fa = await V.forward(XA, yA, W1a, W2a, D, L), ga = V.backward(XA, W1a, W2a, fa, D);
|
| 37 |
+
const fb = await V.forward(XB, yB, W1b, W2b, D, L), gb = V.backward(XB, W1b, W2b, fb, D);
|
| 38 |
+
const avg = T.averageGrads([ga, gb]);
|
| 39 |
+
if (useAdam) {
|
| 40 |
+
V.splitApply(W1a, W2a, oa.step(avg), 1);
|
| 41 |
+
V.splitApply(W1b, W2b, ob.step(avg), 1);
|
| 42 |
+
} else {
|
| 43 |
+
V.splitApply(W1a, W2a, avg, 0.03);
|
| 44 |
+
V.splitApply(W1b, W2b, avg, 0.03);
|
| 45 |
+
}
|
| 46 |
+
loss = (fa.loss + fb.loss) / 2;
|
| 47 |
+
}
|
| 48 |
+
let diff = 0;
|
| 49 |
+
for (let i = 0; i < W1a.length; i++) diff = Math.max(diff, Math.abs(W1a[i] - W1b[i]));
|
| 50 |
+
for (let i = 0; i < W2a.length; i++) diff = Math.max(diff, Math.abs(W2a[i] - W2b[i]));
|
| 51 |
+
return { loss, diff };
|
| 52 |
+
}
|
| 53 |
+
|
| 54 |
+
(async function () {
|
| 55 |
+
const sgd = await run(false);
|
| 56 |
+
const adam = await run(true);
|
| 57 |
+
console.log(`SGD(lr=0.03) final loss ${sgd.loss.toFixed(5)}`);
|
| 58 |
+
console.log(`DaisyAdam(lr=0.2) final loss ${adam.loss.toFixed(5)} replica diff ${adam.diff.toExponential(3)}`);
|
| 59 |
+
const ok = adam.diff === 0 && adam.loss < sgd.loss;
|
| 60 |
+
console.log(ok ? "\nOPTIMIZER TEST PASSED — deterministic replicas, beats SGD through the verified units."
|
| 61 |
+
: "\nOPTIMIZER TEST FAILED");
|
| 62 |
+
process.exit(ok ? 0 : 1);
|
| 63 |
+
})();
|
web/test_verified.js
ADDED
|
@@ -0,0 +1,48 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// Proves 2-peer training THROUGH the verified units (LUTs) converges and stays
|
| 2 |
+
// in sync — the browser will do exactly this, with the matmul on WebGPU.
|
| 3 |
+
const fs = require("fs");
|
| 4 |
+
const path = require("path");
|
| 5 |
+
const T = require("./public/traincore.js");
|
| 6 |
+
const V = require("./public/verified_core.js");
|
| 7 |
+
|
| 8 |
+
function loadLUTs() {
|
| 9 |
+
const p = (f) => path.join(__dirname, "public", f);
|
| 10 |
+
const mul = new Int16Array(fs.readFileSync(p("mul_lut.bin")).buffer.slice(0));
|
| 11 |
+
const requant = new Int8Array(fs.readFileSync(p("requant_lut.bin")).buffer.slice(0));
|
| 12 |
+
const relu = new Int8Array(fs.readFileSync(p("relu_lut.bin")).buffer.slice(0));
|
| 13 |
+
return { mul, requant, relu };
|
| 14 |
+
}
|
| 15 |
+
|
| 16 |
+
function randn(n, rng) { const r = rng || Math.random; const o = new Float32Array(n); for (let i = 0; i < n; i += 2) { let u = 0, v = 0; while (u === 0) u = r(); while (v === 0) v = r(); const m = Math.sqrt(-2 * Math.log(u)); o[i] = m * Math.cos(2 * Math.PI * v); if (i + 1 < n) o[i + 1] = m * Math.sin(2 * Math.PI * v); } return o; }
|
| 17 |
+
function mulberry32(a) { return function () { a |= 0; a = a + 0x6D2B79F5 | 0; let t = Math.imul(a ^ a >>> 15, 1 | a); t = t + Math.imul(t ^ t >>> 7, 61 | t) ^ t; return ((t ^ t >>> 14) >>> 0) / 4294967296; }; }
|
| 18 |
+
|
| 19 |
+
const L = loadLUTs();
|
| 20 |
+
const D = { n: 128, din: 16, h: 16, dout: 4 }, lr = 0.03, steps = 300;
|
| 21 |
+
|
| 22 |
+
// shared deterministic truth + init; per-peer data shard
|
| 23 |
+
const Wtrue1 = randn(D.din * D.h, mulberry32(42)), Wtrue2 = randn(D.h * D.dout, mulberry32(43));
|
| 24 |
+
function target(X) { const hpre = T.matmul(X, Wtrue1, D.n, D.din, D.h); for (let i = 0; i < hpre.length; i++) hpre[i] = Math.max(0, hpre[i]); return T.matmul(hpre, Wtrue2, D.n, D.h, D.dout); }
|
| 25 |
+
const XA = randn(D.n * D.din), yA = target(XA);
|
| 26 |
+
const XB = randn(D.n * D.din), yB = target(XB);
|
| 27 |
+
|
| 28 |
+
const W1a = randn(D.din * D.h, mulberry32(7)), W2a = randn(D.h * D.dout, mulberry32(8));
|
| 29 |
+
const W1b = Float32Array.from(W1a), W2b = Float32Array.from(W2a);
|
| 30 |
+
|
| 31 |
+
(async function () {
|
| 32 |
+
let loss = 0, loss0 = 0;
|
| 33 |
+
for (let s = 0; s < steps; s++) {
|
| 34 |
+
const fa = await V.forward(XA, yA, W1a, W2a, D, L), ga = V.backward(XA, W1a, W2a, fa, D);
|
| 35 |
+
const fb = await V.forward(XB, yB, W1b, W2b, D, L), gb = V.backward(XB, W1b, W2b, fb, D);
|
| 36 |
+
const avg = T.averageGrads([ga, gb]); // <-- exchanged P2P
|
| 37 |
+
V.splitApply(W1a, W2a, avg, lr);
|
| 38 |
+
V.splitApply(W1b, W2b, avg, lr);
|
| 39 |
+
loss = (fa.loss + fb.loss) / 2; if (s === 0) loss0 = loss;
|
| 40 |
+
if (s % 60 === 0 || s === steps - 1) console.log(` step ${s} cluster-avg loss ${loss.toFixed(5)}`);
|
| 41 |
+
}
|
| 42 |
+
let diff = 0; for (let i = 0; i < W1a.length; i++) diff = Math.max(diff, Math.abs(W1a[i] - W1b[i]));
|
| 43 |
+
for (let i = 0; i < W2a.length; i++) diff = Math.max(diff, Math.abs(W2a[i] - W2b[i]));
|
| 44 |
+
console.log(`\nreplica max param diff: ${diff.toExponential(3)}`);
|
| 45 |
+
const ok = loss < loss0 * 0.3 && diff < 1e-9;
|
| 46 |
+
console.log(ok ? "VERIFIED TEST PASSED — trained through the units, converged, replicas in sync." : "FAILED");
|
| 47 |
+
process.exit(ok ? 0 : 1);
|
| 48 |
+
})();
|