ReLU-CLIP

Frozen CLIP image encoders distilled into small ReLU/ReLU6 + BatchNorm CNNs that quantise to int8 and run as a single graph on low-power edge accelerators, whose operator coverage is strongest for convolutional networks.

Code, figures and the full result set: https://github.com/jiaheguo521/relu-clip

This repository holds the weights: the deployable model, and the int8 graphs for all 18 runs of the accompanying study so its central claim can be checked independently.

Headline model — efflite4 distilled from CLIP ViT-L/14

• 68.25% ImageNet-1k zero-shot top-1 in int8 — 68.35% in fp32, so quantisation costs 0.10 pp
• 12.71M parameters, 13.6 MiB as an int8 graph
• 26.49 ms/frame (37.8 FPS) measured, with 7.5 MiB of weights streamed off-chip
• 116 of 116 operators on the accelerator, 0 on CPU — the whole image tower runs as one graph

Evaluated on the complete 50,000-image ImageNet-1k validation set with the TFLite CPU interpreter; on-device int8 agrees to 0.48 pp on average across all 18 runs. Latency measured on a Coral USB Edge TPU, once per run over a 500-frame pack, first frame dropped.

Files

efflite4-vitl14/
  student.safetensors           fp32 weights, 12.71M parameters -- prefer this
  student.pt                    the same model as a pickled nn.Module (see note below)
  student_int8.tflite           per-channel int8, runs on any TFLite CPU runtime
  student_int8_edgetpu.tflite   compiled with edgetpu_compiler 16.0
text/
  imagenet1k_text_emb_vitl14.npz  1000 x 768 class embeddings (keys: embs, labels)
  demo_prompt_emb_vitl14.npz      open-vocabulary demo prompts, encoded verbatim (keys: text_emb, labels)
sweep-int8/
  <teacher>__<student>_int8.tflite   all 18 runs (2 teachers x 9 students)
example_infer.py                runnable example -- numpy + Pillow + a TFLite runtime, nothing else
preprocessing.json              machine-readable input/output spec
matrix.csv, results_full.json, activation_ranges.csv, derisk_table.csv

Only want the deployable model? sweep-int8/ is 384 of the 500 MB:

from huggingface_hub import snapshot_download
snapshot_download("jiaheguo521/relu-clip", allow_patterns=["efflite4-vitl14/*", "text/*", "*.json", "*.py"])

Two things that will silently give wrong results if missed:

• imagenet1k_text_emb_vitl14.npz is not L2-normalised. Each row is an average over the 80 OpenAI ImageNet prompt templates, and averaging unit vectors gives a norm below 1 (here 0.81-0.93). Normalise it yourself before taking cosines. demo_prompt_emb_vitl14.npz is normalised — its prompts are encoded verbatim, without templating.
• The image graph emits an unnormalised embedding. L2 normalisation is deliberately left out of the int8 graph, so the accelerator subgraph stays free of reductions. Do it on the host after dequantising.

student.pt is a pickled nn.Module, so loading it runs torch.load(..., weights_only=False) and needs this project's sources/ importable. It is included because the repository's own conversion and evaluation scripts consume that format. For anything else, use student.safetensors, which carries the identical weights (verified bit-exact) and loads without executing pickled code:

from safetensors.torch import load_file
from students import build_student          # from the GitHub repo
model = build_student("efflite4", embed_dim=768, pretrained=False)
model.load_state_dict(load_file("efflite4-vitl14/student.safetensors"))
model.eval()

Usage

pip install numpy pillow tflite-runtime      # no torch, no CLIP
python example_infer.py your_photo.jpg

example_infer.py is the whole thing end to end, and its preprocessing is verified bit-equivalent to the transform the student was trained with across 40 aspect ratios. Two details in it are easy to get wrong and cost accuracy silently, so preprocessing.json states them machine-readably: Resize(224) truncates the long side, and CenterCrop rounds the crop offset. Off by one pixel in either and the crop shifts.

The image tower is a plain int8 TFLite graph; zero-shot classification is a cosine between its L2-normalised output and a stored text-embedding matrix. No CLIP text encoder is needed at inference.

from tflite_runtime.interpreter import Interpreter   # example_infer.py also falls back to
                                                     # ai_edge_litert or full tensorflow
import numpy as np

itp = Interpreter("efflite4-vitl14/student_int8.tflite")
itp.allocate_tensors()
inp, out = itp.get_input_details()[0], itp.get_output_details()[0]

text = np.load("text/imagenet1k_text_emb_vitl14.npz")
T, labels = text["embs"], text["labels"]                   # [1000, 768]
T = T / np.linalg.norm(T, axis=1, keepdims=True)           # NOT normalised on disk -- see above

# img: [1,224,224,3], resized to 224 bicubic + CLIP-normalised, then quantised with inp["quantization"]
s_in, z_in = inp["quantization"]
itp.set_tensor(inp["index"], np.clip(np.round(img / s_in + z_in), -128, 127).astype(inp["dtype"]))
itp.invoke()
emb = itp.get_tensor(out["index"]).astype(np.float32)
s, z = out["quantization"]
emb = (emb - z) * s                                        # dequantise
emb /= np.linalg.norm(emb, axis=-1, keepdims=True)         # the graph emits UNNORMALISED embeddings
print(labels[(emb @ T.T).argmax(-1)])

Your own classes

Changing the sentences changes the classifier; nothing is retrained. Write the phrases one per line and encode them once:

pip install torch --index-url https://download.pytorch.org/whl/cpu   # CPU wheel: skips ~2.7 GB of CUDA libraries
pip install open_clip_torch

python sources/make_prompt_emb.py --teacher vitl14 \
    --labels-file my_labels.txt --out my_labels.npz               # downloads ViT-L/14 once, 1.6 GB

python example_infer.py your_photo.jpg --text my_labels.npz       # back to numpy + tflite only

This is a build step, not a runtime dependency: run it once on any laptop and copy the resulting few-KB .npz to the device. Text encoding never happens on the accelerator — that is what makes the deployed graph a single int8 CNN.

text/demo_prompt_emb_vitl14.npz is a worked example of the output.

For the Edge TPU binary, load student_int8_edgetpu.tflite with the libedgetpu.so.1 delegate. It was produced by edgetpu_compiler 16.0 and needs a matched-generation runtime; the GitHub README documents which builds work.

The sweep, and why the int8 graphs are published

Across 2 teachers x 9 students, int8 does not preserve the fp32 ranking:

teacher / student	activation	fp32	int8	Δ
vitl14 / efflite4	ReLU6	68.35	68.25	-0.10
vitl14 / r50	ReLU	67.94	60.03	-7.90
vitl14 / r50_relu6	ReLU6	68.13	67.62	-0.50
vitl14 / r101	ReLU	70.70	12.09	-58.62
vitl14 / r101_relu6	ReLU6	69.09	67.27	-1.82

r50_relu6 / r101_relu6 are the same ResNets with every nn.ReLU replaced by nn.ReLU6 — same depth, same parameter count, same pretrained weights, same recipe. Bounding the activation is what recovers the accuracy.

The mechanism is measurable directly from these files. int8 spans a tensor's range in 255 steps, so the step is range / 255; every ReLU6 run has a median activation range of exactly 6.00 (step 0.0235), while the plain-ReLU ResNets sit at 59.2–209.8 with worst tensors reaching 4547.1 (step 17.83).

What that costs shows up once real images flow through the graph. A tensor calibrated for a range the data never reaches leaves most of that range empty: over 2000 images, every ReLU6 backbone puts data across 0.52–1.00 of the range it was calibrated for, while the unbounded ResNets reach only 0.045–0.147 — the two groups do not overlap. In int8 codes that is 256 of 256 against 61–218, with rn50/r101's worst tensor left with 5 — 2.3 bits. Across the 18 runs, how many codes a network keeps tracks how much it loses (Spearman −0.88), and it explains why depth is the wrong variable to read the table by: vitl14/r50 is deeper than r34 yet uses 0.118 of its calibrated range against 0.045 and loses 7.9 points against 28.0.

The code counts are a coverage statistic — a code counts as used once any image hits it, so the number grows with the sample and has not converged even at 2000. Compare networks on the range share, which drifts by at most 0.020 in absolute terms between 64 and 2000 images (read as a relative change it reaches −9.5%); quote code counts only at a stated sample size, and with the image pack they came from.

sweep-int8/ is published so that anyone can measure both properties on the same graphs that produced the accuracy numbers, rather than taking a CSV on trust. The activation ranges need only the graphs; the level usage also needs an image pack, rebuilt from ImageNet-1k val (2000 images, class-strided at 2 per class):

python sources/activation_ranges.py                                   # from the GitHub repo
python sources/quant_levels.py --pack outputs/artifacts/<a device pack> --n-images 2000

Every trained run compiles with 0 CPU operators — all 18, not just the pre-training gate in derisk_table.csv — so none of this is an operator-coverage artifact.

Limitations

• Single seed, no variance estimate. The bounded-vs-unbounded gap is far larger than any plausible run-to-run noise, but that noise was never measured, and rankings within the frontier should not be over-interpreted.
• Latency was originally measured once per architecture and reused across teachers, described as a sub-1% approximation. Re-timing all 18 runs individually put the nine reused 1024-d rows at −0.2% to +3.5% of their copied values — six of the nine off by more than 1.4%, the other three still inside 0.4% — while the reuse across ReLU/ReLU6 variants held to 0.5%. The published numbers are now per-run measurements, with the superseded values kept in latency_measured.csv on GitHub.
• Accuracy is CPU-interpreter int8, not on-device int8. The two agree to 0.48 pp on average across all 18 runs (500 images, worst 1.40 pp), but they are not bit-identical, and the headline pairs a CPU-measured accuracy with a device-measured latency.
• ReLU6 is not demonstrably fp32-lossless — r50_relu6 (68.13) is close to r50 (67.94), but r101_relu6 costs 1.6 pp of fp32 against r101 at an identical epoch count. The int8 conclusions are unaffected, since each drop is relative to that model's own fp32.
• r34 and r50 may be undertrained, as early stopping was driven by a 5k validation subset whose single-eval noise is several times the improvement threshold. rn50/r50_relu6 is the sharpest case — 6 epochs, best at epoch 1 — and it sits inside the 2×2 control.
• Activation ranges are a property of the network and its calibration set: they are what the quantiser observed over 512 class-strided images.
• Bounded activations quantising better than unbounded ones is a known result — ReLU6 was adopted in MobileNetV2 for exactly this reason, and the quantization whitepaper covers the per-channel scheme used here. What is measured is its magnitude and consequences under a specific deployment constraint, with same-parameter controls; no novelty is claimed for the mechanism.
• Operator coverage and the compiler behaviour described on GitHub are specific to one toolchain generation. The activation-range result is toolchain-independent; the compile details are not.
• The unbounded-ReLU graphs in sweep-int8/ are published as evidence, not as usable models — rn50__r101_int8.tflite scores 0.72%.

Acknowledgements

Carried out at LAAS-CNRS under the supervision of Tomasz Kłoda, whose regular discussions and feedback I am grateful for. Thanks also to Mouad Zouhdi, a fellow intern, for the day-to-day exchanges.

License

MIT.