Argus

Argus is a multi-task perception system built on a single compact vision backbone. From one forward pass through the encoder, the model produces classification labels, semantic segmentation masks, metric depth maps, and dense keypoint correspondences, thereby collapsing four domain-specific pipelines into a unified package of roughly 86 million parameters. The system is named after Argus Panoptes, the many-eyed giant of Greek mythology who was tasked by Hera with watching over everything at once.

The underlying backbone is EUPE-ViT-B, which was introduced in Efficient Universal Perception Encoder (Zhu et al., Meta FAIR, arXiv:2603.22387, March 2026). That paper demonstrates that a small vision encoder can be distilled from a collection of larger specialist teachers, yielding features that transfer well to image understanding, dense prediction, and vision–language tasks simultaneously. Argus takes the released EUPE-ViT-B backbone, leaves its weights frozen, and attaches four lightweight task heads that were trained or constructed independently for this project.

Architecture

Image → EUPE-ViT-B (frozen, 86M parameters) → shared features

  ├── Classification   — two methods on the same CLS token:
  │                       kNN over 1000 class prototypes (default)
  │                       trained linear softmax over 1000 classes
  ├── Segmentation     — linear head, 150 ADE20K classes
  ├── Depth            — two decoders: linear probe (default) or DPT multi-scale
  ├── Detection        — FCOS head with simple feature pyramid, 80 COCO classes
  └── Correspondence   — training-free dense feature matching

The segmentation head consists of a BatchNorm layer followed by a single 1×1 convolution, trained with the backbone held frozen throughout. Depth estimation supports two decoders: a linear probe (same BatchNorm + 1×1 Conv architecture as segmentation, the default) and a DPT multi-scale decoder that hooks into four evenly-spaced ViT blocks and fuses their features at four spatial scales via residual conv fusion. The DPT decoder improves RMSE by 8% over the linear probe on NYUv2 and is selected via decoder="dpt" on the depth() method. Classification supports two methods operating on the same normalized CLS token: a kNN protocol that computes cosine similarity against a precomputed matrix of 1000 class prototypes built from the full ImageNet-1k training set, and a trained linear softmax classifier consisting of a single Linear(768, 1000) layer with bias. Both methods run from the same backbone forward pass and the caller selects between them via a method argument on classify(). Detection uses an FCOS-style anchor-free detector built on a ViTDet-style simple feature pyramid that synthesizes five spatial scales (strides 8 through 128) from the backbone's stride-16 patch features, with shared four-layer convolutional towers for classification and box regression across all pyramid levels. The detection head runs at 640-pixel input with letterbox padding to preserve aspect ratio, and returns per-image lists of bounding boxes with class labels, confidence scores, and COCO class names. Keypoint correspondence requires no trained parameters at all: source and target features are extracted from two images, upsampled to pixel resolution, and matched by cosine similarity at each source keypoint.

Reproduction of the EUPE Paper

All four of the paper's reported benchmarks were reproduced as part of building Argus, and the results either matched the published numbers within rounding error or exceeded them modestly.

Task	Dataset	Metric	Paper	Argus	Delta
Classification	ImageNet-1k	kNN k=10 top-1	84.1	84.07	−0.03
Segmentation	ADE20K	mean IoU	52.4	52.72	+0.32
Depth	NYU Depth v2	RMSE (lower is better)	0.391	0.3914	+0.0004
Correspondence	SPair-71k	PCK@0.1	51.3	54.35	+3.05

The classification evaluation used the full 1.28-million-image ImageNet-1k training set as the kNN reference and the 50,000-image validation set as the query. The segmentation and depth heads were trained using the same linear-probe configurations described in the EUPE repository. Correspondence was evaluated on the SPair-71k test split at 512-pixel resolution across all 12,234 test pairs, for a total of 88,328 keypoints, with no failures during the run.

The trained linear softmax classifier was added after the paper reproduction work and is not a paper benchmark. On ImageNet-1k val it reaches 85.53% top-1 and 97.69% top-5, which improves on the kNN reference by +1.46 points on top-1 and +3.70 points on top-5. The top-5 improvement is the more meaningful number: softmax is decisive where nearest-mean kNN is flat on visually similar classes.

Classification method	Top-1	Top-5
kNN (k=10)	84.07 %	93.99 %
Linear softmax	85.53 %	97.69 %

Detection

The FCOS detection head was trained on COCO 2017 train (117,266 images) at 640-pixel input with the backbone frozen throughout. Evaluation on COCO val2017 (5,000 images) with the standard pycocotools protocol:

Metric	Value
mAP@[0.5:0.95]	41.0
mAP@0.50	64.8
mAP@0.75	43.2
mAP (small objects)	21.4
mAP (medium objects)	44.9
mAP (large objects)	62.1

For context, FCOS with a fully-trained ResNet-50-FPN backbone achieves 39.1 mAP on the same benchmark. The frozen EUPE-ViT-B backbone at 41.0 mAP exceeds that baseline while sharing its features with four other task heads simultaneously.

DPT Depth Decoder

The DPT multi-scale depth decoder was trained on the same NYUv2 dataset as the original linear probe. Evaluation on the NYUv2 test set (654 samples, 416×416):

Decoder	RMSE ↓	abs_rel ↓	a1 (δ<1.25) ↑
Linear probe	0.520	0.303	0.860
DPT multi-scale	0.480	0.219	0.872
Delta	-0.040 (8%)	-0.084 (28%)	+0.012

The improvement comes from the DPT decoder's access to intermediate ViT layers (blocks 2, 5, 8, 11) which carry low-level spatial detail that the final-layer linear probe cannot see. Both decoders produce the same output format (metric depth in meters via 256-bin weighted sum) and are selectable per-call via the decoder argument.

Cross-Dataset Segmentation Transfer

To test whether the backbone's features generalize beyond the ADE20K distribution on which the segmentation head was trained, a separate linear probe (identical BatchNorm + 1×1 Conv architecture, same training recipe) was trained on the Cityscapes urban driving dataset using the frozen backbone. Cityscapes contains 2,975 training images and 500 validation images of street scenes captured from a vehicle-mounted camera in German cities, annotated with 19 semantic classes. The backbone was never exposed to driving scenes during EUPE's multi-teacher distillation or during any phase of Argus head training.

Dataset	Classes	Train images	mIoU
ADE20K (original head)	150	20,210	52.72%
Cityscapes (transfer probe)	19	2,975	63.76%

The Cityscapes probe reaches 63.76% mIoU, with road at 96.4%, car at 87.9%, sky at 88.8%, building at 86.7%, and vegetation at 85.6%. The weaker categories are thin vertical structures — pole at 17.8%, traffic light at 36.4%, traffic sign at 48.3% — which is an inherent resolution limitation of the stride-16 patch grid rather than a deficiency in the learned representation. The frozen backbone produces features that transfer to an entirely unseen visual domain with a minimal linear decoder, which is the property that makes a universal perception encoder worth having.

Comparison with Standard Baselines

As a sanity check, Argus was compared against several well-known models on the same 200-image COCO subset. The classification comparison uses a keyword cross-reference between each model's top-k ImageNet predictions and the COCO ground-truth detection labels on those images, which provides a consistent yardstick across differently-trained models despite the label-space mismatch. These hit rates measure agreement with COCO detection labels via keyword matching on the 200-image subset; they are not raw ImageNet accuracy. For reference, all three classifiers exceed 80% top-1 on the full ImageNet validation set.

Classification (hit rate against COCO detection labels, 200 images):

Model	Parameters	Top-1 hit	Top-5 hit	Latency	Peak VRAM
Argus (EUPE-ViT-B)	86 M	42.2%	66.8%	13.1 ms	0.34 GB
ConvNeXt-Base	89 M	40.2%	71.4%	10.4 ms	0.35 GB
ResNet50	26 M	36.2%	61.8%	8.4 ms	0.12 GB

Segmentation:

Model	Parameters	Classes	Latency	Peak VRAM
Argus (EUPE + linear head)	86 M	150	11.8 ms	0.41 GB
DeepLabV3-ResNet50	42 M	21	15.9 ms	0.33 GB

Depth:

Model	Parameters	Latency	Peak VRAM
Argus (EUPE + linear head)	86 M	13.3 ms	0.35 GB
Depth-Anything-V2-Base	98 M	18.8 ms	0.68 GB

Argus produces the top-1 classification accuracy of the three image classifiers, with ConvNeXt-Base edging it slightly on top-5. The Argus row above uses the kNN classification method, which is decisive on top-1 but flatter on top-k than a trained softmax. Argus is faster than DeepLabV3 while predicting a much richer label space, and it is faster than Depth-Anything-V2 while using roughly half the VRAM. Although these baselines and Argus were trained for different objectives on different datasets, the comparison is useful for understanding what the model delivers in practice.

Multi-Task Throughput

The per-task comparisons above measure each head against its single-task counterpart in isolation. A separate question is what happens when a user needs all of the tasks at once, which is the typical situation in dataset annotation, model evaluation, and any pipeline where images pass through multiple analysis stages in sequence. The alternative to Argus in that situation is to load and run four separate single-task models of comparable quality, each carrying its own backbone, its own preprocessing, and its own forward pass. The total cost is the sum of the four individual inference times, plus the memory overhead of holding four independent models on the device simultaneously.

The models chosen for this comparison were selected to match the quality tier of the EUPE-ViT-B backbone rather than to minimize size or maximize speed. ConvNeXt-Base (88.6M parameters) is a widely-used ImageNet-1k classifier at the same parameter scale as EUPE-ViT-B. SegFormer-B3 (47.3M) is a transformer-based ADE20K semantic segmenter that is the standard mid-range alternative to a linear probe on a frozen backbone. Depth-Anything-V2-Base (97.5M) is the current standard for single-image monocular depth estimation at base scale. YOLO26l (26.3M) is the large variant of the January 2026 YOLO release from Ultralytics, representing the state of the art in efficient real-time detection. All measurements were taken on the same RTX 6000 Ada GPU across the same nine example images, with five timed runs after a three-image warmup pass to eliminate cold-start effects.

Pipeline	Parameters	Latency per image	Tasks
Argus unified	116 M	56 ms	5 (classify, segment, depth, detect, correspond)
Four separate models	260 M	68 ms	4 (classify, segment, depth, detect)

The per-model breakdown for the separate pipeline is ConvNeXt-Base at 6 ms, SegFormer-B3 at 19 ms, Depth-Anything-V2-Base at 31 ms, and YOLO26l at 12 ms, summing to 68 ms when the tasks are run sequentially on the same image. Argus completes five tasks — the same four plus keypoint correspondence, which the separate pipeline does not attempt — in 56 ms from a single model load. The total parameter count for the separate pipeline is 260M across four independent weight sets, while Argus carries 116M in a single file.

The throughput advantage comes from the shared backbone. Each of the four separate models pays the cost of encoding the image through its own network before producing task-specific output. Argus encodes the image once through EUPE-ViT-B and then routes the resulting features to five lightweight heads, each of which adds only a few milliseconds on top of the shared representation. The backbone forward pass is the dominant cost in both pipelines, and running it once rather than four times is where the 1.2x throughput improvement and 2.2x parameter reduction originate. The practical consequence for deployment is that Argus requires a single model download (447 MB), a single checkpoint load into VRAM (0.53 GB), and a single Python import, where the equivalent separate-model pipeline requires four downloads totaling over a gigabyte, four loads consuming over a gigabyte of VRAM if held concurrently, and four separate dependency trees to manage.

Usage

from PIL import Image
from transformers import AutoModel

model = AutoModel.from_pretrained("phanerozoic/argus", trust_remote_code=True)

image = Image.open("your_image.jpg").convert("RGB")

# Any single task can be called directly:
top5   = model.classify(image, top_k=5)                  # default: kNN method
top5   = model.classify(image, top_k=5, method="softmax") # alternative: trained linear
seg    = model.segment(image)                            # returns [H, W] class indices
depth  = model.depth(image)                              # returns [H, W] metric depth in meters
depth  = model.depth(image, decoder="dpt")               # DPT multi-scale decoder (8% better RMSE)

# Detection runs at 640px with letterbox padding:
dets   = model.detect(image, score_thresh=0.3)
# returns list of {"box": [x1,y1,x2,y2], "score": float, "label": int, "class_name": str}

# Classification, segmentation, and depth can be run at once:
result = model.perceive(image)
# result["classification"] — list of top-5 {"class_id", "class_name", "score"}
# result["segmentation"]   — numpy array of ADE20K class indices
# result["depth"]          — numpy array of depth values in meters
# result["timings_ms"]     — per-task latency breakdown

# Detection is called separately because it uses a different input resolution:
dets = model.detect(image)

# Keypoint correspondence requires two images and a set of source points:
target = Image.open("other_image.jpg").convert("RGB")
src_points = [[100, 100], [200, 200]]
predicted_target_points = model.correspond(image, target, src_points)

Every single-image method also accepts a list of images. When a list is passed, the return type becomes a list of per-image results in the same shape that a single call would produce:

images = [Image.open(p).convert("RGB") for p in paths]

top5_batch = model.classify(images, top_k=5)           # list of list-of-dict
seg_batch  = model.segment(images)                     # list of [H, W] tensors
depth_batch = model.depth(images)                      # list of [H, W] tensors
perceive_batch = model.perceive(images)                # list of dicts

Per-task confidence and uncertainty are available as opt-in outputs. Classification always carries a margin field (top-1 score minus top-2 score) on the first entry. Segmentation and depth expose confidence maps when return_confidence=True is passed:

seg_map, seg_conf   = model.segment(image, return_confidence=True)
# seg_conf is per-pixel max softmax probability in [0, 1]

depth_map, depth_std = model.depth(image, return_confidence=True)
# depth_std is per-pixel standard deviation of the 256-bin distribution

result = model.perceive(image, return_confidence=True)
# result["segmentation_confidence"] and result["depth_uncertainty"] are populated

The model can be exported to ONNX. This produces three separate graphs — backbone, segmentation head, and depth head — with verification against the PyTorch reference automatically performed when verify=True:

paths = model.export_onnx("/path/to/out_dir", backbone_resolution=224, verify=True)
# paths["backbone"], paths["seg_head"], paths["depth_head"]
# paths["verification"] — max abs diff per component

Classification (kNN over class prototypes) and correspondence run as post-processing on top of the backbone output and need no separate graph.

For reduced VRAM on memory-constrained hardware, INT8 weight-only quantization is available via torchao. This quantizes the Linear weight matrices to INT8 while keeping activations in BF16, avoiding the outlier-channel problems that break naive INT8 quantization of ViT models:

model = AutoModel.from_pretrained("phanerozoic/argus", trust_remote_code=True)
model = model.cuda().eval().quantize_int8()  # requires: pip install torchao

# All methods work identically after quantization:
result = model.perceive(image)       # 100% classification agreement, <0.05 m depth drift
dets   = model.detect(image)         # detection counts identical to FP32

Mode	perceive latency	VRAM	Classification	Depth drift
BF16 mixed (default)	35 ms	0.53 GB	reference	—
INT8 weight-only	39 ms	0.47 GB	100% agreement	0.013 m mean

The model uses HuggingFace's custom-code mechanism (trust_remote_code=True), so the loader code is fetched from the model repo automatically. No additional files need to be cloned.

Training

The backbone is frozen for every task. Only the task heads are trained, and the class prototypes are extracted (not trained at all).

Heads

Component	Source dataset	Trained by
EUPE-ViT-B backbone	LVD-1689M (approximately 1.7 billion web images)	Meta FAIR (used here frozen)
Segmentation head	ADE20K (20,210 training images, 2,000 validation images)	This repository, 40,000 iterations of linear-probe training
Depth head	NYU Depth V2 (24,231 training images)	This repository, 38,400 iterations of linear-probe training
Class prototypes (kNN)	ImageNet-1k (1.28 million training images)	This repository, mean CLS feature per class
Linear softmax classifier	ImageNet-1k (1.28 million training images)	This repository, SGD over cached frozen features
Detection head	COCO 2017 (117,266 training images, 80 classes)	This repository, FCOS with simple FPN, 8 epochs at 640px
DPT depth decoder	NYU Depth V2 (24,231 training images)	This repository, DPT multi-scale, 38,400 iterations
Correspondence	None (training-free)	—

The trainable heads sum to approximately 30.7M parameters (seg 117K + depth linear 201K + linear classifier 769K + detection 16.14M + DPT decoder 13.45M). The unified model.safetensors is 447 MB.

Precision variants

Two safetensors files with the same weights at different on-disk precision. Inference behavior is identical; the smaller file is for users with limited bandwidth or storage.

File	Size	Load
model.safetensors	334 MB	AutoModel.from_pretrained("phanerozoic/argus", trust_remote_code=True)
model.bf16_backbone.safetensors	170 MB	AutoModel.from_pretrained("phanerozoic/argus", trust_remote_code=True, variant="bf16_backbone")

Both files load into the same FP32 model in memory; PyTorch automatically upcasts the bfloat16 stored weights at construction time. The smaller variant saves download bandwidth and disk space but does not reduce inference VRAM.

Architecture details

Segmentation head is BatchNorm2d(768) → Conv2d(768, 150, 1×1) — 116,886 parameters, 1.4 MB on disk. Trained at 512×512 with cross-entropy loss, AdamW (lr 1e-3, weight decay 1e-3), WarmupOneCycleLR with 1500-step warmup, batch size 16.

Depth head is BatchNorm2d(768) → Conv2d(768, 256, 1×1), with the 256 output channels treated as linear depth bins between 0.001 m and 10 m and combined into a metric prediction by weighted sum — 200,961 parameters, 2.3 MB on disk. Trained on 416×544 crops with SigLoss, AdamW (lr 3e-4, weight decay 1e-3), WarmupOneCycleLR with 12,800-step warmup, batch size 16.

Class prototypes (kNN path) are produced by running the frozen backbone over the full ImageNet-1k training set at 224×224 resolution, computing the mean L2-normalized CLS feature per class, and saving the resulting 1000×768 matrix. No training, just feature extraction. At inference, the kNN path normalizes the query CLS token and computes cosine similarity against the prototype matrix.

Linear softmax classifier is a single Linear(768, 1000) layer with bias — 769,000 parameters, about 3 MB on disk. Trained as a two-pass job: first the frozen backbone is run over the ImageNet-1k training set to cache a per-image CLS feature tensor (1,281,167 × 768, stored once at ~3.9 GB), then the linear layer is trained on the cached features alone. The training pass uses SGD with momentum 0.9, weight decay 0, batch size 4096, cosine schedule, 100 epochs, no augmentation, and the best checkpoint by validation top-1 is restored at the end. A small learning-rate sweep over {0.5, 1.0, 3.0, 10.0, 30.0} selects the best configuration; the L2-normalized CLS features and zero-initialized weights demand an unusually large learning rate to grow the weight scale to the point where softmax distributions become sharp. The best run used lr = 30.0 and produced 85.53% top-1 / 97.69% top-5 on ImageNet-1k val, beating the kNN protocol on both metrics.

Detection head is an FCOS-style anchor-free detector on a ViTDet-style simple feature pyramid. The FPN takes the backbone's stride-16 spatial features and synthesizes five levels: P3 (stride 8, via transposed convolution), P4 (stride 16, identity with channel reduction), P5 (stride 32), P6 (stride 64), and P7 (stride 128), each with 256 channels and GroupNorm normalization. Two shared four-layer convolutional towers (classification and regression) with GroupNorm and GELU process each level, followed by three prediction heads: 80 classification channels, 4 box regression channels (left/top/right/bottom distances, exponentiated with learned per-level scale), and 1 centerness channel. 16,138,074 parameters total, 61.6 MB on disk. Trained at 640×640 with letterbox padding, focal loss (alpha 0.25, gamma 2.0) for classification, GIoU loss for boxes, BCE for centerness, AdamW (lr 1e-3, weight decay 1e-4), cosine schedule with 3% warmup, batch size 64, 8 epochs.

DPT depth decoder hooks into four evenly-spaced ViT blocks (2, 5, 8, 11) via PyTorch forward hooks, capturing intermediate representations without modifying the backbone. A reassemble stage projects each block's output from 768 to 256 channels via LayerNorm + Linear, reshapes to spatial grids, and rescales to four target strides (4, 8, 16, 32) via bilinear interpolation. A bottom-up fusion path combines these four scales through residual conv blocks with skip connections, progressively doubling spatial resolution from stride 32 to stride 2. A final conv head produces 256 depth-bin logits using the same bin-weighted-sum scheme as the linear probe. 13,450,000 parameters total, 51 MB on disk. Trained at 416×416 with SILog loss, AdamW (lr 1e-4, weight decay 1e-3), cosine schedule with 3% warmup, batch size 16, 38,400 iterations.

Correspondence has no learned parameters. At inference time, dense patch features are extracted from both images, upsampled to 512×512 pixel resolution, and matched by cosine similarity per source keypoint.

Compute

Task	Iterations	Wall time
Segmentation (ADE20K)	40,000	~5 hours
Depth (NYU Depth V2)	38,400	~3 hours
Class prototypes (IN1k)	1.28M images, single pass	~45 minutes
Linear classifier (IN1k)	100 epochs × 313 steps	~25 seconds (on cached features, extraction amortized with the kNN prototype pass)
Detection (COCO 2017)	8 epochs × 1,832 batches	~6 hours at batch 64, 640px, FP32, frozen backbone
DPT depth decoder (NYUv2)	38,400 iterations	~5.3 hours at batch 16, 416px, SILog loss, frozen backbone
Correspondence (SPair)	training-free	—

Training was done on a single 48 GB workstation GPU. Peak VRAM was approximately 10 GB during segmentation training, 7 GB during depth training, 2.5 GB during class prototype extraction, and 4 GB during the linear classifier training (once features are cached, the training loop only holds the 3.7 GB cached feature tensor on GPU).

Why minimal heads

The decision to use BatchNorm + 1×1 convolution for segmentation and depth, and a single linear layer for classification, is the same one the EUPE paper makes. A minimal head means downstream performance can be attributed to the backbone's features rather than to a sophisticated decoder. The same backbone with a Mask2Former-style head would produce higher segmentation numbers, but those numbers wouldn't tell you anything about the backbone in isolation. The linear softmax classifier added here follows the same principle: one fully-connected layer on top of the frozen CLS token, no intermediate hidden layers, no augmentation at training time.

Notes

The segmentation head was trained on ADE20K's 150-class indoor-and-urban label space. The depth head was trained on NYU Depth v2 and is indoor-biased; outdoor metric depth should be treated as approximate. The detection head was trained on COCO 2017's 80-class label space at 640-pixel input; small-object detection (mAP 21.4) is the expected weakness because the stride-8 P3 level can only resolve objects roughly 12 pixels and larger at that resolution. Classification supports both a kNN protocol (decisive on top-1, flat on top-k because nearest-mean distances compress on visually similar classes) and a trained linear softmax (sharper top-k, calibrated probabilities); the caller picks between them per call via the method argument.

License

The EUPE-ViT-B backbone weights inside this checkpoint were released by Meta FAIR under the FAIR Research License, which restricts use to non-commercial research and education. The task heads and class prototypes in this checkpoint were trained independently by the author of this repository and would on their own be releasable under a permissive license. However, because they are inseparably bundled with the backbone weights in a single file, the unified checkpoint inherits the more restrictive license of its most restricted component. In practical terms, the entire argus.pt file should be treated as released under the FAIR Research License. See LICENSE for the full text.

Citation

If you use Argus or the underlying EUPE backbone in academic work, please cite the original paper:

@misc{zhu2026eupe,
  title={Efficient Universal Perception Encoder},
  author={Zhu, Chenchen and Suri, Saksham and Jose, Cijo and Oquab, Maxime and Szafraniec, Marc and Wen, Wei and Xiong, Yunyang and Labatut, Patrick and Bojanowski, Piotr and Krishnamoorthi, Raghuraman and Chandra, Vikas},
  year={2026},
  eprint={2603.22387},
  archivePrefix={arXiv},
  primaryClass={cs.CV}
}

Acknowledgements

The EUPE backbone was trained and released by Meta FAIR. The dataset loading utilities are from the DINOv3 repository. The Argus task heads, benchmarks, and packaging were done by phanerozoic.