xlerobot-coffee-model-sim-a-vision-pos

Method A — vision + position, trained on simulation data. ACT experts for the Coffee Automata chain on the 17-DoF XLeRobot.

One of four repos on the domain × method grid; the companion is …-model-sim-b-force.

Contents

Each folder is a complete pretrained_model directory (final 100k checkpoint at the folder root) with a <folder>/checkpoints/<NNNNNN>/ step sweep alongside. Read the folder suffix before loading — it encodes both the action space and the camera count, and those are the variables these models exist to compare.

folder	action	cameras	run	step sweep
g135_shared_t1_t3_t5_17dof	17-D	1 camera — head	sim_g135_A	4
g135_shared_t1_t3_t5_6dof	6-D	1 camera — head	sim6_g135_A	4
g35_shared_t3_t5_17dof	17-D	1 camera — head	sim_g35_A	4
g35_shared_t3_t5_6dof	6-D	1 camera — head	sim6_g35_A	4
t1_place_cup_17dof	17-D	1 camera — head	sim_t1_A	4
t1_place_cup_6dof	6-D	1 camera — head	sim6_t1_A	4
t2_push_button_17dof	17-D	1 camera — head	sim_t2_A	4
t2_push_button_6dof	6-D	1 camera — head	sim6_t2_A	4
t3_cup_to_tray_17dof	17-D	1 camera — head	sim_t3_A	4
t3_cup_to_tray_6dof	6-D	1 camera — head	sim6_t3_A	4
t5_tray_to_table_17dof	17-D	1 camera — head	sim_t5_A	4
t5_tray_to_table_6dof	6-D	1 camera — head	sim6_t5_A	4

t4_navigate has no rung — there is no sim-xlerobot/t4_navigate recording.

The two action spaces

*_17dof predicts the whole 17-D action; *_6dof predicts only the arm that performs the task. The pair exists to test whether ACT's loss — a mean over all 17 dims, of which only ~6–9 move — dilutes the signal on the dims that matter.

Measured answer: it does not, to any reliable degree. Over 22 matched pairs (12 sim, 10 real 2-cam), scoring both models on the same 6 acting-arm joints:

set	n	mean Δ	95% CI
sim, 1 cam	12	+4.2%	[−5.1%, +13.5%]
real, 2 cam	10	+9.4%	[−5.2%, +23.9%]
combined	22	+6.5%	[−1.7%, +14.7%]

Positive = 17-DoF lower error. Every interval contains zero and 6-DoF wins 11 of 22, so the two are indistinguishable on accuracy. Prefer *_17dof for deployment — it matches the platform, needs no per-task arm choice, and keeps bimanual behaviour reachable (real t2_push_button genuinely uses both arms) — not because it predicts better.

⚠ Do not compare the two by training loss. ACT's loss is a mean over action dims, so the 17-D and 6-D numbers are different quantities; the 17-D mean is dragged down by frozen channels and appears better for that reason alone.

t2_push_button is a RIGHT-arm task in simulation, not left — the acting arm was chosen per task by measured action variation, not assumed.

Architecture

• 17-DoF: Vanilla ACT · action[17] (12 arm .pos + 2 head .pos + base x/y/theta .vel), observation.state[17], head RGB. No HPI. Loads on stock LeRobot.
• 6-DoF: Vanilla ACT · action[6] — the single arm that performs the task (RIGHT for t2_push_button, left elsewhere), observation.state[6], head RGB.

Recipe

ACT · chunk_size 100 · n_action_steps 100 · batch 8 · 100k steps · seed 1000 — identical for every model here, so domain / method / action-dim / camera-count are each read in isolation. hpi_fps follows the DATA rate (30 for real and 2-cam, 20 for the 20 fps sim recordings); a mismatch puts the HPI window off the frame grid and the dataset rejects it. Trained on idlab_server1 (RTX PRO 6000 Blackwell) for the 1-camera models and idlab2 (RTX 5090) for the 2-camera ladder.

Normalization statistics are floored per the xlerobot_norm_floor rule (1e-2 state/action, 1e-5 HPI) — the fix for the degenerate-std defect in docs/coffee/EVAL_POSTMORTEM.md.

Loading

PreTrainedPolicy.from_pretrained has no subfolder support, so load through the project helper, or snapshot the folder you want:

from huggingface_hub import snapshot_download
from lerobot.policies.act.modeling_act import ACTPolicy

root = snapshot_download("IntelligentDecisionLab/xlerobot-coffee-model-sim-a-vision-pos", allow_patterns="t1_place_cup_17dof/*")
policy = ACTPolicy.from_pretrained(f"{root}/t1_place_cup_17dof")

Method A loads on stock LeRobot.

Caveats

• These are offline-trained policies with no on-robot success numbers yet. The DoF comparison above is action-prediction error, not task success; closed-loop compounding error can rank models differently.
• 1-cam vs 2-cam is not a controlled comparison. The 2-cam dataset has 5–6× the episodes of the 1-cam real set, so camera count and dataset size are confounded.
• No 2-camera simulation data exists, so sim→real cannot be read at two cameras.
• 100k steps on ~50 episodes is ~90–110 epochs and likely overfit; the step sweep is provided so evaluation can pick a checkpoint rather than assume the last one is best.

Part of the X-Lerobot Coffee Automata project. AS-CITI Intelligent Decision Lab.