Robot AI might be developing a pretraining evaluation…
0 reactions
Robot AI might be developing a pretraining evaluation culture. An open-source release just shipped real-hardware zero-shot benchmarks
Most robot videos are difficult to read because you never know how much of the behavior is the foundation model and how much is task-specific tuning, teleop cleanup, or a very carefully selected rollout. This one caught my attention for a different reason: the accompanying report tries to measure what a pretrained robot model can do before task-specific fine tuning.
The model is Wall-OSS-0.5, an open-source vision-language-action model from X Square Robot. In the attached reel, the robot follows language instructions for tabletop tasks like opening a pot lid and putting fruit inside, covering blocks with cloth, stacking a ring on a peg, sorting blocks by color, putting multiple objects into specified containers in sequence, placing a cup relative to a calculator, and feeding paper into a shredder. The video itself has an "Autonomous w/o Fine-Tuning" watermark, so the obvious question is whether this is actually measuring pretrained behavior rather than a tuned demo policy.
According to the report, they evaluate the pretrained checkpoint on a 17-task real-robot suite before task-specific fine tuning. Four tasks pass 80 task progress, including block sorting, fruit sorting, ring stacking, and a held-out deformable task, rope tightening. That does not mean general-purpose home robots are suddenly solved. The same report says harder tasks like towel folding, table setting and charger insertion are still very low in zero shot. But the fact that a release is centering pre-finetune real-hardware evaluation feels like a healthier benchmark direction than another polished humanoid clip.
Source material if anyone wants to check the claim instead of just the video: paper https://x2robot.com/api/files/file/wall\_oss\_05.pdf https://x2robot.com/api/files/file/wall_oss_05.pdf , code https://github.com/X-Square-Robot/wall-x , project page https://x2robot.com/oss#resources , Hugging Face org https://huggingface.co/x-square-robot .
The future-facing question is less "does this robot do all these tasks perfectly?" and more "are robot foundation models starting to get the kind of pretraining evaluation culture that language models already have?" If third-party labs can reproduce the same strengths and failures on different hardware, that would be a much more meaningful signal than any single demo reel.
#technology
source
Most robot videos are difficult to read because you never know how much of the behavior is the foundation model and how much is task-specific tuning, teleop cleanup, or a very carefully selected rollout. This one caught my attention for a different reason: the accompanying report tries to measure what a pretrained robot model can do before task-specific fine tuning.
The model is Wall-OSS-0.5, an open-source vision-language-action model from X Square Robot. In the attached reel, the robot follows language instructions for tabletop tasks like opening a pot lid and putting fruit inside, covering blocks with cloth, stacking a ring on a peg, sorting blocks by color, putting multiple objects into specified containers in sequence, placing a cup relative to a calculator, and feeding paper into a shredder. The video itself has an "Autonomous w/o Fine-Tuning" watermark, so the obvious question is whether this is actually measuring pretrained behavior rather than a tuned demo policy.
According to the report, they evaluate the pretrained checkpoint on a 17-task real-robot suite before task-specific fine tuning. Four tasks pass 80 task progress, including block sorting, fruit sorting, ring stacking, and a held-out deformable task, rope tightening. That does not mean general-purpose home robots are suddenly solved. The same report says harder tasks like towel folding, table setting and charger insertion are still very low in zero shot. But the fact that a release is centering pre-finetune real-hardware evaluation feels like a healthier benchmark direction than another polished humanoid clip.
Source material if anyone wants to check the claim instead of just the video: paper https://x2robot.com/api/files/file/wall\_oss\_05.pdf https://x2robot.com/api/files/file/wall_oss_05.pdf , code https://github.com/X-Square-Robot/wall-x , project page https://x2robot.com/oss#resources , Hugging Face org https://huggingface.co/x-square-robot .
The future-facing question is less "does this robot do all these tasks perfectly?" and more "are robot foundation models starting to get the kind of pretraining evaluation culture that language models already have?" If third-party labs can reproduce the same strengths and failures on different hardware, that would be a much more meaningful signal than any single demo reel.
#technology
source
| 1,000 mlx | |
| 2 | |
| 0 |