Manipulation Benchmarks Researchers Actually Use

How the field actually measures whether one manipulation approach is better than another.

Manipulation benchmarks are standardized tasks and datasets that let different research groups compare their approaches on equal footing, rather than each group demonstrating its own hand-picked, favorable examples.

What a good benchmark measures

Success rate across a large number of trials matters more than any single successful attempt. Good benchmarks also test generalization deliberately — evaluating a system on object shapes, lighting conditions, or task variations it wasn't specifically trained on, rather than only the exact conditions it saw during development.

Why benchmarks matter for reading company claims

A company reporting results on a recognized, standardized benchmark is making a more verifiable and comparable claim than one showcasing an unbenchmarked demo video — a useful distinction to keep in mind when comparing manipulation claims across different companies and research groups.