Simulation
Run a deployed policy on LIBERO’s four suites, RoboCasa365 and RoboTwin 2.0, or on a simulation you write, its environments stepping together on one GPU. Every eval reports each metric with its 95% interval.
How it works
A benchmark is a class you write, like a policy, or one of the templates in the benchmark library. An eval runs a deployed policy on it, on machines of its own, and leaves a report, recordings and replays to come back to.
$ seq init libero-spatial my-libero-spatial
$ seq deploy my-libero-spatial/benchmark.py
$ seq deploy my-libero-spatial/pi05_libero_adapter.py
$ seq eval run my-pi05-libero --benchmark libero-spatial \
--adapter libero-openpi --envs 64
$ seq eval report <eval_id>A GPU benchmark steps its environments together on one card, and the policy answers them in batches. LIBERO-Spatial runs 64 on an L4 by default and up to 256, which step in about 79 ms; more instances run side by side. A CPU simulator runs as it is, one environment per core.
LIBERO and RoboCasa record where every object is at each step, so an eval’s page replays any episode in 3D: turn it, frame it from the front, side or top, and step through it at 0.25× to 4×, beside the camera recording where one was made.
When a benchmark’s observations and actions are not the ones a policy takes, an adapter maps them. It is registered on its own, versioned and named by the eval, so every report reads benchmark vN + adapter vN + policy vN.
Every metric comes overall and per task, with n and its 95% interval. Run another policy on the same episodes and seeds with --compare for the paired difference, or schedule an eval after every deploy and read the trend.
Starter costs $0 a month and comes with $20 of free credit every month. See what an eval will cost before you start it, and give it a spend cap it stops at.