Customize

A preset or your own

A policy says what it takes and what it returns in one of two ways.

• A preset. Declare a contract — action=seq.ActionContract(...) and observation=seq.ObservationContract(...) — with one of the five action spaces below. A preset is a ready-made template for a robot arm and nothing more: no policy is treated differently for using one. What a contract gives you is checks the platform runs for you: the worker refuses an observation the policy cannot read — a missing camera, a frame of another size, a state vector of the wrong length — with a 400 before any inference runs, and refuses a chunk that is not the declared action space or dimension (The I/O contract).
• Your own. Declare no contract. infer gets the dict the caller sent, unchanged, and what it returns goes back as it is; the platform converts nothing and checks nothing. When your code raises, the caller gets your error with your traceback (Your own inputs and outputs). Anything a preset does not describe — another body, outputs that are not joint or end-effector rows, inputs of your own making — takes this way. run() plays such a policy too: tell it how to read the steps out of an answer (actions_of) and at what rate (control_hz) — Your policy’s own inputs.

An eval keeps the same split. A preset policy is sent the benchmark’s observation as the preset’s — its cameras and its state, with the task’s instruction — built by an adapter when the benchmark’s does not already fit. A policy with its own inputs is sent exactly what its adapter produces, or, with no adapter, the benchmark’s observation as it is. The first act shows whether the two fit. When they do not, the eval stops there with stop reason mismatch, billed only for its machines’ start, and its report says which side did not fit in one of these sentences, followed by the error itself:

Action spaces

A preset’s chunk is rows of numbers, one row a control step, and its action space says what a row means. The same numbers mean opposite things in different spaces — an absolute joint position played as a delta throws the arm into its joint limits — and nothing in a row’s shape tells them apart, so every chunk names its space, and connect(model, expect={"action_space": ...}) checks it before the first action.

An absolute row is a target: played twice, the arm stays where it is. A delta row is a step from where the arm is: played twice, the arm moves twice as far. A velocity row is a speed, held for one control step. A space only names what a row means; what its numbers are in, and what a robot does with them, is the template’s own — pi05-droid’s rows, for one, are 7 joint velocities and the gripper’s position at 15 Hz (What the rows mean), and its policy.py and README say so. No preset converts one space into another.

Which robot state a policy reads

A preset’s observation is its camera frames, its instruction and the robot’s state, in two places:

• proprioception: joint_positions (radians), joint_velocities, end_effector_pose (x, y, z and a quaternion qx, qy, qz, qw: 7 values) and gripper (a list: a robot with two grippers sends two).
• state: named vectors that are not joints — an end effector’s position and axis-angle, a mobile base’s pose, an object’s pose — so none has to pass as joint_positions.

Every one of them is optional. A policy names the ones it reads, each with its length, and those are the ones its worker requires; one it does not name is not required and is not handed to it. The official pi05-libero reads its gripper and its end effector, and no joints:

observation=seq.ObservationContract(
    cameras={...},
    proprio={"gripper": 2},           # the proprioception it reads, each with its length
    state={"eef": 6},                # the end effector's position and axis-angle
)

# on the robot
sequence_ai.observation(images=frames, instruction="put the bowl on the plate",
                        gripper=[f1, f2], state={"eef": [x, y, z, ax, ay, az]})

A field it names that is missing, or of another length, is refused before the policy runs — on the robot before it is sent, and by the worker with 400 invalid_request, its details.problems naming each one:

In an eval, a benchmark that does not give what the policy names is said before it starts — my-policy needs proprioception.joint_positions of 7 values (it has no key of that name), which my-bench does not give: run it with an adapter that maps them (--adapter <name>) — and an adapter can give it (Adapters). An eval started anyway stops at its first act, its worker refusing that observation.

Smoothing between chunks

Each chunk is computed from its own observation, so its first row can sit away from the last row the arm played: that step across the boundary is the seam. run() measures it every time — RunOutcome.max_seam_jump, the largest step in any one dimension across a boundary — and changes nothing unless you ask. run(..., smooth_seam=N) blends the first N rows of each new chunk from the last action played toward the chunk’s own, in a straight line, so the arm does not jump.

• Absolute spaces only: joint_absolute and ee_absolute. A joint_delta, ee_delta or joint_velocity chunk passes untouched: blending increments would rescale them and move the arm where the model never asked.
• Never the first chunk of a run: there is no last action to blend from, and where the arm starts is yours to decide.
• max_seam_jump is measured before any blending, so smoothing never hides what it smooths.
• A policy’s own steps are not smoothed: nothing says what they are. When they are absolute targets, say so — run(..., smooth_seam=N, absolute=True) (Your policy’s own inputs).

Chunks and control rate

One act returns a chunk: a block of future rows, so the robot needs no round trip per step. Its fields say how to play it:

run() plays the rows at control_frequency_hz on a fixed schedule — a late row is never made up by firing the next ones in a burst — and asks for the next chunk while the current one plays. A policy with its own inputs declares no rate, so its run names one: control_hz=10, or read from each answer. A chunk that arrives after the last row has played is an underrun: the arm holds (on_underrun) until it comes. Action chunks and Chunks are deadlines say how to keep the next chunk in time.

How much to send each act

Every act uploads its observation before the policy can start, so what you send is part of every chunk’s wait. A larger body takes more round trips to upload and arrives later, and past covers_seconds the arm runs out of rows and holds. Send what the policy reads, and nothing more:

• Frames at the policy’s size. A preset policy declares each camera’s width and height; send frames at that size, not at the camera’s own. policy.prepare_image(view, frame) fits a frame the way the policy was trained to see it (Fit each frame on the robot), and the worker refuses a frame of another size anyway.
• Your own inputs: only what infer reads. Nothing knows which of your keys the policy uses, so a dict carrying more is uploaded whole on every act.

The robot’s SDK says so once when an act is much bigger than that. For a policy that declares its cameras, “much bigger” is past twice what they read: one raw frame a camera (width × height × 3 bytes, × 2 more for a depth frame) times its frames a view (history) — frames sent as JPEG stay well under it. For a policy with its own inputs it is past 1 MB. The warning is a Python UserWarning at your own line, once a session (one connect), from the size of its first act and of every 100th after; the act goes as it is either way:

my_robot.py:42: UserWarning: my-pi05: an act sent 2.4 MB; the policy reads 2 cameras at 224x224, 294 KB at most
my_robot.py:42: UserWarning: my-toy: an act sent 3.0 MB, past 1.0 MB      # a policy with its own inputs

The worker writes what it received on every act’s line in the policy’s log, recv_bytes=, in an eval as when it serves a robot (seq policy logs, or the Logs tab on its page).