Python client

The control loop, in Python

sequence_ai is the client behind Running a policy. connect() returns a handle whose first call opens a kept-alive line straight to the worker; observe() and act() pass what your robot sends straight to the policy and bring back what it returned, and they raise typed errors. Your own loop plays the actions (your own loop). This is the reference for those calls. In the examples, everything ending in _user is yours.

Install

One install for everyone. On a robot (Python 3.9) it is the client and two dependencies, httpx and Pillow, which fits and encodes each camera frame on the robot; on Python 3.10+ it also brings seq, to deploy and evaluate your own policy, with pydantic and the shared contract. This page is the client.

installshell
pip install -U sequence-ai                   # the client: connect / observe / act, and Pillow for the frames

Install sequence-ai, import sequence_ai.

Authentication

One key identifies your account. The client reads it from api_key= first, then $SEQUENCES_API_KEY, and talks to api.generalsequences.com — no host to configure.

set a keyshell
export SEQUENCES_API_KEY="seq_dev_..."

Create one in the console: it starts seq_dev_. A robot connects with its own robot key instead, which starts seq_robot_ and reaches the policies on its list (Robots). sequence_ai.models() needs no key.

Or sign in as yourself: seq login prints a code and opens the console to confirm it, and every command with no $SEQUENCES_API_KEY then runs as you. Making keys, paying and changing the account take you signed in, because a key makes no keys. seq logout signs this machine out. seq login --profile <name> keeps a sign-in under a name of its own, so two people on one machine stay apart; seq profile list shows them and seq profile activate picks one.

Quickstart

A complete control loop. connect() is a context manager, so the connection closes even when the loop raises; observe() hands the policy each observation, and act(), which carries nothing, returns what its infer returned.

control looppython
import time
from concurrent.futures import ThreadPoolExecutor
import sequence_ai

with sequence_ai.connect(model="my_policy_user") as policy_user:
    pool_user = ThreadPoolExecutor(max_workers=1)
    hz_user, valid_user, replan_user = 15, 15, 8   # the policy's hz, action_chunk_valid_len, action_chunk_replan_at
    policy_user.observe(read_obs_user())
    sent_user, ahead_user = time.monotonic(), pool_user.submit(policy_user.act)
    while not done_user():
        rows_user, t0_user, ahead_user = ahead_user.result()[:valid_user], sent_user, None   # what infer returned
        for step_user in range(int((time.monotonic() - t0_user) * hz_user), valid_user):
            if ahead_user is None and step_user >= replan_user:    # replan_user steps since its act went out
                sent_user, ahead_user = time.monotonic(), pool_user.submit(policy_user.act)
            arm_user.move_to(rows_user[step_user])
            policy_user.observe(read_obs_user())
            time.sleep(max(0, t0_user + (step_user + 1) / hz_user - time.monotonic()))
            if ahead_user is not None and ahead_user.done():       # the next one is in: switch to it
                break
        if ahead_user is None:                                 # every row went stale on the way: ask again
            sent_user, ahead_user = time.monotonic(), pool_user.submit(policy_user.act)

When to ask and when each row goes to the motors is your code: this loop asks for the next chunk in a background thread replan_user steps after its act went out (the policy’s action_chunk_replan_at), plays on through the chunk’s valid rows while it comes, and sends each row at its own time point (your own loop). read_obs_user, arm_user and done_user stand for your robot’s own.

An action returned by any model is model output, not a safe robot command. Neither the platform nor this library checks it: no joint limits, no reachability or collision, nothing about whether a step is safe at the robot’s current velocity — your robot gets what the policy returned, as it returned it. Put your own bounds check and a physical e-stop between it and your motors; connect(on_stale=) tells your code the moment the arm runs out of answer (below).

connect()

connect() returns a Policy handle, or, given several names, a tuple of them in that order (Connect a robot). Its first observe(), act() or wait_until_ready() connects, and the robot’s one connection to its worker — a WebSocket (wss) — opens as soon as the platform names the worker and stays open for the life of the loop. A call is two phases; the gateway is in only the first.

"teleop" can be one of the names, for a robot a person drives: a handle with the same observe() and act(), on no worker. connect(hold=…) is what act() hands back while nothing is to be played, and sequence_ai.teleop(key=, operator=) is the person’s side, from your own Python: Teleoperation.

phasereachespurpose
connect(model)the gateway, at the first calladmit, ensure a worker is up, return its worker_url, the cert_fingerprint the client pins, direct_secret and lease
observe(), act()the worker, directlyhand over each observation and run inference on the one WebSocket, in the order you made them, the worker’s certificate checked; Bearer <direct_secret>
reconnectthe gateway, on demandon a rotated secret, expired lease or a worker that scaled to zero — fetch fresh coordinates and replay
keepalivethe gateway, about every 20 s while the Policy is openkeep the robot’s seat, and renew its lease before it runs out — from a thread of its own, which no act waits on

Direct connect and fallback

You do not handle the 401 or 503 yourself. A rotated direct_secret, an expired lease, a worker that scaled to zero, or a dropped connection all surface as one transparent reconnect: the client fetches fresh coordinates and replays the in-flight call — unless its worker is still running it (see how long an act waits). The same Policy handle keeps working.

A WebSocket is the one way a robot reaches its worker. A network that lets none through — a proxy or firewall that answers it with anything but the stream — raises StreamRefused at the first act, saying what answered (an answer of 401 or 403 reads as AuthError, its hint naming the WebSocket, and a 5xx as a worker not reached); wait_until_ready() warns of it as it returns.

A cold worker answers connect() with 503 and a Retry-After while it boots; the client waits that out — see cold starts and warming.

How long an act waits

An act waits for its answer as long as the deployment allows one — the platform’s limit, 300 s unless its version’s check measured slower inferences — and 10 s more, so the worker’s own 504 at that deadline is what you get. Where the platform names none for the deployment, an act waits 30 s. connect(model, timeout_s=…) sets your own bound for every request, acts included.

An act with no answer in time is never sent again while its worker still runs it: that would run it twice. It raises Unavailable (504, details["reason"] == "act_still_running"). A worker that is gone is reconnected around and the act replayed, as above. A worker busy with other robots’ acts (429) is waited out for the same time from its first refusal, in steps no longer than its Retry-After; a wait past 5 seconds says once, on stderr, what it waits for, in the worker’s words (a copy of the policy frees when its episode ends, or when it has had no act for its idle time).

observe() and act()

observe() hands the policy each observation as it is — every input goes this way, an instruction too — and act(), which carries nothing, asks for actions and returns what the policy’s act function returned. When to call each, and how to play what comes back, is your loop (your own loop).

act() returns what the policy’s infer returned, as it returned it — a dict, a list, a number, an array: an array as a read-only numpy array of its dtype and shape where numpy is installed (sequence-ai does not install it; without it, nested lists), bytes as bytes. The instruction is one of the policy’s inputs, in the shape it reads them, so each observe() carries the one in force (changing the instruction).

An agent that calls your loop

An agent of your own can call your loop as its tool: each observation carries the instruction in force, and the loop plays until the sub-goal is done, a deadline, or an interrupt. Wiring instructions together — retries, failure detection, choosing the next sub-goal — is yours. For a plan the platform runs inside the policy, see Planning and External agents.

act()

policy.act() asks for actions and returns what the policy’s act function returned, once every observe() sent before it has reached the policy. It carries no data — act(x) is refused before it goes, saying to hand x to observe() first — and it waits out a starting worker on every call, within startup_timeout_s.

actpython
toy_user = sequence_ai.connect("my_toy_user")  # a deployed toy-policy
# its inputs, as its README gives them
toy_user.observe({"state": [0.0], "target": [0.5]})
toy_user.act()["delta"]                 # [0.1], as infer returned it

act() hands back the answer alone. What befell an act on its way — its episode reset, a move to its standby copy, observations sent again or left out — the SDK says with Python’s warnings.warn; what an act took is on the policy’s page in the console.

A seed. act(seed=7) passes the seed to the policy’s infer(…, seed=7). A policy that samples answers one observation the same way for one seed — the policy draws its flow-matching noise from it (a pi05 template draws openpi’s standard normal noise from the seed and hands it to openpi) — and without a seed draws fresh noise, so two answers to one observation can differ.

episode_id= on observe() or act() puts the call in that episode; without one, a call is the connection’s current episode, which its first call starts and policy.end_episode() ends (Episodes).

Changing the instruction

the instruction, one of the inputspython
instruction_user = "pick up the cup"             # your agent's thread replaces it whenever it decides
while not done_user():
    policy_user.observe({**robot_user.read_observation(), "prompt": instruction_user})
    out_user = policy_user.act()

The sentence is one of the policy’s inputs, in the shape the policy reads it — pi05-droid reads it as "prompt" — so your agent’s thread keeps the latest one and the robot’s loop hands it over with every observation it makes; neither waits for the other. An agent’s call inside your loop holds the arm still for its length; on its own thread, the loop keeps going under the instruction in force.

A new sentence refines or continues, and interrupt() abandons: the first keeps the robot moving, the second stops a call that is waiting, and what the robot does next is your loop’s.

The new instruction is not instant: an act already on its way was asked with what was observed before it, so the first actions that follow the new sentence come with the act after that.

Changing your mind

interrupt — from any thread, at any momentpython
policy_user.interrupt()

Thread-safe. Call it from any thread: an act(), observe() or wait_until_ready() that is waiting — through a cold start or a reconnect too — raises sequence_ai.Interrupted out of its wait, and your loop decides what the robot does next. It is a BaseException, so an except Exception around your own hardware calls does not swallow it.

The flag is cleared as each act() starts, so an interrupt that came between two acts does not cancel the next one.

Labels and episodes

label the episodepython
with sequence_ai.connect(model="my_policy_user") as policy_user:
    policy_user.observe(robot_user.read())
    out_user = policy_user.act()                            # your loop plays it
    policy_user.label(success=robot_user.cup_in_gripper())  # the episode of the last act

What the policy returns is its own and goes to your loop as it was returned: nothing clips or checks it, and holding it to the robot’s limits is your robot’s own code.

A policy’s inference parameters — the keyword arguments its @seq.infer declares — are the same for every robot: their defaults are its settings, seq policy set <name> params.replan_steps_user=3. What differs between robots goes in their observations, policy.observe(x).

label(success, notes="", *, episode_id=None) records how the episode of the last act went, and returns the episode id. With the robot’s Record on, your recorder reads it as that episode’s label (see field data).

end_episode(episode_id=None) ends an episode on its worker now — the connection’s current episode, unless episode_id names another — so a policy that keeps an episode’s state frees it at once and its copy is free for the robot’s next episode. It returns the id it ended, or None when none was open, and never raises. close() ends one left open.

Chunks are deadlines

One call returns a block of future actions: the rows of one answer, the policy’s action_chunk_valid_len of them valid, played at its hz. The robot asks for the next one action_chunk_replan_at steps after it sent this act, or when this one arrives if that is later (a robot has one act on its way at a time), and the next answer has to arrive before the valid rows finish playing; seq deploy --dry-run says whether it would, and what makes room when it would not.

The client reuses one connection — now straight to the worker — for the life of a Policy. seq policy doctor <deployment> checks it from the platform: its state and version, its capacity, its last cold start and a running worker’s /ping; it sends no act.

When the arm runs out of answer

The SDK knows when the answer the robot plays runs out: its act’s send time plus its action_chunk_valid_len steps at hz. Once a robot asks ahead — an act sent while the answer it plays still has valid steps — and that answer runs out with no next one in hand, on_stale is called that moment, from a thread of the SDK’s, and on_resume as the next answer comes, before act() returns it:

def hold_user(stale_user):            # the arm has nothing valid to play
    arm_user.hold()

def resume_user(stale_user):          # an answer again, stale_user.stale_s seconds later
    arm_user.release()

with sequence_ai.connect(model="my_policy_user", on_stale=hold_user, on_resume=resume_user) as policy_user:
    ...

The sequence_ai.Stale each is called with says why in its reason: "late", the act for the next answer is still on its way (its request_id, for seq policy logs --rid); "failed", that act raised and your loop has its error; "not asked", your loop asked for none in time — a loop that hangs mid-run is caught as surely as a late answer. A robot that asks only once its answer ran out waits for every answer and is not watched, nor is an episode end_episode() ended, nor a teleop robot while a person drives it or it is stopped. A stretch that a stop or a person’s taking the robot ends is not resumed then, the arm staying as on_stale left it: on_resume waits for what the arm can play, a person’s chunk or its model’s answer once it drives again. Each stretch is logged on the "sequence_ai" logger and kept on policy.stale (now, events, stale_s, last); with no on_stale set, the first is said once a process.

The platform hears the robot too: from its first observe() or act() on, its SDK sends its worker a heartbeat every 0.4 s saying how the arm stands, and seq robot show says each connection’s — acting (asking ahead), waiting, stale (why, for how long) or silent once it was not heard for 2 s: its process gone, hung, or its link down. A stretch that lasts past a heartbeat is two lines of seq policy logs too, as it began and as it ended, with the late act’s request id. A worker of a version deployed before heartbeats is not reached at all: the robot’s first act raises InvalidRequest, saying to deploy the same code again.

It is no emergency stop: it runs in your robot’s process and needs that process alive. Keep a physical e-stop between the library and your motors.

Where an act’s time goes

One act is the worker’s time plus the network both ways. Measured on a policy deployment on an L4, 102 acts from a robot 87 ms from its worker, the median act:

partmedianwhere you see it
the policy’s inference247 msseq policy logs <name> --all, each act’s line: “act r-… served in 256.0 ms (policy 247.0 · …)” (without --all, each robot’s acts are folded); the same lines are in the Logs tab of the policy’s page in the console
the rest of the worker’s handling: reading the request, decoding the frames, handing them to the policy9 msthe same line
the network, both ways198 msseq policy logs <name> --rid <rid>, the r-… of that act’s line: your robot’s own timing of that act, its round trip and the network in it — for the first act after a connect, the connect, the TCP connect and TLS handshake, and the transfer
the act, as the robot waits for it455 msthe same

The network is the distance and what goes up ahead of the answer. An act carries no data, but the frames observe() handed over just before it go first, on the same stream, and the worker answers the act once they are in. A round trip from that robot took 87 ms, and the act paid two: the connection had sat idle through a chunk’s playback, after which an upload larger than about 14 KB goes in two flights, and two 224×224 camera frames come to about 31 KB. The SDK now keeps that stream from sitting idle, pinging the worker while your robot sends nothing (How long it actually takes). The 232 ms in Batching is the first row alone — the model’s time inside a warm worker, with no network — and not what a robot waits for.

To shorten what the arm waits for:

  • Ask ahead. A loop that asks for the next chunk while the current one plays has the whole act — inference and network — happen while the arm moves, and the arm stops only when an act takes longer than the valid steps after action_chunk_replan_at play; with one act on its way at a time, an answer and the one before it have to come within one chunk together (your own loop).
  • Send less. What an act waits behind is mostly the frames observe() sent just before it: send each camera at the size the policy reads (sequence_ai.resize_image fits it and sequence_ai.encode encodes it; Image size), never larger. An upload that fits in one flight saves the second round trip after an idle pause.
  • Be near the worker. The network both ways is mostly distance. The platform places a deployment’s workers as near its robots as it can; --rid above says what one act’s network took from your robot.
  • Mind your program’s busy threads. Python runs one thread at a time, and each of an act’s threads waits its turn behind a busy one of yours (a loop of pure Python, an arm’s own SDK reading its state). connect() shortens that turn from Python’s 5 ms to 0.5 ms, unless your program set its own (sys.setswitchinterval): beside one busy thread, an act took 190 ms at 5 ms and 96 ms at 0.5 ms, as with none.

Batching serves more robots on a card; it does not shorten one robot’s act.

Cold starts and warming

Warming happens on the first call after connect(), which sends nothing itself, and again whenever a call finds its worker gone — after a connection lost for longer than its idle timeout, or when its copy restarts. A worker that scaled to zero answers connect with 503 and a retry_after_s; in Python that is Unavailable with .warming set. observe() and act() wait it out for you; the try below is what you see with startup_timeout_s=0.

a warming workerpython
try:
    policy_user.act()                    # connect(..., startup_timeout_s=0)
except sequence_ai.Unavailable as exc_user:
    if exc_user.warming:
        print(f"still starting; asking again in {exc_user.retry_after_s:.0f}s")  # when to ask again, not when it is ready

act() waits warming out on every call — by default up to 1800 s from its first ask, as the platform names no deadline for a start — and observe() on its first. startup_timeout_s=0 opts out.

seq policy logs <name> --rid <rid> splits an act’s time into the connect that handed the robot its worker, the TCP connect and TLS handshake to the worker, the transfer and the worker. The rid is on the act’s line in seq policy logs <name> --all, and an error the platform answered says it at its end: deployment 'my_policy_user' is still building; retry when it is ready (hint: its status says when it is ready) [deployment_building, request r-c5b7bb45].

How long it actually takes

A cold start is your deployment’s own: a worker starts with your image, your weights are mounted, and your class is built, its constructor loading the model. For a multi-gigabyte checkpoint that is minutes, and a start may first wait for its card, which can take more than 10 minutes, so a robot waits for a start by default up to 1800 s from its first ask; when wait_until_ready() gives up, it says whether the start was still loading the model or slower than its start usually takes. seq deploy shows each phase as it runs, and seq policy logs <name> shows every worker’s load after it.

Much of a large model’s load can be its own construction: a class that builds the model and then reads its checkpoint over it first fills every parameter with random numbers, which the checkpoint replaces. Built inside transformers’ no_init_weights(), it skips that fill; what the constructor computes for itself, such as rotary frequencies, is unchanged. pi0.5 built through LeRobot loaded in 14.6 s this way, against 170.1 s.

building the model without its random fillpython
try:
    from transformers.initialization import no_init_weights   # newer transformers
except ImportError:
    from transformers.modeling_utils import no_init_weights   # older ones

with no_init_weights():
    self.model_user = MyModel_user.from_pretrained(weights_dir_user)

Pass on_warming to show progress your own way (by default one line goes to stderr; a UserWarning is given as well, which a warnings filter silences). It is called with the seconds waited so far and the seconds until the client asks again; the platform sends no estimate of a cold start:

def warming_user(waited_s_user, again_s_user):
    print(f"starting the model: {waited_s_user:.0f} s so far")

with sequence_ai.connect(model="my_policy_user", on_warming=warming_user) as policy_user:
    ...

A worker stays up while a robot is connected to it, whether or not it acts, with no limit, billed all the while. Once its last robot disconnects it ends after its idle timeout — 60 seconds unless its deployment sets idle_timeout (Scaling) — and the next connection then pays a cold start on a new machine, every time. It can end up to about a minute later, and is billed until it does. A session that pauses keeps its worker and its sub-second latency: one policy act on an L4 takes about 240 ms inside a warm worker, and the round trip at your robot adds the network and the client’s own encoding — seq policy logs <name> shows both for each act; min_workers keeps one running with no robot connected.

You can also check readiness before you call, instead of reacting to a 503. policy.ready() returns (ready, eta_seconds) without blocking. It only asks: it never starts a worker, and while this robot has none it answers (False, None) with the status cold; wait_until_ready() then connects, and the connect starts the robot’s worker in the region nearest it (in line for a seat, it connects again when told to: each connect is its turn). Once a worker serves it also says the card that worker runs on — the deployment’s one card: card and usd_per_hour (what one worker costs an hour on it, as billed). While a worker’s start waits for an available card it is (False, eta). eta counts down to when the start should serve; once the start takes longer than that, it is None and the returned Ready’s eta_unknown is "past_expected", rather than a number the start is already past. A start’s wait for a card can take more than 10 minutes. A robot every place turns away at once waits in line for the card; past 10 minutes in line, ready() raises Unavailable, saying the card and how long it waited. Poll ready() every few seconds, as wait_until_ready() does (every 5 s), or at a queued answer’s retry_after_s (1 s at the head of the line, 5 s behind it): each poll is one request on your key’s requests a minute (600 on Starter, 2,400 on Team), which every robot on the key shares with its connects and keepalives, and a key past it is refused (429) until the minute rolls on. A robot polling every 0.5 s spends 120 a minute. policy.wait_until_ready() blocks until the model can serve — by default up to 1800 s, or your startup_timeout_s (other than 0, which has act() raise at once), or its own timeout_s seconds — and returns the seconds it waited. It returns once the robot’s connection to the worker is open too — its TCP connect, TLS handshake and WebSocket upgrade done — so the first act waits for none of them (one still not open by its connect’s timeout is said in a warning, and the first act opens it). On Linux, a connection that sat idle for longer than its retransmission timeout restarts its congestion window (net.ipv4.tcp_slow_start_after_idle, on by default), so an observation sent after a pause can take one more round trip to upload: the SDK has its connection to the worker use BBR where the robot allows it (BBR does not restart; sudo modprobe tcp_bbr makes it available, and the SDK then takes it on its own), and from the robot’s first observation it also pings the worker on it every 0.1 s that the robot sends nothing, an act’s wait included, for a minute after its last observation or act. Turning the restart off on the robot does the same: sudo sysctl -w net.ipv4.tcp_slow_start_after_idle=0 (in a container, which has its own copy of it: docker run --sysctl net.ipv4.tcp_slow_start_after_idle=0 ...). Once either answers ready, the worker is yours: it is billed from the moment its machine starts, its start included (see what keeping a deployment costs), and stops after its idle timeout (60 seconds unless its deployment sets idle_timeout) once no robot is connected. A worker your client stopped waiting for before it was ready is stopped a minute after it comes up, billed from its machine’s start until then.

check before you callpython
ready_user, eta_user = policy_user.ready()  # (False, 180.0) while warming, (True, 0.0) when live
r_user = policy_user.ready()
r_user.card, r_user.usd_per_hour            # ("L4", 1.14): the card its worker runs on, and an hour of it
policy_user.wait_until_ready()              # block until it can serve; returns seconds waited

Many robots at once

One Policy per robot: a connection takes one act() at a time, and a second one from another thread while the first is out raises SequencesError; observe() goes on from any thread meanwhile.

two armspython
def drive_user(robot_user, model_user):
    with sequence_ai.connect(model=model_user) as policy_user:  # one each
        return your_loop_user(policy_user, robot_user)  # observe, act, play

with ThreadPoolExecutor() as pool_user:
    left_user, right_user = pool_user.map(drive_user, [arm_l_user, arm_r_user], ["my_policy_user"] * 2)

The handshake is paid once per Policy, not once per call.

Concurrent loops are independent: each has its own connection (your own loop). How many share a worker the platform measures within the deployment’s own declaration (the batch sizes @seq.concurrent lists), and max_gpus caps how many workers it runs; a robot that finds every slot full waits while another worker starts, and past max_gpus it waits in line for up to 2 minutes. A loop that has started is never interrupted. Your account’s GPU limit bounds the workers, enforced as a worker starts, not per step.

A slot follows the Policy that holds it. While the Policy is open, the SDK keeps its slot alive in the background, between acts too, so a robot that stops for a minute or two to reset its scene keeps it. The slot is given back the moment the Policy closes (policy.close(), or the end of its with block), is garbage-collected, or its process exits or is stopped (SIGTERM, SIGHUP). A process that is killed, or loses its network, gives it back within 60 s. So is what ready() gives a robot before its first act — the slot it is told ready with, the worker a poll starts for it, its place in line — so a robot that gives up waiting frees them for the robots behind it at once, or within 60 s when it is killed. A robot cut off from the platform for up to about 40 s keeps its slot (up to 60 s when the cut comes just after a keepalive, sent every 20 s); one cut off for longer is told on stderr that its slot went, and its next act takes one again or waits in line. An episode ending does not give it back: the slot follows the Policy object. So close the Policy, or use with, when the robot is done.

Holding a slot keeps its worker running, whether or not the robot acts, with no limit, billed as it runs: a robot can hold several policies this way and act on one at a time, each answering the moment it is asked. So a Policy left open is billed for as long as it stays open, and for its idle timeout and the minute it can take to stop after that. A robot whose slot went takes one again at its next act, or waits in line. When a worker has to start for it, act() waits for it on every call, within startup_timeout_s: call wait_until_ready() before the arm needs to move, so the wait does not land mid-motion.

What your deployment reads

Its own inputs: whatever you hand policy.observe(x), sent as it is — a dict of frames, arrays, numbers and strings, in the shape the policy reads. Nothing on the robot checks them against a declaration: a policy that cannot read an observation answers with its own error. Nothing checks a frame’s size either: the policy is handed each frame at the size the robot sent it (fit each frame on the robot). A camera frame wrapped in sequence_ai.Video travels compressed, as lossy video, and reaches the policy as an array of its shape and dtype, not pixel for pixel; the markers need Python 3.10+ and pip install 'sequence-ai[observe]'.

What the rows mean

They are what your policy’s act function returned, as it returned it: an array as a read-only numpy array of its dtype and shape where numpy is installed (nested lists without it), bytes as bytes. What they mean is the policy’s own — the pi05-droid template answers {"actions": ...}, 15 rows of seven joint velocities and a gripper position — and turning a row into a command for your controller is your robot’s own code: nothing in the platform or this client clips or converts it. A gripper the policy answers as a fraction is the template’s own client.py to turn into open or closed.

a Franka on the DROID stackpython
env_user = RobotEnv_user(action_space="joint_velocity", gripper_action_space="position")

policy_user.observe(read_obs_user())
out_user = policy_user.act()
for row_user in out_user["actions"][:8]:  # openpi plays 8 of its 15
    env_user.step(row_user)           # 7 joint velocities and the gripper position

Which rows are valid and when the checkpoint was trained to be asked again are the policy’s action_chunk_valid_len and action_chunk_replan_at, as an eval plays each answer; how many your robot plays is your loop’s. Reading one kind of row as another is invisible in the shape: never command pi05-droid’s rows as joint positions — that sends the arm toward its zero pose.

Fit each frame on the robot

Nothing resizes a frame behind your back: encode() sends a frame at its own size, and the policy is handed it at that size. Fit the frame on the robot, before sending:

fit a frame to its camerapython
img_user = sequence_ai.resize_image(frame_user, 224, 224, mode="pad", filter="bilinear")
data_user = sequence_ai.encode(img_user)

mode="pad" keeps the aspect ratio and pads the rest black (openpi’s resize_with_pad); "stretch" fills the box. The filter is one of nearest, box, bilinear, hamming, bicubic, lanczos. Neither has a default: they are how the policy was trained, and a frame fitted another way differs pixel by pixel. A frame may be a numpy array, a PIL image, encoded JPEG or PNG bytes, or base64 text.

Nothing on the worker checks a frame’s size: fit each frame on the robot before sending it, with the mode and filter the policy was trained with — sequence_ai.resize_image(frame, 224, 224, mode="pad", filter="bilinear"). Fitted on the robot, a 224x224 camera frame is about 12 kB on the wire; the same frame sent at 640 px was about 45 kB.

Chat for agents

chat() is the vision-language call an agent plans with: messages in, one message out. Every result carries cost_usd — what the call was charged — so a loop can budget per turn without a second request. A model that thinks spends max_tokens on its thinking too: a small one can go to thinking alone, the content empty and the call billed all the same (r.warnings says so, and chat() warns with it), and a request with an image is likelier to be thought over first. With a small budget, turn thinking off where the model allows it, reasoning={"effort": "none"}, or leave it room: "minimal" still reasons (about 10% of max_tokens), so on a small budget — an image makes it likelier — it can come back empty all the same. Which models answer at which effort with a tiny max_tokens is in Reasoning.

chatpython
import sequence_ai

r_user = sequence_ai.chat("claude-opus-5-5", [
    {"role": "system", "content": "You plan pick-and-place for a six-axis arm."},
    {"role": "user", "content": [
        sequence_ai.image("frame.jpg"),
        {"type": "text", "text": "Where is the red block?"},
    ]},
])
print(r_user.content, r_user.cost_usd, r_user.cached_tokens)

image() takes a file path, bytes, a PIL image or a numpy array and returns the inline image block; a URL is not fetched. response_format={"type": "json_object"} (or a json_schema) asks for structured output. tools and tool_choice let the model ask for a function call — it comes back in r_user.tool_calls, and you run it.

Caching is on by default. Your system prompt and tool definitions are cached on the first call and read back on later ones at a fraction of the input price — the part of an agent’s request that repeats every turn. r.cached_tokens shows the hit, and the saving lands in cost_usd, because the charge follows the model’s own cost. On some models writing the cache costs a little more than input (r.usage["cache_write_tokens"]; the model’s page lists the rate), so caching pays off from the second call that reads it. Mark a larger reused prefix — a manual, few-shot examples — with sequence_ai.cached(text); pass cache=False to opt out.

The cache has a minimum. A model caches only a prefix of at least its own minimum length. A shorter one is processed at the plain input price with nothing cached, and r.warnings says so. The example above, with its one-line system prompt, is shorter than every minimum, so its first call writes nothing to the cache. The minimums, as each model’s maker documents them:

modelsminimum
claude-opus-5-5
claude-haiku-5-5
claude-fable-5-1
512 tokens (Anthropic)
gpt-6-sol
gpt-6-astra
gpt-6-luna
1,024 tokens (OpenAI)
gemini-3-1-pro
gemini-3-8-flash
4,096 tokens (Google)
glm-5-3-flash
qwen3-8-27b
no published minimum
perceptron-mk1-5no prompt cache
stream + conversationpython
# stream: text as it is generated, the full result afterwards
s_user = sequence_ai.chat("claude-opus-5-5", messages_user, stream=True)
for piece_user in s_user:
    print(piece_user, end="", flush=True)
print(s_user.result.cost_usd)

# Conversation keeps the history and echoes tool calls back for you
conv_user = sequence_ai.Conversation("claude-opus-5-5", system="You plan robot tasks.", tools=TOOLS_user)
r_user = conv_user.send("Put the red block in the bowl.")
while r_user.tool_calls:
    for call_user in r_user.tool_calls:
        conv_user.tool_result(call_user["id"], run_tool_user(call_user))
    r_user = conv_user.send()
print(r_user.content, conv_user.cost_usd)

Replaced models. Five ids now name newer models: claude-opus-4-8 is claude-opus-5-5, claude-fable-5 is claude-fable-5-1, gemini-3-7-flash is gemini-3-8-flash, qwen3-vl is qwen3-8-27b and gpt-4-1 is gpt-6-luna. An old id keeps working until at least 2026-11-07: the new model answers at its own price, and r.warnings names the id to send instead. qwen3-vl and gpt-4-1 did not reason, so a call by either that sets no reasoning is sent with reasoning off, as they answered. seq models lists the current ids.

How a call is billed. On the tokens it used, at the model’s published rates: its input and output rate per million tokens, and on some models two more — a long-context rate, which bills the whole call once its prompt reaches a threshold (Claude Haiku 5.5 from 100,000 prompt tokens, for one), and a cache-write rate for the prompt tokens it writes to the cache. Each model’s page in the console lists every rate it has, and so does the model library. A call never costs more than its tokens at those rates, and often less: a cache read, or a provider cheaper than the list, is charged at what it costs.

Reasoning

Several models think before they answer. reasoning= controls how much. Reasoning tokens are output tokens: they are billed, and they count against max_tokens. If the budget runs out while the model is still thinking, the reply comes back with empty content and finish_reason="length" — and the reasoning tokens are still charged — so the result carries a warning saying so.

fieldmeaning
effort"max", "xhigh", "high", "medium", "low", "minimal" or "none" — how much of max_tokens goes to thinking, from about 95% at the top to about 10% at minimal; none turns reasoning off, on a model that allows it
max_tokensa cap on reasoning tokens, for a model that takes a token budget instead of an effort
excludetrue: the model still reasons (and is billed for it), but the reasoning is left out of the reply
enabledtrue: reason at medium effort
include_reasoningthe older parameter: true is reasoning={}, false is reasoning={"exclude": true}

Not set, each model uses its own default, as it did before. How the models behave, measured on a yes/no question with max_tokens=5:

modeleffort="none"with a tiny max_tokens
claude-opus-5-5, claude-fable-5-1, gpt-6-astrarefused: reasoning is mandatory (400)answers
gemini-3-1-pro, gemini-3-8-flashrefused: reasoning is mandatoryempty at every effort — give it headroom
glm-5-3-flashrefused: reasoning is mandatoryempty by default; answers with effort="minimal"
gpt-6-sol, perceptron-mk1-5, claude-haiku-5-5accepted — reasoning offanswers
gpt-6-lunaaccepted — reasoning offempty by default; answers with effort="minimal"
qwen3-8-27baccepted — reasoning offempty by default and at "minimal"; answers with effort="none"

A question the model has to think over is likelier to be thought over first, and one with an image above all: claude-haiku-5-5, which answers the yes/no question, came back empty at max_tokens=5 each time it was asked the colour on an image’s left, its 5 tokens all reasoning, and answered with effort="minimal" or "none". "minimal" lowers the thinking and "none" turns it off, so on a model that takes "none" it is the surer of the two.

For a plan that calls the model every couple of seconds, keep reasoning short and leave room for the answer: reasoning={"effort": "minimal"} with a max_tokens of a few hundred, or {"effort": "none"} on a model that takes it (gpt-6-sol, qwen3-8-27b, perceptron-mk1-5, claude-haiku-5-5, gpt-6-luna). When an empty reply comes back with the warning, raise max_tokens or lower the effort.

What the model thought is on the result: r.reasoning (text, when the provider shows it), r.reasoning_details (structured blocks, some signed) and r.reasoning_tokens — also usage["completion_tokens_details"]["reasoning_tokens"], the same count in the nested place many clients read it from. Send reasoning_details back with the assistant turn to keep a reasoning model’s tool loop continuous; a Conversation does that for you. A stream yields only the reply; its reasoning collects on s_user.reasoning, and on the wire it arrives as delta.reasoning / delta.reasoning_details.

The other model parameters are sent as you give them, under their usual names: include_reasoning, reasoning_effort, max_completion_tokens (1 to 32,768, as max_tokens), seed, stop, top_p, top_k, min_p, frequency_penalty, presence_penalty, repetition_penalty, logit_bias, logprobs (the log-probabilities come back on r.logprobs), top_logprobs, parallel_tool_calls and verbosity. Which ones a model takes is its supported_parameters in models(); whether a value is allowed is the provider’s call, and a value it refuses comes back as InvalidRequest in its own words. Requires sequence-ai 0.33.4 or later.

Chat is answered by an external provider, so there is no cold-start wait. A request the provider rejects — a value out of range, an image it cannot read, a prompt longer than the model’s context — raises InvalidRequest (400) with the provider’s own words; a provider error (502 upstream_error) or a rate limit / no answer (503 unavailable) raises Unavailable. None of them is charged. A stream that fails after it started raises inside the loop and is not charged; a stream you stop early is charged for what was generated. Requires sequence-ai 0.28.0 or later.

Decide

decide() is a one-shot call to a scoring model (jev): a state (string, object, or array) and typed questions in, calibrated probabilities out. It scores rather than generates — no max_tokens or temperature, billed for input tokens only, and nothing of it is spent thinking: reasoning is chat’s alone. Use it for routing, ranking and verification rather than chat.

decidepython
from sequence_ai import decide

result_user = decide(
    "jev",
    "Help! My payouts have been failing for 3 days.",
    {
        "is_urgent":   {"type": "noul",   "instructions": "Does this convey urgency?",
                        "criteria": {"true": "Time-sensitive", "false": "Not urgent"}},
        "department":  {"type": "choice", "instructions": "Which team handles this?",
                        "criteria": {"billing": "Payments", "technical": "Bugs", "sales": "Pricing"}},
    },
)

for key_user, ans_user in result_user.answers.items():
    if ans_user["type"] == "noul":
        print(key_user, ans_user["noul"])             # is_urgent 0.96
    elif ans_user["type"] == "choice":
        print(key_user, ans_user["choice"], ans_user["confidence"])  # department billing 0.81

Each question’s type fixes the answer’s shape: noul is a 0–1 probability, choice picks from options you name and carries the full distribution beside a confidence, score is a position on an ordered rubric. The keys are yours and each answer comes back under the same key, so branch on answers[key]["type"] and gate on confidence rather than trusting the single pick. The type also fixes what criteria must be:

typecriteriaanswer
nouloptional; when given, {"true": …, "false": …} — what yes and no mean{"type": "noul", "noul": 0.97}
choice{option: description}, at least one optionchoice, probabilities per option, confidence
score[label, …] from lowest to highest, at least onescore (a position on the rubric), legend, probabilities per label, confidence

Criteria that do not fit their type are refused before anything is sent or billed, with an InvalidRequest (400) that names the field — questions.grip.criteria: a "score" question needs a list of rubric labels, in order, got an object. Like chat(), decide() is answered by an external provider, so there is no cold-start wait: a request the provider rejects is a 400 with its own words, a provider error a 502 and a rate limit or no answer a 503 (both Unavailable), and none of them is charged. help(sequence_ai.decide) has a minimal example of each type.

decide() stands apart from the Policy loop — scoring rather than control. The model behind it is in the catalogue under Decision.

GPU machines

sequence_ai.gpu.add(), .list() and .remove() open, list and shut down GPU machines of your own, root over SSH (GPUs), near a place in the United States: a US city or a latitude and longitude in the US. One outside is refused, InvalidRequest whose details["candidates"] are the US cities of its name. .list() is of those live, each by its place as you gave it; .history(limit=None, cursor=None) is of those over, the one ended last first, a page at a time: {"gpus", "total", "offset", "next_cursor"}, its next_cursor the next page's cursor. sequence_ai.gpu.places(q="", limit=8) lists the US cities a machine’s place can be: those whose name, or another name they go by, starts with q (an airport’s code, as "SFO", only typed whole), case and accents aside, those of their own name first, each the largest first, at most 20; a state after a comma narrows them, and with no q it lists the largest US cities. Each is {"name", "region", "country", "lat", "lon", "also"}, where the region is its state, and also is the other name a city was found by ("Big Apple", New York City for "big apple"), None for one found by its own; its latitude and longitude are a place.

a GPU near the city chosenpython
import sequence_ai

city_user = sequence_ai.gpu.places("Springfield, IL")[0]   # {"name": "Springfield", "region": "Illinois", ...}
gpus_user = sequence_ai.gpu.add(place=f"{city_user['lat']},{city_user['lon']}", gpu="L4")

Errors

Every error the platform sends has one shape — detail, code, retryable, retry_after_s, request_id, hint, details — and becomes one exception class, the same from chat(), connect() and every seq command. Every error the platform sends inherits SequencesError, so catching that catches all of them and none of httpx’s — but for a built-in exception your policy raised, which comes back as itself, and Interrupted, a BaseException. Switch on .code, show .hint, quote .request_id when you report a problem; the full list of codes is in the platform docs. print(e) shows the words, the hint when there is one, then why and the request’s id — deployment 'my_policy_user' is still building; retry when it is ready (hint: its status says when it is ready) [deployment_building, request r-5a5a5a5a] — the same for every refusal, and seq policy logs my_policy_user --rid r-5a5a5a5a finds it.

Class.codeDo
InvalidRequestinvalid_request, payload_too_largeFix it. Deterministic — the identical request fails identically.
AuthErrorunauthenticatedThe key is missing, revoked or expired. Retrying will not help.
PermissionDeniedforbiddenThis key may not do that — a robot key calling a model its policies do not declare, connecting to a policy not on its list (key_bound), or managing anything; a key bound to another deployment.
OutOfCreditinsufficient_credit, budget_exhaustedStop and hold. For insufficient_credit: at the monthly limit, wait for the 1st or upgrade to Team on the Billing page; with no payment method, add one there; at a contract’s credit line, ask us to raise it; at the key’s own limit, raise or remove it on the Keys page (a robot key from before robots came: replace it), or use another key. For budget_exhausted: raise the deployment’s monthly budget or turn it off.
NotFoundnot_foundNo such name — or not yours, which reads the same.
GonegoneRetired on purpose; .replacement names what to use instead.
ConflictconflictThe state is in the way: a name taken; a deployment building, failed or stopped (seq policy start); a limit reached.
NewVersionconflictA Conflict: a new version went live, or a rollback made another serve, which ended this connection, whatever it was doing. Every call on it raises this: connect() again reaches the version that serves, .version.
Unavailablebusy, warming, unavailable, upstream_error, timeoutRetry after .retry_after_s; .warming means a cold worker. A failed call is not billed; a worker it started is, as any worker.
PolicyErrorpolicy_errorYour policy’s own code raised; .remote_traceback holds its frames. A built-in exception (ValueError, …) comes back as itself.
StreamRefusedstream_refusedSomething between your robot and its worker answered its WebSocket (wss) with no stream — a proxy or firewall that lets none through; .status says what it answered. Let wss to the policy’s workers through it; retrying will not help.
InternalErrorinternal, dependency_missingOurs. Report it with .request_id.
CertPinError—The worker’s certificate did not match the fingerprint its connect pinned. Retrying will not help: a fresh connect brings the right one.

A worker that is going away answers 503 unavailable with details.reason: "draining", and a rotated secret or an expired lease sends the client back to connect: neither reaches you as an error — they are the cases the client resolves itself, by reconnecting through the gateway and replaying (see fallback).

Wire endpoints

For a controller not written in Python. connect reaches the gateway; /stream reaches the worker directly.

POST/v1/act/connect

Client → gateway, once. Admits the call and ensures a worker is up; returns worker_url, cert_fingerprint, direct_secret and a lease (ttl_s, expires_at, epoch). A cold worker answers 503 with Retry-After while it boots.

GET{worker_url}/stream

Client → worker, directly — the gateway is not in this path. One WebSocket (wss, Authorization: Bearer <direct_secret>) that the robot’s observations, acts and episode ends travel on, as msgpack messages (x-seq-stream/3) in the order it made them, each answered on the same connection — an act with outputs, what the policy’s infer returned, as it returned it, with perf and warnings. The same token over plain HTTP is refused (400 invalid_request, details.reason http_gone). A 401 (rotated secret), a stream upgrade answered 403, or lease expiry sends the client back to connect to fetch fresh coordinates.

GET/v1/account

Client → gateway, under your API key. With a robot’s key, its own deployment’s usage alone ({"scope": "deployment", …}); with any other, the read-only status of the account the key belongs to: balance (this month’s money — billing is postpaid, each month’s usage billed at its end: spendable, what you may spend now — your credits and any payment method’s line, at most what this month’s limit leaves past what was spent and what running workers and evals hold (held_usd) — what seq account says is left to spend, and the console’s Left to spend; work that would spend past it is refused; period_spent_usd, credit_left_usd, month_left_usd (what this month’s limit leaves before what running work holds) and payment_method), usage (the policies and benchmarks counted against your limits) and limits (the per-account caps that are configured). Creating keys and adding a payment method take you signed in, never a key: in the console, or with seq key add after seq login; so does adding credit ahead, seq billing top-up.

check this month’s room before a big runpython
import sequence_ai

acct_user = sequence_ai.account()
acct_user["balance"]["spendable"]  # what you may spend now: what seq account and the console's Left to spend say
acct_user["usage"]             # {"policies": 1, "benchmarks": 0}

The full three-contract wire — POST /v1/deployments to register a policy, connect, and the direct /stream — is laid out in the platform docs.

Command line

On Python 3.10+ the package installs seq: start from a template, deploy a policy or register a benchmark, evaluate one on the other, and manage what you deployed. Everything runs on the platform. On Python 3.9 — a robot — seq says it needs 3.10, and the client above works as it is, but for the Video, Depth and Tactile markers, which need 3.10.

seq --help
the core flow:
  init <template> [dir]     start from a template: a policy, a benchmark or an adapter (seq init --list)
  deploy [folder]           deploy the folder's @seq.policy, or register its @seq.benchmark or
                            @seq.adapter (--name picks one of several; --dry-run: check it, and price a policy, with
                            nothing built)
  eval run <policy> --benchmark <name> [--adapter <name>@vN]
                            evaluate a deployed policy on a registered benchmark

policy      ls status logs doctor shell debug versions source upgrade rollback stop start rename rm
            set settings fork data
robot       create list show set record rm key            (a robot of your own: make it, set its policies, record it)
gpu         add places list history logs allow ssh save saved remove
                                                          (GPU machines you run yourself, root over SSH)
key         list add set revoke                           (your keys: you, signed in)
benchmark   ls tasks versions source logs shell status upgrade cancel rm
adapter     ls versions source rm check
eval        run status cancel rm report episodes replay ls logs
storage     storage · volume put|ls|show|files|get|rm · secret set|ls|rm
account     login · logout · profile · audit · models · gpus · account [set|delete] · usage
billing     top-up redeem credits plan invoices details card    (the console's Billing page; you signed in)
port        export check-model scaffold compare convert   (a CPU MuJoCo or SAPIEN benchmark to the GPU)

seq init toy-policy my_policy_user writes the template’s stable version when this sequence-ai can deploy it, and otherwise the newest version it can, saying which and why; @stable, @beta or @vN after the name is written as named, with a warning when it cannot. seq deploy my_policy_user uploads that folder, whole, less version control and caches, a virtualenv or build output at its root, .env files and what its .seqignore files list, deploys its one policy, benchmark or adapter class (--name picks one of several), and lists what it uploads; --dry-run prints the request and, with a key, has the platform check it — a policy, a benchmark or an adapter alike — with nothing built or registered, and for a name deployed before says how long its last measured version took to its first start and about what this version’s check costs — its image builds on the machine you declare, billed at its card — or that there is no data yet. seq policy doctor <deployment> probes the direct-connect chain segment by segment. seq models lists everything you can start from: the chat and decide models and the templates seq init writes (--endpoint chat lists the models a plan can call); it needs no key. seq --json <command> prints what a command read, for scripts.