Install
One install for everyone. On a robot (Python 3.9) it is the client and two dependencies,
httpx and Pillow, which fits and encodes each camera frame on the robot; on Python 3.10+ it
also brings seq, to deploy and evaluate your own policy, with pydantic and the
shared contract. This page is the client.
pip install -U sequence-ai # the client: connect / observe / act, and Pillow for the frames
Install sequence-ai, import sequence_ai.
Authentication
One key identifies your account. The client reads it from api_key= first, then
$SEQUENCES_API_KEY, and talks to api.generalsequences.com — no host to configure.
export SEQUENCES_API_KEY="seq_dev_..."
Create one in the console: it starts seq_dev_.
A robot connects with its own robot key instead, which starts
seq_robot_ and reaches the policies on its list (Robots). sequence_ai.models() needs no key.
Or sign in as yourself: seq login prints a code and opens the console to confirm it, and every
command with no $SEQUENCES_API_KEY then runs as you. Making keys, paying and changing the account
take you signed in, because a key makes no keys. seq logout signs this machine out.
seq login --profile <name> keeps a sign-in under a name of its own, so two people on one
machine stay apart; seq profile list shows them and seq profile activate picks one.
Quickstart
A complete control loop. connect() is a context manager, so the connection closes
even when the loop raises; observe() hands the policy each observation, and act(),
which carries nothing, returns what its infer returned.
import time from concurrent.futures import ThreadPoolExecutor import sequence_ai with sequence_ai.connect(model="my_policy_user") as policy_user: pool_user = ThreadPoolExecutor(max_workers=1) hz_user, valid_user, replan_user = 15, 15, 8 # the policy's hz, action_chunk_valid_len, action_chunk_replan_at policy_user.observe(read_obs_user()) sent_user, ahead_user = time.monotonic(), pool_user.submit(policy_user.act) while not done_user(): rows_user, t0_user, ahead_user = ahead_user.result()[:valid_user], sent_user, None # what infer returned for step_user in range(int((time.monotonic() - t0_user) * hz_user), valid_user): if ahead_user is None and step_user >= replan_user: # replan_user steps since its act went out sent_user, ahead_user = time.monotonic(), pool_user.submit(policy_user.act) arm_user.move_to(rows_user[step_user]) policy_user.observe(read_obs_user()) time.sleep(max(0, t0_user + (step_user + 1) / hz_user - time.monotonic())) if ahead_user is not None and ahead_user.done(): # the next one is in: switch to it break if ahead_user is None: # every row went stale on the way: ask again sent_user, ahead_user = time.monotonic(), pool_user.submit(policy_user.act)
When to ask and when each row goes to the motors is your code: this loop asks for the next chunk in a background
thread replan_user steps after its act went out (the policy’s
action_chunk_replan_at), plays on through the chunk’s valid rows while it comes, and sends
each row at its own time point (your own loop). read_obs_user, arm_user and done_user
stand for your robot’s own.
connect(on_stale=) tells your code the moment the arm runs out of answer
(below).
connect()
connect() returns a Policy handle, or, given several names, a tuple of them in that order
(Connect a robot). Its first observe(), act() or
wait_until_ready() connects, and the robot’s one connection to its worker — a WebSocket
(wss) — opens as soon as the platform names the worker and stays open for the life of the loop.
A call is two phases; the gateway is in only the first.
"teleop" can be one of the names, for a robot a person drives: a handle with the same observe() and
act(), on no worker. connect(hold=…) is what act() hands back while nothing is to be played, and
sequence_ai.teleop(key=, operator=) is the person’s side, from your own Python: Teleoperation.
| phase | reaches | purpose |
|---|---|---|
connect(model) | the gateway, at the first call | admit, ensure a worker is up, return its worker_url, the cert_fingerprint the client pins, direct_secret and lease |
observe(), act() | the worker, directly | hand over each observation and run inference on the one WebSocket, in the order you made them, the worker’s certificate checked; Bearer <direct_secret> |
| reconnect | the gateway, on demand | on a rotated secret, expired lease or a worker that scaled to zero — fetch fresh coordinates and replay |
| keepalive | the gateway, about every 20 s while the Policy is open | keep the robot’s seat, and renew its lease before it runs out — from a thread of its own, which no act waits on |
Direct connect and fallback
You do not handle the 401 or 503 yourself. A rotated
direct_secret, an expired lease, a worker that scaled to zero, or a dropped
connection all surface as one transparent reconnect: the client fetches fresh
coordinates and replays the in-flight call — unless its worker is still running it (see
how long an act waits). The same Policy handle keeps working.
A WebSocket is the one way a robot reaches its worker. A network that lets none through — a proxy or
firewall that answers it with anything but the stream — raises StreamRefused at the first act,
saying what answered (an answer of 401 or 403 reads as AuthError, its hint naming the
WebSocket, and a 5xx as a worker not reached); wait_until_ready() warns of it as it returns.
A cold worker answers connect() with 503 and a
Retry-After while it boots; the client waits that out — see
cold starts and warming.
How long an act waits
An act waits for its answer as long as the deployment allows one — the platform’s limit,
300 s unless its version’s check measured slower inferences — and 10 s
more, so the worker’s own 504 at that deadline is what you get. Where the platform names none
for the deployment, an act waits 30 s. connect(model, timeout_s=…) sets your own bound for every request,
acts included.
An act with no answer in time is never sent again while its worker still runs it: that would run it twice. It
raises Unavailable (504, details["reason"] == "act_still_running"). A
worker that is gone is reconnected around and the act replayed, as above. A worker busy with other
robots’ acts (429) is waited out for the same time from its first refusal, in steps no longer
than its Retry-After; a wait past 5 seconds says once, on stderr, what it waits for, in the
worker’s words (a copy of the policy frees when its episode ends, or when it has had no act for its idle
time).
observe() and act()
observe() hands the policy each observation as it is — every input goes this way, an
instruction too — and act(), which carries nothing, asks for actions and returns what the
policy’s act function returned. When to call each, and how to play what comes back, is your loop
(your own loop).
act() returns what the policy’s infer returned, as it returned it — a dict, a
list, a number, an array: an array as a read-only numpy array of its dtype and shape where numpy is installed (sequence-ai does not install it; without it, nested lists), bytes as bytes. The instruction is one
of the policy’s inputs, in the shape it reads them, so each observe() carries the one in force
(changing the instruction).
An agent that calls your loop
An agent of your own can call your loop as its tool: each observation carries the instruction in force, and the loop plays until the sub-goal is done, a deadline, or an interrupt. Wiring instructions together — retries, failure detection, choosing the next sub-goal — is yours. For a plan the platform runs inside the policy, see Planning and External agents.
act()
policy.act() asks for actions and returns what the policy’s act function returned, once every
observe() sent before it has reached the policy. It carries no data — act(x) is
refused before it goes, saying to hand x to observe() first — and it waits out a
starting worker on every call, within startup_timeout_s.
toy_user = sequence_ai.connect("my_toy_user") # a deployed toy-policy # its inputs, as its README gives them toy_user.observe({"state": [0.0], "target": [0.5]}) toy_user.act()["delta"] # [0.1], as infer returned it
act() hands back the answer alone. What befell an act on its way — its episode reset, a move
to its standby copy, observations sent again or left out — the SDK says with Python’s
warnings.warn; what an act took is on the policy’s page in the console.
A seed. act(seed=7) passes the seed to the policy’s infer(…, seed=7).
A policy that samples answers one observation the same way for one seed — the policy draws its flow-matching
noise from it (a pi05 template draws openpi’s standard normal noise from the seed and hands it to openpi) — and without a seed draws fresh noise, so two
answers to one observation can differ.
episode_id= on observe() or act() puts the call in that episode; without
one, a call is the connection’s current episode, which its first call starts and
policy.end_episode() ends (Episodes).
Changing the instruction
instruction_user = "pick up the cup" # your agent's thread replaces it whenever it decides while not done_user(): policy_user.observe({**robot_user.read_observation(), "prompt": instruction_user}) out_user = policy_user.act()
The sentence is one of the policy’s inputs, in the shape the policy reads it — pi05-droid reads it
as "prompt" — so your agent’s thread keeps the latest one and the robot’s loop hands
it over with every observation it makes; neither waits for the other. An agent’s call inside your loop holds the
arm still for its length; on its own thread, the loop keeps going under the instruction in force.
A new sentence refines or continues, and interrupt() abandons: the first keeps the robot moving, the
second stops a call that is waiting, and what the robot does next is your loop’s.
The new instruction is not instant: an act already on its way was asked with what was observed before it, so the first actions that follow the new sentence come with the act after that.
Changing your mind
policy_user.interrupt()
Thread-safe. Call it from any thread: an act(), observe() or
wait_until_ready() that is waiting — through a cold start or a reconnect too — raises
sequence_ai.Interrupted out of its wait, and your loop decides what the robot does next. It is a
BaseException, so an except Exception around your own hardware calls does not swallow
it.
The flag is cleared as each act() starts, so an interrupt that came between two acts does not cancel
the next one.
Labels and episodes
with sequence_ai.connect(model="my_policy_user") as policy_user: policy_user.observe(robot_user.read()) out_user = policy_user.act() # your loop plays it policy_user.label(success=robot_user.cup_in_gripper()) # the episode of the last act
What the policy returns is its own and goes to your loop as it was returned: nothing clips or checks it, and holding it to the robot’s limits is your robot’s own code.
A policy’s inference parameters — the keyword arguments its @seq.infer declares —
are the same for every robot: their defaults are its settings,
seq policy set <name> params.replan_steps_user=3. What differs between robots goes in their
observations, policy.observe(x).
label(success, notes="", *, episode_id=None) records how the episode of the last act went, and returns the episode id. With the robot’s Record on, your recorder reads it as that episode’s label (see field data).
end_episode(episode_id=None) ends an episode on its worker now — the connection’s current
episode, unless episode_id names another — so a
policy that keeps an episode’s state frees it at once and its copy is free for the robot’s next
episode. It returns the id it ended, or None when none was open, and never raises.
close() ends one left open.
Chunks are deadlines
One call returns a block of future actions: the rows of one answer, the policy’s
action_chunk_valid_len of them valid, played at its hz. The robot asks for the next one
action_chunk_replan_at steps after it sent this act, or when this one arrives if that is later (a
robot has one act on its way at a time), and the next answer has to arrive before the valid rows finish playing;
seq deploy --dry-run says whether it would, and what makes room when it would not.
The client reuses one connection — now straight to the worker — for the life of a
Policy. seq policy doctor <deployment> checks it from the platform: its state and
version, its capacity, its last cold start and a running worker’s /ping; it sends no act.
When the arm runs out of answer
The SDK knows when the answer the robot plays runs out: its act’s send time plus its
action_chunk_valid_len steps at hz. Once a robot asks ahead — an act sent while
the answer it plays still has valid steps — and that answer runs out with no next one in hand,
on_stale is called that moment, from a thread of the SDK’s, and on_resume as the
next answer comes, before act() returns it:
def hold_user(stale_user): # the arm has nothing valid to play arm_user.hold() def resume_user(stale_user): # an answer again, stale_user.stale_s seconds later arm_user.release() with sequence_ai.connect(model="my_policy_user", on_stale=hold_user, on_resume=resume_user) as policy_user: ...
The sequence_ai.Stale each is called with says why in its reason:
"late", the act for the next answer is still on its way (its request_id, for
seq policy logs --rid); "failed", that act raised and your loop has its error;
"not asked", your loop asked for none in time — a loop that hangs mid-run is caught as surely
as a late answer. A robot that asks only once its answer ran out waits for every answer and is not watched, nor
is an episode end_episode() ended, nor a teleop robot while a person drives it or it is stopped.
A stretch that a stop or a person’s taking the robot ends is not resumed then, the arm staying as
on_stale left it: on_resume waits for what the arm can play, a person’s chunk or
its model’s answer once it drives again. Each stretch is
logged on the "sequence_ai" logger and kept on policy.stale (now,
events, stale_s, last); with no on_stale set, the first is
said once a process.
The platform hears the robot too: from its first observe() or act() on, its SDK sends
its worker a heartbeat every 0.4 s saying how the arm stands, and seq robot show says each
connection’s — acting (asking ahead), waiting, stale (why, for how long) or silent once it was not
heard for 2 s: its process gone, hung, or its link down. A stretch that lasts past a heartbeat is two lines of
seq policy logs too, as it began and as it ended, with the late act’s request id. A worker of
a version deployed before heartbeats is not reached at all: the robot’s first act raises
InvalidRequest, saying to deploy the same code again.
It is no emergency stop: it runs in your robot’s process and needs that process alive. Keep a physical e-stop between the library and your motors.
Where an act’s time goes
One act is the worker’s time plus the network both ways. Measured on a policy deployment on an L4, 102 acts from a robot 87 ms from its worker, the median act:
| part | median | where you see it |
|---|---|---|
| the policy’s inference | 247 ms | seq policy logs <name> --all, each
act’s line: “act r-… served in 256.0 ms (policy 247.0 · …)” (without --all, each robot’s acts are folded); the same
lines are in the Logs tab of the policy’s page in the console |
| the rest of the worker’s handling: reading the request, decoding the frames, handing them to the policy | 9 ms | the same line |
| the network, both ways | 198 ms | seq policy logs <name> --rid
<rid>, the r-… of that act’s line: your robot’s own timing of that act, its round trip and the network in it —
for the first act after a connect, the connect, the TCP connect and TLS handshake, and the transfer |
| the act, as the robot waits for it | 455 ms | the same |
The network is the distance and what goes up ahead of the answer. An act carries no data, but the frames
observe() handed over just before it go first, on the same stream, and the worker answers the act once
they are in. A round trip from that robot took 87 ms, and the act paid two: the connection had sat idle through a
chunk’s playback, after which an upload larger than about 14 KB goes in two flights, and two 224×224
camera frames come to about 31 KB. The SDK now keeps that stream from sitting idle, pinging the worker while your
robot sends nothing (How long it actually takes). The 232 ms in
Batching is the first row alone — the model’s time inside a warm
worker, with no network — and not what a robot waits for.
To shorten what the arm waits for:
- Ask ahead. A loop that asks for the next chunk while the current one plays has the whole act
— inference and network — happen while the arm moves, and the arm stops only when an act takes longer
than the valid steps after
action_chunk_replan_atplay; with one act on its way at a time, an answer and the one before it have to come within one chunk together (your own loop). - Send less. What an act waits behind is mostly the frames
observe()sent just before it: send each camera at the size the policy reads (sequence_ai.resize_imagefits it andsequence_ai.encodeencodes it; Image size), never larger. An upload that fits in one flight saves the second round trip after an idle pause. - Be near the worker. The network both ways is mostly distance. The platform places a
deployment’s workers as near its robots as it can;
--ridabove says what one act’s network took from your robot. - Mind your program’s busy threads. Python runs one thread at a time, and each of an act’s
threads waits its turn behind a busy one of yours (a loop of pure Python, an arm’s own SDK reading its
state).
connect()shortens that turn from Python’s 5 ms to 0.5 ms, unless your program set its own (sys.setswitchinterval): beside one busy thread, an act took 190 ms at 5 ms and 96 ms at 0.5 ms, as with none.
Batching serves more robots on a card; it does not shorten one robot’s act.
Cold starts and warming
Warming happens on the first call after connect(), which sends nothing itself, and again whenever a call finds its worker gone — after a connection lost for longer than its idle timeout, or when its copy restarts. A worker that scaled to zero answers
connect with 503 and a retry_after_s; in Python that is
Unavailable with .warming set. observe() and act()
wait it out for you; the try below is what you see with
startup_timeout_s=0.
try: policy_user.act() # connect(..., startup_timeout_s=0) except sequence_ai.Unavailable as exc_user: if exc_user.warming: print(f"still starting; asking again in {exc_user.retry_after_s:.0f}s") # when to ask again, not when it is ready
act() waits warming out on every call — by default up to 1800 s from its first ask, as the
platform names no deadline for a start — and observe() on its first.
startup_timeout_s=0 opts out.
seq policy logs <name> --rid
<rid> splits an act’s time into the connect that handed the robot its worker, the TCP
connect and TLS handshake to the worker, the transfer and the worker. The rid is on the act’s line in seq policy logs <name> --all, and an error the
platform answered says it at its end: deployment 'my_policy_user' is still building; retry when it is ready (hint: its status says when it is ready) [deployment_building, request r-c5b7bb45].
How long it actually takes
A cold start is your deployment’s own: a worker starts with your image, your weights are mounted, and
your class is built, its constructor loading the model. For a multi-gigabyte checkpoint that is
minutes, and a start may first wait for its card, which can take more than 10 minutes, so a robot waits for a start by default up to
1800 s from its first ask; when wait_until_ready() gives up, it says whether the start was still
loading the model or slower than its start usually takes. seq deploy shows
each phase as it runs, and seq policy logs <name> shows every worker’s load after it.
Much of a large model’s load can be its own construction: a class that builds the model and then reads its
checkpoint over it first fills every parameter with random numbers, which the checkpoint replaces. Built inside
transformers’ no_init_weights(), it skips that fill; what the constructor computes for itself,
such as rotary frequencies, is unchanged. pi0.5 built through LeRobot loaded in 14.6 s this way, against 170.1 s.
try: from transformers.initialization import no_init_weights # newer transformers except ImportError: from transformers.modeling_utils import no_init_weights # older ones with no_init_weights(): self.model_user = MyModel_user.from_pretrained(weights_dir_user)
Pass on_warming to show progress your own way (by default one line goes to stderr; a
UserWarning is given as well, which a warnings filter silences). It is called
with the seconds waited so far and the seconds until the client asks again; the platform sends no estimate of a
cold start:
def warming_user(waited_s_user, again_s_user): print(f"starting the model: {waited_s_user:.0f} s so far") with sequence_ai.connect(model="my_policy_user", on_warming=warming_user) as policy_user: ...
A worker stays up while a robot is connected to it, whether or not it acts, with no limit, billed all the while.
Once its last robot disconnects it ends after its idle timeout — 60 seconds unless its deployment
sets idle_timeout (Scaling) — and the
next connection then pays a cold start on a new machine, every time. It can end up to about a minute later,
and is billed until it does. A session that pauses keeps
its worker and its sub-second latency: one policy act on an L4 takes about 240 ms inside a warm worker, and the
round trip at your robot adds the network and the client’s own encoding —
seq policy logs <name> shows both for each act; min_workers keeps one running with no robot connected.
You can also check readiness before you call, instead of reacting to a 503.
policy.ready() returns (ready, eta_seconds) without blocking. It only asks: it never
starts a worker, and while this robot has none it answers (False, None) with the status
cold; wait_until_ready() then connects, and the connect starts the robot’s worker
in the region nearest it (in line for a seat, it connects again when told to: each connect is its turn). Once a
worker serves
it also says the card that worker runs on — the deployment’s one card: card and
usd_per_hour (what one worker costs an hour on it, as billed). While a worker’s start waits
for an available card it is (False, eta). eta counts down to when the start should serve;
once the start takes longer than that, it is None and the returned Ready’s
eta_unknown is "past_expected", rather than a number the
start is already past. A start’s wait for a card can take more than 10 minutes. A robot every place turns
away at once waits in line for the card; past 10 minutes in line,
ready() raises Unavailable, saying the card and how long it waited.
Poll ready() every few seconds, as wait_until_ready() does (every 5 s), or at a
queued answer’s retry_after_s (1 s at the head of the line, 5 s behind it): each poll
is one request on your key’s requests a minute (600 on Starter, 2,400 on Team), which every robot on the
key shares with its connects and keepalives, and a key past it is refused (429) until the minute
rolls on. A robot polling every 0.5 s spends 120 a minute.
policy.wait_until_ready() blocks until the model can serve — by default up to 1800 s, or
your startup_timeout_s (other than 0, which has act() raise at once), or its own
timeout_s seconds —
and returns the seconds it waited. It returns once the robot’s connection to the worker is open too —
its TCP connect, TLS handshake and WebSocket upgrade done — so the first act waits for none of them (one
still not open by its connect’s timeout is said in a warning, and the first act opens it). On
Linux, a connection that sat idle for longer than its retransmission timeout restarts its congestion window
(net.ipv4.tcp_slow_start_after_idle, on by default), so an observation sent after a pause can take
one more round trip to upload: the SDK has its connection to the worker use BBR where the robot allows it (BBR
does not restart; sudo modprobe tcp_bbr makes it available, and the SDK then takes it on its own),
and from the robot’s first observation it also pings the worker on it every 0.1 s that the robot sends nothing, an act’s wait included,
for a minute after its last observation or act. Turning the restart off on the robot does the same:
sudo sysctl -w net.ipv4.tcp_slow_start_after_idle=0 (in a
container, which has its own copy of it: docker run --sysctl net.ipv4.tcp_slow_start_after_idle=0
...). Once either answers ready, the worker is yours: it is billed from the moment its machine
starts, its start included (see what keeping a deployment costs), and stops
after its idle timeout (60 seconds unless its deployment sets idle_timeout) once no robot is connected.
A worker your client stopped waiting for before it was ready is stopped a minute after it comes up, billed from its
machine’s start until then.
ready_user, eta_user = policy_user.ready() # (False, 180.0) while warming, (True, 0.0) when live r_user = policy_user.ready() r_user.card, r_user.usd_per_hour # ("L4", 1.14): the card its worker runs on, and an hour of it policy_user.wait_until_ready() # block until it can serve; returns seconds waited
Many robots at once
One Policy per robot: a connection takes one act() at a time, and a second one from
another thread while the first is out raises SequencesError; observe() goes on from any
thread meanwhile.
def drive_user(robot_user, model_user): with sequence_ai.connect(model=model_user) as policy_user: # one each return your_loop_user(policy_user, robot_user) # observe, act, play with ThreadPoolExecutor() as pool_user: left_user, right_user = pool_user.map(drive_user, [arm_l_user, arm_r_user], ["my_policy_user"] * 2)
The handshake is paid once per Policy, not once per call.
Concurrent loops are independent: each has its own connection (your own
loop). How many share a worker the platform
measures within the deployment’s own declaration (the batch sizes @seq.concurrent lists), and
max_gpus caps how many workers it runs; a robot that
finds every slot full waits while another worker starts, and past max_gpus it waits in line for up
to 2 minutes. A loop that has started is never interrupted. Your account’s GPU limit
bounds the workers, enforced as a worker starts, not per step.
A slot follows the Policy that holds it. While the Policy is open, the SDK keeps its
slot alive in the background, between acts too, so a robot that stops for a minute or two to reset its scene
keeps it. The slot is given back the moment the Policy closes (policy.close(), or the
end of its with block), is garbage-collected, or its process exits or is stopped
(SIGTERM, SIGHUP). A process that is killed, or loses its network, gives it back within
60 s. So is what ready() gives a robot before its first act — the slot it is told ready
with, the worker a poll starts for it, its place in line — so a robot that gives up waiting frees them for
the robots behind it at once, or within 60 s when it is killed. A robot cut off from the platform for up to
about 40 s keeps its slot (up to 60 s when the cut comes just after a keepalive, sent every
20 s); one cut off for longer
is told on stderr that its slot went, and its next act takes one again or waits in line. An episode ending
does not give it back: the slot follows the Policy object. So close the Policy, or use with, when the robot is done.
Holding a slot keeps its worker running, whether or not the robot acts, with no limit, billed as it runs: a robot
can hold several policies this way and act on one at a time, each answering the moment it is asked. So a
Policy left open is billed for as long as it stays open, and for its idle timeout and the minute it
can take to stop after that. A robot whose slot went takes one again at its next act, or waits in line. When a worker has to
start for it, act() waits for it on every call, within startup_timeout_s: call
wait_until_ready() before the arm needs to move, so the wait does not land mid-motion.
What your deployment reads
Its own inputs: whatever you hand policy.observe(x), sent as it is — a dict of frames, arrays,
numbers and strings, in the shape the policy reads. Nothing on the robot checks them against a declaration: a
policy that cannot read an observation answers with its own error. Nothing checks a frame’s size either: the
policy is handed each frame at the size the robot sent it (fit each frame on the robot). A camera frame
wrapped in sequence_ai.Video travels compressed, as lossy video, and reaches the policy as an array of its shape and dtype, not pixel for pixel; the markers need Python 3.10+ and pip install 'sequence-ai[observe]'.
What the rows mean
They are what your policy’s act function returned, as it returned it: an array as a read-only numpy array of its dtype
and shape where numpy is installed (nested lists without it), bytes as bytes. What they mean is the policy’s own — the pi05-droid template answers
{"actions": ...}, 15 rows of seven joint velocities and a gripper position — and turning a row
into a command for your controller is your robot’s own code: nothing in the platform or this client clips or
converts it. A gripper the policy answers as a fraction is the template’s own client.py to turn
into open or closed.
env_user = RobotEnv_user(action_space="joint_velocity", gripper_action_space="position") policy_user.observe(read_obs_user()) out_user = policy_user.act() for row_user in out_user["actions"][:8]: # openpi plays 8 of its 15 env_user.step(row_user) # 7 joint velocities and the gripper position
Which rows are valid and when the checkpoint was trained to be asked again are the policy’s
action_chunk_valid_len and action_chunk_replan_at, as an eval plays each answer; how
many your robot plays is your loop’s. Reading one kind of row as another is invisible in the shape: never command pi05-droid’s
rows as joint positions — that sends the arm toward its zero pose.
Fit each frame on the robot
Nothing resizes a frame behind your back: encode() sends a frame at its own size, and the policy is
handed it at that size. Fit the frame on the robot, before sending:
img_user = sequence_ai.resize_image(frame_user, 224, 224, mode="pad", filter="bilinear") data_user = sequence_ai.encode(img_user)
mode="pad" keeps the aspect ratio and pads the rest black (openpi’s resize_with_pad);
"stretch" fills the box. The filter is one of nearest, box,
bilinear, hamming, bicubic, lanczos. Neither has a default: they
are how the policy was trained, and a frame fitted another way differs pixel by pixel. A frame may be a numpy array,
a PIL image, encoded JPEG or PNG bytes, or base64 text.
Nothing on the worker checks a frame’s size: fit each frame on the robot before sending it, with the mode
and filter the policy was trained with —
sequence_ai.resize_image(frame, 224, 224, mode="pad", filter="bilinear"). Fitted on the robot, a
224x224 camera frame is about 12 kB on the wire; the same frame sent at 640 px was about 45 kB.
Chat for agents
chat() is the vision-language call an agent plans with: messages in, one message
out. Every result carries cost_usd — what the call was charged — so a loop
can budget per turn without a second request. A model that thinks spends max_tokens on its
thinking too: a small one can go to thinking alone, the content empty and the call billed all the same
(r.warnings says so, and chat() warns with it), and a request with an image is
likelier to be thought over first. With a small budget, turn thinking off where the model allows it,
reasoning={"effort": "none"}, or leave it room: "minimal" still reasons
(about 10% of max_tokens), so on a small budget — an image makes it likelier — it can
come back empty all the same. Which models answer at which effort with a tiny max_tokens is in
Reasoning.
import sequence_ai r_user = sequence_ai.chat("claude-opus-5-5", [ {"role": "system", "content": "You plan pick-and-place for a six-axis arm."}, {"role": "user", "content": [ sequence_ai.image("frame.jpg"), {"type": "text", "text": "Where is the red block?"}, ]}, ]) print(r_user.content, r_user.cost_usd, r_user.cached_tokens)
image() takes a file path, bytes, a PIL image or a numpy array and returns the inline
image block; a URL is not fetched. response_format={"type": "json_object"} (or a
json_schema) asks for structured output. tools and
tool_choice let the model ask for a function call — it comes back in
r_user.tool_calls, and you run it.
Caching is on by default. Your system prompt and tool definitions are cached on
the first call and read back on later ones at a fraction of the input price — the part of an
agent’s request that repeats every turn. r.cached_tokens shows the hit, and the
saving lands in cost_usd, because the charge follows the model’s own cost. On some models
writing the cache costs a little more than input (r.usage["cache_write_tokens"]; the model’s
page lists the rate), so caching pays off from the second call that reads it. Mark a
larger reused prefix — a manual, few-shot examples — with
sequence_ai.cached(text); pass cache=False to opt out.
The cache has a minimum. A model caches only a prefix of at least its own minimum length. A
shorter one is processed at the plain input price with nothing cached, and r.warnings says so. The
example above, with its one-line system prompt, is shorter than every minimum, so its first call writes nothing
to the cache. The minimums, as each model’s maker documents them:
| models | minimum |
|---|---|
claude-opus-5-5claude-haiku-5-5claude-fable-5-1 | 512 tokens (Anthropic) |
gpt-6-solgpt-6-astragpt-6-luna | 1,024 tokens (OpenAI) |
gemini-3-1-progemini-3-8-flash | 4,096 tokens (Google) |
glm-5-3-flashqwen3-8-27b | no published minimum |
perceptron-mk1-5 | no prompt cache |
# stream: text as it is generated, the full result afterwards s_user = sequence_ai.chat("claude-opus-5-5", messages_user, stream=True) for piece_user in s_user: print(piece_user, end="", flush=True) print(s_user.result.cost_usd) # Conversation keeps the history and echoes tool calls back for you conv_user = sequence_ai.Conversation("claude-opus-5-5", system="You plan robot tasks.", tools=TOOLS_user) r_user = conv_user.send("Put the red block in the bowl.") while r_user.tool_calls: for call_user in r_user.tool_calls: conv_user.tool_result(call_user["id"], run_tool_user(call_user)) r_user = conv_user.send() print(r_user.content, conv_user.cost_usd)
Replaced models. Five ids now name newer models: claude-opus-4-8 is
claude-opus-5-5, claude-fable-5 is claude-fable-5-1,
gemini-3-7-flash is gemini-3-8-flash, qwen3-vl is
qwen3-8-27b and gpt-4-1 is gpt-6-luna. An old id keeps working until at
least 2026-11-07: the new model answers at its own price, and r.warnings names the id to send instead.
qwen3-vl and gpt-4-1 did not reason, so a call by either that sets no
reasoning is sent with reasoning off, as they answered. seq models lists the current ids.
How a call is billed. On the tokens it used, at the model’s published rates: its input and output rate per million tokens, and on some models two more — a long-context rate, which bills the whole call once its prompt reaches a threshold (Claude Haiku 5.5 from 100,000 prompt tokens, for one), and a cache-write rate for the prompt tokens it writes to the cache. Each model’s page in the console lists every rate it has, and so does the model library. A call never costs more than its tokens at those rates, and often less: a cache read, or a provider cheaper than the list, is charged at what it costs.
Reasoning
Several models think before they answer. reasoning= controls how much. Reasoning
tokens are output tokens: they are billed, and they
count against max_tokens. If the budget runs out while the model is still thinking,
the reply comes back with empty content and finish_reason="length" — and the
reasoning tokens are still charged — so the result carries a warning saying so.
| field | meaning |
|---|---|
effort | "max", "xhigh", "high", "medium", "low", "minimal" or "none" — how much of max_tokens goes to thinking, from about 95% at the top to about 10% at minimal; none turns reasoning off, on a model that allows it |
max_tokens | a cap on reasoning tokens, for a model that takes a token budget instead of an effort |
exclude | true: the model still reasons (and is billed for it), but the reasoning is left out of the reply |
enabled | true: reason at medium effort |
include_reasoning | the older parameter: true is reasoning={}, false is reasoning={"exclude": true} |
Not set, each model uses its own default, as it did before. How the models behave, measured on a
yes/no question with max_tokens=5:
| model | effort="none" | with a tiny max_tokens |
|---|---|---|
claude-opus-5-5, claude-fable-5-1, gpt-6-astra | refused: reasoning is mandatory (400) | answers |
gemini-3-1-pro, gemini-3-8-flash | refused: reasoning is mandatory | empty at every effort — give it headroom |
glm-5-3-flash | refused: reasoning is mandatory | empty by default; answers with effort="minimal" |
gpt-6-sol, perceptron-mk1-5, claude-haiku-5-5 | accepted — reasoning off | answers |
gpt-6-luna | accepted — reasoning off | empty by default; answers with effort="minimal" |
qwen3-8-27b | accepted — reasoning off | empty by default and at "minimal"; answers with effort="none" |
A question the model has to think over is likelier to be thought over first, and one with an image above all:
claude-haiku-5-5, which answers the yes/no question, came back empty at max_tokens=5
each time it was asked the colour on an image’s left, its 5 tokens all reasoning, and answered with
effort="minimal" or "none". "minimal" lowers the thinking and
"none" turns it off, so on a model that takes "none" it is the surer of the two.
For a plan that calls the model every couple of seconds, keep reasoning short and leave
room for the answer: reasoning={"effort": "minimal"} with a max_tokens of a
few hundred, or {"effort": "none"} on a model that takes it (gpt-6-sol,
qwen3-8-27b, perceptron-mk1-5, claude-haiku-5-5, gpt-6-luna). When an empty reply
comes back with the warning, raise max_tokens or lower the effort.
What the model thought is on the result: r.reasoning (text, when the provider shows
it), r.reasoning_details (structured blocks, some signed) and
r.reasoning_tokens — also usage["completion_tokens_details"]["reasoning_tokens"],
the same count in the nested place many clients read it from. Send reasoning_details back with the assistant turn to keep
a reasoning model’s tool loop continuous; a Conversation does that for you. A
stream yields only the reply; its reasoning collects on s_user.reasoning, and on the wire it
arrives as delta.reasoning / delta.reasoning_details.
The other model parameters are sent as you give them, under their usual names:
include_reasoning, reasoning_effort, max_completion_tokens (1 to 32,768,
as max_tokens),
seed, stop, top_p, top_k, min_p,
frequency_penalty, presence_penalty, repetition_penalty,
logit_bias, logprobs (the log-probabilities come back on
r.logprobs), top_logprobs, parallel_tool_calls and
verbosity. Which ones a model takes is its supported_parameters in
models(); whether a value is allowed is the provider’s call, and a value it
refuses comes back as InvalidRequest in its own words. Requires
sequence-ai 0.33.4 or later.
Chat is answered by an external provider, so there is no cold-start wait. A request the
provider rejects — a value out of range, an image it cannot read, a prompt longer than the
model’s context — raises InvalidRequest (400) with the provider’s own
words; a provider error (502 upstream_error) or a rate limit / no answer
(503 unavailable) raises Unavailable. None of them is charged.
A stream that fails after it started raises inside the loop and is not charged; a stream you stop
early is charged for what was generated. Requires sequence-ai 0.28.0 or later.
Decide
decide() is a one-shot call to a scoring model (jev): a
state (string, object, or array) and typed questions in, calibrated probabilities
out. It scores rather than generates — no max_tokens or
temperature, billed for input tokens only, and nothing of it is spent thinking:
reasoning is chat’s alone. Use it for routing, ranking and verification rather than chat.
from sequence_ai import decide result_user = decide( "jev", "Help! My payouts have been failing for 3 days.", { "is_urgent": {"type": "noul", "instructions": "Does this convey urgency?", "criteria": {"true": "Time-sensitive", "false": "Not urgent"}}, "department": {"type": "choice", "instructions": "Which team handles this?", "criteria": {"billing": "Payments", "technical": "Bugs", "sales": "Pricing"}}, }, ) for key_user, ans_user in result_user.answers.items(): if ans_user["type"] == "noul": print(key_user, ans_user["noul"]) # is_urgent 0.96 elif ans_user["type"] == "choice": print(key_user, ans_user["choice"], ans_user["confidence"]) # department billing 0.81
Each question’s type fixes the answer’s shape: noul is a 0–1
probability, choice picks from options you name and carries the full distribution
beside a confidence, score is a position on an ordered rubric. The keys
are yours and each answer comes back under the same key, so branch on
answers[key]["type"] and gate on confidence rather than trusting the
single pick. The type also fixes what criteria must be:
type | criteria | answer |
|---|---|---|
noul | optional; when given, {"true": …, "false": …} — what yes and no mean | {"type": "noul", "noul": 0.97} |
choice | {option: description}, at least one option | choice, probabilities per option, confidence |
score | [label, …] from lowest to highest, at least one | score (a position on the rubric), legend, probabilities per label, confidence |
Criteria that do not fit their type are refused before anything is sent or billed, with an
InvalidRequest (400) that names the field —
questions.grip.criteria: a "score" question needs a list of rubric labels, in order, got an
object. Like chat(), decide() is answered by an external provider,
so there is no cold-start wait: a request the provider rejects is a 400 with its own
words, a provider error a 502 and a rate limit or no answer a 503
(both Unavailable), and none of them is charged. help(sequence_ai.decide)
has a minimal example of each type.
decide() stands apart from the Policy loop — scoring rather than
control. The model behind it is in the catalogue under Decision.
GPU machines
sequence_ai.gpu.add(), .list() and .remove() open, list and shut down GPU
machines of your own, root over SSH (GPUs), near a place in the United
States: a US city or a latitude and longitude in the US. One outside is refused, InvalidRequest whose
details["candidates"] are the US cities of its name. .list() is of those live, each by its
place as you gave it; .history(limit=None, cursor=None) is of those over, the one ended
last first, a page at a time: {"gpus", "total", "offset", "next_cursor"}, its
next_cursor the next page's cursor. sequence_ai.gpu.places(q="", limit=8)
lists the US cities a machine’s place can be: those whose name, or another name they go by,
starts with q (an airport’s code, as "SFO", only typed whole), case and accents
aside, those of their own name first, each the largest first, at most 20; a state after a comma narrows them, and
with no q it lists the largest US cities. Each is
{"name", "region", "country", "lat", "lon", "also"}, where the region is its state, and
also is the other name a city was found by ("Big Apple", New York City for
"big apple"), None for one found by its own; its latitude and longitude are a
place.
import sequence_ai city_user = sequence_ai.gpu.places("Springfield, IL")[0] # {"name": "Springfield", "region": "Illinois", ...} gpus_user = sequence_ai.gpu.add(place=f"{city_user['lat']},{city_user['lon']}", gpu="L4")
Errors
Every error the platform sends has one shape — detail, code,
retryable, retry_after_s, request_id, hint,
details — and becomes one exception class, the same from chat(),
connect() and every seq command. Every error the platform sends inherits
SequencesError, so catching that catches all of them and none of httpx’s — but for a
built-in exception your policy raised, which comes back as itself, and Interrupted, a
BaseException. Switch on
.code, show .hint, quote .request_id when you report a problem; the
full list of codes is in the platform docs. print(e) shows the
words, the hint when there is one, then why and the request’s id —
deployment 'my_policy_user' is still building; retry when it is ready (hint: its status says when it is ready) [deployment_building, request r-5a5a5a5a]
— the same for every refusal, and seq policy logs my_policy_user --rid r-5a5a5a5a finds it.
| Class | .code | Do |
|---|---|---|
InvalidRequest | invalid_request, payload_too_large | Fix it. Deterministic — the identical request fails identically. |
AuthError | unauthenticated | The key is missing, revoked or expired. Retrying will not help. |
PermissionDenied | forbidden | This key may not do that — a robot key calling a model its policies do not declare, connecting to a policy not on its list (key_bound), or managing anything; a key bound to another deployment. |
OutOfCredit | insufficient_credit, budget_exhausted | Stop and hold. For insufficient_credit: at the monthly limit, wait for the 1st or upgrade to Team on the Billing page; with no payment method, add one there; at a contract’s credit line, ask us to raise it; at the key’s own limit, raise or remove it on the Keys page (a robot key from before robots came: replace it), or use another key. For budget_exhausted: raise the deployment’s monthly budget or turn it off. |
NotFound | not_found | No such name — or not yours, which reads the same. |
Gone | gone | Retired on purpose; .replacement names what to use instead. |
Conflict | conflict | The state is in the way: a name taken; a deployment building, failed or stopped (seq policy start); a limit reached. |
NewVersion | conflict | A Conflict: a new version went live, or a rollback made another serve, which ended this connection, whatever it was doing. Every call on it raises this: connect() again reaches the version that serves, .version. |
Unavailable | busy, warming, unavailable, upstream_error, timeout | Retry after .retry_after_s; .warming means a cold worker. A failed call is not billed; a worker it started is, as any worker. |
PolicyError | policy_error | Your policy’s own code raised; .remote_traceback holds its frames. A built-in exception (ValueError, …) comes back as itself. |
StreamRefused | stream_refused | Something between your robot and its worker answered its WebSocket (wss) with no stream — a proxy or firewall that lets none through; .status says what it answered. Let wss to the policy’s workers through it; retrying will not help. |
InternalError | internal, dependency_missing | Ours. Report it with .request_id. |
CertPinError | — | The worker’s certificate did not match the fingerprint its connect pinned. Retrying will not help: a fresh connect brings the right one. |
A worker that is going away answers 503 unavailable with details.reason: "draining",
and a rotated secret or an expired lease sends the client back to connect: neither reaches you as
an error — they are the cases the client resolves itself, by reconnecting through the gateway and
replaying (see fallback).
Wire endpoints
For a controller not written in Python. connect reaches the gateway;
/stream reaches the worker directly.
Client → gateway, once. Admits the call and ensures a worker is up; returns
worker_url, cert_fingerprint, direct_secret and a
lease (ttl_s, expires_at, epoch). A cold
worker answers 503 with Retry-After while it boots.
Client → worker, directly — the gateway is not in this path. One WebSocket (wss,
Authorization: Bearer <direct_secret>) that the robot’s observations, acts and episode
ends travel on, as msgpack messages (x-seq-stream/3) in the order it made them, each answered on the
same connection — an act with outputs, what the policy’s infer returned, as
it returned it, with perf and warnings. The same token over plain HTTP is refused
(400 invalid_request, details.reason http_gone). A 401
(rotated secret), a stream upgrade answered 403, or lease expiry sends the client back to connect to fetch fresh coordinates.
Client → gateway, under your API key. With a robot’s key, its own deployment’s usage alone
({"scope": "deployment", …}); with any other, the read-only status of the account the key belongs
to: balance (this month’s money — billing is postpaid, each month’s usage billed at
its end: spendable, what you may spend now — your credits and any payment method’s line,
at most what this month’s limit leaves past what was spent and what running workers and evals hold
(held_usd) — what seq account says is left to spend, and the console’s Left to
spend; work that would spend past it is refused; period_spent_usd, credit_left_usd,
month_left_usd (what this month’s limit leaves before what running work holds) and
payment_method), usage
(the policies and benchmarks counted against your limits) and
limits (the per-account caps that are configured). Creating keys and adding a payment method take you
signed in, never a key: in the console, or with seq key add after seq login; so does
adding credit ahead, seq billing top-up.
import sequence_ai acct_user = sequence_ai.account() acct_user["balance"]["spendable"] # what you may spend now: what seq account and the console's Left to spend say acct_user["usage"] # {"policies": 1, "benchmarks": 0}
The full three-contract wire — POST /v1/deployments to register a policy,
connect, and the direct /stream — is laid out in the
platform docs.
Command line
On Python 3.10+ the package installs seq: start from a template, deploy a policy or
register a benchmark, evaluate one on the other, and manage what you deployed. Everything runs on the
platform. On Python 3.9 — a robot — seq says it needs 3.10, and the client
above works as it is, but for the Video, Depth and Tactile markers, which need 3.10.
the core flow:
init <template> [dir] start from a template: a policy, a benchmark or an adapter (seq init --list)
deploy [folder] deploy the folder's @seq.policy, or register its @seq.benchmark or
@seq.adapter (--name picks one of several; --dry-run: check it, and price a policy, with
nothing built)
eval run <policy> --benchmark <name> [--adapter <name>@vN]
evaluate a deployed policy on a registered benchmark
policy ls status logs doctor shell debug versions source upgrade rollback stop start rename rm
set settings fork data
robot create list show set record rm key (a robot of your own: make it, set its policies, record it)
gpu add places list history logs allow ssh save saved remove
(GPU machines you run yourself, root over SSH)
key list add set revoke (your keys: you, signed in)
benchmark ls tasks versions source logs shell status upgrade cancel rm
adapter ls versions source rm check
eval run status cancel rm report episodes replay ls logs
storage storage · volume put|ls|show|files|get|rm · secret set|ls|rm
account login · logout · profile · audit · models · gpus · account [set|delete] · usage
billing top-up redeem credits plan invoices details card (the console's Billing page; you signed in)
port export check-model scaffold compare convert (a CPU MuJoCo or SAPIEN benchmark to the GPU)
seq init toy-policy my_policy_user writes the template’s stable version when this sequence-ai can
deploy it, and otherwise the newest version it can, saying which and why; @stable,
@beta or @vN after the name is written as named, with a warning when it cannot.
seq deploy my_policy_user uploads that folder, whole, less version control and caches, a virtualenv or build output at its root,
.env files and what its .seqignore files list,
deploys its one policy, benchmark or adapter class (--name picks one of several), and lists what it
uploads; --dry-run prints the request and, with a key, has the platform check it — a
policy, a benchmark or an adapter alike — with nothing built or registered, and for a name deployed before
says how long its last measured version took to its first start and about what this version’s check
costs — its image builds on the machine you declare, billed at its card — or that there is no data yet.
seq policy doctor <deployment> probes the direct-connect chain segment by segment.
seq models lists everything you can start from: the chat and decide models and the templates
seq init writes (--endpoint chat lists the models a plan can call); it needs no
key.
seq --json <command> prints what a command read, for scripts.