Docs

Open source

sequence-ai, the package these docs are about, is open source under the Apache-2.0 license: the client your robot connects with, the seq command, and the components that make a robot policy fast to reach from its robot over the internet. The components run on a GPU of your own, for free, and the same package rents a GPU of ours near your robot or deploys your policy to ours. Its source, its own docs and its issues are on GitHub: github.com/General-Sequences/sequence-ai.

installshell
# the robot's client, the seq command and every component,
# and uvicorn, numpy and zstandard for a server of your own
pip install sequence-ai

The client and the components run on Python 3.9 or later, on Linux and macOS, and the seq command on 3.10 or later. cuda_graphs uses the torch your model already has, 2.4 or later. Each component is one file, and most import only the standard library.

Three ways to run it

Whose GPUWhat you run on itOn the robot
Your own serveryours, or any GPU your robot can reachyour policy server, with the components you addconnect("http://your-gpu:8000")
A GPU near your robotone of ours, by city (Add GPUs)the same serverconnect("http://<its address>:8000")
A hosted deploymentoursnothing: we build your policy, serve it and score it (Deploy)connect("my_policy_user")

The first needs no account and costs nothing. The other two are billed while a GPU runs (Pricing), and their commands, seq gpu add and seq deploy, are in the same package.

On the robot

Given an address, connect() goes straight to a policy server of yours, with the components of the robot’s side on its connection — BBR, TCP keepalive, a ping while it is idle, and msgpack — and nothing of ours in between: no key, no robot, nothing billed. It answers with the same two calls as a policy of ours, observe() and act() (observe() and act()).

a policy server of your ownpython
import sequence_ai

# everything ending in _user is yours
with sequence_ai.connect("http://your-gpu:8000") as policy_user:
    policy_user.observe({"image": frame_user, "state": state_user})
    actions_user = policy_user.act()["actions"]

Why BBR, and a ping while the connection is idle where the server answers it, shorten an upload after a pause: Round trips. The connection’s other components, each one file, on the robot’s side, on the server’s and on both, with adapters for openpi and LeRobot, are listed in the repository: What does each network component do?

A robot that already runs openpi’s own client against openpi’s own server on a GPU of yours keeps both, with one changed line on each side: the server, started with python -m sequence_ai.network.openpi scripts/serve_policy.py, pings each robot every 100 ms and decodes its JPEG pictures, and the robot sends them as JPEG with image_tools imported from sequence_ai.network.openpi in place of openpi_client’s (How do I run π0 (openpi) on a remote GPU?). With openpi’s own server, which answers no ping while it runs inference, a Linux robot without BBR can still have its window halved once before each upload; BBR avoids that (sequence-ai’s measurements).

On the GPU

The components of the GPU’s side wrap the model your server already has:

your own serverpython
from sequence_ai.engine import cuda_graphs
from sequence_ai.engine.batcher import Batcher

# replayed from CUDA graphs, kept only if its outputs match
cuda_graphs.install(model_user, ["action_head.denoiser"])

# several robots' requests in one forward pass
batcher_user = Batcher(infer_a_batch_user, sizes=[1, 2, 4, 8])

# in each request's handler
action_user = await batcher_user(observation_user)
  • cuda_graphs replays a module your model calls many times a run from CUDA graphs captured from it, and checks each shape against eager mode byte for byte: a shape that differs runs in eager mode.
  • batcher runs the calls waiting for a model as one call, in the batch sizes the model lists.
  • replicas runs replicas of a model, each in its own process on the GPU given to it, and replaces one that dies.
  • slack says how many robots a replica can serve, from the time an action chunk leaves each robot and each robot’s own round trip, and how long an open batch may wait.

With openpi, LeRobot or GR00T: How do I use the inference components with openpi, LeRobot or GR00T?

Its source, docs and issues

  • The code, its tests and examples, and a page for each component: the repository and its docs.
  • A pull request can change the components, sequence_ai.engine and sequence_ai.network, with their tests, examples and docs. The rest of the package, the client of our GPUs and deployments, is published there as it ships, at each release: a change to it begins as an issue (Contributing).
  • To cite it: CITATION.cff.

What runs our GPUs and deployments, from sending a robot to a GPU to billing it, is not in the package.