Docs

Teleoperation

A person can drive any of your robots that has teleop on its list, from the console’s Teleoperation tab or from your own Python, and give it back. Pictures and commands go straight between the console and the robot and reach no server of ours. The robot’s loop stays as it is: act() answers with its policy’s chunk while its policy drives, with the newest chunk its operator sent while a person drives, and with its hold while it is stopped. A robot connected to no policy can be driven too, and holds no machine.

Put teleop on its list

its listshell
# the whole list: name every policy it keeps
seq robot set arm_1_user policies=pick_policy_user,teleop

Or add teleop on the robot’s Policies tab in the console. It is one name on the robot’s list, as a policy is. Taken off, the robot is refused at its next observe() or act().

The robot

the robotpython
# pip install 'sequence-ai[teleop]', Python 3.11 or later
pick_user, teleop_user = sequence_ai.connect("pick_policy_user", "teleop",
                                             api_key=ROBOT_KEY_user)
while True:
    pick_user.observe({"head": sequence_ai.Video(camera_user.read()),
                       "joints": read_joints_user()})
    robot_user.apply(pick_user.act())   # its policy's or a person's

"teleop" goes among the names you connect to, and comes back as one more handle. Connected to it alone, teleop_user = sequence_ai.connect("teleop", api_key=ROBOT_KEY_user), the robot holds no machine and act() is a person’s chunk or its hold. Every handle of one connect shares who drives: while a person drives, any handle’s act() hands back their chunk; while no one does, a policy’s handle goes to its policy and the "teleop" handle hands back the hold.

While a person drives, act() hands back at once the newest chunk they sent that it has not handed back yet, and that one only once: a chunk a newer one replaced before the robot’s next act() is never handed back. With none newer, it hands back the robot’s hold, as it does while the robot is stopped. The hold is the policy’s last answer emptied, its keys kept: each array cut to no steps, each list empty, each dict’s values emptied alike, any other value — a string, a number — None, and {} before its first answer and on a robot with no policy. A robot that has to be told to stop — a base that keeps its last velocity — names its own with connect(hold=...). An act on its way when the robot is stopped or taken comes back the same: a person’s chunk or the hold, never the policy’s answer.

pick_user.driver is who drives it now, a string: "model", "human" or "stopped"; on a robot with no policy, "model" means no one does. What act() already handed back is your loop’s: a loop that plays an answer while it asks for the next reads it each step, and stops playing when it is not "model". A key the platform turns away, or teleop taken off the list, raises at the next observe() or act(), and in wait_until_ready(). A network that drops does not: your loop goes on, and the robot connects again by itself.

From the console

Once teleop is on its list, a robot’s page has a Teleoperation tab. Opening it connects your browser straight to the robot, which has to be running with "teleop" among its names.

  • The picture is the robot’s Video items, each camera under its own name: at most 8, in the order the robot first observes them. A ninth and any after it are not sent, and the robot’s log names them.
  • Take, Release and Stop are take(), release() and stop() below. Stop stays on the screen, and anyone connected can press it.
  • The tab sends no actions. A person’s chunks are sent from Python, with send() on a console that took the robot: From Python, below.
  • Latency is Frame, the robot’s observe() to your screen, the p50 and p95 of the latest 256; and the round trip.
  • Leaving the tab ends the connection. If you were driving, the robot stops first.

From Python

the operator’s consolepython
# pip install 'sequence-ai[teleop]', Python 3.11 or later
op_user = sequence_ai.teleop(key=ROBOT_KEY_user, operator="alice_user")
for step_user in op_user.stream():   # watch it, until you break out
    if take_over_user(step_user):
        break
op_user.take()                         # drive it
op_user.send({"actions": chunk_user})   # the robot's next act() returns it
op_user.release()                      # back to its policy, stopped or not
op_user.stop()                         # stop it at any time, watching too
  • key is the robot’s own key, $SEQUENCES_API_KEY when left out. The key names its robot, so the console needs no robot name. operator is the name you go by, in the robot’s audit trail and to anyone else connected.
  • op_user.stream() gives the robot’s newest view each time something new comes, never a backlog: a dict with "episode", "seq", "observation" and "t_us". The observation is the newest of each item: the newest step’s arrays, its Depth and Tactile items, and each camera’s newest frame, at most 8 cameras in the order the robot first observes them. A camera’s frame may be a few steps from the arrays beside it, older or newer: each comes its own way, and a frame is dropped rather than made to wait. It runs until the console closes: break out of it, as above, or watch it on a thread of its own while you drive. op_user.states() gives each step’s arrays alone, as soon as they come.
  • op_user.metrics() says how late each command and step was: command_us and frame_us, p50, p95 and p99. Its connect_s says how many seconds each step of connecting took. op_user.rtt_ms is the last round trip to the robot.
  • op_user.driver is who drives the robot, as it last said, and a dict where the robot’s own driver is a string: {"driver": "human", "by": "alice_user", "since": 1794402000.0}. "driver" is "model", "human" or "stopped"; "by" the operator’s name, "plan" when its plan stopped it, or None; "since" the epoch second it began, or None. op_user.driving is whether this console is the one driving.

Driving it

  • One person drives a robot at a time: while another drives it, take() raises Conflict naming them in details["by"], and send() before a take() raises Conflict (not_driving). Only the person who drives can release it; once it is stopped, anyone connected can take or release it.
  • Any console can stop it, one only watching too.
  • Send a chunk at least as often as the robot’s loop calls act(): it plays the newest you sent, once — one a newer chunk replaced before its next act() is never played — and with none newer the robot holds where it is.
  • A chunk made from a picture older than half a second, or five of its policy’s steps where that is longer, is dropped and not played, so a picture that froze cannot move the robot.
  • A console that drives the robot and goes quiet for 3 seconds stops it, and closing one stops it at once. One that leaves while it watches changes nothing (op_user.close(), or a with block). At most 4 consoles can be connected to a robot at once.

When it cannot connect

The console and the robot have to reach each other directly: nothing goes any other way in its place. details["reason"] says what stopped it.

reasonerrorwhen
robot_not_connectedConflictthe robot is not connected with "teleop" among its names
teleop_not_listedConflictteleop is not on its list
robot_not_directConflictthe robot’s machine cannot take a direct connection: pip install 'sequence-ai[teleop]' there and connect again
direct_refusedConflictits connection details did not pass the platform’s check
direct_failedUnavailablethe two could not reach each other, the message says why: both behind symmetric NAT, a network that lets no UDP out
direct_lostUnavailablethe connection went while you were driving: send() raises
not_drivingConflictsend() before a take()
another_operator_drivesConflictsomeone else drives it

When a plan asks for a person

A policy’s plan that asks for a person (Asking for a person) reaches your console through op_user.requests(). A robot with "teleop" among its names is stopped, as a console’s stop() stops it, and every console connected to the robot hears of it; the console’s Teleoperation tab says Stopped by its plan. The request is answered once someone takes the robot. A plan never stops a robot a person drives.

What is kept

Nothing of what a person sees or sends is kept by the platform. Turn on the robot’s Record and your own recorder reads every chunk a person sent as a step they drove — its did saying whether the robot played it — and every take, release, stop, timeout and ask for a person (Your own recorder). A robot connected to "teleop" alone writes its own file while its Record is on, with each chunk of a person’s that it played: a chunk a newer one replaced before the next act() was never played, and is not in it (Field data). Each step’s driver says who drove it: model, or human with the operator’s name.

To find each other, the two hand the platform their connection details, which hold both sides’ network addresses. They are deleted as soon as the console has the robot’s answer, within 60 seconds at the most (Data safety). Every take, release, stop and timeout is kept in your account’s audit trail — which robot, when, and who: the name the console gave and the robot key it came with, or the person signed in — and so is each console’s connect, with the address it came from (seq audit). Replacing a robot’s key turns away the robot and its consoles together.

What it costs

A robot and a direct connection cost nothing: no machine is started or held for a robot connected to "teleop" alone or for a console, and pictures and commands reach no server of ours. A robot also connected to a policy keeps its seat on that policy’s worker while a person drives it, billed as when its policy drives. Handed back, the robot’s next act goes to that worker at once, with no start and no wait in line. Record costs nothing more.