An agent of your own — one that reads work orders, talks with a person, or runs anywhere else — tells a
robot what to do through the robot’s own connection. It hands its words to the robot’s loop, the loop
sends them with an observation, and the policy’s plan reads them; what
the plan returns is the robot’s next step. In the examples, everything ending in _user is yours,
and so are the strings.
Send what it says
The robot’s loop sends the agent’s words under a key of your own, once, with the next observation. An agent in another process or on another machine reaches the loop the way your programs reach each other.
import queue import threading import sequence_ai said_user = queue.SimpleQueue() # your agent puts what it says here threading.Thread(target=agent_user, args=(said_user,), daemon=True).start() policy_user = sequence_ai.connect("pick_user") while True: seen_user = {"image_user": camera_user.read()} if not said_user.empty(): seen_user["say_user"] = said_user.get() # once, as it comes policy_user.observe(seen_user) robot_user.apply(policy_user.act())
Read it in the plan
wake_on calls the plan as soon as the key comes. It can pass the words on as they are, as here, or ask
a model what they mean for what the robot sees.
import seq @seq.policy(gpu="L4", max_gpus=1, hz=15) class Pick_user(seq.Policy): def __init__(self): self.model_user = ... @seq.plan(every_s=30.0, wake_on=["say_user"]) def plan_user(self, robot): said_user = [o_user["say_user"] for o_user in robot.state["seq.observations"] if "say_user" in o_user] if said_user: return said_user[-1] # None otherwise: its plans stay as they are @seq.infer def act_user(self, obs, episode): plans_user = episode.state.get("seq.plans") step_user = plans_user[-1]["value"] if plans_user else "wait_user" return self.model_user(obs["image_user"], step_user)
What to know
- One connection. Each
connect()is a robot of its own, with its own episode and its own plan. An agent that connects on its own is planned apart and reaches nothing of the robot’s, so its words go through the connection the robot acts on. - A key of your own. A top-level key that starts
seq.is refused before it goes:"seq.plans" is the platform's: keys that start with "seq." cannot be observed, so name it something else. An agent never writes a robot’s plans itself: your plan does. - How long it lasts. Observations hold about the episode’s last minute, and the newest plan stays until the plan returns another. A new episode starts with no plans, so say it again with its first observation.
- Several robots. Each robot has its own plan, which sees that robot alone. An agent that runs several tells each through its own connection.
- What it hears back. What an act returns is yours to make, so a policy can return the step it is on beside its actions, for the loop to hand to your agent.
- When it needs a person. A plan that returns
seq.ask_human(reason)reaches a console, through itsrequests()(Asking for a person). No answer comes back to the plan or to your agent. - With no plan. An agent that only changes a sentence the policy reads needs none: the loop sends the sentence with every observation (Changing the instruction).
- What it costs. Your agent runs where you run it. A plan’s model calls are billed as any chat call, and the plan itself as part of the policy’s worker.