What a request from your robot spends on the network to a GPU of yours and back: the round trip, how many of them an upload takes and what shortens them, and the stretch that is your own network.
Round trips
Two numbers say how far your robots are from a GPU, in network time:
- Expected, as you add it: the median round trip that public probes within 150 km of your place
measured over the last 30 days to where your GPU gets its card, counting only places that really had that card
over the last 14 days (
expected_rtt). Where no probe that near measured it, there is no number, andaddsays so. - Your robots’, while it runs:
seq gpu listshows how many robots are connected and their median round trip, read on the GPU from each connection’s own TCP timing — the same whoever runs the command, from wherever (robots, of the last three minutes).
Across most of the US, the round trip to the nearest region is about 30 ms, and under 50 ms almost everywhere (76 places measured, the network alone; the map). A round trip is one TCP handshake, with no data in it. One request of your robot’s takes about:
round trip × round trips + bytes ÷ upload speed + model time
How many round trips an upload takes depends on your robot’s own network stack, and on whether its connection was busy. Between two Linux machines of ours in data centers, 12 ms apart, for an upload after 500 ms idle, one on a connection pinged every 100 ms, and one sent back to back:
| sends | idle | pinged | back to back |
|---|---|---|---|
| 100 KB | 3.0 | 1.1 | 1.05 |
| 500 KB | 3.0 | 1.1 | 1.1 |
| 1 MB | 3.0 | 1.25 | 1.3 |
That is Linux as it ships: a connection left idle for about 200 ms starts slowly again, and a ping keeps it from going idle. Another robot can show far less of it. From a Mac on a home network, 91 ms from its GPU, the same three uploads after 500 ms idle took 1.6, 2.6 and 4.1 round trips, and 1.5, 2.2 and 3.5 with a ping every 100 ms — 10 to 15% less — and a 900 KB upload after 10 seconds idle was no faster with the pings. There most of the time went to the line’s own upload speed, as far as we can tell, which no ping changes.
Between data centers nothing but that restart slows an upload, so its count is small. From a home it is not so: a large upload’s time is mostly its bytes over the line’s own upload speed. From a Linux machine on a home network — a virtual machine on a laptop there, 30 to 35 ms from its GPU — after 500 ms idle, on a connection pinged every 100 ms, and with BBR and no pings, each in its own round trips:
| sends | idle | pinged | BBR |
|---|---|---|---|
| 100 KB | 5.5 | 1.7 | 2.0 |
| 500 KB | 10.3 | 7.8 | 5.1 |
| 1 MB | 13.4 | 12.7 | 8.3 |
At 100 KB the pings do what they do between data centers, and the kernel setting below does as much (2.0). At 500 KB and 1 MB neither helps much — the setting took 13.5 and 16.0. That line needs about 230 ms to carry a megabyte, six or seven of its round trips, however the connection starts; and Linux’s usual congestion control slows at each lost packet, as far as we can tell. BBR, with no pings, did best on the large uploads. That is one line on one morning, 16 uploads a figure.
Between the two machines in data centers, a new connection, its handshake included, takes about 5, 7 and 8 round trips for those three. A stream — 30 frames a second of 50 KB, or 10 of 100 KB or 500 KB — takes about one round trip a frame, as long as it stays under your robot’s upload speed. The bytes add their own time: two 640×480 JPEGs, about 100 KB, add about 40 ms at 20 Mbps and 8 ms at 100 Mbps; the same two uncompressed, about 1.8 MB, add 740 ms at 20 Mbps.
So, for the shortest requests:
- Keep one connection open, and open it before your robot needs it.
- Have your server ping each robot about every 100 ms while it is connected — a WebSocket ping, or an
HTTP/2 or gRPC PING. The robot’s library answers on its own, and nothing changes on the robot. A robot
that runs Linux then never has its connection sit idle long enough to slow down again while the server’s
event loop is free to send the pings: any interval under about 200 ms keeps it, shorter is no better, and at
100 ms it holds even when a ping is lost. With openpi’s own server, which answers no ping while it runs
inference, a Linux robot without BBR can still have its window halved once before each upload
(sequence-ai’s measurements); BBR, below, avoids that. Between our
machines the pings did not slow the requests; on the Mac above they saved 10 to 15%, or nothing; on the home
line above they helped the Linux machine’s small uploads, not its large ones.
With Python’s
websockets, that isserve(..., ping_interval=0.1); with uvicorn,--ws-ping-interval 0.1. - On a Linux robot, have its connection use BBR, which does not start slowly again after a pause. Load it
once with
sudo modprobe tcp_bbr— a linetcp_bbrin/etc/modules-load.d/bbr.confloads it at every boot — then set it on the robot’s socket: in Python,sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_CONGESTION, b"bbr"), which fails where BBR is not loaded. sequence-ai sets it on its own connection to a policy wherever the robot has it loaded (Pacing on a real robot); to a server of your own on a GPU, your own client sets it. - On the robot, turn off Nagle’s algorithm on its socket (
TCP_NODELAY), or send each message in one write: a small message written in two parts otherwise waits about 40 ms more. In Python, that issock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1). - Have your server send each reply in one write too, and send less: shrink or compress images before your robot sends them. On a home line the bytes are most of a large upload’s time.
- For a raw TCP protocol of your own, with nothing that answers a ping, set
sysctl -w net.ipv4.tcp_slow_start_after_idle=0on a Linux robot instead.
A packet lost on a long connection can leave it slower from then on, as the robot’s own network stack decides: in our test, 1 MB went from about 1.2 round trips to 2.1–2.3. We do not promise any round trip; these are what we measured.
Your network
The last stretch is yours: your robot to its router, and your line out. On Wi-Fi it is what makes the slow requests slower, and by how much changes from hour to hour. We measured it twice on one home Wi-Fi whose line out is a cellular one, late one night and in the evening of the day after: a laptop sending 30 frames a second to the nearest region, each frame timed from its first byte sent to its acknowledgement.
| 30 frames a second | night | evening |
|---|---|---|
| 20 KB: half under | 39 ms | 37 ms |
| 20 KB: 1 in 100 over | 115 ms | 74 ms |
| 20 KB: over 100 ms | 2.6% | 0.3% |
| 50 KB: half under | 48 ms | 48 ms |
| 50 KB: 1 in 100 over | 131 ms | 415 ms |
| 50 KB: over 100 ms | 5.7% | 12.4% |
Each figure is one to three minutes of frames. Half of them took the same both times; the slowest did not, and at 50 KB they took three times as long in the evening. What we have seen help, and what is common practice:
- Plug in a cable where you can.
- Send well under what your line can upload, in smaller frames. Thirty 50 KB frames a second are 12 Mbit/s, on
a line we have measured uploading 16 to 47 Mbit/s: when anything stalls the stream, the frames behind it
queue, and bigger frames take longer to clear. At 20 KB a frame, about 5 Mbit/s, one in a hundred took over
115 ms at most. Fit each frame on the robot to the size its model was trained at, with
sequence_ai.resize_image. - On a Mac, keep System Settings’ Wi-Fi page closed while your robot runs. The night we had it open, the radio was scanning for networks half the time and 39% of the 50 KB frames took over 100 ms; closed, 5.7% did. It is not the only cause: in the evening, with it closed, a quarter of the frames in the first half minute took over 100 ms, and almost none in the second. Turn AirDrop off too: it takes the radio off your network’s channel now and then, which we did not time.
- On Linux — a Raspberry Pi, a Jetson, a laptop — turn off Wi-Fi power saving:
iw dev wlan0 set power_save off, with your interface’s name, orwifi.powersave = 2in NetworkManager. That is common practice; we have not measured it. - Have your robot ask for its next actions while it still has some to play, so a slow answer costs no pause:
for a policy,
action_chunk_replan_atbelowaction_chunk_valid_len(Ask at the replan step). A minute after an answer first comes late, sequence-ai says how many did and, where asking earlier would help, at which step.
Not all of it is the Wi-Fi. That night, with the Wi-Fi hop steady, as it was for 92% of the frames, one 50 KB frame in a hundred still took over 126 ms: what is left comes after the router, most likely on the home’s own line. And the region counts for more than any of these: half the same frames to the far coast took over 111 ms, against 48 ms to the nearest, which is where your GPU is asked for first. One home network, measured twice: what we saw there, not what to expect of yours.