Research · Serving
Many robots, one GPU: async batching for action chunks
Language models get cheap by serving many users at once. Robots each run their own control loop and cannot wait in a queue. We show how the slack inside an action chunk lets several robots share one copy of a model, with no shared clock and no robot told when to send.
Why batching is hard for robots
Large model serving is cheap because of batching. A GPU that answers one request in 30 ms can often answer sixteen in 60 ms, and systems from Clipper1 and Clockwork2 to NVIDIA Triton's dynamic batcher3 are built around one idea: hold a request for a few milliseconds so that it can leave with others, as long as everyone still finishes inside a latency target. Orca4 and vLLM5 pushed this further for language models by rescheduling the batch at every generated token.
A robot is a different kind of client. Every robot runs its own closed control loop, on its own clock, and a late answer is not a slow page load: the arm stops, or acts on a stale plan. Cloud robotics work such as FogROS26 showed that offloading heavy computation is worth it, but a cloud that serves one robot per GPU is too expensive to be the default. So the question we care about is simple to state. Can several robots share one copy of a vision language action model without any of them waiting longer than its control loop allows?
Action chunks create slack
Modern robot policies do not predict one action at a time. ACT7 and Diffusion Policy8 made it standard to predict a chunk of future actions, and today's large models, from OpenVLA9 to π010 and π0.511, do the same. A chunk is a promise: the robot knows what to do for the next few steps, whether or not a new answer has arrived.
Replanning is what keeps a chunked policy closed loop. A policy π maps an observation ot (camera images, joint state and the instruction) to a chunk of L future actions. The robot does not play the whole chunk: after r of them it captures a fresh observation and asks again, and when the new chunk arrives it replaces whatever was left of the old one. A small r reacts faster but calls the model more often. A large L is a longer promise, and the room between r and L is how long the robot can keep going on the old chunk if the next one is late.
That promise is what makes asynchronous execution possible. Real time chunking from Physical Intelligence12 starts computing the next chunk while the current one is still being executed, and freezes the actions that will run during the delay so the two chunks join smoothly. SmolVLA and LeRobot's async inference13,14 decouple the robot from the policy server in the same spirit: the robot sends a new observation when its action queue drops below a threshold and keeps moving while the server thinks.
Prior work uses this slack to hide latency for one robot. Our observation is that the same slack is exactly what a server needs to batch. If a robot's answer is not needed for another 267 ms, the server is free to hold its request for part of that time and let it leave together with requests from other robots.
What goes wrong without it
We first tried the obvious thing on our platform: two robots running π0.5 (LIBERO) closed loop against one copy of the model on an NVIDIA L4, with a short batching window of 10 ms. Each robot sends its next request as soon as the last answer arrives. Over 120 requests per robot, only 2 left in a batch of two.
The batching window was too short to catch the other robot, and making it longer would add the same delay to every request whether or not anyone else was coming. A fixed window is a single latency target for the whole model, which is how web serving thinks. Robots each have their own deadline, set by where they are in their chunk.
Async batching
Our scheduler replaces the fixed window with each robot's own deadline. Robots keep asking on their own clocks; nobody is told when to send. When a request arrives, the worker knows from the policy's declaration how much slack that robot has left: the steps between its replan step and the end of its chunk, at its control rate, minus its measured round trip and a 50 ms margin. The request is held only as long as that slack allows, and leaves with whatever else is waiting when holding any longer would make some robot late.
Every request carries two numbers from the robot: how long its requests spend outside the worker (network, encoding and decoding, measured by the robot itself), and how much of its current chunk is left. From those the worker computes a deadline for each request: the latest moment its answer may leave and still reach the robot before the chunk runs out, with 50 ms to spare. An open batch waits for the other robots on the same copy, but only for robots that can actually arrive before the earliest deadline in the batch. A robot cannot ask again sooner than r steps after its last request, so the worker knows who is worth waiting for. Our first version waited for every robot on the copy; with four robots it formed batches of two that each waited for a partner who could not come, and 45% of answers arrived late. Waiting only for robots that can arrive fixed it.
Each robot on its own clock
Robots are not the same distance from their worker. If one robot has a 300 ms round trip and three have 60 ms, a single rule for the whole copy would shrink every batch to what the far robot can afford. Instead each robot gets its own fit: the largest batch it can join and still be answered in time.
fit(i) = max b such that T(b) + RTT(i) + 50 ms ≤ slack(i)
Here T(b) is the measured time of a batch of b on this worker, RTT(i) the round trip robot i reports, and slack(i) what is left of its chunk, (L − r) / hz for a robot on schedule. A far robot is batched apart, in batches its own fit allows, and no batch waits for it; it costs only its own load and never lowers anyone else's batch size. When a worker starts, it times every batch size the model was compiled for on a sample episode, and then seats robots nearest first, as long as every robot's batches stay on time and all the batches together use at most 90% of the time between one robot's requests.
We picked 90% from measurements. On a DROID policy, two robots per copy at 90% load got 94.1% of their answers back in time, and four robots at 96% load got only 85.7%. Periodic robots naturally settle into gaps between each other's batches until the GPU is nearly full, and then pile up behind each other all at once, so the cap sits just below the knee.
None of this needs a shared clock between robots, a change to the model, or the robot to know that anyone else exists. And a policy that asks only at the last valid step of its chunk has no slack to share, so it gets one robot per copy.
Trading a little wait for density
Some policies are slow enough that a chunk's slack is smaller than one inference. For them we added one knob, max_wait_for_batch: the developer declares how long a robot may wait for its answer, and the robot stops for at most that long if the answer is not in hand. The worker then seats robots by that wait instead of by the chunk.
This is the same trade every serving system makes, with one difference: the developer chooses it per policy, in units their robot understands, and the robot's own code decides what to do while it waits.
What we measured, honestly
The method only helps when the GPU has room to spare inside a robot's budget. For π0.5 on an L4, it does not yet. A 15 Hz robot executing 8 actions per chunk has 533 ms of motion per chunk, and the network round trip from our measured robots was 237 ms at the median, which leaves under 300 ms for inference.
| Setup on one L4, π0.5 | Round trip per robot, median | p95 |
|---|---|---|
| One robot, one copy | 368 ms | 444 ms |
| Two robots, one copy, fixed 10 ms window | 515 ms | 592 to 633 ms |
| Two robots, two copies on the same card | 599 to 605 ms | 705 to 713 ms |
We report this because it is the point of the design. The seat count comes from the model's own timings and the robot's own chunk, so the platform never packs robots onto a card that cannot keep them on time. The same rule seats more robots on faster cards, on longer chunks, on lighter models, and closer to the robot, and every 100 ms saved on the network is 100 ms more for batching.
max_wait_for_batch; the platform does the rest.What comes next
Batching at the level of a whole chunk has a floor: a request that arrives just after a batch starts must wait for that entire inference. Language model servers removed the same floor by rescheduling at every token4. Flow matching policies such as π0 already take a separate timestep for every row of a batch, and generate a chunk in about ten denoising steps after encoding the images and instruction once. We are building a scheduler that admits new requests between denoising steps, so a robot waits for at most one step rather than one inference. Because real time chunking also works step by step during denoising12, the two fit the same loop.
If you run robot policies and want to try this on your own arms, start with the developer docs.
References
- D. Crankshaw, X. Wang, G. Zhou, et al. Clipper: A Low-Latency Online Prediction Serving System. NSDI 2017.
- A. Gujarati, R. Karimi, S. Alzayat, et al. Serving DNNs like Clockwork: Performance Predictability from the Bottom Up. OSDI 2020.
- NVIDIA. Triton Inference Server: Batchers.
- G.-I. Yu, J. S. Jeong, G.-W. Kim, et al. Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI 2022.
- W. Kwon, Z. Li, S. Zhuang, et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023.
- J. Ichnowski, K. Chen, K. Dharmarajan, et al. FogROS2: An Adaptive Platform for Cloud and Fog Robotics Using ROS 2. ICRA 2023.
- T. Z. Zhao, V. Kumar, S. Levine, C. Finn. Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. RSS 2023.
- C. Chi, Z. Xu, S. Feng, et al. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. RSS 2023.
- M. J. Kim, K. Pertsch, S. Karamcheti, et al. OpenVLA: An Open-Source Vision-Language-Action Model. 2024.
- K. Black, N. Brown, D. Driess, et al. π0: A Vision-Language-Action Flow Model for General Robot Control. 2024.
- Physical Intelligence, K. Black, N. Brown, et al. π0.5: a Vision-Language-Action Model with Open-World Generalization. 2025.
- K. Black, M. Y. Galliker, S. Levine. Real-Time Execution of Action Chunking Flow Policies. 2025. See also Real-Time Action Chunking with Large Models.
- M. Shukor, D. Aubakirova, F. Capuano, et al. SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics. 2025.
- F. Capuano, S. Palma, M. Aractingi, et al. Asynchronous Robot Inference: Decoupling Action Prediction and Execution. Hugging Face, 2025.