Skip to content

Commit c00c8bd

Browse files
fracapuanopcuenca
fracapuano
and
pcuenca
authored
Add async inference blogpost (#2926)
* add: blogpost first draft * add: entry to blog yml file * fix: addressing pedro's comments 🥹 * add: what am I doing here * fix: take in merve comments * add: table of contents * fix: authors * fix: links * Update async-inference.md Co-authored-by: Pedro Cuenca <pedro@huggingface.co> * address julien comments * fix: remove duplicates, date * Update async-robot-inference.md Co-authored-by: Pedro Cuenca <pedro@huggingface.co> * fix: minor * Revert "fix: remove duplicates, date" This reverts commit fc7feb9. * fix: duplicated entries, date (again) --------- Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
1 parent f298d55 commit c00c8bd

File tree

3 files changed

+253
-0
lines changed

3 files changed

+253
-0
lines changed

‎_blog.yml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6315,3 +6315,15 @@
63156315
- mcp
63166316
- gradio
63176317
- multimodal
6318+
6319+
- local: async-robot-inference
6320+
title: "Asynchronous Robot Inference: Decoupling Action Prediction and Execution"
6321+
author: fracapuano
6322+
thumbnail: /blog/assets/async_inference/thumbnail_async_blog.png
6323+
date: Jul 10, 2025
6324+
tags:
6325+
- lerobot
6326+
- robotics
6327+
- models
6328+
- open-source
6329+

‎assets/async_inference/thumbnail_async_blog.png

107 KB
Loading

‎async-robot-inference.md

Lines changed: 241 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,241 @@
1+
---
2+
title: "Asynchronous Robot Inference: Decoupling Action Prediction and Execution"
3+
thumbnail: /blog/assets/async_inference/thumbnail_async_blog.png
4+
authors:
5+
- user: fracapuano
6+
- user: imstevenpmwork
7+
- user: aractingi
8+
- user: danaaubakirova
9+
- user: AdilZtn
10+
- user: aliberts
11+
- user: rcadene
12+
---
13+
14+
**TL;DR**
15+
Robotic policies are increasingly bulky, and predict chunks of future actions rather than a single next action. This results in the robot being idle while awaiting new actions to perform, introducing noticeable lags at execution, and lacking of responsiveness. Asynchronous inference tightens the control loop, removing lags at runtime and resulting in more adaptive control by decoupling action prediction from action execution. In this blog post, we cover the basics behind async inference, and how it can be used to improve the performance of robotic policies in the real-world.
16+
17+
## Table of Contents
18+
- [Getting started](#getting-started)
19+
- [Async inference: a deep dive](#async-inference-a-deep-dive)
20+
- [1. Why sequential inference falls short](#1-why-sequential-inference-falls-short)
21+
- [2. Asynchronous inference, in a nutshell](#2-asynchronous-inference-in-a-nutshell)
22+
- [3. System Architecture](#3-system-architecture)
23+
- [Robot Client](#robot-client)
24+
- [Policy Server](#policy-server)
25+
- [4. Analyzing async inference](#4-analyzing-async-inference)
26+
- [5. Using async in your setup](#5-using-async-in-your-setup)
27+
- [Conclusions](#conclusions)
28+
29+
## Getting started
30+
Get started with async inference by following [our tutorial](https://huggingface.co/docs/lerobot/en/async).
31+
32+
<p align="center">
33+
<video width="600" height="400" controls autoplay muted loop playsinline>
34+
<source src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/async-inference/seq_vs_async.mov" type="video/mp4">
35+
</video>
36+
</p>
37+
38+
*Sequential inference (first) versus async inference (second)*. Allowing for replanning and a tigther control loop, async inference results in (1) attempts at recovery, and (2) a ~2x speedup in task completion. Sequential inference keeps acting out the current action chunk even after failure to grasp the object, while async inference can replan and act the new action chunk. Both setups use the same policy!
39+
40+
41+
## Async inference: a deep dive
42+
43+
With async inference, we decouple action execution from action prediction. This is particularly relevant considering the tendency of currently popular models like [[ACT](https://huggingface.co/papers/2304.13705)], [[OpenVLA](https://huggingface.co/papers/2406.09246)], [[PI0](https://huggingface.co/papers/2410.24164)], and [[SmolVLA](https://huggingface.co/papers/2506.01844)] to be outputting chunks of actions \\(a_{t:t+H}\\) rather than single actions \\(a_t\\) given an observation \\(o_t\\).
44+
Convince yourself of this by running all these models using [LeRobot](https://huggingface.co/lerobot).
45+
46+
Using chunks sequentially results in (1) lags at runtime, impacting task execution time and (2) lack of responsiveness, due to acting widely open-loop.
47+
Asynchronous inference avoids mitigates both these limitations by **decoupling action prediction from action execution**.
48+
We introduced asynchronous inference in SmolVLA, and found it to result in a ~2x speed-up in task completion time with comparable task success rate.
49+
50+
In particular, we design a 2-component system where policy inference and action execution are performed in two different processes, possibly on two different machines connected through the network:
51+
* A **`PolicyServer`**, hosted on accelerated hardware and capable of running inference using more computational resources than the ones allocated on a real-world robot.
52+
* A **`RobotClient`** enqueues the received actions and executes them while the next chunk is being computed.
53+
54+
Communication between `PolicyServer` and `RobotClient` relies on **gRPC**, which guarantees ~5× faster performance than a comparable REST API. The result of all of this is a robot that *never* waits for inference.
55+
56+
<p align="center">
57+
<img width="600" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/async-inference/async_scheme.png" alt="Async inference scheme"/>
58+
</p>
59+
60+
<p align="center"><i>Asynchronous inference</i>, highlighting: (1) The client sending the first observation for inference, receiving the first chunk shortly after; (2) The client sending another observation for processing while it has not yet exhausted the current chunk; (3) The client receiving an updated action chunk, which it aggregates with the remaineders of the one it was previously executing.
61+
</p>
62+
63+
---
64+
65+
## 1. Why sequential inference falls short
66+
67+
Suppose a policy \\( \pi \\) maps the current observation \\( o_t \\) to a sequence of \\( H \\) future actions.
68+
Formally, \\(\pi : \mathcal{O} \;\mapsto\; \mathcal{A}, \mathbf{A}_t = \begin{pmatrix} a_{t}, & a_{t+1}, & \dots & a_{t+H} \end{pmatrix} = \pi(o_t)\\).
69+
70+
A traditional control loop would therefore consist of the following steps:
71+
72+
1. Capture \\( o_t \\).
73+
2. Run \\( \pi(o_t) \\) to obtain \\( \mathbf{A}_t = \pi(o_t) \\).
74+
3. Enqueue \\(\mathbf{A_t} \\) and start acting popping actions from the queue.
75+
4. If the queue is empty, wait for \\( \mathbf{A}_{t+H} \\), otherwise repeat step 3.
76+
77+
During step 2 the robot is **idle**. The latency grows with the model size (and models tend to be increasingly bulky over time), and can quickly dominate interaction time (which is typically around 1/`fps`), as shown in the video below (coming from our [Discord community](https://discord.com/invite/ttk5CV6tUw) 🤗):
78+
79+
<p align="center">
80+
<video width="600" height="400" controls autoplay muted loop playsinline>
81+
<source src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/async-inference/lags_videos.mp4" type="video/mp4">
82+
</video>
83+
</p>
84+
85+
This directly results in (1) reduced performance in terms of task completion time---the robot needs to be waiting for the next action chunk to be computed---and (2) reduced responsiveness, due to (2.1) acting widely open-loop while actions are available and (2.2) complete idleness while waiting for the next action chunk.
86+
87+
<div style="display:flex; align-items:center; justify-content:center; gap:12px;">
88+
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/async-inference/sync.png" alt="Sequential inference – idle periods highlighted" style="width:66%;"/>
89+
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/async-inference/time_to_select_action.png" alt="Time to select action – spikes indicate inference" style="width:33%;"/>
90+
</div>
91+
92+
<p align="center">(Left)<i>Sequential inference</i> with highlighted idle periods. (Right)<i>Time to select an action</i> showing spikes when inference is triggered due to local queue exhaustion (inference latency is around ~100ms---~3 frames at 30fps---using an ACT model on a 2021 MacBook Pro).</p>
93+
94+
---
95+
96+
## 2. Asynchronous inference, in a nutshell
97+
Our system removes the idle period by overlapping computation and execution:
98+
99+
1. `RobotClient` streams the latest observation to `PolicyServer`.
100+
2. While the server performs inference, the client executes the **current queue** of actions.
101+
3. New actions arrive, are merged into the queue, and the loop continues.
102+
103+
The key idea is that the robot already knows what to do for the next few timesteps, so it can keep moving while fresh actions are being computed on the server.
104+
105+
<p align="center">
106+
<img width="600" src="https://github.com/user-attachments/assets/6f323660-52b4-4537-8bde-f9b70b7f1bc0" alt="Async inference diagram"/>
107+
</p>
108+
<p align="center"><i>Asynchronous inference</i> overlaps in time the execution of the current action chunk with the computation of the next one, by decoupling these two processes, possibly running them on entirely distinct machines connected through the network.
109+
</p>
110+
111+
This results in a tighter control loop, and a robot that never waits for inference. In turn, this results in ~2x speedup in task completion time with comparable task success rate, and more adaptive control coming from a tighther loop (see video below).
112+
113+
<p align="center">
114+
<video width="600" height="400" controls autoplay muted loop playsinline>
115+
<source src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/async-inference/remi-plays-async.mp4" type="video/mp4">
116+
</video>
117+
</p>
118+
119+
---
120+
121+
## 3. System Architecture
122+
| Component | Role | Technology |
123+
|-----------|------|-------------|
124+
| **RobotClient** | Runs on-board, streams observations, maintains an **action queue**, executes actions | Python, gRPC |
125+
| **PolicyServer** | Hosts the policy, performs batched inference, sends action chunks back | Python, gRPC, possibly accelerated hardware (GPU/TPU) |
126+
127+
Because gRPC is HTTP/2-based and uses protocol buffers, it achieves low-latency binary messaging and bidirectional streams out of the box, which in turn helps us maintain a tighter control loop and sub-100ms round-trip latency (on our local network, and hosting SmolVLA on a NVIDIA RTX 4090).
128+
129+
The `RobotClient` runs on-board, and streams observations to the `PolicyServer` through gRPC. The `PolicyServer` prepares the observations received for inference, and sends back to the `RobotClient` an action chunk.
130+
131+
132+
### Robot Client
133+
134+
<p align="center">
135+
<img width="600" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/async-inference/from_client_perspective.png" alt="From client perspective"/>
136+
</p>
137+
<p align="center"><i>From the client's perspective</i>, observations are streamed to the server according to the local queue status. Incoming chunks are aggregated on overlapping portions with the currently available action queue.
138+
</p>
139+
140+
The `RobotClient` maintains a local action queue and follows a simple yet effective strategy: **send a new observation when the queue length drops below a configurable threshold** (\\(g\\) in the SmolVLA paper, `chunk_size_threshold` in the code).
141+
This threshold value, expressed as a fraction of the maximum chunk size, acts as a trigger condition that balances computational load with responsiveness.
142+
143+
<p align="center">
144+
<img width="600" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/async-inference/client_to_server.png" alt="Client to server"/>
145+
</p>
146+
<p align="center"><i>The client streams observations to the server</i>, according to the local queue status.
147+
</p>
148+
149+
From the client's perspective, the process unfolds as follows:
150+
151+
1. **Queue monitoring**: The client continuously monitors its action queue length against a **chunk size threshold** parameter. When the queue drops below this threshold, it signals that a new observation should be sent for processing.
152+
153+
2. **Observation streaming**: Once the threshold condition is met, the client captures the current observation and streams it to the `PolicyServer` via gRPC. Crucially, **observations are streamed rather than being sent via a unary RPC** because they typically exceed the maximum message size of 4MB (multiple camera captures at high resolution result in this).
154+
155+
3. **Action chunk aggregation**: When a new action chunk arrives from the server, the client merges it with any remaining actions in the current queue over the overlapping portion. This is where **custom aggregators** come into play, handling overlapping sections between the current and incoming chunks differently. As of now, we support flexibly aggregation between the chunks via the specification of a custom `aggregate_fn(chunk1: torch.Tensor, chunk2: torch.Tensor) -> torch.Tensor` function, which is called for each overlapping timestep and can be user-provided.
156+
The overlapping portions (shown in light blue in the diagram) require careful handling. We can design different aggregation strategies:
157+
- **Replace**: Simply replace overlapping actions with the newer predictions
158+
- **Weighted blend**: Combine overlapping actions using temporal weights (closer actions get higher weight)
159+
160+
161+
This system is highly configurable, as the chunk size threshold can be tuned based on network latency, model inference time, and desired responsiveness.
162+
A lower threshold means more frequent updates (and higher computational cost), while a higher threshold reduces communication overhead at the expense of potential queue starvation.
163+
Lastly, we typically receive actions from `PolicyServer` in a thread, and perform them in another one. This keeps the client listening for incoming chunks in a separate thread, without blocking execution and always consuming the current chunk until a new one becomes fully available.
164+
165+
### Policy Server
166+
167+
Upon receiving observations from the `RobotClient`, the `PolicyServer` receives observations from the `RobotClient`, and performs the necessary observation cleaning to make received observations ready for inference. This process is illustrated in the image below:
168+
<p align="center">
169+
<img width="600" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/async-inference/server_pipeline.png" alt="Server pipeline"/>
170+
</p>
171+
<p align="center"><i>The observation cleaning pipeline</i> running on the server, highlighting the three main steps related to (1) Keys matching (2) Preprocessing and (3) Preparation for inference.
172+
</p>
173+
174+
Once the observation has been prepared, it is compared with the last observation used for inference.
175+
This avoids collapsing into a loop whereby very similar observations are processed, thus triggering unnecessary inference and similar actions being executed (which in turn, result in very similar observations being processed again).
176+
We compare observations in terms of their joint-space similarity, which provides us an approximate and quick way of measuring changes in the robot. Clearly, this metric is not adaptive to dynamic changes in the environment (an object changing its position, or disturbances being applied), but we found it to be a good trade-off for the majority of the cases, and to be very effective in avoiding unnecessary inference and state collapse.
177+
Critically, the `RobotClient` retains control over whether a given observation must be processed, to avoid deadlocks.
178+
Observations sent by the client and tagged with `must_go=True` are processed regardless of the similarity metric.
179+
180+
<p align="center">
181+
<img width="600" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/async-inference/policy_workflow.png" alt="Policy workflow"/>
182+
</p>
183+
<p align="center"><i>The policy workflow</i>, in which incoming observations are compared to the last one used for inference, and processed only if different enough, or `must_go`.
184+
</p>
185+
186+
Lastly, to ensure the `PolicyServer` always processes the latest available observation, we block incoming observations until the previous one has been successfully processed. In this, we leverage queues on the `PolicyServer` to ensure incoming observations are not enqueued until the server is ready to process them (see below).
187+
188+
<p align="center">
189+
<img width="600" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/async-inference/client_pings_server.png" alt="Client pings server"/>
190+
</p>
191+
<p align="center"><i>The client pings the server every 1/fps seconds</i>, but observations are not enqueued for processing until the previous one has been successfully processed.
192+
</p>
193+
194+
---
195+
196+
## 4. Analyzing async inference
197+
198+
For all practical purposes, in async inference there are two time-scales that matter:
199+
200+
* **Environment step** \\(\texttt{environment\_dt} = 1/\texttt{fps}\\), depicting how fast the robot can perform an action.
201+
* **Inference latency** \\(\texttt{inference\_time}\\): forward-pass + network round-trip. We can assume the network round-trip to be negligible with respect to the policy inference time, though this might not be the case for every setup.
202+
203+
Importantly, the ratio
204+
\\(
205+
c = \frac{\texttt{environment\_dt}}{\texttt{inference\_time}}
206+
\\)
207+
results in different behaviours:
208+
209+
* \\(c \ll 1\\): environment evolves faster than inference. In this scenario, the queue empties quickly and we degenerate to sequential control.
210+
* \\(c \ge 1\\): server keeps up. The queue is always (nearly) full.
211+
212+
Critically, \\(c\\) influences the number of available actions in the queue at any given time. To avoid the aforementioned sequential limit control, one can:
213+
1. **Use more compute for the policy server**, hosting the server on a GPU, reducing \\(\texttt{inference\_time}\\) as a consequence of allocating more computational resources.
214+
2. **Sending observations to the server more often**, send a new observation when the queue length \\(k\\) drops below a **fraction** \\(g = k/H\\) of its maximum size.
215+
* \\(g=0\\) reproduces sequential inference (empty queue, wait).
216+
* \\(g=1\\) sends an observation every timestep (max compute, minimal lag).
217+
218+
Experiments (see plots below) show that \\(g\approx0.7\\) offers a good trade-off when observations sent are not filtered out (they are all must-go). We recommend setting \\(g=0.5\\) and following [our documentation](https://huggingface.co/docs/lerobot/en/async) to tune this parameter to your needs.
219+
220+
<p align="center">
221+
<img width="600" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/async-inference/queues.png" alt="Queues"/>
222+
</p>
223+
<p align="center"><i>The number of available actions in the queue at any given time</i>, as a function of g. Larger values of g result in more frequent updates, and more computational cost. Values of g closer to 0 reproduce sequential inference (empty queue, wait). We found g~0.7 to be a good trade-off in our experiments.
224+
</p>
225+
226+
---
227+
228+
## 5. Using async in your setup
229+
230+
Async inference is a simple yet effective way to improve the performance of robotic policies. In our experiments using SmolVLA, async inference results in a ~2x speedup in task completion time with comparable task success rate, and more adaptive control coming from a tighter loop.
231+
232+
To run your policy using async inference, you just need to follow our [tutorial](https://huggingface.co/docs/lerobot/en/async) with your own custom parameters (e.g., the policy path or the chunk size threshold). Async inference comes with support for policies supporting action chunking!
233+
234+
---
235+
236+
## Conclusions
237+
238+
We have introduced async inference, a simple yet effective way to improve the performance of robotic policies. In our experiments using SmolVLA, async inference results in a ~2x speedup in task completion time with comparable task success rate, and more adaptive control coming from a tighter loop.
239+
240+
We are excited to share this work with the community, and to see how it can be used to improve the performance of robotic policies. We welcome PRs to improve and extend the async inference framework at [`huggingface/lerobot`](https://huggingface.co/lerobot), and are available to discuss this further in our [Discord community](https://discord.com/invite/ttk5CV6tUw), 🤗.
241+

0 commit comments

Comments
 (0)