Async Retargeter (TeleopSession level) API

[jiwenc-nv]

Background

Isaac Teleop currently exposes only a sync retargeter API (scoped that way for MVP). As retargeters get heavier, downstream consumers (Lab, Arena, Sim) are blocking on every call and hitting real performance issues. Surfaced first in Lab, but it's an IT API limitation, not a Lab-specific problem.

Proposal

Add a first-class async retargeter API in Isaac Teleop, owned by Isaac Teleop and consumed uniformly across Lab, Arena, and Sim.

• Define an async interface alongside the existing sync one (sync stays for backward compat).
• Provide async implementations for built-in retargeters.
• Document the concurrency model (ordering, cancellation, backpressure).
• Migrate Isaac Lab as the validation consumer.

Doing this in Lab would force every other stack to re-implement or depend on Lab. Keeping it in Isaac Teleop puts the abstraction in the right layer.

API

apps (e.g. isaac-lab) can kick of session.async_step(), and have a separate session.sample() call

Open questions

• The operation modes:

  • Overall retargeting runs slow -> you need to run async. We can design an async API that runs in the background, but this should really be trivial for the consumer to do
  • Retargeting graph is slow because it is serial -> we need to execute retargeters in parallel, doable but opens up a can of scheduling worms

• Async primitive: asyncio, futures, callbacks?

• Cancellation when a newer pose arrives mid-retarget?

• Backpressure: drop-stale vs. queue (sim stepping vs. live teleop may differ).

Stakeholders

• @hougantc-nvda or @aristarkhovNV — potential owners
• @nvddr — FYI (on other critical path, not driving this)