What's a realistic timeline for language-conditioned tasks becoming truly reliable?

[niklassantos]

@scott.andersson5 I'd frame this differently. Isaac Lab (the successor to Isaac Gym) is widely used for large-scale parallel RL training thanks to GPU-accelerated physics, while MuJoCo is often used as a secondary 'sim-to-sim' validation step because its contact dynamics are generally considered more realistic than Isaac's, even though it trains slower at scale. Balance-recovery controllers are usually evaluated with push-recovery tests (a known, repeatable lateral push) in demos, but real-world robustness also depends on recovering from unstructured events like uneven flooring, unexpected contact, or a dropped payload shifting the center of mass mid-stride - which is a much harder, less demo-friendly test.

[jhansen]

From hands-on experience, 'Zero-shot sim-to-real' rarely means literally zero real-world tuning in practice - it usually means the policy transfers well enough to be usable with only calibration and minor safety-limit adjustments, rather than needing a full additional training phase on hardware.

[thomas65]

@jhansen Worth being a little skeptical of the marketing angle here. Diffusion policies model the distribution of possible actions and sample from it, which handles multimodal manipulation tasks (multiple valid ways to grasp something) more naturally than a single deterministic action output, at the cost of slower inference. Vision-Language-Action (VLA) models like RT-2, OpenVLA, and Physical Intelligence's pi0 unify a vision-language backbone with an action-output head, letting a robot map a camera image and a text instruction directly to motor commands instead of hand-coding separate perception and planning stages.

[zoeanderson]

@thomas65 Slight correction, though the overall point stands: Physical Intelligence's pi0 pairs a smaller pretrained vision-language backbone with a separate flow-matching 'action expert' module, which is one way to get fast, high-frequency action output without needing the whole giant language model to run at control-loop speed. ROS2 remains common in research and early-stage products for its tooling and ecosystem, but a number of production humanoid companies run custom, more tightly-optimized middleware for their real-time control loops, using ROS2-like tooling mainly for development, visualization, and non-real-time subsystems.

[aliu]

@zoeanderson From hands-on experience, Vision-Language-Action (VLA) models like RT-2, OpenVLA, and Physical Intelligence's pi0 unify a vision-language backbone with an action-output head, letting a robot map a camera image and a text instruction directly to motor commands instead of hand-coding separate perception and planning stages.

[ethan.lewis5]

Slight correction, though the overall point stands: Diffusion policies model the distribution of possible actions and sample from it, which handles multimodal manipulation tasks (multiple valid ways to grasp something) more naturally than a single deterministic action output, at the cost of slower inference.

[aliu]

@ethan.lewis5 Speaking from personal experience here, Sim-to-real transfer still commonly breaks on contact dynamics - friction, restitution, and deformable/compliant surfaces are the hardest things to model accurately in simulation, so policies trained purely in sim often need real-world fine-tuning specifically around contact-rich tasks.

[scott.andersson5]

I can speak to this a bit. Model predictive control (MPC) is still very much alive in production humanoids, often working alongside or underneath learned policies - MPC handles short-horizon dynamically-consistent trajectory optimization while learned components handle perception, task-level decisions, or recovery behaviors that are hard to hand-model.

[giulia.roberts4]

@scott.andersson5 I'd take that specific number with a grain of salt, honestly. OpenVLA is a notable open-source VLA model - roughly 7 billion parameters, trained on hundreds of thousands of real-world robot demonstrations - and has been shown to outperform much larger closed models on some manipulation benchmarks, which says a lot about how much of VLA performance comes from data curation rather than raw scale. Model predictive control (MPC) is still very much alive in production humanoids, often working alongside or underneath learned policies - MPC handles short-horizon dynamically-consistent trajectory optimization while learned components handle perception, task-level decisions, or recovery behaviors that are hard to hand-model.

[sharonschmidt]

@giulia.roberts4 That's the official framing, at least - reality tends to lag a bit. OpenVLA is a notable open-source VLA model - roughly 7 billion parameters, trained on hundreds of thousands of real-world robot demonstrations - and has been shown to outperform much larger closed models on some manipulation benchmarks, which says a lot about how much of VLA performance comes from data curation rather than raw scale.