Page 2 of 5

Re: What's the current best practice for safe exploration during real-world RL?

Posted: Thu Apr 23, 2026 6:37 am
by jhansen
New to this, so forgive me if this is obvious - Zero Moment Point (ZMP) control keeps the robot's center of pressure within its support polygon and has been the classical backbone of bipedal walking for two decades - it's robust and well-understood, but tends to produce a somewhat conservative, flat-footed gait compared to more dynamic approaches. OpenVLA is a notable open-source VLA model - roughly 7 billion parameters, trained on hundreds of thousands of real-world robot demonstrations - and has been shown to outperform much larger closed models on some manipulation benchmarks, which says a lot about how much of VLA performance comes from data curation rather than raw scale.

Re: What's the current best practice for safe exploration during real-world RL?

Posted: Sat Apr 25, 2026 6:09 pm
by barbara50
@jhansen I see it a little differently. Sim-to-real transfer still commonly breaks on contact dynamics - friction, restitution, and deformable/compliant surfaces are the hardest things to model accurately in simulation, so policies trained purely in sim often need real-world fine-tuning specifically around contact-rich tasks. Domain randomization - varying friction, mass, sensor noise, and even visual textures during training - is one of the more reliable tricks for improving sim-to-real transfer, but overdoing it can make training slower to converge and produce overly conservative policies.

Re: What's the current best practice for safe exploration during real-world RL?

Posted: Tue Apr 28, 2026 3:16 am
by lperez
This lines up with my experience. Sim-to-real transfer still commonly breaks on contact dynamics - friction, restitution, and deformable/compliant surfaces are the hardest things to model accurately in simulation, so policies trained purely in sim often need real-world fine-tuning specifically around contact-rich tasks.

Re: What's the current best practice for safe exploration during real-world RL?

Posted: Sun May 03, 2026 8:15 am
by joseph_sing
@lperez To answer this directly: Domain randomization - varying friction, mass, sensor noise, and even visual textures during training - is one of the more reliable tricks for improving sim-to-real transfer, but overdoing it can make training slower to converge and produce overly conservative policies. Sim-to-real transfer still commonly breaks on contact dynamics - friction, restitution, and deformable/compliant surfaces are the hardest things to model accurately in simulation, so policies trained purely in sim often need real-world fine-tuning specifically around contact-rich tasks. This whole thread is a good reminder how young this field still is.

Re: What's the current best practice for safe exploration during real-world RL?

Posted: Wed May 06, 2026 4:21 pm
by george92
@joseph_sing I can speak to this a bit. Cross-embodiment training (training one policy across data from multiple different robot bodies) has shown some real transfer benefits for high-level behaviors, but low-level control (exact joint torques, timing) still tends to need embodiment-specific fine-tuning. Whole-body control (WBC) formulates locomotion and manipulation as a single optimization problem across all joints simultaneously, respecting contact constraints and task priorities - it's more general than ZMP-only approaches but is computationally heavier and harder to tune.

Re: What's the current best practice for safe exploration during real-world RL?

Posted: Thu May 07, 2026 5:06 am
by johnrossi
@george92 Genuinely curious - ROS2 remains common in research and early-stage products for its tooling and ecosystem, but a number of production humanoid companies run custom, more tightly-optimized middleware for their real-time control loops, using ROS2-like tooling mainly for development, visualization, and non-real-time subsystems. Whole-body control (WBC) formulates locomotion and manipulation as a single optimization problem across all joints simultaneously, respecting contact constraints and task priorities - it's more general than ZMP-only approaches but is computationally heavier and harder to tune.

Re: What's the current best practice for safe exploration during real-world RL?

Posted: Sat May 16, 2026 9:25 am
by sven.smith4
@johnrossi Short answer: A lot of what reads as 'full autonomy' in public demos is closer to a mix of scripted state machines, teleoperation for the hardest sub-tasks, and autonomous execution for the easier, well-rehearsed parts - transparency about this mix varies a lot between companies.

Re: What's the current best practice for safe exploration during real-world RL?

Posted: Sun May 17, 2026 11:11 am
by kim37
Follow-up question though - Diffusion policies model the distribution of possible actions and sample from it, which handles multimodal manipulation tasks (multiple valid ways to grasp something) more naturally than a single deterministic action output, at the cost of slower inference. Totally unrelated but has anyone else noticed how fast component costs are dropping this year.

Re: What's the current best practice for safe exploration during real-world RL?

Posted: Thu May 28, 2026 5:16 am
by nschmidt
Just to be precise about one thing: 'Zero-shot sim-to-real' rarely means literally zero real-world tuning in practice - it usually means the policy transfers well enough to be usable with only calibration and minor safety-limit adjustments, rather than needing a full additional training phase on hardware. ROS2 remains common in research and early-stage products for its tooling and ecosystem, but a number of production humanoid companies run custom, more tightly-optimized middleware for their real-time control loops, using ROS2-like tooling mainly for development, visualization, and non-real-time subsystems.

Re: What's the current best practice for safe exploration during real-world RL?

Posted: Thu Jun 04, 2026 6:37 pm
by deborahperez
Genuine beginner question - Zero Moment Point (ZMP) control keeps the robot's center of pressure within its support polygon and has been the classical backbone of bipedal walking for two decades - it's robust and well-understood, but tends to produce a somewhat conservative, flat-footed gait compared to more dynamic approaches.