Page 3 of 6

Re: How much does perception latency budget actually constrain controller design choices?

Posted: Thu Apr 09, 2026 9:58 pm
by servoken70
@charlesbianchi Tangent, but worth mentioning: Zero Moment Point (ZMP) control keeps the robot's center of pressure within its support polygon and has been the classical backbone of bipedal walking for two decades - it's robust and well-understood, but tends to produce a somewhat conservative, flat-footed gait compared to more dynamic approaches.

Re: How much does perception latency budget actually constrain controller design choices?

Posted: Thu Apr 16, 2026 2:41 am
by nicole57
This matches something I went through recently. Model predictive control (MPC) is still very much alive in production humanoids, often working alongside or underneath learned policies - MPC handles short-horizon dynamically-consistent trajectory optimization while learned components handle perception, task-level decisions, or recovery behaviors that are hard to hand-model. Isaac Lab (the successor to Isaac Gym) is widely used for large-scale parallel RL training thanks to GPU-accelerated physics, while MuJoCo is often used as a secondary 'sim-to-sim' validation step because its contact dynamics are generally considered more realistic than Isaac's, even though it trains slower at scale.

Re: How much does perception latency budget actually constrain controller design choices?

Posted: Sun Apr 26, 2026 1:49 am
by forgecam45
@nicole57 Just to be precise about one thing: Physical Intelligence's pi0 pairs a smaller pretrained vision-language backbone with a separate flow-matching 'action expert' module, which is one way to get fast, high-frequency action output without needing the whole giant language model to run at control-loop speed.

Re: How much does perception latency budget actually constrain controller design choices?

Posted: Tue May 05, 2026 7:27 am
by george92
@forgecam45 I'd push back on this a bit. A lot of what reads as 'full autonomy' in public demos is closer to a mix of scripted state machines, teleoperation for the hardest sub-tasks, and autonomous execution for the easier, well-rehearsed parts - transparency about this mix varies a lot between companies. Domain randomization - varying friction, mass, sensor noise, and even visual textures during training - is one of the more reliable tricks for improving sim-to-real transfer, but overdoing it can make training slower to converge and produce overly conservative policies. Kind of makes me think about how different this all looked even three years ago.

Re: How much does perception latency budget actually constrain controller design choices?

Posted: Fri May 08, 2026 7:26 am
by erik_novi
@george92 Not to derail, but this reminds me of something adjacent: Zero Moment Point (ZMP) control keeps the robot's center of pressure within its support polygon and has been the classical backbone of bipedal walking for two decades - it's robust and well-understood, but tends to produce a somewhat conservative, flat-footed gait compared to more dynamic approaches.

Re: How much does perception latency budget actually constrain controller design choices?

Posted: Sat May 09, 2026 9:55 pm
by scott.andersson5
One nitpick - 'Zero-shot sim-to-real' rarely means literally zero real-world tuning in practice - it usually means the policy transfers well enough to be usable with only calibration and minor safety-limit adjustments, rather than needing a full additional training phase on hardware.

Re: How much does perception latency budget actually constrain controller design choices?

Posted: Sat May 16, 2026 8:21 pm
by thomasmitchell
@scott.andersson5 Slight correction, though the overall point stands: 'Zero-shot sim-to-real' rarely means literally zero real-world tuning in practice - it usually means the policy transfers well enough to be usable with only calibration and minor safety-limit adjustments, rather than needing a full additional training phase on hardware.

Re: How much does perception latency budget actually constrain controller design choices?

Posted: Mon May 18, 2026 11:34 pm
by garcia51
From hands-on experience, A lot of what reads as 'full autonomy' in public demos is closer to a mix of scripted state machines, teleoperation for the hardest sub-tasks, and autonomous execution for the easier, well-rehearsed parts - transparency about this mix varies a lot between companies.

Re: How much does perception latency budget actually constrain controller design choices?

Posted: Sat May 30, 2026 9:27 am
by karen_kim
@garcia51 This matches what I've seen too. 'Zero-shot sim-to-real' rarely means literally zero real-world tuning in practice - it usually means the policy transfers well enough to be usable with only calibration and minor safety-limit adjustments, rather than needing a full additional training phase on hardware.

Re: How much does perception latency budget actually constrain controller design choices?

Posted: Tue Jun 09, 2026 5:11 am
by lbianchi
This matches something I went through recently. Physical Intelligence's pi0 pairs a smaller pretrained vision-language backbone with a separate flow-matching 'action expert' module, which is one way to get fast, high-frequency action output without needing the whole giant language model to run at control-loop speed. This whole thread is a good reminder how young this field still is.