Page 2 of 2
Re: Anyone using language models purely for high-level task planning, not low-level control?
Posted: Sat Feb 08, 2025 5:01 pm
by nicole57
Minor factual note:
Cross-embodiment training (training one policy across data from multiple different robot bodies) has shown some real transfer benefits for high-level behaviors, but low-level control (exact joint torques, timing) still tends to need embodiment-specific fine-tuning.
Re: Anyone using language models purely for high-level task planning, not low-level control?
Posted: Sun Feb 09, 2025 7:54 am
by chloe_jack
From hands-on experience,
Balance-recovery controllers are usually evaluated with push-recovery tests (a known, repeatable lateral push) in demos, but real-world robustness also depends on recovering from unstructured events like uneven flooring, unexpected contact, or a dropped payload shifting the center of mass mid-stride - which is a much harder, less demo-friendly test.
Re: Anyone using language models purely for high-level task planning, not low-level control?
Posted: Mon Feb 10, 2025 12:11 pm
by kwilliams
Speaking from personal experience here,
Sim-to-real transfer still commonly breaks on contact dynamics - friction, restitution, and deformable/compliant surfaces are the hardest things to model accurately in simulation, so policies trained purely in sim often need real-world fine-tuning specifically around contact-rich tasks.
Re: Anyone using language models purely for high-level task planning, not low-level control?
Posted: Tue Feb 18, 2025 10:17 pm
by nicole57
@kwilliams One nitpick -
OpenVLA is a notable open-source VLA model - roughly 7 billion parameters, trained on hundreds of thousands of real-world robot demonstrations - and has been shown to outperform much larger closed models on some manipulation benchmarks, which says a lot about how much of VLA performance comes from data curation rather than raw scale.
Re: Anyone using language models purely for high-level task planning, not low-level control?
Posted: Wed Feb 26, 2025 8:35 pm
by erik_novi
@nicole57 Follow-up question though -
Physical Intelligence's pi0 pairs a smaller pretrained vision-language backbone with a separate flow-matching 'action expert' module, which is one way to get fast, high-frequency action output without needing the whole giant language model to run at control-loop speed. Whole-body control (WBC) formulates locomotion and manipulation as a single optimization problem across all joints simultaneously, respecting contact constraints and task priorities - it's more general than ZMP-only approaches but is computationally heavier and harder to tune.
Totally unrelated but has anyone else noticed how fast component costs are dropping this year.