Page 2 of 2

Re: VLA models explained for people who aren't ML researchers

Posted: Fri Oct 11, 2024 5:28 pm
by chenperez
Worth being a little skeptical of the marketing angle here. Vision-Language-Action (VLA) models like RT-2, OpenVLA, and Physical Intelligence's pi0 unify a vision-language backbone with an action-output head, letting a robot map a camera image and a text instruction directly to motor commands instead of hand-coding separate perception and planning stages.

Re: VLA models explained for people who aren't ML researchers

Posted: Sat Oct 12, 2024 5:47 am
by dubois35
This is a great summary, thanks. Sim-to-real transfer still commonly breaks on contact dynamics - friction, restitution, and deformable/compliant surfaces are the hardest things to model accurately in simulation, so policies trained purely in sim often need real-world fine-tuning specifically around contact-rich tasks.