How Robots Use Reward Shaping to Learn to Walk

Small changes to what a learning algorithm is rewarded for can produce dramatically different walking styles.

Reward shaping is the practice of designing a reinforcement learning reward function carefully, term by term, to guide a policy toward the kind of behavior an engineer actually wants — not just any behavior that technically maximizes reward.

A concrete example

A pure 'maximize forward distance' reward might produce a policy that learns to fall forward repeatedly, technically covering ground quickly while looking nothing like walking. Adding penalty terms for excessive joint torque, unnatural joint angles, and falling teaches the policy to find solutions that are both effective and stable.

Why it's an iterative process

Engineers typically train, observe the resulting gait, notice some undesirable quirk (a strange arm swing, an inefficient shuffle), adjust the reward function to discourage it, and retrain — often through many such cycles before arriving at a gait that's both robust and looks natural.