Purely information based tasks have mountains of training material. Physical world interactions, less so and harder to instrument.

Driving is one of the more favorable scenarios, vision and hearing are the only sensory inputs and the only operation is a wheel and two pedals. Easy to instrument and passively capture millions and millions of miles worth of human driving to imitate.

But in other tasks, people are largely not instrumented.. Meta hopes everyone will wear their AI glasses which could boost passive data capture, but even then they are not capturing sense of touch and human dexterity. Some are paying people to try to tele operate machines, but that frequently goes embarrassingly badly because it is really hard for a person to navigate things without touch.

If you put on mittens and a task becomes impossible, well you still have way more to work with than the models can access training data for.