Wired wrote about robots that can learn a task on the spot. That framing has appeared before and mostly not survived contact with reality, so it is worth understanding what actually changed in how robots are taught, and what remains genuinely hard.
The old way: describe every motion
Traditional industrial robots are programmed explicitly. An engineer specifies positions, paths and forces, and the robot repeats them with great precision.
This works superbly when the world is controlled. A car factory positions each part identically every time, so the robot never needs to perceive anything.
It collapses immediately outside those conditions. A kitchen contains objects in arbitrary positions, of unknown weight, that deform when gripped. You cannot enumerate the cases, which is why robots have been in factories for decades and not in homes.
Imitation learning
The approach behind most recent progress is to show rather than tell.
A human teleoperates the robot through a task dozens or hundreds of times. Each demonstration records what the cameras saw and what the joints did. A neural network then learns the mapping from observation to action.
The robot is not following a plan. It is predicting, moment to moment, what a demonstrator would have done given this view, which is why the resulting motion looks noticeably more human and slightly hesitant.
The significant recent shift is that these models transfer. A network trained across many tasks and many robots picks up new tasks from far fewer demonstrations, because it has already learned general things about objects and contact. That is what “learning on the spot” usually refers to: adaptation on top of extensive prior training, not learning from nothing.
Why simulation only half works
Training in simulation is attractive because you can run millions of attempts in parallel with no broken hardware.
The obstacle is the reality gap. Simulators model rigid-body dynamics well and model friction, deformation, cloth and granular materials poorly. A policy trained in simulation frequently fails on a real table because the real friction differs slightly.
The standard workaround is domain randomisation: vary lighting, textures, masses and friction wildly during training so the policy cannot depend on any of them. It helps considerably for locomotion and coarse manipulation, and much less for fine contact.
The genuinely hard part
Perception has largely been solved by the same advances that transformed image recognition. Robots identify and locate objects well.
Manipulation has not been solved. Specifically, tasks involving contact, force and deformation remain difficult: folding a towel, plugging in a cable, opening a bag, handling anything soft.
This is Moravec’s paradox, observed decades ago and still holding. Tasks humans find intellectually demanding, such as chess, turned out to be easy for machines. Tasks a two-year-old performs without thought, such as picking up a sock, turned out to be extraordinarily hard, because the underlying competence is ancient, unconscious and never written down.
Reading the demonstrations sceptically
Robot videos are easy to misread. A few questions separate a real capability from a rehearsal.
- Is it teleoperated? Impressive dexterity frequently has a human in the loop. This should be disclosed and often is not prominent.
- Is it real time? Speeded-up footage hides long pauses for planning.
- How many attempts? A single success from many tries is a very different claim.
- Was the scene arranged? Objects placed in known positions on a clear surface is closer to the factory case than the kitchen case.
- Does it generalise? The meaningful test is a different object, in a different place, in different light.
The progress is real and the trajectory is genuinely different from previous cycles. The reasonable expectation is competent robots in structured commercial settings, warehouses and back-of-house, well before anything useful in an ordinary untidy home.
Join the discussion