How This Robotic Leg Learned to Walk By Itself | News & Opinion

(Photo: Matthew Lin)

Most robots have to be programmed to perform specific, repetitive tasks—from making coffee and pizza to giving high fives. But what if robots could improve over time, like toddlers who turn unsteady steps into confident sprints.

That’s the goal of two USC researchers who built a 3-tendon, 2-joint robotic limb that can teach itself how to move through trial and error. “We want to reverse-engineer brains and bodies and create awesome robots,” says Dr. Francisco Valero-Cuevas, professor of Biomedical Engineering and professor of Biokinesiology & Physical Therapy.

He and USC Viterbi School of Engineering doctoral student Ali Marjaninejad just published a paper on their work, which made the cover of the March issue of Nature Machine Intelligence. We spoke to them ahead of its publication; here are edited and condensed excerpts of our conversation.

Can you explain how your new robotic limb ‘learns’ to walk?
[Ali Marjaninejad] We call this algorithm General-to Particular or G2P because we begin by letting the system play at random to internalize the general properties of the leg, like children [learning to walk]. We then give it a reward every time it approaches good performance of a given task. In this case, moving the treadmill forward. This is called reinforcement learning as it is similar to the way animals respond to positive reinforcement.

Robotic limb in pre-test mode (Photo: Matthew Lin)

Talk us through the experiment, which uses something called motor babbling, like newborn ponies who ‘figure out’ how to run ASAP to avoid predators?
[AM] The process is two-step: first babble and then perform. But in more detail, this has interesting consequences. First, it allows fast learning of good-enough solutions—like ponies who need to walk ASAP. At another level, the motor babbling is similar to how animals train lower parts of the nervous system such as the spinal cord, which is what controls muscles directly. So the babbling allows the creation of a pre-tuned system that a “high-level” controller can then use—like how your brain uses your spinal cord to control your body. If you combine these two, then the robot will learn to walk quickly, even if not very well. Subsequently, the algorithm will continue to refine how to exploit the complex dynamics of the system. It will continue to learn to improve its performance from every time it does the task, just like you and I do.

And this is significantly different to current robotic controllers?
[AM] Yes. This is in contrast to how robots are controlled today, mostly relying on exact equations, sophisticated computer simulations, or thousands of repetitions to refine a task. Nature does not have this luxury of time; animals need to learn quickly to do things well enough to live another day.

Ali Marjaninejad and Dr. Valero-Cuevas set up a robotic limb test (Photo: Matthew Lin)

How does the limb create a ‘neural network’ so it knows how to move?
[AM] An internal part of the algorithm is that the learning is encoded as the continual training of a simple 3-layer neural network. During the babbling phase, the system will send random commands to motors and sense the joint angles. Then, it will train the 3-layer neural network to guess what commands will produce a given movement. We then start performing the task and reinforce good behavior. Even then, the motor commands and joint angles from every time it does the task—i.e., its “experience”—will be used to refine the weights of this neural network.

This limb has tendons too, right? Just like humans?
[Dr. Francisco Valero-Cuevas] Indeed. Most robotics systems control the joints of legs and arms with a motor that rotates them directly. Nature does not have that luxury, as it has muscles that must act at a distance via tendons. Therefore, the physics, mechanics, and mathematics of biological limbs are fundamentally different from those in traditional robots. We therefore want to explore what it is about these “complex” anatomies that allows versatility and agility in animals, which we would love our robots to have. We chose to control such bio-inspired legs because it is only then that we confront the actual problems brains confront to produce locomotion.

Is it harder to build?
[FVC] Yes, and using 3D printing simplifies this very much. However, even then some would say that we are complicating our lives unnecessarily by using these tendon-driven limbs. We use this approach, however, because our guiding philosophy is to understand how animals can be so successful even when facing two unforgiving arbiters of success: Newton and Darwin. You and your offspring live to fight another day only if you can use tendon-driven limbs to walk (Newtonian mechanics), while using as few trial-and-error attempts that find good-enough solutions to beat your competitors at that moment in time (Darwinian evolution). Solving these problems will also help us understand how and why animals, even with such “complicated” bodies, are much more versatile than today’s robots.

Talk us through the engineering challenges. Did you 3D print all parts in a fabrication lab at USC?
[AM] Thanks for the opportunity to talk shop. We designed the legs in-house based on the concepts in the book Fundamentals of Neuromechanics. The real challenge began by how to build these systems while also using off-the-shelf motors and electronics so that we could focus on the algorithm. So we 3D-printed the parts and used simple data acquisition systems that also provide research-grade accuracy and sampling rates to enable an effective use of G2P. All told, we could build this over a few months with a low budget. This allowed us to also quickly build our own mini-treadmill, which was also 3D-printed. We then were able to quickly focus on developing the G2P algorithm.

In the spirit of openness, are you sharing your code?
[FVC] Yes, and gladly. The best place to start is ValeroLab.org/G2P, where you will find links to the GitHub repository for the source code, run data for experiments, as well as files for you to 3D print your own system. Science has now moved to more transparency so that others can replicate what you claim you did. But also, given the low cost of our systems, we want to transform how roboticists around the world work by allowing them to use our results. This empowers the field as a whole, as well as young scientists around the world who may not have access to research-grade infrastructure, but who can benefit from our work to move the field forward.

This research is funded by NIH, DARPA, and the DoD. Will your limbs be used on future battlescape robots or to enable wounded warriors to walk again?
[FVC] Our emphasis and philosophy is to understand how brains and bodies collaborate so we can help people with disabilities by creating better assistive robots, prostheses, exoskeletons, and other assistive technologies. This may lead to a class of robots with unique advantages in design, versatility, and performance to help wounded warriors and civilians to walk or use their arms again. Consider the huge problem of millions of landmines scattered around the world that put warriors and civilians at huge risk of disability. Our open-source approach can also help develop low-cost technology for those victims of war.

Essentially can your robotic limb think for itself?
[FVC] In a way. Programmers can predict and code for multiple scenarios, but not for every possible scenario. This makes pre-programmed robots prone to failure. However, we let our robots learn from “personal” experience that is tied to their particular environment and tasks they are exposed to. They will then eventually find a solution within a short time, and will continue to be refined and adapted as they continue to work. I would not say a robots will “thinking for itself,” but it is definitely “learning by itself” and even share its knowledge with other robots to create a “culture.”

Finally, do you hope to see your robotic limbs deployed on future space missions?
[FVC] Space exploration and rescue missions are definitely a critical frontier because robots are technically in “hazardous” or time-delayed environments. Robots need to do what needs to be done without being escorted or supervised as they land into a new planet from which it takes 30 minutes to send/receive a signal, or venture into uncertain and dangerous terrain in the wake of natural disasters. These robots would in principle be able to adapt to walking on loose rocks one day and mud after it rains on Earth, or in space adapt to the low or high gravity of moons versus planets, hard ground versus windblown sand around craters, etc. They would be our very best ambassadors and scouts to help people and expand our knowledge.