How to control robots with brainwaves and hand gestures

Getting robots to do things isn’t easy: Usually, scientists have to either explicitly program them or get them to understand how humans communicate via language.

But what if we could control robots more intuitively, using just hand gestures and brainwaves?

A new system spearheaded by researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) aims to do exactly that, allowing users to instantly correct robot mistakes with nothing more than brain signals and the flick of a finger.

Building off the team’s past work focused on simple binary-choice activities, the new work expands the scope to multiple-choice tasks, opening up new possibilities for how human workers could manage teams of robots.

By monitoring brain activity, the system can detect in real-time if a person notices an error as a robot does a task. Using an interface that measures muscle activity, the person can then make hand gestures to scroll through and select the correct option for the robot to execute.

The team demonstrated the system on a task in which a robot moves a power drill to one of three possible targets on the body of a mock plane. Importantly, they showed that the system works on people it’s never seen before, meaning that organizations could deploy it in real-world settings without needing to train it on users.

This work combining EEG and EMG feedback enables natural human-robot interactions for a broader set of applications than we've been able to do before using only EEG feedback. By including muscle feedback, the gestures can be udes to command the robot spatially, with much more nuance and specificity.”

In most previous work, systems could generally only recognize brain signals when people trained themselves to “think” in very specific but arbitrary ways and when the system was trained on such signals. For instance, a human operator might have to look at different light displays that correspond to different robot tasks during a training session.

Not surprisingly, such approaches are difficult for people to handle reliably, especially if they work in fields like construction or navigation that already require intense concentration.

Meanwhile, the team harnessed the power of brain signals called “error-related potentials” (ErrPs), which researchers have found to naturally occur when people notice mistakes. If there’s an ErrP, the system stops so the user can correct it; if not, it carries on.

For the project the team used “Baxter,” a humanoid robot from Rethink Robotics. With human supervision, the robot went from choosing the correct target 70 percent of the time to more than 97 percent of the time.

To create the system the team harnessed the power of electroencephalography (EEG) for brain activity and electromyography (EMG) for muscle activity, putting a series of electrodes on the users’ scalp and forearm.

Both metrics have some individual shortcomings: EEG signals are not always reliably detectable, while EMG signals can sometimes be difficult to map to motions that are any more specific than “move left or right.” Merging the two, however, allows for more robust bio-sensing and makes it possible for the system to work on new users without training.

The team says that they could imagine the system one day being useful for the elderly, or workers with language disorders or limited mobility.

Trends in Robot Accessories

The right accessories can boost the flexibility and reliability of robotic assembly operations.

The efficiency and reliability of an industrial robot largely depend on its end-of-arm tooling (EOAT). However, we often forget that the interface between the end-effector and the robot arm also plays a critical role. Devices, such as tool changers, compensation units, and force and torque sensors, have a great influence on the robot’s performance, flexibility and fields of application.

Tool changers lend speed and flexibility to a robotic assembly line.Even an experienced operator requires 10 to 30 minutes to manually change the EOAT on a robot. With a tool changer, the same operation takes less than 10 to 30 seconds.

A tool changer consists of two components: a quick-change head, which is mounted on the robot arm, and a quick-change adapter, which is connected to the tool. The tool changer provides feed-throughs for electric, pneumatic and hydraulic connections between the arm and the EOAT. During a tool change, both components are automatically or manually coupled. When selecting a tool changer, engineers should look for a device with a low weight-to-force ratio, short change times, and precision-made energy-transfer points.

Integrating micro valves into a quick-change system can replace a complete valve terminal. At every cycle, the cylinder of the actuator will fill with compressed air, which saves a lot of time. If you are using a feeding hose that’s 3 meters long and 4 millimeters in diameter, micro valves will reduce air consumption by 90 percent. Instead of cable and wire bundles, just one pneumatic line for compressed air and power supply is required. Both lines can usually be fed through a center bore with a diameter of 12 millimeters, and they can be integrated into the arm of a SCARA robot.

To avoid problems during robotic insertion operations, compensation units provide some compliance between the end-effector and the robot arm. This prevents damage to the assembly or the EAOT and increases the reliability of the process. The latest compensation devices work without pneumatics. Compliance in two directions is adjusted via springs with adjustment screws. Compliance in three directions is adjusted via elastomeric elements. Because the units work without pneumatics, they are flat and particularly suitable for use in confined areas. Smooth-running roller guides compensate for smaller forces without stick-slip effects.

The latest trend in robotics is to equip the robot with force and torque sensors. Such sensors give the robot the ability to precisely adjust its path based on feedback. This is critical for obtaining consistent results in robotic grinding and finishing applications.

The Exoskeletons of the Future Might Just Be Comfortable Pants

When the term “exoskeleton” comes to mind, it’s easy to think about science fiction or bulky military enhancements. The idea of military driven by men with the strength of robots has conjured up dreams for decades. But what if the future of exoskeletons isn’t Iron Man or Master Chief, but grandma maintaining her self-reliance by getting some help with simple acts like walking up the stairs?

The soft exosuit uses a combination of sensors, including a hyper elastic strain sensor (1) and sensors around the wearer’s hip, calf and ankle (2)-(5), all secured by straps. Flexible membranes cover sensors and straps (6) 

“Soft” exoskeletons, developed by researchers at the Wyss Institute at Harvard University, certainly have military applications.  But these exoskeletons are pants. Comfortable pants. These pants would have a waist belt, two thigh pieces, and two calf straps. Cables are attached through to a motor placed on the wearer’s back in a backpack. The thigh pants monitor the human gait, which the researchers place into “human kinematics,” the study of the motion of multi-joint systems. The sensors use electro-myography, a procedure used to assess the health of muscles and the nerve cells that control them, and to determine the parts of the body that need the most assistance.

Participants in the experiments carried thirty percent of their body weight at three miles per hour on a treadmill, slightly faster than the average human walking speed of two and a half. They would walk normally, walk with the weight, and then walk with the weight and the exosuit. The efforts seemed to put a strain on the body that without the suit would be manageable but still noticeable. The results were also noticeable. Measuring the metabolism and strain of each participant, the researchers determined there was “a significant metabolic power reduction” while wearing the exosuit. The participants’ hips, knees, and ankles were also all working significantly less while the exosuit was on.

This wasn’t the Wyss Institute’s first foray into robotic wearable, they’ve also worked with exo gloves. And while their work has been put into a military milieu, it’s clear that the Institute has an eye towards commercial sales. “A key feature of exosuits is that if the actuated segments are extended the suit length can increase so that the entire suit is slack, at which point wearing an exosuit feels like wearing a pair of pants and does not restrict the wearer whatsoever.”

Meet the Tiny Origami Robots That Will Unfold in Your Stomach

Science often has a way of catching up with science fiction. While it never claimed the fan fervour of Star Trek, the 1966 movie Fantastic Voyage offered a tantalizing promise: shrinking scientists down so they could extract information from a human body from the inside. MIT scientists haven’t gotten the shrinking figured out, but by making ingestible microscopic robots that can enter human organs, they’ve managed to get at least half of it down. The robots are based on the concept of origami. You take a capsule, and the robot unfolds itself into a more functional form. MIT first announced the development of these origami robots in 2014, with the hopes of making robots as small as possible. At the time, Daniela Rus, Andrew and Erna Viterbi Professor of Electrical Engineering and Computer Science at MIT, talked about how gaining control of the geometry of the robots was crucial to the project: “We can do the sequencing, we have a lot more control,” she said.

The construction of the robot 

That control is on full display in the latest developments from Rus and her team. “In our calculation,” says Shuhei Miyashita, a lecturer in electronics at the University of Yorkwith whom work Rus worked, “twenty percent of forward motion is by propelling water—thrust—and eighty percent is stick-slip motion,” a method of locomotion where the robot sticks to a surface through friction when it moves and slips free when it changes its weight distribution. “In this regard, we actively introduced and applied the concept and characteristics of the fin to the body design, which you can see in the relatively flat design.” Once inside the body, the robot unfolds to a rectangular design with accordion folds. Crucial to the robot’s success inside the body is the magnet at its core. Outside magnetic forces are used to steer the robot through intestines, and the magnet was also critical to the robot’s success in its test mission.

The size is small enough to be easily ingested 

The team got pig intestines from Boston’s Chinatown and used them to construct a synthetic stomach, in which they placed a battery. It was a realistic situation; there were a reported 11,940 incidents between 2005 and 2014 of children under six swallowing batteries. After navigating the robot with outside magnetic fields, the robot used its central magnet to grab the battery. It’s a system which is very reliant on outside observation, which is what Rus and her team are hoping to work on next. They are working on adding sensors to the robot, redesign the robot so it can control itself. Watch the video. So while the chances of humans shrinking down to get things out of your body are slim, at least something with physical autonomy could be pulling batteries out some time soon.

MIT Device Makes You a Superhuman With a Third Hand

What if instead of one hand on each arm, you had two? That’s what Sang-won Leigh is trying to achieve with his Robotic Symbiotic wearable that he developed as a project for MIT’s Fluid Interfaces course. The robotic hand (described as a “body integrated programmable joints interface”) is equipped with 11 motors that can be rearranged and reprogrammed to satisfy different uses. The device can serve as a large extra finger or as an entire hand. It can sit below your wrist and clamp onto things to hold them still, or pick up objects while your fleshy human hand is left free to, I don’t know, play with your phone.

The robotic hand is controlled separately from your real hand 

The neat thing about Leigh’s robotic hand is that the machine is controlled separately from your real hand. You can hold your hand completely still and move the robotic appendage, and vice versa. How? The device senses electrical signals sent to the muscle in your forearm. This muscle is not used to move your hand, but after practicing for a few hours, you can flex and move it in ways that will cause the robot hand to respond. 

You can also use the device as a joystick or trigger, actually manipulating it with your hand to control a computer interface or play a video game. While there certainly could be applications for people who have disabilities or were injured in combat, Leigh has something else in mind for his human-augmentation device. “A lot of people think about machine augmentation in terms of rehabilitation,” Leigh says, “But we envisioned it as an assistive technology that wasn’t just for people with challenges, but which could turn people with normal physiology into super humans.”

IBM’s Librarian Bot Swaps Data Tapes Like a Madman

There’s no such thing as the cloud, the saying goes—there’s just other people’s computers. How intensely data-intense medium-sized companies—says, banks or insurance companies—can handle huge volumes of data on-site using an IBM system called TS 4500, which comes with a handy robot to fetch files for you.

Huge data storage enabled by one plucky machine

         The tape library in the data storage unit is meant to maximize the amount of 

1s and 0s a company can store in a small space, which is especially useful for organizations that must keep years of older data around. The IBM specs can house more than 16 petabytes (at 3:1 compression) in a 10-by-10-foot space and expand as needed. That’s at least 16 million gigs to organize and protect in roughly the area of a freight elevator, with all that hard storage existing on perhaps tens of thousands of tapes. It’s a job for the robots.

This Tactile Robot Diver Lets Explorers Feel Around Shipwrecks

Stanford engineers have built a robot scuba diver that’s just the sort of sturdy, sensitive sort that you need if you want to pick your way through a coral reef or a decaying shipwreck. Named Ocean One, the robot commands some of the most human like qualities of any remote-controlled machine designed to withstand ocean pressures at depths of hundreds of meters.

Ocean One transmits sensations back to its operator for delicate explorations 

The robot’s key traits: OceanOne is roughly person-sized (about 5 feet long), has binocular vision in a head that looks like Mega Man’s, and trawls the deep with the use of two arms and hands that send back haptic data to its operator. This allows a person controlling Ocean One with joysticks to feel, roughly, the weight of the objects the robot’s picking up. The computer scientist leading this project, Oussama Khatib, uses the word “avatar” to describe the relationship between the operator and Ocean One. Its seems apt, and it opens up a different frontier, perhaps, in how we explore the deep. More than a mere drone (no offense, Sentry) or a chunky human-occupied pod Ocean One is nimble, tactile, and seemingly intuitive. It can comb a fragile environment, in tight confines, and report its findings directly into the hands of its guide. Khatib told Stanford’s news service: “You can feel exactly what the robot is doing. It’s almost like you are there; with the sense of touch you create a new dimension of perception.”

 Its maiden voyage into the live ocean was to prowl a La Lune, a frigate of Louis XIV’s that sank off the southern coast of France in 1664 and was rediscovered only in 1993, some 300 feet below the surface. French underwater archaeologists have been probing the site for years, but it’s still striking to see OceanOne hustle down and retrieve a vase, with Khatib at the controls. No wonder the scientist hopes to build a small team of these robots in the near future. Watch the Ocean One.