Researchers at Cornell University in New York have developed a robotic feeding system that uses computer vision, machine learning and multimodal sensing to safely feed people with severe mobility limitations, including those with spinal cord injuries, cerebral palsy and multiple sclerosis.
‘Feeding individuals with severe mobility limitations with a robot is difficult, as many cannot lean forward and require food to be placed directly inside their mouths,’ said Tapomayukh ‘Tapo’ Bhattacharjee, assistant professor of computer science in the Cornell Ann S Bowers College of Computing and Information Science and senior developer behind the system. ‘The challenge intensifies when feeding individuals with additional complex medical conditions.’
Advertisement
A leader in assistive robotics, Bhattacharjee and his EmPRISE Lab have spent years teaching machines the complex process by which we humans feed ourselves. It’s a complicated challenge to teach a machine – everything from identifying food items on a plate, picking them up and then transferring it inside the mouth of a care recipient.
‘This last five centimetres, from the utensil to inside the mouth, is extremely challenging,’ Bhattacharjee said.
Advertisement
Some care recipients may have very limited mouth openings, measuring less than 2 centimetres, while others experience involuntary muscle spasms that can occur unexpectedly, even when the utensil is inside their mouth, Bhattacharjee said. Further, some can only bite food at specific locations inside their mouth, which they indicate by pushing the utensil using their tongue, he said.
‘Current technology only looks at a person’s face once and assumes they will remain still, which is often not the case and can be very limiting for care recipients,’ said Rajat Kumar Jenamani, a computer science doctoral student.
Advertisement
To address these challenges, researchers developed and outfitted their robot with two essential features: real-time mouth tracking that adjusts to users’ movements and a dynamic response mechanism that enables the robot to detect the nature of physical interactions as they occur, and react appropriately. This enables the system to distinguish between sudden spasms, intentional bites and user attempts to manipulate the utensil inside their mouth, the researchers said.
The robotic system successfully fed 13 individuals with diverse medical conditions in a user study spanning three locations: the EmPRISE Lab on the Cornell Ithaca campus, a medical centre in New York Cit, and a care recipient’s home in Connecticut. According to the researchers, users of the robot found it to be safe and comfortable.
‘This is one of the most extensive real-world evaluations of any autonomous robot-assisted feeding system with end-users,’ Bhattacharjee said.
Advertisement
The team’s robot is a multi-jointed arm that holds a custom-built utensil at the end that can sense the forces being applied on it. The mouth-tracking method – trained on thousands of images featuring various participants’ head poses and facial expressions – combines data from two cameras positioned above and below the utensil. This allows for precise detection of the mouth and overcomes any visual obstructions caused by the utensil itself, researchers said. This physical-interaction-aware response mechanism uses both visual and force sensing to perceive how users are interacting with the robot, Jenamani said.
‘We’re empowering individuals to control a 20-pound [nine-kilogram] robot with just their tongue,’ he said.
Advertisement
He cited the user studies as the most gratifying aspect of the project, noting the significant emotional impact of the robot on the care recipients and their caregivers. During one session, the parents of a daughter with schizencephaly quadriplegia, a rare birth defect, witnessed her successfully feed herself using the system.
‘It was a moment of real emotion; her father raised his cap in celebration and her mother was almost in tears,’ Jenamani said.
While further work is needed to explore the system’s long-term usability, its promising results highlight the potential to improve care recipients’ level of independence and quality of life, the researchers said.
‘It’s amazing,’ Bhattacharjee said, ‘and very, very fulfilling.’