A Brazilian professor has created an Android app capable of translating sign language to speech and vice versa. Using inbuilt sensors in the smartphone, algorithms and artificial intelligence, the app can help the hard of hearing better communicate.
There are approximately 10 million Brazilians with hearing deficiencies, half of whom are entirely deaf. Many of them do not know any Portuguese, communicating exclusively through Brazilian sign language (LIBRAS).
But Professor Manuel Cardoso, of the Federal University of Amazonas (UFAM), may be about to change that. Giulia, a smartphone app invented by the professor, uses three of the phone’s sensors to capture LIBRAS gestures.
Attached to the users’ arm, the app uses the phone’s gyroscope, accelerometer and magnetometer to read the user’s LIBRAS gestures. It can also translate spoken Portuguese into LIBRAS, with an avatar on screen conveying the meaning of the words. The technology is even capable of accommodating different accents and speeds, as well as variations in gestures.
Without interpreters, daily tasks become difficult for many hearing-impaired Brazilians, from food shopping to doctor’s visits. And although factories frequently employ deaf workers to fill disability quotas, they often miss out on training and development.
“Deafness is an invisible disability, and when you see a blind person, you know that you are a blind person, and the deaf person you do not know unless you try to talk to him,” Cardoso told Brazilian media. “Communication becomes more and more important, the deaf are even more excluded.”
Cardoso, who had already built eye-controlled computer mice for quadriplegics decades ago, hopes to internationalize the technology. For the professor, being able to communicate between different sign languages forms part of his plan for the technology’s future.
The app also has different features for different contexts. One such element is the ‘electronic babysitter’ feature, which vibrates when babies cry and was developed at the request of deaf mothers. It also has a tool for generating signals from QR codes, as well as plans to develop alerts for car horns.
Cardoso’s project, which already has support from the National Institute of Education of the Deaf (Ines), is developing wider usages. But Honda and Whirlpool factories in Manaus are also testing the app with their deaf employees. The app has also received support from Alcatel, and Brazilian service provider TIM has included the app in its own innovation programs.