About the American Sign Language Avatar Project.
Dr. Rosalee Wolfe: [Deaf] people don’t use English as their preferred language. They use American Sign Language. This is the language of the Deaf. Being able to communicate is such a basic fundamental that we need. When I say Deaf here I’m saying Deaf with a capital D. I’m not talking bout say your grandparents who may have gone to too many rock concerts in their youth. I’m talking about people who were born deaf.
What we’re doing is developing noninvasive technology to bridge that deaf hearing communication gap. We have a voice activated avatar where you can speak various words and phrases and it then produces the American Sign Language. A person who is deaf can read what the avatar is saying in the animation and it promotes better communication between the two communities.
So, for example, if I am not deaf and you are Deaf, I could speak into this program, which is on a desktop computer, and an animation avatar would actually sign it back.
Exactly. What’s wonderful about today’s technology, is that now the actual components, the physical hardware you need for this is quite reasonably priced. This is something that ultimately we’d like to have on all portable devices, so it’s just simply an add on to the cell phone you already have.
Basically this is like a Google Translate for the Deaf, between American Sign Language and spoken English. How different is American Sign Language from spoken English?
They are very different. In American Sign Language, it's its own independent language with a unique syntax and grammar like any spoken language. There are parts of speech in American Sign Language that we as speakers of English haven’t even imagined yet. For example, you sign a statement such as "You’re going to the store." If you add say some movements of your head and eyebrows, you can ask “Are you going to the store?” but it’s exactly the same signs you make on your hands. The attitude of the body, so how you position your spine, how you position your head, how you move your eyes, how you move your face all contribute to the language. There are rules--just like in spoken or written English--there are rules to how to use your body in order to create this wonderfully complex and expressive language.
It’s sort of like, what we create using our voices in terms of tones and expression, that’s done in three dimensions.
Yes, that’s done in three dimensions. And also we do things in word order that don’t necessarily correspond to sign order in sign language.
I think about how when we use Google Translate, if I do English to French it will still use words. I type English words into one side and French words come out the other side. But when you’re translating to American Sign Language, words aren’t enough.
When you’re going from English to French, you’re going from typography or words, to another set of typography and French words. Here, when you’re going from English, which is words you can type, into this visual modality, this gesture. Then yes, you need to use computer animation in the form of an avatar in order to produce that so it is legible.
Right now you have a desktop version of this program people can interact with. Where are people using it and how are people using it?
We’ve found a lot of use in helping hearing people learn how to sign effectively. We have copies of our avatar in all of the state-supported developmental residential centers in Illinois. At the Jack Mabley Developmental Center in Dixon Illinois, now over 95 percent of all the caregivers can communicate with residents using sign language. that’s really amazing because when they’re hired, they’re hired for their caregiving capabilities not for knowing sign language. They have to learn on the job, and the avatar gives them the chance to practice.
So they can speak into the program, and it can sign it back to them and they can learn in that way.
And also, the avatar can sign something and say “Ok, what did I just sign?” so you can learn recognition of sign language in that way.
So it’s also a bit of a Rosetta Stone for sign language.
That would be a great goal.
So tell us about your long term vision for this. You mentioned a mobile app in the future. Is that something you’re actively working towards?
Yes we are. Along with my co-leader Dr. John McDonald, we’re working toward the idea of having this app available universally. It would be something you could download form the App store, have it on their phone. Say someone doesn’t have an app—you’re talking about the hearing hotel clerk and a Deaf client who’s going to be staying there that night, if the Deaf person has this on their phone, they could hold it out and say speak into this.
The idea is we could make this easy to download, available at a free or a very nominal cost, so that everyone will download and use it.
How far away are we from something like that happening?
We’re a little ways away yet, but we’re working toward it. I actually just had a productive conversation for onboarding the technology onto an Android platform.
So the project originally started in 1998.
Tell us about some of the challenges to creating tech like this. What are some of the obstacles that got in the way, or some of the things that are trickier about creating this very dynamic technology?
A Deaf student came to me and said, "Speech recognition has gotten better. We have video. We could hook up speech recognition to video." I didn’t know anything about sign language and I started asking, "Do things in sign language change form depending on how they’re used, like verbs in English?" We talked about 'I am' and 'You are'. She said "Yes it does," and I said "We should be using computer animation for this. We should be using avatar technology. There’s so much in avatar technology that exists, there’s commercial programs. We should just be able to take it off the shelf and it should work."
[Then] everything broke. It turns out the demands for realism and flexibility are much much higher for sign language. The things that is lovely about animated movies—beautifully expressive. But what Snow White did in 1937, she is still doing today. She hasn’t changed a thing. It’s always the same. For gaming avatar, they have different moves. You can control the various things that the avatar does, but it’s coming from a limited repertoire. It’s not like speech, which changes, it’s a living thing, it changes everyday. We needed to be able to create an avatar that can create very realistic utterances that are easy to read for the Deaf community.