AI Can Interpret and Translate American Sign Language Sentences

According to the World Health Organization (WHO), there are an estimated 360 million people worldwide with disabling hearing loss. To help with sign language translation,  Researchers from Michigan State University developed a deep learning-based system that can automatically interpret individual signs of the American Sign Language (ASL) as well as translate full ASL sentences without needing users to pause after each sign. The work has the potential to help translate some of the 300 sign languages in use globally. 

Using the NVIDIA GeForce GTX 1080Ti GPUs, with the cuDNN-accelerated TensorFlow deep learning framework, the researchers trained their algorithm on a self-collected ASL dataset, comprised of approximately 7,000 samples covering 56 commonly used ASL words and 100 ASL sentences. 

The researchers used an NVIDIA Jetson AGX Xavier to implement their deep learning model, allowing the system to translate the ASL sentences in real time.

“If a deaf person were in the hospital and needed to communicate with a doctor they would have to wait for the hospital’s translator – if they have one – to arrive, connect with a toll-free service or rely on a family member to be present. This compromises their privacy and could worsen a health emergency,” said Biyi Fang, who created the DeepASL system as part of his Ph.D., along with Professor Mi Zhang of Michigan State University.

“This is just one example demonstrating the critical need for this type of technology. We are providing a ubiquitous solution to sign language translation. Hard-of-hearing individuals who need to communicate with someone who doesn’t understand sign language can have a personalized, virtual interpreter at anytime, anywhere.”

The system architecture of DeepASL.

According to Zhang, their algorithm named DeepASL has enabled a wide range of applications.

Virtual Interpreter, the first application enabled by DeepASL, is an always-available virtual ASL interpreter that allows deaf individuals to use their primary language to communicate with others in a natural and convenient manner. 

Beyond its ability to help the hard-of-hearing communicate, DeepASL can help those virtually learning ASL by giving real-time feedback on their signing.

Read more > >