Sign Language Robot: Fingerspelling Hands to ABB

A sign language robot has to do something most robots never attempt: move fingers, wrists and arms clearly enough that a deaf person can read meaning from the motion. Engineers were already building mechanical fingerspelling hands for deaf-blind people in the 1970s and 1980s, and in September 2026 two new projects, one in Nairobi and one in Auckland, put signing robots in front of deaf children. This post covers how signing was handled before machines, the first hands that tried it, what is being built now, and where ABB's dual-arm YuMi fits as a possible research tool.

A sign language interpreter at work during a Wiki4Inclusion campaign event in Tanzania. Photo: Rwebogora / Wikimedia Commons (CC BY-SA 4.0)

How deaf-blind fingerspelling led to the first mechanical hands

For most of history, signing between people needed another person. Deaf-blind people often use tactile fingerspelling: the speaker forms letters of the manual alphabet against the listener's palm, and the listener reads them by touch. Anne Sullivan taught Helen Keller this way in 1887, spelling words into her hand until Keller linked the shapes to things around her. The method works, but it ties the deaf-blind person to whoever is beside them. If no one who knows the alphabet is present, a phone call, a typed message or a news broadcast stays out of reach. Engineers in the 1970s started looking for a machine that could do the spelling.

Helen Keller with her teacher Anne Sullivan, who spelled words into her hand, 1897. Photo: Library of Congress via Wikimedia Commons (public domain)

In 1977 the Southwest Research Institute in San Antonio built a mechanical fingerspelling hand. A person typed on a keyboard, logic circuits translated each key into finger positions, and the hand formed the letters for a deaf-blind user to feel. It proved the idea could work, but users found it slow and stiff, and it could not form most letters well. In 1985 four Stanford mechanical engineering graduate students built Dexter, sponsored by the Smith-Kettlewell Eye Research Foundation. Dexter used pneumatic cylinders and cables, with an Intel 8085 microprocessor switching 22 valves, and spelled about two letters per second.

A Shadow dexterous robot hand holding a light bulb, a later research hand with fingers driven by tendons. Photo: Richard Greenhill and Hugo Elias / Wikimedia Commons (CC BY-SA 3.0)

The hands kept shrinking. Dexter-II, built by another Stanford student team in 1988, swapped air cylinders for DC servo motors, cut the volume by about ninety percent and reached four letters per second. Gallaudet University received two third-generation hands in 1992, designed with the Rehabilitation R&D Center in Palo Alto. A fourth generation called RALPH, short for Robotic ALPHabet, replaced pulleys with lever arms and linkages so the hand could be faster and more compact. All of these machines did one job, fingerspelling single letters, which is a small slice of what a full sign language needs.

Chapel Hall at Gallaudet University in Washington, DC, which received two third-generation fingerspelling hands in 1992. Photo: Sdkb / Wikimedia Commons (CC BY-SA 4.0)

Signing robots in Nairobi and Auckland classrooms

The newer projects try to produce whole signs. In Nairobi, the start-up ZeroBionic, co-founded by 22-year-old Norah Kimathi, has built a robotic hand that turns a teacher's speech into sign language in real time, AFP reported in September 2026. The hands are 3D printed from recycled plastic, cost about $350 each and are already in 78 schools across Kenya. Deaf teachers wear an imported motion-capture bodysuit to record signs, building a dataset of technical terms for subjects like maths and biology that Kenyan Sign Language lacked.

Deaf students in a classroom at Kayieye school in Kenya. Photo: Moving Mountains Trust / Wikimedia Commons (CC BY 2.0)

On 21 September 2026 the University of Auckland described a robotic interpreter for Turi Māori, Māori who are deaf. Dr Ho Seok Ahn and Dr Tauwehe Tamati lead the work, which uses signs developed with Turi Māori, their whānau, iwi and hapū, and translates them to and from text and speech. There are three robots: an adult-height humanoid, one sized for primary school children, and a smaller one for early childhood centres. Signers were filmed and photographed to capture mannerisms and lip movement, because sign language depends on the face as much as the hands.

An InMoov humanoid built from 3D printed parts, shown at Imagine RIT in 2017. Photo: DanielPenfield / Wikimedia Commons (CC BY-SA 4.0)

ABB does not sell a signing robot, and no source shows ABB working on one. Its dual-arm YuMi IRB 14000 is still a sensible platform to think about for research. Each arm has seven axes, which gives the elbow-and-wrist freedom that signing depends on, and the robot was designed to work next to people. In 2017 YuMi conducted the Lucca Philharmonic with Andrea Bocelli in Pisa, copying a human conductor's gestures that were recorded with lead-through teaching. A lab could mount small dexterous hands, like the ones in our prosthetic hand post, on YuMi's wrists and study arm movement and handshape as separate problems.

ABB's YuMi conducts the Lucca Philharmonic with Andrea Bocelli in Pisa, 2017. Photo: ABB

What a sign language robot needs before it reaches homes

In homes, the first likely use is a tutor, a small signing robot that helps a hearing family learn the signs their deaf child uses, or that repeats a lesson for a child who missed it. Auckland's early childhood robot points that way. The hard part is quality. Deaf signers judge signs on speed, rhythm, facial expression and the use of space around the body, and a robot that signs stiffly can be as hard to read as a bad interpreter. Every project so far has needed deaf people to record and check the signs.

A sign language interpreter talks with a deaf participant at a workshop in Tanzania. Photo: Rwebogora / Wikimedia Commons (CC0)

For factories and labs, the near-term value is the hardware lessons. A hand that forms clear handshapes thousands of times a day needs durable small actuators, good wrist control and safe contact with people, and those are the same needs collaborative robots have in assembly. Programs for arm gestures can be laid out and tested in simulation first, the way robot cells are built in RobotStudio, before anything moves near a person. Teaching by demonstration, as YuMi did for the Bocelli concert, is also how no-code tools let non-programmers show a robot what to do.

Students working with ABB YuMi robots in an Italian school lab. Photo: ABB

How far this goes depends on cost, data and trust. ZeroBionic's $350 hand shows the price of a single hand can drop, while a two-armed signer with an expressive face costs far more. Most sign languages have no large, checked datasets, which is why Nairobi and Auckland both had to build their own. Deaf communities have long pushed back on technology made for them without them, so the projects that involve deaf signers from the start, as both of these do, have the best chance. Human interpreters will be needed for a long time yet, and robot signers are most useful in the places where no interpreter is available at all.

Programming a single-arm YuMi with Wizard on the FlexPendant, without writing code. Photo: ABB

Innovation

AI solutions for effortless ABB robot control.

Automation

Robotics

ceojohntran@chatwithrobot.net

+84905311611

© 2025. All rights reserved.

qtran1215@gmail.com