DeepMind’s new AI model brings sign-language-to-text translation to Pixel 11

Google is bringing a new form of accessibility technology to its latest smartphones, with the Pixel 11 family becoming the first consumer devices to use DeepMind’s new sign-language-to-text (SL2T) model. The technology is designed to let Deaf and hard-of-hearing users communicate with their phones using sign language instead of relying solely on a physical or on-screen keyboard.

SL2T is integrated into Gboard and Live Transcribe, allowing users to sign in situations where they would normally type. This can include searching the web, composing messages and interacting with Google’s Gemini assistant. In Live Transcribe, users can also sign responses during conversations, potentially reducing the need to type replies manually. Google says the feature is intended to provide a more natural input method for people who use sign language.

A different approach to translating sign language

DeepMind says the new model was trained on more than 100,000 hours of multilingual sign language data. About a quarter of that training material is in American Sign Language (ASL), which explains why ASL-to-English is the only translation pair available when the feature launches. Google says it plans to expand support to additional sign languages and devices in the future.

The company says its approach is designed to account for the fact that sign languages contain considerably more information than hand movements alone. Facial expressions, body position, and movement can all contribute to meaning, making sign language translation more complex than simply recognising individual gestures. DeepMind says the system was developed with input from members of the Deaf community to make the technology more useful in everyday situations.

Rather than sending raw camera footage to its servers, the system uses a separate on-device model to convert the video into a representation made up of geometric landmarks. These coordinates describe relevant movements of the hands, body and face. The resulting data is then sent to Google’s servers for translation, while the original video remains on the device. This approach is intended to strike a balance between the translation model’s computational requirements and user privacy.

SL2T also takes a different approach from many earlier sign language translation systems. Traditional approaches have often converted signs into intermediate representations, known as glosses, before translating them into written language. DeepMind argues that glosses can struggle to represent the non-linear and spatial characteristics of sign languages.

“Glosses fail to capture rich, non-linear aspects of sign languages such as non-manual markers and spatial constructions,” DeepMind explains. “Translating directly from landmarks removes artificial vocabulary limits and allows translation quality to scale directly with data.”

Pixel 11 marks the beginning of a wider rollout

The arrival of SL2T on Pixel 11 represents a move from experimental sign-language AI towards technology built directly into consumer devices. Instead of requiring a separate application or specialised hardware, the feature is available through familiar Google tools such as Gboard and Live Transcribe. The Pixel 11 range was unveiled alongside other new Google hardware and software features, with sign language recognition as part of the company’s broader accessibility push.

DeepMind says the technology can be used in a variety of everyday situations. A user could sign a search query rather than type it, compose a message using ASL, or communicate with Gemini through sign language. In a face-to-face conversation, Live Transcribe can provide another way for a Deaf or hard-of-hearing person to respond without repeatedly switching between signing and typing.

The launch is nevertheless limited compared with the global range of sign languages. More than 70 million Deaf and hard-of-hearing people worldwide use around 200 different sign languages, according to DeepMind. American Sign Language is therefore only one part of a much broader accessibility challenge, and ASL is not interchangeable with other major sign languages such as British Sign Language.

DeepMind says the multilingual training behind SL2T could help provide a foundation for future expansion. The model was trained across multiple sign languages to learn shared structures among them, although those additional languages are not yet available to Pixel users. Google has indicated that more languages and devices will follow, potentially extending sign-language-to-text technology beyond the initial Pixel 11 launch.

The introduction of SL2T also highlights a broader shift in how smartphones can be used as accessibility tools. Voice dictation has become a common way for hearing users to enter text without typing, while sign-language input has historically been far less accessible. By bringing sign-language recognition into everyday smartphone software, Google is attempting to make that type of interaction more widely available.

The technology is still an early step, not a universal sign language translator. Its initial ASL-to-English limitation means many potential users will have to wait for support for their preferred sign language. However, the Pixel 11 launch establishes a consumer platform for DeepMind’s research and provides Google with a path to expand the technology as its multilingual capabilities evolve.