AI-Powered Expressive Captions for Android Have Improved Google Live Captions

To help the millions of individuals who are deaf, dumb and those with the challenge of hearing watch television, captioning was first made popular in the 1970s. Whether they’re watching movies in a noisy public setting, on the metro, or just to better understand what’s being said, 70% of Gen Z now use subtitles the majority of the time. Livestreams, social media posts, and videos from friends or relatives may occasionally become inaccessible due to the absence of pre-loaded captions. However, throughout the previous 50 years, there hasn’t been much of a shift in the way captions are displayed. This implies that subtleties of sound and language, such as personality, tone, and emphasis, are frequently missed.

With the release of Expressive Captions on Android, a new feature in Live Caption that will tell you not just what someone says but also how they say it, we hope to change that today. This release significantly advances our range of Google Captions products, which includes Sound Notifications, Live Transcription, and more. Because you ought to be able to sense it even if you are unable to hear it.

A new feature named ‘Expressive Captions’ was introduced by Google on Thursday as part of an AI upgrade. Its Live Captions feature for Android is implementing the feature. Users will be able to view live captioning of videos in a new manner that better explains the meaning of the sounds as they are played throughout the device. The AI function will display text in all caps to convey loudness, yelling, and excitement. In the US, Expressive Captions is currently accessible in English for Android 14 and Android 15 devices.

The ‘Expressive Captions’ feature on Google is powered by AI, although captions were first made popular in the 1970s as an accessibility tool for the deaf and hard-of-hearing population, the search engine giant revealed specifics of the new AI function that would be added to Android’s Live Captions. It also claimed that the way captions are presented has not changed in the past 50 years.

Nowadays, many individuals use subtitles to better comprehend what is being said, when streaming information online in noisy public places, or when watching content in a foreign language. Google noted how popular captions are with Android users and announced that it is now using AI to reinvent the information that captions transmit.

The live subtitles will be able to convey human noises, ambient signals, tone, and volume thanks to expressive captioning. “These little things have a big impact on communicating ideas that are beyond words, especially for social media and live content without high-quality or preloaded captions,” which has been shared by Google.

Expressive captions will be innovative in that they will use full capital letters to convey the intensity of speech, such as loudness, excitement, or rage. Additionally, by identifying sounds like sighing, groaning, and breathing, these captions will assist users in comprehending the subtleties of speech. Additionally, it is going to record background and foreground ambient sounds, like cheers and applause.

According to Google, Expressive Captions is a feature of Live Captions that is integrated into the operating system and will work on any Android device, regardless of the app or interface the user is using. As a result, users may discover real-time AI subtitles for films shared on messaging apps, social media postings, live broadcasts, and memories in Google Photos.

The AI processing for expressive captions is notably done on-device, so users will be able to view them even when the device is in airplane mode or not online.

Your Android device’s expressive captions use artificial intelligence (AI) to convey human sounds, ambient signals, tone, and loudness. Particularly for live and social media footage without pre-loaded or excellent captioning, these minor details have a significant impact on communicating ideas that transcend words.

• All CAPs: Captions will now accurately convey the passion of speech using capitalization, so you’ll be able to tell when a friend exclaims, “HAPPY BIRTHDAY!”
• Vocal bursts: Additional sounds, such as sighing, groaning, and gasping, will be recognized, providing you with crucial tone expressions.
• Ambient sound: To help you understand what’s going on in the surroundings, we’ll name other sounds in the background and foreground, such as cheers and clapping.

Since expressive captions are a component of live captioning, they are integrated into the phone’s operating system and accessible through all of its apps. This implies that you can use expressive captions for the majority of the content you watch, including social media livestreams, memories in your Google Photos reel, and video greetings from loved ones. You can utilize captions even when in airplane mode because they will appear on the smartphone and in real time when activated.

Our Google DeepMind and Android teams studied how we interact with material on our devices without sound in order to develop Expressive Captions. Expressive Captions uses several AI models to record spoken phrases and convert them into stylistic captions. It also labels a greater variety of background noises. Because of this, captions are as lively as sounds. It’s just one way we’re leveraging AI to develop for everyone while also taking into account the actual experiences of individuals with impairments.

Expressive Captions will be accessible in English in the United States starting today on any Android device with Live Caption that is running Android 14 or later. This is a component of our efforts to discover even more methods for adding context and emotional expression to captions.