Apple’s AI-powered “audio intelligence” technology for the Apple Watch was first announced at the company’s Surprise and Shine hardware event in September, and is initially in beta testing for developers with the iOS 27.3 update. The technology includes four opt-in AI features that enable Apple Watch Series 12 wearers to detect important sounds, identify songs, follow conversations, and take notes.
Sound recognition helps hearing-impaired users identify important sounds like sirens, alarms, and doorbells from their watches, while song recognition leverages Apple’s Shazam app to detect music playing in the background and identify song names and artists.
However, a potentially more controversial feature is the ability to listen to human conversations, namely Live Rewind and Siri Recaps.
Given the backlash against AI recording devices like Meta’s AI glasses and other AI wearables, Apple is trying to carve out a space where the feature is useful for Apple Watch owners while respecting the privacy of those around them.
The idea of Live Rewind is to be able to go back in time and see what someone said in case you missed it. For example, if someone is giving you verbal instructions or reading you a long list of specials at a restaurant, pressing the watch’s digital crown twice will reveal what was said in the past 15 seconds as a text snippet displayed on the watch face. If you view this snippet and do nothing, it will disappear completely. Or you can save it to the standalone Siri app that ships with iOS 27 for later access. This is useful if you want to remember someone’s name or take quick notes from a fast-paced presentation.
Apple says users can also tap the “Ask Siri” option that appears below the text snippet if they have questions about what was said. For example, jargon you don’t understand or sauce options your waiter mentions that you don’t understand.
Press the Digital Crown twice to hear a chime and see an animation on your watch. It aims to let people know that what was said has been recorded through a combination of gestures, sounds, and visual displays.
These snippets will be automatically deleted after 30 seconds if you don’t save them.
Another option, Siri Recaps, takes this ambient listening technology even further by allowing Apple Watch owners to take notes on summaries of conversations.
Importantly, Apple says it does not identify speakers, save the live audio, or provide verbatim transcripts. Instead, this feature summarizes the conversation with short sentences and a few bullet points that identify important points. For example, depending on the topic, bullet points might include important dates and deadlines, pricing and payment plan information, and other factual information.
Apple suggested that this feature could be used in work meetings, parent-teacher conferences, personal training sessions, or other places where you need to complete a task or remember what was said. It’s basically supposed to mimic a user’s ability to write down their own notes without adding any emotion or tone.
This is especially useful when you can’t write notes yourself because your hands are full or you’re trying to concentrate on what’s being said at the time. If you’re listening to music with headphones or using the noise-cancelling feature of your headphones, Summaries won’t record your conversations.
At the end of the day, you’ll be asked to review the Siri summaries you saved while the feature was enabled and select any you’d like to save. These notes can also be saved to your Notes app or exported to your Journal app if they’re more personal. If you do nothing, your abstract will be automatically deleted within 7 days.
This reminder to review your recorded Siri summary also appears the first time you unlock your Apple Watch in the morning.
Users can choose to enable Siri Recaps manually or configure a schedule, such as having Recaps run only during work hours or not at night. You can turn off the Recap feature at any time from Control Center, even if it’s running on a schedule.
Apple notes that Siri Recaps is designed to omit sensitive information such as government-assigned identifiers, financial and authentication data, certain personal identifiers, and language that promotes self-harm.
The audio intelligence features of Apple Watch Series 12 and Apple Watch Ultra 4 rely on a combination of Apple hardware, software, and services. This includes the S11 chip, a built-in microphone, the Apple Intelligence model running on the iPhone, and Apple’s private cloud computing.
The always-on processor includes a secure enclave (a hardware-protected memory space that is inaccessible to the rest of the operating system) and a dedicated audio buffer to process raw audio. This allows anyone, including Apple, to process the audio without actually having access to it. Audio can only be accessed through a secure pairing mechanism between a paired iPhone and the watch.
To work, a lightweight AI model on the Apple Watch’s S11 chip detects whether a conversation is taking place in which the wearer is participating, and routes it to a secure area on the iPhone. iPhone’s on-device processing converts audio into compressed text, or conversational text with fillers removed. This compressed text is encrypted and sent to Apple’s Private Cloud Computing (PCC), which creates a summary of the conversation and sends it back to your phone. (The abstract and the abstract used to create it will be removed from the PCC immediately).
The summaries you can access on your iPhone can only be accessed by you, not Apple. It’s also encrypted when syncing between multiple Apple devices.
Similarly, the Apple Watch transfers audio to your iPhone for processing and creates a 15-second snippet for live rewind.
Apple says none of these features will affect the battery life of your watch or phone. These features will be rolled out in public beta for extensive testing before being released to the public later.
If you make a purchase through links in our articles, we may earn a small commission. This does not affect editorial independence.
