How Apple is trying to normalize always-on AI

Apple does seem to have tried to ensure its controversial Audio Intelligence feature isn’t abused. Alongside the iPhone Duo and new iPhone 18 Pro range, Apple on Wednesday introduced a new Apple Watch equipped with a brand new feature it calls Audio Intelligence. The existence of this surprising tool was confirmed only on the eve of the launch event. Enabled by the new S11 chip on the latest Watch devices, this is actually a collection of four features: Siri Recap, which creates high-level summaries of conversations during your day. Live Rewind, which transcribes the previous 15 seconds of a conversation as text. Music recognition, which relies on Shazam. Sound recognition, which can identify things like sirens or alarms. All of these features must be consciously enabled by the user and are not on by default. What they have in common is use of artificial intelligence along with always-on microphones. While all four of these tools are being presented as opt-in, their existence will inevitably — and justifiably — raise privacy concerns.  Apple saw those concerns coming, and to make the features work it has placed a Secure Enclave on the S11 chip inside the latest watches. This is a dedicated secure buffer on the chip that processes audio privately on the fly. It does so without creating a recording of ambient sound around you, and the data it works with is isolated from the rest of the system. That Enclave pairs with the iPhone using “a new audio-verified pairing mechanism that exists in addition to Bluetooth pairing,” Apple explains in a detailed white paper about how it works.  For Siri Recap, the process works like this: Apple Watch recognizes a conversation is taking place. If so, audio flows into the Secure Enclave, encrypted, and transmitted to a similarly Secure Enclave on your paired iPhone. The audio is then immediately deleted from the Watch.  The phone will decrypt and transcribe the audio, condense it, and send that transcript to Private Cloud Compute for processing.  The short summary is then made, returned, and deleted after seven days. Speakers are not identified, recordings are not made, and detailed transcripts are not created or retained. The company has also introduced safeguards for those around you. The system will deliberately omit some information, including potentially harmful content, financial data, or personal identifiers. When you use the Live Rewind tool, an audible chime plays on your device to alert nearby people that you are using the feature. Challenges will emerge For one thing, we don’t yet know whether the company will provide enterprise IT with device management tools to disable the feature on managed devices. It’s crystal clear that devices that are constantly gathering data will be seen as potential security risks — particularly in regulated industries. And it seems far more logical to provide new APIs to disable Audio Intelligence on managed devices than it would be to insist anyone wearing an Apple Watch put it in a lead-lined box before beginning the next safeguarding, healthcare, or product development meeting. Another problem I see concerns Europe’s Digital Markets Act. It is, after all, inevitable that competitors (maybe including Meta) will want their devices to have equal access to the information gathered by Audio Intelligence. Based on the decisions Apple has made so far, it seems equally likely it will want to refuse such access; this is why these new features will not initially be available in the EU.  There is also no doubt hackers will attempt to break into the system, though doing so will not be at all easy on account of the intentional way the company has built in security. I suspect, but do not know, that attempts will focus on the points at which data is exchanged across devices, rather than when the information sits within secure enclaves on those devices. It also makes sense that everyone who uses iCloud services for their data should put 2FA in place and pick strong passwords.  What about iCloud storage? The other challenge will be encryption, specifically ongoing attempts to penetrate iCloud data encryption by some nations, led by the UK. Even so, if access could be achieved to iCloud-stored Audio Intelligence text, what is obtained would only be summary data, not a recording. The system is architected so recordings are never made. Apple’s white paper on the feature explores this in more depth: “Audio from the microphone enters the Secure Exclave of Apple Watch, where it is initially processed for speech, sounds, or music, without transcribing or storing the raw audio. This buffer is a continuously overwritten stream that exists only within the protected hardware and never creates an audio recording.” Once you decide to keep a Recap or Life Rewind text, the raw audio is not saved to iCloud, only the text, and then only if you use 2FA and a device passcode. No one else, including Apple, can access the encrypted data you save, Apple said. Where does this go next? One pre-event rumor that didn’t come true concerned AirPods equipped with cameras and Vision Intelligence to understand physical context and surrounding, with the aim of enhancing Siri AI. I’ve expressed reservations about this idea, but do think the security model Apple has put in place for Audio Intelligence will turn out to be an echo of its intentions to secure Vision Intelligence transactions. This implies a new chip with Silicon Enclave for AirPods Pro, a similar exchange of information in real time for summary and determination of context and content, and a process in which no recordings are made and media access beyond brief, time-limited summaries, is available. In short, Apple will use the same security model. What about the rest of them? That’s fine as far as it goes. But the other subtext is that while Apple seems genuine in its attempt to deliver relatively intrusive tech advances right, others will show less commitment to privacy and security. So, while it’s furthering the conversation about using these tools, Apple is also pushing public acceptance of such technologies. I’m not entirely sure we’re ready. Now please subscribe to my daily, human-curated Apple-related news headline feed at The Core, or follow me on BlueSky, LinkedIn, or Mastodon