WWDC 2024 Recap: AI, More AI, and Some Not AI

Intro

Today, Apple told us what to expect in the next round of software updates, coming this fall. While the focus was, unsurprisingly, AI, there is more than just that to be excited about. Let's dive into what you can look forward to on your iPhone, iPad, Apple Watch, Mac, and Apple TV later this year. Or next year, in some cases; as has become common in the last few years, Apple announced features it won't be rolling out until well after the software's initial release.

Siri and Apple Intelligence

Siri got a lot of attention this year, and for good reason. AI is being used to upgrade and enhance Siri in a lot of big ways. Many of the improvements can be summarized quite simply: real, natural interactions. Basically, the goal seems to be to let users ask for something to happen in natural speech, and Siri will do it. It feels a lot like Apple's old "it just works" philosophy--no thinking about how to phrase a request, no having to know what actions Siri can and can't do. Just ask that this contact's name be changed, or this photo be set as your wallpaper, and it happens.

Siri can even handle mistakes. If you ask, "Hey Siri, what's the weather in San Diego? Wait, I mean San Francisco." Siri will correctly identify that you wanted the weather in San Francisco. It'll even keep the conversation in mind for future queries, so if you then ask, "How long would it take to drive there?" Siri will know that "there" is San Francisco.

Siri will also be able to keep track of, well, pretty much everything. If you and a friend agreed to have lunch in a text message thread, Siri knows about those lunch plans. If your boss sends you an email with a time-sensitive task, Siri knows it. Ask it to play "that song Jim texted me about", and Siri will figure out which song you mean. This even extends to photos and video--Siri can handle a command like, "show me pictures of my husband from that trip we took last year where he's wearing a hat."

Apple Intelligence (which abbreviates to AI, which definitely won't get confusing to talk about) is behind all this. Apple Intelligence runs locally, noting everything it can about you and your life. Does that sound creepy? Apple was careful to emphasize that this is all local. It'll never send your aggregated information anywhere outside of Apple's own infrastructure, or sell it, or do anything else with it. If Siri can't handle a request on your phone, it'll use a purpose-built server for the job. These servers run Apple Silicon, and are designed to never save data or allow anyone to read any data while a request is being processed. Apple even open-sourced the code they use on these servers so it can be independently audited.

AI (Apple Intelligence, not artificial intelligence) can do pictures and audio, too. It can generate emoji-like images from text prompts, which Apple calls Genmoji. It can generate images inside notes, or clean up images you draw yourself on iPad. It can automatically OCR images and PDFs, and save transcriptions of audio from voice memos or audio you record in the Notes app. You can even record phone calls now, and have transcriptions available almost immediately, transcriptions AI can add to Siri's data set for use later.

The tricks don't stop there, though. Apple has integrated ChatGPT into its Siri revamp. If Siri determines that ChatGPT is better able to help you, it'll ask for your permission, then send your question (anonymized) to ChatGPT 4O.

What else can AI do? It can summarize emails, and show those summaries in place of the message previews we have now. It can categorize your emails by topic (newsletters, receipts, marketing, important, personal, and so on). It can generate text, and help you rewrite text you type yourself, in just about any text field you might use. It can decide if a notification is important enough to show you when you have a specific focus mode enabled. It can summarize a bunch of notifications, such as a text thread that blew up, so you can get the gist of what was discussed without having to read a bunch of messages. It can do a lot more, too, or will be able to eventually. Within the next year, Apple says Siri will be able to understand what's on your screen, and take natural language commands about what you want to do. Delete this email, copy this summary and put it into a note, save this document with this name, you get the idea.

All these features are coming to iPhone, iPad, and Mac. We don't yet know exactly what can run locally on which devices, or which features might not work at all on older products.

iOS 18

Apart from all the Siri and Apple Intelligence stuff, iOS 18 has more new features.

First up, the home screen and Control Center are getting a makeover. You can now put app icons wherever you want on your home screen, instead of having to use a grid that fills in from the top left. If you want all your apps on the edges of your screen, or the bottom, you can do that. You don't even need to keep them in a grid anymore if you don't want to. App icons will now change color when dark mode is on, or you can choose your own color that will tint all your icons.

Control Center now has multiple pages, similar to the lock screen. Perhaps more exciting is the ability for app developers to create their own Control Center widgets. For example, if Uber takes advantage of this, you might one day be able to put a widget to "get a ride home". Open your Control Center, tap the widget, and your ride is on the way. These new widgets can also be tied to the action button on iPhone 15 Pro, letting you issue any Control Center command with the push of a physical button. You can also replace the camera and flashlight buttons on the lock screen with any Control Center widgets you want.

At long last, you can lock any apps you choose behind authentication. Some apps already offer this, like journals or banking apps, but now you can decide. If you want to make it so someone has to use Face ID or Touch ID before they can get into your book player, you can do that. You can even take it a step further and hide the app completely. No one will even know the app exists without authenticating first. This is perfect for hiding apps you find embarrassing or overly personal, so you can hand your phone to someone else without worrying that they'll ask awkward questions.

Messages is getting some new features, in addition to the Genmoji we've already talked about. There are more quick reactions available, you can apply bold, underline, and italics to text, and you can use special effects on parts of your text. RCS is finally on the way, letting non-iMessage conversations take advantage of typing indicators, read receipts, and other features. You can schedule a message to send later, and if you have a compatible iPhone, you can exchange messages via satellite.

We've been over some of the Mail app's new features, but there's more. I told you Mail can automatically categorize messages and offer AI-generated replies or edits to replies you type. Mail can also group mail by business, into what Apple calls a digest. All the messages from a particular company, for instance, might make up a digest. You can act on all these messages at once--that is, move or delete them.

When Mail puts a message into a particular category, you can manually move it if the AI was wrong. Apple didn't say if this will train the sorting of future messages, but I hope it will. One category to pay attention to consists of messages Apple Intelligence thinks are important. Requests for you to do something, messages you should reply to, that kind of thing.

Photos is the other app that got a huge update in iOS 18. The top of the screen shows the familiar grid of photos, with the ability to filter by objects, places, and more. You can also have the grid show only photos from all time, this week, and other timeframes, just as you could before.

Below the grid is the new library, where you can select a collection of pictures. Collections are put together by Apple Intelligence. It might be a trip you took, or pictures of someone specific, or some other commonality. You can choose collections to pin for easy access, and make slideshows (aided by on-device AI) complete with music. You can share collections with others if you want to.

There are other changes, too. Filter out "clutter", such as pictures of receipts. Hide screenshots. Make AI-assisted edits, such as removing a distracting object from the background of a shot. The presentation didn't describe the layout of the new app that well, though, so I can't really tell you what to expect beyond that.

There are some smaller, but no less useful, changes coming across the OS.

• Tap to Cash will let you send or receive money through Apple Cash with NFC, by putting your phone near someone else's.
• There will be topographical maps of 63 U.S. national parks, useful for planning hikes. These can be used on watchOS as well.
• Game mode will focus your phone's power on the game, minimizing background operations, and will decrease the latency on AirPods.
• Games can build in support for personalized spacial audio on certain AirPods models.
• With AirPods Pro 2, you will be able to nod or shake your head in silent response to yes/no questions from Siri, and voice isolation will be improved.
• There's now a dedicated app for managing your saved passwords. It is coming to iPhone, iPad, Mac, and Windows.
• The Journal app will have writing streaks, a search function, a mood tracker, and some other new features.
• Reminders have been integrated into Calendar. There's still a Reminders app, but you can add reminders while inside the Calendar app as well.
• You have the option of choosing larger Home Screen icons.

watchOS 11

With watchOS 11, Apple continues its fitness focus. You can set up your activity rings by day, such as giving yourself harder goals on Sundays when you have some extra time. You can pause your goals, so you can have a rest day every so often or not have an illness break your move streak. You can also customize the summary screen shown in the Fitness app on your watch, focusing on the metrics you care about most.

The new Vitals app tracks your overnight metrics--temperature, heart rate, breathing rate, and more--and alerts you if anything is out of the ordinary. It can also help you link these aberrant values to causes, such as suggesting that alcohol consumption makes you get less sleep or that your heart rate has been high while you sleep, which may be due to your body temperature also being high at night.

Training load is the other big change. Now, you can get an understanding of how your workouts are affecting you. This is especially useful for those on a specific training plan or who are working toward a goal, but it can help anyone better understand their fitness. Some workout types will generate an effort rating from 1 to 10. If a workout can't auto-generate a rating, you can enter one manually. You can also adjust an automatic rating, such as bumping it up if you were sore during strength training and you feel that Apple's estimate is too low. Eventually, these numbers will be used to help you get insights. For instance, if you logged five days in a row with effort ratings of 8-10, maybe you should give yourself an easier day. You will also be able to compare your most recent 7 days to the previous 28, to see if your effort has been below, above, or about the same in the last week compared to the last month.

While fitness changes are the big ones, watchOS 11 has other improvements I'm looking forward to.

• The Smart Stack is more intelligent. It offers more widgets, and uses time, location, and other factors to bring up widgets that may be useful.
• Checkin, a feature that can be used to tell friends and family when you get home or that you didn't arrive when expected, is now available on Apple Watch. It's also integrated into the Workout app, so you can use it to let someone get an alert if you don't get home from a workout.
• Live activities will work on watchOS now. This makes it much easier to track flights, rides, sports scores, and other events on your watch.
• In they Cycle Tracking app, you can mark that you are pregnant. This will start tracking your gestation age, and will suggest things for you to do. It might ask if you want to increase your high heart rate threshold, or remind you to check your mental wellbeing more often.
• Apps can be told when you perform a double tap. Finally, third party apps will be able to do things when you double tap with the app open.

iPadOS 18

In addition to all the Apple Intelligence, Mail, Messages, and other features we went over in the iOS 18 section, iPad is getting some new tricks of its own. We'll start with the one many of you probably won't care about: Apple Pencil.

iPad can figure out your unique handwriting style. When you write, it can keep that style in place, but clean up your text to make it more legible. If you write with Apple Pencil and then paste text from another app, the text will be transformed to look as though you wrote it. A similar automatic cleanup process can be applied to sketches you make in the Notes app.

Math Notes is a feature that was presented as though it's tied to Apple Pencil, but I don't think it is. I think it works if you type, too. Math Notes basically detects when you're writing math-like text, and will auto-fill answers. If you write an equation, then add an equals sign, iPad will solve the equation and add the answer to your text. This feature will come to macOS as well, but it's unclear if iOS will get it.

Speaking of math, iPad finally gets a calculator app! It hasn't had one since its launch back in 2010, and the lack of a calculator has been something of a joke around the internet for years. Now it has one, with everything from basic math to scientific functions to unit conversions.

SharePlay has been updated. You can now take control of the other person's device if you're in a screen sharing session, to help them do something. You can also use your Apple Pencil to point to your device, and the other person will see the pointer on their screen.

The only other feature that was discussed is one I didn't quite understand. Tab bars now float, and can transform into sidebars and back again automatically. I don't quite know what this is about, but Apple seems very excited about it, so I guess it's a good thing. Hurray for the magic tab bars!

macOS Sequoia

macOS Sequoia isn't a major update, but it has a headline feature you'll definitely want to hear about. It also gains Apple Intelligence, Math Notes, the passwords app, and the same topographical maps that the other Apple devices get.

The big new thing in macOS is iPhone mirroring. Choose this from the macOS Control Center, and a life-sized image of your iPhone screen appears on your Mac. You can use the mouse/trackpad to perform gestures on the virtual screen, and the keyboard to type. Your phone's audio comes through your Mac's speakers. All this happens while your actual iPhone stays locked and tucked away--this is all wireless. Your Mac will now show iOS notifications; clicking one will open iPhone Mirror with the notifying app already displayed.

Other features Apple mentioned are:

• Safari can pull highlights from a webpage, such as the contact information for a business, and show them to you. It can also summarize a page and show an auto-generated table of contents when Reader Mode is activated.
• Presenter Preview works with video conferencing apps to show you what your camera is about to share with everyone else. You can replace your background with a premade one, or one generated from one of your own photos.

Other Announcements

There were a few other things covered by Apple that I didn't feel warranted their own sections. My apologies to all you Apple Vision Pro owners in the audience.

• visionOS 2.0 will increase the resolution of the Mac displays that can be mirrored on Apple Vision Pro.
• Machine learning algorithms can turn any photo into a spacial one, suitable for viewing on Apple Vision Pro, and SharePlay can now be used to let a group of people all look at the same photos on their headsets.
• There are new visionOS gestures for common tasks, such as checking the battery.
• If you're watching an Apple TV Plus production, you can get a live feed of which actors are on screen, their characters' names, and what the name of the song playing is.
• Apple TV has wider support for speakers, better enhanced dialog, and automatic subtitling if you skip back or mute the volume.
• Apple TV has the ability to use your own photos as screen savers, or clips from Apple TV Plus content you've been watching.

There You Go

And there you have it. A huge focus on AI, which isn't a surprise, with Apple's typical focus on user privacy. But we also got some great new features on all of Apple's devices. I'm tentatively optimistic about AI and Siri, and I'm looking forward to seeing how good it winds up being. I'm thrilled by the fitness features on watchOS, intrigued by Math Notes and some other changes on iPad, and would be more excited by iPhone Mirroring on macOS if I used a Mac more.

How are you feeling? Does all this AI stuff creep you out? Will you find iPhone Mirroring useful? Are you looking forward to being able to arrange your icons however you want? Are you one of our rare Apple Pencil users who's excited to get your hands on iPadOS 18?