A Demonstration of the New Image Description Features in iOS 27

One of the most significant VoiceOver improvements in iOS 27 is the arrival of much richer, AI-powered image descriptions. In this episode, David Nason takes the feature for a hands-on test drive, showing how VoiceOver can provide far more detailed information about images than the simpler descriptions users may be familiar with. Rather than having to share an image with another app, these descriptions are integrated directly into VoiceOver and can be accessed using the rotor or a gesture of your choice.

David demonstrates the different ways you can interact with visual content. Image Description provides a detailed account of an image, while Ask About Image lets you pose a specific question about what is shown. Explore Image combines the two approaches, first providing a description and then letting you type or dictate follow-up questions. These capabilities aren't limited to individual pictures: VoiceOver can describe an entire screen, answer questions about it, and even describe an individual item under VoiceOver focus. That last option could prove especially useful when encountering an unlabeled button or another inaccessible interface element.

The episode then turns to customization. David walks through adding Image Recognition to the VoiceOver rotor, choosing which recognition options appear there, and rearranging them to suit your preferences. He also demonstrates assigning Intelligent Image Description directly to a VoiceOver gesture for almost instant access. Alternatively, you can assign a gesture to Show Recognition Options, which presents a menu and lets you choose the type of description or exploration you want each time.

There's also a particularly useful option in Photos. David demonstrates generating an intelligent description of a photograph and saving it as that photo's VoiceOver label, meaning the description will subsequently be announced while browsing your photo library. He also discusses some areas where he would like to see the feature improve, including the ability to edit saved descriptions and receive more complete information about people and certain types of images. Despite these limitations, he comes away impressed by how quickly and conveniently the feature makes visual information accessible from within VoiceOver itself.

If you're a blind, DeafBlind, or low-vision iPhone user who regularly encounters photographs, app artwork, unlabeled controls, or other visual information, this episode offers a practical look at what these new VoiceOver capabilities can do. It's also worth a listen if you already use third-party AI image-description tools, as David demonstrates why having this functionality integrated directly into VoiceOver could make exploring visual content considerably quicker and easier.

Transcript

Disclaimer: This transcript was generated by AI Note Taker – VoicePen, an AI-powered transcription app. It is not edited or formatted, and it may not accurately capture the speakers’ names, voices, or content.

David: Hallo there and welcome to the AppleVis Podcast. My name is David Nason. This podcast is to demo one of the new and I think most important features, certainly for those of us using voiceover, in iOS 27. That feature is the new AI-powered image descriptions. Now you may be thinking we already had image descriptions in voiceover, but this is taking it to a new level. So those descriptions we had before were when you'd land on a photo and it might say table, chair, two people, grass, those kinds of things. So very, very basic descriptions we had before. Now within voiceover, we have proper AI-powered image descriptions similar to those we've become used to from things like BMIAI in the BMIs app or Pixibot and other such apps. So again, you might be thinking, why do I need this if I have those apps? And of course, you can continue to use those apps, and they still work brilliantly well. But one of the advantages of these new Apple built-in image descriptions I guess are that it is integrated with voiceover so you can access them via a gesture of your choice or using the rotor, you don't need to find a share button and then share it out to another app. You can just very quickly very easily access it. There are some caveats in the sense of some things I might like to see improved, but on the whole I found this to be a really useful feature as I've been testing iOS 27 and I'm delighted it is now available to everyone in iOS 27 full release. So before we jump in and look at some of the settings, let us take a look at the feature in action and give you a flavor for it. Where I'm going to go first is into the app store and I am going to find some artwork on an app entry.

Voiceover: Story graph app store. App store artwork libre.fm audiobooks image.

David: So this is the Libre.fm app and I am going to find the artwork on its app page.

Voiceover: Artwork image. Artwork image.

David: And you hear the automatic image description just said screenshot, but I can now get a full AI-powered image description on this. Believe it or not, there are six different options or ways of using these image descriptions. So we'll take a look at what I mean by that now. So the first one is just a straight up image description. So I'm going to use the rotor for this.

Voiceover: Headings. Image recognition. Image description.

David: So image recognition and then it's set to image description. So now I'm gonna double tap.

Voiceover: A promotional graphic displaying two smartphones against a plain white background with large text on the left reading, by audiobooks to your local bookshop in black and teal. A teal icon resembling an open book with sound waves rising from it sits below the text. The lower smartphone displays a choose a bookstore screen with a search bar and a list of stores near you, including Papercuts JP, Porter Square Books, Brookline Booksmith, Harvard Bookstore, and Trident Book Sell.

David: So quite a detailed description there from the artwork which is in the app store here. I'm going to flip over to the next picture in this artwork.

Voiceover: Artwork image adult screenshot.

David: And again, I just got the basic image description, but I can run the new AI powered image description on this as well. Let us go down and check the next option in the rotor after the straight up image descrip Image description.

Voiceover: Ask about image.

David: We've got ask about image. If I select this, it will allow me to ask a question rather than have it the picture described to me first. So straight up I ask it a specific question. So I am just going to do that now. I'm gonna double tap. Are there any people in this picture?

Voiceover: Yes, there is one person visible on the book cover display on the phone screen.

David: So that's the other way we can use it is we can actually ask a question up front. We can also, I suppose, get the best of both worlds, which is to first get an image description and then ask follow-up questions as well. So I'm going to go over to another app here for a change of scenery and we're going to go to overcast.

Voiceover: Overcast active. Overcast. Overcast one second with one hour.

David: And here we have some podcast artwork that we can try the feature out on and like say this time we're going to use the explore image feature which is that kind of best of both worlds.

Voiceover: The second captain's world surface image.

David: So this is artwork for the second captains world service podcast sports podcast I listen to here in Ireland. So I'm gonna find image descriptions in the rotor again.

Voiceover: Characters, image recognition, image description. Ask about image, explore image.

David: And the third option down is explore image so let's double tap.

Voiceover: A black and white circular logo featuring a vintage style microphone in the center, surrounded by six jagged lightning bolts radiating outward. The microphone and bolts are enclosed within a thick black border containing the white text the second captain's curving along the top and world service along the bottom. A small white globe icon sits on the left side of the border, and a white shamrock icon sits on the right.

David: Description, but this time there's also a text field available to me to ask any follow-up questions.

Voiceover: A black and white sir. Ask text field.

David: So then we have ask text field.

Voiceover: Ask dictate button.

David: Or you can dictate your question either.

Voiceover: Ask text field. So I can in text field is editing ask insertion point at start. Next keyboard flow type flow type ready.

David: Type in a question.

Voiceover: Keyboard area. Safe. Safe. R R there any colors in the occurs occurs picture. Period com question mark.

David: Are there any colors in the picture?

Voiceover: New line. Cap U. Text field is edit. Submit button. Thinking, are there any colors in the picture? Black and white.

David: And it just says black and white. So again, and sometimes you'll get a more detailed answer than that, so it depends on your question and so on. So that is the three main image descriptions that you can get using this feature. You've got just a straight up image description, you've got ask a question in advance and just get an answer to that question, and you've got the explore image, which it gives you that image description to start with plus then a text field or dictation where you can ask a question or a follow up question after that and again you I could ask another question again. So they are the three main images. I said there were six though, and what I mean by that is that we can also ask questions or get descriptions of the full screen. So what I'm gonna do is go back to

Voiceover: the app store.

David: home screen here.

Voiceover: Silhouette of two people mode app store.

David: And I'm gonna find the image descriptions again in the rotor.

Voiceover: Image recognition item description.

David: Or image recognition. And I'm gonna swipe down

Voiceover: Ask about item explore item ask about screen.

David: And here now you'll see we have ask about screen.

Voiceover: Explore item ask about screen. Explore screen.

David: Explore screen.

Voiceover: Screen description.

David: And screen description. So again, it's the same three concepts. Screen description gives me this the straight description of what's on the screen. Ask about screen lets me ask a question in advance, and then explore screen gives me the description plus a text field to then ask any follow-up questions. So let's do um explore screen

Voiceover: screen description.

David: Screen description here on my home screen.

Voiceover: A smartphone home screen displaying a grid of application icons and folders against a solid black background. The top rows feature individual apps including app store, passwords, photos, settings, story graph, call sheet, files, and notes, followed by two rows of folders containing smaller icons, some with red notification badges showing the numbers 4 and 30 Below the folders are individual icons for music, overcast, audiobooks, YouTube, be my eyes, live red, vibe, and games with the search button positioned just above the bottom dock. The dock at the bottom contains four primary apps, phone with a red, one notifications badge, mail, safari, and WhatsApp.

David: So a good detailed description of my screen and this can be really useful. You could be in an app where you have where you can't maybe focus on a specific image or you just want to know the layout of a screen, so there's various circumstances where rather than getting a description of just a particular image, you want a description of the whole screen. So again, this is a really useful feature and one definitely recommended. And another cool feature of this is that it's not only images or full screens that you can have described. You can actually have any item under voice over focus described. So I can go to an app icon here, for example, let's pick story graph.

Voiceover: Bandit water story graph.

David: And I can again run the image description.

Voiceover: Image recognition characters actions activate image recognition. Ask about item.

David: This time it says ask about item because it's not an image, instead of saying ask about image, it says ask about item, but it's the same thing and works exactly the same way. We have the same three options where I can just get a description, I can ask about it, or I can explore, which is to get the description and have follow up.

Voiceover: Item description.

David: So I'm just gonna select item description.

Voiceover: Three white minimalist bookshapes resting on a white horizontal shelf line inside a dark, rounded square icon. The two books on the left stand perfectly vertical, while the third book on the right leans outward at an angle. The word story graph appears below the icon in a clean white sans serif font against a solid black background.

David: But of course this can be useful in really practical ways too for example you could be in an app and there could be a button that has no label on it and of course sometimes voiceover will detect text on those but sometimes it doesn't sometimes it's just button and now you can run the item description on that button and it will at least describe the picture and if there is any writing it'll tell you any writing as well so again there, are really practical uses for why we're able to use this for any item with voiceover focus and not only for images. Okay, next we will take a look at those settings so that you can access this feature in the way that you want. As I said earlier, you can use the rotor, and/or you can use a gesture of your choice. There is no default gesture but you can go in two settings and assign one so let us go to settings settings set personal

Voiceover: hotspot back vpn switch general button accessibility button and then accessibility

David: accessibility heading personalized

Voiceover: vision heading voiceover on button gonna go to voiceover voiceover switch button on adjust the speak speech button.

David: And then we're gonna look for rotor to make sure it's in our to show you how to make sure it's in your rotor. I think it should be by default, but if you're not finding it in your rotor, this is what you can do to make sure.

Voiceover: Braille but voice over boss audio commands rotor

David: back There's rotor.

Voiceover: Rotor heading.

David: And then we find rotor items button. Rotor items.

Voiceover: Selected characters.

David: And here we have the list where we can select which items we ha we want in our rotor. I'm going to tap the bottom of the screen with four fingers because it's I think the last one in the list.

Voiceover: Dictate button. Search search field.

David: So tap with the four fingers and then swipe left a couple of times.

Voiceover: Reorder image recognition. Selected image recognition.

David: There we have image recognition. So just make sure that is selected and it will be in your rotor or if you would prefer to only use gestures and don't want to clutter in your rotor you can double tap select and select that to remove it. There is one other option in here as well so let us go back.

Voiceover: Audio audio docking rotor back button.

David: And back on the rotor settings screen we are going to find the following live

Voiceover: recognition rotor button. Image recognition rotor button.

David: Image recognition rotor.

Voiceover: Selected intelligent image description.

David: And this allows you to select which of those six items you want to appear in your rotor. So if you decide I never use the ask feature, for example, or I always want to have the option to ask follow-up questions so I don't want the straight up image description. Then you can deselect the ones you do not want in your rotor.

Voiceover: Image recognition rotor heading selected intelligent image reorder intelligent image description button drag selected ask about image reorder ask about image button drag selected explore image reorder explore image button drag selected ask about screen reorder ask about screen selected explore screen. Reorder explore screen. Selected intelligent screen description. Reorder intelligent screen description button draggable.

David: So you can hear there. But as I say, you can choose as you wish. You'll also have heard that you can reorder them as well, so you can go to the reorder button and then swipe up or down, move down, move up, and select to move up or move down again if you want to reorder them. So a really useful way to really customize it to how you want. Okay, now we will take a look at making sure we have a gesture assigned if that is what you want to do.

Voiceover: Rotor back button so I'm gonna go

David: back to voiceover settings voiceover back button and we are going to find the options to customize our commands, our touch gestures.

Voiceover: Verbosity button. Audio button. Commands button.

David: So we find commands and double tap.

Voiceover: All commands button. Touch gestures button.

David: And then touch gestures.

Voiceover: Tap one finger heading and then we

David: will select the gesture we want. As I said before, by default there is no gesture assigned to this, so it is up to you to find a gesture that you want to use. You may already have something else assigned to it and you might be replacing that or you may find one that's not currently in use and assign that. It can be any of the typical gestures be that taps or multi-finger taps or swipes or the even the back tap either is whatever you wish to use. I have assigned a one-finger quadruple tap to mine.

Voiceover: One finger one finger double, one finger triple tap,

David: one finger quadruple tap intelligence image description button and I've set that to intelligent image description which is just the straight description so let's go back to the home screen for a second home settings and I will just find an app icon.

Voiceover: Vibe.

David: And I'm gonna quadruple tap here with one finger.

Voiceover: A stylized, pixelated letter, M centered on a dark, rounded square icon against a black background. The letter features a gradient of warm colors, transitioning from bright yellow at the top to orange to deep red at the base. The word vibe appears below the icon in a clean, white, sensarif font.

David: So, very cool there that all I had to do is just do a gesture on the item or on the picture as it may be and you get that description. So let's go back into the settings. And I'm gonna select this and I'm gonna change it because there's another cool thing you can do here.

Voiceover: Settings active. Perform long press. Secondary activate. Edit. Image recognition headings.

David: So I'm gonna use headings and we need to f I'll show you as well where to find these options.

Voiceover: Basic navigation heading. Next block quote advanced navigation heading. Item text editing heading. Copy. Rotor. Heading. Next rotor. Scrolling. Heading. Scroll down. Speech and audio. Next audio output heading. Read all from current item braille. Heading. Decrease auto voiceover. Heading.

David: So it's under the voiceover heading. So you've just wiped down several headings to find it and we will find each of our six options, our image recognition options here, plus one extra one. So let us have a look here.

Voiceover: Ask a question. Ask about image.

David: So there's our ask about image.

Voiceover: Ask about screen.

David: Ask about screen.

Voiceover: Braille screen input. Braille screen input explore image. Explore screen.

David: Explore image and explore screen.

Voiceover: Selected intelligent image description.

David: And then we have what I have selected there, image description.

Voiceover: Intelligent screen description.

David: Or intelligent screen description. And then there is, as I said, another one.

Voiceover: Item chooser. Keyboard help, label item, live rec open quick settings, open voiceover settings, show recognition options.

David: Show recognition options. So I'm gonna select this.

Voiceover: Alert. Gesture in use. The gesture is assigned to another command intelligent image description.

David: And assign.

Voiceover: Selected one finger quadruple tap show recognition options home.

David: And now let's go to photos and I'll show you how this works.

Voiceover: Files. Call sheet. Photos.

David: And this is an old photo we found from my old school yearbook, believe it or not, many, many years ago. That was sent to me recently in a WhatsApp group. So I'm going to this time do the quadruple tap and what will come up rather than a image description is a menu. So let's do that.

Voiceover: Alert. Screen description button. Explore screen button.

David: And we can then see that we have a menu that has popped up and allows us to select which of the image description options we want to use.

Voiceover: Ask about screen button, explore image button, ask about image button, image description button.

David: So they're all there in a list in a pop-up menu. So that's a really useful way to use it as well. If rather than assigning a gesture to a specific one of these you can have it bring up a menu so you can use it as you wish and the final piece I want to show again is in the photos app so I'm just going to dismiss this menu. Photo date the set when you are in the photos app and only the photos app, you can also have it create the image description and save it as a label for that photo. So I'm on this photo, I'm gonna go to the rotor again.

Voiceover: Photo, date, the set image recognition, image description.

David: And I'm gonna swipe up.

Voiceover: Save intelligent image description.

David: And the last one of the list is save image description so if I double tap this a black and white half tone printed portrait of a

Voiceover: person with short dark hair and a neutral expression looking directly forward the person wears a dark colored garment over a light-colored shirt. The image is framed by a thin black border against a plain, light background.

David: And now that becomes the label on this picture. So even if I went back to my main library.

Voiceover: Date photo, date, the eighteenth of photo, date, the eighteenth, photo, date, the eighth, photo, a black and white, half-tone printed portrait of a person with short, dark hair and a neutral expression looking directly forward. The person wears a dark, colored garment over a light-colored shirt. The image is framed by a thin black border against a plain, light background. Date, the 17th of August to home.

David: And that is the label for that photo. So that's a pretty cool feature that you can use on your photos should you wish. Personally, I would like if you could also edit that description. You might know who the people in the picture are for, example, and want to add their names into that description, or you might want to shorten it a little bit, that kind of thing. Hopefully, that's something they'll add in the future, but at least you can do this if you wish. My one other little gripe with this feature is that it can be a little bit conservative in terms of giving you full information particularly around people. For example things like it won't tell you the gender or skin tone and those kinds of details about people in pictures. Sometimes you can get that information with follow-up questions, but it can be a bit hid and miss. So I would definitely like if it was a bit more upfront with information, full information about people in pictures. The other thing is that it can be a little bit conservative with what it decides are potentially inappropriate pictures or potentially explicit and that kind of thing. Again, I would think it should be up to the user what images they wish to have described to them. So that's my only kind of gripes with it. But on the whole I'm finding this feature really useful. I think the kind of deep integration with voiceover is brilliant. It's really quick, really easy to get descriptions of images when you're on the web or in an app or that people sent to you in messages, that kind of thing, as well as it working on items as well, so things like buttons and things like that. So I found this feature as I've been testing iOS 27 absolutely brilliant. I hope you find it useful too. Thank you for listening. I hope you found this useful and I will speak to you in the next one.