Proactively learning sighted gestures

I have run into the situation of not being able to communicate with a sighted person about IOS icons when trying to explain how to use apps; to close a tab in Safari, for example. A few times, I've asked a sighted person to turn on Voiceover and directed them when trying to explain something over the phone.
It might be helpful to have some sort of a guide of basic icon descriptions for Voiceover users.

Yes, I've had a similar problem.
I have a sighted friend who I'd love to help with her mac but we have problems because I don't know what the icons look like to tell her which one to click on.

How can we learn this?

This is a good discussion. I think what's happening here is common in the blind community. You're used to doing things as a blind person that you forget to think of things from the visual perspective.

I, too, would like to suggest creating a kind ov guide, either as a text post or podcast, that describes the icons.

In the meantime, one thing all of us can do is ask a friend to describe what the icons look like. Use your iPhone or some other note making tool to record the descriptions.

On an impulse, I did this the other day. I figured it would also help my friend as much as it would help me. I think I'll do that again this weekend. This way, I will have a guide nearby.

Given this, if anyone wants to contribute to this description guide idea, please do what you can. Thanks.

How about all of us work together to build such a descriptive icon guide, starting with iOS?

Each comment could contain just two pieces of information for one control, icon or other visual element:
1. A visual description of the icon. Include descriptive elements such as its color and shape.
2. A concise description of the action(s) performed when the icon is selected. Include descriptions of actions that take place when the icon is tapped, tapped and held, dragged, or acted upon using any other gesture.

Hope this helps.

When iOS 9.2 came out, one of the items mentioned as being changed was in the Music app, which "finally" showed which songs were downloaded to the device and which ones were only in the cloud. I thought that was strange, because VoiceOver always told me that a track was "available offline in this iPad." Apparently that was something only VoiceOver was telling me, that there was no visual indication of this status on the screen. Incidentally, they did change the VoiceOver label so it now says "downloaded" instead.

Over the last couple of years, I have transitioned tfrom using devices visually to using them with speech, as my vision has become progressively worse. This has, among other things, shown me how different these devices are when used without sight. It's not just knowing how an icon looks like or where it is located on the screen. The way certain tasks, including common ones, is performed by the user changes as well. I have had to transition from using drag and drop to using cut and paste, for example, since the latter is much easier when using keyboard navigation.

I know the share icon in Safari looks like an arrow.
Does the ones that says "pages" look like a book?

The way we do things is indeed totally different.

We can still learn though and help others and ourselves bad doing it.

The same thing happened when I tried teaching my Dad how to work Whatsapp, thus I showed him with VO on, while I performed the steps, in this case, to write to a Contact within Whatsapp's Favorites; thanks to this approach, I was told that when VO says Write Message (remember I'm using my IPhone in Spahisn, thus maybe in English it says something else, but hopefully you get the idea), a Pencil icon is onscreen, thus something new learned thanks to this experience.

If it's an app on my phone and someone else has it on their phone, I open the app and tell them which thing on the screen I'm talking about. A generic example, although it may say something about Bookmarks on the screen. Let's say someone called me up who was sighted and asked me where the show bookmarks in Safari was. I would tell them to open the Safari browser and then tap the second button from the right which should be the show bookmarks button. I hope this explanation helps a bit.

I'm really surprised that there isn't a way to get VO to announce the shape of an icon you touch, as well as the shape of the cursor when you touch said icon. I would think this a common enough request that it would have been implemented by now, but obviously not. Should this be something we ask Apple to incorporate? For now, I guess we'll need the sighted to help us help the sighted lol.

This subject actually came up on the thread I posted about the Apple At Home Advisor position . Someone made a very valid point that it would be hard for a blind person to provide troubleshooting of an Apple product to a sighted person since we use the products very differently . There is currently a speak hints option in VoiceOver so that you hear an announcement about what a control does if you pause after selecting it. How about we request another option where we also can hear a visual description of the control, as it appears to cited users on the screen?

The option of adding shape descriptions is an interesting idea, but this leaves me to ask some questions. How long should these descriptions be? Also, how much programming would it take? If it can't be done on the device itself, where else would you get these shape descriptions?

I did have someone try to describe the graphics visually, and I was made aware of how complex a job it is. I hope to ask another friend to describe them tomorrow and get a feel for what it sounds like.

Also, even if you do get a friend to describe these graphics, use your own imaginations. It may not help anyone else, but you can come up with some idea of what the graphics look like based on the descriptions, and combine that with your imagination.

I'm pleased this discussion is coming up. I'll do what I can for my own understanding, and if it's worthy of sharing, I'll share it with you. Thanks.

Hi,
One option that I do not think people have brought up yet here, is to read through the user guides for the apple products.
From what I have noticed, the descriptions have some of this information right in them.
Hope this helps. :)

Many assistive technology trainers that I know of use a system of teaching people who are blind to use mainstream computer applications using assistive technology. This is the approach they take:
1. Divide each screen of an application into "rows" including a title bar, menu bar, content areas and a status bar.
2. Describe each element on each "row" of the application including a visual description, the description that the assistive technology uses for this element, and the element's function.
3. Explain how each element can be activated for both the assistive technology user and people who do not use assistive technology.
Some of this detail cannot be applied to iOS, though it could be certainly applied to OS X.
For example, using this description we can describe the start page of Safari for the iPhone:
1. At the top of the screen is a long bar where the address is inserted. There is also a button for reloading the page to the right of the address bar. When the address field is activated, a "Clear Text" button appears to the right with the on-screen keyboard below it.
2. Below the Address Bar are four buttons pointing to frequently used websites.
3. Users can add more websites that they like below this.
4. There is a large area in the centre of the screen that contains nothing.
5. At the bottom of the screen are five buttons that VoiceOver identifies as "Back", "Forward", "Share", "Show bookmarks" and "Pages".
For example, if I were telling someone to go forward one page in Safari, I would tell them to tap the second button from the left in the bottom of the Safari window.
Sorry for this very long explanation. I am not trying to come across as being superior; it is just a method that I use and seems to work quite well. This method is also supported by Apple's Human Interface Guidelines for app developers.

Eight years later and I'm just discovering this helpful information.

Some good tips here. I wonder if there is more discussion on Apple viz. and if anyone has actually made a chart.

I've struggled with helping people with their devices for a long time. I don't know why I didn't look for this earlier.

What about the gestures that people with site use without voiceover?
I've tried reading manuals and help files for the iPhone for sighted people, but I just don't get how they do things.

I gave up trying to help sighted people who ask me things about the iPhone seven and a half years ago.

These days, at least on iOS, you can do a gesture to get image descriptions, and it will describe icons on your home screen.

Pretty cool, eh? 😆

Hi Brian,
You say, 'you can do a gesture to get image descriptions, and it will describe icons on your home screen.'
What gesture is that? I'm googling and nothing is coming up. I've got screen recognition and image recognition enabled. Sometimes, with apps like X for example, if I stay on the icon long enough, VoiceOver plays a sound and it announces 'close'. Is this what you're referring to, or is that something else? Incidentally these days, you could get Be My AI to describe the icons on your screen anyway. But I'm curious about this other gesture now.

I would love to learn more of these. Mostly what the Share buttons look like in different apps. I try to use other ways of helping my sighted family members, but they get frustrated and end up just bringing their phones here so I can do it instead of trying to talk them through it.

Hi Tara,

Apologies, I realize my last post was kinda vague. So this is how I have my VO custom gesture setup for image description, or recognition, or whatever's clever.
~ Under the 3 finger tap, I have tap one time with 3 fingers set to Image description/recognition.

Now, I realize that by default VO has it either as an unassigned gesture, or it is set to report VO position. Here is the really neat thing about this gesture; the VO position function is hardcoded. So you can, as I have done, set the 3 finger single tap gesture to be image recognition/description, and if you hold 3 fingers onto the screen, you get VO position.

Pretty cool, huh? 😃

Just a preview of what I get on my primary Home screen for a few random app icons, in no particular order:
Messages
Tapping 3 fingers (Image Recognition) An illustration of a speech bubble with white text on a black background. "Chat".
Holding 3 fingers (VO position) "Group. Messages. Row 1, Column 1. Home, page 1 of 2. Top of screen. Double tap to open".

Weather
(Image Recognition) An illustrationg of a cloud with white text on a black background.
(VO Position) "Group. Weather. Row 1, Column 4. Home, page 1 of 2. Top of screen. Double tap to open".

Clock
(Image Recognition) An illustration of a clock above text.
(VO Position) "Group. Clock. Row 2, Column 1. Home, page 1 of 2. Top of screen. Double tap to open".

Calculator
(Image Recognition) An illustration of a calculator above white text.
(VO Position)"Group. Calculator. Row 3, Column 1. Home, page 1 of 2. Center of screen. Double tap to open".

Settings
(Image Recognition) An illustration of a gray gear above white text on a black background.
(VO Position) "Group. Settings. Row 4, Column 4. Home, page 1 of 2. Center of screen. Double tap to open".

Mail
(Image Recognition) An illustration of an envelope in a blue square. "Mail".
(VO Position) Dock: Mail, in Dock. Position 2, bottom of screen. Double tap to open.
Phone
(Image Recognition) An illustration of a white phone in a green square. "Phone".
(VO Position) Dock: Mail, in Dock. Position 1, bottom of screen. Double tap to open.

These are admittedly very vague, however top this with something like Seeing AI or Be my AI, and you can get a detailed reference of every single icon on your screens.

HTH. 🙂

I've set this up now, and I really like it. The description I get for messages is a bit different to yours, not quite so detailed, and the colours for a lot of the descriptions are different to mine. But the general gist is the same. I wonder if it makes a difference what phone you have. I haven't got a pro, I'm on an iPhone 12 mini. My next phone will be a pro though. And yes, you could get some amazing descriptions with Be My AI. This sort of thing makes helping sighted users possible. You could always use Aira's Access AI to verify descriptions with an agent. I've done that a couple of times now, and the AI description was spot on each time. It's just nice to have that human verification, especially when you're preparing for a professional setting.

I neglected to mention above, that I always have my phone in dark mode. So that might change the descriptions a little, transitionally due to the nature of dark mode.
also I am on an iPhone SE3, if it helps any. 😀

Hi Brian,
I just tried that, but it didn't really help. I'm not too worried though, since I could just ask Be My AI anyway. I'll get a better description that way anyway. Thanks for your advice.

Yeah, Be my Ai is definitely one way to go about this, as you can take a screenshot and immediately go into image description with Be My Eyes from the Share sheet.

No idea why my descriptions are a little different, but then again, it is probably best to use Apple's image recognition as a starting point, rather than the final result. 😇

I'd never thought of customizing the gestures in that way. I like it, off to make the changes now.

Glad I could help. 😀

I have a feeling the differing levels of detail in VoiceOver's image descriptions is at least partly influenced by the machine learning performance and capabilities of the device's chip, as processing of this information happens entirely on-device. For example, the iPhone SE 3 is equipped with the A15, whereas the iPhone 12 Mini is equipped with the A14. While the chip differences are subtle and likely not noticeable when performing everyday tasks, machine learning workloads like image description could potentially benefit from the relative improvements in the slightly newer chip.

I've been wondering for a long time what became of that function. Thanks. My icons were slightly different too, but on an SE3 2022. Nothing significant, though. I think it was a green circle on yellow square with chat not a talking bubble on the messages app.

If you use the 4 finger single tap near the top or bottom of the screen, which will move VO to the top or bottom of the screen respectively, do not add custom gestures to the 4 finger single tap option, it will break the original function.

Fair warning.

Brian, this is an awesome modification. Thanks for telling us about it.

A couple notes on modifying the 3-finger tap gesture, as Brian described.

For me, it's not called Image Description / Recognition," it's called "Describe Image." And in order for it to do anything, you need to enable it in Accessibility, VoiceOver, Image Descriptions. Otherwise, the 3-finger tap will just give you the frustrating thunk sound.

I kinda feel like everyone should have enabled Image Recognition. Still, it is probably wise to mention it. 😳

She has been a journalist, a professor, a great writer of many things. Please give this a read. It is rather detailed and helpful. Please ask questions. I will ask them to her myself.
The following is, essentially, alt text for visual icons and iconography used so commonly on the iPhone they’ve become conventions— and because they have, designers overlook descriptions for the blind and visually impaired.

I hope these descriptions are helpful; if there are more you need, I’ll try to find time to add them to the list.

Safari iOS/Google front page
(A note on design, in case any reader is interested or needs this information: the iconography is minimalist, and if a user is in “light mode” — that is, reading black text and icons on a white background— then the top icons are black outline illustrations; if a user opts for the battery-saving “dark mode”, the Google icons on top are white, while the Safari icons below are light blue.)

When a new browser page is called up, the following icons are displayed in the following positions:

Top right (Google icons):
These icons are “flush right,” or “right justified,” which means they line up on the right hand side of the screen.

Going left to right:

● A bell icon, which is a simple but nevertheless very clear outline of a bell, from the crown head down to the bell bottom. The icon is drawn with outlines only, and represents notifications; if a user presses this with their finger directly, they are taken to a new page which reads either “no new notifications” or lists whatever notifications the user may have.
● A grid of 3 x 3 small squares, representing the many connected apps Google offers users as Our Benevolent Overlords.
● The last “icon” is whatever a user has chosen and uploaded for their own profile image. Tapping this will take you to “manage your Google account.”

Safari
There are five icons on the bottom of a standard Safari browser page.

The icons are outline illustrations, which has become the standard for these icons. In fact, a rule of effective visual design in general is “don’t make me think.” (There’s a highly-regarded book on the topic that uses that phrase as its title, if that tells you anything,)

None of these icons are large; they are each smaller than a pinky nail.

From left to right:

1 & 2: The first is a less than symbol, and the second is a greater than symbol, which to sighted people has come to mean, respectively, “go back” and “go forward” — or, using gestures, “swipe left,” or “swipe right.”

<
go back
gesture: swipe left

>
go forward
gesture: swipe right

3. The third icon is the “share” icon, and clicking it will take you to the standard choices of email, message, notes, etc,

This icon is perhaps the most… well, iconic… because it’s used so prolifically all over the online world. It’s universally recognized, and because of this, it’s probably why it hasn’t been labeled, so here’s what it looks like (spoiler alert) and it’s truly mediocre.

The share icon is a rectangle that’s a little taller than it is wide (think around the proportions of a sheet of printer paper), with an arrow pointing up that starts from the approximate center of the rectangle and crosses the top line.

4. The bookmarks icon looks like a drawing of an open book. When you click it, you get treated to more icons which head three tabs.

The left is the bookmark icon, repeated. The center is a pair of rounded eyeglasses, which is the reading list. The third and rightmost tab is a clock face, which displays your recent history.

5. The fifth icon is simply two squares laid offset on top of each other; this is another common icon— although it may sound odd.

Tapping this one lets a user zoom out and scroll through all open browser windows. It also serves up additional bottom row of icons (of course) like:

1. A plus sign: tapping this lets you open a new browser window
2. A “hamburger” (stacked) menu that lets you create new tab groups, private windows, etc
3. The word DONE.

Please let me know if you would like to have any other parts of the iPhone described. Thank you so much.