Screen Recognition

In all the times I’ve tried to use screen recognition to make something thats inaccessible become accessible its only worked once. I mean great that it did work that 1 time but not great that it almost never seems to improve things and almost always makes accessibility worse. It’s designed to OCR the screen and turn items it recognises into usable controls but that as I said just doesn’t work out for me. The time it did work was only because of a failing of voiceover to allow me to tap a button on a website so it managed to cover for a problem with web accessibility caused by voiceover as the button is accessible with JAWS. Others might have had different experiences and I’m interested to find out what others use it for but for me it has been a massive failure except 1 instance where voiceover was the failure and screen recognition saved the day.

sometimes, and some apps, turning it on swiping the screen, and then turning it off, makes the app actually work more or less. Screen recognition needs a lot of work. But occasionally it can help. It’s not a cure all.

The only use I ever had for Screen Recognition was when an update to the McDonalds app broke the ability to navigate and select individual promotions using VoiceOver. In that instance, it was able to successfully parse each listed promotion as a distinct element that I could then double-tap. Once that bug was fixed, I disabled screen recognition and haven't used it since. My general recommendation regarding Screen Recognition is that it should be used in the most narrow of apps and circumstances as possible, as it can degrade the accessibility of otherwise accessible interfaces, and with it being in the rotor by default, it can be just a little too easy to activate inadvertently.

The only instance I can get it to work is with chapters on Apple Podcasts. Otherwise, it's been a waste of time. I guess you just have to experiment for yourself and see what works and what doesn't.

I leave it turned off unless I'm encountering problems. Even then, it usually doesn't help.

I tried to use it with an app that was not accessible and did not do anything. It screw up my iPhone and I turn it off.

https://support.apple.com/en-gb/HT211899#:~:text=Go%20to%20Settings%20%3E%20Accessibility%20%3E%20VoiceOver,options%20to%20turn%20them%20on.

people are so conflated about screen recognition, image description and text recognition. the latter two should be left on, while screen recognition should only be use as the last resort.

something you cannot select like "agree or disagree", something not responding to tapping like the "next" button, or a switch bar where three or or more selections are on it, but voiceover just read all of them like a line of text? that's when you turn on screen recognition to deal with it. if you keep screen recognition on, you only make most apps unusable.

so screen recognition is like a third line therapy. you've tried everything and it didn't work, so you turn on screen recognition. when it works, it works like magic; but if it doesn't, nothing is lost. like the third line therapy, the patient is going to die anyway, so let's try something, maybe miracle will happen.

I've only found it useful in a few apps. Ironically, 2 of those are Apple's own built-in apps.

As someone said above, I've found it to help a little bit with navigating chapters in the native podcasts app. This used to work fine before iOS 16.4 I think it was, but remains broken as of the current beta of 16.6. I have reported it to Apple.

I also have to use SR in the music app to confirm that a spatial track is playing. The little Dolby atmos icon is supposed to show up right above the play button, but VO doesn't see it at all. I can find it by turning on SR and swiping around the screen. VO sees the lossless icon fine when I'm playing a song on my homepod though.

I also used to have to use SR to find the update button in the home app when there was a homepod software update, but this has since been fixed.

I've lately been using it in the door dash app to complete the check out process. VO sees the "buy with Apple pay" button fine, but double tapping it does nothing unless I turn on screen recognition first.

It’s supposed to make an accessible apps accessible, but screen recognition makes accessible apps in accessible, it messed up my Dice World once
,,,,,,,,,,,;((((

I believe text recognition is on by default, and you can check if it's on for you in Settings > Accessibility > VoiceOver > VoiceOver Recognition. Then, whenever you focus on an image with text, VoiceOver will attempt to analyze and speak the text content.

I just spent some time this afternoon attempting to reset my password on the client portal for the fitness studio in my building, and it seems that all or some of the VO Recognition features have to be turned off in order for this to work properly. I'm still figuring it out myself, but VO recognition seems like another very useful thing and I can definitely see the need for it. At first I was afraid the portal might've been inaccessible, but it doesn't appear that way.

I've used it from time to time to try new apps, it's useful and will probably become very useful in IOS 18.

I'm thinking they use AI and screen recognition together to make something interesting but we'll see how it goes.