Seeing AI

Category

Description of App

Seeing AI is a free app that narrates the world around you. Designed with and for the blind and low vision community, this ongoing research project harnesses the power of AI to open up the visual world by describing nearby people, text and objects.

Seeing AI provides tools to assist with a variety of daily tasks:

• Short Text - Speaks text as soon as it appears in front of the camera.

• Documents - Provides audio guidance to capture a printed page, and recognizes the text, along with its original formatting.

• Products - Scans barcodes, using audio beeps to guide you; hear the name, and package information when available.

• People - Saves people’s faces so you can recognize them, and get an estimate of their age, gender, and expression.

• Currency - Recognizes currency notes.

• Scenes - Hear an overall description of the scene captured. Explore the photo by moving your finger over the screen to hear the location of different objects.

• World - An Audio Augmented Reality experience to explore an unfamiliar environment, including hearing objects announced around you with Spatial Audio (requires a device with a LiDAR, and iOS 14+).

• Indoor Navigation - Available on the World Channel, enables you to create routes through a building, like "entrance to classroom", and navigate by following the sound (requires a device with an A9 or later processor, and iOS 14+).

• Colors - Identifies colors.

• Handwriting - Reads handwritten text like in greeting cards (available in a subset of languages).

• Light - Generates an audible tone corresponding to the brightness in the surroundings.

• Images in other apps - Just tap “Share” and “Recognize with Seeing AI” to describe images from Mail, Photos, Twitter, and more.

• Browse Photos - Hear descriptions of photos saved on your device.

Seeing AI continues to evolve as we hear from the community, and AI research advances.

Check out tutorials with this YouTube playlist: http://aka.ms/SeeingAIPlaylist.

Questions, feedback or feature requests? Email us at SeeingAI@Microsoft.com.

Version

5.4

Free or Paid

Free

Apple Watch Support

No

Device(s) App Was Tested On

iPhone

iOS Version

18.1

Accessibility Comments

This app is very accessible

VoiceOver Performance

VoiceOver reads all page elements.

Button Labeling

Most buttons are clearly labeled.

Usability

The app is fully accessible with VoiceOver and is easy to navigate and use.

Other Comments

This is an amazing app! I was pleasantly surprised to learn about this app from mainstream tech sites this morning.

Recommendations

69 people have recommended this app

Most recently recommended by Casey 2 weeks 5 days ago

Options

Comments

By Erion on Friday, July 21, 2017 - 09:31

The fact that one can read a print book live is quite impressive, but that's not everything. This application is truly amazing.

By Joseph on Friday, July 21, 2017 - 09:31

I just got it. And wow! Very, very nice. This one and taptapsee are probably gonna be my go-to things.

By jrjolley (not verified) on Friday, July 21, 2017 - 09:31

It'll be interesting to see how this app works out. I question it being free though but we'll see. It's a bit surprising that it's not available worldwide, doesn't look good in 2017 when we're all mobile all of the time.

It's Microsoft. I don't foresee MS ever charging for this app. It's using AI technology that isn't just designed for the blind.

By jrjolley (not verified) on Friday, July 21, 2017 - 09:31

In reply to by Carlos Taylor

Don't you think it looks a bit wierd that MS launch this in just a few countries? Surely if this is all just AI, why not just release globally - seems strange but we'll see what happens.I think Prizmo's using MS OCR for the cloud OCR option, I used to use it when it was project oxford.

By JeffB on Friday, July 21, 2017 - 09:31

This is great wow! I'm impressed!

By Toonhead on Friday, July 21, 2017 - 09:31

I hate to rain on everyone's parade here but so far, I ain't impressed. I tried, for about 15 minutes to get it to recognize different bar codes, and after following the instructions in the help, my phone beeped just as the instructions said it would, but that's all it did, and it heated up my phone to the point where I thought it was going to shut off but luckily, it didn't. Then again I am not very good with using my camera so it could certainly be operator error. To all of you having good luck with the app, yay! But for me, it's about as useless as a paperweight, at this point anyways.

By Kevan on Friday, July 21, 2017 - 09:31

This is excellent! I played around with it for a few minutes and I'm very impressed. I especially like making funny faces at the camera and getting it to recognize me. Only thing is it ages me around 10 years, LOL.

I'd update the AppleVis app listing to include the full description from the appstore.

By JeffB on Friday, July 21, 2017 - 09:31

How close are you holding the phone? When I got it to read the barcode I held it about a half a foot or a foot away from the object. Then I turned the object slowly in 1 direction.

By Carlos Taylor on Friday, July 21, 2017 - 09:31

In reply to by Kevan

Yes, I plan to update the apps description when I get home. I was in the car leaving town when I posted it this morning. I was so excited about this app that I wanted to get the word out as soon as possible. I will update the app description later this evening when I get home.

By Carlos Taylor on Friday, July 21, 2017 - 09:31

I will say that scanning barcodes is a feature I haven't tried yet. I will do so later when I get home. However, taking pictures of your surroundings, reading text off of a printed page, and recognizing people, all works great here. It is amazing how it recognizes text in a live video stream. This is great as it can start reading much quicker and doesn't rely on you to snap the picture as it can do so automatically when using the document mode.

By Dawn 👩🏻‍🦯 on Friday, July 21, 2017 - 09:31

Hi! I downloaded this as soon as I heard about it! I really love it! I can't wait to play with it and try it in different circumstances! That being said, I do have a couple questions.

1. How do you change the camera from back to front facing in the app? Is that even something you can do in the Seeing AI. app, or do you have to go to the camera app and change it?

2. I'm having an issue with the product channel. I select it, but then I can't take a pic and scan the barcode. Any help would be appreciated! I can't wait until someone does a podcast on this app! I listened to the tutorial videos and found them helpful but I couldn't find the solutions to the issues I'm having.

Thans for any help in advance! Also, many thanks for sharing this app! I'm glad Microsoft made this, and I'm looking forward to the updates to this app and what knew channels that are added!

By Greg Wocher on Friday, July 21, 2017 - 09:31

Hello,
When you are scanning barcodes there is not a take picture button or scan button. All you need to do is move the product under the camera and it starts scanning automatically. The app will start beeping when you get close to the barcode. I believe the faster the beep the closer to the barcode you are. I really am liking this app. It is working much better for me than digit eyes does.

Regards,
Greg Wocher

By Remy on Friday, July 21, 2017 - 09:31

Okay, maybe not, but this thing is all sorts of amazing so far. So far I've only used facial recognition and the live text reader. I'm extremely impressed with the live text reader. Obviously no private documents unfortunately, but wow, everything else! It's not super accurate all the time yet, but I think a part of that is because you need to hold the camera very still. I could even read bits of a computer screen with it, as well as random notes and pieces of paper around the office. Considering this is 1.0, I am highly impressed.

By JeffB on Friday, July 21, 2017 - 09:31

I watched a video that said you can find out what pictures are from Facebook and other things like that. However I did not see Scan with Seeing AI when I picked a photo from messages. Has anyone had any luck doing this? If not I guess its something coming soon.

By Carlos Taylor on Friday, July 21, 2017 - 09:31

In reply to by JeffB

I opened my photos app and shared a few pics to seeing AI. I was amazed with the descriptions I got! Not sure how to make use of Facebook photos.

By JeffB on Friday, July 21, 2017 - 09:31

I can't find that option either in the photos app.

By Mattie SV on Friday, July 21, 2017 - 09:31

Hello all!!!

I am really loving this app so far however it really heats up my phone and it gets really hot and the battery drains fast!!! However overall I really love it!!! I really love the short text mode I really love when it reads live instead of me having to take a picture!!! So far it is really great and I'm looking forward to it getting better and better!!!

By Carlos Taylor on Friday, July 21, 2017 - 09:31

In reply to by JeffB

Locate a picture in the photos app. Double tap the picture. You should find share in the bottom left corner of the screen. Seeing AI should appear as one of the share options.

By Troy B on Friday, July 21, 2017 - 09:31

I've just installed this app, but when I try to take a picture of myself using the frontfacing camera the app never sees my face when I snap the picture. Where is the camera on the front of an iPhone 7?

By Chris on Friday, July 21, 2017 - 09:31

Hi,

I can't find the product category. When I swipe up on the slider, I have short text, document, person and scene. Why isn't the product category showing up? I'm using an iPhone 5S running iOS 10.3.2.

By smhy on Friday, July 21, 2017 - 09:31

I also cannot find the extension for this app in the share sheet. I'm on an iPhone 6 running 10.3.1. But I do have the product channel.

By That Blind Canuck on Friday, July 21, 2017 - 09:31

As instructed earlier, open the Photos app, select a photo, select the Share option near the bottom, but in order to have it appear, go at the bottom of the list, select the More option. From here, find the option to Open in SeeingAI, and turn it on. It seems to be turned off by default. You can even reorder the list of options on the Share Sheet this way. I've positioned it higher on the list of options. I'm guessing that this may be the same in other apps, you may have to enable it in other Share Sheets. HTH!

By JeffB on Friday, July 21, 2017 - 09:31

I found it thanks for the help!

By smhy on Friday, July 21, 2017 - 09:31

Yes, I do know how to use the more menu in the share sheet. It is not there.

By Holger Fiallo on Friday, July 21, 2017 - 09:31

Please will be nice if someone did a podcast. For a strange reason I can not get the idea of taking a picture. Back and front camera. I am good with technology but taking photos is hard to get for me.

By LaBoheme on Friday, July 21, 2017 - 09:31

so what is your favorite channel? mine is facial recognition. it seems to work pretty good, it can also recognize face in a photo. right now it only works with physical photos, i.e. paper photos, and not pictures on your phone.

By DMNagel on Friday, July 21, 2017 - 09:31

Who is to blame when an app is not in all countries? Is it Apple or is it The developer?

By That Blind Canuck on Friday, July 21, 2017 - 09:31

If some are looking for a podcast that talks about this app, Cool Blind Tech has just released an episode this week with Jessica Rickers going through how to use the app. As for availability, it appears to be available in North America as here in Canada, we had it.

By DPinWI on Friday, July 21, 2017 - 09:31

I read in the description that the app can identify currency. I assume this would be in the Product channel, but had no luck. I tried the Short Text, and i could read some words, but still no luck with identifying the bill. I tried Canadian and American bills.

Anyone else?

By mendi on Friday, July 21, 2017 - 09:31

I used it successfully at first and had a great time snapping pics of my team at work. But then, I tried it later that evening and/or yesterday. The app would say the verbiage about one face and where it was in the frame, but then when I snapped a pic it came back and said no people were detected. I'm baffled by that, as it said it saw a face but then when the pic is snapped nothing? Nonetheless, I do think this app has great potential, and I am looking forward to doing more with it!

By DPinWI on Friday, July 21, 2017 - 09:31

On some of the screens, I hear reference to a help video. While the app is simple and easy to navigate, I am interested in the video. however, I have yet to get it to play. Has anyone had success?

By Jesse Anderson on Friday, July 21, 2017 - 09:31

I wondered about this too the other day, and tried the various channels. I learned via Twitter that currency identification isn't yet in the current build but coming soon.

By Mattie SV on Friday, July 21, 2017 - 09:31

Hello!

I have successfully watched all of the videos! Most of the time it does work when I push the link for play video but if not if you have a 3D Touch capable iPhone I would 3D Touch on the link and push start playback or sometimes it will just start playing on its own unfortunately I have no experiences with non 3D Touch iPhones I hope this helps!

By Justin on Friday, July 21, 2017 - 09:31

Hi,
I've successfully gotten all the vids to play on my end. You select the category you want, find the help button, then there should be a button for playing the vid. Hit that, then your default video controls should appear, hit start playback and the video should automatically start playback.

By DPinWI on Friday, July 21, 2017 - 09:31

Thanks for the ideas about the videos. I have an iPhone 7, but did not try 3d touch. I will give that a go. All other forms of tapping, split tapping, and pass through tapping did not work for me.

By Toonhead on Friday, July 21, 2017 - 09:31

Here's exactly how to access the video tutorials, no 3d touch required. 1. With seeing AI open, double tap on the menu button. 2. Next, double tap on the help icon. 3. on the next screen, there'll be some text describing what Seeing AI is, and if you keep swiping you'll find a link that says "play the video tutorial. This launches the one that gives an overview of the entire app. Double tap on that link, and next, double-tap on the start playback button. You can do this for each editional video tutorial in the app. Just remember that there's a play video tutorial link for each video. I hope this helps!

By Troy B on Friday, July 21, 2017 - 09:31

I haven't tried much with this app yet, but I did try to take a picture of my surroundings while outdoors earlier and it told me it looked like a flock of birds sitting on something wood. Well it was half right because it was a wooden deck, but the flock of birds was actually a bunch of flowers.

I may not use that part of the app a lot, but I do think it'll be a great addition to the other apps on my phone.

By Remy on Friday, July 21, 2017 - 09:31

That is one feature I wish readers like this would be able to get better about. I can't tell you how many times I want to read a block of text on my 32 inch HDTV only and end up having various degrees of success. This one sort of does it, but even still, not very well.

By Karok on Friday, July 21, 2017 - 09:31

Hi all, i am sure it will come to the uk but we won't know when and it is a bit strange that it's not world-wide you would have thought a company like microsoft could handle requests. also i hope it comes to android to

By Jordan on Friday, July 21, 2017 - 09:31

In reply to by Erion

I heard about this app right about the time I was leaving for the NFB convention and was going to wait to get it. Well, it was during the convention that I ended up downloading it and it is now my go to app when I need to figure things out.

By Becca on Friday, July 21, 2017 - 09:31

This app is great! I was able to read my computer screen and the description was pretty accurate. Great app!

By Troy B on Friday, July 21, 2017 - 09:31

I just went to the product channel in seeing AI and scanned the side of a can that I knew had cashew nuts in it and I found the bar code easily and the app identified the product, but then I got a box of rice and although the bar code was detected and I believe that the code was in good focus given how fast the app was beeping the product was never identified. Is this just due to the work that still needs to be done with the app?

I will say that I've tried 2 or 3 different bar code readers in the past, both on the iPhone and otherwise, and given how easily I just found the bar codes on these 2 products I just tried seeing AI is by far the easiest thing I've used for finding bar codes and I'm very excited to test the app further.

By JeffB on Friday, July 21, 2017 - 09:31

Does anyone know if bar codes are universal or have any overlap? When I scanned the bar code on a spray can of sunscreen it seemed to think it was Bunratty Botcheen which turns out to be a type of Irish moonshine. I did submit this under feedback but was just wondering if there could be some overlap when it came to bar codes?

By Troy B on Friday, July 21, 2017 - 09:31

It would be nice if somebody could do a walk-through podcast on this site for seeing AI. Somebody did a podcast on CoolBlindTech, but I've listened to it and didn't get much out of it because the person is talking as she's flicking through the different areas on her phone so it's hard to hear what's going on.

By Sean Terry on Friday, July 21, 2017 - 09:31

Hi all, I just spoke with one of the app developers on Twitter. They said that a fix for iOS 11 beta will be coming soon. I also suggested a nice feature that would really help out. What do you think about this! A prescription barcode reader! There are a few of these, but I think it would be very useful for this app and especially iOS to have something like this!

I searched for it and could not find.; Where I can find it? Did search typing blind cool tech and nothing about it.