Facebook Now Providing Automatically Generated Photo Descriptions to Blind and Low Vision Users on iOS

This is the sort of thing that makes me feel excited. I deleted my Facebook account because too many photos without sufficient descriptions were appearing in my newsfeed. I may consider getting a new account to try this out. Does this work on the Facebook website?

How please, does this work? for me, if someone has put up 4 photos i still just hear photo photo photo etc. what am i doing wrong? even clicking on the photo just saying image? i am set to english using the facebook app in ios 9,3,1 thanks to all, William.

I don't have any option allowing me to do this. I'm running the latest versions of both IOS and Facebook as far as I can tell. Only thing I'm thinking is that I am set to English UK. Do they in fact mean this will only work with US English?

Probably a release bug.

It does work, in response to bingos post, set the actual language of your phone to u.s. and region to u.s. worked for me. whilst wont describe every nuance of a photo at least we get an idea of what a photo is. it cool!

I wonder how many decades it will take until it finally reaches Sweden? Shall we take a guess at around 2040?

I guess one must actually live in the US.

While I can't say whether this works for other languages besides English and other regions besides the United States, I did have to close the app in App Switcher and re-open it for the Automatic Alt Text to start appearing.

It's clearly stated in the blog post that for now this feature is only available for english language, hense my maybe somewhat sarcastic statement.

Just tried it a few minutes ago, took me a while to find a pot with a picture, everyone is sleeping in my part of the world, but I found it, and while its not the very discriptive things from TapTapSee, its enough for you to know whats in the picture. Close and open the app from the app switcher, or just weigt. I think I heard somewhere that its coming to other english speeking contries. Maybe its a rollout feature.

I live in the United States. My iPhone and Facebook Settings are set to English. I am running the latest versions of IOS and Facebook. I have powered my phone off and back on. I have closed Facebook and re-opened it. And I did just find a photo a friend posted day before yesterday. VoiceOver says, "Image may contain two people smiling." On the other hand, Seedlings Braille Books for Children just added five new photos to their Facebook page. On four out of five, VoiceOver says, "Image may contain text." On the fifth, VO says "No automatic alt-text available."

Thanks for confirming my suspicions. Only available in US English I believe would be more accurate. Goodness knows what Facebook think our official language over here is! Thanks for the suggestion re changing my language and region but I don't want to do that just for the sake of Facebook. I'll wait for it to come to me rather than going to it. Completely sympathise with the comment re Sweden. it seems that some countries are always last and I would be very aggrieved if I were you.

Its cool!!!! It describes how many people are in the photo. Even says if its a selfie lol. I'm wondering if it's supposed to read the text though? It says "contains text" but doesn't read it. Also what is alt text? Sometimes it says "no alt text available". Or how do I get it to read the text on the photo?

I'm in South Africa, checked for any Facebook updates, closed the Facebook app and reopened it again, but just said image.
Then I changed my region to US and language to English US, closed and reopened the app again but still no luck.
I hope it's a rollout thing.

I have had this feature for a couple of weeks. Facebook must have opted me in. I also had a survey request and I filled it out. So far, I like it, but, it has its limitations.

In my experience, and noted in my survey response, I would like to have the text in a photo read. If it can tell there is text, then it would be nice to have it OCR'd.

I have not found it to be overly specific. For exampl,e I will hear things like "four people smiling" or, "Food on a plate." I wish it identified the people, but I'm probably good not knowing what the food is...

Hiya, it's working perfectly here in the UK… I have set my language to British English!
Cheers,
Shannon

I would like to have text read to. Especially with all these memes people post. But this just started. It will get better and better with time.

I'm still determinedly in the UK and it's still not working for me. When did this Facebook update come out? I last updated Facebook on 2nd April or something like that.

Last night people were posting things from the basketball game facebook told me basketball court national champions banner pretty cool.

The facebook aat works in the uk also , but does not know what a guide dog is just states this image may contain one person , clouds , sky , nature and i am sure i was smiling , a great first step can only improve

Hi.

This is something great.

I've used it several times and I now feel like I know what people are liking etc.

The feature is in its very early stages so it can only get better.

I read on the BBC that facebook would like to try and match photos from people who use facebook and try and tell us who is in the photo. how cool is that?

Bryan

Not working here either and I'm in the uS. will try to close and reopen the app though.

Well, to me, image may contain text, or image may comtain a plate of food isn't really a description at all, it's more of a guess. But, as everyone else is saying, give it time. This isn't ever going to be perfect, I think some of y'all are expecting too much too soon.

My iPod set to English, I also try to restarted the Facebook app but not working. Do I need to change my region also?

For those that say it doesn’t work, simply closing and opening Facebook may not work. They have said that the feature will be rolled out over the next two months and so all might not get it right away. Don’t worry, it will happen. I had it from the first day but my fiancé got it several weeks later.

So I'll have to wait

For you guys who have to wait for this functionality, by the time you get it it might be even better than it is now. Like I say it's more info than we've had before and it's only getting started so hang tight. It'll work for everyone eventually.

Closing and reopening the app didn't do it. I had to empty the app switcher and then reboot the iPhone. Now I have it, I'd say it's a start. The descriptions are not very informative really, but they are better than nothing. As a beginning it's not bad.

I'm excited to try out this feature when it comes to the Mac. I can't try it out at this time, as I don't own an iOS device. But this is so cool and those of you who haven't listened to the podcast accompanying this blog post should do so.

Hi,
I don't know why but I have gotten this feature from a previous FB update last month. I was not beta testing FB so I don't know what happened there.
It's an ok feature. Let's give it time to see if it improves.

You can hear another interview with Matt King from the Facebook Accessibility team about this new feature as well as some demos by listening to this week's episode of Eyes On Success (to be posted 8:00 a.m. eastern time on Wednesday, April 6): 1615 Accessibility of Facebook Products (Apr. 6, 2016) Do you struggle using Facebook with a screen reader? Matt King, a member of the Accessibility Engineering Team at Facebook, describes many tips and hotkeys for using Facebook more efficiently. Hosts Nancy and Peter Torpey also talk with him about an exciting new feature launching this week to help make photos more accessible to people with vision loss. The show can be heard or downloaded from:

It's working for me wish it was more descriptive. Neat though!

Great what kind of food. lol

Looks like I'm one of the unlucky ones who has to wait for it to be rolled out. I'm in the US, and have restarted the app, my iPhone, and uninstalled and reinstalled the app. Nothing yet.

I experienced it last night. It was saying 'image looks like, cat".
And then it was trying to differentiate between in and outdoors, I'm guessing as in if the photo was taken in or outdoors.
It does need to improve but I was surprised to all of a sudden get the extra information since I didn't know it had been released yesterday.

If you hear "No automatic alt text available", that means that the Facebook object recognition engine attempted to analyze the image but was not confident enough about what it found to provide any alt text.

All the words you hear after the phrase "Image may contain" are what we refer to as concepts. We assign a minimum acceptable confidence level, AKA precision, to each concept. There are some concepts where we require a very high level of precision, say above0.95 on a scale of 0 to 1, and other concepts where we allow lower precision, e.g., 0.80.

If none of the concepts recognized by the system meet their minimum threshhold, we will generate the phrase "No automatic alt text available."

Further, we are currently limiting the number of concepts to a small set, fewer than 100, where the system has been very well trained. Naturally, this number will grow over time. And, the accessibility team can provide input to the AI team regarding which concepts we think should have a special training focus.

One of the most critical decisions on this project was to decide when to go live -- when was it good enough?

At Facebook, we lean toward some value is better than no value and sooner is better than later. That said, going live too soon can be disasterous.

We prefer data-driven decisions so we conducted some user research and studies. The data seem to indicate that we could be providing value even with a relatively small number of concepts included in the automatic alt text, and even without text recognition, even though text in images is very common.

The ability to recognize text is part of the back-end system, but we do not believe the results are yet good enough to turn it on for all images. We are surfacing the fact that the image may include text. And, when auto alt text is available in desktop and laptop browsers, people with the ability to perform OCR on those images could choose to do so. You should know, however, that we want to turn on Facebook OCR for all images that contain text as soon as we have enough confidence in the feature. At this time, I can't venture a guess as to when that will happen.

Yesterday's launch was only the first of many steps down a long, new path. It is kind of like 30 years ago when OCR required a perfectly clean, flat page with crystal-clear block fonts. In a few years, or sooner, what we have today will seem like child's play. For now, we would like to step, learn, iterate, step, learn, iterate, etc. so we can make the future happen as fast as possible.

Woohoo, I've got the feature now. I noticed it this morning!

For people it's not working for, you might try going to the updates tab of the app store and making sure the device has downloaded the latest version of the Facebook app. It's working for me now in the UK with UK region settings. Most of the photos just have "Image may contain x number of people" but I guess that's what mostly people take pictures of. I have had some photos with descriptions of water and sky and food and drink and articles of clothing. Definitley a positive step in the Facebook experience for blind users.

Glad to hear there will be further updates. Perhaps to get around the privacy issue of saying people's names in posts to tell you who the person is there could be a prompt that people could enable or disable it? I say this because it would be neat to hear <Person name> doing such and such. Or perhaps it will only say the name if your friends with the person just some thoughts.

Hi Matt.
congratulations for this great work.
Matt, please, don't forget Portugal, and brazil.

make this feature available to Portugal and brazil.

thanks Matt. cheers.

Hi, On the off-chance that a developer will see this: I sent a long email to Facebook accessibility about the possibility of the automated image descriptions being paired with a mechanism for human-generated image descriptions. It would almost be like Be my eyes, except the user could just click on a button next to the image and request more information. Also, unlike Be My Eyes, I think that blind people should form relationships with specific sighted describers, or at least have the image described by someone in their demographic, so that there's a better chance of salient details being captured. If anyone would be interested in developing something like this, I'd love to help as the idea girl (I can't code, unfortunately). Tasha dot chemel at gmail.

Hello all,
Reading the comment about human and automated descriptions together made me think of this: Why can't I add captions on my phone when posting a photo?
Whenever I took a picture and wanted to add it to my Facebook status using the Facebook app, there was a text field to add a caption to the photo so that VO wouldn't just say, "Photo." This was different from the text field labeled "Say something about this photo...".
Captions can still be added on the Facebook website, but I can no longer add captions using my phone, which is where I do 99% of my Facebook activity.
Any help with this would be greatly appreciated. :)
Kira

Hi everyone, first of all, let me say that I've got the feature working, and I like it so far as a first step. I think we all very much appreciate the efforts in this area! I agree with the comments above, in that it would be nice to have our friends add captions, which become alt text. I use Zoom along with VoiceOver since I have some residual vision, and I have to say that often what I see is a lot more than what is automatically described. I think it would be a great step forward if humans could add descriptions to pictures, and have these descriptions help “train” the computer algorithm. For instance, one of my friends posted a stunning shot of a dolphin in mid-jump. The description said, “Image may contain ocean”, but nothing about a dolphin. If I could have added that, and if enough people added that on other pictures of dolphins, then maybe the neural network could learn what a dolphin looks like. Just a few thoughts. And again, this is only a suggestion for how we can add details in the future. This is all very exciting!

One thing I'd like to see the accessibility team at fb work on as well is when someone sends you a sticker right now vo will just say sticker, it would be nice to know what that sticker is.