Facebook Now Providing Automatically Generated Photo Descriptions to Blind and Low Vision Users on iOS

By AppleVis, 5 April, 2016

Member of the AppleVis Editorial Team

Update 19 May, 2016: Facebook has today announced that automatic alt text is now available globally in English.

*** Facebook has today officially launched it’s much anticipated tool for automatically generating descriptions of photos posted to its site.

Called Automatic Alternative Text (AAT), the tool has been developed by Facebook’s Accessibility Team and uses object recognition technology to identify the contents of a photo. With photos such a core part of Facebook, this could transform the user experience for blind and low vision users, giving them the opportunity to know what others are liking and sharing.

According to a report by TechCrunch:

The object recognition powering Facebook’s AAT is based on a neural network with billions of parameters, and one that is trained with millions of examples. Neural networks are one type of model for machine learning. When it comes to images, you can think of a neural network as a pattern recognition system. In Facebook’s technology for AAT, it recognizes images and words in transportation (“car,” “boat,” “motorcycle,” etc.), nature (“outdoor,” “mountain,” “wave,” “sun,” “grass,” etc.), sports (‘tennis,” “swimming,” ‘stadium,” etc.) food (“ice cream,” “sushi,” “dessert,” etc.) and descriptive words for appearance (“baby,” “eyeglasses,” “smiling,” “jewelry,” “selfie,” etc.).

In a reply posted below, Matt King from Facebook’s Accessibility Team says that they are “… currently limiting the number of concepts to a small set, fewer than 100, where the system has been very well trained. Naturally, this number will grow over time. And, the accessibility team can provide input to the AI team regarding which concepts we think should have a special training focus.”

In a second reply, Matt states that recognition of text in images is also in Facebook’s plans for this tool:

The ability to recognize text is part of the back-end system, but we do not believe the results are yet good enough to turn it on for all images. We are surfacing the fact that the image may include text. And, when auto alt text is available in desktop and laptop browsers, people with the ability to perform OCR on those images could choose to do so. You should know, however, that we want to turn on Facebook OCR for all images that contain text as soon as we have enough confidence in the feature. At this time, I can't venture a guess as to when that will happen.

The BBC reports that “… Facebook would like the system to go one step further and use face recognition to identify people in a picture by name with help from their database of users, but others are resisting the idea on privacy grounds.”

AAT will initially only be available to users in the United States, Canada, the United Kingdom, Australia and New Zealand; and who are using English as their language. The tool should become available globally in a few weeks to anybody else using English language. Work on supporting other languages has already begun, with Facebook expecting to be able to announce more information soon.

For now, AAT is only available on the iOS app; with support for Android and Facebook’s website expected to be rolled out in the next few months.

If you are using English in one of the currently supported countries but not yet seeing descriptions being included with photos, it’s been reported that simply closing the Facebook app via the App Switcher and then re-opening it may get things working properly for you. However, Matt King has said below in the comments that this should not be necessary.

For more information about Facebook's Automatic Alt Text feature, check out this AppleVis Extra Interview with Matt King, Accessibility Specialist in UI Engineering at Facebook.

We would love to hear your personal experiences and views of this new tool, so please post a comment below.

Options

Comments

By TJT 2001 on Sunday, April 24, 2016 - 06:19

This is the sort of thing that makes me feel excited. I deleted my Facebook account because too many photos without sufficient descriptions were appearing in my newsfeed. I may consider getting a new account to try this out. Does this work on the Facebook website?

By Karok on Sunday, April 24, 2016 - 06:19

How please, does this work? for me, if someone has put up 4 photos i still just hear photo photo photo etc. what am i doing wrong? even clicking on the photo just saying image? i am set to english using the facebook app in ios 9,3,1 thanks to all, William.

By Bingo Little on Sunday, April 24, 2016 - 06:19

I don't have any option allowing me to do this. I'm running the latest versions of both IOS and Facebook as far as I can tell. Only thing I'm thinking is that I am set to English UK. Do they in fact mean this will only work with US English?

By Karok on Sunday, April 24, 2016 - 06:19

It does work, in response to bingos post, set the actual language of your phone to u.s. and region to u.s. worked for me. whilst wont describe every nuance of a photo at least we get an idea of what a photo is. it cool!

By Krister Ekstrom on Sunday, April 24, 2016 - 06:19

I wonder how many decades it will take until it finally reaches Sweden? Shall we take a guess at around 2040?

By DMNagel on Sunday, April 24, 2016 - 06:19

I guess one must actually live in the US.

By Michael Hansen on Sunday, April 24, 2016 - 06:19

Member of the AppleVis Editorial Team

While I can't say whether this works for other languages besides English and other regions besides the United States, I did have to close the app in App Switcher and re-open it for the Automatic Alt Text to start appearing.

By Krister Ekstrom on Sunday, April 24, 2016 - 06:19

It's clearly stated in the blog post that for now this feature is only available for english language, hense my maybe somewhat sarcastic statement.

By Jesus R on Sunday, April 24, 2016 - 06:19

Just tried it a few minutes ago, took me a while to find a pot with a picture, everyone is sleeping in my part of the world, but I found it, and while its not the very discriptive things from TapTapSee, its enough for you to know whats in the picture. Close and open the app from the app switcher, or just weigt. I think I heard somewhere that its coming to other english speeking contries. Maybe its a rollout feature.

By Kathy Blackburn on Sunday, April 24, 2016 - 06:19

I live in the United States. My iPhone and Facebook Settings are set to English. I am running the latest versions of IOS and Facebook. I have powered my phone off and back on. I have closed Facebook and re-opened it. And I did just find a photo a friend posted day before yesterday. VoiceOver says, "Image may contain two people smiling." On the other hand, Seedlings Braille Books for Children just added five new photos to their Facebook page. On four out of five, VoiceOver says, "Image may contain text." On the fifth, VO says "No automatic alt-text available."

By Bingo Little on Sunday, April 24, 2016 - 06:19

Thanks for confirming my suspicions. Only available in US English I believe would be more accurate. Goodness knows what Facebook think our official language over here is! Thanks for the suggestion re changing my language and region but I don't want to do that just for the sake of Facebook. I'll wait for it to come to me rather than going to it. Completely sympathise with the comment re Sweden. it seems that some countries are always last and I would be very aggrieved if I were you.

By TamagotchiTune on Sunday, April 24, 2016 - 06:19

Its cool!!!! It describes how many people are in the photo. Even says if its a selfie lol. I'm wondering if it's supposed to read the text though? It says "contains text" but doesn't read it. Also what is alt text? Sometimes it says "no alt text available". Or how do I get it to read the text on the photo?

By JN.V on Sunday, April 24, 2016 - 06:19

I'm in South Africa, checked for any Facebook updates, closed the Facebook app and reopened it again, but just said image.
Then I changed my region to US and language to English US, closed and reopened the app again but still no luck.
I hope it's a rollout thing.

By DPinWI on Sunday, April 24, 2016 - 06:19

I have had this feature for a couple of weeks. Facebook must have opted me in. I also had a survey request and I filled it out. So far, I like it, but, it has its limitations.

In my experience, and noted in my survey response, I would like to have the text in a photo read. If it can tell there is text, then it would be nice to have it OCR'd.

I have not found it to be overly specific. For exampl,e I will hear things like "four people smiling" or, "Food on a plate." I wish it identified the people, but I'm probably good not knowing what the food is...

By Shannon on Sunday, April 24, 2016 - 06:19

Hiya, it's working perfectly here in the UK… I have set my language to British English!
Cheers,
Shannon

By TamagotchiTune on Sunday, April 24, 2016 - 06:19

I would like to have text read to. Especially with all these memes people post. But this just started. It will get better and better with time.

By Bingo Little on Sunday, April 24, 2016 - 06:19

I'm still determinedly in the UK and it's still not working for me. When did this Facebook update come out? I last updated Facebook on 2nd April or something like that.

By Joe on Sunday, April 24, 2016 - 06:19

Last night people were posting things from the basketball game facebook told me basketball court national champions banner pretty cool.

By blindgary1 on Sunday, April 24, 2016 - 06:19

The facebook aat works in the uk also , but does not know what a guide dog is just states this image may contain one person , clouds , sky , nature and i am sure i was smiling , a great first step can only improve

By bryan mcglashan on Sunday, April 24, 2016 - 06:19

Hi.

This is something great.

I've used it several times and I now feel like I know what people are liking etc.

The feature is in its very early stages so it can only get better.

I read on the BBC that facebook would like to try and match photos from people who use facebook and try and tell us who is in the photo. how cool is that?

Bryan

By Joseph on Sunday, April 24, 2016 - 06:19

Not working here either and I'm in the uS. will try to close and reopen the app though.

By Toonhead on Sunday, April 24, 2016 - 06:19

Well, to me, image may contain text, or image may comtain a plate of food isn't really a description at all, it's more of a guess. But, as everyone else is saying, give it time. This isn't ever going to be perfect, I think some of y'all are expecting too much too soon.

By Socheat on Sunday, April 24, 2016 - 06:19

My iPod set to English, I also try to restarted the Facebook app but not working. Do I need to change my region also?

By thunderhorse82 on Sunday, April 24, 2016 - 06:19

For those that say it doesn’t work, simply closing and opening Facebook may not work. They have said that the feature will be rolled out over the next two months and so all might not get it right away. Don’t worry, it will happen. I had it from the first day but my fiancé got it several weeks later.

By Socheat on Sunday, April 24, 2016 - 06:19

So I'll have to wait

By Toonhead on Sunday, April 24, 2016 - 06:19

For you guys who have to wait for this functionality, by the time you get it it might be even better than it is now. Like I say it's more info than we've had before and it's only getting started so hang tight. It'll work for everyone eventually.

By Bingo Little on Sunday, April 24, 2016 - 06:19

Closing and reopening the app didn't do it. I had to empty the app switcher and then reboot the iPhone. Now I have it, I'd say it's a start. The descriptions are not very informative really, but they are better than nothing. As a beginning it's not bad.

By Ekaj on Sunday, April 24, 2016 - 06:19

I'm excited to try out this feature when it comes to the Mac. I can't try it out at this time, as I don't own an iOS device. But this is so cool and those of you who haven't listened to the podcast accompanying this blog post should do so.

By Nina on Sunday, April 24, 2016 - 06:19

Hi,
I don't know why but I have gotten this feature from a previous FB update last month. I was not beta testing FB so I don't know what happened there.
It's an ok feature. Let's give it time to see if it improves.

By peter on Sunday, April 24, 2016 - 06:19

You can hear another interview with Matt King from the Facebook Accessibility team about this new feature as well as some demos by listening to this week's episode of Eyes On Success (to be posted 8:00 a.m. eastern time on Wednesday, April 6): 1615 Accessibility of Facebook Products (Apr. 6, 2016) Do you struggle using Facebook with a screen reader? Matt King, a member of the Accessibility Engineering Team at Facebook, describes many tips and hotkeys for using Facebook more efficiently. Hosts Nancy and Peter Torpey also talk with him about an exciting new feature launching this week to help make photos more accessible to people with vision loss. The show can be heard or downloaded from:

By JeffB on Sunday, April 24, 2016 - 06:19

It's working for me wish it was more descriptive. Neat though!

By Brooke on Sunday, April 24, 2016 - 06:19

Looks like I'm one of the unlucky ones who has to wait for it to be rolled out. I'm in the US, and have restarted the app, my iPhone, and uninstalled and reinstalled the app. Nothing yet.

By rdfreak on Sunday, April 24, 2016 - 06:19

I experienced it last night. It was saying 'image looks like, cat".
And then it was trying to differentiate between in and outdoors, I'm guessing as in if the photo was taken in or outdoors.
It does need to improve but I was surprised to all of a sudden get the extra information since I didn't know it had been released yesterday.

By Matt King on Sunday, April 24, 2016 - 06:19

In reply to by TJT 2001

TJT, the current launch of automatic alt text is for the IOS app. The Android app will be next followed by the Facebook web sites. I don't have set dates for each yet, but if all goes well, they will all be rolled out in the next month or two.

By Matt King on Sunday, April 24, 2016 - 06:19

In reply to by Krister Ekstrom

Krister, The English version will be available globally in a few weeks or less. Today it is in United States, Canada, United Kingdom, Australia, and New Zealand. Work for more languages has already begun and we should know dates availability soon.

You don't have to update your app to make it work. When it is available in your location it will just start appearing in your feed.

By Matt King on Sunday, April 24, 2016 - 06:19

If you hear "No automatic alt text available", that means that the Facebook object recognition engine attempted to analyze the image but was not confident enough about what it found to provide any alt text.

All the words you hear after the phrase "Image may contain" are what we refer to as concepts. We assign a minimum acceptable confidence level, AKA precision, to each concept. There are some concepts where we require a very high level of precision, say above0.95 on a scale of 0 to 1, and other concepts where we allow lower precision, e.g., 0.80.

If none of the concepts recognized by the system meet their minimum threshhold, we will generate the phrase "No automatic alt text available."

Further, we are currently limiting the number of concepts to a small set, fewer than 100, where the system has been very well trained. Naturally, this number will grow over time. And, the accessibility team can provide input to the AI team regarding which concepts we think should have a special training focus.

By Matt King on Sunday, April 24, 2016 - 06:19

One of the most critical decisions on this project was to decide when to go live -- when was it good enough?

At Facebook, we lean toward some value is better than no value and sooner is better than later. That said, going live too soon can be disasterous.

We prefer data-driven decisions so we conducted some user research and studies. The data seem to indicate that we could be providing value even with a relatively small number of concepts included in the automatic alt text, and even without text recognition, even though text in images is very common.

The ability to recognize text is part of the back-end system, but we do not believe the results are yet good enough to turn it on for all images. We are surfacing the fact that the image may include text. And, when auto alt text is available in desktop and laptop browsers, people with the ability to perform OCR on those images could choose to do so. You should know, however, that we want to turn on Facebook OCR for all images that contain text as soon as we have enough confidence in the feature. At this time, I can't venture a guess as to when that will happen.

Yesterday's launch was only the first of many steps down a long, new path. It is kind of like 30 years ago when OCR required a perfectly clean, flat page with crystal-clear block fonts. In a few years, or sooner, what we have today will seem like child's play. For now, we would like to step, learn, iterate, step, learn, iterate, etc. so we can make the future happen as fast as possible.

By Brooke on Sunday, April 24, 2016 - 06:19

Woohoo, I've got the feature now. I noticed it this morning!

By Paul Hunt on Sunday, April 24, 2016 - 06:19

For people it's not working for, you might try going to the updates tab of the app store and making sure the device has downloaded the latest version of the Facebook app. It's working for me now in the UK with UK region settings. Most of the photos just have "Image may contain x number of people" but I guess that's what mostly people take pictures of. I have had some photos with descriptions of water and sky and food and drink and articles of clothing. Definitley a positive step in the Facebook experience for blind users.

By JeffB on Sunday, April 24, 2016 - 06:19

Glad to hear there will be further updates. Perhaps to get around the privacy issue of saying people's names in posts to tell you who the person is there could be a prompt that people could enable or disable it? I say this because it would be neat to hear <Person name> doing such and such. Or perhaps it will only say the name if your friends with the person just some thoughts.

By mario_hardrock on Sunday, April 24, 2016 - 06:19

Hi Matt.
congratulations for this great work.
Matt, please, don't forget Portugal, and brazil.

make this feature available to Portugal and brazil.

thanks Matt. cheers.

By Tasha on Sunday, April 24, 2016 - 06:19

Hi, On the off-chance that a developer will see this: I sent a long email to Facebook accessibility about the possibility of the automated image descriptions being paired with a mechanism for human-generated image descriptions. It would almost be like Be my eyes, except the user could just click on a button next to the image and request more information. Also, unlike Be My Eyes, I think that blind people should form relationships with specific sighted describers, or at least have the image described by someone in their demographic, so that there's a better chance of salient details being captured. If anyone would be interested in developing something like this, I'd love to help as the idea girl (I can't code, unfortunately). Tasha dot chemel at gmail.

By Kira McCall on Sunday, April 24, 2016 - 06:19

Hello all,
Reading the comment about human and automated descriptions together made me think of this: Why can't I add captions on my phone when posting a photo?
Whenever I took a picture and wanted to add it to my Facebook status using the Facebook app, there was a text field to add a caption to the photo so that VO wouldn't just say, "Photo." This was different from the text field labeled "Say something about this photo...".
Captions can still be added on the Facebook website, but I can no longer add captions using my phone, which is where I do 99% of my Facebook activity.
Any help with this would be greatly appreciated. :)
Kira

By Jessica on Sunday, April 24, 2016 - 06:19

Hi everyone, first of all, let me say that I've got the feature working, and I like it so far as a first step. I think we all very much appreciate the efforts in this area! I agree with the comments above, in that it would be nice to have our friends add captions, which become alt text. I use Zoom along with VoiceOver since I have some residual vision, and I have to say that often what I see is a lot more than what is automatically described. I think it would be a great step forward if humans could add descriptions to pictures, and have these descriptions help “train” the computer algorithm. For instance, one of my friends posted a stunning shot of a dolphin in mid-jump. The description said, “Image may contain ocean”, but nothing about a dolphin. If I could have added that, and if enough people added that on other pictures of dolphins, then maybe the neural network could learn what a dolphin looks like. Just a few thoughts. And again, this is only a suggestion for how we can add details in the future. This is all very exciting!

By Troy on Sunday, April 24, 2016 - 06:19

One thing I'd like to see the accessibility team at fb work on as well is when someone sends you a sticker right now vo will just say sticker, it would be nice to know what that sticker is.

By Fatima.Hamoud10 on Sunday, April 24, 2016 - 06:19

In reply to by TJT 2001

Why not just reactivate your old facebook account instead of having to create an new one? That would be much easier instead of going through so much information. To make your new account you would have to add your birthday, email address, new password and so on. Just reactivate your old account again. You do that by loggin in with your email address and password. Or did you delete your facebook account permenantly?