Update 19 May, 2016: Facebook has today announced that automatic alt text is now available globally in English.
*** Facebook has today officially launched it’s much anticipated tool for automatically generating descriptions of photos posted to its site.
Called Automatic Alternative Text (AAT), the tool has been developed by Facebook’s Accessibility Team and uses object recognition technology to identify the contents of a photo. With photos such a core part of Facebook, this could transform the user experience for blind and low vision users, giving them the opportunity to know what others are liking and sharing.
According to a report by TechCrunch:
The object recognition powering Facebook’s AAT is based on a neural network with billions of parameters, and one that is trained with millions of examples. Neural networks are one type of model for machine learning. When it comes to images, you can think of a neural network as a pattern recognition system. In Facebook’s technology for AAT, it recognizes images and words in transportation (“car,” “boat,” “motorcycle,” etc.), nature (“outdoor,” “mountain,” “wave,” “sun,” “grass,” etc.), sports (‘tennis,” “swimming,” ‘stadium,” etc.) food (“ice cream,” “sushi,” “dessert,” etc.) and descriptive words for appearance (“baby,” “eyeglasses,” “smiling,” “jewelry,” “selfie,” etc.).
In a reply posted below, Matt King from Facebook’s Accessibility Team says that they are “… currently limiting the number of concepts to a small set, fewer than 100, where the system has been very well trained. Naturally, this number will grow over time. And, the accessibility team can provide input to the AI team regarding which concepts we think should have a special training focus.”
In a second reply, Matt states that recognition of text in images is also in Facebook’s plans for this tool:
The ability to recognize text is part of the back-end system, but we do not believe the results are yet good enough to turn it on for all images. We are surfacing the fact that the image may include text. And, when auto alt text is available in desktop and laptop browsers, people with the ability to perform OCR on those images could choose to do so. You should know, however, that we want to turn on Facebook OCR for all images that contain text as soon as we have enough confidence in the feature. At this time, I can't venture a guess as to when that will happen.
The BBC reports that “… Facebook would like the system to go one step further and use face recognition to identify people in a picture by name with help from their database of users, but others are resisting the idea on privacy grounds.”
AAT will initially only be available to users in the United States, Canada, the United Kingdom, Australia and New Zealand; and who are using English as their language. The tool should become available globally in a few weeks to anybody else using English language. Work on supporting other languages has already begun, with Facebook expecting to be able to announce more information soon.
For now, AAT is only available on the iOS app; with support for Android and Facebook’s website expected to be rolled out in the next few months.
If you are using English in one of the currently supported countries but not yet seeing descriptions being included with photos, it’s been reported that simply closing the Facebook app via the App Switcher and then re-opening it may get things working properly for you. However, Matt King has said below in the comments that this should not be necessary.
For more information about Facebook's Automatic Alt Text feature, check out this AppleVis Extra Interview with Matt King, Accessibility Specialist in UI Engineering at Facebook.
We would love to hear your personal experiences and views of this new tool, so please post a comment below.
Comments
Try updating
Try updating the Facebook app to the latest version.
Works now.
It's works for me since facebook version 54.0
Have Enjoyed the Concept
Although I have enjoyed the concept, I am never quite sure when a photo is described that it is truly accurate. I've asked my friends if they wouldn't mind posting what a photo really is so I could compare that with what FB was telling me. So far, I've not been successful with that as a couple of friends said they would but have neglected to do so. A couple examples have been, however, when one friend said it was two guide dogs playing, the photo recognized it as a dog and a person. Perhaps the one dog was standing on his/her back legs but don't know. Another time a photo said it was outdoors, a sky and a tree when in fact it was a person taking a selfy of herself showing her hair blowing in the wind commenting on how windy it was. Again, because i haven't been really successful in getting people to work with me on real descriptions vs. what FB is telling me, I think if not already being done, some comparisons with people who can really describe photos vs. what FB is saying might help. One time, it was right on, however, when a friend said she still had a Christmas tree set up in her office, good old FB said Christmas Tree. I thought that was cool.
Hope in an improvement, but this feature is not inclusion
Hello,
I have the descriptions feature here in Italy too, descriptions are in english Language but they have come here with latest app update.
I am perfectly aknoledged that it's a preliminary version, a beta which needs further improvements and it's just matter of time. Even because every artificial intelligence needs to roll out publicly and learn from people's usage, not just being tested by internal engineers.
But, whatever improvement it can have, it is just a first step, this is not real inclusion of visually impaired people, we should encourage sighted people to make us part of their social networking experience, by describing situations they are taking pictures of.
I mean:
if the picture represents a dog lying down on a sofa, a very clever artificial intelligence should say: "dog lying on the sofa".
People who just remember they have a visually impaired in their contact, will say: "for blind users: this image represents my dog lying on the sofa".
But, a person who really is interested in making us part of his/her life, would say:
Look at Argo. When he's on the sofa, there is no way to sit down, he takes all space for him and it's a challenge to try moving 40 kilos of furry beast from there. Poor dog, life is hard for him.
Not a solution
For users in countries where English is not an official language, providing image descriptions cannot be considered a solution, even if it is in beta form. Users who do not know English would likely be confused and possibly be annoyedat the information. For such users, it would be advisable for image descriptions to not be presented at all.
I have it
it's amazing. it needs to be improved though, but that's a great start. for those who don't have it yet. it says something like photo may contain food or photo may contain 1 person. it's fine. I hope it will be improved in the future
for non-english speaking users: just needed to flick away
Hi, I answer to the user talking about non-english speaking people.
Well, I am a non-english speaking person, I know the Language but it's not my native, but as far as descriptions are not Always so detailed, I just Flick away from the post when VoiceOver starts to say Photo, or a comment says "with a picture".
This should not be a problem, the real problem is that it is still in development and marketing campaigns have treated it as it was already advanced. Matt King is definitely right when he says Facebook had to launch the service on the market, so that feedbacks can be collected from a large public, it is the characteristic of artificial intelligences, to learn from a large usage.
But let's allow this service to grow up, and also add languages.
Great feature!
I find this feature terrific. It's Amazing what technology can do. I can't wait for the feature allowing recognition of text in images. Right now I need to save images and scan them with an OCR software installed on my Iphone, which doesn't get it right most of the time...Is there a quicker way?
Anyway it's cool.
Now on OS X As Well?
Hi everybody. I'm revisiting this topic to proudly report that it seems Facebook is expanding this feature. I was just on the Facebook page for my job to see how well I could navigate it with VoiceOver, and noticed that several of our photos contain alt text of some kind. I'm not sure how widely available this feature is yet on OS X, but big kudos to Mark Zuckerberg and Company.
So nearly a year later
Hi All, Facebook launched it's photo recognition system nearly a year since. Yet, still no sign of anything to translate text. This baffles me as surely if at the time they stated they hoped to launch as soon as possible something should be around now?. I've searched everywhere and can find absolutely nothing to indicate that A this is still on the cards and B any, even vague timescale. Has anyone heard anymore about this?
translation of auto alt text
Lee, as of now, auto alt text is translated to 27 languages. It has been so since last August.