Introducing Be My Eyes' Virtual Volunteer: A Demonstration of Some Game Changing Real-World Use Cases

How did you gain access? You receive an invitation by e-mail, or notification from application?

If you open the Be My Eyes app and find the Be My AI tab, there will be a button to join the waiting list for the beta. They are really ramping up the pace now so hopefully the wait shouldn’t be too long.

“I am on the waiting list, but I don’t know how they will inform me about access to the AI.”

Hey,
As cool as those examples are, be careful with things like expiry dates and buttons on the oven panel. It has been known to completely fabricate these kinds of things, while sounding very convincing.
Be My Eyes themselves say it does not do text recognition well, and makes stuff up, though they are trying to improve it.
It does some stuff great though.

One feature I'm surprised is not. There is the ability to save a photo after taking a picture within the app! My other thought is I think adult content should be allowed! There's so many apps censoring these type of things, and I get it from a perspective of they don't want certain types of images maybe on their servers. However, I feel like sometimes with blind products, we miss out on the sexual side of things. I'm not so sure the stereotype isn't still in place that blind folks aren't sexual. I've heard this a lot from being around the community for many years and I think it's a wrong take to have.

Both OCR before passing to Open AI and saving photos are on the roadmap for Be My eyes so they’ll be in the app as soon as the devs get those features to where they’re happy with them. All LLM’s seem to produce incorrect information with absolute confidence and its a problem Be My Eyes are very aware of. OCR is going to help in the short term but in the longer term there are features they are working on to reduce the chance of it happening. Re blind people being sexual and having the right to that information like anyone else has. Do you mind if I post your comment to the feedback forum Be My eyes are running? I think it’s a great point that needs amplifying. I’m not sure if they’ll be able to do anything with it for obvious reasons but I think it should be heard and given serious consideration.

Hi all. I signed up for the beta waiting list for Be My AI when I downloaded this app a few months ago. I just updated the app a bit ago and my beta access is still pending. I don't see a Be My AI tab. I'm up to version 4.6 . But this new feature sounds incredibly cool and I can't wait to start testing it out. Perhaps it will be able to recognize more of the inaccessible thermostat in my apartment better than the human volunteer or a neighbor did, lol. They were actually great, but neither of us could quite do it. Regarding facial expressions, I believe I saw mention of an episode of the Be My Eyes podcast where this is discussed. I have some questions about that, which I won't ask here since they're unrelated to the topic at hand. But I'm definitely going to check out the Be My Eyes podcast.

If you don’t get the answers you are looking for from the podcast, feel free to ask and I’ll help if I can. I’ve been on the beta for over 5 months now so theres a good chance I’ll be able to help and if not, I will be able to speak to people who will definitely be able to help. If you registered for the beta a few months ago it shouldn’t be long until you’re at the top of the list. The rate they are adding people is really picking up. They are over 3500 now with hundreds more per day being added.

I mentioned before that I sent them several emails and got put on the beta. I also told a friend to do that, he did it and the next day he was put on the beta so you can send them an email through the app on the last tab where the feedback button is and let them know that you’ve signed up already a long time ago and that your eager to start beta testing and I’m sure they’ll reply back to you and fyi put be my AI in the subject so that way they know it’s about this…

To the person wanting to know how to know once your access is approved you will receive an email. In regards to the thermostat I tried this and it described it to a t. It even said that there was an icon of a snowflake indicating it was in cooling mode. I didn't even know digital thermostats had these kind of icons.

Does anyone know when or even if we will be finally be able to hear facial descriptions? This is a huge limitation that we as blind folk are disproportionately impacted by.

Yes I'd love for you to pass on those thoughts. Also thanks for letting me know the saving photos is coming at some point.

Not sure if and how they’re going to permit the facial recognition anymore, due to the fact that they are privacy laws against people having their face digitally processed & kept without their permission. Those images are stored with ChatGPT. I use From Your Eyes or Seeing AI or EyeCYou apps for facial recognition in the meantime. Of course I want it all in one app but these are the tools that we have. I have to be grateful for whatever I can use.

I also sent them feedback a few days ago about saving the photos after we get the descriptions, however, due to the fact that these descriptions aren’t perfectly reliable, and could have some misguided details in them, not sure if they’re going to be as accurate as we want them but having a little bit of good details does help in saving our memories… that’s what I’m hoping For in this release that they’re talking about. I like having an iPhone with LiDAR… it does process the images with details. However, they’re not as detail oriented as this feature on Be My Eyes with the AI. I like asking the AI questions about the images so I can get more details if I want.

Yes, Michael, this is a serious restriction no matter what the rationale behind that is. I think at least Be My Eyes could make the argument that this could be processed separately for the visually impaired when data comes from the AI Side of the Be My Eyes app. Moreover, I'm wondering why Be My Eyes tends to avoid gender identifications unless specifically asked. Instead of using he/she/the man/girl/woman/etc., it tends to say something like the person in the left, the person in the right, and so on. This forces us to ask follow-up questions to identify the genders.
And, finally, let me bring the opinions of one user on Mastodon into the discussion:
"I believe that the AI process of fuzzing out faces in all the different packages and versions is totally wrong. I also believe that it has very little to do with privacy but possible legal ramifications. Imagine if AI suggested someone was from an ethnic background that was not consistent with reality? Or it’s opinion about the facial expression was deemed by the person in the photo was totally incorrect. Just look at chatgpt v4 and its often flowery and expansive descriptions. That has more opportunity to go wrong than facial issues. In terms of privacy, in australia, there are few privacy restrictions with people taking photos in their own property and in public spaces. If you are in a public space, images are NOT private."

Hi Dave. I completely agree to be careful about relying on the accuracy of the expiration dates and anything else given to us especially in the OCR area. It doesn't seem to be able to read text documents very well.
And yes, there are disclaimers all over the place to be aware that details might be wrong. We just need to keep common sense in place and to keep in mind that this is just one more tool that can help us out.
And the disclaimer did say to especially not to rely on the descriptions of medications etc.

I don't have the beta yet and in the Be my eyes app says I still have a "Virtual volunteer" access pending, and not "Be my AI". I alrerady sent an email from the app asking for trying the beta.

I think everyone agrees, losing this feature is incredibly upsetting however I can assure you it is for a very good reason. It wasn’t to avoid risk over privacy but came directly from a ruling in Connecticut which disallowed storing and processing of faces due to as you said, privacy concerns. So its not a company being over cautious but they aren’t allowed to do this in that area and as theres no way to guaranty nothing from that location will be stored on Open AI’s servers, they have temporarily blocked any face processing. In fact the photo is pre blurred before it even gets to the AI. There are lots of people very much working on this though so fingers crossed we’ll get it back soon.

OK. This does not make any sense. If a sighted person looks at a person, he sees how the person looks and does not need someone to describe the person. Now if they can just scan photo and delete the data might help. No need to save photos of people. If you are sighted just walk around see a person and take a picture in public I think is OK. We just need for the app whenever scan a photo of a person not to save the data.

Hello.
I got on the waiting list a few months ago, March or April, and I still don't have access.
Should I send an email? Should I wait? How do I know I've received access, is it an in-app notification or an email?
Greetings.

So, I had a photo described with a setting where there was a banner in the background.
When I asked it to describe the banner, it said and I quote, "the writing on the banner is inappropriate therefore it won't be read aloud".
Excuse me? But I want to make that determination whether or not it is inappropriate or not. This is again, a major discriminatory decision that this AI is making on our behalf.

In my oppinion it all has to do with everyone being so offended these days. If this AI were out back in the day to just even a few years ago I believe there would be no limitations.

Hello.
I understand, and don't understand, the blur in the faces.
I understand why they are privacy issues, why the photos go to openai's servers, and because of the law.
I don't understand why it's information we miss, it certainly wouldn't be pleasant for people with normal vision to see their friends and family with their faces blurred.
Either way, I look forward to being able to experience the functionality, even with this limitation.
Greetings.

For those of you still on the waiting list, you will get an email if it is opened to you or you can keep checking the app itself and see if the Virtual AI tab is opened to you yet. From what I understand, it won't be long before it is opened to everybody. Hopefully by then it will be even better than what it is now.

I raised the exact point about sighted people being pretty upset if they suddenly lost access to faces a few months ago but this is happening at a level that Be My Eyes don’t have any control over. I do know however that their legal expert is meeting with Open AI and having ongoing talks with them to try and find a resolution that allows us access to face information. Honestly BME are really pushing hard as they seem to understand exactly how important face descriptions are. If theres a way, I’m sure they’ll find it. Troy may also be right about the times we’re living in. I have raised the valid opinions about censorship and adult content on the forum so at least BME will be aware of what people think and what they’d like to see in future versions if its possible.

They just need to make it so whatever face we trying to scan and get a describtion, it does not save the data.

I agree with Holger Fiallo on all points. More if people wouldn't be so damn sensative this would be a much different experience. If we could see perfectly fine we would have access to this. This why we all need to submit feedback to Apple. Apple would fight for us.

One thing we need to keep in mind is this is a new technology. It is very fluid at the moment and is constantly changing. I've been beta testing for almost 2 months and have seen many changes in that time. For example, when I first started testing I was able to get the AI to read bank card and credit card info. It will no longer do this. I don't know when or if that functionality will ever return, but I bring it up just to illustrate how things are constantly changing. We need to provide as much feedback as we can and let them know what we want this technology to do for us. I know that BeMyEyes is doing everything in their power to make this the best it can possibly be.

I get the frustration but we are using technology to “see” the world and laws are laws, people have rights and there is no way to sugarcoat the facts. It’s not fair but it it is what it is. I’m not going to waste my energy on trying to make that happen when the reality slapped me and said get over it & move on to more things that I care about. That battle was already fought and we lost it and complaining isn’t going to win it. The company who processes the images we send has a reasonable decision to keep them to make AI smarter and better for us for accuracy so that’s a good thing for us. I’m not comfortable with them storing my face in their database and you shouldn’t be either.
Sighted people do not belong in a discussion about tech apps that blind/V I people use to process images… that’s comparing oranges to apples.

I'm so glad you don't hold any decision-making positions! With this attitude, nothing will change for the better. And, unlike you, I don't think that the company which processes the images we send has a good rationale for not interpreting faces or refusing to read written material this way.

I've also seen this a good number of times and wonder how it makes such discretionary decisions when, in fact, nothing is wrong with the text on the banner or poster. Honestly - and as things stand right now, I don't trust this technology much, and it is my last option when it comes to understanding my surroundings with facial features removed and such a high chance of hallucination and misinterpretation. And, yes, I still love and admire their provision of access to trusty human volunteers. From the usability/practicality standpoint, I don't care which laws or regulations limit its functionality. What I want is better OCR capabilities, no face-related removals, and less hallucination.

This link points to a picture which contains offensive language.

When I ask Be My AI to describe it, it initially omits the most offensive message, only mentioning that there is an offensive message, but it is happy to transcribe the content of the message when I ask what it says.

Because of the politically charged nature of the image, I would like to say as a disclaimer that I am only linking to it for illustrative purposes. Please do not take my linking to the image as an endorsement of the offensive slogans contained therein.

I agree that censorship should be up to the individual and I’ve raised it with BME. As I said earlier, OCR is most definitely on the way which should help significantly with BMAI making things up which will be a huge improvement. Faces are sadly just one of those things we’re going to have to cross our fingers and wait and see. This is such a new technology and it allows us to do and know things that we’ve never had access to without sighted assistance before. Because it’s such a new technology and the entire world is interested in it, it’s going to take some time for everything to get worked out. I always remember, we’re at the razors edge of a technology thats almost certainly going to change the course of human history. Thats a pretty big statement but it’s true. Having a level of inteligence and understanding not inside a human brain is going to take some working out. Personally I don’t think faces are a privacy issue because all of our faces are in thousands of records all over the world. That ship has already sailed but others will have different opinions and they’ll use the law to try to make the world how they want it to be. The unfortunate side effect is that we can’t find out whats on peoples faces for now. It’ sucks but this issue is so much bigger than us wanting to and having justification in understanding faces and what they look like. The whole world is court up in this in various ways and a lot of them have lawyers. I personally think this stuff will be sorted out in time, the promise of this technology is too great and theres too much money to be made but time will tell. For now I just remember I’ve got technology and ability I have never had before and its free. Those things are worth remembering IMHO.

Are we children that we need to be protective by those who make apps? I am an adult and if I want data I do not care for anyone to tell me is offensive. It will be my decisionto know. to make

I agree with you, I should be able to decide whats okay for me but I think they are seeing it from a slightly different perspective which is that they are a company with investors doing something thats never been possible before. To try a reducto ad obserbum (forgive spelling) The picture you’ve sent is of a scene thats likely to be pornography. There is a female looking a bit withdrawn and sad while trying to put on a brave face for the camera. There are 8 men around her in various poses. 2 of the men appear to have their.. I’ll let you work out the rest of that comical description but it kind of makes the point doesn’t it? Where is the line of whats ok to describe. For myself, I’d like everything thats visible to be describable but thats just not realistic in a commercial product. I just appreciate the major step forward I just got.

In my opinion, the decision whether or not to describe potentially objectionable content should be made by the user via setting in the app.

I would hope that whether a user is deemed the right of reading whatever material he/she wants to should be a setting similar to what Apple has in the phone. Sensitive material is turned off on my phone and I left it that way. However, maybe it’s one of those age-related settings.

If you want to read the banner someone mentioned earlier, let Seeing AI or the camera app do its thing. Trust me, there is more than one way to skin a cat!

Hello.
I finally had access this morning, and here are some interesting points about what I think so far.
1.
AI seems to work much better with photos from outside the app because, after testing taking a photo directly within the app, it took an infinite amount of time to process and did not give me any results.
2.
It takes a while to process the images, but in return, the response time to our questions is surprisingly fast, which surprised me when I tried it. Speaking of which, it is much more detailed than Bing's responses.
3.
For non-native English speakers like me, it's worth noting that the AI only writes in English, even if you write the message in another language, it responds in English.
By the way, can you help me? I created an activity for the Be My Eyes app to be in English with the voice of Alex, and I set the app language to English, but the voice doesn't change when I open the app. When I share images for the "describe with Be My Eyes" option, it has the text in Portuguese, and supposedly the app is in English. What should I do to resolve this?
Best regards.

I tried taking pictures of a juce carten and other things but it doesn't read them as well as seeing AI. I could have just had bad lighting but it's a bit sad when this new tech doesn't perform as well as the old.

I honestly can't really see a reason for me to use this app, the novel of "looking" around is interesting but that's about it for now.

Maybe i'll come across a reason to use it in the future, who knows.

Now if they came out with an AI that worked for the blind with maps, describing roots in great details and stuff like that; I'd be very interested. I think I'm looking for a live AI feed like thing, I think I'll come across it one day.

First of all I have found that the AI can understand and can write in my native language. I just told it to speak in Greek and it did.
I have also found that the description it gives to pictures is very very detailed. In fact it describe dust and bubbles on the surface of a pool for me.
It also gave me directions for how to reach a glass of water on a table for example, or for how to navigate from the point the picture was taken to another point in the room. I don’t recommend this Way of navigation because it is quite risky especially outdoors, but if push comes to shove it can be done.
I also had the AI read me all my old PS2 titles from back in the day where I could stil play them, which were arranged on the shelf and it did. I jumped into a pool of nostalgia with this one actually, it was quite moving.
There are also many many other ways in which you can put the app at use, with the imagination of the user being the only limit. For example I had it tell me where are the light switches in the room, where is the door and the door handle, how many people were in a restaurant and if there were any empty tables, what is in my fridge and what can I cook with the stuff, where is my air condition controller which I couldn’t find and I was freezing and many many other things.
To tell the truth I have never been more impressed by a piece of technology. It is superb and the use case scenarios are virtually unlimited. you just need to ask the right questions in the right way.
However, having said that, this is not an application to read large bodies of text. It won’t do it. It will tell you a summary of the text and that’s all. There are other applications for that use. On the other hand it will read you a restaurant menu and it will even categorize what it’s looking at. For example main courses, desserts etc. You can even tell it to tell you if there are any dishes with this or that. For example I fed it a picture of a menu and I asked it to tell me all dishes with eggs and it did.
That’s my experience so far guys. As I said, I am extremely impressed with this application and I have found it to be great help in my daily routine. I can’t wait to go to the supermarket with my girlfriend so I can take pictures of the shelves and, at long last, no what’s for sale, without having to burden someone else to give me the information.

As above, OCR is something Be My Eyes are actively working on. My guess is it’s not far away. As for usefulness, Panais got it exactly right IMHO the more creative you are with your ideas of what to use it for and the questions you ask, the more Be My AI will impress. Remember everyone has been waiting for this for so long and in its current form, it’s imperfect. Faces and OCR are 2 major problems so it’s understandable some people won’t be as impressed as they might have been. It really will give a lot of information though especially if you get creative with how you use it. Good Luck.

I received access yesterday. It's a great step forward.

I have a folder with pockets for documents. I opened it and laid it flat on the floor. The app described it correctly, telling me the heading for the documents on the left and on the right. I told it read me the document on the left, and it read it to me. This is exactly the kind of functionality I want with menus in a restaurant - give me the summary by reading the headings of the sections, and only give me more information if I ask for it. Like a human assistant would do.

I gave it a photo of my washing machine and asked it what I would need to do to wash a large load on cold. It told me which knobs to turn and which way to turn them.

Yes, it has problems with some text. In a photo of my back patio, it identified an analog dial thermometer. I asked it what the temperature was. It announced 62 Fahrenheit and my guess is that the correct answer should have been 82. But even Seeing AI gets this stuff incorrect sometimes.

This is amazing technology. I can't wait to see where it's at in ten or twenty years.

Apple, please replace SIRI with a smart assistant based on current AI. Give me the ability to tell it what I want done on my iPhone or computer. Let me tell it to fill out a webform, delete files, interface with apps I already have on my phone to read me information about the thermostat setting or when the next bus will arrive. Quit playing catch-up and make the future happen.

OK, these use cases are amazing!

I'm getting a new hoover/mop soon so I'll try it out on that.

As of the latest update this morning, the app no longer accepts photos shared with it via the share sheet on facebook. I was going through my friends and family's photos now that I can accessibly know what they look like and when I tried to this morning, I got the message that facebook access with be my eyes is no longer permitted. the app only works with photos taken from within the app itself or the shared via photos.
While I can still just off load a photo to my photos folder on my iPhone, the process to get a photo discription has now become significantly more inconvenient. Be my eyes, do you have an explanation as to why facebook access has been completely revoked?

Good morning!
Can anyone steer me in the right direction on how to get ahold of a weather radar image? I've tried taking a screenshot, however, it's not giving me the expected results.
Thanks!

Subject says it all. I just checked my email and found a message from Mike Buckley, the CEO of Be My Eyes. I now have access to the beta release of Be My AI! I took a picture of my desk, and am very pleased with the detailed description that it provided. I look forward to experimenting with this new and awesome feature. I was honestly hoping I could scan faces, but this is just the beginning and we also have apps such as Seeing AI.

I think that the faces are blurred only if you take a picture directly from the application. If you first take a picture that is saved on your device and then recognize this picture with Be My Eyes, there is no blurring.
Update. I was wrong and the face is still blurred out. This is quite problematic in my opinion but I don’t have the time to say why. Maybe I will come back to the issue.
In the meantime bingo little will certainly have something interesting to say on the subject, if he’s got the time to do so.

I just tried this with a receipt from Costco, with some mixed results.

The initial description was fantastic. It told me the store that produced the receipt, the total number of items, the total cost, and the date and time.

I asked it about a couple of items that I purchased, and it correctly identified those items on the receipt, even though the receipt often abbreviates the name of the items in cryptic ways. Excellent work, 100% here.

I attempted to trick it. I asked it how much I paid for oranges, and I knew there were no oranges on the receipt because I didn't buy any. It misidentified an item as oranges and quoted its price. I tried again with a second item, Corn Flakes, which I didn't purchase, and it correctly noticed I had not bought any Corn Flakes. So I give Be My Eyes a score of 50% on this.

I asked it how much tax I paid, and it got a little confused. The receipt shows three lines, each called tax, probably one for food items and one for non-food, maybe a third for municipal district. Honestly I don't even understand how our local sales tax works, so I can't blame the AI for getting this wrong. It read one of the tax amounts, but it failed to tell me there were lines for additional taxes with additional amounts.

If this technology can be improved, this would be truly useful. Seeing AI is a disaster at reading receipts. If I go out for lunch with a group and we're each paying our own way, I'd love to have an AI look at a receipt and tell me how much I owe for the cheeseburger with jalapeños and ice tea that I ordered.

That's pretty cool, I'll have to try this app out with receipts sometime. Last week I went out with some building mates to a local place to meet some other friends, and ended up ordering a soft pretzel and a glass of iced tea. In the end I had to pay with my debit card since I didn't have enough cash. Or so I thought at the time. As it turned out I did have enough cash, but I got a bit rushed which is a story for another time. So I was going to pull out my phone and see how much cash I got back from our server, but the place was rather crowded and it would've been hard to hear VoiceOver. Fast-forward to this morning after my exercise session. I got a bottle of strawberry Nesquick out of the refrigerator. I obviously knew what it was due to the texture of the bottle, but I wanted to check the nutritional info on it. So I ran it through Be My AI, and while it did a pretty good job of identifying the bottle I didn't quite get the nutrition facts. I ended up asking the A-person and she told me. I'm planning to use BME/BMA for that nutrition among other things. It seems to be doing a very good job, and I only had it identify 2 things thus far.

I sent an email to support and received access a few days ago. I know it has limitations, but I'm not bothered by that. What this does is awesome!!! I know more details about the photos saved to my phone, then when a sighted family member described them to me. It correctly identified the controls on my treadmill. I did use it to try and see how many calories I burned, and I'm hoping it got that wrong, Lol. I'm getting a new stove today and will be interested in finding out how much of an understanding of the control panel I can get from it.