Staying Oriented: Using AI-Powered Apps as a tool for further Independence

By Deborah Armstrong, 19 November, 2024

I remember the first time I used Seeing AI to help me in the airport. I had 45 minutes to get from my gate to my connecting flight. I knew I could have asked the airline for assistance, but previous experience has been quite frustrating. I often got stuck simply sitting and waiting for someone to guide me to my next gate. Often, they arrived with a wheelchair and I had to argue that it wasn't a good idea since I had concerns about the chair rolling on my guide dog's toes!

Instead, I had Seeing AI set to detect text, and I simply took off waving the phone around. Sure enough, it read me gate signs. When I heard gate 11 and then gate 10, I knew I was going the wrong way, since I needed gate 29. I turned around, told my dog "Forward," and we headed past gates 10, 11, 12, etc. Guess what! We did get to gate 29 only ten minutes later with 35 minutes to spare.

While this happened five years ago, my usage of these tools continues. I work on a 112-acre college campus. And right now, major portions are under construction. So walking a familiar route, I often encounter a fence blocking the path. Trying to find a new route to my destination, I often get lost. However, with the power of AI, I can do what the sighted do when they get lost. I can read signs and/or have the app describe the environment. For example, when the sign reads "Audio Visual Services," I know I'm at the back end of the library.

In preparation for the upcoming construction, I walked all over campus asking AI to read many many signs to me. Once I had to travel new routes, I knew what signs belong to which buildings and could improve on my mental map of the campus.

Many of us use our phones for navigation, in conjunction with apps like BlindSquare, Nearby Explorer, Google Maps and Apple Maps. These apps can help us know about intersections and addresses. AI, though, can ad a lot more information. Suppose you find yourself stranded in a large parking lot, attempting to locate a store. Or maybe you want to take a walk in the park, but worry about finding your way back to the picnic spot. Maybe you are in that store but all you are finding is piles of women's clothing when what you really need is some shampoo.

Remember first that sighted people get lost all the time. They forget where they parked their car. Stores are designed so that the shopper must wander around observing sales before they find what they were originally seeking. Now with apps like Seeing AI, the Be My Eyes "Be My AI" feature, The Aira Explorer "Access AI" or even the new Speakaboo, you too can see your surroundings.

There are also built-in tools with iOS 18, such as its Live Region Detection, which can offer you even more flexibility. You can point the phone to items in real time to get a description or have it detect text.

If you are new to all this, try working with these AI-enabled apps and iOS features first in familiar environments like your home, yard, neighborhood or workplace. Then branch out and explore the world. You will find at times, AI can describe even better than your sighted friends. And for you, the wider world will come alive.

Options

Comments

By SeasonKing on Monday, November 25, 2024 - 08:14

5 years ago, she navigated an Airport using Seeing AI!
I have tried navigating a shopping mall using the same, and I can't imagin doing that on an airport. But again, I use a cane, and, people do crash in to me allot. Or may be I crash in to them, but I like to think the former.
The thing with Seeing AI is that I often don't know where to point the camera at, specially at airports, where the boards might not be those shiney bright shop signs. If it was like those bright shop signs, I would know to point camera at it.
And, in some of our airports, layout of the gates, it's not a strait line, it's some times U shaped, L shaped, T Shaped, and what not.
Now, if something like live recognition was there hands-free, and it gave me hints to adjust camera angles, then I might be successful at navigating airports like that. I would be damn scared of missing my flight though.

By Brian on Monday, November 25, 2024 - 08:14

First, this was an enjoyable and inspirational read. Thank you for taking the time to write and post this. 😃
Second, navigating the world with a guide dog is so, so much easier than navigating with a cane. I have had to navigate airports both as a cane user, and as a guide dog user, and I will take the guide dog every single time, over the cane. The fact you're able to navigate an airport on your own, with just your guide dog and the Seeing AI app, is just beautiful.
Finally, I hope they add such services to devices such as Meta smart glasses. Oh how wonderful that would be. 🤔

By OldBear on Monday, November 25, 2024 - 08:14

Yes, I remember the wheelchair ordeal at airports...
I was reminded to look up something by the story, and I'm delighted to find that Seeing AI does not require a network connection for its Short Text function. In the back of my mind, I really, really worry about becoming dependent on AI that requires a network connection to function, especially when the network isn't working for what ever reason.
I also use the compass on the phone from time to time. Not that I'm completely sure that would work in all situations with no signal of any sort.

By Ekaj on Monday, November 25, 2024 - 08:14

First, the disclaimer: I'm one who for reasons I shall not go into, does not have very good outdoor travel skills. I have been a cane user practically all my life, and have gotten quite good at that. I had some very good O&M training, but not recently. Now with that out of the way, here are my comments. I love this post. I started out using an iPhone 7, and became very comfortable using it. But I am now the proud owner of an iPhone 14, and yes some of these AI apps are on there. I have used them for various tasks with mixed results. I'd very much like to do more traveling one of these years, but based on my somewhat limited experience these apps are a joy to use and they're only getting better. The only problem is that I've had trouble knowing exactly where to point my phone so that these apps can more effectively do their job. But I think an iPhone stand might be in my future. Hint hint: possibly Christmas or my birthday? I'd really like that. Having said all this, I do very much enjoy my iPhone and hope to do more in the coming months. It's honestly a bit daunting to see all the things these apps alone can do. A few years ago I took a car ride with 2 sighted neighbors to a breakfast meeting, and used Soundscape to get an idea of the route which we were traveling. It was a great experience. At the time Microsoft still owned Soundscape and I don't think it was open-source yet. As for Live Recognition, it seems a bit confusing to me but I'm glad Apple included it in iOS 18.

By Tara on Monday, November 25, 2024 - 08:14

Hi Deborah and all,
That's such an awesome way to use SeeingAI, and it was five years ago too!
I've got a question for any of you who wish to answer: what happens if you're pointing your phone in different directions, and someone thinks you're trying to take a picture of them or something? Has this ever happened to anybody? I've never used SeeingAI or BeMyAI in a public place like this; I'm worried somebody might think I'm trying to take a picture of them or something. I wouldn't mind sitting at a table in a restaurant and reading the menu that way if I had to, but walking around a place waving my phone around doesn't seem like a great idea. That's why I really want a good AI to come to an affordable pair of smart glasses. I've heard of people waring a chest harness for their phone which doesn't sound very comfortable to me.

By Lee on Monday, November 25, 2024 - 08:14

Hi Tara,

Don't know for sure but I suspect the answer is stereo typing. How many sighted people think us blindies can take pictures? So they probably haven't a clue what we are doing as long as they see the White cane.

By Tara on Monday, November 25, 2024 - 08:14

Hi Lee,
Good point. Maybe they just wonder what on earth we're doing with our phone. But I don't know, I'm still cautious of using my phone like that. I'd much rather a pair of smart glasses.

By Rusty Perez on Monday, December 2, 2024 - 08:14

First of all, I applaud your courage. Though I travel with a dog, I would not have the courage to venture off in an unknown airport relying on my pointing a phone camera around and hoping I get useful information with only 45 minutes to catch my next flight.
I have been in enough airports to know that there is a very real possibility of needing to take a tram, change levels, change terminals or concourses ETC. ETC. to get to a connecting flight.

On the other hand, while I always need to politely refuse a wheel chair, I have never missed a flight when being guided by an airport provided assistant. Are there communication barriers? Often. But even so, they know where they are going.

So, I'm impressed, mostly by your dogged self determination.

By Rusty Perez on Monday, December 2, 2024 - 08:14

I'd like to know more about this. I'm gonna search the site for this now.

By Ekaj on Monday, December 2, 2024 - 08:14

There've been times when I waved my phone around, but nobody was with me when I did it. Again I think this is a confidence thing. I myself--having only had light perception since the ripe old age of 6 months--am still rather overwhelmed by the fact that a person like me can even use the photo and camera apps since they work with VoiceOver. This despite having a brother who has successfully done it for several years. He acquired his first iPhone before me, and was in fact one of the people who convinced me to get one of these. I mean, that wasn't one of the reasons for getting an iPhone in the first place. Having said that, I asked a sighted neighbor to send me at least one pic of a sunset which he took while on Thanksgiving vacation. He sent it to me, as well as some other pics and I ran them through Seeing AI. I've done this in the not-so-distant past and have shown friends and family some of these apps. Everybody has always been impressed. But I suppose there are also those people who doubt this fact, despite anything that they're told. As a matter of fact, I think I just might know some of those people.

By Brian on Monday, December 2, 2024 - 08:14

Sighted person, "Hey, how do you use a smart phone? Do you braille the screen?!"
Blind person, "... no. I use a screen reader to navigate the device, audibly".
Sighted person, "hey, how do you take pictures with your smart phone? Do you braille the camera?!"
Blind person, "... go away".

/end sarcasm

Thank you for reading. 🙇

By Mister Kayne on Monday, December 2, 2024 - 08:14

I know, but if they were offering it to me with no extra cost; why not get wheeled around like a prince on a chariot? Airport staff and disability inclusion rules for the airlines industry if you think are bad out there, you must experience India and it's airports. I once actually flirted with a staff member and she was extra kind with me by bringing me an extra buttered toast. I am not a guide dog owner, make do with a cane that is so not right for my height but hey sometimes you are the windshield, sometimes the bug. Here they ask for extra money if we need a sighted guide to assist us on the airport. Humanity is dead, if a fellow passenger would just be an ally all this would not concern us as travelers. Not all apples are bad, the passenger on my flight, the curious Joe, kept me engaged in talking to him about my disability and assistive technology. His mission was to convert me to a Christian; well I obliged and now I do everything in the name of Jesus including walking into moving obstacles

By Brian on Tuesday, December 3, 2024 - 08:14

I no longer have a guide dog. Retired him 2 years ago, and he now lives with a family with 3 kids between middle school and college ages, whom spoil the life out of him. So, the few times I have to travel by plane, I always take the offered wheelchair.
1. It gets me through TSA super quick.
2. I always make my gate.
3. I can respond to texts, etc, while being wheeled around.

Why yes, it is true. I have no shame.

thank you for reading. 🙇

By PaulMartz on Tuesday, December 3, 2024 - 08:14

Nice to see this blog, Deborah. Sorry I missed it when it was originally posted.

The barrier to using Seeing AI as Deborah did is that you need one hand for the iPhone and another for the came or dog. Unless you've got one of those cool harnesses and strap your phone to your chest. This leaves no hands for your bag, or that latte you picked up at Starbucks. The Meta glasses feel like they are ever so close to being able to do what Deborah did, but hands-free. Meta's AI could improve on the Seeing AI experience by adding location information: approximate distance and direction to the signs it is reading.

Another way AI could help: The AI could access a map of the airport. Combined with what it sees through the smart glasses, the AI could then give you turn by turn walking directions. This would beat the crap out of existing indoor navigation systems.

I take a wheelchair in an airport. Works great with a folding cane. No dog paws to run over. But if Meta can make AI smart glasses for US$300, why not build that technology into a motorized wheelchair? "Hey wheelchair, take me to my Detroit flight." You wouldn't even need the gate. The wheelchair could look that up. Or it could scan your boarding pass. Or you could share your boarding pass with the wheelchair.

We're really close, folks, and it's only going to get better.

By sechaba on Tuesday, December 3, 2024 - 08:14

I can't imagine walking around and waving my phone in South Africa.