Well, it's the integration of GPT into the traditional JAWS picture smart that got me thinking about the whole thing. With the advent of AI, we have systems in place that can understand even complex graphical information and provide text output. But many a time, this becomes not enough. For instance, I work in a profession where I have to deal with highly localised maps, and other geospacial information on a daily basis. There may be data scientists among us who certainly will have to work with complex data in the form of graphs and whatnot. So I was wondering, what are the different ways in which we can generate output such that the comprehention of graphical information by the visually impaired can come out of the limitations imposed by a textual output paradigm? We already have VO providing audio graphs in certain IOS apps. While I find these interesting, it's scope and extend is limited as of now. Can we have systems in place such that such audio graphs can be generated for complex information such as say, an infographic? Can we push for a multisensory approach of output in emerging areas like wearable technology? For instance, like someone suggested the other day, can we integrate something like vOICe into the Seleste glasses? Then can we also go ahead and make tactile(as in both vibratory and braille-like) feedback also part of the loop? and powered by AI and computer vision, could these kind of break new fronteers in how visually impaired folk understand the world around them? Since it seemed like the right time we start thinking about this, I thought it'd be interesting to get everyone's thoughts and ideas on this.
Comments
Custom GPTs
@Lottie. do we have any tangible examples/working projects employing custom GPTs for data mining/analysis around? One issue I have with the dialog becoming the diagram is that we're still depending on an external agent IE the LLM to get the analysis.
the search for Vibrotactile solutions is nothing new
So suddenly this came back to memmory:
https://en.wikipedia.org/wiki/Optacon
I don't know where or why we left research in that direction and holely focused on text to speech?
Apple's Audio Graphs
Very interesting topic here. Regarding the audio graphs that are now in iOS, they are indeed interesting and it'd be cool to expand them. For those of us who use Braille in addition to speech, it would also be cool to implement tactile graphs/charts. I briefly checked out BrailleBlaster and Zephyr, both from the American Printing House for the Blind. I've not actually downloaded them yet, but the documentation for BrailleBlaster says that it can produce raised-line drawings or something like that. Have any of you checked this out in detail and if so, what are your thoughts? Additionally, I do remember the Optacon. I used it when living in Pennsylvania, and liked it. It was probably ahead of its time, and it's a shame that the company is gone. I also used the VersaBraille from them.
I tried Optacon 3 years ago
I agree, it's very sad that people just forgot this device. As old as it is now, it was really groundbreaking for its time back in the good old days (I wasn't even there yet).
Basically someone draw a table with a couple of lines on a paper and asked me to test if the device is still working. I was indeed able to follow the line along and adjust the camera/scanner.. whatever, angle, at the surface of the paper.
Unfortunately for me, I wasn't delicate enough with the device and it has actually been broken since then.
I am curious about orbit grafity and the device aph and humanware are working on. As long as we don't see more than a lab prototype I'll still be skeptical though.
PS: is it just me but humanware are making very expensive device but with very outdated and crappy hardware/software even in their category of devices? Like heard that their new braille display/note taker takes about 20-40s to boot! Bi 40x or something.
It's just how the industry is
I have a zoom Max, gonna send it back tomorrow as if autoscroll won't work, no point. Anyway you're correct. we have comps that boot in under a minute but a simple braille display, no notetaking functions whatsoever, takes ages and can't find wifi you know you're connected because your phone is. Plus at least with the zoom max, you can only have one book open at once. Why? Sorry to get off topic.
Ai is certainly exciting
So far, I've seen AI generating text descriptions for visual content, and that too, only for images. Haven't seen a single instant where an AI is capable of interpreting a video. I am guessing that would require a terrifying amount of processing. But on the other hand, if it can generate a video out of text, why can't it do the other way around?
I don't think with hapticks we can represent a detailed enough picture. IPhones have such a nice haptick moter, I can imagin some application which picks up the point where your finger is touching on the map or image, and provide appropriate haptick feedback. The learning curve for such a thing would be very high in the beginning.
May be you put all that haptic in a pair of gloves and with AR/VR tech, let the gloves vibrate as soon as they touch something in the vertual world. Again, how accurate this kind of representation would be, god knows.
The only meaning full way to provide a 2D image would be with a larger braille display, the likes of dotpad and such. Haven't seen any of those in public so far.
For a 3D representation, visually we've been trying to get functioning holograms for decades. Tactile is going to be even more challenging. Even with a fancy dotpad 3.5 or 5.5 version, which can may be change the amount of height for individual dot, still,would it be able to represent, say an airplain in an understandable manner? I am assuming only 1 side would be there at a time. Again, big lurning curve, and I am assuming complex things would be next to impossible to understand in this way.
I was hopefull with 3D printers, but it hasn't reached masses. May be AI can help us in that regard, creat 3D models for 3D printers, based on text descriptions etc. Or based on an image, creat a tactile map. Currently this is a manual process, very expencive.
I don't suppose anyone is looking at tactile holograms yet? Can't imagin how on earth that would work. May with electromagnetic fields, lots and lots of them, you hold magnetic particles in a specified shape. How would the person touch that thing safely would be the first question. Further, how much energy something like this would take? Rather I think the haptic gloves would be a nice starting point.
Just running wild with my ideas, don't bash me. Everyone's got right to imagin.
A couple of things.
I wouldn't depend on an Optacon for reading today. The speed and flexibility that text to speech offers is too convenient for that. All the same, if that research continued and there was something like a wearable device, say like a glove or something, that would let me perceive printed text and graphics? definitely!
Also, generating text out of videos...No, certainly not impossible. If I'm not wrong, Microsoft Azure already offers a similar service?
@Lottie so what does your NVDA guide do? Talking of custom GPTs, there're all kinds of things we could do with those. For instance, there could be one such GPT for totally blind people who use make-up. you click a picture and it tells you if it looks fine. Just thinking of possibilities.
And as much as haptics kind of has a huge potential, I guess auditory feedback could also be holding the key especially since there the learning curve would kind of be less since we've been, consciously or otherwise, using it for the entireties of our lives.
Posted in the wrong thread
You can find my original post here:
https://www.applevis.com/forum/ios-ipados/more-posative-spin-things
Posted on march 17th.
@SeasonKing thanks for the innitial inspiration.
A use case for the Optacon
I've occasionally needed to check whether text in a document is aligned correctly. I use JAWS, and even its INSERT+F and ALT+DELETE keystrokes are sometimes not enough.