Announcing a New Project: Building a JAWS-Inspired Screen Reader for macOS

My questions stand. It took me a single week to build the version of Vosh that I used to make its initial teaser video. and I did it without AI assistance or prior examples to learn from, so it's not even hard. What's hard about the whole thing is the amount of work that it takes to support lots of individual applications that change over time, which is why I question the viability of this project that at the moment has absolutely nothing to show for it and doesn't seem to expect community assistance.

My work on Vosh landed me a job at a US company using accessibility on macOS to build an AI service to summarize and advise people on their own lives, so although I did not provide support for any application in particular in Vosh, I do know how cumbersome it is to address the quirks in lots of applications and maintain support over time, because the company that I worked for had this problem. Although in the case of an AI assistant it's possible to rely on the AI itself to make sense of the content, which was something that I suggested doing shortly before leaving, a screen-reader is a completely different beast.

The reason why I ask whether this is a joke is because this comes right in the heels of Turnover, a vibe coded screen-reader project. In addition I know the public accessibility API all too well to know that this isn't viable without using the private CoreAccessibility framework. One example that I gave on the other thread are the custom accessibility rotors, which to my knowledge are not made available by the public accessibility API.

To me this sounds a lot like someone promising to accomplish a very ambitious goal without even researching the subject. It is likely that, like the developer of Turnover, they believe that vibe coding will carry them over the whole process, but the reality of vibe coding is that current AIs are very bad at software architecture so the code ends up becoming an unmaintainable mess as it grows. It's easy to build some initial version that impresses, but like they say, the devil is in the details, and in this case there isn't even an initial version to show.

Why won't I because of comments like above. The majority of the community doesn't appreciate what people try to do for it so screw it why should I spend hours to try and help? I just can't believe the attitudes of the community. and yes I'm speaking directly like the negative comments earlier in this thread. so when your comments make you come across as an ass remember that. I agree with the other person who said you could've been a lot more constructive and how is this diffent from a main stream company saying version 8 or 9 of x product will be out a year or 6 month before it is. Sighted people don't complain like the blind community.

You shouldn't do it to get appreciation, you should do it because it's useful to you. I didn't start Vosh to be appreciated, I started Vosh to prove that building third-party screen-readers on macOS was possible and hopefully get the community to assist building one, which didn't materialize so I abandoned the project. I'd do it all over again under the same circumstances regardless of how well it was received, because at the end of the day it's the contributions not the opinions that matter, and at the time I did not know that the community wouldn't engage.

Editing to switch the formatting to mark down, as well as to add that the reception I had to my initial teaser video for Vosh was a bit scary, because people had very high expectations that I knew that I wouldn't be able to fulfill alone. Also, I have no idea why you think that this kind of announcement is well received by sighted people, because it's not. Vaporware is very poorly received by the sighted, and for that reason alone sighted developers rarely announce things before they have initial implementations to show. In case you don't know most open-source projects fail to gain traction and are simply ignored, which is worse than getting negative criticism because people never end up learning why their projects fail.

I honestly wonder though with one of latest apple paper about this comuter vision model about UI detection for app intents if attempting the off screen model trend again could go somewhere, a question for you @choom and other knowledgeable developers out there.

I have considered training such a model myself, it is actually something that I have planned to do at some point to work on once I get my Mac Studio back. My first idea will be to just build an OCR trained on rasterized characters using the San Francisco and Menlo fonts for variable and mono space characters respectively on a 32x32 pixel square, that will be then vectorized and classified based on intersections and curves so it will be a very robust OCR also capable of solving the old CAPTCHAs based on blurry contorted text. My second idea will be to recognize the shape of buttons, switches, sliders, and so on. The general idea is to build an iPad application that can recognize and generate accessibility elements for images captured using an HDMI dongle, to make navigating inaccessible interfaces like PC UEFIs possible, but I have no plans to target macOS specifically both because I've never experienced Liquid Glass visually and because it would require screen capturing privileges, which last I heard are automatically revoked after a certain amount of time by the system thus providing a bad user experience, but answering your question, I think that's a very viable option for Apple.

yeah at least ternover had a virsion that we can look at.

also im not gonna say anything about this guys account, because this person has apparently been on this site for 4 years and 4 months, but this is his/her only post here

Just as you have the right to criticize us for apparently being "asses we have the write to criticize someone who has most likely vibe coded this and hasn't even shared anything to show for it, which is even worse than turnover because at least turnover had something to show for it and at least we were able to actually discuss what the issues were I do agree about the blind community though there is some people who are definitely a lot more entitled than others, but that doesn't mean you put the branding on everybody. doing that, in my opinion just makes you as bad as them and that's something I really do not want to say on somewhere as public as this

This is why I respect you so much. I didn't understand some of what you said there 😂 but thank you! this is the kind of discussionbs applevis deserves. So cool my idea has something at least.
PS: my accessibility tree dump idea was more to get headless chromium and, still idea stage, then pipe everything in emacs / emacspeak with something that can then pass the events / other inputs back in the real browser behind so at least us blind can get everything inside one monolithic that can work pretty similarely everywhere. Will probably go nowhere more than the ideation itself, but who knows. I think tv raman theorized this a couple years ago, and he has always said that screen reader apis are a lossy way for us blind to in modern ui computing unlike the previous terminal world.

I did not intend to make my comment cognitively inaccessible. Feel free to ask questions by private message if you wish because this is off-topic here, both you and anyone else wanting to understand what I said.

To be fair, it is only like 8:30 AM for him so

Looks like Linux has entered the conversation too! I recently came across gorca, a new screen reader for Linux, written from scratch in Rust by an experienced engineer working in DevOps and neural networks.

The project was started to address Orca's responsiveness issues under heavy workloads. The developer is openly using AI-assisted coding, but the project also has a clearly defined architecture and an engineering approach behind it.

I haven't tested it myself, as I don't currently have Linux installed, but I thought you'd find it interesting given your comment.

https://github.com/Ravino/gorca

I don't think I can agree with "the vast majority..." I think it is actually a small minority but they are vocal enough for it to feel like that because it can derail the conversation. You probably do need to be thick skinned enough to put out anything in the public domain, but I don't agree that the majority of people are like that.

In a similar way, if I go out walking with my white cane and someone runs straight into me and then has a go at me, I can become of the opinion that no sighted people understand the white cane and that most people are rude. But I haven't noticed the 99% of people who just got silently out of my way. But even if I had, it would be the one incident I would personally dwell on, not the many that are fine.

Anyway, having said that I do thin it is reasonable to have questions, preferably if we could keep them courteous that would be best so we don't end up in another argument. Being kind should not take more effort than being rude.

I would be interested to know for sure if this is a 100% vibe coded project, and how far along it is. I would like to know whether it is a viable long-term option. These days there are so many apps and things being thrown out there because someone got excited vibe coding, that I personally wouldn't want to devote my own time to anything without knowing that it had a more solid foundations - particularly something so essential like this. I'm not saying this particular project is like this, but I would like to know more.

I'm not sure I personally need Jaws on the Mac, but I would like us to have a solid competitor to VoiceOver to give us some more choice, so it is definitely of interest, whether or not it is something I would personally use.

I did not call anyone by name.

you said.
"when your comments make you come across as an ass remember that."

just remenber that you don't have to personaly name someone to still insult a person

I currently have a prototype ready. Even though some folks around here might say "there's nothing concrete yet," I wanted to share it with you all, even though it's still in its infancy. It's currently open for testing in apps like Finder, System Settings, Mail, Safari, and Chrome—but believe me, the roadmap I have in mind for this project goes way beyond that.

My question stands, now reinforced by the fact that you have a prototype ready. What's the point of announcing and not showing what you built, not even in a teaser video?

Also I don't think that you will get far using AppleEvents, since most applications including Apple's Messages don't support it properly. I tried that path while working on the AI assistant for the American company, and found it to be a dead end, not only due to the lack of support but also because users wouldn't feel very comfortable with constant prompts asking whether they would allow our service to control their applications after they had already granted accessibility privileges.

It is one thing to promise to try building something possibly with community assistance while having an initial prototype to share with the world, and another completely different thing to make a likely empty promise a solution that many others, including the JAWS developers themselves, have allegedly tried and failed to produce.

I don't care about your roadmap, because everyone including myself can have grandiose ideas which aren't really worth anything since all the value is in putting the effort into turning them into reality.

Talk is cheap, show me the code! -- Linus Torvalds

Very puzzling indeed. But I completely agree.

FYI, makers of console games for Xbox , switch, n Playstation, do this as well. and heck, even Apple. remember the jurnal app anyone? it was supposed to lonch with iOS 18.1 but was delayed until 18.2. we never gave Apple Negative feedback about that? even the new health app isn't here yet, with Apple, saying, it will be available later this year. so don't trample this developer, the cited world has this all the time.

I don't think that anyone here is against someone building an alternative screen-reader for macOS, but if someone whose only experience is in building Ikea chairs and tables tells you that they're gonna build a skyscraper next, it it's perfectly reasonable for you to doubt them.

To the developer, by all means, do prove me wrong, because me being wrong here is the best for the community! Go ahead and build that skyscraper! Show me that unlike myself and others, you really have what it takes to succeed! I genuinely want to be wrong here!

yea but tbf who uses the gernal app, no idea why it was even made just to give us less storage to play with ig, also i never new there was a new health app