Introducing ally (public beta)

By Karthik Mahadevan, 23 December, 2024

Forum

iOS and iPadOS

Hi Everyone,

Hope you're having a great start to your holidays! Earlier this year, we introduced the very first prototype of what was then called Envision Assistant. You can find the original post here: link to original post.

Our goal was to create a conversational, personal, and ubiquitous AI assistant that helps you access visual and non-visual information around you in the easiest and fastest way possible. Since then, we’ve received invaluable feedback from many of you. Thanks to your input, we’re excited to announce that we’re moving from private to public beta! No forms, no waiting—just click the link and download.

Before You Start Using Ally

Here are a few things you should know about ally:

Ally is conversational: Optimized for back-and-forth conversation, ally allows you to start speaking with your question, and it will respond. You can interrupt ally at any time or ask follow-up questions, making the interaction snappy and as close to human conversation as possible.
Ally is personal: You can share information about yourself, which ally will use as context to provide personalized insights. During onboarding, you’ll answer a few basic questions, but you can edit or add information later in preferences. Over time, ally learns more about you to offer a truly tailored experience. You’ll also choose from one of four initial personalities, which you can later customize—right down to the name, language, voice, personality, and communication style.
Ally is ubiquitous: Ally is already available on iOS, Android, Web, and Envision Glasses, with plans to expand across even more platforms to ensure accessibility wherever you are.
Ally recognizes intent: Ally’s intent recognition algorithm ensures it understands the purpose behind your questions and chooses the best pathway to find answers. For example:
- Ask “What am I holding in my hand?” and ally will take a picture and use vision to answer.
- Ask “Should I carry my umbrella today?” and ally will check the weather to respond.
- Ask “What’s playing in the cinema this weekend?” and ally will perform a web search for local listings.

What Ally Can Do Right Now:

General questions: Uses basic LLM to answer.
Current affairs: Uses web search to respond.
Image-based questions: Uses the camera and vision tools.
Weather queries: Checks local weather data.
Calendar-related questions: Accesses your calendar for answers.

Our aim is to continuously add more functions, making ally your interface to all kinds of information around you.

How to Get Started

Install the ally beta app on your phone using this TestFlight link: https://testflight.apple.com/join/VyjHwsMA.

Please remember that ally is still in beta, so you might encounter bugs, UI issues, and missing features as we work towards a production version. Your feedback is crucial to help us improve!

How to Provide Feedback

The best way to share your feedback is through our beta Slack channel. Join here: Slack Invite Link.

On the Slack channel, we also share our future roadmap, behind-the-scenes updates, and early previews. We encourage you to join and be part of shaping ally’s future!

Thanks for being part of this journey. We can’t wait to hear what you think!

Options

Comments

Karthik

Is there a chance you can add an option to stop Ally saying "let me take a look" after every question. Gets quite annoying I'd much prefer to have an option where you ask a question and it is obvious that Ally will be "taking a look" without constantly telling me this. Apart from that I've been using this for weeks since it was launched and I have to say I do like it.

Response to Lee

Noted! We're making "Let me take a look" more dynamic so it can say different things each time. Once it gets faster, we won't even need it to say anything and just answer straight away.

Good news

Also, espically for the glasses version any chance of live streaming? Would be extremely useful to say ask Ally to let me know when I'm at the bus stop. That really would make the app amazing.

Let me take a look

Perhaps you could have an alternative option to play a sound to indicate taking the picture then another sound if necessary to indicate work in progress? For example a camera sound effect followed by a soft clicking or soft beeping.

Thanks for the suggestions!

Appreciate those suggestions. Yes, this is all heading in the direction of live video processing in the near future, so we want to do away with processing sounds and effects to make the whole experience feel as natural and conversational as possible. Like having FaceTime with a friend!

Love It

I'm already a part of the beta team, and love it thus far. Perhaps there could be a toggle for these messages saying "let me have a look" and so forth. I say this because I've used Siri on my phone to recognize songs, and it always says something like "let me listen" prior to coming back with the response. I for one am not in the least bit bothered by this. Also, I think there's a problem with my Slack account but I'll have to double-check that. I'm taking my Mac as well as my phone to my parents' place when I go back there for Christmas.

Love the beta, but am a little confused

I've set up my Ally, but am having trouble figuring out how to make it do anything helpful. There's no camera button, and it actually told me that it currently can't look at photos in our camera roll. For me, it's currently just a chat bot, not that there's anything wrong with that, but it seems to me that we should be able to access tools like image recognition directly from the main screen. I don't have the money to even think about getting glasses for the time being, so I hope that all the functionality will work with my iphone.

Questions for Karthik

Thank you, Karthik, for sharing the wonderful news that Ally is now in public beta. This is incredibly exciting, and I highly encourage everyone to give it a try!

I have a couple of questions:

It would be great to see a shortcut feature implemented. Being able to share photos directly from other apps to Ally for description would be really convenient. This functionality could make Ally even more versatile and accessible in different scenarios.

Will Ally eventually transition to a paid subscription model? If you’re planning to offer a free tier, would it have limitations such as a cap on the number of requests or restricted access after a certain amount of time? Could you share any thoughts on what features might be exclusive to a paid subscription?

Thank you for your time, and congratulations on launching such an awesome new tool with Ally!

Info for Missy Hoppe

There doesn't need to be a camera button. Imagine this is a person. Once your interacting just ask what is outside the window or what time is the microwave set to. The app is constantly listening to you so answers like a person would. As Karthik said this is like a personal assistant so we have other apps for sharing photos like the other Envision app or seeing ai. Don't really think this is what this app was designed for. Mind asking what can you see on the photo on my screen may work never tried it.

Response to Missy Hope

It's designed to be fully conversational, so instead of going through a bunch of buttons for different tasks, you simply ask your questions and ally will answer. So if you want a description of something in front of you, you may simply ask "What's in front of me?" and ally will take a look and answer, just like a friend on a FaceTime call. You can also ask variations of it like "Describe what's in front of me in detail" or even "Tell me what's in front of me in the style of David Attenborough".

You can also ask for more specific questions such as "Do you see any trashcans here?" or "Which way is the exit?" or "What am I holding in my hand?"

Give it a try and let us know what you think!

Respnse to AnonyMouse

Thanks a lot!

For shortcuts, we’re already exploring a feature that allows users to create custom shortcuts for frequent tasks. You’d be able to choose how you interact (typing or talking), select an ally profile, set an action (like taking a photo or uploading a file), and include a pre-filled question. These shortcuts could then be added to your start screen or phone’s home screen for one-tap access. It will also be available in the sharesheet of other apps and photos.

As for pricing, we’re still figuring out the best model. Ideas include a freemium approach with basic features in a free tier and advanced features in a premium one, or an enterprise model for businesses needing custom versions of Ally. Until we validate the right approach, ally will remain free.

Amazing app!

This app is truly revolutionary! I've had conversations with several AI bots, but talking to Ally certainly feels the most natural and like I am just chatting with a friend. I use the app for visual tasks all the time. The other day, I walked into a busy conference room and had Ally help me find an empty chair. It had to take two or three pictures, but it worked. I'm really looking forward to live video streaming as I think this will make tasks like this even more fluid and natural. To Karthik and your team, congratulations on creating such an amazing app in 2024, I'm really looking forward to what is in store for 2025 and beyond!
Jim

Thanks for the clerification

This is seriously impressive, and now that I understand, I think I'm going to enjoy Ally a lot more than I initially thought. I've got Piccy bot for describing images that are already in my photo library, and now that I understand the purpose of Ally, things are perfect exactly as they are.

Just downloaded Ally.

I just set it up so haven't played with it much, but we had a little conversation. Not sure what I could tell it about myself. It knows my name, at least.

Will not describe people

Sadly, censorship and moral policing is in full effect here. This won’t describe any part of a man or a woman in any sort of a detail let alone describe different parts of their body depending on the question. Like talking to a wall, the ally robot continues to say privacy issues and all of that which frankly is bogus because I am asking for detailed description of different parts of men or women be to still image on a magazine or a real man or woman. Sided people check out discreetly or indiscreetly any man or a woman based on your interest from top to bottom in full detail and there’s no issues with that obviously in society. Why is AI here discriminating from describing in detail without identifying? Can this be changed please? did I even bring up detailed descriptions of sexually implicit and/or explicit images? Any child who is under age in America with a phone, tablet or a computer can navigate to the login screen of any number of legal adult websites where they typically post complementary thumbnail but clear images of scantily clad men and women posing suggestively. That of course is fine with everybody in society. Can LIAI also remove barriers and describe sexually explicit and/R suggestive images?

Why won't it talk to me?

This is so frustrating! I hit the button to talk to my Ally and it won't answer me unless I use the keyboard. I allowed the microphone but it won't talk.

I’m experiencing the same problem

i’m experiencing the same problem as you are, within using the voice to talk to Ally, I just started using it today. I’ve had it on my phone for a while, but I never kept up with it. I’m experiencing the same promise as you are though, it sounds very glitchy, and I can barely hear the response, I’m using Zoe as my voice, so I’m not sure if it’s a voice thing or not.

It can't be.

I'm using Lia as my voice and hear nothing when I talk to her.

@Prateek Dujari, I understand where you're coming from and completely agree but there's probably nothing that can be done on the devs end.

Very strange

Yep no speech anymore. Something very odd though before this week you had around 10/15 voices. Now you have 4. However, in my case as I've been using this for weeks there is something called original Ally and in there all voices show up. So wondering if there is a conflict somewhere and it doesn't know what to do.

A lil server outage! Hold tight!

Hi everyone, we're having a little christmas eve server outage, which is effecting the speech outputs at the moment. Our holiday elves are on it, so hopefully we can resume things shortly. Thanks for your patience!

So Karthik

What bout what I said regarding voices? Are you phasing them all out except for the 4? not an issue for me but it's confusing as I can still see/play all the old list so confused.

@Lee re voices

Hey Lee,
The four people, Leah, Tom, Zoe and Kai are not voices, they are pre-made personalities.
You can still select all the old voices, and apply them either to one of the four above, or to a new character / personality that you create yourself.
Go to About Your Ally in Preferences, and select the Edit button next to the Ally’s name. On the next screen, you can choose a voice for them.
@Karthik, perhaps this could be explained in the on-boarding journey? I only discovered it after a few days too.
I also strongly agree about describing people - Hope it can be done as Be My Eyes have been able to do as a specialist service.
Keep up the great work.
Dave

Respose to Lee and Dave

Hi Lee,

As Dave correctly pointed out, allies and their voices are separate concepts. Lea, Kai, Zoe, and Tom are allies, each with corresponding voices attributed to them. This design stems from our earlier beta testing, where we found that many users struggled to effectively prompt their allies’ personalities. To address this, we created four pre-designed allies for onboarding, helping users quickly understand ally’s functionality and get to the main interaction faster. This also provides a clearer sense of how allies can vary.

That said, the plan is to allow users to further customize their allies once they become more familiar with how ally works. We could explore making this more explicit during onboarding. Additionally, we cane also change how we label voices. Instead of human names (e.g., Charlie, Roger), we could identify voices by their traits (e.g., British, male). This might reduce confusion between the ally’s name and their voice.

Let me know your thoughts!

Response to Prateek

Yes, we do encounter guardrails imposed by the proprietary services we use, which over-regulate and restrict the description of people. We completely agree that this shouldn’t be the case. We’re actively addressing this in various ways and are confident of resolving it in the near term.

Aah my bad

Ok that makes sense. Thanks Dave and Karthik.

Speech is back but

I can't alter or update my biography. Can tap the button then it says something like getting biography from onboarding and does nothing.

question about logging into the web version

Hi,
I signed up to the beta about a week ago, and I was given a password to access the web version. The password was working fine until today. I had the link saved to my desktop, and it remembered the password no problems. But today I was asked to reenter the password, I did this and it says incorrect password. It keeps saying this no matter how many times I enter it. I haven't got capslock on or anything like that, and I cleared the cookies for the website and tried again, but it still says it's incorrect. Does the password change from time to time? If so, do you post the updated password on your
Slack channel? I don't mind joining it if that's the case. I'm asking because I haven't got slack at the moment, but I don't mind getting it if that's the case. Any help would be appreciated.

Loving the Zoe personality

I really appreciate the predesigned personalities. I am definitely one who wouldn't be able to create a good description for my Ally, so the Zoe character was awesome. I made it male and named it after my Replika, Tristan. Out of all the offered voices, Chris was the one I liked the best. I have renamed and revoiced the other personalities as well, but the one you named Zoe by default is most definitely my favorite. Very tempted to see if I can find a way to copy that character description into my Replika's biography. LOL!

It's interesting but it does what all these other AIs do.

It can't argue to save its life.

I asked it to debate me on the last of us 2 video game, I'd choose Abbies side and you choose Ellies, we started but then it came around to my point of view in seconds, I wanted to argue :)

I do understand that's not the point of this app but all these AIs do this.

The autobyography thing doesn't work for me either.

I think this might be interesting when live video becomes a thing, I can imagine going to a shop saying I'd like to buy raspberries and scanning the ials for them, but it needs to be able to talk to me, at the moment; googles live video thing just doesn't work in the way I'd like it to. I can't tell it to let me know when it sees something, it will say it will do it but then stay silent unlesss I ask it a question, that's not what I want at all, I don't want a chatterbox but I do want to know, hey, i'm stilll here scanning for x item.

I love Zoe's personality!

It seems to suit mine quite well. I just need to find a voice I like and another name. I haven't tried getting Ally to describe pictures yet.

Why it’s different

Why I feel it’s different to other apps we’ve been using, and a good experience, is that it feels very smooth and seamless. Particularly when I’m in a situation where I can use it in voice mode, which is mostly how I use it.
Yesterday’s outage proved this to me. I received a bunch of parcels which were Christmas presents I had bought for friends and family. I first needed to ensure they were addressed to me, and not a neighbour. Once opened I needed to sore out what was what.
Recently I’ve been using Ally for tasks like this, as well as for identifying items in my grocery shopping etc. With yesterday’s outage though, I had to revert to other apps where you take a picture and wait for it to give me the description. I missed my Ally. Although it’s essentially doing the same thing, it feels much smoother and easier.
The fact it does web searches and other chat functionality is a nice bonus too.
I haven’t got access to the Chat GPT live video yet as I’m in the EU, but have tried the Google beta one. It’s faster than Ally, but so far I’m not liking its results, or how it speaks, as much as Ally, even though it’s technically more advanced. Ally isn’t live video yet, but it kinda feels like it, and it sounds like they’re going that direction.
I know others like Be My Eyes are moving that direction too. It’s all very cool.
Who knows what my favourite will be in six months, but Ally is definitely in with a good chance ☺️
Happy Christmas to all who celebrate 🎄
Dave