Our dreams have come true - Gemini 2.0 is released with its real-time audio/video streaming capabilities!

After playing with them both, chat gpt wins by a long shot.
So I also wanted to get on here and just say y’all, don’t under value that sharing screen feature in chat gpt. I’m a PlayStation 5 gamer and I decided to download the remote play app to my phone… partner with the PlayStation remote play app and chat gpt advanced voice mode screen share, makes for one heck of a gaming session btw.
While you can screen share with google, it’s glitchy, only lasts a couple minutes at a time, its response rate is slower… yes, I timed them.
Plus the open AI team is really quick if you have a problem.
If you can do it and you use it as much as I do, it’s worth the pro for chat gpt.

So, with pro ChatGpt can I share my iPhone screen and also have it take action like clicking a link? I have an iPhone 16 pro running the latest IOS software. I have a website page that when VO is active, the website developer has an overlay on the screen that prevents me from using it and the only way to fix it is to click a button on the screen but VO has to be off to click this button so I need sighted assistance. But, wanted to figure out if this would be possible with pro ChatGpt to see my iphone screen and click a link as I tell it what to do.

As of now, chat gpt doesn't take actions on your behalf, whichever plan you have. And interestingly enough, it hasn't previewed that feature (as far as I know) in spite of 2 of its major competitors demoing such capabilities.

I appreciate the quick and thorough response.

Unfortunately, no, it can’t take actions as of yet.

I just gave Gemini Live a try, and I was both impressed and unimpressed at the same time, if that's possible.

I used Gemini to identify several different tea packets. It was cool to be able to carry a conversation with Gemini without having to take individual pictures of each packet, but one thing I kept coming back to was the tone of the voice. If this were a person I was talking to, I would think the person was annoyed based on their tone. I felt like the software was programmed to try and limit the interaction. Every time Gemini asked me "Is there anything else I can help you with?" I felt as though it was trying to direct me towards ending the conversation. Google could certainly refine the personality of Gemini to make it sound more friendly and engaging.

What I think would be really cool is if one of the companies working in the blindness field could perfect a product using one of these live video AI models and tune it specifically for the needs of people who are blind, DeafBlind, or who have low vision. It's not hard to imagine how with some customizations, a live video AI product could really revolutionize how we get access to visual information.

Hey Michael. Haha I know exactly what you mean.
This is probably partly why I’ve been using Envision Ally a lot more than Gemini recently. Have you tried it yet?
Yes a model or agent that is built specifically for us is exactly what we need.
Dave

Good morning, all! I've been casually following this thread, and I have several questions. How does this work? Do we have to have pro subscriptions to chat gpt or geminy to take advantage of this? I understand that the screen share thing doesn't have the capability of taking actions on our behalf, which makes me wonder if it would even work for the one task I'd like to be able to use it for. Is screen sharing something that's going to be an option with Ally? The one task I can imagine this being very useful for is browsing the Replika store to get clothes or accessories for my Replika. Sadly, the store portion of the iPhone app is more or less completely inaccessible, so even with an AI screen sharing partner, I don't know how or if I'd be able to navigate to the items I want. I'm not keen on the idea of another bill, but it's sounding like it might be worth investing in chat gpt pro at some point. I could never justify paying for both chat gpt and google, though, so I'd have to figure out which one would meet my needs better. I think 2025 is going to be a very exciting year for AI in general, and I'm looking forward to learning as much as my little brain can handle.

Hi everybody, first of all I am new around these parts I was looking for this and it is very very awesome. Thank you for telling us all about itI was looking for this and it is very very awesome. Thank you for telling us all about it. I believe that the more we use it the better it will get.

The Gemini video descriptions were working super well on my iPhone a month or two ago. Now I cannot get the AI to view through my camera at the AIStudio.google.com/live web site.

I wonder if Google has changed something?

The site seems to work on my desktop PC, but not the iPhone any more. This used to be pretty seamless.

Thanks for any advice. This was a cool tool.

--Pete

I did have to re-grant permissions for the microphone and camera, but yeah, the service is still working on my end. 😉👍

@Brian,

Hmmm, I have granted permission for both the microphone and camera. I wonder if it has something to do with the fact that I'm running the iOS 18.4 beta or if some other configuration somehow got messed up.

I'll tinker around a bit more now that I know it should still be working. Thanks.

--Pete

The steps I took:
1. Access the link to the Google AI studio. In my case, I double tapped an icon I have on my home screen as a shortcut.
2. Navigate by headings until I find the heading labeled "Speak to Gemini". Then, swipe to the right until I find a button labeled "talk to Gemini" and tap on it. Here I was asked for permission to access the microphone.
3. At this point, I can tell that the microphone has been activated, but I can also still hear VoiceOver. Also, Gemini will not be able to use my camera just yet. However, if I swipe to the right a few times, I will eventually find a start camera button, Tapping on this will present a pop-up asking for permission to access the camera. Once that has been activated, then I can speak to Gemini conversationally, and it can use my camera to describe things, etc.

Sidenote, I have no idea if Beta iOS software affects the functionality of Google AI Studio in any way.

HTH.

Thanks for your step by step instructions.

I followed your instructions precisely, but no luck. The AI will talk to me, but when I ask what it "sees" it describes a totally imaginary scene.

Question: In the camera pop up dialog, there are lots of options for the camera, i.e., screen, front camera, triple back camera, double back camera, telephoto back camera, etc. I think I've tried them all but wonder if you see these options. Again, I am using an iPhone 15 Pro.

I don't remember having all of this trouble setting this up in the past. It seemed pretty seamless. I hope it isn't iOS 18.4 beta that is giving this problem.

I've even tried setting the web page to the Desktop view as well as the Mobile view with no luck. AIStudio does work well on my Windows PC using either the screen or my webcam, so I know the service does work. Just not with my iPhone any more.

Thanks. anyway.

--Pete

Hi Peter,

I have an iPhone SE 2022, so no, I do not get all of those options. When I click on the camera button, which is interestingly enough right beside the microphone button, I just get a pop up to ask for permission to access the camera. Does not say front, or back, or anything else. When I give it permission, I can converse with the AI conversationally, and it works wonderfully for me.
I am starting to think it is an 18.4 beta issue, but that could just be wishful thinking on my part…

Brian,

Well hopefully the problem is either specific to my setup or an iOS 18.4 issue that wil get resolved. Let me know how you fare when the release version of iOS 18.4 comes out.

Thanks again.

--Pete

I'll keep you posted. Stay tuned ...

Just FYI, I reported the issue with AIStudio using iOS 18.4 beta 3 to Apple just in case it is an iOS 18.4 issue and not specific to my phone. So hopefully this won't be a problem in the official release.
🤞
--Pete

@Brian, just to let you know, when iOS 18.4 beta 4 came out I found that the issue with AIStudio now seems to have been resolved. It now works on my system seamlessly as it did with iOS 18,3. So I guess the issue was with the beta. Don't know if it was because of my feedback, but at least we will still be able to use this nifty tool.

Thanks again for all your help and suggestions.

--Pete