New AI app for describing images and video: PiccyBot

I can't seem to ask further questions for either videos or images. On an iPhone SE 2020 running the latest version of IOS.

Also, could we have a haptic feedback option when images or videos are being processed? Similar to the Chat GPT app if you're familiar.

I can't find any option to capture a video or upload one. How/where do I locate it?

Ever since this was posted, I was convinced this app was called Pixie Bot. Last week I made the mistake of letting my wife use my phone and she found that it is actually called Piccy Bot. Well, that's not how VoiceOver says it! Very disappointing.

Anyway, I just wanted to say I was using this quite a lot over the last 2 weeks on holiday - it was nice to be able to take pictures with the Meta Ray-bans and then later on send them to the pixie for a proper interpretation. (Meta AI is OK for a quick note about what you are looking at but not a patch on these models). I hadn't noticed that the app had switched me to Gemini and it was incredibly good. I should also say that I probably use this in preference to other apps because of the tts voice it uses. My wife enjoyed listening to it too, particularly when I turned on personality mode and it made her sound like a celebrity.

The only thing I found is that if you are listening to a lot of photos you took on the same day it can be a bit tiresome hearing the same old comments about the sky or overall impression etc etc. In some ways it would be nice if I could have pointed it at a day's photos and the pixie could have given me a single detailed description encompassing all the photos. But that might be just this specific use case.

It's unbelievable to have a video option suddenly appear. I tried a few videos - they mostly worked fine in that they told me some of what the video was about but the level of detail is nothing like with photos. I'm hoping this might improve a little over time as it's an exciting new feature.

I love seeing the new things being added to this app - thanks so much to the dev.

Make sure you are on the latest app (pull down to refresh in the app store when looking at updates just to be sure). Then there is a Video option that is just before Microphone in the app. So four finger tap at the bottom to get to Camera, then swipe back twice.

As far as I can tell there is no way to do this via the share sheet, but hopefully that will come soon!

for example, in safari, one can tap and hold an image and tap share and choose any of she sharing option without download the image first. usual suspects such as mail, message, etc. all showing up, but piccybot is not there. i don't know why it only shows up in the photo viewer and not other places.

First let me say, I love how much update it's being done on this APP. It makes me feel that even when there's glitches and bugs, or things we don't like, it'll get addressed sooner rather than later.
Personally, I favor using Claude Opus (I miss my Claude Opus!) and/or Lama3 cause those two are the best when dealing with (sometimes) nude or erotic artwork as I have to identify and label/catalog them in my work. While Gemini and GPT 4O have consistantly the higher quality of description, they are so heavily censored they'll even reject none-nude art subjects even if when it's just a semi-erotic statue. Lama3 does give really weird description at times, such as the other day I asked it to describe me and it said I was covered in hair (which I'm definitely not), and it said I was wearing "A dark shirt, a dark gray shirt, blue pants, and pants" ROFL. WTH
Now on to the latest. I love the potential this video description will bring, but needing to have the video on your phone first is a drag. Be allot better if we can say, find a youtube video, and then use the "share" function to share it to the APP. Also I think the APP only describe the first 30 seconds of the video. I thought if I scrub the video to say, 10 minutes 30 seconds, it'll describe the scene there, but it doesn't appear to do that. Also many videos are simply returning "server error."
There again, this video description is very new, so I expect in 3 to six months all this would become none-issues.
Thanks for the continuous update and improving of the APP!

Hi Martijn.
This was fun but the descriptions are lack luster and bland when personality is off & the description of 1 video by using AI model LLAMA 3 was strange! It told me something that did not happen after I confirmed it with a friend who sent me the video.
After that I stopped using LLAMA LOL

I took various short videos of my apartment which is nicely decorated, before I lost my eyesight, I was heavily into interior decor and watching lots of HGTV for all these great ideas so I decided to make my place stylish but using this app to get the descriptions by video was disappointing. It's not your fault.
I hope this gets more advanced but in the meantime you have done very well. Bravo! I'm proud of you for a cool new feature...
even though there's bugs to kick out but keep up the good work.
Please add a share sheet option for the photo gallery situation for videos, all the error messages are terrible also.

I mean, it's a singular description. I think what it's doing at the moment is summerising the whole video... It's a video of a man pratting about dressed as a mushroom... For example.

I'm not sure how we would asociate the timeline with what is going on unless we do indeed have something like audio description where the description syncs up with the video as it is playing.

From the looks of the vision aspect of GPT 4.0 with the conversational feedback, this might be possible but here, with a singular description after processing, unfortunately form doesn't quite fit function. Don't get me wrong, it's really cool, but I think tempering expectations is needed. The fact that it can do it at all would have been mind blowing a year ago, now we're demanding so much more. We adapt so quickly to the new norm, it's scary.

I followed the developer of this app on YouTube and stumbled upon this on my feed.
https://youtu.be/AnigBgi-_Ow?si=X_1pGQ5O8-hmfCEz

It's not trained to be an audio describer, if it were then we'd get a lot more out of this.

We'd need to run the video through an app to get the markers and all that sinked up but once that's done, it would be pretty cool.

I think Lottie asks a good question, but like photos it depends a lot on specific use cases.

For example, I mainly use this for looking at memories - either holiday snaps or other things. And I was trying to use the video option in a similar way. So in my particular case I wasn't really needing audio description, I just wanted it to be able to give me a decent amount of detail to help me know what was going on. For example, it told me one video had my wife on a beach playing fetch with my dog, and it was good to get that but it hardly made me feel like I was there and getting a sense of what was going on. Another video was of my other dog no longer with us who was making funny noises in the dog bed whilst the recent cup final was on 5 live. The description picked up on the audio and told me the dog was there and looking tired but that was about it. I'm not sure if she was looking tired because she was in a bed or if she was actually looking ill, but it wasn't really what I was hoping for. I guess I wanted something that would give me more of a sense of her personality.

But if I was looking at an instructional video or trying to accomplish a task specifically then my requirements might be entirely different.

I think it might be good to have different options for videos - in terms of verbosity, AI model and so on. For example, Claude seems good for images, but it told me that there was a voice commentating on my dog in the video and she certainly wasn't heading in the winner against Chelsea as far as I was aware.

@Martijn yes, having something like a slider as to the amount of compression would be the optimum. The problem before the update was that the video was being described almost like how a photo would be. Meaning the AI wasn't picking up on any movement or progression within the video.

I noticed since the last update, I'm no longer seeing the app in the share sheet.

As of last update, the quary history is not showing up to be used as quick prompts. Also, using Lama3 and switching to Chinese language, it would randomly generate in English rather than Chinese, and many times the Chinese generated text would bug out and start repeating itself like a broken CD. Like it'd say, The person has a long beard and is wearing a blue shirt, the person's shirt is blue and he has a long beard, the person's beard is long and his shirt is blue, the person has a long shirt and is wear a blue beard...and on and on and on. LOL Quite amusing actually.

Having fun with the new updated version.

I used one of your other apps called VoiceArt and decided to gather several photos of a dream home that I gave good detail description to and it created the images for me so I put all of those images into a new photo album and created a video and it described the video which was about 30 seconds long, it gave wonderful scene descriptions of each room, details about the furnishings etc. and it improved the whole experience. Loves it.
I'm proud of the work you have put into this.
For now, I don't think you need to do anything else but enjoy the fruits of your labor until you feel like you need to improve or add some other feature.
You're welcome!
Thanks for all that you do.

The video showcases a tour of a modern, luxurious home. We start in a living room with floor-to-ceiling windows revealing a stunning view of the ocean. The room is bathed in warm sunlight, and the soft white curtains billow gently in the breeze. The focus then shifts to the kitchen, a space brimming with sunlight streaming in through a large arched window. White cabinets line the walls, contrasting beautifully with the dark countertops. The camera pans over a home gym, equipped with a treadmill, a stationary bike, and a weight bench, showcasing a dedicated space for fitness. Next, we step into a home office, featuring a spacious desk, comfortable chairs, and a large window overlooking a lush green forest. A plant sits on a small table, adding a touch of nature to the room. The video takes us through a luxurious bathroom with twin sinks, a freestanding tub, a large shower enclosure, and a skylight overhead. The walls are adorned with tasteful tiles, and the room is both elegant and functional.
We then see a home theater with a large projection screen, plush leather seating, and dim lighting. The screen displays a vibrant scene with a woman laughing and holding popcorn. Next, the video showcases a master bedroom, with a king-size bed draped in soft linens. The space is punctuated by a stunning chandelier, a plush armchair, and a large rug. We then glimpse another bedroom, with a similar king-size bed, but this one features more modern décor. The room is brightly lit with a large window offering a tranquil view of the outdoors. The tour concludes with a peek into a laundry room, with a stainless-steel sink and a set of modern washing machines. The video ends with an exterior shot, showcasing a relaxing patio with a hot tub, a fireplace, and a grilling station. The entire video offers a glimpse into the spacious and elegant lifestyle of the homeowner.
This was usingPiccyBot AI Model , GPT 4o
However, when I did each photo individually and got a more detailed description then I knew exactly what was being referred to in the video because I did get detailed descriptions of each room & outside backyard using Be My AI.
It's funny using all this technology to see the world through my iPhone.

Hi Martijn and all,
I finally decided to give this app a go after hearing a demonstration on the Double Tap podcast. I always thought it was a pixie too, untill I actually looked at the spelling! The guys were pronouncing it as 'picky'. I didn't try this app at first because I thought it would be as good or as bad as Be My AI, but I liked the voice so decided to give it a go. I have some feedback if you don't mind. First, could you get the app to autodetect languages? Be My AI and GPT do this. The reason I say this is because I deal with texts in multiple languages, for example, English and something else. By the way, it reads text from screenshots of ebooks really well. It can read lots of languages, but I have to tell it which language apart from English I want it to recognise. Also, could you please allow people to share a URL to a video, or to share videos directly from sites like YouTube. So Olly and Lottie, you were asking or suggesting what we want from video descriptions. How about something like this.
The YouTube video is titled, "5 Best Languages To Learn in 2024." a man is speaking. He is wearing a gray sweater and is positioned slightly to the right of the center. Around him are five colorful circles, each with a number inside:
1. Red circle with the number 1
2. Blue circle with the number 2
3. Green circle with the number 3
4. Yellow circle with the number 4
5. Purple circle with the number 5
The background shows a cozy room with bookshelves, a couch, and a lamp. That was from Be My Eyes on Windows. I know this is like a picture, but isn't audio description on TV describing pictures on screen anyway? The only real difference is that audio descriptions on TV or as part of a film tell you which character is doing what. Maybe the descriptions could be divided into frames? Yeah I know, this is like sci-fi. I get it if Martijn, this sort of thing isn't possible! This app is worth supporting. I purchased a lifetime subscription today. This has so much potential. It's even great as it is now. Thank you for making an app like this! It's yet another tool we can add to our arsenal. I think I've quoted that from someone else.

I think you hit on the problem with your example. That is an instant in time. It can summerise the video but, without having a live blow by blow description of aspects that are changing, an actual audio description track,it's, as yet, not very useful. I suppose we could use it to interrogate the video, IE, what happens when he opens the box covered in skulls and crossbones? ... Still, we need to know that there is a point such a box appears.

I think the real time voice and video might be the solution to this, as you say, adding in a link and watching it chummily with an ai who is telling us what is happening real time is the end game.

I think your suggestion of links to video feeds is a good one though I think the back end could get quite bloated if it's having to scrape youtube videos first, download then upload. It's a shame there isn't some sort of shareplay thing we could do to feed a video into the app, which it can watch, uploading real time, and then we can ask it about it.

Hi Olly,
Yes, real time would be the best solution. The closest we'll get is video with Be My AI and Copilot on Windows at the moment. PiccyBot is still a nice app though.

My last post I mentioned the prompt history appears to be missing. I finally found it LOL. It used to be you just swipe right, past the text field, and they'd show up there. Now you have to tap on the keyboard and then they appear above the keyboard. I still think a proper set of "pre programmed" prompts you can add, remove, select from would be better. This works but because it only stores so many, I find myself scared to try new questions or prompts cause it'll bump off the "tested and true" ones. Also when I restart my phone all my prompt history goes bye-bye. It's no big deal since I got my favorite prompts in a text thread to myself so I just copy and paste them in, but still a bit of a drag.

PiccyBot has been installed on my device for quite a while, but only yesterday have I actually started using it as I was interested in getting videos described. I did go through the whole thread from the very first post to the very last one though, and one thing I notice is how responsive the developer is. I appreciate that and hope the app gets better, but would like to touch upon some points, several of which have already been mentioned by others:

• While I can share photos with PiccyBot from the Photos app or elsewhere, this is not the case for videos.
• There should be an option to toggle whether audio feedback is switched on or off depending on the status of the ringer switch.
• Personality should be disabled by default (in the free version), as this is something impacts the app's usability and basic functionality rather than what may be called convenience or extra features, some of which might even be redundant to some. The remarks that the voice makes don't often make sense at all, and it is not really preferable to wait for it to finish speaking so that one can then listen to the text description.
• Can free users not have the option to watch ads and get access to some paid features?
• I presume one can ask only one follow-up question per description in the free version, and would like to ask whether this is also the case for the paid version.
• The app should have better localization support. I can confirm that this is definitely true for Turkish, my mother tongue.
• Finally, I would like to know how exactly our data and images are handled.

"I presume one can ask only one follow-up question per description in the free version, and would like to ask whether this is also the case for the paid version."
In the paid version you seem to have unlimited questions you can ask- at least, I haven't hit a limit yet and I am very chatty with Piccybot LOL.

Still, it could be better if we could at least ask multiple questions in the free version while using GPT, as in Be My Eyes.

I can't seem to find the feature to export images with the description back to my photo album. I've looked under share on the main page but there isn't a 'save button which I think was how it worked?

Call me MM as Martin seems to be a popular name on this thread.
Heard about this on Double Tap and what interested me was the video description.
After reading the comments you may have answered the question.
The video I played is 46 seconds long. It's of me and a girl walking through lots of people clapping.
It described this very well.
The next video is 2.5 minutes long.
It only described the outside of the building, the shop and what two people were doing.
It did not tell me about the people in the corridors, offices and especially more importantly me laughing on the telephone.
This is a video from my last place of work and I would loved to have had it all described to me.
My question: is this all it is capable of describing or will it get better in the future?
I must say in the first video it described the girl I was with better than any human being had done!!!
Brilliant work Devs
MM

So I might actually consider upgrading to the paid version as it may be great to be able to make use of various AI models and use GPT-4o without any third party like Be My Eyes in the middle to store and process my data. Thing is, will PiccyBot eventually let us get complete descriptions for videos longer than 30 seconds? Also, will we be able to get real-time/live descriptions, and can we or will we be able to use LLM's other than GPT-4o to get video descriptions or is it only GPT-4o that can provide video descriptions? By the way, I am not sure we both got one another right. Does the mute switch currently control audio feedback? What I suggested is an option to toggle whether it does or not; not whether audio feedback is on or off. Finally, I can help with improving the Turkish translation.

I think this ap is great. The personality stuff is excellent for novelty value but the factual descriptions are superb. One question I had though: if I save images back to my album with the descriptions attached, the only disadvantage is I lose the dates and times of the photos. That's unfortunate as what I quite like about Apple Photos is the date, time, location information. Losing the original date disarranges them a bit. I don’t suppose there’s a solution to this one, is there?

On video descriptions, I’ve found I’ve got a longer and better description by asking a different question: "Describe the visual details of the video from beginning to end, for the benefit of a blind person, in 1000 words." Or, of course you could have it shorter if you wanted. It seems to do considerably better with that. I ran a video Mrs Bingo and I took of our new house, circa 6 minutes long, and although the AI reported Bingo saying things he certainly did not say (the archway is 14ft wide", "The architecture is genuine Regency" etc. it got the gist of it. I have to say I did not appreciate some of the included implied criticisms—short, scruffy beard, there is a lot of clutter around, the man is mumbling so we can't hear what he’s saying. Still, it was the whole video…sort of.

.

Two things I'm wondering about: Does the flashlight turn on automatically? I'm not clear about that because with other apps I've used, the flashlight doesn't show up at all, and I just take the pic. Second, I don't see a way to clear the picture. For now I'm using the free version, but chances are pretty good I'll subscribe so I can play with the other voices and personalities.

@Bingo Little - that was a great suggestion, thank you.

And for some reason I'd never really picked up on the microphone so wasn't really asking any follow-up questions. (Yes the tiny amount of effort to edit the text did put me off, pathetic though that is)

Yesterday I was watching a video, then used the microphone to correct some details (gender, names etc). When I played it back I was in buckets as it was such a bittersweet memory. It's not the same as being able to see them but considering just a short time ago photos and videos on my phone were just useless files taking up space and now they are memories again. It's astonishing how far we have come in such a short space of time.

I love this app, and I am still gobsmacked that a video feature was added to it - it is incredible, thank you so much.

Bingo's point is a good one, and for the first time in Apple VIS history, I'm going to agree with him... Mark this day, people! :)

If the date could be maintained in the meta data that would be wonderful though I realise what it's doing is creating a new date for when the image is saved.

On that point, I've got a bunch of photos here and processing them is prooving a little difficult as I processed and saved the same photo several times before I realised it was, in fact, the same photo. I'm not sure how to get around this, but saving to a different location or folder, batch processing etc might be great.

to be honest, I think the aspects of this app I use most, is not taking photos, but organising photo albums. an app specific to this, for me, would be great. Batch processing, creating folders, background processing in that I don't need the description or even be able to interrogate it. Literally, all I want to be able to do is point this magic at my photo library, press go, and it populates a new library or folder with these wonderful meta descriptions.

I know it's a big ask, but I think you mentioned you were considering a separate app for such things in the past. It's an app I'd certainly buy. The nice thing then is I could go through my marked up album, pull out the image I want and ask your original app about it. PiccyBatch might be the name... Now, go make it. :)

I love this idea, but less so if I end up with two lots of everything. For some reason when I originally tried the Save Picture option under Share I thought it was updating the original image to add the meta data. So I did it for a couple of pictures, then stopped because I wasn't confident enough about going back and deleting the photos that I had already done, even though it's fairly obvious if you are doing them one at a time.

I'm never really sure how to do batch things in the Photos app. Probably if I could get my head around the Mac Photos app it might be better but I always got totally lost there.

I guess ideally I just want a load of photos that I can browse without needing to pass them off individually. So for me personally I would even accept a different app where it had its own index of my photos with the descriptions against them and I could just browse that way.

But maintaining the date/time would be important. I think if Pixie Batch worked exactly the same way as Pixie Bot I probably wouldn't use it but that might just be out of laziness and a lack of confidence that I'm not going to delete things by mistake.

Your work, as always, is really appreciated. God, i wish you worked for apple accessibility...

Fingers crossed for PiccyBatch. No matter what Open AI comes out with, we will still need good devs to cut through the clamour and give us the application of this new tech.

I downloaded this app and tried to run a short video through it. Immediately after getting the result, I bought the lifetime subscription. Haven't changed any settings, yet. Wow! This app will certainly get used a lot. Thanks for this exciting way to kill spare time! GREAT APP!

Hello, I am very happy to see this post here, but I have been using this App for a month. I discovered several problems during use. First of all, I'm using Simplified Chinese and I found that the voice used to describe the pictures sounded very strange, with a weird accent. Secondly, when using VoiceOver to read the displayed description content, I often feel that the information is incomplete, that is, the description is gone before it is finished. Third, it takes a very long time to wait for results to be returned after uploading images. I'm using the free version now and I'd like to know if a few of the issues I mentioned can be solved if I pay. I'd be more than willing to pay for lifetime use if I could. Finally, thank you for developing the function of video description. This is really amazing. Although video description is not perfect yet, I believe this is a great start.

I completely missed your comment in reply to mine. apologies for that; but I am curious - about what have we disagreed previously? or have you kept your disagreement with me unspoken? either way, I'm glad I kept trying and didn't give up. Finally, I have made an observation to your liking! First of many, perhaps - that's a cracking Devonian beer, by the way.

I think PiccyBatch in principle is a fabulous concept. I wouldn't want two copies of all my images, though. When I think of the thousands and thousands of photos that I have accumulated over the years - think of Whatsapp chats, for example, where folk are forever exchanging pictures - it just depresses me. I did try and organise my photo library a year or so ago, deleting videos I did not want. It took about a century, but I got there. I had a look yesterday and, like weeds on Bingo's new lawn, surplus videos have made a comeback.

Decided to check this app out (have been reluctant due to too many apps already). I guess this has been answered somewhere in this thread already, but it is a bit much to read through so I would appreciate if someone could give me a quick answer.
Are all settings hidden behind the subscription, because I don't see a settings button anywhere? Also, when I have taken a picture in the free version and gotten the description, how do I start a new conversation? Is that even possible without closing the app?
I will probably buy the lifetime subscription anyway since it is not that much, but I want to know if I have understood things first or if I'm just a bit stupid here.

So after a bit of digging I found out that yes, all settings are hidden in the free version, so I bought the Premium one, but I still have questions/bugs:
* How do I clear the current conversation to start a new?
* I read in one post that there is supposed to be some kind of feedback from VoiceOver if you hold the camera straight or not when taking a photo (I guess like in the Camera app), but I don't hear anything of the sort.
* When I try to get an image (from Twitter) described via the share sheet, the app opens but nothing watsoever happens.

Except these things I think this is a really nice app. The ability to choose models, the volume button shortcut to take a photo and video descriptions sets this one apart.

When I try to share a photo via the share sheet to the app, it ow puts me in a window with an edit field and a "post" button with no additional info as to what I am "posting" about. I tried to type in something random and hit "post" then it just sits there doing nothing.

Hi.
I installed the app, and I really liked it, although unfortunately, I can't buy the lifetime license at the moment. What models and voices are available? Is it worth it?

"What models and voices are available? Is it worth it?"
IMO it's totally worth it because it is constantly being improved and is already a very cool APP, even with what little bugs there are :)
Off the top of my head, there's 5 voices to pick from, personality versus no personality settings, and as for models, there's gemini, GPT4O, Lama3, Reca, Claude Haiku, and Claude Sonnett. It used to hve Claude Oppus which was my favorite, but that got buggy and got taken off for now I think.

First of all, I want to thank the developer of this app for all of their hard work. It is very much appreciated. I bought this app a few months ago, but have only recently started playing around with it. I am loving the various personalities and how they describe pictures. I'm not sure, but I think the current version has a bug. In settings, I can't find the place to change which AI model is being used. It used to be that I could double tap on the ai model button and a list would appear. Now, if I t ap on that button, it says selected, and if I tap it again, selected goes away, but unless I'm missing something, a list of ai models is no longer available. I've chosen for the picture descriptions to be as long as possible, and have heard some really great descriptions of some of my old pictures. I'm not overly keen on any of the voices currently available; the Envision Assistant beta has tons of voice options available, but since that app is currently in beta, it's very, very buggy. I'm going to continue discovering pictures from my photos app with Piccie bot, but I look forward to maybe eventually getting more voices and figuring out how to change the ai models again.

Do you think one day we'll get to share youtube videos?

...but can someone please answer these two questions?
* Is there any way to "clear" a conversation except restarting the app?
* Is there supposed to be some kind of VoiceOver feedback in the camera view or have I misunderstood something?

Someone on here mentioned a few comments back that they produced a slideshow and imported it (as a video) to PiccyBot. Simple question; how did that person do that? I can't find any way in the photos app that allows me to save a slideshow I have created. slightly off-topic I know but I've asked elsewhere and it didn't seem to register.

I have to say that since installing this app I have loved saving pictures with the descriptions alongside. I hope we will get the batch processing idea off the ground one day. I've got thousands of photos which I would love to receive the PiccyBot treatment, if we can avoid duplication and preserve the time and date stamps and the location information. I realise that might be a bit much to ask.