hey everyone
i'm matt, i'm one person in northern california and i built an iphone app called RealTime AI Cam. i emailed michael first and he gave me the ok to post it here and ask for feedback rather than just drop a link.
what it does, you point the camera at something and it names what it sees, it reads printed text out loud, and if you press the what's this button it describes the whole scene or the whole page in one sentence out loud. bills, packages, street signs, a room you just walked into. on phones with a lidar sensor it tells you how far away something is, and it translates spanish text to english.
the part i care about most, all of it runs on the phone. no account, no signal needed, nothing leaves the device, it's free and there's nothing to buy inside it. the model that does the describing is one i fine tuned myself and it ships inside the app instead of calling a server, which is why the download is around 700 mb and why it needs ios 17 or later. it's a small model so it does get things wrong sometimes and i'd rather you hear that from me than find it out cold.
i went through every control about a month ago after a talkback user on android told me several buttons weren't labeled, and i think everything is labeled now, but you all will find what i missed. if you try it i'd really like to know what's unlabeled or awkward, what gets read in the wrong order, and where the descriptions just aren't useful. i read everything and i'll fix what i can.
https://apps.apple.com/us/app/id6751230739
thanks
matt
Comments
Matt
I think one option that would be useful would be to have the camera automatically take the picture after a few seconds.
a few seconds before it fires
thanks dennis, that is a good idea. so you would aim it, press once, and get a few seconds to hold steady before it fires? curious what feels right to you, three seconds or five, and whether you would want a spoken countdown or just a beep. writing it down either way.
a spoken count down and no pressing
No pressing required and a spoken countdown. It would be useful if you had to take a selfie or wanted or needed to snap a picture of something. The reason I say no pressing is you could accidently move the camera while pressing. I would have a 3 second countdown.
response to prev comment
i agree on this :)
AI camera app
Hey there Matt. I like the idea of the on device ai model.. I have just started playing with it and I notice one thing right off. I think it would be easier to have a turn on flashlight button, then a slider so you can adjust the flashlight brightness.
AI camera app
Hey there Matt. I like the idea of the on device ai model.. I have just started playing with it and I notice one thing right off. I think it would be easier to have a turn on flashlight button, then a slider so you can adjust the flashlight brightness.
Some more thoughts
After playing with this app around my apartment, I have some more thoughts. The what's this picture mode is pretty respectable for an on-device model. For instance, I took a picture of my kitchen and got the follow: You are facing a corner of a room. A closed white door is on your right, and another closed white door is at the far end of the room. A red fire extinguisher sits on a shelf above a black trash can, which is against the wall. All of the object it mentioned were correct, and in the places it said they were. So, definitely not bad for offline picture taking. The object detection however... was something else. It confidently said that my bathroom contained a skyscraper, a tower, and a jelly fish. Lol. So, I think the object detection could definitely use some work. But, overall, I'll keep this in my toolbox in case I really need to take a picture of something offline.
A few more thoughts
Also, perhaps you could make so we can have the app describe photos we have on our devices. Also, maybe there could be an option to save the photo along with its description to the camera roll.
it's built, three seconds and a spoken count
dennis you talked me into it, and it's built.
you tap what's this once, it counts three two one out loud with a little buzz you can feel each second, then it takes the shot, so your hand is off the phone when it fires. three seconds like you said. there's a switch in settings if you ever want it off.
the part that sold me was you saying the pressing itself is what moves the camera. i had been thinking of it as a convenience and it isn't, it's the thing that was blurring the shot. harmonicaplayer, thanks for the second vote, that's what made me quit asking and go build it.
it went to apple tonight so it'll come through as an update once they clear it.
flashlight, library photos, and yes the object detection
matt, this is exactly what i was hoping someone would do, thank you for actually walking around your apartment with it.
the flashlight is changed. one tap turns it on all the way, one tap turns it off. it used to make you pick a percentage first, which is a handful of extra flicks before any light happens, and i don't think anyone really wants a dim flashlight. i skipped the slider because a slider is an adjustable element and that felt like more work rather than less, but if you'd still rather have one say so and i'll put it back as an option.
describing photos you already have is in there now too. there's a photo button next to the flashlight, you pick anything out of your library and it describes it the same way. saving the picture and its description together to the camera roll isn't built yet, that one i'm writing down.
on your bathroom containing a skyscraper and a jellyfish, there is an indoor and an outdoor setting on the detection screen and that is the fix. it isn't a sensitivity slider, it swaps the whole list of things the detector is allowed to name. skyscraper and jellyfish are not on the indoor list at all, so in indoor mode it cannot say them and what you saw would not have happened. it defaults to the broad list, which is my fault for not making that obvious, and i'd rather that be a better default than something you have to know about.
try it on indoor and tell me whether it settles down or just picks a different wrong thing. that tells me whether this is a defaults problem or a detector problem. either way the what's this button is the model i actually trained and it's the part i'd trust most.
all of this went to apple tonight and the android version went up to google play too.
want early builds, just email me
also, open offer to anyone reading this. if you want to try things before they go out to everyone, email me at matt@ineedhemp.com and i'll put you on testflight. dennis is already on it.
no catch and nothing expected back. if you poke at it and tell me what's wrong, that's worth more to me than anything else i could be doing with the time.
option to turn off the built-in voiceover
Hey, I think what would be a great feature if you had a Settings menu and the home screen where you can turn on or off the built-in voice guide so voiceover doesn’t speak over it
done, there's a switch for that now
cash, yeah, no problem, that's a good catch and it's built.
there's a switch in settings now called speak in the app's own voice. turn it off and the app stops talking entirely and hands everything to voiceover instead, so you hear it in your voice at your speed and nothing is talking over anything. it covers the descriptions, the countdown and the object detection screen, not just one of them.
it's going into testflight right now and i'm resubmitting to apple tonight. if you want it before it clears review, email me at matt@ineedhemp.com and i'll add you to testflight.
Matt Macosko
Congratulations on the application. Could you expand the language support and add localization for French, Italian, Russian, and Bulgarian?
Excellent work
Thanks for your quick feedback response. The fact we can have local AI models on our phones is impressive. And thanks for making this free. I'll submit more feedback once I can download the new build you've submitted. Again, excellent work and keep it up.
Mac
I am trying it on the mac and it works great.
mantanini, on the four languages
mantanini, thanks. you asked for french, italian, russian and bulgarian, and i want to make sure i'm answering the right thing, because i don't think you're asking me for a cosmetic translation.
everything this app does, it does out loud. the buttons are english, the descriptions are english, and the object names are english, so if your phone is set to bulgarian the app still turns up talking to you in a language that isn't yours. that's the actual problem and it's a fair one.
so here's where each piece really stands. the object names are a list of six hundred and one words sitting in a file, and translating that file is the easiest thing on your list and probably the one you'd feel the most, since that's the part that talks constantly while you're moving the camera around. the buttons and menus are the same kind of work. the app can speak all four of those languages, apple ships a bulgarian voice and i went and checked.
reading printed text out loud is where it splits. french, italian and russian are already supported by the text reader built into the phone, so those three are close to free for me. bulgarian apple doesn't support at all. bulgarian and russian share almost the same letters though, so the russian one may read a bulgarian sign well enough to be worth having, and i'm not going to tell you it works until someone points it at one.
the scene description is the honest no for now. that model was trained to write english sentences and it can't write anything else, so translating what it says is a separate job i haven't done yet.
rather than do four languages halfway, tell me which one is yours and i'll take that one all the way through. i'd want you reading the wording before it ships too, so you're not the one stuck listening to something a machine wrote. email me at matt@ineedhemp.com and i'll put you on testflight.
Hi
I speak French, Italian, and Spanish, as well as a little Russian. Regarding the Bulgarian language, the mistake was caused by dictation. I apologize for that. It was my fault for not reviewing what was written.
response
going to email you keep an ear out matt
french first, and bulgarian is off the list
mantanini, no need to apologize, dictation does that to all of us. and it actually makes this easier, because bulgarian was the only one on your list i had a real problem with. french, italian and russian are all handled by the text reader on the phone, so everything you actually asked for is doable.
you named three though, so let me pick one and you tell me if i picked wrong. i'd start with french. it's first on your list, you speak it, and it means when i hand you something to check you're reading your own language instead of doing me a favor in your third one.
what you'd get first is the object names and the buttons, because that's the part that talks constantly while you're moving the camera around. reading printed text out loud comes with it. the scene description stays english for now, that one's a bigger job and i'm not going to pretend otherwise.
one thing you can use today, the app already reads spanish text and speaks it back to you in english. that part is built.
email me at matt@ineedhemp.com and i'll put you on testflight. i'd want you reading the french wording before it ships, not after.
harmonicaplayer, maldalain, and matt
harmonicaplayer, got it, i'll watch for your email.
maldalain, you're the first person to tell me it runs on a mac, and honestly i didn't build it for that, so that's good news to me. if anything is laid out wrong with a trackpad and a keyboard instead of a finger, tell me. that's the part i've never looked at.
matt, you don't have to wait on apple. email me and i'll put you on testflight and you'll have the countdown, the one tap flashlight and the library photos today. still writing down saving the picture and its description to the camera roll, that one isn't built yet.
burgundy!!!!
some good thoughts, I'd love to have a burgundy phone again. I used to use an se3 in product: red, and loved the color but burgundy is my favorite color, and that's because I can still see it. It makes me feel safe. As far as duo goes, I'd love to see one, maybe get one, because of the touch ID. Face ID is useable, but I'm not fond of it. I do wonder, as someone with small hands though how useable braille screen input would be on a huge phone like that. How much bigger, unfolded, would it be from the 16e? I can't believe how high the damned price is though, that's ridiculous. Maybe when I'm in better straights financially (whenever that will be) I'll get one.
Fi
I hope you're doing well.
This post caught my attention but is not quite what I'm looking for.
The problem to be very honest with you is that already exists.
That doesn't mean get rid of the feature
but I think there's just lots of those popping up now.
however.
there is something I do need.
I need to be able to take good photo's
where it instructures using the ai move the camra a bit right
tilt it up because it's not showing the lighting
This is a perfect view
and if it's got a feature where once it's perfect it has a feature where it just snaps the photo for you.
You tell it a bit about what you want
but it has the photography proffesional knowledge and can help me with lighting
and setting the right settings for me so that I can take good photos.
It's diffcult right right now where in the camra app I have the view finder.
I used to have to screen shot then ask be my eyes is it right
by then I'd reposisioned the camra in my hand just from holding it there not being able to keep it still.
Now even still I have to get it described in the new update which is quicker but I would still have to use 2 hands to ask it should I ajust it? and how. what settings would be best?
So if you could do that!
I'd absolutely download your app.
I'd be first to give you feature ideas requests and test it with real uses in my every day work.
You know?
Do you think you can do that?
if not can you put me on to something that can?
on aiming the shot
hey amrit, that's the clearest version of this problem i've read, and you're right that it isn't what the app does today. it describes, it doesn't help you aim.
here's the honest shape of what i can do. the app already measures whether a shot is dark, blurry, washed out by glare, or cut off on an edge, it just does that after you take the picture. moving it to before the shutter is the real change. so it would say one line, a page cut off on the right, or too dark, or a person near the left edge, and give you a few seconds to fix it before it fires.
on knowing the subject, i'd rather it tell you what it locked onto than guess what you meant. if it says a person centered and that's what you wanted, you hold still. if it says the wrong thing, you move. that way it never confidently points you the wrong direction, which here is worse than saying nothing at all.
the part i can't do is the settings side. i can't give it a photographer's judgment about exposure or what looks good and i don't want to pretend otherwise. it can tell you it's too dark and offer you the light, that's the honest edge of it.
the full version you described, where it keeps talking until it's right, takes the shot itself and saves the photo for you to actually use, is where i want to end up. email me at matt@ineedhemp.com and i'll put you on testflight so you see it as it gets built. i'd rather shape this one with you than guess at it.
flashlight
Hi,
If I turn the flashlight on, get descriptions of what I want described, do I need to turn the flashlight back off before I close the app from my app switcher?
flashlight, good catch
moopie, thank you for asking this. you shouldn't have to think about it, so i'm changing the app so the flashlight turns itself off whenever you close the app or leave a camera screen. that goes out in the next update.
Realtime reading
Thanks for developing this tool Matt, really appreciate it! I'd also love to be able to read the text in real time while being offline if possible.... I'm talking SeeingAI but offline mode. Comes in handy when the connection is unstable/nonexistent
already in there, works offline
thanks so much, glad it's useful! good news, that one is already in there. on the home screen tap English Text to Speech and point the camera at any printed text, it reads it out loud live as it sees it. it all runs on the phone with no connection needed, so it works fine with bad signal or airplane mode. if anything about it doesn't work well with voiceover let me know and i'll fix it
This app sounds great! Can't wait to check it out!
I always enjoy adding new apps to my vision assistance toolbox, and ths sounds like a worthy addition.
glad you're here
thanks missy, that means a lot. we're adding and testing new things every day, so it keeps getting better. if anything trips you up or you think of something you'd love it to do, just post it here and i'll get on it.
Document scanning mode
Hey Matt. Great work with the app. I'm definitely keeping it around. Quick question. Could you add a document scanning mode, so, say, when scanning something like a sheet of paper, a piece of mail or whatever, the phone could give you guidance to make sure the whole thing is in the camera's view? Something like, move left, move up, hold steady, etc. Also, I noticed that in the what's this menu you can switch between normal and wide angle cameras. Does the wide angle work even with phones like mine that have only one rear camera? I'm using an iPhone 16e. Again, thanks for all your good work. Keep it coming.
document mode, and the wide angle button on your 16e
matt, thanks for sticking with it, and both of these are good ones.
the document guidance is already being built. it's a new button on the home screen called help me aim, and one of its choices is picture or page. you tell it you're going for a page, it finds the edges and talks you in, move the phone left, back up, hold steady, then it fires on its own and keeps the sharpest, best centered frame. it's on my phone right now and still in testing, so i'm not going to put a date on it, but that is what i'm working on this week.
on the wide angle button, the straight answer is that on your 16e it does nothing. that phone has one rear camera, so when you tap it the app quietly falls back to the same lens and only the label changes. that's my fault for showing a button that can't do anything on your phone, and i'm going to hide it when there's no second rear camera so it stops being confusing. thank you for catching it.
if you want help me aim early, email me at matt@ineedhemp.com and i'll add you to testflight.
App not working on iPhone SE 2022
Given this app is fairly small for an app that bundles a local AI model, and the iPhone SE 2022 should actually meet the hardware requirements to run it, I would expect the app to launch and be usable without any issues, but that doesn't happen at all. I came across way before you posted about it on here, but then I had to delete it. Another reason that urged me to delete the app is that it translates text from Spanish but not any other language. You can definitely use Apple's own built-in offline language packs rather than fine-tuning the model yourself. So the app should perform OCR and translation for any language that the device can handle rather than only Spanish and English.
maybe feedback?
What if things like the flashlight button also was an adjustable value? For example, let's say someone needed dimmer light. You could control the amount of light with swiping up and down on the flashlight button or if you just want to turn it on you can just double tap the flashlight button. And that could be the case for other settings as well for example the object descriptions, You can swipe up or down on the button to switch between indoor and outdoor.
Re: maybe feedback?
Ernesto, thanks for this. Your second idea is built: in object detection, All, Indoor and Outdoor are now one control called What to look for, and you swipe up or down on it to change it. It's in the next update.
I kept the flashlight at one tap, full brightness, on purpose. A tester asked for that last week, and the camera reads best with all the light it can get. If you ever need it dimmer for a real reason, tell me and I'll look at it again.
Re: App not working on iPhone SE 2022
Enes, thanks for being straight about it. You're right that an SE 2022 should run this fine, and if it won't open, that's a bug on my end. A lot has changed since the early versions, so if you're willing to try the current version one more time, I'd really like to know whether it opens. If it still doesn't, tell me what happens when you tap it and I'll chase it down.
You were right about languages too, so I built it this morning. The translator can now read French, German, Italian, Portuguese, Chinese, Japanese, Russian, Arabic and more, using Apple's offline packs. Each one downloads once, then works with no internet. Spanish still uses the built-in translator with no download. It's in the next update.