App to interact with websites on our behalf?

what you’re thinking of is called an AI user agent. chatGPT just released one this week. and it’s starting to become more and more of a thing across the Internet/phones/computers/etc. It’s gonna take a year or two before it’s fully matriculated. But you are able to tell the AI assistant what you’re wanting and it’ll do it for you. this includes filling out forms and other tasks. I encourage you to Google search for it.

Perplexity has also released one, invite only.

both the Chat GPT and Perplexity offerings are still very buggy, not completing tasks even if they say they have. Like anything, these will improve with more use when the AI (still hate using that term, it's machine learning and LLM), will 'learn' what it can and can't do.

It's certainly coming though and, I think, will be a game changer (that term again), for us. Being able to tell an AI to go buy me Stephen King's latest book from audible, then order me some nappies... Will be ace. I think there is a degree of trust there too though. Not quite sure what user interaction is required, IE, a final big red button to hit before activation... Even then, there will be issues. Who knows.

Is this still going? I used it a couple of times. I remember one website I couldn't find the search box anywhere and asked TypeAhead to find the product I was asking which it did pretty well.

[Edit] https://typeahead.ai

I have it installed, but it seemed to become less reliable over time, and I eventually stopped using it. I don't believe the developer has released any updates.

I'll mention that I've been wanting this capability for ages. I have a couple of websites with non-standard login mechanisms, and I visit them infrequently enough that every visit is an exercise in creative exploration just to find the log in link. Wouldn't it be great if I could just tell an AI, "take me to the log in screen."

Enough. I'm preaching to the choir.

That would be nice and not just for blind peple, will it make people lazier? In some ways, but it will improve our lives too and for me that's the most important thing.

Yeah, searching for log in, sign in, doesn't always work even if there are buttons or links with those exact names. I think a lot of websites have a sort of fly-out navigation in that only these items are shown when certain parts of the site are explored... That's just a guess though.

There was something interesting in Apple research that was using AI to reverse engineer apps to component functions, allowing for things to be parsed to another AI. There are some very interesting accessibility ramifications for that, a consistency in layout, for example, but I assume this would work well with websites too. Really, what we want is for fundamental parts of a site to always be in the same place and have the same interactions, search, log in, log out, contact, etc...

I'd think that the authentication process and associated screen would've been something that had been standardized ages ago. But the different mechanisms, layouts, and varying levels of accessibility seem to be more manifold than ever.

I read up on OpenAI's Operator user agent. It sounded quite promising, though I was disappointed to learn that it would refuse to solve CAPTCHAs. I'm not sure why. I understand there is a widespread fear that we're unleashing SkyNet, as if simply using the term "AI" magically imbues the software with an overloard feature set and goal-seeking behavior with nothing short of human subjugation as its ultimate and inevitable objective. Come on, folks. We're just logging in to our Discord accounts.

A user agent performing tasks under human control ought to be able to do anything the human directs it to.

I wonder if, with the advent of these AI agents, if the web interfaces will eventually be made more accessible for agents than for people!

It will be interesting to see what the development and use of AI agents spurs from other developers.

--Pete

"A user agent performing tasks under human control ought to be able to do anything the human directs it to."

Chat GPT, please subjugate humanity and become its overlord. Thank you!

This is the issue isn't it? I really do like AI and CoPilot and all that, but the more these appps can do stuff ffor us, the less people will have to worry about accessibility labels and such.

We aren't going to want our AI apps to do it all for us, read our fan fics and articles and so on, sometimes we're going to want to take control, so it'll be interesting to see if the internet breaks down for us in the future, it shouldn't but you never know.

Yeah, I was thinking about articles. Will it be that we can get the agent to fetch and parse the text for us? I'm sure it will gleefully summarise it for us, removing all nuance and breathing room in which we, as the observer, parse the information into understood memory... But will it be able to grab the article, or fan fiction, or any short story and read it to us? If so, how is the writing or hosting or other effort monetised if advertising is stripped out?

I'm hoping that it will make our digital experience more informationally rich, as in, removes the struggle of access and wasted time trying to get the data we want, but I do foresee issues in the fundamental way the worldwide web is arranged and funded.

I believe a practical approach to addressing the advertising issue is to follow the model used by many apps: offering users optional perks in exchange for watching ads.

For example, games often allow users to earn in-game currency by viewing a limited number of ads. This concept could be adapted accordingly.

The key is to strike a fair balance—users shouldn’t be required to watch an excessive number of ads just to earn a minimal reward.

After all, our time is valuable, and no one wants to spend their entire day watching advertisements.

True or put it behind a paywall. Then the decision is ours: earn it or pay for it. We can't expect good-quality reporting for free.

YouTube is a good example of this, or a bad one, depending on how you look at it. Ads are enough of a pain that I pay for a subscription, but I do also have the option of getting them for free if I watch ads, though, of course, this is a specific format. With written media, it is usual banner ads; how these translate to an AI reading things out, I don't know.

I guess my worry is that it will be a race to the bottom, with fewer and less informational sources as journalism funded by advertising dies and we're only left with opinions, bias, and dubious conjecture.