Absolutely. When my mom's instagram got hacked it was instantly apparent that the automated message it sent to me wasn't from her.
Train an AI on our conversation history instead, and I'd likely click any link it sent me.
"I ordered basketball photos" -> "OMG these basketball photos turned out so cute!"
I'm fairly convinced the long term solution to this will be advocate AI. Basically AI antivirus, a service you pay for that is on your side and defends you and your interests.
And it basically has to be paid for at a fair rate by you, otherwise there will always be an incentive for someone to manipulate it against you.
In other words, protection money.
"Protection money" is a term used when the threat is coming from the same agent that is collecting the payment. In this case they are two separate agents.
And sure they'll protect you from malware, but they'll also have some opinions about who you should vote for, what businesses you should patronize, etc...
Most websites that aren't recently built are getting close to impossible to use due to software bloat, things like forced 2FA, cookie bars, ads everywhere.
Add to that an infinite supply of AI generated content, and you end up with a situation where 99% of content is just bots speaking to each other, and the rest is frankly terrible UX.
Maybe this is how we get our non electronic lives back. We just overdo the machine part.
Ah - perhaps therein lies a solution! We all get a personal AI bot. Mine would sound like me, would be able to imitate my voice and my writing style, and would be able to distinguish bots from humans better than me. It would screen my calls and messages.
"Putting you through!"
Most brands are screaming over each other with bright colors and discount badges and badges promoting this or that benefit that the same product from the brand next to to this one totally doesn’t have. The aisles are completely overwhelming and you can barely even tell what’s in the box anymore.
Then you see the brand aimed at less price conscious consumers. The price is higher, but the box is a plain solid color with a nice font and just says “Rice”. Nothing else. Just rice. Or cereal. Or whatever.
You see similar effects on the internet. The good stuff just doesn’t feel as shouty (look at HN). Calm is a signal of quality. Noise gets ignored.
In the U.S you have brand name with their often noisy labels full of marketing - and then store brand (which is often the exact same stuff as the brand name in different packaging - or stuff that was rejected from quality control of the name brand so is almost the same stuff just slightly worse by some unknown metric). I buy store brand for basically everything but a few food items where the quality is worse enough to be off-putting. Store brand labels aren't quite as plain as No Name labels but are usually a bit more bespoke or just "lower quality" like they didn't want to pay for a graphic designer so had some intern whip up a graphic for the label.
I know the U.K has No Name brand which to my understanding also markets itself as the cheaper no-fuss alternative?
You have simple but cheap looking and you have simple but clean and sophisticated looking. Look at Apple’s branding as a good example of the latter.
You're right, but if it is really good at this sort of thing it's going to be difficult to incentivize anyone with more money than they can get from manipulating you. There's a crapton of money and power in this.
There is also some protection in that I don't care why most of my contacts would send me a link to buy something, it's going to be out of place. An AI can match me or my wife's internal communication style but there's basically no amount of style matching it can do to make me buy cryptocurrency, or send 10 $500 gift cards somewhere. Honestly current spam already takes a decent run at it through plain ol' human evolution anyhow, and phishing/smishing already has a non-trivial hit rate. The space of such messages is already small enough that humans can blindly hit reasonably well.
There appears to be a huge risk of this technology completely devastating the last bastion of trust -- trust in the people we personally know.
I wonder how to mitigate that problem.
eg. prepend the conversation to the message being generated with some instruction to get you to click on a link