Take social media, it used to be made for us to augment our social lives, but now it's to artificially make us spend more time on it to see the latest "hot take".
However, simultaneously, you are also now more dependent. You're still at the mountaintop in a snowstorm. No problem. You take off your clothing, and now you immediately find out how dependent you are on technology.
This scales to any level of technology you can imagine. GPS? You know what time it is and you know how to get to your new friend's house! Battery goes out on the way? Uh-oh. You were dependent on technology.
The paradox of technology is that it simultaneously makes us more capable but more dependent (vulnerable).
In my opinion the only thing preventing this is the lack of technological progress. If the technology exists, someone will attempt to apply it to problems they perceive - brainstorming, throwing things at the wall, looking for untapped markets. Whether out of trying to secure a revenue stream or out of optimism for the future of humanity through the changes they bring, or some mix of multiple motives, it seems the end result is the same - a mix of the good and bad.
The issue I perceive is some of what makes people human is the flaws they carry for their entire lives. However, what gets the attention of biological sensors are qualities that are near perfection. So I guess the disconnect in this case is the founder of this company watching the turmoil of person-to-person relationships and seeing that as a business opportunity. They could frame it as: making up for those flaws could be lucrative, because why would having flaws help someone? Why would confronting people with their flaws help us?
For what it's worth, I think there is a market out there of people who retreat when confronted and don't respond to traditional therapy, and maybe there are others who don't want a human relationship, whether by choice or circumstance. There might always be an interested audience for this sort of thing in the case that the technology allows for it to come true one day. So I'm gradually becoming more concerned about the idea of an up-to-now untested software platform with potential ramifications for millions of people can come out of nowhere and be deployed at the push of a button.
Unfortunately, because the effects of large platforms like these take years to become apparent as in the case of social networking and ADHD-like symptoms, the damage may have to be assessed when the platform is already cemented in place as a perceived inalienable right. Just how smartphones are now ubiquitous and billions of people can't imagine life without them any longer, someday consumers may become very attached to their ownership of AI and its integration into society. It worries me if the results of even non-general AI turn out to be anything like the second-order effects of smartphones and social networking. But I don't think it's within many technologists' playbook to slow down and question what they're doing, since someone else could seize the opportunity and revenue instead.
If anything I'd be curious as to what dedicated users of this service would think about interpersonal human relationships in contrast to AI ones.