Even in the absence of AI, what's stopping this friend from going to a quack 'natural therapist'? If Steve Jobs can be fooled, what about an ordinary person?

ChatGPT is motivated by sycophancy, humans are motivated by profit. I would say the latter is much more dangerous than the former when it comes to medicine.

Remember, totally legit doctors were peddling opioid pills to patients in the name of "Treating undertreated pain". Profit distorts human minds very easily.

AI doesn't need to be perfect in every situation, just much better than the status quo.