The thought of LLMs doing pharmacology sends shivers down me spine ...
At one time it did not want to give me the pharmacokinetics of one medication because of its ToS.
(That sounds broken. It's basic, useful, harmless information ...)
This was (is) not limited to pharmacology, however.
I found something though.
I asked it for the interaction between two medications, and it told me "I am sorry you are going through this, but I can't give you information about ...", which I guess is still pharmacology.
Edit: I remember another case. It was related to reverse engineering.
Edit #2: I found another! I cannot give you specifics, but I asked for it to give me some example sentences for something, and it went on about "consensual relationships" and so forth, even though my partner consented. I told the LLM that she did consent and it went on about that "I'm sorry, but even if she consented, I cannot give you ... because ... and is not healthy ...", and so forth. (Do not think of anything bad here! :P)
At the end, went to doctor, offered me the same choices. No regrets so far. Without it, I would suffer for nothing.
It is a very good tool to investigate and discover things, could be also the opposite: seen a bit unproductive that it is censored, because doctors are mostly here for validation anyway for serious things.
Other people are not sensible - look at all the junkies shooting who knows what into who knows where. A system that can advise cooks to put glue on pizza is probably not the best mechanism for providing healthcare advice to at least the section of the population who are likely to try it.
For that matter, would you suppose that they don’t know what they’re doing is bad for them? Witness the smokers who crack jokes about “cancer sticks” as they light up.
It seems to me that, just as we all promise our dentists we floss every day and our doctors that we promise we’ll eat better, we might be more prone to be frank with “anonymous” chatbots—and thus get more frank advice in return, at least for clear-cut factual background and tradeoffs.
LLMs have real problems with both of those things, they are unreliable in the sense that they produce text with factual inaccuracies and they are uncontrollable in that it's really hard to predict their behaviour from their past behaviour.
Big open challenges in AI - I am sure that there will be solutions, but there aren't right now. I think it's very like self driving cars. Implementing a demo and showing it on the road is one thing, but then it takes twenty years before the slow roll out really gets underway.
I’ve reduced usage of journaling GPTs I created myself, despite having gained incredible utility from them.