Would a company be liable to uphold its promises if a rogue human customer service agent promised something ridiculous such as 1 million dollars worth of free flights?
We live in an interesting world. In the US, a corporation is legally a person, and a chatbot is not a person[0]. I'm looking forward to the first Supreme Court case involving a corporation consisting of chatbots.
[0] I'm handwaving in this lead-in to the fantasy here, so, dear reader, please give me a break for oversimplifying and ignoring technicalities.
Or in this case they need to take the AI out of service immediately until they can get a corrected version that does not do such a thing. I will accept that the AI can be tricked to do such a thing and remain in service, but only if they can show the tricks are something an honest human wouldn't attempt. (I don't know what this is, but I'll allow the idea for someone else to propose in enough detail that we can debate if a honest people would ever do that)