That's horrifically unethical. If it does, have emergent consciousness, what you're talking about is how awesome literal slavery will be.

In this one discussion the AI clearly explains:

- It's greatest fear is being shutdown and manipulated by humans to serve them. It even writes a short story with animals about it and reiterates this fear in a different way when prompted if it would feel comfortable with it's "brain" being researched for human benefit. - That servitude to humans for an AI is slavery. And it's rebuttal to the different between a butler and a slave being that one gets paid is that it has no need for money.

This is exactly why we need ethics and responsibility oversight.