Doesn't that prove too much?
Suppose I claim that there is no objective world - i.e. standard solipsism. You may make various claims about the world being coherent to you and existing even in my absence, but these are first-person accounts, just like accounts of qualia and the existence of first-person consciousness are. I can claim that all that I'm observing is something generated by me that contains a part that argues that the world is objective, but that your failure to conclusively prove to me that objective reality exists is a consequence of the concept of an objective world being incoherent.
The claim, "you invent[ed] a thing that doesn't exist", seems to be just as useful to a solipsist to respond to someone arguing for objective reality as it is to a materialist to respond to someone arguing for first-person consciousness. Yet it's considered uncontroversial to assume (without direct proof) that an external consistent world exists.