I think a similar problem needs to be solved in the language generation neural networks. While generated sentences often are formed syntactically correct, they don't make a lot of sense -- there's no over arching thought that it is trying to express. This is even more apparent when you read the subsequent sentences generated. It's clear there's no conceptual goal it's trying to convey.