This is the network working as designed.
I mean, if you wrote an autocomplete system for written english and asked it to complete the sentence "O Romeo, Romeo" what would you expect to happen?
You'd expect it to complete to "O Romeo, Romeo, wherefore art thou Romeo?" - a very famous quote.
How else could you produce the single right output for that unique input, other than memorising and regurgitating?