The name of the city as a whole is simply London. You could even call it the city of London if you were being poetic. But as soon as you capitalise it, you're talking about the City of London, and that's something different.
Yes, that is confusing, arbitrary, irritating, and only justified by arcane details of history which predate the printing press by five hundred years. That's because this is England we're talking about.