Is the first "Using a Configuration File" example a bit misleading? The example is:
... Host remoteserver HostName remoteserver.thematrix.io User neo Port 2112 IdentityFile /home/test/.ssh/remoteserver.pub
IdentityFile typically (always?) specifies a file with a private key, right?