To make use of one different key per host, you can use some ssh_config(5) magic[0]:

    IdentityFile ~/.ssh/keys/%h

See also https://try.popho.be/ssh-keys.html , discussed: https://news.ycombinator.com/item?id=32510475

[0] https://man.openbsd.org/ssh_config.5