My company Foxpass (YC S15) has a product to manage SSH keys. It serves as access control too -- the keys are only available on the hosts where a user should have access.
Aren has been awesome with responding to emails and helping us set it up too.
This means you can set up a linux group with sudo capabilities (sudo or wheel, usually) in /etc/sudoers. Then using Foxpass you can manage the membership of that group by adding users on a permanent or temporary basis.