i am somewhat scared to use smart cards because they might break at some point and i am loosing access... if i would use some kind of local ca to provision the keys i could just as well use them directly... so i would need some kind of secured (air gaped and what not) device to provision them. which will be unavailable when i am abroad... all in all it seems to me it doesn't solve more problems than it makes...
The CA or smart card software is only needed for provisioning smart cards for clients, and is likely to be of little concern for setting up a new server. The card itself stores a non-phishable credential which you can physically secure.