You don't use your PIV anywhere because you don't want your creds--certs, etc--to be swiped. You use it for facility access and PKI.
The real reason not to use your PIV for ID in random places is that it's meant to be used as an ID for you acting as your official capacity. This can also be seen in the case where people have multiple PIVs to represent their multiple identities, like National Guard who may have a PIV as a contractor and a PIV as a National Guard -- they would use the correct one depending on what capacity they are acting, or none if it's not part of their official duties.