The IOMMU hardware ensures security of the VM's raw device access while the VM is running. The only significant security risk I'm aware of is if the VM has a chance to update some firmware (like an option ROM) on the device you give it access to, which may enable it to do evil things when the host system next reboots.