The kernel has to interpose at least a little bit because some of these hardware devices can read and write anywhere, so you can't just let random users sent them commands.
Sorry. I hate it too.
It is analogous to passing a pointer to a buffer to a library function so that the library can write the contents of the buffer directly. You rely on the library (device) operating properly and not writing outside of the designated buffer. That is the baseline.
As you state, you could add a enforcement mechanism that defines the extents of memory the external actor is allowed to store like how many language runtimes check and disallow out-of-bounds accesses. However, if you have multiple outstanding buffers, control structures, or other complex device-accessible structures then enforcing precise "bounds" checking rapidly demands very complex "bounds" definition/enforcement. Language runtimes can do this because they support arbitrary code, but enforcing in hardware that, say, every access lies within the nodes of a "user-constructed" red-black tree rapidly becomes infeasible.
You basically get one of two options at that point, either you rely on the hardware working properly and do nothing, or you design your driver to only require coarser isolation that fits within hardware-definable boundaries. Most opt for the former. If you do the latter then there are various ways of actually enforcing the isolation such as IOMMUs or you could have a DMA controller that basically functions as a IO MPU (define ranges the device can access) (I am not actually aware of any DMA controllers that actually do this as a security measure, but it is theoretically possible).
You do have to be careful that they actually enforce the isolation. For instance, I question if the x86-64 IOMMU implementation is actually safe against a malicious device due to certain supported features such as device IO TLBs, but I do not know about the actual hardware implementation to know for certain.