Over time, yes. Your statement about physical access == compromise is missing that crucial detail. There's no reason why someone should be able to access all your data just because they have physical access to your device for a short period.

If you really want to do the analogy thing, the DMA vulnerability would be the equivalent of a safe with a door where no key is needed in the back. It would not be a very good safe.

Just sayin'.