Sounds very cool! How do you implement efficient random deletes?
Efficient random deletes and contiguous hole-free storage are completely at odds with each other.
If you don't care about the map part, you can get the same behavior by just moving the last element in the place of the newly removed element. This invalidates all indices, which is what the DenseMap's overhead is meant to avoid, but Vec's remove also invalidates indices. Vec's remove is strictly worse except that it preserves the ordering if the Vec is sorted.
You could add that functionality via 'tombstones': when you delete an element, you replace it with a 'tombstone' marker in the structure.
Whenever you clean up the structure (eg for a resize), you skip the tombstones when you copy the old contents over.