Lack of update is a show stopper. In general I would expect Immutable JS to trounce anything via Object.freeze with respect to performance. Real hash maps and sets are useful to organize programs - the semantics of Object.freeze on ES6 Map and Set are unclear and still suffer from above problems. Again Immutable JS has a big advantage.

In the context of building real applications with immutable data structures and a library like React that makes them easy to integrate - many of your concerns don't apply at all.