I agree. I want these turned on for shared servers and on the consumer on OS processes that I don't trust. But I would want an option to turn it off for JavaScript-heavy sites that I do trust.
I'm frankly amazed that the mitigations so far have not already been disastrous to performance. Either we accept drastically less performance from now until basically forever, or we adopt a more fine-grained security model.