Agree. There are many extensions that I want to use on arbitrary websites, but rarely. This could be handled by the browser locking the extension out from everything, until the moment I manually invoke it, at which point it’s allowed access to the page I’m currently looking at.
Now, maybe that’s how it already works, but I have no confidence in it.