In a nutshell: "In most cases you won't need to do this often" is a HUGE fallacy. It depends on the security rules you work/live by. Plus, it would make it really annoying to use if on top you're using TOR.
Yes, passwords need to be fixed. They are weak, problematic and a security cheddar cheese. It is why we are now implementing two factor authentication. Changing the "fixed password" strategy to a "random and time limited password" strategy isn't exactly solving more issues than it raises. Again, from a security-wise stand point.
May be if this was implemented with something different than your email. Like, for example, a bank tokens or cell phone verifications... which, again, are part of a two factor authentication because by themselves they would be too easy to break.
Think about the following scenario: You use X site with this email auth system and, for example, Thunderbird. Stand up and go to the bathroom or a meeting or whatever without locking your computer. Presto! I won't even need to guess a password and get access. Of course getting access to X site would be the least of your worries in that example, but it illustrates the point I'm trying to make.