If you're not throttling the number of incorrect password guesses per IP, or per IP per account, or both, you're opening your users up to a dictionary attacks anyway. If I can get a box close your yours (~5ms is doable if we use the same colo), then it's completely feasible.

You're right however that this is going to be painful, because most web frameworks don't have any of this throttling architecture (like persistent in-memory hashtables) in place, and as soon as they switch to Argon2/Scrypt, it's going to be an easy DoS vector... particularly for services running on weak VPSs.

It's another reason why solid, secure password authentication protocols, that do client-side hashing, will eventually happen. Even in fancy zero-knowledge asymmetric protocols however, online rate-limiting is essential.