The solution to that threat, the real threat, is scrypt, bcrypt, or PBKDF2 --- the "adaptive" hashes that can be tuned to trade a marginal increase in defender cost for an untenable increase in attacker cost.
The solution to that threat, the real threat, is scrypt, bcrypt, or PBKDF2 --- the "adaptive" hashes that can be tuned to trade a marginal increase in defender cost for an untenable increase in attacker cost.
For as long as there's people using unsalted MD5 hashes in their PHP applications, Rainbow Tables are a real threat.
> incremental crackers [were used to] harvest thousands of passwords [and] forced Unix systems in the 1990's to adopt "shadowed" password files.
As someone who is interested in security but has not spent significant time studying it, I'd be interested to hear more about this. How did Unix solve the rainbow table issue? What is an incremental cracker and how does it relate to shadowed password files? (I'm familiar with the latter but not the former, and a Google search generates more noise than signal without more keywords to go on.)
Incremental password crackers, like John the Ripper and Crack, take a single password hash, and an actual dictionary, and hash each entry in the dictionary looking for a match. They take days to run instead of seconds, and until people started wanting to break into Windows boxes, they were the only way people cracked passwords.
Incremental crackers have improved substantially over the past few years primarily due to the introduction of GPU programming (in some cases algorithms port easily, in other cases they need some work first to be optimal on a GPU), easier distributed programming and rainbow tables. There's some interesting projects that use GPU technology to optimise the rainbow table reduction function (see http://www.cryptohaze.com/ for an example). As GPU technology improves and as hardware becomes cheaper and more powerful these technologies bring capabilities previously limited to three-lettered agencies into the commercial and home space.