For anything bigger than that, you'll quickly run into issues. Try talking to your ops team and telling them you want to set up a mid-range desktop PC with extra RAM and a high end graphics card as a DL workstation. I think you'll very quickly find some friction, especially once you want to go into production with it.
IIRC their original hardware plan was based around commodity hardware and getting the most performance per dollar since they built the crawler to be distributed pretty early on.
Point being, consumer grade hardware has come a long way since then, and if you're doing something cutting edge like DL it's not outlandish to expect that rolling your own might be worthwhile and totally justified.
Google's start up between 1996 and 1998 was also over twenty years ago. There were fewer than 200 million Internet users on the Internet and less than 2.5 million web sites [0] in 1998. Google was also started by two college grad students, meaning gasp it was initially just a research project. I'd also point out that while, yes, the initial production servers were cheap and used commodity hardware, this [1] is what they looked like, which is hardly the type of setup that the article is suggesting.
[0]: http://www.internetlivestats.com/total-number-of-websites/
[1]: https://en.wikipedia.org/wiki/History_of_Google#/media/File:...
I do recall a time as recently as 4 years ago where it was difficult to find, for example, rack-mountable servers that could hold a large number of these GPU cards, but substituting for a mid-range PC with 2 or 4 of them would not have been an issue, even then.
Cost would be higher but not necessarily outrageously so, since large server chasses often have startlingly large PSUs. It might be an extra 50% on top of that $3k, but that's still competitive, even with the pre-builds the article mentions.