I did this with used parts and cheaper consumer cards (3090s) and did much of the same calculations. I found it was way cheaper for me as well.

The main advantage, however, is that the friction of "this is going to cost me in tokens to even try" goes away. I was so much more willing to take chances and try new things on my own hardware than I would have been if I were paying API costs. I feel like this point isn't made clearly enough by those of us who run these absurd self-hosted inference systems.

Thanks for the write up, was a fun read. I spent an order of magnitude less, but I could relate to your story from beginning to end.

Epyc (Milan), 512gb ram, 4x 3090

You kind of bury the lede in that Article, it's a good article, well done getting interest in your work.

Will you now be selling these GPUs for a profit?