Not everyone needs the largest model. There are variations or R1 with fewer parameters that can easily run on consumer hardware. With 80% size reduction you could run 70B on 8-bit on an RTX 3090.
Other than that, if you really need the big one you can get six 3090s and you're good to go. It's not cheap, but you're running a ChatGPT equivalent model from your basement. A year ago this was a wetdream for most enthusiasts.
You need a model that has specifically been extended for larger context windows. For Llama-3 there's Llama3-gradient with up to 1M tokens. You can find it at ollama.com
You're missing the most critical part though. Memory bandwidth. It hasn't been announced yet for Digits and it probably won't be comparable to that of dedicated GPUs.
You're oversimplifying the narrative. It's not ads per se that bother. It's the tracking part. Make ads contextual like they used to be and there's no problem at all, actually they might turn to be interesting. Keep tracking me all over the web and you grow the desire in me to block the living shit out of everything.
Well that makes sense. If you bother to look at how much engagement all those fake female accounts are generating it's no wonder that Facebook wants to own that game too.
I consider the RTX 4060 Ti as the best entry level GPU for running small models. It has 16GBs of RAM which gives you plenty of space for running large context windows and Tensor Cores which are crucial for inference. For larger models probably multiple RTX 3090s since you can buy them on the cheap on the second hand market.
I don’t have experience with AMD cards so I can’t vouch for them.
Get a kindle and start reading books. That’s what's keeping me from staring at a phone all the time. I might be spending similar amounts of time looking at a screen but at least I get something valuable out of it.
Bought a Lexmark E232 laser printer back in 2006. 18 years and 110k pages later the damn thing still works flawlessly. I have nothing but admiration for their printers.
Cold emails are based on the premise that you have a large targeted list of B2B emails. You've scraped for example a whole category of from Facebook or LinkedIn or whatever. Then you cold email them. You aim for a conversion between 1% and 3%, and if you send thousands of emails it soon becomes very lucrative.
As a general rule of thumb, for 8-bit quantization you’ll need as much VRAM as the parameters of the model. Half that if you run it at 4-bit. System RAM should be a bit more because you need RAM for the OS too. What most users do is go for second hand 3090s. You get two or three of them depending on quantization.
Everyone rants about the price, but if you’re handling large numbers of documents for classification or translations $200/month for unlimited use seems like a bargain.
Let’s say for the sake of argument that you could build such a card and sell it for less than $5k. Why would you do it? You know there’s huge demand in the tens of billions per quarter for high end cards. Why undercut so heavily that market? To overthrow NVidia? So you’ll end up with a profit margin way low and then your shareholders will eat you alive.
Would it though? How many people are running inference at home? Outside of enthusiasts I don't know anyone. Even companies don't self-host models and prefer to use APIs. Not that I wouldn't like a consumer GPU with tons of VRAM, but I think that the market for it is quite small for companies to invest building it. If you bother to look at Steam's hardware stats you'll notice that only a small percentage is using high-end cards.