HNHacker News
TopNewBestAskShowJobs

elorant

35,994 karma · joined January 18, 2011

.Net developer

cbarak@gmail.com

submissionscomments
elorant··on Run DeepSeek R1 Dynamic 1.58-bit
Not everyone needs the largest model. There are variations or R1 with fewer parameters that can easily run on consumer hardware. With 80% size reduction you could run 70B on 8-bit on an RTX 3090.

Other than that, if you really need the big one you can get six 3090s and you're good to go. It's not cheap, but you're running a ChatGPT equivalent model from your basement. A year ago this was a wetdream for most enthusiasts.

elorant··on Nvidia, ASML plunge as DeepSeek triggers tech stock selloff
No, I haven't. I've used Perplexity's R1 but I don't know how many parameters it has. It's quite good, although too slow.
elorant··on Nvidia’s $589B DeepSeek rout
There are multiple variations of the model starting from 1.5B parameters.
elorant··on Qwen2.5-1M: Deploy your own Qwen with context length up to 1M tokens
You need a model that has specifically been extended for larger context windows. For Llama-3 there's Llama3-gradient with up to 1M tokens. You can find it at ollama.com
elorant··on Nvidia's Project Digits is a 'personal AI supercomputer'
You're missing the most critical part though. Memory bandwidth. It hasn't been announced yet for Digits and it probably won't be comparable to that of dedicated GPUs.
elorant··on Nvidia's Project Digits is a 'personal AI supercomputer'
Well obviously it has to be low otherwise they would cannibalize their high end GPUs.
elorant··on Nvidia announces next-gen RTX 5090 and RTX 5080 GPUs
You don't need to run them in x16 mode though. For inference even half that is good enough.
elorant··on Ending our third party fact-checking program and moving to Community Notes model
My gut feeling is that this will be accompanied by a relaxed policy on fake profiles too.
elorant··on Perplexity got ads
You're oversimplifying the narrative. It's not ads per se that bother. It's the tracking part. Make ads contextual like they used to be and there's no problem at all, actually they might turn to be interesting. Keep tracking me all over the web and you grow the desire in me to block the living shit out of everything.
elorant··on A path to O1 open source
How is China training models without access to cutting edge GPUs?
elorant··on The GPU, not the TPM, is the root of hardware DRM
Whatever their motivation is, disabling TPM from Bios is the safest way to avoid upgrading to Win11.
elorant··on Meta Wants More AI Bots on Facebook and Instagram
Well that makes sense. If you bother to look at how much engagement all those fake female accounts are generating it's no wonder that Facebook wants to own that game too.
elorant··on I Run LLMs Locally
Not always, because system RAM also has to be equally adequate, but mostly yes, it's about the total VRAM of the GPU(s).
elorant··on How I run LLMs locally
I consider the RTX 4060 Ti as the best entry level GPU for running small models. It has 16GBs of RAM which gives you plenty of space for running large context windows and Tensor Cores which are crucial for inference. For larger models probably multiple RTX 3090s since you can buy them on the cheap on the second hand market.

I don’t have experience with AMD cards so I can’t vouch for them.

elorant··on The Average American Spent 2.5 Months on Their Phone in 2024
It's not just discipline. Kindles are more convenient too for reading. No refresh rate so you can spend hours reading without eye strain.
elorant··on The Average American Spent 2.5 Months on Their Phone in 2024
Get a kindle and start reading books. That’s what's keeping me from staring at a phone all the time. I might be spending similar amounts of time looking at a screen but at least I get something valuable out of it.
elorant··on Xerox to acquire Lexmark
Laser printers print a status page that describes most of their settings, how many toners you've changed and also includes the total page count.
elorant··on Xerox to acquire Lexmark
Bought a Lexmark E232 laser printer back in 2006. 18 years and 110k pages later the damn thing still works flawlessly. I have nothing but admiration for their printers.
elorant··on The NBA's Problem Is Economics, Not Basketball
Euroleague seems way more interesting these days
elorant··on New LLM optimization technique slashes memory costs
For the general public, not the HN audience.
elorant··on Cold Email Handbook
Cold emails are based on the premise that you have a large targeted list of B2B emails. You've scraped for example a whole category of from Facebook or LinkedIn or whatever. Then you cold email them. You aim for a conversion between 1% and 3%, and if you send thousands of emails it soon becomes very lucrative.
elorant··on Sora is here
MidJourney (commercial), Standard Diffusion XL
elorant··on Beekeepers halt honey awards over fraud in global supply chain
I regularly buy honey from the supermarket from a long established Greek brand and it’s as good as any honey I get from local markets, if not better.
elorant··on Romanian court annuls result of presidential election first round
So when Russia interferes with our politics it’s cold war, but when CIA does it’s just another day?
elorant··on Llama-3.3-70B-Instruct
As a general rule of thumb, for 8-bit quantization you’ll need as much VRAM as the parameters of the model. Half that if you run it at 4-bit. System RAM should be a bit more because you need RAM for the OS too. What most users do is go for second hand 3090s. You get two or three of them depending on quantization.
elorant··on ChatGPT Pro
Everyone rants about the price, but if you’re handling large numbers of documents for classification or translations $200/month for unlimited use seems like a bargain.
elorant··on Intel announces Arc B-series "Battlemage" discrete graphics with Linux support
Let’s say for the sake of argument that you could build such a card and sell it for less than $5k. Why would you do it? You know there’s huge demand in the tens of billions per quarter for high end cards. Why undercut so heavily that market? To overthrow NVidia? So you’ll end up with a profit margin way low and then your shareholders will eat you alive.
elorant··on Intel announces Arc B-series "Battlemage" discrete graphics with Linux support
AMD has a 192GB GPU. I don’t see them eating NVidia’s lunch with it.
elorant··on Intel announces Arc B-series "Battlemage" discrete graphics with Linux support
Would it though? How many people are running inference at home? Outside of enthusiasts I don't know anyone. Even companies don't self-host models and prefer to use APIs. Not that I wouldn't like a consumer GPU with tons of VRAM, but I think that the market for it is quite small for companies to invest building it. If you bother to look at Steam's hardware stats you'll notice that only a small percentage is using high-end cards.
elorant··on Alibaba releases an 'open' challenger to OpenAI's O1 reasoning model
Care to say how many tokens per second you're getting?
← PreviousPage 7 of 34Next →