Be interesting to see the efficiency metrics on this compared to high aspect ratio planes.
2,289 karma · joined April 8, 2019
Gemma4:31b, which is the better model with more complex architecture (and thus can't be compiled into a megakernel), is gonna be way slower in prefill, and ~50% slower than a 3090 even with oMLX. You can't get away from much slower shared ram vs vram, and less parallelism compared to 3090.
The other advantage of an eGPU setup, you can simply add/swap performance. Dual 4070s give you 24b of ram and you can still get about 80% of the performance of a single 3090, and actually you get faster prefill due to more parallelism.
So in terms of money/usefulness, Mac always loses.
It was done plenty. Just with extensions to python like numpy and opencv. There is no reason to translate everything versus just the critical pieces.
Try more like 70-130 tok/sec. https://github.com/L-Forster/open-jet/tree/master/megakernel.
And also Qwen models famously chase benchmarks. They are good for writing code, but fall behind in other areas. Gemma4:31B is 6gb larger and does a much better job across all areas. https://www.reddit.com/r/LocalLLaMA/comments/1vyzopv/gemma4_...
Go ahead and run Gemma4 on your M1, and you will find that its unusable at 12 tok/sec.
And this is also not touching prefill, which is the biggest thing that you need fast, and Apple sucks in this regard compared to nvidia cards. With gfx cards, you can throw big codebases at the local models and it will happily crunch through them in very little time.
Sure, spending 1.5k on a Macbook enables you to run a local model, and if you really have a good reason for running local (like running heretic/abliterated models) for tasks you don't mind waiting for, it works. The difference is, spend 1k more and let you never have to pay for tokens, and you can run inference all day long without your computer thermal throttling, like your Macbook.
And I don't need to make assumptions when you say idiotic stuff like "We're getting close to local AI being really useful", like you have any idea what you are talking about. On par with being an Apple fan though.
Mini PC with Occulink, EGPU dock, 750w power supply and a refurbished 3090 is 2500. It can run Gemma4:31b at 1000+ tok/sec prefill, ~40 tok/sec generation (which is slow, but a fully connected model is way better for all tasks)
if you want more specialized partial activation models like Qwen, you can get 100+ tok/sec generation.
This beats anything that Apple has by a long shot. You don't need to run large models for usefulness. Large models can get things write the first time, but thats their only advantage. Running small models that are good enough like gemma4:31b fast allows you to do faster iterations.
And if you really want, you can also do multi GPU setups if you want slightly slower model loading with usb4/thunderbolt ports.
TLDR; local ai has already been useful for quite some time if you arent brainwashed into believing that Macs are the best for running local inference.
Dunno how much investment is needed into language optimization anymore.
https://news.ycombinator.com/threads?id=ActorNightly&next=49...
TLDR its not so much about the transformer, as much as expanding dimentionality. You can chose any which way to do so.
Hate to burst your bubble, but behind Trump there are quite a few people that are ready to take his place, with no real chance of voting ever being able to stop them. Don't get your hopes up that voting blue matters. Everyone had the chance in 2024, half the eligible population did not vote. Now we get to live with those consequences.
US is done for.
There are 3 big factors in play
* 2020 attempted coup gave the Republicans valuable information on what to do this time around. Basically, the game is to put US into legal limbo and remain in power. They know nobody on the democrat side is going to think about using armed forces, and they are not going to use armed forces to keep their public image and keep economy from panicking, so they are going to just declare some emergency and keep power under the pretense of fixing that emergency. Reminder that Supreme Court is on his side.
* Trump has shown that he has no issue with having a private army operating with extrajudicial authority in public. Now imagine with what he is ok doing with a non-public army. You should fully expect people going against him to get "arrested" under some made up charges.
* Even with Trump gone, there are smarter, younger, non senile conservatives ready to take his place and grab power and do the same things, with way less mistakes. Vance is right up there with him.
As such, even if everyone gets out and votes blue, in the end its all pointless. Trump can get impeached, convicted, and either Supreme Court says nah, or he just won't leave. In 2028, he (or Vance) will declare election fraud again and do it successfully this time to remain in power.
And even if somehow Dems win the house and presidency, and manage to pack supreme court to take control of all the branches, the thing is the will be left with a insanely shitty situation to clean up that will result in economic downturn, and the idiots in US will vote republican again in 2032.
If you have ADHD, you can have a lot of secondary effects from your inability to motivate you to go do things.
ADHD medication, which is basically meth, basically prevents the reuptake dopamine, meaning that it says bound to the neurons. You fundamentally get the tool to go and actually do things. But a tool is not the solution without knowing how to use it, for that you need additional support, which is most commonly is therapy.
The local inference threshold has been crossed already. Its possible to achieve big model like performance through agent harnesses with local models.
This isn't suprising. After all, you can use 2 neurons to model a NAND gate, and you can build up a CPU from NAND gates only.
You don't even need transformers to model a CPU using neural nets. The only problem becomes training them, which becomes very tricky from end to end, especially from shell command to output because the gradients are often going to reverse quite a bit due to binary nature, and the recurrence of memory outputs back into the input (we still don't know how to accurately train RNNS)
With LLMS, uou have to do something like have the LLM forget the KV caches after every invocation and instead rely on injected context of a state, which it then also outputs at the end of the terminal command, and then train it on that. That way given enough parameters which translates to resolution, and some command preprocessing (like splitting a complex chained shell commands in a single line), you can guarantee that for every possible input you get accurate output and state change. So instead of an efficient RNN, you basically end getting a much wider lookup table of sorts.
In today, with internet anonymity and prevalence of entertainment, you have shit like Fox news, which is not a news organization but an entertainment organization, solely for the purpose of being able to tell lies and not get sued because its just "fiction".
2. If you are defining MAGA as Trump supporters sure, but the ones that identify as conservatives that don't support Trump enlarge still vote for Trump or anyone republican who supports/empowers trump, and they would love to see someone who is more competent and charismatic doing like getting rid of non white people in US just through less questionable means. Conservativism in itself a ideology stemming from insecurity. And the few things that conservatives get right (like gun control approach, economic policies) are all equally suppored by the moderate Biden/Kamala type democrats.
3. All you have to do is make it more expensive for them to follow their ideology than have a normal life where they can enjoy their vices. Nobody cares THAT much more about starting a civil war en large. You are not trying to erase them from existence, you pretty much just trying to lessen their outreach.
This is what most people don't understand about Trump today. Everyone is so quick to think that the administration is the source of the problems. Even if the pendulum swings the other way, the people that voted for Trump will still exist and carry those ideals into the future.
Once you have reviews and other people say its plausible, you can go present to VCs and acquire the funding.
The main problem is that the exploits from js to code execution or data exfil are possible in the first place. The issue is that you have JS, which is generally pretty isolated, able to talk to bundled libraries (like libwebrtc), which often contain the exploits and share the same memory space. There needs to be a intermediate layer between JS talking to any such library, like process isolation with IPC for communication.
Recession is coming no matter who is in charge. If Dems manage to win, and they will do the same undoing of damage, you are going to have 2024 again where Trump era problems are blamed on Dems, and then you will get another republican who is going to use the Trump era as a guideline on what to do, except he will do a better job.
Its almost better if Republicans steal the election tbh, because more people need to feel the hurt enough to start taking action IRL in fixing things. When people who are MAGA start getting fired from jobs for being MAGA, then we can start moving forward.
The only solution that doesn't involve a decline is for someone like Newsom to come in and be as bombastic as Trump and actually take authoritative action to suppress conservative media, and make big spectacle of all the positive things he does even if they are minor. It seems that people just want to belong to an ideology around a strong leader these days.
In the same way, you can also go look at the public transcript of the Trump Jan 6 court hearing, where Trump did not deny anything he did, like putting the election counts into question, organizing a rally with very energizing language, sitting and watching people storm the Capitol hoping that the certification will be delayed with the chaos, while others are begging him to go on record and call it off - he admitted to everything, with his only claim being that as the sitting President, its with his right to do this in official capacity with immunity. I.e the worst possible example of authoritarian rule that you conservatives are so seemingly scared of, but you still vote for him because apparently all of that is justified as long as there is less brown people in US.
The fact that you haven't done so and instead are automatically are going on defense saying my claims are invalidated pretty much proves everything I said about you.
Theis issue with conservatives isn't the policy you believe in, the issue is that you are just so unbelievably ignorant of how the world works, to where you see an issue like migrants coming in, and think that its such a big problem that the best course of action is giving a flamethrower to a convicted felon rapist and likely a pedo (all the things you supposedly firmly stand against), and trusting him to burn just the right parts down with authoritative force (something you are also supposedly firmly against)
Furthermore, if you bothered to actually care, you would have paid attention, Dems completely would agree with you that undocumented migrants were a problem - Biden had a border bill that was bipartisan, supported by border patrol, that would have allocated all the correct funding to reign in the asylum process, which would have correctly filtered the people that were coming here to work, versus those taking advantage of the system. And consequently, Trump told the Republicans to vote against that bill because it would hurt his chances of getting elected.
So for people like you, it was never about the migrants - you don't actually care. It was always about racism and sexism. At least there isn't any questions about it anymore.
Right now, all thats being done are LLMS being RL trained on specific tasks, which are usually drawn from human curated data sources.
This is why you can ask an LLM questions and it can write code, but you can't just leave it running with a complex task that would involve research and expect it to solve the problem.
Its obvious that part of AGI will be the mapping of language to latent space, and then transformations on either responses or actions. But the issue is that until the model can self learn to become better at coding without going through manual RL tasks designed by humans, you will never get AGI.
And if a model can do that, that means that its possible to have basically the "genesis" code - i.e something that can be run on a computer, with no training on english language, but aware of existance of things it can do, can basically figure out how to train itself to the standard of a current LLM and beyond.
And in figuring that part out, we don't even know where to begin. It very well may be a case of running LLMS in a genetic algorithm and simulating evolution to figure out what LLM+Agent harness works the best.