loudmax

You're right. Tinybox's 1.36 petaflops is FP16 so that is a significant difference.

Also, the Tinybox's memory bandwidth is 8064 GB/s, while the Digits seems to be around 512 GB/s, according to speculation on Reddit.

Moreover, Nvidia's announced their RTX 5090s priced at $2k, which could put downward pressure on the price of Tinybox's 4090s. So the Tinybox green or pro models might get cheaper, or they might come out with a 5090-based model.

If you're the kind of person that's ready to spend $40k on a beastly ML workstation, there's still some upside to Tinybox.

Going by the specs, this pretty much blows Tinybox out of the water.

For $40,000, a Tinybox pro is advertised as offering 1.36 petaflops processing and 192 GB VRAM.

For about $6,000 a pair of Nvidia Project Digits offer about a combined 2 petaflops processing and 256 GB VRAM.

The market segment for Tinybox always seemed to be people that were somewhat price-insensitive, but unless Nvidia completely fumbles on execution, I struggle to think of any benefits of a Tinygrad Tinybox over an Nvidia Digits. Maybe if you absolutely, positively, need to run your OS on x86.

I'd love to see if AMD or Intel has a response to these. I'm not holding my breath.

The hate is for Musk. Tesla hate is just collateral damage.

People tolerated Musk's narcissism when he was largely apolitical and his entrepreneurship seemed to benefiting the world at large. Since 2020, Musk's brain seems to have been pretty severely infected by the woke/anti-woke mind-virus, and it's far less clear that his impact on the world is an unmitigated good thing.

YouTuber Ryan McBeth has a pretty good analysis of available evidence: https://www.youtube.com/watch?v=HKiJwuY79Q0

I tend to agree with his conclusion that it's unlikely that the Russians attempted to force the jet to crash in the Caspian Sea to cover up the event. Not because Putin's government has any moral qualms, but because they lack the level of organizational competence it would take to come up with a plan like this and carry it out on short notice.

The two nearest airports in the direction away from Russia's war of aggression are the flight's origin at Baku, and Aktau. Baku is closer, but it's on the other side of a mountain range. Aktau is slightly further, on the other side of the Caspian Sea. A water landing is at least possible. Choosing to fly a damaged aircraft the slightly longer route over a lake rather than the shorter route over a mountain range is an eminently sensible decision.

A point of anecdata: my 2010 Prius has nearly 200,000 miles on it and it's still going strong.

The Prius has a traditional lead acid battery for starting the engine that's been replaced a couple of times. The traction batteries (the ones that help push the car forward and are charged from regenerative braking) are the originals.

If I understand correctly, Hugging Face is exploring approaches to tuning the output quality of a given model by tuning how long to let it run.

Normally when you run an LLM, you set your prompt and whatever tunable parameters, and the LLM software (eg. lamma.cpp) spits out tokens at whatever rate it can. If you want higher quality, you run a bigger model (though you're limited by the amount of memory you have available). If you want higher speed, you run a smaller model. Hugging Face seems to be looking at ways to make this tradeoff without switching between different models.

The American healthcare system is responding to the incentives that are in place, and those incentives are designed (if not intended) to drive up costs.

Doctors can be sued for malpractice if they miss something, so they're incentivized to provide as much care as possible, regardless of cost.

Hospitals and other healthcare organizations are likewise incentivized to charge as much as they can, since the bills are passed on to insurance companies.

Insurance companies have every incentive to lower the bills they pay. They negotiate volume discounts with healthcare corporations, and the billing system that gets worked out between these large organizations has little connection to the actual cost of healthcare provided to consumers.

For most consumers, insurance coverage is connected to employment, so it's very difficult to shop around. It could mean leaving one job for another, just to get different health coverage.

Inefficiencies abound. We can do better, but there is not a simple solution to this.

You could probably eliminate a lot of these inefficiencies by eschewing any sense of morality or fairness and simply letting more people die. Society isn't quite ready for this level of dystopia, but it's the direction we're headed if we don't change course. Nationalizing the US healthcare system may be the best answer, but it is guaranteed to be enormously complicated, and not guaranteed to improve outcomes.

The History of English podcast is worth a listen. It's about the development of the English language, so it covers a lot of history and prehistory, and also linguistics. The presenter Kevin Stroud has a deep passion for the subject matter. Unfortunately, he also has a tendency to repeat himself and over-explain simple examples so the effect can be somewhat soporific.

The Centre Place youtube channel has some pretty good lectures, including some very good lectures on Judaism and early Christianity: https://www.youtube.com/@centre-place/playlists

The videos are presented by a pastor of the Community of Christ church in Toronto, but they're from a historical rather than religious perspective.

This of course depends on your budget and what you expect to do with these models. For a lot of people, the most cost-effective solution is probably to rent a GPU in the cloud.

The limiting factor for running LLMs on consumer grade hardware is generally how much memory your GPU has access to. This is VRAM that's built into the GPU. On non-Apple hardware, the GPU's bandwidth to system RAM is so constrained that you might as well run those operations on the CPU.

The cheapest PC solution is usually second-hand RTX 3090's. These can be had for around $700 and they have 24G of VRAM. An RTX 4090 also has 24G of VRAM, but they're about twice as expensive, so for that price you're probably better off getting two 3090's than a single 4090.

Llama.cpp runs on the CPU and supports GPU offloading, so you can run a model partly on CPU and partly on GPU. Running anything on the CPU will slow down performance considerably, but it does mean that you can reasonably run a model that's slightly bigger than will fit in VRAM.

Quantization works by trimming the least significant digits from the models' parameters, so the model uses less memory at the cost of slight brain damage. A lightly quantized version of QwQ 32B will fit onto a single 3090. A 70B parameter model will need to be quantized down to Q3 or so to run entirely on a 3090. Or you could run a model quantized to Q4 or Q5, but expect only a few tokens per second. We'll need to see how well the quantized versions of this new model behave in practice.

Apple's M1-M4 series chips have unified memory so their GPU has access to the system RAM. If you like using a Mac and you were thinking of getting one anyway, they're not a bad choice. But you'll want to get a Mac with as much RAM as you can and they're not cheap.

I found this part particularly interesting:

> Also, with 512 gigs of RAM and a massive CPU, it can run a 405 billion parameter Large Language Model. It's not fast, but it did run, giving me just under a token per second.

If you're serious about running LLMs and you can afford it, you'll of course want GPUs. But this might be a relatively affordable way to run really huge models like Llama 405B on your own hardware. This could be even more plausible on Ampere's upcoming 512-core CPU, though RAM bandwidth might be more of a bottleneck than CPU cores. Probably a niche use case, but intriguing.

Jeff says as much. The AMD EPYC core does offer better performance per watt, which says more about AMD than it does about Ampere.

But also this: "The big difference is the AmpereOne A192-32X is $5,555, while the EPYC 9965 is almost $15,000!"

So you have to look at TCO over a period of years, but that's still quite significant.

Yeah, agreed. If you think artificial intelligence is going to be an important technology in the coming years, and you want to get a better understanding of how it works, it's useful to be able to run something that you have full control over. Especially since you become very aware what the shortcomings are, and you appreciate the efforts that go into running the big online models.

What you describe is very similar to my own experience first running llama.cpp on my desktop computer. It was slow and inaccurate, but that's beside the point. What impressed me was that I could write a question in English, and it would understand the question, and respond in English with an internally coherent and grammatically correct answer. This is coming from a desktop, not a rack full of servers in some hyperscaler's datacenter. This was like meeting a talking dog! The fact that what it says is unreliable is completely beside the point.

I think you still need to calibrate your expectations for what you can get from consumer grade hardware without a powerful GPU. I wouldn't look to a local LLM as a useful store of factual knowledge about the world. The amount of stuff that it knows is going to be hampered by the smaller size. That doesn't mean it can't be useful, it may be very helpful for specialized domains, like coding.

I hope and expect that over the next several years, hardware that's capable of running more powerful models will become cheaper and more widely available. But for now, the practical applications of local models that don't require a powerful GPU are fairly limited. If you really want to talk to an LLM that has a sophisticated understanding of the world, you're better off using Claude or Gemeni or ChatGPT.

An old laptop can make a great homelab server if you don't need that much processing power. It's quiet, and it's got a built-in KVM and battery backup.

That's nice, but it would be even nicer if Red Hat were contributing to Wine or Proton to get Windows applications running reliably on RHEL.

Really wish people were migrating to Mastodon, or even to Threads, since Threads can federate.

IMHO, the lesson from the corruption of Twitter should be don't build your social network on a proprietary walled garden.

The fuel cost to launch is significant here, as you'd want to put this into quite a high orbit. You absolutely don't want this stuff in low orbit where it will be taken down eventually by atmospheric drag, or by slamming into some other spacecraft.

Another issue is the reliability of your launch vehicle. Sometimes these things blow up on launch, which you really don't want happening when it's loaded with radioactive waste. Developments made by SpaceX might contribute positively to both of these, by lowering the cost and increasing the reliability.

A third issue is that this stuff might turn out to be valuable after all. Hopefully not for making bombs, but future technology might come up with socially positive uses for material that's currently considered dangerous waste.

That is a reasonable question. I think the answer has a lot to do with why the dead are being dug up. If the dead are being exhumed by archeologists seeking to make information about our ancestors available to everyone, that's very different from grave robbers looking for gold.

There are some tasks that I would have used a search engine for in the past, because that was more or less the only option. Say, searching for "ffmpeg transcode syntax" and then spending 15 minutes comparing examples from Stack Overflow with the official documentation to try to make sense of them. Now I can tell Claude exactly what I'm trying to accomplish and it will give me an answer in 30 seconds that's either correct, or close enough for me to quickly make up the difference.

I'm still going to turn to Google to find out what a store's opening hours or phone number is, as well as a lot of other tasks. But there are types of queries that are better suited for an LLM, that previously could only be done in a search engine.

There's also a non-technical reason for LLM search. Google built its business on free search, paid for by advertising, which seemed like a good idea in the early 2000's. A few decades later and we have a better appreciation for the value of the ad-driven business model. Right now, there's a whole lot of money being thrown at online LLMs, so for the most part they're not really doing ads yet. It's refreshing to make a query and not have sponsored results at the top of the list. Obviously, the free online LLM business model isn't going to last indefinitely. In the pretty near future, we'll either need to start paying a usage fee, or parse through advertisements delivered by LLMs as well. But it's nice while it lasts.

How do you even determine whether or not an an AI-generated image is what you desired?

If a human being is making that determination, their time is probably worth more than the 6 cents it cost to generate the image.

I think the 'Pro' part of the name is a good indicator that this is a business-to-business venture, not something aimed at consumers. In that respect, 6 cents per image seems quite reasonable. If you're generating images for a business, paying a human to even visually review image quality at 6c per image is basically sweatshop wage.

If you're generating images for personal use, there are lots of open weight models that are quite capable.

Thanks for this excellent overview of the current available models.

I'd only note that when you say a model is `terrible`, all of this image generation technology is mind-blowing compared to what you could expect to do on consumer grade hardware just a few years ago. We've come a very long way in a very short time.

The trade-off may not be worth it today, but the processing power we can expect in the coming years will make this accessible to ordinary consumers. When your laptop or phone or AR headset has the processing power to run these models, it will make more efficient use of limited bandwidth, even if more bandwidth is available. I don't think available bandwidth will scale at the same rate as processing power, but even if it does, the picture be that much more realistic.

The speed of light through a vacuum is about 300,000 km/s. The speed of light over a fiber optic cable is about 200,000 km/s. The distance between, say Singapore and New York, is about 15,000 km. If we pretend the cable runs all the way to the East Coast, this gets us to 0.075 seconds for light to travel that distance around the Earth over fiber.

So that's a theoretical minimum of 75 milliseconds additional lag in addition whatever network equipment exists in between you and your friends. Actually not too bad, even though we concede that the premise is slightly absurd.

I really dig namechecking Sid Meier for the name of the project. I'm also skeptical that this project actually works as presented, but building a Civilization game off of a Minecraft engine is a deeply interesting idea.

Along similar lines, Steve Buscemi's character Seymour in "Ghost World" (2001) is pretty clearly heavily inspired by Robert Crumb. Seymour collects old blues records and it put off by rock music, not to mention normal mainstream American culture. "Ghost World" was based on a comic book and was directed by Terry Zwigoff, the director of "Crumb".

Pride is also one of the deadly sins.

I tend to agree with this take. Shifting from home cooked meals to processed meals probably has far more impact on diet and health than published guidelines.

The only nuance I'd add is that it doesn't follow that the optimal solution is for women to leave the workforce and get back in the kitchen. Domestic responsibilities can be shared by husbands and wives.

Also, when eating processed foods it's a good idea to pay attention to the nutrition information. Most people don't do much of this, which is why those published guidelines probably don't have much impact anyway. Those guidelines should be updated, and most people should probably start paying more attention to them.

When I run `ssh-keygen`, I can remember the options `-t rsa` or `-t dsa`. I simply cannot remember the flag `-t ed25519`. I have to look it up every time.

I just remember the flag as being vaguely similar the name of the monster robot from RoboCop.