kristopolous

This is news? Didn't we all just presume that was what was happening? I mean they're a data mining company.

well no of course not ... but they do this for qwen 3.8 27b, and decision models like this https://artificialanalysis.ai/articles/artificial-analysis-d...

this one is especially annoying ... i can run these on lots of hardware but there's vast differences in many aspects.

Whether I'm getting a good deal isn't what I'm thinking about if I'm getting garbage results. It's more about why am I wasting my time on the wrong tool.

I mean what I really want is a more reliable classic eval benchmark for the decision models that I can easily deploy, like the official ollama ones: https://ollama.com/library?sort=newest

perplexity has a 27b decision then there's a 3b from liquidai and another 600M one. Then there's one called kev and bespoke has nimble-v3. clef from cloudflare is there along with strands-decider from aws. Then there's laya and tev1 ... I mean the AA page seems to be the only real third party doing benchmark and evals on it but most of these are missing from it. It has a lot to be desired.

And you go over to HF and it's even more mysterious https://huggingface.co/models?pipeline_tag=text-classificati...

Maybe an agentic eval framework that tests anything that hits 100 downloads would be a decent weekend project...

Also there's like wildly different tasks. choices versus noul have way different performances and also if you are doing things like text classification such as using it for theme extraction, I mean this does work with nimble-v3 but d1 gives mostly noise and pplx-decider is in between and their performance degradation curves are dramatically different depending on the input-token length.

This is all going to be bitter lessoned eventually but I can't just sit on my hands and wait around 12 months.

So now I have a proxy that looks at the input size and do dynamic routing and sometimes fire off both and do ensemble weightings ... I mean why do I have to be clever here, this is a generic problem.

I had to discover all this myself and I really shouldn't have to do that...I guess I'm just expecting some quant results from AA that they aren't in the business of providing.

AA prices are always kind of hand wavy for open models. I mean if you rent a cheap gpu and run inference yourself...or heck, what about a cpu instance with a bunch of ram, you can do that with llama.cpp for under a $1/hour. Ok not that, so are we talking about the median spot price per token from a basket of inference providers? Which ones? They actually vary quite a bit, even on openrouter, for this particular deepseek checkpoint it's between $0.15 and $1.20 ... that's like 8x spread...

They have some math here https://artificialanalysis.ai/methodology but ok how are the prices weighted among the providers... and even if you do that 7:2:1 math it doesn't match the providers. Specifically the model in question: https://artificialanalysis.ai/models/deepseek-v4-1-flash-101... doesn't match the first listed provider https://deepinfra.com/deepseek-ai/DeepSeek-V4.1-Flash-1017

Do the math yourself, it's actually $0.08 not the AA claimed $0.10 (and that's the more expensive of the two pricing tiers). That's a pretty large error bar.

Also many of the providers are now doing "peak" and "offpeak" pricing (deepseek.com itself for instance) so it can differ dramatically. These "peak" times depend on where the service is most popular. You can basically ride "off-peak" pricing by cascading from global provider to global provider.

The majority of the time taken in an inference call is doing the inference so the extra latency of traveling across an ocean is actually a fairly small wall-clock tax here. People still seem to be partial to nearby servers for ai-related workloads but really, try some decent ones 10,000 km away, you probably won't notice.

I know AA thinks highly of the price metric (I've been to their events, even have their hat, they brag about it) but I have learned to ignore it as noise.

It's an especially strange metric with open-weights decision models. I mean I can run those on my laptop ... cost is very abstract there.

But for closed-source models with only a handful of providers (sometimes just 1) great, the price is a very reasonable number.

For the open models a more useful thing would be like "100 tok/s spot price build" and then it's using pcpartpicker or something in some kind of partnership and can query a build for buying a machine right now that can achieve a certain benchmark number and maybe something for less interactive workloads like "5 tok/s". Of course getting some base unit of power cost in there might be important as well since the power bill is the majority of what you pay for when you rent a commodity gpu.

I thought they meant bus width ...still impressed for entirely unexpected reasons.

just go rent one for a few minutes and try it out ... it'll cost you well under $0.50 if you're just trying to get a feel for if it's right for you. There's a lot of things like context window that this doesn't capture.

Depending on who you sign up for, that's within the introductory credits (usually like 5-10$) so it's actually $0.

Being on the side of human rights watch, amnesty international, doctors without borders, the red cross, oxfam, and the un about the classification of a war strategy isn't antisemitism

People with Nazi tattoos who make songs called heil hitler and chanting "Jews will not replace us" on the far right passing around protocols of the elders of Zion calling human rights watch "the actual antisemites" doesn't make it so.

Words have meaning and lying criminal hate mongering scam artists who are viscerally and proudly antisemitic slandering human rights organizations... I mean you take scam artists and conmen and all of a sudden there's this shroud labeled political identity that they tie around their hips and now their crimes qualify them to scam the civic government?

No.

Look what the Europeans did from their antisemitism in the 1940s. We need to criminalize hate politics like we've banned other antisocial behaviors such as assault and rape. This just feels so obvious to me. Political hatemongering isn't some mystifying slippery ghost.

But no we're like "oh so we have one candidate that wants to adjust the budget to fund some libraries and this other one that wants to build concentration camps and imprison transgendered people. Alright, let's allow both on the ballot"

Being a violent hateful ignorant bigot isn't a qualifying condition for administering civic life.

Has that caught on? I last used it 15 years ago and it was great but I haven't seen it anywhere

reminds me that fortran has a modern website: https://fortran-lang.org/

Yes. usenet, however, used to be social media. It was like text-only reddit.

sarcasm?

like in enterprise? the browsers stopped shipping with ftp support I think 5 years ago.

I thought of that ... you could put it in the list but I wouldn't unless you're talking cron. I mean when I started it was like mail bob@froggy.ucla.edu and bob would open up pine or emacs and not get any spam or have to do anything crazy and we'd have a conversation.

that modality is long dead. we still have email but it's not that same ... I however could take mirc from like 1995 and still use everything.

If you look at "old" internet: fidonet telnet bitnet ftp fsp wais netnews whois++ gopher relaynet ... i mean irc is one of the only ones that is still used, and mostly as it was 35 years ago. Pretty impressive.

People love knocking on IRC but people are still on it, still using it...

still researching them ... some of these look like garbage ... i'll come back to this thread in a few days when i find something decent. I don't want high nits or resolution, just want to be able to send pixels and not have it be noisy and get a response in under 500ms.

I've got something similar to this using cheap ONVIF security cameras and kasa smart bulbs. I have an nvidia card process the capture and for the smart speaker I use just a microphone and speaker hooked up to a normal computer. The architecture is spread across many machines and it's nowhere close to this realtime. Also the video capture is fairly lousy because the lighting and resolution and coverage isn't great. The cameras can change the lighting to try to get a better shot. (well it's qwen with toolcalls)

I also have (I think, haven't counted recently) 12 cameras that are all ballhead so they move around and look for me when I say something to the smart speaker.

But I didn't think of the projector ... there's ~$30 ones ... might be worth buying a few.

Getting this level of response, feedback and accuracy, that's the real magic. The latency is garbage. I'm just using cheap OTS hardware though.

Anyone can do this ... it's not that expensive. Smart bulbs are cheap, wifi cams are cheap, omnidirectional usb microphones are cheap and projectors are cheap too. I'm talking like ~$100 total

I'm fully open to the idea of having to fly to China to buy hardware. If we're seeing $25,000 US prices versus like $4,000 in China, might as well go visit the great wall and have a vacation.

I don't have faith in the political parties. Everything is insane. You look at platter recently? It's up 3x in 12 months, not just ssd or nvme, but straight up traditional platter.

https://web.archive.org/web/20250612003557/https://diskprice...

https://diskprices.com/

After 70 years of decreasing computer prices all of a sudden it's gone 3x, 5x, 10x up in 1 year, we are in total clown world and saying "dur AI" is lazy and doesn't map to reality.

It's Argentina style inflation - as if Honda said "we're only making $500,000 luxury cars now. Everything under $50k we've stopped." and then those cars shoot up to $125k.

It's destroys the market, destroys the consumer, destroys the company, dismantles everything, and they do it for the short term payday.

There's nothing raving about it. I was trying to communicate that I've got all the hardware I want. I'm still pissed these companies are ripping people off.

It was supposed to be over by now and then they said 2027, then it's 2028, and now I hear "oh it's going to continue to rise the rest of the decade".

I'm likely going to be flying into Shenzhen to put my next computer together. The one I put together in 2025 would have cost me about $25,000 right now. I paid under $5,000.

I can round trip to China for $750. So once they ramp up production that's the strategy.

Other countries already do this. Apple and Google aren't in every country and those people buy new electronics when they travel.

The USA is soon to be on that list

You aren't engaging in good faith and there's no reason to continue with you.

This is all hypothetical and debating hypotheticals isn't productive so let's roll back to markets.

Let's say ram used to cost $100 and now that same unit costs $1000. You paid say $500x1,000 for that unit during the price increase or some price where you can currently flip for profit.

You have a very expensive data center and you're in debt financed on the premise that you have these special computers.

Now a new technique comes out and it turns out you only need 1 memory unit for something that used to require 8 or 4 or some meaningful multiplier.

This stuff happens all the time. It's why we don't use BMP files on websites or serve giant MOV files on YouTube. It's why postgres queries are faster now than they were 10 and 20 years ago.

You rent out your machines. You need to service your debt.. Demand may 8x overnight to accommodate but you have a monthly bill to pay and that's unlikely. It's likely going to drop.

Think about it. Your customers are paying maybe $10,000 a month and serving their customers. Now they can drop that to $1,250.

On market if you were to sell some of that ram you have 100% profit right now but not for long.

Jevons paradox assumes unlimited capitalization, zero debt servicing, infinite time horizons...

We live in the real world so what do you do?

Historically the answer has been "sell that shit"

There's an aphorism for this "stairs on the way up elevator on the way down"

If we had a healthy market with sane prices where you can't flip the thing you bought for 100% profit the answer would be "create more value."

Capitalism eats itself this way. Second and third order effects will collapse the demand.

You need to keep the market healthy, not some insane Bitcoin style HODL pump - that's how you get wrecked.

I mean I'm not a neoclassicalist but I've read all of them. I'm in consensus with them here. There's a bunch of theories on what a healthy market is but what we're currently seeing matches none of them.

It's short term profitable but long term disastrous, especially in a world where new mathematics and techniques could literally collapse the demand overnight.

Imagine if some paper hits arxiv and the 256 GB requirement for some model now becomes 64. Woops!

Some clever trick about how attention heads and context Windows work could potentially slash a bunch of requirements by giant margins and all they're doing is firing the starting gun at that global race with every obscenely priced unit they sell.

But if prices were reasonable, this wouldn't be an apocalypse. It'd be fine. Consumers wouldn't rush to 64GB, they'd say " Cool I can multitask now at 256" or " great I can do horizontal scalability' or something else.

But no they created the market conditions so now what would happen is the consumer will immediately flip the 192GB they don't need on eBay, hoping to snatch a profit before the prices tank and the second hand market will be flooded the rug will be pulled out from the luxury pricing and everyone will get screwed.

This has happened in electronics markets before. Many times.

When Engels talked about the grave diggers of capitalism they were looking at it through a 19th century labor/manufacturing lens but arguably this same dynamic is at play here.

https://pcpartpicker.com/trends/price/memory/ is better. pcpartpicker has the data.

The AI boom started in 2022. Prices rose THREE years later after 2025 sanction and tariff style legislation to protect the market during a price hike.

I got a 4090 in 2023 for 1600, a 5090 in 2025 for 2000 with 256 DDR5 for about $1,000 ... and then, after some protectionist legislation passed, these prices quickly shot to the moon.

Connect the dots.

wait until holiday shopping...it affects the price of almost everything with a battery or power cord.

Seriously, if a single politician stepped forward and said "i'll bring down ram prices" they could then shoot a puppy and call me a slur and I'd still go out and doorknock for them.

Memory companies have price fixed multiple times. They've paid hundreds of millions in fines. wikipedia even has a page on it. https://en.wikipedia.org/wiki/DRAM_industry_price_fixing.

Look at the financials of these companies, they're all making obscene margins and do they plan to increase production? No. Micron is doing a stock buy back to pump the price of their share.

The Micron CEO just recently said this is the exact plan https://www.theregister.com/systems/2026/10/01/ram-supply-se...

There's sanctions, tarrifs, and a DOJ who doesn't give a shit. Until we can fix that the insanity will continue. Phones will be unaffordable. Laptops will be obscene. Gaming consoles will be thousands of dollars. Desktops will be dead.

If you're waiting for some David Ricardo equation to happen, tough cookies, it's not coming.

The market is legally locked down and we're in hostage pricing mode.

And what's the story? You can't afford electronics because we're using it to build robots to take your job? I mean ...

Nobody is coming to save us. That's our job.

I'll vote and doorknock for any candidate who says they'll end this bullshit and I simply do not care in the slightest about any other issue.

ds4 is by Salvatore Sanfilippo ... this is by what looks like a highschool child who signed up for github last week based on the commit history and artwork. Not the same thing.

I'm sick of these 4 day old vibe coded projects becoming hyped to hell while I spend a year on something and it goes nowhere

I don't know who this kid is but I've looked at the code, it's all Claude. Saying you can run on 2-bit quant with some optimization flags at 100t/s isn't like some "oh my goodness" ... it's wasting everyone's time with noise. I mean give me a break.

What are they doing? Is it Tiktok? Discord? LinkedIn?

I want to do high quality work but apparently I should be dicking around on social media

For historical technology, one of the must reads along with Soul of a New Machine and Hackers

it's going to be mostly open models running on consumer grade GPUs except for a minority of use cases. Depending on what eval you use, the best 20-30b parameter models are at about 5-8 month lag behind the frontier closed source models and that's been consistent for a while now.

There's a longer talk I give on this but briefly we are in a time period where the development of the technology is quickly outstripping people's needs and they will be satisfied by models runnable on commodity sub-$10k machines.

That threshold has arguably been met for many users over the past 6 months and it will continue to be met for the majority of use-cases in the upcoming months.

So not only does Anthropic and OpenAI have no moat - unless they have other compelling products, the demand for their token-based subscription and metering products isn't sustainable because consumer preferences go elsewhere after any product quality reaches a sufficient baseline.

Luckily there's many many options. Faster, cheaper, stronger they're there but these are all zero-profit condition qualifiers. The labs current push, and this is across the board, is to have a compelling suite of applications where people will have a preference for them or there will be some side hustle. Look at Meta muse - they have an app for your phone that will hoover up your personal data in the name of convenience. 5 million people said yes in 3 weeks.