andy99

Nature abhors a vacuum

You obviously know what you’re looking for better than me, but personally I’d want to see a narrative that made sense before accepting that a smaller model somehow just performs better, even if the benchmarks say so. There may be such an explanation, it feels very dicey without one.

What does that look like in your opinion, what do you use?

How much you value control is one part of the optimization problem. Obviously self hosting gives you more but it costs more, and re evals, I trust GPT, Gemini, and Claude a lot more than some smaller thing I self host, and would end up wanting to do way more evals if I self hosted a smaller model.

(Potentially interesting aside: I’d say I trust new GLM models similarly to the big 3, but they’re too big for most people to self host)

Depends on what you’re doing. Using the smaller / cheaper LLMs will generally make it way more fragile. The article appears to focus on creating a benchmark dataset with real examples. For lots of applications, especially if you’re worried about people messing with it, about weird behavior on edge cases, about stability, you’d have to do a bunch of robustness testing as well, and bigger models will be better.

Another big problem is it’s hard to set objectives is many cases, and for example maybe your customer service chat still passes but comes across worse for a smaller model.

Id be careful is all.

Imo this is what happened once protests became a “right”. I know most people here won’t agree with the Canada trucker protest, but I remember when it happened, people were saying “ok, you’ve had your protest and exercised your rights, you’ve been heard, you can go home now” - framing it just like that, as a rally to show an opinion rather than a threat. It felt to me like “the establishment” just treats them as peformative, because as you say they usually are, and then doesn’t know what to do when it’s actually something they have to react to.

Is anyone aware of a more thorough argument for why this must be the case? Is it a commonly held view? It sounds realistic, but not necessarily and immutable law, I’d like to know what thought has been given to this.

Yes, just clicked on the video, instantly nauseous. I get motion sick generally.

There was a COBOL LLM eval benchmark published a few years ago, looks like it hasn’t been maintained: https://github.com/zorse-project/COBOLEval

At least I think that’s the repo, there was an HN discussion at the time but the link is broken now: https://news.ycombinator.com/item?id=39873793

I recently learned Bear blog (a small blogging platform, posts on which often appear on HN) has a “discover” section with a front page style ranking. Their algorithm is on the page

  This page is ranked according to the following algorithm:
  Score = log10(U) + (S / (B * 86,400))

  Where,
  U = Upvotes of a post
  S = Seconds since Jan 1st, 2020
  B = Buoyancy modifier (currently at 14)

See https://bearblog.dev/discover/

I just saw a sibling post about Kagi, maybe this is how the industry will end up, with a main provider like OpenAI and niche wrappers on top (I know Kagi is not just a google wrapper but at least they used to return google search results that they paid for).

There would be competition from API wrappers, if you want to pay there will always be lots of options to chat without ads. I hate to think what they and others might come up with to try and thwart this.

Everything about this is so stupid, from the fact that these pills are expected to change population weight so much to the silly napkin math that some analyst has pulled out of their ass to turn it into airline EPS.

What if the only moat is domains where it’s hard to judge (non superficial) quality?

Code generation, you don’t see what’s wrong right away, it’s only later in project lifecycle that you pay for it. Writing looks good to skim, is embarrassingly bad once you start reading it.

Some things (slides apparently) you notice right away how crappy they are.

I don’t think it’s just better training data, I think LLMs apply largely the same kind of zeal to different tasks. It’s the places where coherent nonsense ends up being acceptable.

I’m actually a big LLM proponent and see a bright future, but believe a critical assessment of how they work and what they do is important.

  it only gleans logic from human language

This isn’t really true, at least how I interpret the statement, little if any of the “logic” or appearance of such is learned from language. It’s trained in with reinforcement learning as pattern recognition.

Point being it’s deliberate training, not just some emergent property of language modeling. Not sure if the above post meant this, but it does seem a common misconception.

No, this describes the common understanding of LLMs and adds little to just calling it AI. The search is the more accurate model when considering their actual capabilities and understanding weaknesses. “Lossy compression of human knowledge” is marketing.

I had no idea they still had which I understand to be a “metaverse” division, especially with thousands of people. People there must have realized they were on borrowed time.

I canceled prime ~8 years ago because where I am, half the stuff I wanted was considered an “add on item” that could only be shipped free if you had > $35 of other stuff, which is a complete scam because you get that without prime.

Maybe that was just for me (in a large Canadian city at the time) or maybe they don’t do that anymore?

I haven’t considered getting prime since, it would be a lot more interesting if it actually provided the shipping terms they advertise.

Why wouldn’t you just ask them, and particularly for media like this that has a native language, default to that? I don’t want software to think for me about what I want to see, if I want something different I’ll change it.

Big tech has changed over the last bit to trying to tell us exactly what and how we should think - or maybe more precisely to see thought as “friction” and want to remove it all together. This is a very minor example in the scheme of things, but I see it everywhere now.

Canadian healthcare is horrible, pretty sure public healthcare in the UK is also very bad. It’s not a given that somehow switching to public healthcare will make the US like Finland, Canada is much more likely.

Government is probably the worst actor to run healthcare facilities. It’s not that different from PE, except with more administrative bloat. I’d be curious to compare US PE run facilities with government run facilities in Canada.

There is not an easy answer here, it basically a cost centre that whoever runs it, the welfare state is incentivized to spend as little as possible on it. PE is almost certainly a bad solution. If they can destroy a restaurant or other low impact business, I hate to think what they’d do to businesses that care for people. You’d get the healthcare equivalent of Burger King. But with government you get the equivalent of the DMV.

These should be useful signals to regulators. Regulation is imperfect but somehow we blame companies that take advantage, either of immature or nonexistent regulation (which might be the case here) or of poorly written regulation (e.g. circa 2000 California energy regulation that let Enron and company run wild).

These companies, in finding essentially arbitrage opportunities, are, perversely, helping strengthen regulation, but only if regulators pay attention to what is going on and do something about it, instead of just watching it happen.

I don’t buy it. Obviously LLMs have a role and are powerful, but they (and more importantly the people that think they know something because they can prompt an LLM) are more like the kid in 2008 that thinks he knows something because he has Wikipedia. Fact lookup isn’t intelligence, it’s not even idiot savant intelligence which I think is the point of the article.

You should try doing 1/2 hour of exercise after you get back. It will make you feel way better, especially mentally.

Re DGX, I’m mostly interested in local inference, it might have been nice to try but it was more expensive for similar performance (or so I think).

I do lots of different experiments, synthetic data generation along the lines of Magpie is one of the things I wanted a local machine for, as well as just general access to a decent sized LLM to try different things, without having to spin up a cloud machine each time.

I would prefer PyTorch / HF transformers to llama.cpp as I fine the latter less flexible if I want to change anything.

Just bought a Strix Halo (framework desktop), waffled a long time between that and a Mac Studo but I got tired of waiting for the M5 and don’t really like Apple.

I work with ML professionally, almost all in cloud, I just wanted something “off grid” and unmetered, and needed a computer anyway so decided to pay a bit more and get the one I want. It’s “personal” in that it’s exclusively for me, but I have a business and bought it for that.

Still figuring out the best software, so far it looks like llama.cpp with Vulcan though I have a lot of experimenting to do and don’t currently find it optimal for what I want.

People would rather blame external factors and not take responsibility.

It’s actually insulting to people who work hard that some people assume they have it easy somehow, like the “must be nice” comment upstream. Not everyone takes the view that you can’t control what happens to you, it’s pretty easy to see who does.

My (Canadian) bank extorted me into installing their app, literally blocking me from doing transfers of my own money without it - I had to install it and take a picture of myself and my ID. After this I was able to switch to sms authentication and delete it, but they’re obviously trying to force people onto the app, and eventually they will do so more aggressively.

Of course in Canada we have a banking oligopoly that is effectively there just to rob people, but ironically any of the “challenger” startup banks are 100% app based afaik

I don’t know the backstory but Cloudflare arguing for an open internet is super ironic, presumably he means they want the be the one to close it off and are upset that someone else is ruining their monopoly on it.