andy99

Interesting to see the claudeslop reply as the first comment to the gh post and the reaction to it.

Not sure the relevance of this comment, but normally if someone built a classifier that bad they’d be fired. Anthropic obviously thinks they have some monopoly power they can use to foist garbage on consumers, I think they don’t.

I realize hallucination has no precise definition but this doesn’t sound at all like anything I’ve ever heard called hallucination. Hallucination is usually plausible wrong answers or made up info that ends up fitting the most likely response (like a manufactured citation) and comes from the way LLMs work at predicting tokens. This example demonstrates completely implausible output, it’s not something that fits with hallucination.

All that said, it doesn’t require cross session leakage, it could just be training data or like those nightingale (probably the wrong bird*) data generations where they just prompt an LLM with nothing and it starts spitting out conversations.

I see a bunch of downstream comments about caching, sounds like maybe there’s an error where it loads nothing instead of the cache and so starts spitting out random generations.

* edit: it’s magpie. Worth looking at the concept, I’m not sure people realize they LLMs generate random conversations when prompted with nothing, this seems at least as likely as sessions leaking: https://github.com/magpie-align/magpie

The people that are homeless now would have been institutionalized or dead back then.

Read “The Gervais Principle” https://ribbonfarm.com/2009/10/07/the-gervais-principle-or-t...

I think it explains everything, most companies are optimizing for “confused” - a class within the framework of people who work hard for no reason.

Not saying it’s the best but it’s certainly a way to do it, and probably becomes inevitable once the company is big enough.

I also think it closely parallels the practice of companies being actively hostile to their customers (or Pareto optimal as they might see it). Big companies would rather provide an offering that suits them, actively mistreating their customers, and just target customers that will go along with it rather than wasting time on customers that want some less profitable version of the offering.

I worked in a grocery store meat department in high school. I wasn’t a butcher, my main job was wrapping the meat for the counter, weighing and pricing it, but over time I learned to do basic meat cutting, still nowhere near a full butcher but getting some basic experience.

Point is it’s an apprenticeship that takes years and starts with helping a butcher do low value stuff. Grabbing half a cow and trying to cut it up probably isn’t the best entry point or test of aptitude.

You can run one on a cloud provider. You’re correct that intelligence orgs probably still can access them, but if you’re that high value of a target then you have bigger problems and / or can afford to build an air gapped system or whatever. If you’re just concerned about other companies mining your messages, self hosting in the cloud solves that.

Reminds me a bit of the old “is your adversary Mossad or not Mossad” decision matrix https://www.usenix.org/system/files/1401_08-12_mickens.pdf

It’s not no reason. At a fundamental level I don’t trust the companies any differently. But at a professional level, nobody is going to question my using Claude or OpenAI in a professional capacity - to work on customer projects, analyze their data, etc.

I also consider Microsoft to be the biggest industrial spy in the world, them and google both are no doubt mining everything you type into office / gsuite, all your emails, etc. But nobody bats an eye when you write a word doc about some sensitive matter.

If my customers thought I was feeding their data into a Chinese owned LLM API (which to be clear I’m not), I don’t think it would go over well, and I’d be exposed legally to all sorts of things.

So the reason is risk aversion and desire to participate in US / western commerce. One can debate the actual threat, but why would you ever risk sending your data to a processor perceived as dodgy?

I’ve never had a refusal coding, and in some areas (AI red teaming specifically) I’ve found it quite good at recognizing and discussing “white hat” stuff that in the past I think would have got refusals.

But when there was the Hantavirus thing a while back, I asked it if there was a vaccine under development and got a refusal immediately. I’ve had a few like that. It seems they’ve implemented really poor guardrails on certain topics (CBRN and cyber) that have lots of false positives. But if you actually chat with the model itself it’s quite lucid about what is legitimately dangerous and what is just performative “AI Safety” style refusal.

Do you guys use it through open router? Do you have any concerns about how the data you send is being intercepted? Not that I trust Anthropic but it’s widely agreed that it’s kosher to use them for commercial work, I can’t see comfortably sending any customer data to openrouter.

Edit- I see down-thread you use z.ai directly. Same concern, aren’t you worried about using it for professional stuff.

All the discussion this week have been about GLM, Qwen, etc. Both over 1000 comments in the last couple days.

https://news.ycombinator.com/item?id=48709670

https://news.ycombinator.com/item?id=48721903

Of course Anthropic is still relevant, but people have realized they’re not special, and between this and the ID verification thing, they’ve given up a ton of their relevance vs a month ago.

By “abusive” they probably mean “doesn’t let us track them”. I’m surprised they’ve kept old.reddit.com going this long, it’s actually an un-enshittified version of the product, which is basically unheard of.

That was with the MTP version

I get ~55 Tok/s on my framework desktop with the 35B A3B q8 model, and so far am also very happy with the coding performance.

I wondered if the whole thing was just a vibe coded weekend project. It’s hard to gauge the seriousness and level of effort but without some evidence to the contrary I’m guessing it’s all LLM generated.

Right, but is there any evidence of intelligence behind any of these (government) decisions? It’s just regulatory capture + marketing (plus some people living out an imaginary fantasy that they’re in Neuromancer or something), absolutely no reason to think they won’t try and target open models as part of this.

Other than maybe some in-the-moment cybersec wrappers, is this really true? Does anyone think a startup with a good product is going to be materially disadvantaged by not having access to an incrementally better security focused LLM release? It’s lots of fun to pretend it’s some step-change that’s too dangerous for general release, but in real life it’s not conferring some massive advantage that any real startup would need to compete. Almost everyone would be best just to ignore it and keep building.

(Just to be clear, I think the gatekeeping is ridiculous, especially given the above)

> Chinese labs must entirely retool from harvesting frontier model data to producing the data systems and efforts to produce novel data

Even if your characterization is accurate, they could do this tomorrow and are not so myopic that they wouldn’t have thought about it. I don’t see this as a barrier, and I see a lot of the same underestimation of Asia that’s been happening for 50 years. There’s not some innate American advantage to building LLMs, and personally I think whatever head start the US has is going to be squandered on delays from the export control “to dangerous for release” LARPing we’re seeing.

In the early days of the LLM era, there was lots of talk about how big incumbents, in particular google would be disadvantaged relative to “startups” like OpenAI because of their valuable legacy businesses that could be destroyed if something went wrong. Mainly people thought about big lawsuits but government action is similar.

Now OpenAI and Anthropic are big incumbents with Trillion dollar valuations at stake, so they can’t take any risks. Unlike google they don’t really have a thriving primary business to protect though, so without being able to continue to take risks and ignore regulation startup-style, it’s going to be a lot harder for them to stay relevant.

How compatible is never replying with the threat model you are trying to avoid? Attack success is probably more likely when the attacker can iterate based on replies or engage in multi-turn conversations. Here they’re just taking stabs in the dark with no feedback. Does that accurately represent the access a real attacker might have?

I think they may have overplayed their hand so to speak. The end consequence is that their best model isn’t available right now, people are exploring alternatives, and realizing they work fine.

It’s such a fast paced and competitive industry, anyone who takes even a short break is going to have a hard time coming back from it, and that’s basically what they’ve done.

This seems to come up in every discussion, in practice it’s irrelevant both because it’s too complicated for normal people to understand, and because the point of all this nonsense really is identification so anything that defeats that will be a non starter.

Unfortunately I don’t think it has the public’s attention, it’s still very niche. Nowhere near enough to change anything yet.

I generally agree with the post but also don’t see a problem with what it claims is going on. It could be summarized as “kids don’t have any respect anymore” or whatever other lament by someone entrenched about new ways of doing things. I think the behavior he calls out is classless, but have absolutely no problem with (and I think encourage) people who aren’t deferential to VCs. It’s sort of the ultimate “aristocrats complain about the proles”.

It’s supervised learning rather than RL, you’re just training to labels. It doesn’t work (doesn’t generalize) because there is no guarantee or even expectation that any causal relationship is learned, it’s just whatever convenient pattern gets the lowest loss. There is lots of research on this for those unaware.

At least in my academic experience, there are academics that are uniquely suited to academia (for better or worse) and there are “academics” that know what to say to be the business version of whatever the discipline is. Neither is necessarily better or worse, but they’re very different and not doing the same thing. Presumably it’s the latter that fit in well with bigco philosophy orgs.

I will say though I went to and taught at lower tier schools, if you’re going to Stanford or whatever it might be so competitive that everyone has already been screened to be the second kind and will do just fine working in industry.

I wonder what it is they feel LLMs can’t do for them and they need a human for? I’d like to see the spec, like are they expecting the philosophers to use Claude all day, maybe in a philosophy harness and will be judged by their token use? Or have they isolated to high value philosophy task that they’ll work on without using AI? I think it would be interesting to contrast with developers and understand how vs philosophers they are seen differently. How much of philosophy is boilerplate vs original thinking and how does it compare to writing software?

That sounds like providing context rather than anything philosophical, and it stands to reason that it would lead to better decision making.

In a world where everyone is using LLMs, the only way to differentiate oneself is to actually think. I don’t know if this is part of the idea behind having some in-house philosophers but it would be interesting. If I was a big lab I’d definitely want some “clean room” humans providing input that’s not just what a model regurgitated.

If we start letting trees communicate by underground signalling though a network of mycorrhizal fungus, it’s going to be the death of French. It’s important that every living being learns to adopt the culture.