davesque

I have a lot of questions about methods here. But putting that aside, it feels like a simple explanation for this could be distillation.

Might never come to fruition? Is there any chance at all that such a car would ever be street legal? Oh, wait the US govt is literally for sale nowadays, so hmm...maybe that's not an obstacle actually.

Because it's creepy and Orwellian, not to mention hypocritical (MAGA is supposed to hate thought policing). Do you like people telling you what words you can and cannot use?

Does any American actually think this way about lobbying?

> If you're not going to do that don't post your results online, just keep it to yourself

...I mean...I just don't really know how to respond to that. Who the heck cares if someone posts this online? If you don't like it, just move on. Sheesh. Lookout everyone, this guy doesn't approve of your side project.

But C isn't as nice to work with. Maybe OP just thought it was fun to see it take shape and wanted to share it. Why do any sort of little side project like this at all? Just seems like everyone is judging this too harshly. If an LLM is reasonably good at this sort of thing, and you think it would be cool to see Quake written in Rust running in the browser, I say just do it and have fun and don't be shy about doing show and tell about it.

I mean...why not do it? Are you implying it would only have value if someone painstakingly did it by hand?

This article's argument in a few sentences:

* Silicon valley gave us DraftKings, Polymarket, and crypto.

* Modern business planning and venture capital firms make "bets" based on the expected value of investments, i.e. probability of success.

* LLMs compute a probability distribution on the next token.

Therefore everything that silicon valley does looks like gambling and shame on them. Sad times seeing stuff like this in the paper of record.

Yes, it would be correct not to pursue a technology that has that chance of wiping out the world. But where are the people who are suggesting these probabilities getting their numbers? I personally can't imagine where, and I'm an engineer with a specific technical interest in LLMs. And I haven't heard one clear description of the methodologies used to calculate these chances.

The much more likely explanation to me is that people are just spitballing, either because they've watched too much sci-fi, or they have some weird counterintuitive agenda (e.g. Anthropic and OpenAI trying to position themselves as the amazing, trustworthy keepers of this dangerous technology before their IPOs).

Yeah I wish we would focus on concrete risks like job displacement and disinformation. The apocalyptic stuff feels either misguided or like some kind of weird, toxic, reverse psychology marketing by OpenAI and Anthropic. I wish we would just move on from it.

And the folks from podcastistan are never clear on the details of how human extinction would happen exactly. It's always something like, "Well, how do humans regard chickens? AI is way smarter therefore it wants to conquer and control us." An ASML lithography machine is also way better at making chips, but we don't consider it a threat.

I'm a bit concerned that a government chatbot like this would include a user-feedback signal like thumbs up/down. I think this could be gamed by bad actors to promote/demote specific types of content in the training data, assuming they use those signals in some sort of post training. Hopefully they're accounting for that.

AI-generated writing does not bother me, unless it is intended as prose writing, where the flow communicates as much as the specific facts that are described. For technical writing, I usually just scan for important information anyway, and in that sense the only thing that bothers me about AI-generated writing is if it's too verbose and obscures the important information by leaning ineffectively on jargon and high-minded verbiage. I find that Claude Opus is very guilty of this, whereas GPT 5.6 variants are not.

Can we please make it so that AI CEOs are smart enough not to optimize everything according to the focus group that is their shareholders? Seems like we could solve the enshittification problem with this if we play our cards right.

Unless humanity can figure out a way to make a binding international agreement to pace development, it doesn't seem practical to expect US frontier labs to do it.

Regarding automatic formalization of proofs using AI, how do we know the formalization doesn't contain errors?

Isn't it also a simple idea that a model designed to recall relevant information from its training data, which is also known to have been trained on data from user transcripts, would, in fact, reproduce directly relevant work by leading experts in the field? Seems like Occam's razor would apply to that situation as well.

We know that LLMs are trained to recall relevant info. We know AI vendors are using user transcripts to train models. Two plus two equals four, right? I mean, an LLM that failed to recall the transcripts of those researchers would be a bad model.

Honestly this whole thing is so fucking weird. I feel like there's an argument that absolutely no one involved in the final crossing of the finish line to the proof actually did any work (other than just intelligently directing an LLM) and deserves any credit. As the author of this doc mentions, the mathematicians who did the actual work that led to the formulation of this approach (without the use of LLMs; just good ole' fashioned human intellect) are the ones who deserve the credit.

Imagine that a no name janitor used their time in the evenings to go spelunking through the literature to push an LLM to this result. No one would care because that person isn't an anointed expert. So why would the expert deserve any more credit? Because they sort of understand the result, even if they couldn't have achieved it on their own? The whole issue of credit for AI-assisted discoveries seems like it's going to run into a brick wall pretty soon.

And then he wanted New Mexico to be New America. Capitulation is not just about what they're asking for now, but also what they'll be emboldened to ask for in the future.

Corruption in the US Census Bureau? No fuckin' way!

The article doesn't address what I think are the main complaints about H-1B: that it is mostly used to cut costs and hire people who have very little freedom to switch jobs. That and all the trickery with ghost jobs meant to only give the appearance of hiring stateside. The example given is also not great. Von Braun was one remarkable foreign expert in a leadership position. On the other hand, in my experience, many H-1B hires tend to occupy low-level jobs; ironically the very ones that the author says will be filled by motivated Americans. I mean...just try working at any American FAANG. The evidence is totally in your face.

In a way, it's nice that Trump is giving such a good, practical demonstration of why being a bully never works out the way you think it will. A good education for those who need it. Just sucks that we all have to pay for it out of pocket.

The voting public is not the same as the general public.

Who's clapping?? I think the man should be in jail. A minimum of half the country has always hated this guy's guts.

That's not saying much. The real distribution is probably heavily skewed by risk factors such as poverty, age, drug abuse, etc.

I actually don't feel like Tao's recently published conversation is the best example of this idea. As intelligent as Dr. Tao is, and surely more so than me, I got the feeling that he wasn't running up against failure states of the model, which I'm not sure you could attribute entirely to his expertise. I honestly think it was more a matter of luck that the model apparently had so much training data on the topic or that it was architecturally so well suited for it. On the other hand, I've had really surprising moments where Claude was just failing terribly to execute simple dev ops tasks having to do with log processing. And I'd be so bold to say that I don't think it could have been explained by a lack of expertise on my part, or even a misuse of the model.

So yeah, sometimes LLMs reward expertise, sometimes they don't. I guess either way it helps to have it.

Everything he does benefits his adversaries because they're all playing him like a fiddle because he's basically the dumbest person alive.

I disagree. In fact, I think the field of dev ops gives a clean analogy with mathematical proofs. My point was that my work often requires that I figure out a consistent way to prove to myself what the condition of a system is by asking the right probing questions about it. What I've seen is that even the best LLMs lack a good intuition about what questions they should be asking and instead reach for the quickest and most obvious checks that leave edge cases uncovered. Maybe it is something about the domain of the problem; I don't know. But it makes it hard for me to imagine that an LLM wouldn't make similar errors in other cases, especially when generating mathematical proofs that will soon be too dense for humans to review.

Exactly. I really wish people wouldn't use this. If this becomes popular, anthropic will just modify their cache policy to be much less fair. It's not like they have infinite cache.

Without any more context, "keep going" seems to be doing a lot of work. The user is placing a lot of faith in the LLM to not make subtle logic mistakes and to take good approaches to each problem. In my experience, even frontier models (such as Fable) are quite capable of getting confused during even simple technical work I've done in the dev ops world. For example:

LLM: This package hasn't made it to production.

ME: are you sure? i see it right here!

LLM: You're right to push back. I inferred that based on weak data. I see now that the package has been deployed!

If the above conversation is typical for me, how could one expect to achieve a sound result by repeatedly prompting an LLM to simply "keep going" in dense mathematical proofs? Perhaps the user in this case had actually checked the LLM's work before issuing the prompt, but I think you see my point anyway.

Not sure I follow. I can't imagine humanity has evolved much since 1974.