Jweb_Guru

Believe it or not, using arenas does not provide free memory safety. You need to statically bound allocations to make sure they don't escape the arena (which is exactly how arenas work in Rust, but not Zig). There are also quite a lot of ways of generating memory unsafe code that aren't just use after free or array-out-of-bounds in a language like Zig, especially in the context of stuff like DOM nodes where one frequently needs to swap out pointers between elements of one type and a different type.

Respectfully, for browser-based work, simplicity is absolutely not a good enough reason to use a memory-unsafe language. Your claim that Zig is in some way safer than Rust for something like this is flat out untrue.

Yeah people dramatically overestimate the difficulty of getting one's definitions correct for most problems, especially when you are doing an end to end proof rather than just axiomatizing some system. They are still worth looking at carefully, especially for AI-generated proofs where you don't get the immediate feedback that you do as a human when something you expect to be hard goes through easily, but contrary to what seems to be popular belief here they are generally much easier to verify than the corresponding proof (in the case of formally verified software, the corresponding analogy is verifying that the spec is what you want vs. verifying that the program matches the spec; the former is generally much easier).

This website made me suddenly have a huge feeling of loss for what the web could be like. It is so snappy (in the way old static sites could be) but without the page transitions that made them fall out of fashion.

> if you spend too much time interacting with LLMs, you eventually resemble one

Pretty much. I think people who care about reducing their children's exposure to screen time should probably take care to do the same for themselves wrt LLMs.

It makes people uncomfortable. That's the long and the short of it.

Yeah this is something I think a lot of people tend to overlook. People are far too quick to rewrite "we don't know of any reason why it would be impossible" to "we know how to do it" in their heads.

The reason people had these assumptions is because people have been trying (unsuccessfully) to find a constructive interpretation of this stuff for a very long time. Even very fundamental results in measure theory like the Heine-Borel theorem typically require some extension to traditional constructive axioms. Like I absolutely get where you are coming from, but there are a large number of "nice" classical results that definitely do not have constructive counterparts. It's cool that descriptive set theory is not one of them but it's not obvious by any stretch of the imagination, and the pattern you're using to say that it's probably true ("Curry Howard maps nice to nice") is not great process IMO since it would fail in a lot of other cases.

I've done a bunch of theoretical PL work and I find this to be a very surprising result... historically the assumption has been that you need deeply "non-computational" classical axioms to work with the sorts of infinites described in the article. There was no fundamental reason that you could give a nice computational description of measure theory just because certain kinds of much better-behaved infinities map naturally to programs. In fact IIRC measure theory was one of the go to examples for a while of something that really needed classical set theory (specifically, the axiom of choice) and couldn't be handled nicely otherwise.

This is the quality of discourse I expect at this point from anti-Rust folks, yes.

Or maybe consumers do not want to (because it doesn't make economic sense for them, or because they don't want to spend every second of their free time optimizing) spend a significant amount of their time doing this kind of research before every purchase decision? I have never understood this argument that as long as information is technically available companies should be absolved of meeting some minimum level of baseline quality. You're also describing an industry that's unusually transparent (in part because there are not that many models of planes); for a lot of online goods there is absolutely no reasonable way you can know what a particular brand will be like before you buy.

These are just backdoors as a vulnerability class. It does happen occasionally even in high quality widely used libraries, but it's very rare and usually quite sophisticated (e.g. https://en.wikipedia.org/wiki/XZ_Utils_backdoor in 2024). A lot of these attacks, even when highly sophisticated, only really work when the attack payload can be hidden in a binary--it's very very hard to insert a useful backdoor into the source code of an actively maintained project that receives regular code reviews without anyone noticing (which is not to say it's impossible).

Okay. A lot of the blame for this has been previously placed directly at the feet of stuff like AP programs that claim to teach, e.g., calculus to a bunch of people who aren't ready for it; it becomes a prestige thing or expected for admittance to college, which results in a bunch of people being taught to the test and not actually gaining the foundational mathematics education they actually need to understand the subject. I don't see how encouraging this sort of thing actually helps with the problem of universities needing to do remedial math education to people who supposedly know the material already, but it appears to be what you're arguing for. If you're arguing for lower education reform in general, great, but that has little to nothing to do with how highly elite colleges weigh the math portion of SAT scores.

I feel compelled to point out that for people who want to learn mathematics, there are more and higher quality resources than there ever have been before. For the most part they are absolutely free, and unlike virtually every other subject on the planet they are generally not the sort of thing where you can be led astray by misleading material. I simply don't see how such people are being suppressed in any way, or why (from the perspective of advancing the state of the art of mathematics) I should care about the "non Terry Taos" in your words, who are merely above average at math but don't actually intend to pursue it as a career. There are plenty of other skills that are actually eroding at a high rate, or have huge startup and lab costs, or are otherwise underappreciated and underpaid relative to their importance to society; I don't think mathematics is one of them.

Nobody is saying "don't teach math to high schoolers." My point is that this whole admissions rant appears to be entirely about admission to "elite" schools no longer being fully determined by SAT scores (or whatever--I've never been completely clear on what people are actually arguing for), when anyone good enough at math to get a really high SAT score can easily gain admission to a ton of universities with great math departments. As for the real "gifted" kids, there will always be some middle schoolers taking calculus etc. with or without a structured gifted & talented program. The majority of people in these programs are not so far beyond their peers as you seem to think, and my experience in math departments has been that there's a pretty even mix of kids with precocious math backgrounds and people who developed their skills at a later point.

I have never met a single person who wanted to learn advanced mathematics who was prevented from doing this by not getting into the college of their choice, and I know a lot of mathematicians. Do you genuinely believe that people who get an 800 on the math portion of the SATs (which is actually a pretty large number of people) are struggling to get admission to university at all?

In software, where the consequences for initial failure are extremely minimal, sometimes it does actually work well.

I'm sure your friends want to secure a future for children, but unfortunately this motive is not mutually exclusive with being evil.

Your best guess is that the true rate is 20x higher than the observed rate? This seems unlikely to me given the number of samples (outside of systematic biases towards certain types of memory safety bugs that probably apply to C++ code too). 10 per hundred MLOC is closer to what I would have guessed too, but that is because I've historically been very conservative with my assumptions about the memory unsafety rate per unsafe LOC being similar to that of C++. The evidence here suggests that the true rate is probably much lower than that.

> Graydon was ejected out of his own language

That's not what happened at all. Graydon voluntarily stepped down because he didn't want to be in the role of BDFL. It's true that he wasn't a huge fan of the "C++-ification" of the language, but he wasn't pushed out or anything, and definitely could have stayed on steering the project as long as he wanted to. I think there are a number of other languages that would have benefited from a similar approach, actually.

The fact that the safe subset of Rust can indeed be made safe, and that unsafe code can be encapsulated like this, have already been formally verified. This is an empirical demonstration that matches these formal results across a large company with a large amount of code (technically, it exceeds them, but this is only under the assumption that memory safety issues per line of unsafe Rust are identical to memory safety issues per line of C, which is really an unwarranted simplifying assumption).

Do you really believe that "they're not actively looking for memory safety issues in Rust" is (1) true (at least outside of Google, there is actually a ton of security work done specifically targeting just the unsafe blocks, since those are obviously where the memory safety issues lie) or (2) could possibly be responsible for a literal thousandfold reduction in memory safety issues? Remember that the Rust code is often integrated with C++ code--there is not necessarily a way to just test the C++ part even if you wanted to. Additionally, Google has explicitly prioritized code that interacts with untrusted data (like parsers and networking code) meaning it's likely to be easier to fuzz most of this Rust code than most new C++ code.

Also remember that, again, there are mechanized proofs of memory safety for a large subset of the safe portion of Rust, which constitutes 96% of the code under consideration here. The rate of memory safety bugs would have to be 25x as high per LOC in unsafe Rust code as in C code for the number of vulnerabilities to match. It would be far more shocking if we didn't see a dramatic reduction. Google is empirically demonstrating that the observed memory safety bugs per line of unsafe Rust is actually far lower than per line of C, but my point is that even if you think that is the result of bias or them not applying the same scrutiny to Rust code (something that is certainly not true of Rust vs. C code in the wild), the effect of this underrepresentation cannot possibly explain most of the reduction they observe.

Google deployed numerous state of the art mitigations prior to adopting Rust and still found that 70% of their CVEs were due to memory safety issues--your assertion that they are engaged in motivated reasoning and just wanted Rust to work out is pretty ill-founded. In fact when I worked for Google prior to Rust's release, they were strongly averse towards adopting any new language and believed that good engineering practices, automation, and a rigorous review process always outweighed the benefits of adopting a new language past their core ones, whatever its purported reliability or performance benefits. Security researchers are highly skeptical as a rule of these sorts of claims and have a lot of say at Google. They themselves changed their minds based on this sort of internal evidence.

I agree that skepticism is warranted, because we are constantly being sold things by industry. At a certain point, though, when the effect size is massive and persistent and the mechanism extremely clear, that skepticism (not in general, but of a particular claim) becomes the unscientific position. We are well past that point with Rust wrt memory safety.

Is it a usual thing you do that when you're given data about a literal thousandfold improvement, in a context where there are well-understood theoretical and practical reasons why you might have expected to see such an improvement, you make up reasons why it is actually not an improvement at all, without either investigating to see whether those reasons are actually true or demonstrating that even if they were true, they could possibly explain more than a tiny fraction of the improvement?

It wasn't really possible. We had neither the PL techniques nor the computational power to make something like Rust work at the time. All the answers people are throwing around showing it would have been possible rely on a garbage collector and require a runtime, or have many other unacceptable compromises (e.g. no use after free because you aren't allowed to free).

The actual best response would be to run any "unsupported" codecs in a WASM sandbox. That way you are not throwing away work, Google can stop running fuzzers against random formats from 1995, and you can legitimately say that the worst that can happen with these formats is a process crash. Everybody wins.

It feels like maybe people do not realize that Google is not the only company that can run fuzzers against ffmpeg? Attackers are also highly incentivized to do so and they will not do you the courtesy of filing bug reports.

Thanks for engaging thoughtfully! FWIW I actually looked this article up because I was interested in your claim that even experts couldn't perform these tasks, something I hadn't heard before--I'm not actually ignoring what you're saying. It's actually very nice to have a productive conversation on HN :)

I am pretty much certain that parts of it were LLM-written, yes. This doesn't imply that the entire blog post is LLM-generated. If you're a good Bayesian and object to my use of "100%" feel free to pretend that I said something like "95%" instead. I cannot rule out possibilities like, for example, a human deliberately writing in the style of an LLM to trick people, or a human who uses LLMs so frequently that their writing style has become very close to LLM writing (something I mentioned as a possibility in an earlier reply; for various reasons, including the uneven distribution of the LLM-isms, I think that's unlikely here).

Human experts can reliably detect some kinds of long-form, AI-generated text using exactly the same sorts of cues I've outlined: https://arxiv.org/html/2501.15654v1. You may take issue with the quality of the paper, but there have been very few studies like this and this one found an extremely strong effect.

I am making an even more limited claim than the article, which is only that it's possible for "experts" (i.e. people who frequently interact with LLMs as part of their day jobs) to identify AI generated text in long-form passages in a way that has very few false positives, not classify it perfectly. I've also introduced the caveat that this only applies to AI generated text that has received minimal or no prompting to "humanize" the writing style, not AI generated text in general.

If you would like to perform a higher-quality study with more recent models, feel free (it's only fair that I ask you to do an unreasonable amount of work here given that your argument appears to be that if I don't quit my lucrative programming job and go manually classify text for pennies on the dollar, it proves that it can't be done).

The reason this isn't offered as a service is because it makes no economic sense to do so using humans, not because it's impossible as you claim. This kind of "human" detection mechanism does not scale the way generation does. The cues that I rely on are also pretty easy to eliminate if you know someone is looking for them. This means that heuristics do not work reliably against someone actively trying to avoid human detection, or a human deliberately trying to sound like an LLM (I feel the need to reiterate this as many of the counterarguments to what I'm saying are to claims of this form).

> I’m not going to write another detailed explanation of why your “slop === AI” premise is flawed.

This isn't a claim that I made. I believe that text written with LLM assistance is not necessarily slop, and that slop is not necessarily AI generated. The only assertion I made regarding slop is that being written with LLM assistance with minimal prompting or editing is a strong predictor of slop, and that the heuristics I'm using (if present in large quantities) are a strong predictor of an article being written with LLM assistance with minimal prompting or editing. i.e. I, I am asserting that these kinds of heuristics work pretty well on articles generated by people who don't realize (or care) that there are LLM "tells" all over their work. The fact that many of the articles posted to HN are being accused of being LLM generated could certainly indicate that this is all just a massive witch hunt, but given the acknowledged popularity of ChatGPT among the general population and the fact that experts can pretty easily identify non-humanized articles, I think "a lot of people are using LLMs in the process of generating their blog posts, and some sizable fraction of those people didn't edit the output very much" is an equally compelling hypothesis.

I would never claim that we can reliably detect all AI generated text. There are many ways to write text with LLM assistance that is indistinguishable from human output. Moreover, models themselves are extremely bad at detecting AI-generated text, and it is relatively easy to edit these tells out if you know what to look for (one can try to prompt them out too, though success is more limited there). I am happy to make a much narrower claim, however: each particular set of models, when not heavily prompted to do otherwise, has a "house style" that's pretty easily identifiable by humans in long-form writing samples, and content written with that house style has a very high chance of being generated by AI. When text is written in this house style, it is often a sign that not only were LLMs used in its generation, but the person doing the generation did not bother to do much editing or use a more sophisticated prompt that wouldn't result in such obvious tells, which is why the style is commonly associated with "slop."

I find it interesting that you believe this claim is wildly conspirational, or that you think the difficulty of reliably detecting AI generated text at scale is evidence that humans can't do pretty well at this much more limited task. Do you also find claims that AIs are frequently sycophantic in ways that humans are not, or that they will use phrases like "you're absolutely right!" far more than a human would unless prompted otherwise (which are the exact same type of narrow claim) similarly conspirational? i.e., is your assertion that people would have difficulty differentiating between a real human's response to a prompt and Claude's response to a prompt when there was no specific pre-prompt trying to control the writing style of the response?

> It was absolutely an example of the pattern, just more wordy. Spare me the ad hominem.

LLMs simply don't generate the syntactic pattern I used consistently, but they do generate the pattern in the article. I'm not really sure what else to tell you.

The rest of your post isn't really that interesting to me. You asked why nobody was giving specific examples of why it was generated. I told you some of the specific reasons we believe this article was generated with the assistance of an LLM (not all--there are many other sentences that are more borderline which only slightly increase the probability of LLM generation in isolation, which aren't worth cataloguing except in a context where people genuinely want to know why humans think a post reads as AI-generated and are not just using this as an excuse to deliver a pre-prepared rant), mentioned that the reason people don't typically bother to bring it up is that we know people who demand this sort of thing tend to claim without evidence that humans write in the exact same way all the time, and you proceeded to do exactly that. Next time you don't get a response when you ask for evidence, consider that it might be because we don't particularly want to waste time responding to someone who isn't interested in the answer.

Again, this is why I don't bother explaining why it's very obvious to us. People like you immediately claim that human writing is like this all the time, which it's not. Suffice it to say that if a large number of people are immediately flagging something as AI, it is probably for a reason.

My reply wasn't an instance of this syntactic pattern, and the fact that you think it's the same thing shows that you are probably not capable of recognizing the particular way in which LLMs write.