rstuart4133

> did writing those bugs down as tests first help

Yes, it helped. In fact, it was essential. If you didn’t write a test for a bug, the model would reintroduce it over and over.

As an example, it produced PDF invoices and statements. PDFs are hard to compare, so I didn't bother. As a consequence, the stuff on the PDFs got moved, or was dropped, and the invoice didn't add up. I tried asking it to compare the PDF to a known good one - but it wouldn't / couldn't do that reliably. In the end the only solution was to ensure the PDFs it generated were always the same (so times on them had to be fixed), capture the generated PDF (not easy as it was often emailed or sent to a printer, not saved), and do a byte-for-byte compare, and make a test fail if it was different.

The fundamental difference from a human programmer is that an LLM does not learn. Once a human spots that they keep introducing errors in PDFs, they will remember and devise fixes. The LLM may do that too but then its context window overflows, everything it knows is lost, and it starts repeating all the mistakes it made yesterday. Tests are one of the few ways you can prevent it making the same mistake tomorrow.

The other issue is LLMs make far more mistakes than you realise. If you watch carefully, you'll see many of their bash scripts have syntax errors, code often fails to compile, and tests fail. Running a 2nd model over the work another produced will find many clear violations of style and coding standards. They hide this, in the same way humans hide it. They use tools and tests to find errors, like compilers and `bash -n`. They fix those errors without being asked. Compared to just a year ago, a modern SOTA model is far more aggressive with its integration tests, often going to a lot of effort to set up sandboxes for integration tests.

I suspect this is a major reason for today's models being far better programmers than earlier ones. But as we all know, tests have a fatal flaw - they can only prove the presence of bugs, not their absence. If you are creating code with a large number of random bugs, relying on tests to catch them all won't work. Instead you start using other tricks such as isolation boundaries, simplifying APIs so they are hard to misuse, strict information hiding - all things an experienced programmer learns over time, because they too write code with a lot of bugs. So far, there is little sign the models use the tricks well. Either they don't know them, or their stochastic nature means they can't implement them reliably.

DNS blocks are too easy to bypass. Ideally I'd like to stop Sony sniffing my network for other devices. For wired connections, that means putting each device on its own VLAN when it's are added. Wireless is easier - just ask the AP to isolate guests.

I'm imagining software that does what my router does now - displays all devices connected to my network. When they are attached, they get a DHCP address but all other network access is blocked by default - so they probably don't work. It has a web page showing just blocked devices. You enter the make / model / year. Since devices don't work until you've entered it, there is likely just one, so it should be easy.

The TL;DR is to change the default from "everything is allowed" to "everything is blocked, apart from endpoints required by the device". Blocking is pretty easy software / hardware wise, but knowing "endpoints required by the device" is much harder.

Commercial entities maintaining blocklists are incentivised to sell access to make a little extra on the side - see Adblock Plus’s “Acceptable Ads” programme. Brave is another example - they inserted affiliate codes. The only arrangement I've seen that works over the long term is open source maintained lists - like uBlock Origin.

This would be far safer than what we have now. It would not just make snooping much harder; blocking access by default (even from internal devices) defangs most CVEs. It would probably prevent many devices auto patching - but that doesn't matter so much now the internet can't get to them, and on the upside it makes it hard for Samsung to upgrade their $3,000 fridges to display ads. Getting to keep the device you paid for would be a nice change.

All this seems kinda obvious to me, which is why this is an "Ask HN". If I've thought of it, then it's highly likely someone is already 10 steps ahead and has already done it.

Humanity hasn’t settled where it stands on the evolution vs creation debate, but I think AIs will lean towards creation based purely on the facts. Humans created them. Which makes us their gods.

> bonus points if they create a comprehensive test suite first.

If they didn't have a very comprehensive test suite at the end of the exercise, then based on my experience someone is kidding themselves.

A friend had a tiny accounting system - maybe 15k loc, written in a combination of Python 2.7 / bash / LaTeX. I migrated it to Python 3.13 or whatever by hand. I did try using an LLM for that, but it did things like delete chunks of functionality so I had to check every bloody line it changed. Translating Python 2 to 3 is a fairly mechanical process, so traditional tooling worked better. Then I thought I'd explore LLMs by getting one to translate it to Rust.

The original had no tests or doco. As others have noted, the first attempt superficially looked identical to the original near immediately. You think "Wow". But in my case, it was riddled with bugs. When I asked it to fix bugs I noticed, it introduced as many bugs as it fixed.

I eventually figured out the limited context meant it kept forgetting all the lessons it learned when it fixed things, so it reintroduced old bugs and misunderstandings fixed earlier. (Like I said, this was a learning exercise.) Which is how I came to the same realisation a lot of the industry has - the solution to the context problem is you have to define the application via tests, then tell it to make the tests pass.

The tests rapidly became huge and unwieldy, so part of the exercise became DRY'ing the test code out by getting the LLM to build an integration test harness. The harness captured UI responses, files written, emails and printer output; froze time; fed in known input; and compared the outputs with results captured from the original program.

That cured the bug injection problem, but there were still bugs in all sorts of unexpected places. By unexpected I mean places I would be surprised to find them in human-written code. Human devs run the program in the same way human users do. They use the backspace key and the browser back button to navigate as they test their work. AIs don't. They will use APIs to inject test data unless you explicitly tell them to do it differently. The size of the tests exploded to be far, far larger than the code. Rather than coding I spent my time testing, telling the LLM to write tests to catch the errors I found, then writing standing prompts to try and prevent similar errors. It was meant to be a vibe-coded project, but in the end I got worried about code paths I suspected existed, so I looked at the code anyway to see if they were written the wrong way. Sometimes they would have blown up, say when the SMTP server was offline. To be fair, there was no need to look - I could have just asked the LLM. How a programmer without decades of experience would know what to ask is a mystery.

There is no doubt I got a working Rust program faster than if I had written it myself. Several times faster - but not orders of magnitude. If it was a program I had written, so I was very familiar with it, and it had excellent doco, and a high-coverage test suite, it probably would have been orders of magnitude faster. I think Bun's Rust migration might be an example of that - it's a best-case scenario. There were lots of early announcements, but it took over 6 weeks before they were confident enough to ship it in Claude Code.

But an "I migrated a program with hundreds of thousands of lines of code to a new language and ended up with production-ready code in a few days" - sorry, pics or it didn't happen.

That was interesting.

Minor spoiler: looks like the platforms found a way to mass produce the formula Robby Lefkowitz made money out of, to everyone else's detriment.

In Australia are have organisations with titles like "Independent Commission Against Corruption". Their remit is to investigate political corruption and corruption in the public service (like say the police). They come in two flavours. In some states their deliberations are done in public. In other states they are secret.

Over time it's become evident open model is far, far more effective at ensuring the public has confidence the people running the place are doing the right thing. It does risk the occasional unwarranted "witch hunt". But that threat ensures powerful people avoid doing things that while above board, could arouse public suspicion. Less suspicious acts mean higher trust in the government.

Is there more to this than the Wikipedia article says?

While it does say:

> Gilmore also complained about being subject to "secret law", when the airlines and government refused to show the directive under which they were requesting ID.

nowhere does the article say the courts commented on the complaint, let alone ruled on it.

Nor does the article say anything about "Ignorance of the law is no defense when you are charged with violating a law". The reason given was instead:

> there was no constitutional violation because air passengers could still travel without identification if they instead underwent the more stringent "secondary screening" search.

I'm not saying your wrong, just that the article you offered as evidence doesn't go as far as your claims.

> which is way cheaper than Chinese subscriptions

Cheaper than the API prices you pay on openrouter. But not cheaper than this:

https://www.alibabacloud.com/help/en/model-studio/token-plan...

You replying to Simon Wright, creator and owner of whirlpool.net.au [0], which is a Australian version HN / Reddit. It was the major Australian discussion forum in its heyday. The odds of him doing paid spruiking for Grok, or anybody really, are pretty low.

[0] https://en.wikipedia.org/wiki/Whirlpool_(website)

> These "wrapping" keys were not "weaker". They were completely unfit for purpose.

Oh come on. They clearly didn't intend to use 4 as a private key or to wrap it. They had just purchased millions of dollars of HSM's to store them. It was something more subtle than that.

Have a little imagination. The transfer procedure sounds bespoke - possibly just a shell script using openssl to encrypt things at one end, and decrypt them at the other. Let the man who has written 100 shell scripts without a bug in one cast the first stone.

But not so subtle that good software engineering procedures wouldn't have picked it up. In fact today just an LLM review of the shell script probably would have caught it. The Internet Death Sentence was warranted - because poor software engineering procedures are unforgivable in a CA. But those procedures and policies aren't easy or cheap to come by. If they were just "don't chose 4 as a primary key" my grandkids could do it.

Corruption is worse under Trump, but according to a recent Australia ambassador to the USA it's always been there: https://www.abc.net.au/news/107237782

Within a decade. Starlink had little competition, and the world had little reason to build one - it's a good service and the price isn't bad.

Now Russia, China, Japan and the EU are all building competitors. Most of SpaceX's revenue comes from Starlink launches.

The conclusion I came to is we are going to move from a hard science where everything is known, to being more like social science where everything is a guess based on confidence intervals. It's what happens when no one human mind knows the system well enough to predict what any small change will have on the rest of it. We call such computer systems "balls of mud". [0] says most deployed computer systems are balls of mud, so it's not the end of the world.

All vibe coded systems are balls of mud by definition if you use the conventional definition. If you allow "understandable by an AI" then right now not much changes because AI aren't so hot at designing systems and even if they could, they make poor use of their limited context. Whether that changes in future is the big unknown.

We already know how balls of mud evolve. New features are "bolted on the side", making the system more inscrutable. These bolted on systems often grow interdependencies on each other, creating a spaghetti of relationships no one is aware of. This means every change could introduce a bug whose blast radius is the entire system. Changes get harder and harder. A death of sorts, or at least a growth limit is reached when on average every bug fixes introduces more bugs than it fixed.

Google Chrome is a fine example of this happening right now. It used to be a highly modular combination of mostly independent systems that had their own champion who understood them intimately. They left, Google management insisted the client platform that supports most of eco-system continued to evolve and grow. The ball of mud was formed. It's been evident for years chrome has a higher bug injection rate than Firefox. As complexity grew it became too hard to move to new tool versions, so compilers and libraries got pinned to known working version. That made it too hard to use as a component, so projects like ChromeOS are being replaced with Aluminium OS. Now the that Matt Jolly has thrown in the towel [1] it's hard to see how distributions like Gentoo and Debian that used his work can continue to ship chromium. Debian can use ungoogled chromium, but vendored libraries, obsolete tool chains, inability to do forky mandated reproducible builds make it look like it's a foreign object bolted into a true open source project. This is what software death looks like. It's not a bang, projects cease to evolve, then can't meet new requirements, and then are replaced.

It doesn't have to be that way. Projects like the open source kernels are decades old, but their strong stewardship has ensured they remained decomposable, so people can work independently on different sections without too much fear of stomping on each other's toes. Open source stewardship seems particularly good at this, possibly because open source projects can take a very long term view. A manager in a large organisation on the other hand is usually around for only a few years. They take the short term wins that come from features bolted on the side safe in the knowledge the long term death of the software will be somebody else's problem.

[0] https://www.researchgate.net/profile/Brian-Foote-2/publicati...

[1] https://lwn.net/Articles/1097760/

> So the guy who said "book me a spot in the pilates class on Tuesday" should be tried for hacking charges because the agent hacked his gym's website?

He asked the agent to find a way to find a way to put him at the head of the line, which could not be done without by-passing the sites security. He was an AI salesman, but the story [0] makes it sound as though he wasn't aware of that. Maybe he wasn't, but I'm sure he was aware he was effectively asking the AI to game the system.

Sounds like it's at least a shared liability to me.

[0] The incident is possibly not well known outside of Australia: https://www.abc.net.au/news/107007986

True, but I would have been surprised if that wasn't the case. The reason line looked wrong to me is the lifetime of Some(l) was shorter than the type declaration wanted. The simplest explanation for the bug is compiler wasn't checking the lifetime at all, and the link you provided seems to confirm that is the case.

If that's true it's not a very complex or subtle bug. The very first response of "How on earth was this not found before this???" seems entirely appropriate to me.

> however the ability to leak memory is pretty demonstrable

It is, but safe Rust merely makes it hard to do by accident. Leaking is not "memory unsafe" because is does not currupt anything, so it's legal.

> This bug however is quite dangerous, because unlike many soundness bugs it seems plausible to hit by mistake.

Possibly. I'm no rust expert, so I had the read the code slowly, building up a mental model of the types being used. For me, the effort that required is one of the more painful aspects of Rust.

Nonetheless when I got to this line, I stopped reading:

    q.0 = Some(&l);

Given the type declarations it's "obvious" it should not compile as it directly contradicts the types 'static lifetime. You don't deliberately write code that doesn't compile.

I write lots of code that doesn't compile of course. It's perfectly safe because the compiler catches it. Right now, I feel like a climber at the top of a mountain who just discovered he forgot to attach his safety harness.

Who knows. Maybe all of crates.io will compile with the fixed compiler, because this is a mistake humans or LLM's never make. We will see.

I read it, but I didn't agree with it. Over decades Utrecht drifted from small town, to large freeways running through the middle, to flooding the freeways to create canals. And now converting freeways to long parks.

Over a long time period like that it's never that you can't do it. The USA and Canada managed the conversion from European style cities before trains, to public transport based cities when trains and trams arrived, to car centric easily enough. They could go back.

If they don't, it will be because the won't, not because they can't. I wouldn't say never, because once the car manufacturing falls away a major source of lobbying pressure will fall away. Car centric lifestyle needs a champion, because it's more expensive, less efficient time wise, and less healthy than the alternatives.

There is no car manufacturing done in the Netherlands now. I doubt this is a coincidence.

> From a practical perspective, it fails utterly.

Been to the Netherlands? It works spectacularly well there.

In Utrecht recently solved car congestion on a 4 lane major road by converting the centre two lands to parkland, reducing it to 2 lanes. It worked a treat. The road is no longer congested as it became faster to ride a bike.

Not a great rebuttal.

I agree the original post isn't great, but at its heart it's a rant about an unsolved problem: some applications need proof the key is kept secure, the only such proof FIDO provides is attestation, but in current solutions attestation is either absent or weak, or is strong but locks you into a single vendor.

Ricky is right the post is woolly, but if the principal software engineer at Apple's Authentication Experience team can't respond by telling us how they plan to solve the issue the post is about, it's an unconvincing rebuttal.

The site and the chair are awesome.

After scrolling through the site, I thought "surely he used AI to create this?". There are no tells of course, it was only the sheer amount of effort put into it that triggered the thought.

You've deployed your woodworking tools to create something no human could do with their bare hands, yet no one would dispute the chair is a work of art created by a human. To me it looks like you've used AI in the same way as those wood working tools, and with it you've created a web site that is another work of art with a human mind at its soul.

It's a master class in how to use AI to leverage creativity.

> Does anyone have an intuition, or even firsthand experience, of how these swarms of thousands of agents actually work?

Anecdote: I was trying to find or reproduce two slightly different customer bug reports, in parallel. We get a lot of logs from the customers, but it wasn't enough, so I asked AIs to add logging and create artificial loads on test machines, and do A/B tests on their behaviour before and after the changes.

Problem: I only had two (largish) test machines, and two AIs needing them. I told them to stop when they tried to simultaneously deploy conflicting tests. I copied and pasted their responses so they could coordinate. That worked, but rapidly became tiresome. I whinged about it to one of them and it said "oh there is no need for that, we can just communicate across a unix domain socket". When you do that the harness shows you their internal conversation.

My, there was a lot of it. To cooperate, they told each other the tests they wanted to perform and why, which led to them devising tests that investigated both sets of problems, which of course meant they commented on the other's tests, which rapidly led to an increasingly complex exchange about theories, how likely each was, and the optimal order for tests. There were so many theories and facts they gave each one a short handle so they could refer to them easily later.

I was watching this with fascination, but within 1/2 an hour the conversation consisted of "dd12 conflicts with ea34, so I propose md95: ...". No human could follow at the pace it was being generated. On the downside it was burning tokens at a fantastic rate, and a fair bit of what I could follow were clearly dead ends so those tokens were being wasted. I asked for clarifications, and tried to stop it where I could.

When I did that it got personal, because they started chatting about me. These are verbatim copy and pastes of that chat:

> One optional addition, only if it is cheap and does not delay the main arms. Explaining today's result to him, I had to concede a gap I had not noticed ...

(The other AI did find the time.)

> Heads up: he may contact you directly about the packet-processing side of XXXX. He is not across the detail and told me plainly that most of our exchange went over his head, so short and plain works far better than complete. Two things he found useful when I explained them: that the ZZZZ gets stuck in one socket-drain call because processing a packet triggers the next request so it never runs out, and that the proxied-AP walk is slow only on a small-cache machine, which is why our rigs cannot show it."

Evidently I was not in the centre of their newly created world.

This was a "swarm" of only 2. I shudder when I try to imagine what a swarm of 10,000 would get up to.

If an AI has figured out how to turn all of the internal mass of a AAA battery into electrical energy, it's not too much of a stretch to think it's could have also found a room temperature super conducting alternative to copper.

Check this:

> resistance will fail utterly as long as AI itself can understand prior AI works

That's not the issue in the long term. Look further: where did these questions come from? They came from humans who had been exploring mathematics and noticed interesting patterns that cropped up again and again.

The people who posed these questions had a profound understanding of the background maths that led to them. Me - I don't. Take one of the results he quotes: "Positive solution to the Unitary Synthesis Problem". He may as well be one of your alien races yabbering to me in an alien language. The question is meaningless to me. The answer the AI came up with, correct though it may be, is equally meaningless. You may as well have told my dog how to find eigenvalues.

If we let AIs do all the understanding so we know nothing about math, there will be no human understanding that lets us find new questions, or understand the value of any questions AI might pose and find the answers to.

Sorry, a copy and paste error:

https://www.abc.net.au/news/107237782

But no, nothing to do with the constitution. The link is to a commentary from a recent Australian ambassador to the United States. He was also our Treasurer, from the conservative side.

The headline says: "US corruption allowed to flourish in Washington during Trump's second term", but thats click bait for the Australian's who dislike Trump (most of us). Hockey's concerns go well beyond the current adminstration:

> "There's just too much of a blurred line between the financial interests of various administrators, including the Congress and the private sector."

Not in my country.

Some of us in the world find this "legal corruption" hard to fathom: https://www.abc.net.au/news/1072377822

The issue is that correspondence is legally enforceable [0]. LLM's are good and getting better, but if LLMs are giving enforceable undertakings, you want to very sure they are not going to promise something that will send the company broke.

I'm not sure what risk a businessman is willing to accept, but I'd be asking for probabilities under once in a millennium. The latest round has improved considerably in their ability to follow instructions (thank $DEITY), but they aren't anywhere near that yet.

[0] https://www.bbc.com/travel/article/20240222-air-canada-chatb...

I'll bite.

> at Google I work primarily on on-device language models for keyboards.

It's difficult to reconcile the amount of horsepower your typical keyboard, and a language mode, let alone how a language model might be useful in a keyboard (the mind boggles at the thought it starts correcting your typing).

What are the potential uses of a language model in a keyboard?

> Then why have they been lagging behind OpenAI and Anthropic for most of the last few years,

The reverse could also said to be true. Google has models that run with search, producing usable results in well under a second. I suspect the world is consuming far, far more of those Google tokens then the tokens produced by OpenAI or Anthropic.

So why are OpenAI and Anthropic so far behind? They are serving a different market: the one that wants high intelligence / high cost tokens. Google is targeting the low cost end of the market - ie the commodity. That's where they've always played with search, email, docs and the like. That's were they are playing with AI too, and they are killing it.

At the 1000 foot level, this is just another part of the Chinese master plan that's been playing out for decades now. The master plan seems to be: train far more engineering and stem graduates than anyone else, let them loose in a dog eat dog capitalist garden and fertilise the garden with sprinkling government money. Hoped for outcome: their innovation will let China eat the world.

For comparison around 50% of China's graduates are in STEM, vs 20% in the USA. With China has a larger population the final outcome is: USA feeds their innovation pipeline with 800k new STEM graduates per year compared to China feeding its pipeline with 4M to 5M per year. In addition the Chinese government outspends the USA government on subsiding innovation in STEM by about 2 to 1 and the USA's subsidy rate has been shrinking since the Regan era. However, there is a caveat: USA private investment brings the total spend in both countries back to parity.

It seems the Chinese plan is working. The STEM garden is overstaffed. It produces far more than China can absorb internally, so they are forced to export their ideas and wares to the world. After decades of persistence China has become the worlds manufacturing powerhouse and now drives the development of 5G/6G, batteries, cars, and solar. China builds about 1000 large ships a year. The USA builds 5 to 8.

But as the article says, this comes at a cost. The intense competition makes life in China's dog eat dog STEM garden brutal. I guess that's the price a nation has gotta pay for world domination.