Hacker News Reader

Comment by ohelno | Hacker News Reader

> it may be very difficult for us as users to discern which model is better

But one thing will stay consistent with LLMs for some time to come: they are programmed to produce output that looks acceptable, but they all unintentionally tend toward deception. You can iterate on that over and over, but there will always be some point where it will fail, and the weight of that failure will only increase as it deceives better.

Some things that seemed safe enough: Hindenburg, Titanic, Deepwater Horizon, Chernobyl, Challenger, Fukushima, Boeing 737 MAX.

View on HN

Don’t malign the beautiful Zeppelins :(

Titanic - people have been boating for two thousand years, and it was run into an iceberg in a place where icebergs were known to be, killing >1500 people.

Hindenburg was an aircraft design of the 1920s, very early in flying history, was one of the most famous air disasters and biggest fireballs and still most people survived(!), killing 36. Decades later people were still suggesting sabotage was the cause. It’s not a fair comparison, an early aircraft against a late boat.

Its predecessor the Graf Zeppelin[1] was one of the best flying vehicles of its era by safety and miles traveled, look at its achievements compared to aeroplanes of that time period. Nothing at the time could do that and was any other aircraft that safe?

If airships had the eighty more years that aeroplanes have put into safety, my guess is that a gondola with hydrogen lift bags dozens of meters above it could be - would be - as safe as a jumbo jet with 60,000 gallons of jet fuel in the wings. Hindenburg killed 36 people 80 years ago, aeroplane crashes have killed 500+ people as recently as 2014.

Wasn’t Challenger known to be unsafe? (Feynman inquiry?). And the 737 MAX was Boeing skirting safety regulations to save money.

[1] https://en.wikipedia.org/wiki/LZ_127_Graf_Zeppelin

> Wasn’t Challenger known to be unsafe? (Feynman inquiry?). And the 737 MAX was Boeing skirting safety regulations to save money.

The AI companies have convinced the US government that there should be no AI safety regulations: https://www.wired.com/story/plaintext-sam-altman-ai-regulati...

Guarantee we'll be saying this about a disaster caused by AI code:

> everyone knows you need to carefully review vibe coded output. This [safety-critical company] hiring zero developers isn't representative of software development as a profession.

> They also used old 32b models for cost reasons so it doesn't knock against AI-assisted development either.

I'm particularly salty about the Hindenburg and don't feel as strongly about Chernobyl, Fukushima, Challenger, so if you're referring to those, that's different. The Hindenburg didn't use Hydrogen for cost reasons, it was designed to use more expensive Helium and the US government refused to export Helium to Nazi controlled Germany, so they redesigned it for Hydrogen. I'm not saying that it wasn't representative of air travel at the time, I'm saying air travel at the time was unsafe and airships were well known to be involved in many crashes, and the Hindenburg was not particularly less safe, it's just that aeroplanes were much smaller and carried fewer people and the accidents were less spectacular so they somehow got a pass and aeroplanes were . I'm saying air travel became safer and so would Zeppelin travel have become, by similar means - more careful processes, designs improved on learnings from previous problems, etc.

Look at the state of the world today, AirBus have a Hydrogen powered commercial aircraft[1]. Toyota have Hydrogen powered cars on the streets. People upload safety videos to YouTube of Hydrogen cars turning into four-meter flamethrowers as if that's reassuring[3]. There are many[2] Hydrogen refuelling gas stations in cities in California where ordinary people can plug high pressure Hydrogen hoses into the side of their car and refuel it from a high pressure Hydrogen tank on a street corner. That's not going to be safer when it's a 15 year old car, a spaced-out owner, and a skeezy gas station which has been looking the other way on maintenance for a decade, where people regularly hear gunshots and do burnouts and crash into things. Analysts are talking about the "Hydrogen Economy" and a tripling of demand for Green Hydrogen in the next two decades. But lifting something with Hydrogen? Something the Graf Zeppelin LZ-127 demonstrated could be done safely with 1920s technology? No! That's too dangerous!

Number of cars on the USA roads when Hindenburg burnt? Around 25 million. Now? 285 million, killing 40,000 people every year. A Hindenburg death toll two or three times a day, every day, on average. A 9/11 every couple of months. Nobody is as concerned as they are about airships because there isn't a massive fireball and a reporter saying "oh the humanity". 36 people died 80 years ago in an early air vehicle and it's stop everything, this cannot be allowed to continue! The comparisons are daft in so many ways. Say airships are too slow to be profitable, say they're too big and difficult to maneouvre against the wind. But don't say they were believed to be perfectly safe and turned out to be too dangerous and put that as a considered reasonable position to hold.

Some of the sabotage accusations suggested it was a gunshot, but you know why that's not so plausible? Because you can fire machine guns into Hydrogen blimps and they don't blow up! "LZ-39, though hit several times [by fighter aeroplane gunfire], proceeded to her base despite one or more leaking cells, a few killed in the crew, and a propeller shot off. She was repaired in less than a week. Although damaged, her hydrogen was not set on fire and the “airtight subdivision” provided by the gas cells insured her flotation for the required period. The same was true of the machine gun. Until an explosive ammunition was put into service no airplane attacks on airships with gunfire had been successful."[4]. How many people who say Hydrogen airships are too dangerous realise they can ever take machine gun fire into their gas bags and not burn and keep flying?

[1] https://www.airbus.com/en/innovation/energy-transition/hydro...

[2] https://afdc.energy.gov/fuels/hydrogen-locations#/find/neare...

[3] https://www.youtube.com/watch?v=OA8dNFiVaF0

[4] https://www.usni.org/magazines/proceedings/1936/september/vu...

> Decades later people were still suggesting sabotage was the cause.

Glad you mention it. Connecting back to AI: there are many possible future scenarios involving negative outcomes involving human sabotage of AI -- or using them to sabotage other systems.

Hindenburg indeed killed hydrogen blimps. Of everything else on your list, the disaster was in the minority. The space shuttle was the most lethal other item -- there are lots of cruise ships, oil rigs, nuke plants, and jet planes that have not blown up.

So what analogy with AI are you trying to make? The straightforward one would be that there will be some toxic and dangerous LLMs (cough Grok cough), but that there will be many others that do their jobs as designed, and that LLMs in general will be a common technology going forward.

I have had gemini running as a qa tester, and it faked very convincing test results by simulating what the results would have been. I only knew it was faked because that part of the code was not even implemented yet. I am sure we have all had similar experiences.

Reply on news.ycombinator.com