Perseids

Trust relationships are fundamental even in the most abstract view of the Web PKI and without the band-aids of Certificate Transparency etc. that trust relationship is: Some random commercial entities that are completely independent of the DNS, make fully trusted statements about the validity of DNS record ownership. It's bonkers to the core. If someone would design it from scratch, of course the domain registrars would be the ones issuing the certificates via their established relationship to the customers, and they would only be able to issue them for the domains they are responsible for. And no: DNSSec does not solve the same problem and is a tragedy all of its own.

In addition to what my sibling comments have said, there is also the advantage of pushing the ecosystem to safe and fast approaches as a whole. Imagine if the default was to run C/C++ software under Fil-C, if no other measures where taken by the developers, e.g. if the consensus interpretation for one of the current security regulations (NIS2 etc.) mandated it. There would then be an enormous incentive for software to provide faster alternatives to Fil-C, as there where 2x-4x speedups for the taking!

Come to think of it, I believe it would actually be a good idea to lobby for exactly that. Memory safety issues have been a plague in IT for far too long.

I'm of two minds about this. On the one hand, you get miraculous results with AI, and that is what you are referring to with "be able to {my-field}". On the other hand you regularly get useless, if not toxic, crap out of the miracle machine. If you work with AI, you know these moments: "no that is not what I meant at all!" [1] or "how could you have already forgotten the central motivation I laid out just ten back-and-forths ago?". We're calling this the jagged frontier, but the jaggedness itself is not the problem, it's how deep these cuts go into aspects that are important. These are, why AI is still not "obviously superior" in its abilities, can't replace mathematicians and software developers and lawyers and everyone else who is primarily working with text. (It's also why people say it is ruining their fields, instead of replacing them in their fields, because it only replaces some part of the work, and what's left is not tenable or pleasant (e.g. doing reviews all the time).)

Over all, I'm not quite sure whether I see these deep cuts occurring less and less with progressing AI. It might just be, that I'm giving them progressively harder tasks as well. But the fact of the matter still is that they aren't there yet and it needs proof they will get there ("there" being able to do competent work that is not strongly limited in their scale by human review and correction). So I'm still not convinced about the point you want to make, even though I see AI disrupting about every part of society, and am hoping for the benevolent super AI to lead us to utopia (and afraid of the cliff of human dictatorship on one side and extinction on the other).

[1] I've started to use these moment to reflect on whether a human in a comparable situation would fare better. Having only a few text inputs and small glimpses into a code base and knowing next to nothing about you or your project personally. Quite a few times I've concluded: No, they would not. But more than 50% they would, and also, these limitations are technological limitations of LLM, so you should probably not excuse them for that.

> I find this unconvincing. Isn't this the exact same security posture as a native app with automatic updates turned on?

Mostly. But the alternative shouldn't be a native app that updates itself, but rather a native app that is updated by a package manager. The big difference is traceability and auditability: With a trusted package manager (and even in good app stores) the company can only decide to push an update to everyone or to no-one. There is no way to push an update just to the pesky journalist or whistle blower. A covert attack is really hard to accomplish this way.

Human languages are a mess. Some mix left-to-right and right-to-left. Some have variable amount of diacritics. Some have non-well-defined character sets where just trying to map it to a manageable complexity still loses cultural heritage. When you are fondly remembering the good old times of ASCII, you just wish back to be able to ignore anyone not speaking English. (Which is in your right to do, if you so desire.)

You seemed to be deeply confused about encodings and character sets.

> The fact that unicode maps the lower 7 bits to its own character set is a nice touch but none of the unicode sets are plain text.

That is only true for the mapping of Unicode character (code point to be exact) to UTF-8, which is an encoding of Unicode characters.

> Unicode are multibyte characters with variable byte length and endianess at play.

That is only true of the UTF-16 encoding. UTF-8 does not have endianess, UTF-32 does not have variable length per code point.

None of that is true for Unicode, because it is abstracted away from any byte representation. Furthermore, getting back to the start:

> ASCII (7-Bit) is the only widely understood charset there is. Everything beyond this point depends on the loaded charset.

ASCII also depends on how you try to decode your text. If you interpret two bytes as one character, ASCII will never be correct. There is nothing magical about one byte mapping to one character. I'd even argue that the only reason ASCII support is so universal internationally is because of UTF-8. Otherwise many countries would default to encodings where ASCII is not a subset (as they did before UTF-8 became common). So IMHO, UTF-8 and Unicode are the only widely understood encoding and character set.

How about https://ai-2027.com/ ? Don't look at the specific years (they are on the extreme low end IMO), but at the story.

If you don't believe in international conflict as a driving scenario, instead think about simple human greed and hubris. Citing myself:

> If AI gets intelligent enough, it will be incredible useful to connect to real world machinery. Think about how much cheaper building houses could be, if all the labor would be close to free. In general, dirt cheap, competent and abundant labor would revolutionize all parts of the economy. People are already trying out near autonomous AI companies today. When AI gets intelligent and cheap enough, no human-led company can compete with AI-led companies. When AI gets competent enough with real world interactions, human blue collar work can't compete. Imagine economic growth not in the single digits, but 80% or 300%. Countries not participating in (reckless) AI growth will quickly be left by the wayside. At this point, we don't even need to allure to military concerns to see how human oversight gets sidelined.

> All of this is only ("only") contingent on sufficiently intelligent and cheap AI. If you don't accept this premise, the rest doesn't follow. (There are multiple arguments, why this could be, but that is another discussion.)

> If you accept the premise, how would AI 'extinct' humanity? With 99%+ of the economy under AI control, the possibilities are endless. And given its enormous GDP, cheap to accomplish. Probably even for a single AI company in the above scenario. Killer drones? Engineered virus? Poisoned water supply? Let your creativity run wild. You just need an entity that is persistent and well-resourced to reach every last human settlement.

> The why is a question about alignment (and out of scope of this comment). As a simple comparison, humans are only mildly aligned with preserving nature. It takes up so much space, protecting it takes an annoying amount of resources, etc.

https://news.ycombinator.com/item?id=49624360

This debate is so broken. AI "sceptics" say: OpenAI should be punished for hacking, because there are no rogue AI agents. AI "believers" say: OpenAI should be punished for hacking, because their agents went rogue. Both argue with each other whether agents went rogue. Can't we unite behind "OpenAI should be punished for hacking"?

> Imagine having a virus escape a sandbox, why are we worried about the virus but not the incompetency of those who are responsible for setting up the sandbox?

> Should we worried about people using LLMs for attacks? Yes, but not in the premise of LLMs going rogue but someone with the intention of abusing it to cause harm.

[Why-not-both?-meme]. To use your example, when you discover prions (a class of pathogen that is much more robust to standard disinfection methods than viruses) you should both be worried about your concrete outbreak of BSE (UK in the 80s and 90s) as well as the wider implication (e.g. do we need to change the sterilization methods for our surgical instruments?).

Seriously, I find the way these discussions are done to be super frustrating, because often people implicitly form tribes that oppose everything the other tribe says. When someone believes AI companies push greatly exaggerated stories of dangerous rogue AI to force out competition via regulation they often implicitly conclude that their argument is fundamentally wrong, whereas in reality the lies that work best are those that distort the truth.

Companies should be punished harshly for the deeds of their AI agents AND we should not allow them to force out competition AND we need take the threat of autonomous AI agents as a new class of danger serious AND we need to worry about the socioeconomic implications of AI companies privatizing new means of production.

Yes, there is competition of these ideas in the attention of the general public, but the methods we can use to solve these problems don't compete with each other. AI slowdown for example helps with all the other topics.

As I've written above, I've found https://ai-2027.com/ to be a compelling description of how this could happen.

In my words: If AI gets intelligent enough, it will be incredible useful to connect to real world machinery. Think about how much cheaper building houses could be, if all the labor would be close to free. In general, dirt cheap, competent and abundant labor would revolutionize all parts of the economy. People are already trying out near autonomous AI companies today. When AI gets intelligent and cheap enough, no human-led company can compete with AI-led companies. When AI gets competent enough with real world interactions, human blue collar work can't compete. Imagine economic growth not in the single digits, but 80% or 300%. Countries not participating in (reckless) AI growth will quickly be left by the wayside. At this point, we don't even need to allure to military concerns to see how human oversight gets sidelined.

All of this is only ("only") contingent on sufficiently intelligent and cheap AI. If you don't accept this premise, the rest doesn't follow. (There are multiple arguments, why this could be, but that is another discussion.)

If you accept the premise, how would AI 'extinct' humanity? With 99%+ of the economy under AI control, the possibilities are endless. And given its enormous GDP, cheap to accomplish. Probably even for a single AI company in the above scenario. Killer drones? Engineered virus? Poisoned water supply? Let your creativity run wild. You just need an entity that is persistent and well-resourced to reach every last human settlement.

The why is a question about alignment (and out of scope of this comment). As a simple comparison, humans are only mildly aligned with preserving nature. It takes up so much space, protecting it takes an annoying amount of resources, etc.

Let me argue on a technicality first: None of these are extinction level events. If global warming disrupts 99% of all crop production, the remaining 1% is still plenty enough to sustain a stable, if miserable, population. In fact you just need about 5k people for a stable gene pool[1]. Of the classical threats, only bioweapons got a shot at extinction, but even that is hard, given the (few) remaining truly secluded settlements.

But most probably care less about human survival, and more about survival of civilization.

In this regard the strongest argument for pushing AI safety is that it is cost-effective. The climate crisis has near 100% likelihood of doing incredible harm to humans, the economy and our ecosystem. Solving the issues behind are also incredible complicated and are a multi decade coordination effort of restructuring the way most of our infrastructure and production works. The AI apocalypse might have a small likelihood of occurring, but could dwarf any issue we have encountered so far. All we have to do to significantly reduce the danger is to negotiate the equivalent of a nuclear weapons control treaty which would reduce the bottom line of … what? Ten significant companies world wide?

Its akin to discovering you are seriously ill and have a 30% chance of dying and the treatment costs 10k bucks vs. having a 1% chance of dying and the treatment costs a cent. In both cases you should pay obviously pay for the treatment (given western levels of wealth).

To be clear, above I'm presenting the argument I find most persuasive for AI control / AI slowdown. Personally, I'm beginning to fear of much higher likelihoods for catastrophic AI, given how incredible irresponsible major players like OpenAI have turned out to be. If you can't imagine how this might come to be, read https://ai-2027.com/ . It's a concrete story of how this could play out, and sometimes stories are more convincing than abstract arguments. Don't let yourself be hung up on the stated dates, though, the moral is identical if you stretch the timeline.

[1] https://www.youtube.com/watch?v=1qIvFpcGkqc

That would be the FernUni Hagen: https://www.fernuni-hagen.de/english/

> The FernUniversität in Hagen is Germany’s only state distance-learning university, and its largest in terms of student numbers. Its flexible degree programs and continuing education options make higher education accessible to students in a variety of life situations. Its five faculties conduct a wide range of theoretical and applied research in their subject areas of Humanities and Social Sciences, Psychology, Mathematics and Computer Science, Economics and Business Administration, and Law.

I believe some exams have to be taken non-remote: https://www.fernuni-hagen.de/english/study/distance-learning...

You're making a good point. I was also of the opinion that these books don't land well, because I'm missing the context. And I'm actually still thinking that about quite a lot of these classics. But what I believe I'm also concluding from your comment, is that these books aren't good hooks to get people, especially teenagers, interested in the context and in the society of the time. And that is where YouTube/TikTok/TV might shine. Or a well written and narrated podcast.

Hoping to not sound like a broken record, this is why having full ownership of your device and OS is important. My stock Pixel Android recently told me something along the line of "A security policy blocks screenshots for this app. Talk to your administrator if you want to change the policy". I looked in the mirror and my administrator said "time for a policy change, we're moving to GrapheneOS".

And just as important: Help your friends and family to move as well, so they can have ad blockers, NewPipe etc. We need a critical mass of users invested in their freedom, otherwise its going to be crushed by malicious/dumb security measures of their banking apps, corporate greed ("oh, a simple misunderstanding, when you clicked 'buy' you rented a limited license. Did you not read the ToS?") and police overreach. It's a perpetual battle.

> The bike with the torque sensor behaves a lot more like what you're used to from riding a conventional bike

I agree, but would like to point out that sometimes this might not be what you want. I had a bike with a front motor and a cadence sensor that I used for my commute for a few years, and was very happy with the motor just pulling me along while I lazily pushed the pedals with ~zero torque, barely enough to let the bike know I'm there. Basically like pressing a throttle. If all you want is to get from A to B comfortably, I've found this to be superior to a torque sensor, and it was also dirt cheap.

The thing is, I explicitly wanted to not focus on current legality, because I believe this is leading us astray. This is about how we want to shape our society, not about how their actions fit into a legal system that wasn't designed to handle AI training. If, by chance, all of this was legal, the same arguments would still apply. For example, see how Adobe is training their image generating AI on licensed work. The result -- privatized intelligence -- is the same as when illegitimate training data would have been used. Or, say, an AI company makes a deal with all the publishers to license their books for training. Or a court decides that training AI with MIT licensed code is totally fine. Or an AI company pays legions of software developers to train their AI. All of this still ends in a power grab. All of this still privatizes knowledge and skills accumulated over centuries, if not millennia.

I think they are doing us a disservice by perpetuating this focus on intellectual "property" [1] in regard to AI.

The real issue with Anthropic, OpenAI etc. is not that they have used all of our public knowledge for training their LLMs. Creating new work from old and learning from prior generations is what we all do. The issue is that they want to claim all of the benefits for themselves. They are standing on the shoulders of giants and have contributed an inch themselves, yet want to privatize the power of the whole giant. We shouldn't let them "own" these models.

The influence on society by AI is so novel that it's reasonable to craft new laws specifically for them. There are a lot of ways to deal with their power grab. We could force them to open source the models after two years. Or we could tax tokens or compute. We just need to agree that the power grab is the problem, the privatization of our cumulative knowledge, and not some details about copyright infringement.

[1] I know I'm going to risk dissent just by putting quotation marks here. But I think for this topic specifically it is crucial to understand that intellectual property is an arbitrary social/legal construct. With physical stuff, there is an inherent scarcity. If you steal my smartphone, I no longer have it. If you steal the character from my book, I... have a harder time selling my next book? Our ancestors have invented copyright to solve a specific problem, but the solution has become perverted over time. There are a lot of egregious cases out there (looking at you, Disney), but even relatively tame success cases don't look good. Society has paid J.K. Rowling a literal billion for her work and still this cultural touchstone of a generation remains privatized. Imagine what other authors could have build upon her stories, if only they were allowed to publish their own stories with these characters. She has not been a particularly good steward in the past decades.

Adding to my sibling comments: It also has been conquered in some sense of the word. There are effective vaccines for the 9 most problematic HPV types: https://en.wikipedia.org/wiki/HPV_vaccine . It is worth getting, regardless of your gender.

I'm usually not one to focus on technological solutions given sociological problems, but this one seems to be a good exception. If we "just wanted to" [1] all this fake calls could be stopped by requiring strong authentication/authorization. We are very much used to just anybody being able to call my number, but that doesn't need to be the case. At the very least, cold calls should be treated as skeptical in the UI as instant messengers like Signal treat first messages. Probably this isn't enough, though, as it wouldn't have prevented the cases described in the article. Cold calling someone should probably require the caller to be traceable to a real, government-ID-verified person [2]. Even if that person is being defrauded themselves ("get a thousand bucks by installing this app and clicking a few screens") is would destroy the economics of the attack, as it would make each call expensive again.

[1] Structural inertia is the killer here. It will certainly not happen until the problem is huge enough.

[2] Exceptions can of course apply to numbers that are meant to primarily be cold called, like doctors offices. The callee possibly have to be specially trained to withstand this kind of attacks.

> It’s only expensive and brittle because environmentalists have choked it to death. They’re the third biggest villains of climate change, after consumers and oil companies.

Do note, though, that it was the unbelievable irresponsibility of past operators that has spurred the anti-nuclear movement in the first place. See e.g. https://youtu.be/929B8sgOOTM?si=FttZr_MsbQ1hB4Nj&t=1664 from 27:44 to 31:35.

And as a third answer: You actually don't need to take the flywheels of the power plants offline, when you take the rest of the power plant down. The locations for them are perfect, as the whole grid has historically been build around them.

The limit is not how cold the water needs to be for the power plant, but how hot it can be for the river ecology to not take (too much) damage. Even in Germany power plants regularly need to be shut down.

> 2022 was another consecutive year in which water levels of major European rivers – such as the Rhine, the Danube, and the Rhône – were dangerously low and the water temperatures very high. This caused severe problems for the operation of nuclear power plants across continental Europe. Energy companies in France, Germany, Switzerland, Belgium and elsewhere had to shut down their nuclear power plants partly or fully because there was not enough cooling water available or, more commonly, because the cooling water that was returned to the river became too warm (Barber, 2022; Limb, 2022; Miller and Vladkov, 2022). Environmental regulations, designed to protect the riverine flora and fauna as far as possible, stipulated that nuclear power plants were not allowed to release cooling water above a certain temperature (European Parliament and European Council, 2000; IKSR, 2022b). The resulting unplanned outages — and the efforts by nuclear operators to avoid such disruptions — highlight pressing concerns about the sustainability of nuclear energy, particularly its impact on river ecosystems in an increasingly warming world (see Fig. 1).

https://www.sciencedirect.com/science/article/pii/S030142152...

You are very conservative with the upper limits, probably because you are limiting yourself to medicine. With bioengineering the upper limits are hard to grasp. Why build a house, when you can grow one. Re-imagine all machines as custom-made biology. Why upload your conscience to silicon, when your body can be anything, your brain can experience anything, you brain can be reshaped to be anything.

How much is possible there, is only constrained in our understanding of biology. How difficult that turns out to be for a super intelligence… who knows? If we are actually on the cusp of the AI singularity, the future is going to be weird and/or wonderful and/or horrible, but definitely unimaginable different than today.

My mind is kind of shooing away from this intuitively. Too hard to believe. My whole life experience has been living in a different world. But imagine if "we" could actually create human level intelligence, say, for the price of a 100k USD/EUR/GBP. It could only do knowledge work, of course, but it would easily pay for itself and thus be mass produced. What is the market cap for cheap knowledge work? I would be surprised if it is only a billion human-equivalents, given humans find new creative ways to pay for themselves all the time. That explosion alone is mind-boggling and it does not rely on super-intelligence.

All of this should make one point clear: At no other point in history has it been more important to have our power structures be aligned with the interests of society.

But why? I'd understand (though not approve) them tightening down everything about the car firmware to the max. They are responsible for the app, sure (it's a "digital element"), but they aren't responsible for the OS the app runs on. The CRA should not be used as an excuse to enact stupid restrictions.

Yes and that would be completely correct in many cases.

> Almost like there are aligned interests there rather than a purely adversarial relationship.

You might very well be the exception, but for something like 99% of marketing content that reaches me, our interests aren't aligned. First of all, they want to generate "needs" where there weren't any before and probably shouldn't be. A pizza ad produces the wish for unhealthy food. A fashion ad produces the wish for new clothes (even though I have enough) and probably even changes the societal dynamics of individual expression and personal style to be consumption oriented.

Second, even if I have a legitimate need for a solution, they still want me to buy their product, consume their media, give my attention to them. I, on the other hand, want to be informed by a neutral third party about the pros and cons of some product. Sure you can say "but unhappy customers are bad for us", but there are actually very few niches, where this signal is powerful enough to align incentives, because information and power asymmetry limit the customers understanding of product quality and their leverage to correct harmful market dynamics.

All of their issues are self-inflicted. What benefit is there to their cloud backend except getting around the home NAT? If you want to build your IoT product privacy-friendly, your cloud offering can be reduced to a STUN/rendezvous server and a proxy server as fallback [1]. Ship your devices with individual tokens to rate limit the proxy, have the STUN/rendezvous/proxy server address configurable and publish their source code for users to not be dependent on your continuous operation.

You can even go so far and have a public sub domain for each devices ( serialnumber.manufacturer.com ) which you only operate as a dumb proxy so that even the TLS certificates are negotiated end-to-end between the IoT device and Let's Encrypt. (The devices connect to your backend via Wireguard and you rate limit with their device individual key, whose public key you read out during the end-of-line production step.)

Hell, with today's browser heavy applications you can even run the whole slicer in the browser. Let the app be distributed via CDN so the code does not need to go through the proxy.

[1] In the case of non-battery operated and always or mostly on devices, like 3d printers at least.

Do try to follow the advice of my sibling comments, but its also okay to find out you are simply really bad at remembering names. I think I'm in the bottom 10% percent in that regard. The only way I can somewhat manage to remember the names of the people I would like to is to use Anki (spaced repetition) on a semi-daily bases. This comes down to what others would consider a crazy amount of work, but at least it is somewhat successful. It frustrating for the long tail of people I might not meet again, but where it still would be really helpful to know their name. Where I really fail is situations that don't allow me to write down names shortly after they were used, which is often the case in introduction rounds. Trying to constantly repeat all names in my head means I'm missing on the other stuff people say.

A side aspect of this drama is the root feature which enabled this bug:

> ugh sorry this was a bug with the 3rd party harness detection and how we pull git status into the system prompt

Claude wants to exercise control of how I use the "inclusive volume" that I purchased with my monthly subscription. This harms competition (someone else could write a more efficient or safer coding agent) and is generally not in the best interest of society. Why do we allow this?

This specific case is interesting, because it is so clear cut. There is no cross financing via ads, they already have the infrastructure to measure usage and even the infrastructure to bill extra usage. I also don't see how you can plausible make the argument that restricting usage to their blessed client is necessary for fair use or for the basic structure of their business model (this would be the standard argument for e.g. Youtube: Purposefully degrading the experience of their free client to not support background playback enables the subscription model).

This is the important point. You need the right to not be discriminated when you withhold your consent, otherwise your consent is effectively meaningless, as it is forced on you by your impossible bargaining position. This is one of the central pillars of the GDPR without which it wouldn't work at all. Be advised to make asking customers for consent that doesn't directly benefit them illegal as well, lest you risk creating another wave of malicious cookie banners.

> This is like saying we should have halted all RSA deployments until improvements in sieving stopped happening.

Absolutely not. If people were advocating for ECC only, you would have a point. But this thread is about hybrids vs ML-KEM-only (for key exchange!). Everybody here wants to deploy the algorithm your favoring and wants to deploy it now, just not without a safety net.