Comment by ThrustVectoring | Hacker News Reader

Comment by ThrustVectoring | Hacker News Reader

ThrustVectoring

The second half of the article that talks about Yudkowsky, rationalists, and Roko's Basilisk is absolute trash and a gross mischaracterization of the actual positions taken.

>Yudkowski has for more than a decade pursued the possibility of perfect human reasoning

His website is literally called "Less Wrong". It's fundamentally a quest for improvement, not perfection.

>His system of coldly logical reason, it turned out, was by many accounts completely undone by a logical paradox known as Roko’s Basilisk.

Roko's Basilisk is a thought experiment designed to import your "don't negotiate with terrorists" intuitions into a really weird corner of decision theory. It's possibly why prior decision theories held were rejected as flawed, but it's not a current issue with their logic. Roko's Basilisk has, unfortunately, gotten way more coverage and fame than it deserves in terms of actual importance. This is because of the regretful (but understandable) decision to censor discussion of it on Less Wrong, which backfired spectacularly.

>For super-nerd bonus points, it’s also arguably a spin on Godel’s incompleteness theorem, which argues that no purely rational algorithmic system can completely and consistently model reality, or prove its own rationality.

That's not at all what Godel's incompleteness theorem argues. That's much more narrowly about formal logic.

> "This is because of the regretful (but understandable) decision to censor discussion of it on Less Wrong, which backfired spectacularly."

Can you explain the (understandable) rationale behind Less Wrong censoring discussion of Roko's Basilisk? I've been lead to believe it was censored out of fear that discussing it could somehow inspire its actual creation.

Honestly from a distance Less Wrong has always struck me as vaguely cult-like, but I'm open to the possibility I've just gotten the wrong impression.

I think the more charitable interpretation is that the discussion was taking over everything. If people are fixating on a narrow topic for a long time (without much progress) then sometimes it's helpful to say "alright people, let's move along now."

You're right about Less Wrong feeling like a cult, though. It gives me the heebie-jeebies.

>You're right about Less Wrong feeling like a cult, though. It gives me the heebie-jeebies.

They even have a succession plan similar to finding the next Dalai Lama:

https://www.youtube.com/watch?v=d9_2qhkWHN8

Sure thing. The basilisk can easily be misread as an argument to do everything in one's power to support AI research of some sort. It's more accurately an argument that you should ensure your decision theory doesn't coerce you into putting all your effort into supporting AI research. LessWrong has a bunch of really scrupulous people reading it, who take pride in taking ideas seriously, so censoring the discussion seemed worthwhile for avoiding the failure mode of having readers make the mistake mentioned previously.

The entry on the less wrong wiki has some info on it:

https://wiki.lesswrong.com/wiki/Roko%27s_basilisk

Cults don't write about how to avoid being a cult, while the LW crowd has reams of texts and blog posts about how they are purposefully avoiding trying to become a cult.

Cults do talk quite a bit about why they're not a cult.

But I haven't seen cults talking about how to personally ensure you're not falling into trap of cult-like thinking. LW is a rather self-conscious community.

Sorry for the tangent, but from the outside I've also felt very "outside" the LW community. I'm not exactly sure how to "join" it. I know that I'm supposed to read the various required readings first, but then what? How do I see what LW is "talking about" or "fighting about" at a point in time? I mean this in a serious, logistical sense. Thanks!

The LW community has decentralized and deformalized things to a huge extent. It's not just you - the logistics of participating are much less clear than it used to be, and the whole system is far less legible.

Honestly the best strategy is attend local meet-ups, come across as intelligent and interested to the people who hold gatekeeping power over the informal non-public groups, and get in the loop that way. This is significantly easier in Berkeley, where a lot of folks end up moving to. You can also skip some of these steps by attending a CFAR workshop, which will both introduce you to many members of the LW community and put you on an alumni-only mailing list which houses a chunk of the general discussion.

I'm out of the loop now, but it used to be like everywhere else (HN included) - you follow the posts and participate in comments, and after a while you suddenly find yourself to be an established community member.

Some pretend to, though.

https://en.wikipedia.org/wiki/New_Cult_Awareness_Network

They even have an essay about _cultish countercultishness_: https://www.lesswrong.com/posts/gBma88LH3CLQsqyfS/cultish-co...

I think some of the feeling of cultishness might stem from a homogeneity of expression, but it kinda makes sense for a community of people dedicated to being rational -- after all, two rational agents with identical priors cannot "agree to disagree" in the mathematical sense. The community would tend to converge on common "correct" things and would have similar rebuttals for "incorrect" things, right?

Have they ever addressed the criticisms of rationality, technological rationality and instrumental rationality of the Frankfurt School in the 50s and 60s? Or, more generally, the critique of positivism and the collapse of rationality to facts connected with formal logic? These are fairly big topics in philosophy, so it would seem really naive if they haven't covered them.

I'm not personally familiar with those critiques and I haven't exhaustively read anything close to all the content from LessWrong, but I can point out that Yudkowski himself has a number of criticisms of Traditional Rationality. I think the most helpful thing I can add is that the paradigm embraced at LW is Bayesian rationality and not pure formal logic. Being massively reductionist, it seems to boil down to updating your expectations to be more in line with reality -- that being less wrong is about having an accurate account of your uncertainty of the beliefs that you hold, taking into account your priors, inherent bias, model complexity, etc.

This just seems like a very strange and restrictive way to model real-life problems, and it just seems to export the meat of the problem (taking into account priors, what exactly is bias, and if it's possible to be aware of it); for instance, I imagine that many people who subscribe to LW thought are probably uncritical of the ideological function of liberal democracy, for instance - or rationality being used to justify the status quo. These sort of things were taken up by critical theory, so there seems to be some overlap between critical theory and what it is to be rational.

I fear that a faith in mathematical reasoning for all problems can easily make you come out the other end thinking your opinions are mathematically correct, with minimal reflection on core issues such as the applicability of classical logical models or a mechanistic view of how people do (or ought to) behave - this kind of uncritical historically divorced reflection has not fared well in the past, with even economists admitting their model of the economic man is as much fiction as their worldwide Robinson Crusoe from a century earlier.

I don't fuck with LW, but SCC definitely has that culty vibe.

Overall I think Scott has some interesting articles and the general idea of rationalism is .. fine. But that's not at all what the communities around it are like. You just get tons of 'rational racism' with HBD (human biodiversity for those not in the know, which is basically just race realism with a nerdy veneer), IQ worship, and a group that collectively abandons anything like the rationalist ideal and becomes a post-hoc clusterfuck.

The community can almost be thought of like these racist AIs: garbage in, garbage out. They cling to any bit of rational-sounding data that supports their preconceptions, and have to adopt severe anti-academic stances to reject the mountains of data that contradict their views.

I suppose the deeper rot is that these rationalists fancy themselves like scientists, able to interact with primary data and draw conclusions, but any actual expert (i.e. someone that has performed research and published it in a community of peers, not whacko journals) is just confused how anyone could be so wrong. You regularly see posts where someone comes in who actually knows what they're talking about, authoritatively shows that all the nonsense they've been talking about makes no sense, and at best they get a 'huh, interesting' before the nonsense picks back up.

The idea of a rationalist community is to organize around questioning your own assumptions, confronting diverse views, and engaging in good-faith debate. But the reality is that basically none of that happens for anything controversial to the community, so it's just a veneer of legitimacy for in-group circle-jerking. And it just so happens that the in-group tends to be pretty damn racist.

My guess would have been that by discussing it, you leave a trace in the networks allowing an eventual AI to prove that you knew about it. Since you knew about it, it can hold you accountable, and take revenge on all of your descendants if you didn't devote sufficient energy on bringing the AI into existence.

Read Eliezer himself, 4 years ago: https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibo...

The "meat" of interest here for those too lazy to click (I encourage people click through, though, really, and maybe even take a glance at https://arxiv.org/abs/1401.5577) is probably this long parenthetical:

> (But taking Roko's premises at face value, his idea would zap people as soon as they read it. Which - keeping in mind that at the time I had absolutely no idea this would all blow up the way it did - caused me to yell quite loudly at Roko for violating ethics given his own premises, I mean really, WTF? You're going to get everyone who reads your article tortured so that you can argue against an AI proposal? In the twisted alternate reality of RationalWiki, this became proof that I believed in Roko's Basilisk, since I yelled at the person who invented it without including twenty lines of disclaimers about what I didn't necessarily believe. And since I had no idea this would blow up that way at the time, I suppose you could even read the sentences I wrote that way, which I did not edit for hours first because I had no idea this was going to haunt me for years to come. And then, since Roko's Basilisk was a putatively a pure infohazard of no conceivable use or good to anyone, and since I didn't really want to deal with the argument, I deleted it from LessWrong which seemed to me like a perfectly good general procedure for dealing with putative pure infohazards that jerkwads were waving in people's faces. Which brought out the censorship!! trolls and was certainly, in retrospect, a mistake.)

Edit: this plus more is all covered by the LW wiki's page someone else linked, check that out too if you really care. https://wiki.lesswrong.com/wiki/Roko%27s_basilisk

Also the neoreactionary bit. I literally paused on that in total surprise. As a long-time LW lurker - yes, the rationalist community plays with plenty of non-mainstream ideas and try to work them in both directions (up from first principles, and towards extreme conclusions). But characterizing rationalists as equal to neoreactionaries? That's a complete mischaracterization of the community.

If you don't see the NRs in the rationalist community, you're not looking.

Ultimately I agree that they're not equal; I think many people involved in LW or SSC are good-faith rationalists. But there's also a strong segment that has adopted - post hoc - a crazy set of priors to support NR views.

What strikes me is how antithetical this post-hoc "pick your priors, any priors!" is to the rationalist ideal, but it doesn't get called out. It's given equal treatment, and it means that the community is categorically incapable of discussing culture-war issues.

There are neoreactionaries, but they aren't anywhere near as important or central as the article implies. A huge chunk of this is the fault of the discourse norms, which allow for a lot of discussions that would be prohibited or highly discouraged elsewhere.

Reply on news.ycombinator.com