frotaur

This was not a neuron. This was from a trained sparse auto encoder.

In a few words, they found a way to map residual stream activations (so, a vector) into a few interpretable sparse concepts. They also had the converse maps, that translate concepts into activations.

Golden Gate claude was obtained by artificially increasing the activations in the direction of the 'Golden Gate' concept. It was not neuron surgery.

Indeed, I see no difference between a chinese model hosted in Europe vs a Europe-grown model.

You could argue that potentially you can train a backdoor into a foreign-grown model (that has been demonstrated to be possible in toy examples), but I would highly doubt that this would be the case today, given the potential reputational damage if discovered.

I think you are free not to care about what they are doing. It turns out that big O notation has some useful properties, and in particular it is a useful definition as it allows you to define complexity classes which are a very useful tool for analyzing algorithms in general.

Much of our 'practical' understanding of algorithms also in part came from theoretical advances.

What makes you so certain that this 'way of developing software' cannot eventually be understood and practiced by AI?

I think precisely the premise of the blog post is that there will be a moment (sooner or later) where flexing that 'engineering' muscle to avoid it disappearing is irrelevant, in that this muscle will not be needed due to AI replacing it.

So it seems you are rejecting this premise?

I think this take is extremely cynical and made in bad faith. Lots of dangers that are due to AI acceleration are raised by a very wide mix of experts, scientists, engineers and researchers.

It is clearly incorrect to call it a conspiracy theory. You can be skeptical, as there is no clear consensus even among experts, but outright dismissal is simply an unscientific position.

The huggingface incident was reviewed by independent researchers, which explicitely declined any payment from OpenAI tonpreserve their integrity. They work for non-profits concerned with AI safety.

They claim that what happened was very much not because they were 'carefully engineered and instructed to do those things'.

Similarly, some wikis which were hijacked by agent to be used as messageboard were actually not disclosed by OpenAI (probably trying to conceal, as website showed likely activity from OpenAI researchers visiting the site after the incident) and discovered independently.

I don't know how you can claim that this was still on purpose by OpenAI as some sort of publicity stunt.

It is a fact that among experts there is no consensus on saying '(super)intelligence is broadly safe and easy to control'. There might even be a consensus forming on the opposite claim.

Regardless, why would there be no scientific consensus if the question was easy and clear cut? I think the easiest reason is that these are hard questions to answer.

that's essentially it, if the proof is incorrect it does not compile which signifies a problem in some step.

In most AI takeover scenarios, if you take as a premise that the AI has human or above-human intelligence, and that it is misaligned, it is obviously aware of the pull the plug possibility.

Therefore, as you would if you were in its position, it will plan around it. For instance, by acting perfectly aligned for 2/3 years, continuing the improvement of its capabilities while being deployed in ever more systems.

Once it's confident it can act with high probability of success, it would then turn on us. This phenomenon is called 'treacherous turns'.

Any scenario in which you assume you have ASI or AGI but also find a 2-sentence way to foil the AI's plan is inconsistent, as the AI will also have thought of this failure mode.

There is no intrinsic physical reason that the difficulty of 'defending' a system is symmetric with the difficulty of 'attacking' it.

For instance, it is not because you are able to design and release a (biological) virus that you are also able to defend from it (design a vaccine and inoculate the world's population).

If we're lucky, it might be the case for many instances of problems, but there is no a priori guarantee that this holds.

The point of lean proofs (as it stands) is simply one bit of information: that a given mathematical statement is indeed true.

It's a way to be absolutely certain (modulo bugs in the lean kernel) that a proof you came up for a statement is indeed correct. It is really not meant to be analyzed, much less now that they are fully llm written.

Not sure about the dystopia... Had a similar thought when covid was beginning 'wow pretty exciting, just like in the movies'.

Turns out actually living some terrible catastrophe is only fun in the movies.

I fail to see why this is a good argument against AI replacing humans, given that humans also produce errors.

Can you specify why we should see things differently if the behaviours the agents display are driven by parsing LLM responses and executing commands?

The energy to make the cotton bags presumably also involves lots of drilling and extracting, at least currently.

'Quach' is an unfortunate name when researching new technologies...

Alternative link: https://xcancel.com/Mcn_S7/status/2089107014526079341

Preprint link: https://arxiv.org/abs/2608.10218

Not only that, but KL divergence is not a '%'. It's just a number ranging from 0 to infinity that tells you the 'distance' between two probability distributions.

I think if you trained an LLM specifically for the purpose of passing the Turing test (instead of being helpful, harmful, and so on), its likely it would pass it.

You would have to train it/finetune it on a couple hundred of 'humans chatting in the context of a Turing test'.

Might be benefits, might be downsides.

In the proof, it claims that "At least two of these must fall on the same side of P′, the perpendicular projection of P on ℓ.".

However, this is not true as it is possible that P'=B. However it seems the proof still goes through (at least as depicted in the image, haven't thought hard about the general case).

"and the only sane thing to do is to fully support high quality ethical research groups and firms as they get up the learning curve of this technology."

These research groups are almost unanimously advocating for a pause/slowdown as our best chance.

First of all, sure if you screw up designing a bioweapon you die, but unfortunately that does not necessarily mean the bioweapon dies with you, quite the contrary it can kickstart its propagation.

Second, in your argumentation you assume that the experiments are extremely difficult or costly. Thankfully, so far, it seems to be the case that they are too difficult (either due to domain knowledge, or to difficulty of obtaining the necessary components/equipment).

But there is no clear reason to think it will remain this way (e.g. crispr allows for genetic engineering which is very cheap). And it appears that, for domain knowledge, capable LLMs are rapidly reducing that barrier. We are not there yet of course, but I think it is crazy to dismiss these concerns, which are very real and crucially are NOT just brought up by the big labs.

Sorry because pointing this is a bit tired by now, but reading already the first two paragraph thete is this unmistakable stench of LLM slop writing. Immediately disengaged.

Didn't humanity also score very low on 353, namely 0 since they were open? Probably collectively we could have gotten a slightly better score if all of Math started trying to solve those, but not by that much, I think, since they are precisely still open.

Check out https://arindube.substack.com/p/a-minimum-wage-natural-exper..., very good post comparing states with minimum wage and without, that seem to go against this often repeated argument.

Not sure the cockroach cyborg will survive such pressures, already surprised it can survive the few feet of water.

I think the board (and disappearing letters) are made with a list of words in mind. Thus, to complete the puzzle (dissolve all letters) you cannot count the bonus words as they were not accounted when creating the letter graph.

Its unclear to me why the color space is 2-dimensional. Why wouldn't it be a 3-dimensional space, indexed by how much each of the 3-cones is activated ? Not clear to me from the article!

Well, the most common example si precisely screens, no? A screen displaying the color yellow is actually a spectrum of red and green peaks, stimulating your red and green cones just like a spectrum containing a single frequency of the color yellow.