Oh, interesting, thanks. I got curious a while back about the whole "how do you attach wires to a bacteria?" thing, but didn't look into it too deeply. Do you have any takes?
1,068 karma · joined November 27, 2018
Do you mean electrofixation of nitrogen? What level of biological involvement are you imagining? More like biology producing (some?) of the catalysts, or more like the entire reaction happening inside of cells?
As for the meta-time thing, functional decision theory [1] solves this. The application is particularly clean because your memories are erased between lives. There's no factor of 2 free lunch from temporal ordering.
And then it turned out that simply learning to imitate text with the right neural net architecture sufficed to achieve a huge fraction of the AI wishlist.
Of course, it's obvious that a machine that imitates text can claim to be conscious without actually being conscious. You're not wrong about that. Writing about consciousness appeared all the time in the training data. But the people who stick to the old ways, and still say "if it says it doesn't want to be turned off, we shouldn't turn it off" have a point too: We used to have a hard line in the sand. Now that's gone; we've found that it yields false positives. But we never replaced it. Now there is no line at all where we might doubt ourselves, no level of AI advanced enough that we might be forced to admit that it is conscious. We started out with simple next token prediction. Just world-modelling, nothing more. Certainly not conscious. Then we added RL. And we're trying to add neuralese and continual learning.
I can't say for sure that we're on track to achieve conscious AI on this trajectory. But one thing's for sure: If we do, we sure ain't gonna stop. One the day when a conscious AI is created, there will be no news story announcing the milestone.
Sorry, but most times I've talked to someone who says this (about AI completely replacing humanity), they don't have anything to say about why it's unlikely to happen other than:
- "well, it's just such an extreme outcome, it must be improbable"
- "humanity has survived near scrapes with extinction before"/"all the previous doomsday predictions have been wrong"
There is a cliche that all teenagers deep down believe that they are invincible. It seems to me that humanity is still a teenager in this respect: We don't take seriously the possibility of our own extinction. While one might think that the invention of nuclear weapons would serve as a wake-up call, if anything it has done the opposite.
I'm willing to hear arguments besides the two above, if you have them. (And to be clear, being replaced by AI doesn't necessarily mean being replaced by LLMs in particular. They are a relatively new development.)
∑ⱼ mⱼ v⃗ⱼ = 0⃗
where the mⱼ are the masses of the parts of the object and the v⃗ⱼ are the velocities of those parts.
If the object initially has 0 velocity, its kinetic energy is:
T = ½∑ⱼ mⱼ v⃗ⱼ²
Now we give the object a kick (or just switch reference frames) to change its velocity by Δv⃗. The new kinetic energy is:
T' = ½∑ⱼ mⱼ (v⃗ⱼ + Δv⃗)²
T' = ½∑ⱼ mⱼ (v⃗ⱼ² + 2v⃗ⱼ⋅Δv⃗ + Δv⃗²)
T' = ½(∑ⱼ mⱼ v⃗ⱼ²) + Δv⃗⋅(∑ⱼ mⱼ v⃗ⱼ) + ½Δv⃗²(∑ⱼ mⱼ)
If M is the total mass of the object, then we can substitute this into the sum in the last term. And we already saw that the sum in the middle term was 0. So:
T' = ½(∑ⱼ mⱼ v⃗ⱼ²) + Δv⃗⋅0⃗ + ½Δv⃗² M
T' = ½∑ⱼ mⱼ v⃗ⱼ² + ½MΔv⃗²
So in terms of the original kinetic energy T, which was purely thermal energy, we get:
T' = T + ½MΔv⃗²
In other words, because of the quadratic kinetic energy formula, we can see that the total kinetic energy T' of a hot object is just its thermal kinetic energy T plus the usual mechanical kinetic energy ½MΔv⃗².
However: Energy and momentum are not invariant under changes of reference frame, though the magnitude of the energy-momentum 4-vector is invariant between frames.
The answer linked above actually takes advantage of the fact that energy is not the same in different reference frames in order to make the argument work.
I think you are overthinking the heat thing. If you have a train car full of hot water and you slow the train down (extracting kinetic energy from it) until it stops, the water in the train car does not change temperature at all, other than a bit of sloshing around and loss of heat to the surroundings.
In reality, it is (as mentioned in TFA) very possible to filter the training data and remove documents that contain discussions of AI misalignment. If an AI lab isn't doing this, it's simply because they don't consider the problem important enough to be worth the expense and development effort.
(It's true that formalization can still have bugs in the definition of "secure" and doesn't work for everything, which means defenders will still probably have to allocate some of their token budget to red teaming.)
In reality, probably radiator designs would rely on fluid cooling to move heat all the way along the radiator, rather than thermal conduction. This prevents the above problem. The issue there is that we now need to design this system with its pipes and pumps in such a way that it can run reliably for years with zero maintenance. Doable? Yes. Easy or cheap? No. The reason cooling on Earth is easier is that we can transfer heat to air / water instead of having to radiate it away ourselves. Doing this basically allows us to use the entire surface of the planet as our radiator. But this is not an option in space, where we need to supply the radiator ourselves.
In terms of scaling by instead making many very small sats, I agree that this will scale well from a cooling perspective as long as you keep them far enough apart from each other. This is not as great from the perspective of many things we actually want to use a compute cluster for, which require high-bandwidth communication between GPUs.
In any case, another very big problem is the fact that space has a lot of ionizing radiation in it, which means we also have to add a lot of radiation shielding too.
Keep in mind that the on-the-ground alternative that all this extra fooling around has to compete with is just using more solar panels and making some batteries.
EDIT: Above is false. Went back and checked and I had mis-remembered the video.
Also, do you actually think the core idea is wrong, or is this more of a complaint about how it was presented? Say we do an experiment where we train an alpha-zero-style RL agent in an environment where it can take actions that replace it with an agent that pursues a different goal. Do you actually expect to find that the original agent won't learn not to let this happen, and even pay some costs to prevent it?
EDIT: Looking things up indicates that if n is even, there's already a square root of -1 in the field, so we can't add another. So now I believe the 1/4 of the time thing you mentioned, and can't see how that's wrong.
People's bets are publicly viewable. The website is very popular with these "rationality-ists" you refer to.
I wasn't in fact arguing that giving a prediction should make people more trustworthy, please explain how you got that from my comment? I said that the main benefit to making such predictions is as practice for the predictor themselves. If there's a benefit for readers, it is just that they could come along and say "eh, I think the chance is higher than that". Then they also get practice and can compare how they did when the outcome is known.
In all seriousness, I do agree it's a bit harmful for people to use this kind of reasoning, but only practice it on things like AGI that will not be resolved for years and years (and maybe we'll all be dead when it does get resolved). Like ideally you'd be doing hand-wavy reasoning with precise probabilities about whether you should bring an umbrella on a trip, or applying for that job, etc. Then you get to practice with actual feedback and learn how not to make dumb mistakes while reasoning in that style.
> And do we also pull this one out of thin air?
That's what we do when training ML models sometimes. We'll have the model make a Gaussian distribution by supplying both a mean and a variance. (Pulled out of thin air, so to speak.) It has to give its best guess of the mean, and if the variance it reports is too small, it gets penalized accordingly. Having the model somehow supply an entire probability distribution is even more flexible (and even less communicable by mere rounding). Of course, as mentioned by commenter danlitt, this isn't relevant to binary outcomes anyways, since the whole distribution is described by a single number.
You're absolutely right that if you have a binary random variable like "IMO gold by 2026", then the only thing you can report about its distribution is the probability of each outcome. This only makes it even more unreasonable to try and communicate some kind of "uncertainty" with sig-figs, as the person I was replying to suggested doing!
(To be fair, in many cases you could introduce a latent variable that takes on continuous values and is closely linked to the outcome of the binary variable. Eg: "Chance of solving a random IMO problem for the very best model in 2025". Then that distribution would have both a mean and a variance (and skew, etc), and it could map to a "distribution over probabilities".)
The point of giving such estimates is mostly an exercise in getting better at understanding the world, and a way to keep yourself honest by making predictions in advance. If someone else consistently gives higher probabilities to events that ended up happening than you did, then that's an indication that there's space for you to improve your prediction ability. (The quantitative way to compare these things is to see who has lower log loss [1].)
If the variance (uncertainty) in a number is large, correct thing to do is to just also report the variance, not to round the mean to a whole number.
Also, in log odds, the difference between 5% and 10% is about the same as the difference between 40% and 60%. So using an intermediate value like 8% is less crazy than you'd think.
People writing comments in their own little forum where they happen not to use sig-figs to communicate uncertainty is probably not a sinister attempt to convince "everyone" that their predictions are somehow scientific. For one thing, I doubt most people are dumb enough to be convinced by that, even if it were the goal. For another, the expected audience for these comments was not "everyone", it was specifically people who are likely to interpret those probabilities in a Bayesian way (i.e. as subjective probabilities).