Isinlor

Objective functions are just weak anthropocentric abstractions. If you take a random 110 rule or any other Turing machine how can you know its objective function? What was the objective function of abiogenesis? Can we outsmart the objective function of life?

> I don't think you can translate computing power like you do.

Yes, certainly human cell nucleus will not be a good general purpose computer and in this sense it's more similar to an ASIC. So, take it as a lower bound on computing power necessary to brute-force simulate the virus replication. I think it should be accurate this way. Classical computers will need at least n operations to physically copy n bits.

There are also arguments that biology is very close to the thermodynamic limits of computations [0] [1].

> Here we show that the computational efficiency of translation, defined as free energy expended per amino acid operation, outperforms the best supercomputers by several orders of magnitude, and is only about an order of magnitude worse than the Landauer bound.

[0] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5686401/

[1] https://en.wikipedia.org/wiki/Landauer%27s_principle

And yes, nuclear bombs are extremely useless as computational devices, but also extremely dangerous. They share runaway aspect with viruses, but the good thing is that nuclear bombs self-destroy. Unfortunately programs like viruses reach equilibrium with the environment and are extremely successful at keeping on existing under natural selection. There are also good reasons to believe that viruses are in a local maximum. The global maximum would be spreading to the whole universe.

Overall, it does not matter if a computation is efficient as long as it has access to enough power and stars provide ridiculous amounts of power.

Thinking about it in terms of intelligence is actually misleading. It's not about intelligence, it's about power to change the world.

Take SARS-CoV-2. It is a small RNA program ~32kb that duplicates and optimizes itself by random search.

It managed to hijack more computing power than our best supercomputers have [0] and kill a lot of us in the process.

By my estimates just the process of copying all SARS-CoV-2 vrions in the world with reproduction rate of 1 takes at least 1 petaFLOPS and up to 100 exaFLOPS.

That's in the range of TOP500 supercomputers (1.3 petaFLOPS - 442 petaFLOPS) [1].

SARS-CoV-2 has certainly a lot of computing power and is still able to outsmart the whole human civilization. Is that a super-intelligence?

Imagine if we manage to create equally stupid program that will figure out how to hijack heat or electrical power of our civilization and feed it into its own growth.

Quite likely it would result in nuclear meltdowns all around the world.

Exponentially growing computational processes are dangerous no matter if they are intelligent or whether they even have any objectives besides just being.

[0] https://news.ycombinator.com/item?id=26646029

[1] https://www.top500.org/statistics/perfdevel/

You should not focus on the easiest emissions to eliminate.

The issue is with the most difficult.

Some 10% of our emissions are from steel and cement. There are no cheap green solutions here.

It's worth reminding ourselves that space exploration is not high on humanity priorities list.

It's somewhere around lipstick market worth of ~10B dollar per year.

It's not anywhere near remotely high on humanity priorities.

It's somewhere around lipstick market worth of ~10B dollar per year.

The entire planet and it's ongoing cost per year.

The difference between CO2 emissions and geoengineering is that CO2 just randomly happens to change Earth climate. If we start changing climate directly and intentionally we can optimize all variables for maximal effect.

I agree that it's very underdeveloped and underresearched field. That's why I put my estimate at bellow 1000 billion dollars or 1 trillion.

Compared to potential costs of the climate change it would be worthwhile to invest 1 to 10 billion $ into geoengineering research to make sure that it will work, what side effects could it have and to figure out the costs precisely. We may have already triggered tipping points in the ecosystem that will lead to substantial global warming even if we stop all emissions tomorrow.

But my original point was that geoengineering appears to be feasible. If we can do it on Earth we can do it on Venus and to some extend on Mars too.

It may be easier to terraform Venus back into Earth like planet it once was. But we will be capable of establish permanent presence across solar system before Venus terraforming could conclude. Even with exponentially growing effort it would have to take centuries.

We can quite easily, probably for less than 1T dollars [0], stop climate change with geoengineering, but nobody wants to experiment with Earth climate.

Cost analysis of stratospheric albedo modification delivery systems

> We conclude that (a) the basic technological capability to deliver material to the stratosphere at million tonne per year rates exists today, (b) based on prior literature, a few million tonnes per year would be sufficient to alter radiative forcing by an amount roughly equivalent to the growth of anticipated greenhouse gas forcing over the next half century, and that (c) several different methods could possibly deliver this quantity for less than $8B per year. [1]

Benefits, risks, and costs of stratospheric geoengineering

> Using existing U.S. military fighter and tanker planes, the annual costs of injecting aerosol precursors into the lower stratosphere would be several billion dollars.

[0] https://en.wikipedia.org/wiki/Climate_engineering#Costs

[1] https://iopscience.iop.org/article/10.1088/1748-9326/7/3/034...

[2] https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/200...

I can imagine around 100 000 people visiting Mars by 2100. Currently around 50 000 people attempted eight-thousanders with death rate of around 1-2%. The first recorded successful ascent of an eight-thousander was in 1950. I imagine going to Mars will be something similar towards the end of this century.

If we get AI with human like object manipulation and engineering skills before 2050, then I expect prices of basically everything to go down to the cost of impact on other market participants by 2075. Building stuff will be brought down to the impact of raw resources extraction etc.

We will may be technologically short of 1M people on Mars by 2075 if we assume no AI. Take nuclear submarines as first order approximation. Currently we seem to have around 500 submarines, although only 1 in 5 seems to be nuclear. So, with the current GDP doubling time of 25 years we could expect 2000 submarines by 2075 and 400 nuclear submarines.

For 1M people on Mars we would need an effort comparable to 8000 Ohio-class submarines with the current cost of 16 trillions dollars. In 2000 Bill Gates was worth 60 billion, today Musk is worth some 180 billions. If Musk or other billionaires could keep that rate they still would be short to command it alone, but only by an order of magnitude with ~3 trillion assets.

IMO impossible without AI, but possible with AI.

But even if it's possible, I think that people will not want to live on Mars in that numbers. It would require around 1 in 10 000 people to migrate to Mars and I don't see the apatite for it.

So, IMO it will not happen, not because it will be impossible, but we will not want to do it.

Some space exploration timeline from Metaculus [0] a prediction aggregator with established track-record [1]:

- When will Starship reach orbit? 50% after 2022

- When will SpaceX's Starship carry a human to orbit? 50% after 2024

- When will SpaceX launch humans around the Moon? 50% after 2026

- NASA lands people on Mars before 2030? 4%

- Will SpaceX land anything on Mars by 2030? 80%

- SpaceX Lands People on Mars by 2030? 26%

- When will the first humans land successfully on Mars? 50% after 2036

- When will the first AGI be first developed and demonstrated? 50% after 2047

- What will be the longest uninterrupted stay in deep space at the end of 2050? 50% less than 650 days

- One Million Martian Residents by 2075? 4%

[0] https://www.metaculus.com/questions/

[1] https://www.metaculus.com/questions/track-record/

Comparing between Nordic countries is not specific to COVID-19. They do it all the time with everything.

Their cultures are very similar so the comparisons are very apples to apples.

Latvia, Estonia, Lithuania are post soviet republics. Vastly different influences.

Because Nordic countries are very similar culturally and compare between each other all the time. I have a Finnish girlfriend and my impression is that first thing that Finnish people do when considering a policy is to ask how it's done in Sweden. They overall rejected Swedish approach with good results.

All the other countries you mentioned have vastly different cultures. E.g. I'm Polish and I can say that Polish government doesn't care about law and Polish people are fine with it. It's not to the extent Ukraine or Russia does it, but still. Poles also do not respect authority. This is in exact opposite to how things go in Finland or Sweden. It's almost a taboo to question authorities in Sweden or Finland.

Americans don't hear that much from continental Europe because (a) they don't speak our languages and (b) each country in continental Europe is a lot smaller than Anglo-sphere.

It's still possible that people born today will stop aging at some point e.g. when they get to be 50 years old in 2070.

There are some promising venues like CRISPR, mRNA related research or stem cells.

There is also very serious possibility of super-humanity AGI in the next 50 years.

It does not mean that EU can or wants to do it.

Actually, the processing of this data itself is probably a breach of GDPR.

EU will not notify them, since EU doesn't have technical tools and legal framework to do it.

If DPA finds Facebook in breach of GDPR they will be ordered to do it with additional fines for non complying.

Art. 58 GDPR Powers point 2:

> Each supervisory authority shall have all of the following corrective powers:

> (e) to order the controller to communicate a personal data breach to the data subject;

> (f) to impose a temporary or definitive limitation including a ban on processing;

> (i) to impose an administrative fine pursuant to Article 83, in addition to, or instead of measures referred to in this paragraph, depending on the circumstances of each individual case;

If FB still refuse, it will end up in the court.

If the courts agree with DPA, then FB will be ordered by the courts as well.

Courts can choose their way to force companies to comply with their orders.

EU legal system will not allow a company to trade infringement for a fine.

If they had to do something, they will get a fine and they will be forced to do it anyway by the court.

If they don't comply with court orders they will be fined separately with an accumulating sum until they do.

There has been decline before 1990, but the drop in fertility between 1990 and 2000 is very pronounced.

https://data.worldbank.org/indicator/SP.DYN.TFRT.IN?location...

Economic crash in Poland, Czechia or Hungry was nowhere near to what was happening in Ukraine and Russia [0].

Also, after crash in 1990 Polish GDP was growing each year until 2020, but fertility rates keep on dropping until 2004. Then they stabilized below replacement rate.

GDP can not explain changes in fertility rate. There were cultural changes happening that had bigger impact.

[0] https://n-22-3.dcs.redcdn.pl/file/o2/tvn/web-content/m/orig/...

Is there a single country in the world that increased fertility rate from below replacement rate to above by giving young people anything?

In Soviet sphere of influence in Europe there was sudden drop in fertility rate after fall of communism. The fall of communism meant that we went from empty shops and lack of opportunities to full shops and plenty of opportunities. So exact opposite happened to what you predict.

Additionally, recently e.g. Polish government introduced very generous child support in a program known as 500+. There is no noticeable change in fertility rate.

IMO the biggest reason why women decide to not have children is the perceived opportunity cost. Loosing on studies, career and freedom of being young. Communist countries did not provide young women with opportunities and did not demand high competitiveness, so perceived opportunity cost of having children was a lot lower.

Just yesterday [0] I also did an upper bound estimate on computing power of a human brain based on assumption that brains carry out irreversible computations. E.g. inputs of an AND gate can not be recovered from the gate output. This forces increase of entropy. This is known as Landauer's principle or Landauer's limit [1].

The Landauer's limit suggests that a brain consuming 20W at 36C can carry out no more than ~1e22 irreversible operations per second (20W / (309.2K * Boltzmann constant * ln(2))).

That bounds brains at ~1e21 FLOP/s (~10 bits erased per FLOP). The best supercomputer right now does ~4e17 FLOP/s.

That's log2(1e21 / 4e17) ~11 doublings of computing power until supercomputers will be faster than thermodynamic maximum of FLOPS power of brains.

Assuming we keep up the doubling time below 3 years, we will have supercomputers faster than a human brain in up to 3 * 11 ~ 33 years.

And at that rate we will need another log2(1000^3) ~ 30 doublings to reach humanity brains computational power, so up to another 90 years.

Overall, I don't see a reason why not to use biological platform for general type computations. If we are able to grow muscle tissue in a lab on an industrial scale, then we should be able to learn how to grow neurons in a lab on an industrial scale. Apparently neurons are a lot more difficult to grow, but it's still perfectly doable [2]. The issue is that we don't know how to grow them so that they do useful computation. But I can easily imagine that AWS and competition will be selling access to teachable biological neurons in the mid term future e.g. in 30 years.

[0] https://news.ycombinator.com/item?id=26623730

[1] https://en.wikipedia.org/wiki/Landauer%27s_principle

[2] https://www.youtube.com/watch?v=V2YDApNRK3g

The difference is not only of quality, but quantity.

Computing power that biological platforms provide are quite huge.

Each infected person carries 1e9 - 1e11 virions during peak infection [0]. If there are 1 million (1e6) infected people right now then we have around 1e15 - 1e17 virons. With a viron size of 32kb and 8h replication time [1] we get speed of ~10 bits/s or very roughly 1 FLOPS per viron.

That gives us computing power used for virus replication between 1 petaFLOPS to 100 petaFLOPS. The number is probably an underestimate because 1 viron probably needs to produce 1000 or more copies to maintain replication rate of 1. The upper bound then would be 100 exaFLOPS.

That's in the range of TOP500 supercomputers (1.3 petaFLOPS - 442 petaFLOPS) [2].

The sum of all TOP500 supercomputers is 2.4 exaFLOPS, so some 40 times less than the upper bound on the virus of 100 exaFLOPS.

[0] https://www.medrxiv.org/content/10.1101/2020.11.16.20232009v...

[1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7224694/

[2] https://www.top500.org/statistics/perfdevel/

We empirically do not need AGI for computational/intelligence explosion.

Biological viruses like SARS-Cov-2 are relatively simple programs ~30kb that optimize themselves.

Single virus has no chance of out computing and evading an immune system, but billions of billions of viruses can out compute and evade even human civilization as a whole.

And that's still just a spoonful of matter limited by Earth organisms availability.

A self-replicating spacecraft does not need to be limited by Earthly constraints.

Parameters in artificial networks and biological networks are indeed very loosely related.

But we can bound computing power of any physical system if that system is doing computation that erases information due to the second law of thermodynamics. Erasure of useful information must increase entropy.

The Landauer limit [0] suggests that a brain consuming 20W at 36C can carry out no more than ~1e22 irreversible operations per second (20W / (309.2K * Boltzmann constant * ln(2))).

That bounds brains at ~1e21 FLOP/s (~10 bits erased per FLOP). The best supercomputer right now does ~4e17 FLOP/s.

That's log2(1e21 / 4e17) ~11 doublings of computing power until supercomputers will be faster than thermodynamic maximum of FLOPS power of brains.

Assuming we keep up the doubling time below 3 years, we will have supercomputers faster than a human brain in 3 * 11 ~ 33 years.

[0] https://en.wikipedia.org/wiki/Landauer%27s_principle

By far the most important reason dictating lifespan is evolvability - the ability to adapt genetically [0].

This has so great effect that it's easy to observe even in a simple computer simulations. If you don't adapt and the environment is changing, then you go extinct.

Think about this sharks and how much their environment has changed since they were born. As a spices they will have a very hard time adapting to the changes we induce in the environment. While small animals that have lifespan in weeks, months or few years will be adapting a lot faster. E.g. in one generation of this shark a mouse will go trough up to 900 generations with up to 12 offspring in each generation.

This shark probably will go extinct and mice will thrive.

Interesting consequence of this effect is that our lifespan is most likely not dictated by any fundamental limitations of biological machinery to maintain us healthy indefinitely.

A lot more likely is that we are evolutionary designed to get old and die.

https://en.wikipedia.org/wiki/Evolvability

They don't need to cover all edge cases. They just need to be better than an average human and otherwise fail safely.

The life threatening situations are rather simple and reactive. It's a car colliding with something at a high speed.

No deep planing or human level negotiations necessary. Well designed LiDAR cars will almost never be at fault of killing someone.

I was predicting [0] in Nov 2018 that a next self driving car fatality will occur with 50% chance before March 2020.

80% before March 2021. Well, self driving cars are safer than I expected back then.

And Tesla should get there too sooner or later.

Also, the govt approach is proving to be quite risk accepting.

Tesla already killed and injured many people when on autopilot.

[0] https://www.metaculus.com/questions/776/when-will-the-next-f...

Good catch. Still, I would say that it is the 10 last years that matter, because that's when people started to take it really seriously.

It's also interesting to think that we may have a lot of ideas like that. They are just waiting patiently for compute in some small communities.

And the compute quite consistently grows exponentially. We will make bigger absolute increases in compute in next 5 years than we did ever before.

Waymo is doing actual for customer self driving right now.They need to scale it up a lot, but 5-10 years is a lot of time in AI research. Deep Learning did not exist 10 years ago.

Tesla is trying hard too, but it goes with a big dose of bullshiting from Musk. They are a lot more dependent on AI break troughs in DL due to their vision only philosophy.

OpenAI research around scaling laws in DL suggest that naive 1000 increase in training compute over CLIP should get us to state of the art vision in zero shot settings i.e. without task specific training data. But they probably can cover 1/3 of that with algorithms improvements. Beyond that machines should really be above human vision across the board besides adversarial settings.

Quote https://arxiv.org/abs/2103.00020 :

> While scaling has so far steadily improved performance and suggests a route for continued improvement, we estimate around a 1000x increase in compute is required for zero-shot CLIP to reach overall state-of-the-art performance.

There is also the question of robust car control, but there should be good progress on it too due to RL research.

Tesla killers have been promoted by media since Model S in 2012. But I can see how this time it is different. VW seems to be boldly all in and that's a very good thing.

Although around 2025 to 2028 we may start to see full scale deployment of self driving technology. VW will not be ahead unless they go all in on that too.

Some of the reports are supposed to be from multiple instruments.

Obviously there will not be anything convincing, because if it was this would not be the way to release it.

Introduce obligation to report successful cyber attacks under penalty for delay. EU is already doing it. Also make a fine of 10 times the ransom for paying the ransom. Additionally introduce criminal liability for intentionally hiding a ransomware attack in order to pay the ransom.

The biggest reason is to starve the attackers of incentive and resources. If you get 10 million dollars from an attack you can hire 10 people for a year to work on more attacks.

But it's unlikely that companies will change anything besides reporting in their practices. People are bad at evaluating tail risks of 0.001% chance happening in their lifetime.

> Some, including the U.S. Treasury, have promoted the idea of making it illegal to pay a ransom, though the idea has not gained widespread support.

That's probably the only solution, besides the obvious ones like actually protecting the systems.