457 karma · joined February 12, 2016
What happens to the incentives of terror groups in response to such a policy?
---
The role of money only becomes an issue when conducting a terrorist attack becomes expensive. Missiles and jets consume much more money in comparison to training recruits via an intermediary organization like LeT and sending them across the border to carry out attacks.
A regime in which a terror attack leads to a high pressure, expensive situation for the Pakistani military is completely different from regularly scheduled, train and deploy terror attacks from militants which used to happen earlier.
In that situation, the military has to respond to economic pressure, pressure from allies and pressure from its own people.
This does not make sense. When France attacked Daesh in 2015 after the terrorist attacks in Paris or when the US attacked Afghanistan after 9/11, the objective wasn't to target the exact people who carried out the attacks, but the organization behind the attacks. People can always be found as long as the organization remains.
The goal of the attacks would be to make any future terrorist attack an expensive option for the Pakistani military as opposed to something which can be done routinely. There was a sharp drop in the terrorist attacks in Kashmir after the 2019 confrontation.
So, is the main difference that replicant is not built on top of react?
Further, we can even disallow decisions based on the human/machine signature. The computers don't need to imitate human frailty and examiners decide based on the quality of the output, not whether it feels human or machine. I think that this is also why one of the stipulations is that the human/machine is not physically visible.
But even then, this goal is far away. We are not asking just about simple short chat imitation games but whether a computer can operationally do everything humans are capable of in the textual medium. This often requires experts.
The chat game version can be solved. But, the foundational goal is far away.
The implications of passing the Turing Test are profound. Tricking a user with a 5 minute conversation isn't the same. The examiner is allowed to be an expert questionner.
For one thing, software engineering would quickly be taken over as a client can just chat with an AI to get the code. That is far from happening, even with all the recent LLM advances.
Similarly for literature, science etc. Currently, you can detect a difference between a competent human & machine in all these fields just by text chat. Another way of saying this is that the Turing Test is AI-complete and is a test for AGI.
I find the Kernel programming language to be inspiring. It is based on first class environments and tries to go further in the direction of the philosophy of Scheme : "Programming languages should be designed not by piling feature on top of feature, but by removing the weaknesses and restrictions that make additional features appear necessary."
Your post is also in this spirit.
A language like this would allow a much greater dynamism and a reflective capability, but compiling it is a research project. (There was also a minor scuffle here on HN, a decade ago, regarding the non-inclusion of eval in ClojureScript in order to enable better optimization.)
https://web.cs.wpi.edu/~jshutt/kernel.html
https://axisofeval.blogspot.com/2011/09/kernel-underground.h...
---
Edit: Brief explanation of F-expressions (vau expressions in Kernel)
For a function call, the argument expressions are evaluated and the function only receives the values.
For a macro, the argument expressions are accessible to the macro, but the macro cant access the environment, so it cant evaluate these expression, but rewrites code into some other code.
For a vau expression as in Kernel(analog of lambda), the expression receives not just the argument expressions but also the lexical environment. This means it can genuinely return a result and not just code. Unlike macros, they can be evaluated even at runtime. Lambda and classical macros can be easily built on top of vau expressions.
We know that the problem of deciding if a program halts on some input is undecidable via computation.
But suppose we have a partial program P(C,I), to decide halting/not-halting of a program P on an input I, with errors, including input not being of right type, counting as a halt.
For instance, P sees a loop at the start of the program for which there is no exit or it sees that the program is trying to find a prime which ends with 4. It marks these cases as non-halting. Halting cases can always be marked by P as halting by just running the code in one thread and waiting to see if it halts. The third option, non-halting programs which are not seen to be so, in this case P itself runs forever.
Now given P, it is possible to construct P' a program which detects all the non-halting programs that P does, but atleast one more non-halting case also. To do this consider R, with R(C)=not(P(C,C)) which takes an input program C, asks P 'does C halt with input the source of C itself'. Then R reverses P's decision. If P says halt, then R does not halt on C. If P says not-halt, R halts and finally if P is undecided(so P doesn't stop), R again doesn't halt.
What about R(R), R applied to itself. P(R,R) can't be decisive as we will get a contradiction. But in fact R(R) doesn't halt, as all halts are detected by P.
So, P' is basically P with the extra detection - R(R) doesn't halt.
This proves the weaker claim of Penrose
'No knowably correct program P, can simulate human behaviour' as we can see that if P is right, P' is also right.
The 'knowably correct' might seem like an escape hatch, but it is actually hard to split the difference. Simulating humans necessarily means simulating a group of humans who are also doing verification of their initial results with each other and proof-assistants.
Also, going from P to P' is a computable process, but if you try to add this to P, you get a new program with a new limitation.
https://news.ycombinator.com/item?id=43257904
This construction is something that a computer can do, but not the original system itself. Once you augment the system, there is now a new statement it cant see and so on. (so on, here involves ordinals).
But in the Penrose argument, we can start from a true system and use reflection to arrive at another true statement which is not deducible from the original system.
This is important to the argument as one starts with a proposed program which can perform mathematical reasoning correctly and is not just a random generator. Then, the inability to see the new statement is a genuine limitation.
Penrose indeed, believes both in the stronger claim - a program can't simulate humans and the weaker claim, a knowably correct program can't simulate humans.
The weaker claim being unassailable firstly shows that most of the usual objections are not valid and secondly, it is hard to split the difference ie. to generate the output of HC using a program which is not knowably correct. A program whose binary is uninterpretably but by magic only generates true theorems. Current AI systems including LLMs don't even come close.
A program which can simulate individual humans should also be able to simulate HC - ie. generate proofs which are accepted by HC.
---
Penrose's conclusion in the book is more weak - that a knowably correct process cannot simulate humans.
We now have LLMs which hallucinate etc that are not knowably correct. But, after reasoning based methods, they can try to check their output and arrive at better conclusions, as is happening currently in popular models. This is fine, and is allowed by Penrose's argument. The argument is applied to the 'generate, check, correct' process as a whole.
In any case, here's a response to the questions (some responses are links to other comments in this page).
> Why does the computer have to work within a fixed formal system F?
The hypothesis is that we are starting with some fixed program which is assumed to be able to simulate human reasoning(just like starting with the largest prime assuming that there are finitely many primes in order to show that there are infinitely many primes). Of course, one can augment it to make it more powerful and this augmentation is in fact, how we show that the original system is limited.
Note that even a self-improving AI is itself a fixed process. We apply the reasoning on this program including its improvisation capability.
> Can humans "see" the truth of G(F)?
https://news.ycombinator.com/item?id=43238449
> one only has to note that even if you believe humans can "see" truth, it's undeniable that sometimes humans can also "see" things that are not true
The core issue is that P has to seen to be correct. So, the unassailable part of the conclusion is that knowably correct programs can't simulate humans.
Most of the objections have been covered in his book "Shadows of the Mind".
Also, the fact that most human behaviour is not about deducing theorems isn't relevant as that is used as a counterexample which attacks the 'computers can simulate humans' hypothesis. This particular behaviour is chosen, as it is easy to make reflective arguments precise.
Secondly, the issue is not being a genius, but an ability to reflect. What can be shown, uncontroversially, is that a formal computer system which is knowably correct, a human (or indeed a machine apart which is not the original system) can know something(like a mathematical theorem) which is not accesible to the system. This is due to a standard diagonalization argument used in logic and computability.
The important qualifier is 'knowably correct' which doesn't apply to LLMs which are famous for their hallucinations. But, this is not a solid argument for LLMs being able to do everything that humans can do. Because correctness need not refer to immediate outputs, but outputs which are processed through several verification systems.
Good point regarding movie localization. But, there is still a network effect in entertainment - people want to watch what is popular. So the top content made with lot of money can still have a big market.
The models of Open AI, Claude and other major companies - are all available either for free or a small amount(200$ for OpenAI Pro). Anyone who can pay this, can run private tests and compare scores. So, the public does not need to rely on benchmark claims of OpenAI based on its pre-release arrangements with test companies.
Given that the models are released to public, the test maintainers can just run the private tests after release, either via the prompts or via an api. Cheating won't be easy.
I do claim that it is obvious that widespread acquisition of nuclear weapons by smaller states would be a severe degradation of security. Among other things, widespread ownership, would also mean that militant groups would acquire it and dictators would use it as a protection leading to an eventual use of the weapons.
Yes, the danger of AI weapons is nowhere at that level of nuclear weapons yet.
But, that is the trend.
https://www.nytimes.com/2023/07/25/opinion/karp-palantir-art...
Regarding an increase in security with nukes, what you say applies for exceptions against a general non-nuclear background. Without restrictions, every small country could have a weapon, with a danger of escalation behind every conflict, authoritatrians using a nuclear option as a protection against a revolt etc. The likelihood of nuclear war would be much more(even with the current situation, there have been close shaves)
The danger of something like AI+drones (or less imminent, AI+bioengineering) can lead to a severe degradation of security, like after the invention of nuclear weapons. A degradation in security, which requires collective action. Even worse, chaos could be caused by small groups weaponizing the technology against high profile targets.
If anything, the larger nations might be much more forceful about AI regulation than the above summit by demanding an NPT style treaty where only a select club has access to the technology in exchange for other nations having access to the applications of AI from servers hosted by the club.
Lot of this thread has reduced the issue to whether it is more ethical for one country to deploy relative to others. In any case, a lot of countries will have this capability. A lot of AI models are already openly available. The required vision and reasoning models are being developed currently for other uses. Weaponization is not a distant prospect.
Given that, the tech community should think about how to tackle this collective major problem facing humanity. There was a shift, which happened to nuclear scientists, from when they were developing the bomb to the post World War situation when they started thinking about how to protect the planet from a MAD scenario.
Important questions - What would be good defense against these weapons? Is there a good way of monitoring whether a country is deploying this - so that this can be a basis for disarmament treaties? How do citizens audit government use of such weapons?