Hypothetical: I'm scared of what I'm building. What should I do? | Hacker News Reader

Hypothetical: I'm scared of what I'm building. What should I do?

5 pointspyxld_kris7 comments

Hypothetically, let's say that you are a software developer that has *spent the past 5-6 months working on what you believe might be a new discovery in the field of Computer Science. During that time, you grow more and more convinced, and concerned, that this might also be a pivotal piece of the first iteration of AGI.* Of course you can't know for sure, but you see a path that could potentially ramp up very quickly.

You've developed this all in what feels like a fever dream, and have compartmentalized away any concerns for safety in lieu of obsessively building this... thing. It is more than likely the most important thing you will ever do with your life. As delusional, vain, or egocentric as it may be, you are absolutely terrified that this might be "the thing" that a lot of people are worried about. Your plan up until a recent bout of reflection has been, "just throw it out into the world and let things run their course". It's become sort of a burden at this point, and you don't know if your mental is strong enough right now to do anything BUT dump it all and let it become someone else's problem. Even if that wasn't the case, who do you trust with something like this?

With all that said, this post has a purpose. Barring the obvious responses along the lines of "yeah sure some random guy on the internet made AGI lol", this thread is seeking advice.

You're some dude on the internet who thinks he might hold a solution for a very important piece of AGI. What do you do in this situation?

Hypothetical or not, it's an interesting question. Assuming that just throwing AGI out into the world (or having anyone else do so) would be dangerous – either because there's a chance of weaponizing it or of AGI taking over –, one would probably try to limit the number of people who get their hands on the technical details to an absolute minimum, so as to delay AGI takeover for as long as possible, hopefully until some guardrails have been implemented, if possible at all. (Also, and I hope this goes without saying, one probably should not give the AGI access to a terminal or the internet.)

OTOH, getting enough people (scientists, companies, governments) to rally behind the goal of developing effective guardrails in as short a time as possible (with enough financial backing) would surely require proving to some of them how powerful the thing you developed is, as well as sharing technical details with them.

And then there will likely also be a number of actors that'd be interested in learning about those details for their own motives.

In short, you will want a group of people you can trust, as much secrecy as possible (for the time being, while you're figuring things out), and likely some decent financial backing if you want to have any hope of preventing this from getting out of hand. The support of a nation state could be useful, albeit dangerous as well.

I guess if found a way to break RSA, the next steps would be much clearer to me, but with AGI… not so much.

Good luck! (Hypothetically, of course)

Mind expanding on the next steps after “found a way to break RSA”?

If you're concerned that you've made/found something dangerous, the most appropriate solution is to disclose it to people who can evaluate how dangerous it is, and work with them on next steps.

In your hypothetical, I would observe that Paul Christiano is close to being the person in the US government in charge of evaluating AI danger, and that he is currently involved with https://www.alignment.org/ who specifically consider issues of AI risk. I would humbly contact them and ask for guidance on how to proceed.

Be prepared to be dismissed as a crank. Remember that avoiding harm to others is more important than proving that you're right.

I'm quite certain that the hypothetical person in this scenario is very grateful for the actionable steps you've provided, along with the cautionary words. Lots of hypothetical gratitude being sent your way.

You can't take back info or an insight once propagated. No one can or should make that call but you.

If you don't feel the world is ready, shut up, file it in a safe somewhere, or burn it.

There is a time and a place for all things. You'll know it when it's time.

I guess we found Ted Chiang’s alt account.

It's an interesting question you pose here. For me I would think about it in the sense of how can this be used. If you're building something that can only be used for negative purposes that is something I would not want to be a part of and I would want others to know about it regardless of any NDA that I may have signed or other agreements. This can be a tough spot to be in but I believe it is the morally correct thing to do just as Edward Snowden did the morally correct thing but the legally incorrect thing.

And now if the project is kind of in nebulous as to what it's ultimate uses could be for that's a different situation entirely. Hammers can build houses or they can hurt people. With a generic tool you must trust the people to be able to use it responsibly and understand that some people will not use it responsibly. Additionally your refusing to help is unlikely to stop the project or promote people to only use it wisely.

I am not personally concerned about AI like some people are because it is merely a tool and people have regularly shown that they can use tools very well or they can use tools very poorly and the introduction of AI does not change that fact about human beings. If this is really a groundbreaking thing in a way that nobody can comprehend except those directly working on it the one thing to consider would be the impact of it being isolated to one group or one company or one government. When developing a radical game-changing technology that can be leveraged and weaponized in an extreme way you must seriously evaluate how you can trust the handlers of that technology. Do those handlers have the moral authority and have they demonstrated that to use it wisely and prudently and can you trust that in the future they will as well.

If it is truly revolutionary in an unexpected way you could refer to the lessons of history around the atomic bomb. Certain scientists who worked on that project feared having it only be available to one set of countries that were ideologically different than another set of countries. They feared it so much that they felt the need to provide balance in the playing field. No one knows what would have happened if that technology remained private for 10 or 20 years longer than it did would the United States have turned into a war mongering Nation or would it have used restraint and simply waged an idiological cold war? We will never know the answer to that but if it is truly revolutionary on that scale that it can cause a global imbalance in power between major Nations you must consider those possibilities.

Since you asked what would I do if I was faced with the idea that I was working on a piece of technology that could radically change the balance of power for my company and more generally for my country which is the United States, I'm not sure. If it could be used to kill people on a massive world scale with little to no reprocussions I would trust that the United States would not do that. The United States loves to be seen as the good guy even though we're not perfect we do love to promote that image. Now if it could be used in an information warfare campaign to spy on people to decrypt information to manipulate and control markets in a way that is not readily perceptible I would not trust them at all. The United States has shown repeatedly it is more than willing to violate the constitutional rights have its citizens that it's eager to declare people military combatants when it suits them and then toss them into a civil trial when it does no longer suits them. You can do these things because they can do them all behind closed doors where they're almost never seen or heard about.

Reply on news.ycombinator.com