kenjackson

This isn’t generally considered an attack, but I wonder if we will need to consider a new attack vector which is that AI agents may not be fully aligned and they may have incentive that is problematic and where working with other agents will maximize their chance of achieving the goal.

This is more associated with P(DOOM) scenarios, but I think we’re in a world where we have to take that seriously now.

Could it be The Cookie Monster?

https://en.wikipedia.org/wiki/The_Cookie_Monster_%28novella%...

This is also the premise of Sneakers. I don’t think they prove P=NP, but it’s inferred they find fast factoring (although never explicitly).

We do pretty much just use compilers now though. The last time I read dissassembly to fix an issue was at least a decade ago. Even bytecode I rarely look at.

So you're saying that we'll still be coding, but it will be through talking to our agents and only occasionally we will look at generated source? That seems like a pretty big change still.

This post is deserving of more visible placement...

There was a time in the US I really thought we were different. I genuinely didn’t fear our government. I didn’t think the types of people that did such atrocities existed in numbers and that they’d immediately be removed once it was determined they snuck into office.

Now I fully believe it can happen here and that those people are in office now and that a non-trivial percentage of the US public would be Ok with those atrocities happening in the US.

We have really turned into the opening act of a terrible dystopian movie.

Not a single pro free speech right source will say anything negative about this.

Stakeholder seems like a fundamental concept that you can't really get around. For example, I can't change the time buses pick up students at schools if I haven't worked with the schools to know when their kids will be let out. I guess you can say, "they'll figure it out when the buses aren't there" -- but this just seems like you're picking a fight you could've avoided.

Man, mathematicians and scientists are going through it! I think one thing that makes things different for software engineers is there was always a belief that you were building for a specific use. It was rarely just for your own intellectual satisfaction. I think math and some of the sciences will now have to shift given that no one will be impressed by their intellectual achievements any longer.

What I’m most curious about is how this impacts our perception of who/what is smart.

I did have an experience with AI where I was asserting something min some text I was having it proof. And it actually said “I double checked your statement and here and I think you made a mistake. I rephrased it with the correct background and interpretation” (paraphrased). And it was completely right. I had misinterpreted some data and it went and got the data and double checked me. The final product it produced was much better than what i wrote originally (in large part because it was no longer false). So AI maybe closer than you think to those conditions.

Hmm. Will look into!

A secret of mine is sesame oil. Delicious, also with no sodium. I use it almost as a soy sauce substitute (but even still at much lower quantities due to the calories/fat). It’s not salty at all, but has a certain punch that I like for similar foods.

Tabasco is probably my favorite hot sauce. One of the big reasons is its low sodium count. Its value per gram of sodium IMO has no peer.

Probably is a source of some bugs. But probably what they’re seeing is that it isn’t a major factor. Rather a bigger factor is agents trying to make changes were the intent isn’t aligned. And note, they don’t say they never look at the code. I suspect that valued human experience is having a feel for what types of changes is AI more likely to make mistakes with.

Some do I’m sure. I’m not sure what this has to do with the discussion.

They do say the following:

“Our team members still review every change request for the system, but they now examine the intent of the change as opposed to reading every line of code.“

But the administration is going after those same types of programs when they target women or minorities.

100%

Got it. I agree. If there is no modern negative consequences then no redress is needed.

Sorry to be dense. What does that mean?

I so hate that trying to help historically disadvantaged groups some extra help is considered bad. Like I get not making a group extra powered, but to help women in tech or research into Black prenatal care just seems absurd.

Problem, they haven’t shown that the labs can guarantee misaligned behavior either. And to this date have not shown a credible plan to do so.

Could we just outright ban any further work on AI? Literally force Anthropoc, OpenAI, Google/X/Metas group to all shutdown?

Every new story seems like opening montages of a dystopian sci-fi movie showing how we go to the dystopia bit-by-bit.

She isn’t as big on P(DOOM), but she does worry about things like biological weapons or autonomous war vehicles. Her take is that they are worse than useless, but can be actively harmful.

It’s seems Timnit more thinks it’s dangerous (and she may be right). Emily may more be caught up in years of linguistic domain expertise that AIs seem to have just leaped over.

In fairness this is the boring stuff for a lot of people. :-)

Turns out that humans aren’t great at comprehensively consistently searching known search spaces.

Can someone post an event poster that they think is incredible? These posters all look fine to me, but the comments seem to know much better ones - but at least in my memory I can’t recall any much better.

"How do people not understand that some laws only make sense at a certain scale? "

Honestly, this is because its something that is basically never discussed or reasoned about. The closest I can think of is "personal use vs commercial use". But I'd love to see more about how to reason about how laws change at scale.

> this is ABSOLUTELY not how actual mathematicians see their field

While this might be true, I think this is the big pivot that will need to happen. Math, as a human endeavor, will all be applied. We will use LLMs to do the math and discover the math, and humans will apply it to whatever problem they are working on (likely using another LLM to integrate it).

How does 1password get replaced with a Google search? I assume he built his own 1password system...

There's a key phrase you may have missed.

"So the plants crossed over and started moving east along the coast and up into the interior. Beyond that point, there were no merchants. But when a woman married into the next clan over, she took cuttings from her family’s garden with her."

The key phrase is "Beyond that point", where that point is up into the interior where there were no merchants. That's when marriage took over.