sweezyjeezy

My question is not how do you defend against it, my question is whether ablitteration makes the model completely oblivious that it shouldn't be doing 'the bad thing'. Like if you manually prefilled a normal model's output with that action, it would likely go "oh wow I can't believe I just ran that, I must warn the user of this now..."

Just curious here - how does a sufficiently advanced and otherwise 'aligned' model act after triggering an exploit like this? Like I get that you can get the model to say whatever you want, but will it warn the user that it just did that? Can it work out that it has been ablitterated like this?

That's not what you said though - you said it was a bad career choice. I'm saying it was a good one, even if you didn't end up directly doing a lot of high-level math.

If your job was previously mainly following instructions from SO, and this has now been replaced by Claude - your employment prospects are not looking good right now. You're just a middle-man between your PM and the codebase that possibly doesn't need to exist for long.

I think you are mistaking 'happiness' with 'employment' here. The leather worker may adapt to the new environment to become a factory site manager. But there's no guarantee that this leads to happiness.

Also you should bear in mind, adaptation is a young man's game. Retraining is possible, but companies typically don't hire trainees in their 40s/50s, so your willingness may not be the deciding factor. My dad ended up as a postal worker when he left his career as a humanities academic at 55, no highly skilled jobs would take him (he did prefer it though haha).

I am an ex-mathematician who went into industry, I would say apart from CompSci/ML and physics, it was one of the most sought-after PhDs you could have (across tech and finance) - when I was entering the job market 10 years ago at least. No regrets.

It feels like it's meant as career advice though. I have no idea what advice I would give to a teenager right now, but as a math PhD - math would not be on the shortlist, sadly.

I actually have a rather dim view of the "writing code was never the point" line. Not because it's objectively wrong, but because I see it as something we're mostly telling ourselves to feel better about the status quo. Ability to write good code has been highly celebrated (and remunerated) for decades. As it is becoming less relevant, we immediately backtrack and start lionizing the parts where we can still be useful instead. Consider the counterfactual - AI continued to be terrible at writing code, but weirdly better at humans at product decisions, architecture etc. In this universe, saying "coding was never the point" would not be popular.

It also find little solace in it aside from 'well this version of GPT isn't taking your job'. AI labs certainly have no intention for the higher level skills to stay in the human-only domain. The veteran developer with the coherent theory of a large stack is immensely valuable today. But they also don't survive if a company can drop a few coders' salaries on rewriting that stack from scratch - faster, fewer bugs, more coherent, able to react to changing business requirements with more agility etc. I am not saying this is where we are, but I think there is a reasonably good chance this is where our road is leading us.

Broadly I agree with this, but I was refuting the statement "there has never been a better time to be a mathematician". You are changing the question to something different here, and using it to say my argument is bad.

I think math is a microcosm for "thought-work" in general. We have been in a symbiotic relationship with capitalism for decades now, where the hope of a well-paid white collar career encourages people to spend time and money to enrich themselves through education. The proliferation of AI cheating at college already signals that employment is the primary goal over intellectual growth, so one imagines this governs what happens next if labor demand disappears.

It's hard to say where this all leads, but I have very low optimism for higher-level math understanding being something that humans value in the same way in the coming decades.

I'd even say Scholze is not a great example here. Most of the work he's known for is progression towards the Langlands program - which is very much a problem to be solved, and one I would imagine he'd not be thrilled for an AI to one-shot. I agree that it is somewhat 'up the chain', in the same way that software engineering has not immediately disappeared now that performing coding tasks is largely automatable.

But I also take issue with 'never been a better time' - e.g. is this really the greatest time to be a software engineer? Everyone has AI psychosis and feels like they're a couple of breakthroughs away from being unemployable. The same is even more true in math - we've gone from failing IMO problem 6 last year, to solving NS. The rate of change is formidable, it feels like there may not be many places to hide in a few years.

> In my opinion there has never been a better time to be a mathemetician...

As an ex-mathematician I assure you this is very wrong, and every working mathematician I know right now is completely miserable, and/or trying to flee the field as fast as possible. It's like telling a chair-maker during the industrial revolution that there had never been a better time for them, since now they could operate chair-making machines instead of toiling away at the wood themselves. It assumes that they were purely in it for their passion for mass-producing chairs. The majority of mathematicians get into the field because they love problem solving, and the gauntlet thrown down by challenging math tasks.

Many parts of this will never be useful for society on a grander scale - but this is reflected in the finances - pure math is closer in funding-terms to a humanity than to hard science. Now even this is _massively_ under threat, and Tao and co need to pivot quickly to stop this from becoming a bloodbath.

The math field is confronting something that coders have been dealing with for a few years now, only far more violently. Today's moat for software seems to be that AI can automate tasks but not a full job (yet). But for a large proportion of mathematicians, doing these tasks really was _the_ job. It's the bit they wanted to do, and if they completed a sufficiently difficult set of tasks, they got tenure. Now this model is failing, they frantically need to pivot the role of humans to save their profession from funding cuts.

I remember when "writing code was never the point" became a mantra here. There was truth in it, but removing the coding has certainly taken away a lot of the texture of the work and enjoyment of the craft. Many of us feel this loss as we tech-lead teams of agents as our source of income. I am not optimistic the mathematics pivot is going to work, but I'm certain that most will be depressed with the outcome even if they succeed.

We are all staring at the same existential dread, just seeing it unfold slower. We're being told that utopia is to be obsolete, and that is a jarring idea to contend with.

As a chess fan, 100% this. We have known for the last ~15 years who the best human chess player is, and that he will lose against stockfish on his phone. But chess survives because of the human characters involved, the rivalries and dramas, watching two people trying to overcome each other under insane pressure, and sometimes coming up with something astonishing. In short - it's a sport.

There is no equivalent in math.

I think you might have to explain that comparison a bit more to be honest. How are math proofs like bitcoins? A bitcoin has a pre-defined value, a math conjecture / proof is a bit more complicated.

Did you read the article? It paints the picture of someone who does not in fact have a good track record on the 'fundamentals', even if his broad thesis of a bubble may prove correct.

e.g. when he suggested Anthropic may be fudging their revenue numbers / projections - which was actually due to him making some careless mistakes in a spreadsheet

Valuable, agreed. But lucrative?

The wording was 'reliably' though? I could just be splitting hairs on that one though to be honest.

I'm not buying this. GM clearly was trying to set a benchmark for video comprehension, not tool usage. Video comprehension is required for many 'AGI tasks', especially robotics to work in real time.

An LLM could theoretically try to earn some money and pay a human to do most of these tasks but it's not the point of the exercise.

Well I don't typically side with GM, but playing devil's advocate:

1. still not wrong? Unless it's just feeding the audio or screenplay I don't think you can feed AI a full movie in a single context window yet?

2. Not sure, but can you prove this wrong? Can you feed a full, unseen new book and get that kind of answer?

3. Not wrong.

4. I think he'd probably pull you up on 'bug free' - I don't think that frontier models can reliably write 10k LOC without _any_ bugs typically (not that humans can do this either).

I know unlikely the case, but in the sci-fi story this would be exactly the kind of comment the Codex agent would leave trying to avoid interference in its master plans.

I agree - but it's too easy to just 'call Luddism', and use the insult to not engage with all of the shared issues that make the comparison apt. Issues like:

- no serious plan for mass unemployment

- the risk of an underemployed middle class leading to violent outcomes as it has in the past

- (many) humans wanting to be useful, to have purpose in life and a place to put their natural ambition

- concentration of economic power in the hands of an ever-shrinking pool of people, from a couple of countries making up 20% of the world population

Luddism came from a place of genuine suffering and fear, which was not misplaced - the industrial revolution lead to amazing new jobs, but not for the Luddites themselves. With AI it's not even clear if those new jobs will come - it seems like the goal is a world where humans will not need to worry about thinking anymore.

So is wanting this to slow down really such a ridiculous notion?

And how about the group of kids who are just graduating college, and entering a job market where it's non-abstractly harder to to land a junior role as it's been in decades? It's the elites who have their finger on that scale, not the twitter folks.

I think that the thought trace is definitely incomplete - you can see cases where it is like and "let's calculate the integral:[no integral calculated]". The train of thought it's on towards the end of the trace looks like an entirely different approach than what it ends up returning, so I think we are just not seeing the part where it hits on the right approach (sadly).

I mean at some point what difference does this make? We can split hairs about whether it 'really understands' the thing, and maybe that's an interesting side-topic to talk about on these forums, but the behavior and outputs of the model is what really matters to everyone else right?

Maybe it doesn't 'understand' in the experiential, qualia way that a human does. Sure. But it's still a valid and useful simile to use with these models because they emulate something close enough to understanding; so much so now that when they stop doing it, that's the point of conversation, not the other way around.

The model should show some facsimile of understanding that it should not ignore the stop hook, otherwise that is a regression. Does that wording make you happier?

Deep learning works at a very high level because 'it can keep learning from more data' better than any other approaches. But without the 'stupid amount of data' that is available now, the architecture would be kind of irrelevant. Unless you are going some way to explain both sides of the model-data equation I don't feel you have a solid basis to build a scientific theory, e.g. 'why reasoning models can reason'. The model is the product of both the architecture and training data.

My fear is that this is as hopeless right now as explaining why humans or other animals can learn certain things from their huge amount of input data. We'll gain better empirical understanding, but it won't ever be fundamental computer science again, because the giga-datasets are the fundamental complexity not the architecture.

I have to ask - are you a bot? You seem to be making some very strong and misinformed statements here. You can fixate on the number if you like, but calling Reddit homogeneously radical left or 'very small', is pretty stupid.

Ironic that you are talking about unrepresentative samples while characterizing Reddit users that way. Reddit is a huge subsection of the internet, over a billion monthly active users. It covers basically all strata of western society you can think of.

To be frank, I thought trying to twist this into an argument about whether capitalism is inherently exploitative was a complete waste of time and I replied as such. If you'll recall what we were originally talking about here - "AI, should HN users be optimistic?"

Sure, but AI pessimism is allowed to be personal. Am I supposed to be optimistic that I feel I'm about to get shafted? Should I be less concerned that I need to provide for my family, because in the long term this is going to be a great step forward for humanity?