rakejake

This precisely - AI takes away your leverage as a software engineer. AI models are very well trained at coding today so I'm not sure what further improvement really looks like. But even if humans stick around to guide the AI, I expect the job itself will get more unpleasant because capital/management have no reason to treat this labor class well anymore.

+1. It is my favorite Radiohead album and the one that I revisit the most.

Where I work, PR counts are a major metric come perf reviews. Surprise, surprise every team has a pointless migration going on. The number of promotions though...

This is a great question. I am not a mathematician but I've had similar questions about my own field (CS/Elec) as well.

I have always felt that a huge amount of brain power has to be expended in understanding the jargon and getting used to the machinery. Once we get past that, what people would think are highly challenging subjects become rather simple and logical. An analogy I like to use is old people struggling with simple tech tasks. I remember an uncle of mine who approached me to fix their laptop which wasn't connecting to the internet. Turns out the laptop had not been booted in a while and the public certs shipped with the machine had expired. Said person is a highly accomplished person in their field but was helpless when it came to this task. After I solved the issue, the uncle was eager to know what the problem was and I tried to ELI5 it but to no avail. It is possible I did a really bad job of explaining it but there was too much basic machinery to cover. Why is there a public and private key, what is symmetric/asymmetric, why do certs even expire etc. I couldn't even touch on the foundational math underpinning all of it so I left him in a confused daze.

To any SWE, all this is table stakes but for non-tech people it is way too much jargon. Today, I'm confident my uncle could prompt his way to a working understanding of PKI - heck maybe even some aspects of cryptography and prime number math that makes it work. This is where I see LLMs making a massive difference - they can crunch a massive amount of jargon and leave only the actual trick/idea for you to understand.

This is also what people like Szegedy and Tao have also said - LLMs will make math closer to what SWE is - a lot more people doing it in workmanlike fashion rather than a small community writing papers. At one point, dynamic programming was some research topic that Bellman worked on and now every 20 year old works on tens to hundreds of these problems in the hopes of cracking a job. But I'm sure the Math field will go through some turbulence and I wish them the best. There will always be a need for SMEs who deeply understand a subject and can come up with interesting problems.

Tbf OP said they have the tools to understand what was going on, not that they could build a model to compete with OpenAI tomorrow. EE (especifically DSP and communications) uses a lot of the same building blocks as ML so yes it does prepare you.

You are right in that the emergent abilities come from humongous amounts of training data and compute which only a handful of companies (and the people working in them) deal with today. I'm sure there's a lot of secret gotchas which only arise at a certain scale so one cannot self-study for them. Companies also guard these wherever possible because of economic incentives. But that applies to most other engineering disciplines. You can take a course in civil engineering and have enough knowledge to reason about why a skyscraper will not fall apart in the wind; doesn't mean you have the ability to build one.

Plausible deniability - The line of attack is in their sessions/prompts data. Just make the prompt pointed enough that the search space is tractable and use your ginormous compute.

> "Of course we don’t know whether that is true"

Yep. Who is verifying these claims? We all know how trustworthy Altman & Co are.

Yeah, I think you can't just throw money randomly at problems and expect results unless you know a line of attack that can get you all the way. OpenAI chose the line of attack only after it became known to them via rumors. They "front-ran" the researchers.

Exactly! This is the real Occam's Razor explanation.

Are you saying there is no search space intractable to LLMs? That wouldn't be possible. AIs are statistical pattern-matchers on steroids. The prompt is key to getting anything useful out of them. They are incredibly useful and major game changers but ultimately that does not alter this fact. People (including OAI) have already tried to solve Millenium Problems with it. That OAI woke up last week and suddenly decided that throwing their researchers armed with millions of compute on one particular idea to a problem is highly suspicious in itself.

Even if OAI had zero data from Buckmaster's sessions, this is in very poor taste and highly unethical. You are front running a researcher just to be able to say you did it first? Tao is right - OAI is treating math results like oil. This is the like Exxon getting a whiff of a massive oil field and racing to the punch by deploying their full crew.

OAI started working on this only after they found out it was close to being solved. They threw a team of researchers who spent sleepless nights + a ton of compute. This is not exactly healthy academic competition - it's like if you spend a year hunting for oil fields and finally find a very promising area to be explored, only to find that Exxon tapped their entire exploration unit to go all in and and find it overnight just to stake claim to the discovery. Tao said it right - math should not be treated as a non-renewable resource to be mined.

OAI doesn't need to mention Buckmaster's name directly in a prompt. They just need to select a basket of sessions that is guaranteed to contain Buckmaster's and then direct the LLM to attack only a specific method/angle. This is trivial to do while maintaining plausible deniability about not using his work.

I'd think nothing is "safe". Anything you say can and will be used by the LLM if it has enough statistical similarity to the prompt. Call it "Ma Random Rights"

Research equivalent of front-running.

I don't think OAI should be given the benefit of doubt. They are doing the research equivalent of front-running. Knowing where to look is one of the main challenges in research. Tristan's argument from his essay was that it is hard to brute force with a vanilla prompt (even for seasoned mathematicians) unless you knew very specifically what to mention i.e the search space would have been intractable even for OAI's compute budget.

"deidentified data" isn't much to go by. Say I prompted the internal model this way - "Hey there's a solution to a unsolved problem X. The solution uses a less known Method Y so don't bother wasting time with the usual methods. Take papers A, B and C as references. Oh btw, here's the last year's worth of data of all prompt sessions that mention this problem. Pay special attention to the ones that mention Method Y and sub-keywords Z,W".

This is obviously all speculation but the timing is very suspect. If OAI actually did this (and I suspect whatever they did is pretty much close to this), I think it is highly unethical.

Try to increase the complexity of the work you're doing. It's not that AIs don't makes mistakes but they are able to pattern match to larger and larger pieces of the problem as more people use it and more iterations of the model are released. So your challenge is now in steering it in a way that these errors are minimised while reducing bloat.

Try to think of yourself as a professor who's trying to come up with a problem statement worth solving. The AI is your "lab" that will help you run experiments.

Yes, that does track with my personal experience. More context, more params and no quantization is probably it. But my hunch is that all the training data they've been getting in the past year also plays a part here. More than any other lab, anthropic's focus on coding right from the beginning gives them access to the best training data (several githubs worth). Most of this code comes with human feedback and anthropic even has data on how many went to production, got reverted etc. No need to pay for human labeling when your customers are doing it for you. This is their secret sauce.

>>> the model is (obviously) good at security

Out of curiosity, are you one of the people who has access to the model? If yes, could you write about your experimental setup in more detail?

Sure, I am not precluding the possibility that they've trained a genuinely great model. All I am saying is that the "this model better than that model" is moot when on one side you have model weights, and on the other side a whitepaper and some accompanying comments on the danger.

I'm not that old but have been here long enough that I remember when GPT-3 was considered too dangerous to release. Now you have models 10x as good, 1/10th the size and run on 8GB VRAM.

>> Test it yourself, GPT 120B OSS is cheap and available. BTW, this is why with this bug, the stronger the model you pick (but not enough to discover the true bug), the less likely it is it will claim there is a bug.

I guess this is the crux of the debate. All the claims are comparing models that are available freely with a model that is available only to limited customers (Mythos). The problem here is with the phrase "better model". Better how? Is it trained specifically on cybersecurity? Is it simply a large model with a higher token/thinking budget? Is it a better harness/scaffold? Is it simply a better prompt?

I don't doubt that some models are stronger that other models (a Gemini Pro or a Claude Opus has more parameters, higher context sizes and probably trained for longer and on more data than their smaller counterparts (Flash and Sonnet respectively).

Unless we know the exact experimental setup (which in this case is impossible because Mythos is completely closed off and not even accessible via API), all of this is hand wavy. Anthropic is definitely not going to reveal their setup because whether or not there is any secret sauce, there is more value to letting people's imaginations fly and the marketing machine work. Anthropic must be jumping with joy at all the free publicity they are getting.

Maybe they did use small models but you couldn't make the front page of HN with something like this until Anthropic made a big fuss out of it. Or perhaps it is just a question of compute. Not everyone has 20k$ or the GPU arsenal to task models to find vulnerabilities which may/may not be correct?

Unless Anthropic makes it known exactly what model + harness/scaffolding + prompt + other engineering they did, these comparisons are pointless. Given the AI labs' general rate of doomsday predictions, who really knows?

The word "profound" is a bit overused when it comes to movies. I agree that The Battle of Algiers is an excellent film, one of the best ever made even. One Battle After Another is also excellent but it is not really political in the way the TBoA is. It uses a political setting very effectively in a chase thriller. A movie like The Parallax View is a better comparison. That movie used the post-60s paranoia very effectively in a great suspense thriller.

Yeah, I'm watching a lot of Charlie Chaplin movies in preparation for my new role as a tramp.

I agree on principle. But there is going to be a painful transition where people are still reckoning with the new capabilities so I understand where all the fear/sadness is coming from.

Tbf I think the golden days of being a software dev are over even if the AI were to stagnate and never improve. The spectre of AGI is enough for higher ups to demand more output which will in turn require more hours to be put in by devs. A project that required 2 months will now be allotted 3 weeks because "Agentic coding increases productivity".

> I don't know if the market will have fully internalized that knowledge soon enough

This exactly. I am neither a boomer nor a doomer. It has helped a lot both at work and in accelerating my personal projects. But now that the C-suite and middle management has jumped on the agentic bandwagon, I'm unsure where this will go and what casualties ensue. At the very least, in the short term there's going to be a lot of "Now that we have agents, this project should be achievable in half the time".

This started decently enough and then the author went all over the place. I'm not sure why detailed explanations of neural nets and smart contracts were needed here. It really feels like trying to ram in a tech solution for what is effectively a market/social problem.

Using computers to aid in designing is not specific to Kanchipuram saris. While I realize people always approach it from the POV of saving a dying art, I'm unsure if K.saris can really fall under that umbrella. Clearly the demand is there and the issues here arise due to inefficient and possibly corrupt market practices rather than the art itself dying. A lot of space was used to explain the lopsided economics on the supply side but there's not enough attention paid to the demand side and the marketplace dynamics.

Excellent comment that really gets to the crux of the matter. Countries like China and India see themselves as civilizational, America sees itself as a perfect marketplace - it exists to feed its customers's wants and whims as efficiently as possible. I don't necessarily mean this in a demeaning way, it is what it is. In some sense, America is a state-level example of hedonic adaptation with its positives being improvements in quality of life and development of new tech, negatives being a bully in world politics, endless wars and bloodshed.

In general, hedonic adaption ends either with internal retrospection (shifting from pleasure to purpose) or an external disruption. In America's case, the former is extremely unlikely IMHO - the American people will not put their money where their mouth is because they enjoy the wealth generated this way. It will be upto external disruptors to check on Uncle Sam's endless thirst.

Rather than think of it as a pivot to hardware, I looked at it as MS trying to corner their share in the consumer market. Mobile and Social were the hot things back then and mobile threatened MS's dominance of the OS market. MS ultimately failed but they still owned the enterprise market and continue to keep their lead in desktop market share.

Ballmer doesn't strike me as an idiot and definitely not bland. He's one of the more colorful tech personalities. MS's almost unassailable lead in enterprise could be attributed to him and the pivot to cloud could not have happened without this. But he definitely fumbled hard on mobile (Windows Phone), Surface (IIRC the initial ARM laptop was a major flop and had a close to 1B+ writeoff) and the disaster that was the Nokia acquisition. I'd say he left at the right time, just as it was becoming clear that MS's bets on Windows Phone and hardware in general weren't paying off.

My point was more that MS hasn't had an industry changing product in a while. Google became joint-SOTA in AI and seems poised to take the crown with the next Gemini, and also in self-driving cars and quantum computing. They've kept their cash cows going while also being up to date on the tech that might upend their business model, so in a way they've cracked the innovator's dilemma which is definitely not an easy thing to do. A lot of HNers even wrote them off after ChatGPT and the disastrous Bard. Apple has a successful mass product in Airpods, a moonshot in Vision Pro and the insane Apple Silicon which they executed over more than a decade.

Nadella did well in the last decade to consolidate the MS stack (Teams, Azure, Office) and to invest in OpenAI when he realized MS's internal efforts wouldn't yield the expected output. He has protected their turf and made some strategic acquisitions like Linkedin and Github to keep their lead in enterprise software. From the POV of Wall Street performance and stock returns, he is a definitely a great CEO but so are Cook, Pichai even Ellison.

I often wonder why Satya Nadella is so venerated on HN compared to say, Cook or Pichai. As innovators, MS lags way behind both Google and Apple. I can't think of one bleeding edge product released during Satya's tenure. Say what you will about Apple and Google, they still consistently put out products that make you sit up and pay attention. What has MS been doing other than squeezing the MS Office and Azure cash cows?