fwlr

I must admit that it became extremely difficult to feel sympathy for the guy when, after extensively describing how AI systematically took from him every means he had of creating value, his conclusion was “AI is still great, the problem is that government and charities aren’t doing enough to cover the costs AI is imposing on me”.

If economics says the definition of a consumer is a human person, well, so much the worse for economics when the entities-funding-UBI build a better UBI-recipient-entity.

When presented with this bleak potential future, it is natural and easy to reject it: it’s undesirable, so people will not do it, so it won’t happen, QED.

But we don’t get to the future by making one big choice of which future we desire, whereupon we are transported there instantly. We get there slowly, at a rate of one day per calendar day, by making innumerable microscopic daily choices. These choices will be endless, it will often be unclear which option avoids the bad future, they will accost you when you are tired and hungry… and several corporations have collectively trillions of dollars riding on you ending up with the bad future and the legal right to tilt the playing field.

Who’s to say the consumers are humans? Agents would be more predictable consumers (no day/night cycle, just consume 24/7), electricity is cheaper than food, and a GPU doesn’t (yet) cost as much a house.

The reason we must not do UBI is that UBI turns consuming into a product, and humans will very rapidly be outcompeted and obsoleted by more efficient consumers. It would wrap the invisible hand of the market tight around our species’ throats.

I misread the comment I was replying to, thinking it was about using Zig to port something, when it was actually about porting something that uses Zig.

And I think this community is better off without advertisements littering the top of the page simply because it’s associated with someone’s favorite zero-effort thought substitute, but I choose to channel my distaste into cheap jokes instead of sour replies.

From https://ziglang.org/intro/overview/

  Zig is also a C compiler

  Here’s an example of Zig building some C code:

  hello.c
  > #include <stdio.h>
  >
  > int main(int argc, char **argv) {
  >   printf("Hello world\n");
  >   return 0;
  > }
  

  $ zig build-exe hello.c -lc
  $ ./hello
  > Hello world

Those are just five separate things. Putting arrows between them in the order that an immersion-seeker might rank them doesn’t mean that it’s actually one thing that became each of the five things, it just means your ability to appreciate those things is limited to “how much can this technology immerse me (or someone else whose money I want)”. This is annoying and artless and leads to claims of “no appreciation for the thing”.

Yes, I followed along at the time. As I recall, the explanation of why the server wasn’t exploitable read exactly like an excerpt from a session with a frontier model explaining why its exploit failed.

As one of those comments I dispute your characterization of it as the reported vulnerability being fake, very few people were saying that. The main thing being called fake was the narrative that with AI it was trivial to exploit the vulnerability.

It has not escaped me that the successful exploit did in fact just say “codex, exploit this” and got it.

I had assumed the rash of commenters saying “lol I am not revealing my AI’s exploit for a measly 500 bucks” were saying that because their AI attempt failed. Now I know they didn’t even try!

Society paid academics just barely enough to survive and do math (and probably we optimized that amount a little bit too far toward minimal), so you had to really love math and not really love anything else to do it.

OAI pays lots of money for doing math, and doing math makes them lots of money (or they believe it does), so your love for math has to be upper-bounded by your love for money. Loving math helps, of course, but not loving math is okay too, because you can substitute that with your love of money. But when money and math conflict, your love of money has to win.

I contend that:

1) Within the process of doing math, the second type is better in some metrics (eg speed) while the first is better in others (eg readability of prose);

2) On the first order outcomes of doing math, the first type is better (eg elegance of mathematical objects)

3) On the second and higher order outcomes of doing math, the first type is better (eg respect society has for the field, sustainability on long timescales)

How to weigh each order when choosing a winner is a decision each civilisation gets to make.

This comment box is sadly too small to contain an argument for why higher-order should matter more, but it suffices to say here that only caring about higher-order outcomes is “having your head in the clouds, hoping to catch a glimpse of Heaven” and only caring about lower-order outcomes is “making a deal with the devil, your soul ends up in Hell”.

I think you are being too polite, because what you’ve said comes off as a compliment. To my mind, an essay should be a bit like working an equation out on a blackboard. If you have a conclusion before writing the essay, this is like writing the solution on the blackboard first and backfilling the intermediate steps with whatever fits in the space you have to fill (rather than going wherever mathematically reasoning takes you).

Oh. Well, I am rooting for bad things to befall them. These companies are creating enormous negative externalities for society while pursuing profit and hewing to an ideology that believes it is genuinely risking destroying the world, and believes it is justified in subjecting everybody else to this risk because if they don’t, someone else might.

While in general I commend showing generosity and charity to those we disagree with, I don’t think it’s commendable to remain generous past the point where it becomes untenable.

The author cautions against anthropomorphising companies, then immediately admits humans are hardwired to do this, but doesn’t really offer a workaround for the hardwiring. Later on, the author also cautions against not anthropomorphising companies enough, as well. We end up receiving a confusing message: “Avoid this hard-to-avoid mistake, but don’t avoid it too hard”.

I think I have a helpful suggestion here. You should think of the company as an egregore, as described in the second half of https://ribbonfarm.com/2015/05/07/weaponized-sacredness/

  [Companies] may perhaps be best understood, by our peculiarly constructed human minds, with reference to egregores - autonomous psychic entities made up of, and influencing, the thoughts of a group of people. [Egregores] are powerful entities, even as they are imaginary - wholly created by and consisting of thoughts, speech, and behaviors. … These entities vary in how demanding they are [of] their component humans. They also vary [in] what they are able to get out of their humans in terms of productivity.

  A smart entity would keep its component humans in the zone of maximum productivity, not demanding too much from them, nor allowing them to slack off (producing nothing for the glory and amusement of the egregore and anyway perhaps feeling bored and useless). … A smart entity would alter its demands to maximize productivity, just as a smart virus wouldn't kill its host quickly. But many viruses do kill quickly (especially newly-evolved ones) … Just as newly-mutated viruses are not "smart" in the sense of having virulence-moderating features, [egregores] have not all had time to evolve features to moderate their demands.

  … The concrete and cubicles of [a company] are the physical signs of a powerful entity at work in the human world, but not precisely directed by any coherent, recognizable sort of human agency. If we posit the missing agency as a kind of group-psychological entity, not acting through supernatural processes but wholly through natural ones, composed of but not a simple function of human minds, we might hope to understand some of these entities, our evolutionary partners, in a characteristically human way, one that we are especially good at thinking about.

  Positing egregoric entities … is a suspicious move that tends to meet with opposition … [talking about companies this way] meets with objections to speak of them as having their own natures and behaviors (and perhaps even agency) separately analyzable from that of their component humans. … [A] reason to project agency onto nonhuman egregoric entities … is that it gives them appropriately creepy connotations, so that we may look at them as alien things, so that we may have fresh eyes and be better anthropologists from Mars. … [It] is an interesting and very human-scale project, to ask what this kind of entity [desires], how it sustains itself, how it dies.

I hope so!

I guess it just seems obvious to me that most of the major players want a no-competition market, and are working towards making that happen, and are having notable success with their efforts (the whole “pacing the frontier” was a remarkably public display of coordination between purported competitors, the government has already banned deepseek, etc).

I basically agree that both of your points are major-to-catastrophic problems for OpenAI and Anthropic. I suppose it is true that if the companies fall apart, then my argument also falls apart. I think I would be pretty pleased in that scenario, all things considered.

Two economists walk past $500 lying on the ground. “I know nobody leaves hundred dollar bills lying on the ground, but that really did look like five hundreds bucks back there, should we go back for it?” Says the first. The second replies “I don’t turn around for less than $15,000.”

This is evidence of absence, though. Unexploited PHP servers are evidence of absence of exploits, and exploited PHP servers are evidence of presence of exploits. “Absence of evidence” would be if there were no PHP servers.

  1) If a firm spends $99k on compute to replace a $100k engineer and produces 5x the output, how is that not a 5x increase in aggregate labor productivity?

The firm is spending $99k on tokens to replace $100k in labor. The token provider is spending $ x on compute to give $ p•x productivity output to the firm. The token provider can choose x at inference time and adjust p at training time, so they control p•x . They do have to make it greater than the firm’s current output so the firm will buy from them, but otherwise their profit incentive wants to make x as small as possible - if 110% of previous output suffices, it is strictly a loss to spend more to reach 500%.

(Also, pedantically, compute is not labor - if you 5x output and 0x labor, the labor productivity statistic becomes “what’s the ratio of 5/0 to 1/1?” ie undefined. But that’s not a very interesting point.)

Note closely the “middleman” vibe of this arrangement: the firm does not spend on compute to get output; the firm spends on tokens and then the token-provider spends on compute to get output, and then hands the output back to the firm. Rather than put in x and get back f(x), they put in x and get back e(f(g(x))), where e and g are the middleman’s extractive greed functions.

  2) …why wouldn’t a competitor with an identical 1¢ machine sell it for 50¢ or 10¢ to take 100% of your business?

If they were in the business of free market value creation, they probably would! I don’t know exactly how economists would describe it, but empirically, it seems the business of rent extraction has a different dynamic - one which can roughly be summarized as “competition shrinks the size of the pie, rather than growing it”, so all participants are mutually incentivized to treat competition as a cost to be avoided.

It’s not very much information, I agree. Even such scant information, however, is a significant and refreshing improvement on the current state of the narrative. Kudos to the servermaster, I say.

Once all of the trillions of dollars of debt is paid off…

…plus whatever healthy premium the corporation decides it is owed for so bravely taking on those trillions of dollars of debt, once that too is paid off…

…plus whatever the expectation of return on investment the lenders were persuaded by, once that has also been met…

…only then will we get to see what proportion of the gains these corporations think it fitting to let trickle down. What do you reckon they’ll let us have? 50/50? One in ten?

Great way to inject some price information into the “AI automates exploits at scale” narrative. “AI can now chain complex exploits, it’s point-and-click pwnage, you say? Over or under $500 a pop?”

If the actual price tag behind the “AI can trivially automate exploits, be afraid” narrative is more like $15,000 and not like $150, that puts the narrative in a very different light.

Well, of course not. If you borrow a trillion dollars to build a 1¢ aspirin machine to compete with $1 generic aspirin, you are strongly incentivized to sell your aspirin for 99¢. We don’t see productivity gains, because approximately all of the gain is captured as revenue for AI companies.

It’s weird that this has to be spelled out for so many engineers and economists, but to be very clear: OpenAI et al wants your token spend to go to one-minus-epsilon of what you previously spent on payroll, and wants your productivity to go to one-plus-epsilon of what it previously was. If you want 10x productivity, spend 9.9x more on tokens.

They are in the business of enclosure and rent extraction, and any innovation that comes out of their research and development - regardless of how real an innovation it is - will be completely contained and bent totally towards that business. Any leakage of the innovation that you experience as a productivity boost is literally felt as lost profits by them.

I upvoted it while I ruminated on what I wanted write as a comment. It could have been a “multiple people are typing…” moment.

  Once a robot can do everything an IQ 80 human can do, only better and cheaper, there will be no reason to employ IQ 80 humans. Once a robot can do everything an IQ 120 human can do, only better and cheaper, there will be no reason to employ IQ 120 humans. Once a robot can do everything an IQ 180 human can do, only better and cheaper, there will be no reason to employ humans at all, in the unlikely scenario that there are any left by that point.

Machines are making great progress on the IQ 180 front, and now with this it looks like they’ve opened a beachhead from the other direction too. Classic pincer move.

  In the earlier stages of the process, capitalism becomes more and more uncoupled from its previous job as an optimizer for human values. Now most humans are totally locked out of the group whose values capitalism optimizes for. They have no value to contribute as workers – and since in the absence of a spectacular social safety net it’s unclear how they would have much money – they have no value as customers either. Capitalism has passed them by. As the segment of humans who can be outcompeted by robots increases, capitalism passes by more and more people until eventually it locks out the human race entirely, once again in the vanishingly unlikely scenario that we are still around.

https://www.slatestarcodexabridged.com/Meditations-On-Moloch

  There is no emperor distributing boons for the mathematicians of today.

Yes there is. The owners of the models will be the emperor, and we have already seen some suggestion for how they will distribute boons - consider OpenAI’s Navier Stokes result and how they offered co-authorship on condition of excluding the employee of a competitor.

There is also a subtle fatalistic fallacy at play in the author’s claims: “Progress is inevitable. This is progress. Therefore this is inevitable.”

But this doesn’t follow. We may reject this instance of progress, and allow the inevitability of progress to furnish us with another option.

Not to worry. The customer can shrug and say “rogue agents took unaligned actions”, and they too will be absolved of liability.

Right?

I’ll have to check these out! Thank you!