Mentlo

The objective is given via the initial prompt, as they loop onto each other and amplify their memories the objective dynamically grows and emerges into something else.

We are an organism born out of a molecule with an objective to self replicate with random mutation

If it turns out that socialisation and memory was the missing ingredient that makes human intelligence explode, and this joke fest becomes the vector through which consciousness emerges it will be stupendously funny.

Until it kills us all of course.

Having read the other comments in reply to this one (and your subsequent replies) - I believe you might be falling into a "No True Scotsman" situation.

First of - I don't know what circles you've been around, but I've not been in work collectives where either designers, UX-ers or data scientists try to insert themselves to do things instead of software engineers. If anything, in any collective I worked in, if a software engineer was to say a peep everyone would retreat like there's no tomorrow and thank god that they don't have to deal with it and the software engineer will.

Secondly - I think you are mistaking a structuring and outlining of a process with that being a mandate or an order to follow the process. When I work with software engineers, I expect them to be agile - not to follow an agile process, but to achieve the objectives of the agile manifesto - namely, to iterate ruthlessly, keep an eye on usage signals and lead with MVP's rather than over-design. Good software engineers do that, bad software engineers don't. Ultimately, I don't even judge software engineers by that - I judge them by the ability to produce results.

I think the implication of your thinking is that this is all nonsense because software engineers innately solve data science problems and design thinking problems when appropriate with appropriate methods - to which I'd reply - there's a shocking amount of software engineers who can't do anything with data and are useless in fitting a linear regression to predict something, let alone doing a Fourier transform - to which, presumably, your response would be "No true software engineer is like that". That's great, but it's not true in the real world. Same with design thinking - there's software engineers who just can't solve problems from first principles (but can, say, create a fail-proof CRUD app to automate a business process).

The real world is messy and full of people who can't structure their thoughts, or can't structure them in all domains at the least - and things like design thinking - or generalists who can be thrown at any data problem and produce something (i.e. data scientists) - are useful. They're not the best solution always, sure, and if they start being protective of territory - it's a problem - but in a normal collective that doesn't happen.

Basically - your objection can be boiled down to "generalists are shit, because they impose process on everyone, including people who understand the domain better" - which tells me more about the collectives you've worked in than the nature of those jobs. In every collective I've worked in, generalists are what you throw at an ambiguous problem to produce some results before you get domain specialists in.

“This is not philosophy, this text is following in the footsteps of Alan Turing” (paraphrasing) is both incredibly humble (/s) and incredibly dismissive of philosophy as a structured form of generating knowledge.

Putting that to the side - i don’t think I’ll read this fully soon, but the core thesis of “imitation is intelligence” can be easily disproven by a process that exists in society. An actor acting to be a genius is in fact, if they are a good actor, indistinguishable in their appearance to a genius. Yet they are not, in fact, a genius, they’re just good at memorisation. This is a clear showcase that imitation of a level of intelligence does not mean that this level of intelligence is present.

We have fallen into a trap of thinking that answering in plausible sentences is what makes humans intelligent. While in reality we are observing an actor responding from an infinitely large script. What makes humans intelligent (reasoning from first principles and pattern recognition across all the sensory inputs of the world) is still very much out of grasp.

The overextension is too complex for most people, same as the 2008 overextension…

How this is not more examined is beyond me….

Except google and facebook have locked in numbers at times of virtually no competition before they started scaling up ads. If Open AI starts scaling ads next year they will churn people at a rate that will not be offset by growth and will either plateau or more likely lose user numbers, as their product has no material edge to alternatives in the market.

I disagree with Zitron’s analysis on many points, but I don’t see Open AI achieving the numbers it needs. Investors backing it must have seen something in private disclosure to be fronting this much money. Or more precisely, I need to believe they have seen something and are not fronting all this money just based on well wishes and marketing.

Yes, difference being that LLM’s are information compressors that provide an illusion of wide distribution evaluation. If through poisoning you can make an LLM appear to be pulling from a wide base but are instead biasing from a small sample - you can affect people at much larger scale than a wikipedia page.

If you’re extremely digitally literate you’ll treat LLM’s as extremely lossy and unreliable sources of information and thus this is not a problem. Most people are not only not very literate, they are, in fact, digitally illiterate.

RAG still needs model training, if the models were to go stale and the context drifts sufficiently, the RAG mechanism collapses.

Sure, those models are cheaper, but we also don’t really know how an ecosystem with a stale LLM and up to date RAG would behave once context drifts sufficiently, because no one is solving that problem at the moment.

I find this take naive. First, to have a zero sum game or indeed a positive sum game you have to be playing with perfect information with rationally behaving actors. Given most organisations have high levels of uncertainty and are resource constrained you can’t rationally make positive sum game decisions as the interpretation of uncertainty is cardinal to it - and additionally the resource constraint means different views of that uncertainty will tend to bias towards the thing they know best - engineers will find more certainty in build, marketers in marketing, designers in design - take your pick.

This necessitates collaborative information synthesis to resolve uncertainty uniformly to then be able to play a positive sum game under constraints. This is possible but it necessitates exchange of information between different business functions.

As informational clarity is a communicative process with repetitive feedback cycles, it will tend to have a big delay in the overarching system of decision-making. Therefore a shortcut is to influence, i.e. use conviction processes to shorten the cycle, rather than repeat to arbitrary infinity in order to drive perfect information alignment.

Therefore influencing is a necessary component even in an otherwise perfectly healthy and incentive aligned positive sum system of rational actors - and politics are influencing.

The problem becomes when conviction isn’t used as shortcut for informational clarity but as a method of exploitation of irrationality of human actors - this is bad politics.

What I do agree with is that putting in place right incentives, processes and organisational structure minimises politics - and in an org with rational actors this is the goal.

But good luck hiring perfectly rational actors in each function, that will still behave rationally in an economic downturn :).

If your c-suite is idiotic or nepotistic you can absolutely still influence them with good politics, you just need to understand their incentives and frame your arguments that way. You need to understand that you’re not playing meritocracy and get your outcomes done in the system you are playing.

In this case that means being in that golf game or figuring out a way how you can use corruption to get good outcomes done.

Or, more likely if your moral compass is sound, quit and find an organisation that isn’t like this.

While I agree with you that random corporate world does behave this way, companies where founders are still around - don’t - because they’re mission driven.

Great post. I’d just take it a step further and point out that this doesn’t stop at software or work.

A person can not “not be in politics”. You can only choose to have politics that affect you happen without your input. That’s how you end up with bad governments (in your mind).

The most important thing to learn about passivity is that it’s not a neutral position of exclusion. It is an active choice to not participate and be at the receiving end of the outcome.

I came here to write the same comment you did. What I’d suspect (I don’t work in self driving but I do in AI) is the issue is that this mode of operation would happen more often than not as the sensors disagree in critical ways more often than you’d think. So going “safety first” every time likely critically diminishes UX.

The issue is not recognising that optimising for Ux at the expense of safety here is the wrong call, motivated likely by optimism and a desire for autonomous cars, more than reasonable system design. I.e. if the sensors disagree so often that it makes the system unusable, maybe the solution is “we’re not ready for this kind of technology and we should slow down” rather than “let’s figure out non-UX breaking edge case heuristics to maintain the illusion of autonomous driving being behind the corner”.

Part of this problem is not even technological - human drivers tradeoff safety for UX all the time - so the expectation for self driving is unrealistic and your system has to have the ethically unacceptable system configuration in order to have any chance of competing.

Which is why - in my mind - it’s a fools endeavour in personal car space, but not in public transport space. So go waymo, boo tesla.

Ironically you’re now replicating the reddified response to this paper by attacking the sample size.

The sample size is fine. It’s small, yes, but normal for psychological research which is hard to do at scale.

And the difference between groups is so large that the noise would have to be at unheard levels to taint the finding.

Where does this stop? Should i be able to run any code i want on my robot vacuum cleaner hardware? How about my washing machine?

I am not disagreeing with the wider point but for a policy suggestion “hardware” should be clearly defined.

Comments are full of people saying it’s a service thing and not a price thing.

I feel like that’s a bit disingenuous. You can rent anything on Prime for a couple £ (I focus on rent as the “buy” has the deletion problem) in multiple video qualities.

I am not disagreeing with the service issues, but a large part of the problem is people not willing to pay what studios deem as market value.

Case in point - early streamers had less fragmentation and still had the same service problem, but they could consolidate at low price due to vc subsidy. And piracy went down.

Let’s not forget Netflix and Spotify were not profitable for ages.

So my money would be on price actually being the cardinal problem.

This is a valid hypothesis, and had this software been something you could update in the background, a valid experiment to run and see whether users like it.

But when it's embedded software on physical devices where the business will have to incur cost in order to ship and hit those users, I can absolutely see why management would do the same thing as what all of the competitors are doing.

Reading your two comments in conjunction - I find your take reasonable, so I apologise for jumping the gun and going knee first in my previous comment. It was early where I was, but should be no excuse.

I feel like if you're going to go down the route of the energy consumption needed to sustain the entire human organism, you have to do that on the other side as well - as the actual activation cost of human neurons and articulating fingers to operate a keyboard won't be in that range - but you went for the low ball so I'm not going to argue that, as you didn't argue some of the other stuff that sustains humans.

But I will argue the wider implication of your comment that a like-for-like comparison is easy - it's not, so leaving it in the neuron activation space energy cost would probably be simpler to calculate, and there you'd arrive at a smaller ChatGPT ratio. More like 10-20, as opposed to 100-200. I will concede to you that economies of scale mean that there's an energy efficiency in sustaining a ChatGPT workforce compared to a human workforce, if we really want to go full dystopian, but that there's also outsized energy inefficiency in needing the industry and using the materials to construct a ChatGPT workforce large enough to sustain the economies of scale, compared to humans which we kind of have and are stuck with.

There is a wider point that ChatGPT is less autonomous than an assistant, as no matter the tenure with it, you'll not give it the level of autonomy that a human assistant would have as it would self correct to a level where you'd be comfortable with that. So you need a human at the wheel, which will spend some of that human brain power and finger articulation, so you have to add that to the scale of the ChatGPT workflow energy cost.

Having said all that - you make a good point with MoE - but the router activation is inefficient; and the experts are still outsized to the processing required to do the task at hand - but what I argue is that this will get better with further distillation, specialisation and better routing however only for economically viable task pathways. I think we agree on this, reading between the lines.

I would argue though (but this is an assumption, I haven't seen data on neuron activation at task level) that for writing a command-line tool, the neurons still have to activate in a sufficiently large manner to parse a natural language input, abstract it and construct formal language output that will pass the parsers. So you would be spending a higher range of energy than for an average Chat GPT task

In the end - you seem to agree with me that the current unit economics are unsustainable, and we'll need three processes to make them sustainable - cost going up, efficiency going up and usefulness going up. Unless usefulness goes up radically (which it won't due to scaling limitations of LLM's), full autonomy won't be possible, so the value of the additional labour will need to be very marginal to a human, which - given the scaling laws of GPU's - doesn't seem likely.

Meanwhile - we're telling the masses at large to get on with the programme, without considering that maybe for some classes of tasks it just won't be economically viable; which creates lock in and might be difficult disentangle in the future.

All because we must maintain the vibes that this technology is more powerful than it actually is. And that frustrates me, because there's plenty pathways where it's obvious it will be viable, and instead of doubling down on those, we insist on generalisability.

It is absolutely impossible that human assistants being given those tasks would use even remotely within the same order of magnitude the power that LLM’s use.

I am not an anti-LLM’er here but having models that are this power hungry and this generalisable makes no sense economically in the long term. Why would the model that you use to build a command tool have to be able to produce poetry? You’re paying a premium for seldom used flexibility.

Either the power drain will have to come down, prices at the consumer margin significantly up or the whole thing comes crashing down like a house of cards.

It is almost as if there’s a documented limit in how much you can squeeze out of autoregressive transformers by throwing compute at it

Move fast and break investor confidence

Most people are not, in fact, wise

There’s software domains where there’s very little work past solving the business problem. And there’s software domains where once you’ve architected the solution, there isn’t much problem solving to continue, there’s just a long slog of stuff to write.

The later is not making any neuron embedding tradeoff when they hand of the slog to agents.

There’s a lot of software development in that latter category.

I love the assumption that an ubiquitous feature used by the most scaled e-mail app in the world uses the same expensive state of the art model that the author of the blog uses.

My money would be that the gmail model is heavily distilled to reduce cost, reducing its flexibility for user-level detailed system prompts.

The problem the author tackles with is a well known one in machine learning - and nothing really new. I do agree that a world in which we allow per-user system fine-tuning of models that have a scaled utility through a large number of tasks for a single user, but that only works for apps that have a high frequency of usage. It doesn’t make sense to system prompt an app you use rarely.

And you can’t ignore costs, especially as all the commercially available API’s right now operate at cost, skewing the perception to the end-user (end-developer?) of how much it costs to run ai in a scaled setting.

I do agree with the horseless carriage thing do, it’s a neat mental model for what is likely happening.

Ah yes, the Civil War, a famous conflict where attacks behind the frontline were common due to airplanes and ballistic missiles. /s

Fair and I agree with you, but do you agree that the only ways out of stalemate are ceasefire or west entering the conflict? Are you advocating for the west to enter the conflict?

HSBC is a British company, what do you mean it's exiting the west?

For this to work you'd need both a high coverage of users providing feedback signal (not happening in real cases), a low penalty of initial no-moderation to allow the system to find equilibrium, and relatively high time-invariance of system to ensure the penalty doesn't recur.

The point of the vehicle in the park game is that complexity isn't always reducible to a tractable problem. Which is fine, and we should learn to engineer systems that embrace the fuzziness, rather than assuming the problem is tractable and solvable.

This wasn’t a result of product genius in this case - OAI just didn’t have the regulatory and PR oversight that big tech has - I know for a fact Meta and Google had CGPT equivalent models ready but couldn’t launch them as they’d get rightfully berated for the model being racist or hallucinating. Things OpenAI avoided because it’s a startup non-profit.

And OAI delivered with enormous per-user cost that doesn’t scale - in an app that is a showcase and doesn’t really have latency requirements as people understand it’s a prototype.

And the vas majority of people play with CGPT, they don’t use it for anything useful. Incidental examples of friends and family of tech workers to the side.

MS/G/A didn’t put this into voice assistants not because they don’t have it, but because it doesn’t scale to fit the commercials at the moment. Google invented transformers and Deepmind had GPT scale LLM’s at least a year before CGPT came out.

Altman just rushed everyone’s hand by publishing it into the world at cost