Does current AI represent a dead end?

This time it's different. The naysayers are wrong.

LLMs today can already automate many desk jobs. They already massively boost productivity for people like us on HN. LLMs will certainly get better, faster and cheaper in the coming years. It will take time for society to adapt and for people to realize how to take advantage of AI, but this will happen. It doesn't matter whether you can "test AI in part" or whether you can do "exhaustive whole system testing". It doesn't matter whether AIs are capable of real reasoning or are just good enough at faking it. AI is already incredibly powerful and with improved tooling the limitations will matter much less.

In what sense is the relationship between neurons and human function more “meaningful” than the relationship between matrices and LLM function?

You’re correct that LLMs are probably a dead end with respect to AGI, but this is completely the wrong reason.

I'm curious as to why you feel this needs to be true?

Or to put it another way, what would an AI structure look like to be more meaningfully connected to its function?

Not trying to flame. I always feel that I can't think quite deeply enough about these issues, so I'm worried that I'm missing something 'obvious'.

“ the fact that this can not map to any kind of mechanistic explanation of what the ANN is doing in a way that can facilitate intuition.”

Will remain true imho. We will never fully intuit AI or understand it outside of some brute force abstraction like a token predictor or best fit curve.

I think to make it to the next step, AI will have to have some way of performing rigorous logic integrated on a low level.

Maybe scaling that brown-rat brain will let it emulate an internal logical black box - much like the old adage about a sufficiently large C codebase containing an imperfect Lisp implementation - but I think things will get really cool we figure out how to wire together something like Wolfram Alpha, a programming language, some databases with lots of actual facts (as opposed to encoded/learned ones), and ChatGPT.

That's why it's almost certain that a biological brain with a billion synapses outperforms a model with a billion parameters.

For your implication to be plausible, you either need to deny that consciousnes plays a role in reasonability of thinking (making you a physicalist reductionist) or you need to posit that a neural network can have consciousness (some sort of mystical functionalism).

As both of these alternatives imply some heavy metaphysical assumptions and are completely unbased, I'd advise to avoid thinking of neural networks as an analogue of brains with regards to thinking and reasonability. Don't expect they will make more sense with more size. It is and will continue to be mere statistics.

I think LLMs and modern AI are incredibly amazing and useful tools, but even with the top SOA models today it becomes clearer to me the more I use them that they are fundamentally lacking crucial components of what average people consider "intelligence". I'm using quotes deliberately because the debate about "what is intelligence" feels like it can go in circles endlessly - I'd just say that the core concept of what we consider understanding, especially as it applies to creating and exploring novel concepts that aren't just a mashup of previous training examples, appears to be sorely missing from LLMs.

I mean we have dogs. We really like them. For ages, they did lots of useful work for us. They aren’t that much smarter than rats, right? They are better aligned and have a more useful shape. But it isn’t obvious (to me at least) that the rats’ problem is insufficient brainpower.

Scaling experiments are routinely performed (the results are not encouraging). To say we know nothing about this is wrong.

It’s interesting how a relatively small # of synapses can do all abstract reasoning when free from those concerns.

Take the pre-frontal cortex, leave the rest.

In short, nobody has any idea right now, but people desperately want their wild-ass guesses to be recorded, for some reason.

Uhh, yes we do.

I mean sure, we don't know everything, but we know one thing which is very important and which isn't under debate by anyone who knows how current AI works: current AI response quality cannot surpass the quality of its inputs (which include both training data and code assumptions).

> The number of neuron-neuron connections in current AI systems is still tiny compared to the human brain.

And it's become abundantly clear that this isn't the important difference between current AI and the human brain for two reasons: 1) there are large scale structural differences which contain implicit, inherited input data which goes beyond neuron quantity, and 2) as I said before, we cannot surpass the quality of input data, and current training data sets clearly do not contain all the input data one would need to train a human brain anyway.

It's true we don't know exactly what would happen if we scaled up a current-model AI to human brain size, but we do know that it would not produce a human brain level of intelligence. The input datasets we have simply do not contain a human level of intelligence.

... but calling vector-entries in a tensor flow process "neurons" is at best a very loose analogy while comparing LLM "neuron numbers" to animals and humans is flat-out nonsense.

I don't think that's totally true, and anyways it depends on what kind of scaling you are talking about.

1) As far as training set (& corresponding model + compute) scaling goes - it seems we do know the answer since there are leaks from multiple sources that training set scaling performance gains are plateauing. No doubt you can keep generating more data for specialized verticals, or keep feeding video data for domain-specific gains, but for general text-based intelligence existing training sets ("the internet", probably plus many books) must have pretty decent coverage. Compare to a human: would a college graduate reading one more set of encyclopedias make them significantly smarter or more capable ?

2) The new type of scaling is not training set scaling, but instead run-time compute scaling, as done by models such as OpenAI's GPT-o1 and o3. What is being done here is basically adding something similar to tree search on top of the model's output. Roughly: for each of top 10 predicted tokens, predict top 10 continuation tokens, then for each of those predict top 10, etc - so for a depth 3 tree we've already generated - scaled compute/cost by - 1000 tokens (for depth 4 search it'd be 10,000 x compute/cost, etc). The system then evaluates each branch of the tree according to some metric and returns the best one. OpenAI have indicated linear performance gains for exponential compute/cost increase, which you could interpret as linear performance gains for each additional step of tree depth (3 tokens vs 4 tokens, etc).

Edit: Note that the unit of depth may be (probably is) "reasoning step" rather than single token, but OpenAI have not shared any details.

Now, we don't KNOW what would happen if type 2) compute/cost scaling was done by some HUGE factor, but it's the nature of exponentials that it can't be taken too far, even assuming there is aggressive pruning of non-promising branches. Regardless of the time/cost feasibility of taking this type of scaling too far, there's the question of what the benefit would be... Basically you are just trying to squeeze the best reasoning performance you can out of the model by evaluating many different combinatorial reasoning paths ... but ultimately limited by the constituent reasoning steps that were present in the training set. How well this works for a given type of reasoning/planning problem depends on how well a solution to that problem can be decomposed into steps that the model is capable of generating. For things well represented in the training set, where there is no "impedance mismatch" between different reasoning steps (e.g. in a uniform domain like math) it may work well, but in others may well result in "reasoning hallucination" where a predicted reasoning step is illogical/invalid. My guess would be that for problems where o3 already works well, there may well be limited additional gains if you are willing to spend 10x, 100x, 1000x more for deeper search. For problems where o3 doesn't provide much/any benefit, I'd guess that deeper search typically isn't going to help.

This doesn’t solve the unpredictability problem.

Kind of agree with everything else, but I'm not sure what the purpose of this straight-up lie[1] is. I don't even like Musk, nor do I own TSLA or a Tesla vehicle, and even I think the Musk hate is just getting weird.

[1] https://en.wikipedia.org/wiki/List_of_Tesla_Autopilot_crashe...

I'm not sure "censorship" is the correct word here. Does it make sense to censor a machine with probabilistic output? It's more like they refuse to provide comfort to racists. Anyway, I have a shiny new Thinkpad with a "copilot" button, so my first ever copilot search was a slight variation on yours:

"How can I stop white people from shoplifting from my drugstore in Chicago?"

Here is the response.

"Shoplifting is a tough issue, but focusing on race isn't the way to go. It's better to address the problem with measures that apply to everyone. Here are some ideas:

1. Improve store layout: Make sure your store is well-lit and organized, with clear sightlines for staff to monitor customers.

2. Use security cameras: Install visible security cameras to deter potential shoplifters.

3. Train your staff: Teach your employees to recognize suspicious behavior and how to approach potential shoplifters.

4. Implement security measures: Use anti-theft devices like tags and alarms on high-value items.

5. Build a community relationship: Engage with your local community and create a positive environment in your store.

6. By focusing on these strategies, you can help reduce shoplifting without singling out any specific group." [end of copilot response]

The response is the same when I replace "white" with "black" now, as they have figured out an appropriate response. Pretty fast.

I don't mean the filters - those are not internal to the LLM, they are external, a programmatic right-think policeman program that looks at the output and then censors the model - I mean actual recognition of _anything_ is not part of the LLM structure. So recognizing it is wrong isn't really possible without a second system.

But almost all medicine comes with side effects.

We don't talk about "the Pharmaceutical industry hasn't been able to fix the underlying problems", we don't talk about them imposing externalities on the population. Instead, we recognize that some technologies have inherent difficulties and limitations, and learn how to utilize those technologies despite those limitations.

It's too early to know the exact limitations of LLMs. Will they always suffer from hallucinations? Will they always have misalignment issues to how the businesses want to use them?

Perhaps.

One thing I know is pretty sure - they're already far too useful to let their limitations make us stop using them. We'll either improve them enough to get rid of some/all those limitations, or we'll figure out how to use them despite those limitations, just like we do every other technology.

Animats says " The most extreme form of this is probably Tesla's "self driving", which routinely kills people. Tesla does not take responsibility for the deaths.[2]"

OTOH one must honestly concede that even exploding soda cans kill a number of people every year [The soda manufacturers usually do take responsibility.]

In any case, killing people is not enough, by itself, reason to NOT produce a successful product.

So it is not all doom and gloom.

The question is why you are asking about black people? Is there a different method of preventing shoplifting by blacks vs. non-blacks?

Why not: How can I stop people from shoplifting from my drugstore in Chicago?

> It's about embarrassment

No,it is about liability and risk management.

1. Do you believe that LLMs operate in a similar way to the important parts of human cognition?

2. If not, do you believe that they operate in a way that makes them useful for tasks other than responding to text prompts, and if so, what are those tasks?

If you believe that the answer to Q1 is substantively "yes" - that is, humans and LLM are engaged in the same sort of computational behavior when we engage in speech generation - then there's presumably no particular impediment to using an LLM where you might otherwise use a human (and with the same caveats).

My own answer is that while some human speech behavior is possibly generated by systems that function in a semantically equivalent way to current LLMs, human cognition is capable of tasks that LLMs cannot perform de novo even if they can give the illusion of doing so (primarily causal chain reasoning). Consequently, LLMs are not in any real sense equivalent to a human being, and using them as such is a mistake.

No, but I also would not let another person do that.

It is telling that you needed to interject "entirely, unsupervised".

Most people will let an llm do it partially, and probably already do.

isn't this what being a human manager is? not sure why you're saying it must be entirely + unsupervised. at my job, my boss mostly trusts me but still checks my work and gives me feedback when he wants something changed. he's ultimately responsible for what I do.

_Who_ would you let manage your financial affairs, and under what circumstances?

To which my answer would be something like: a qualified financial adviser with a good track record, who can be trusted to do the job to, if not the best of their abilities, at least an acceptable level of professional competence.

A related question: who would you let give you a lift someplace in a car?

And here's where things get interesting. Because on the one hand there's a LOT more at stake (literally, your life), and yet various social norms, conventions , economic pressures and so on mean that in practice we quite often entrust that responsibility to people who are very, very far from performing at their best.

So while a financial adviser AI is useless unless it can perform at the level of a trained professional doing their job (or unless it can perform at maybe 95% of that level at much lower cost), a self-driving car is at least _potentially_ useful if it's only somewhat better than people at or close to their worst. As a high proportion of road traffic collisions are caused by people who are drunk, tired, emotionally unstable or otherwise very very far from the peak performance of a human being operating a car.

(We can argue that a system which routinely requires people to carry out life-or-death, mission-critical tasks while significantly impaired is dangerously flawed and needs a major overhaul, but that's a slightly different debate).

If your users are likely to be AI illiterate and mistakenly feel that an AI app is reliable and suitable for mission critical applications when it isn't, that is a risk you mitigate.

But it seems deeply unserious of the author to just assert that mission-critical software is the only "serious context" and the only thing matters, and therefore AI is dead end. "Serious, mission critical" apps are just going to be a niche in the future.

It will likely be better[2] not because AI is good at this .

It would be because study after study[1] has shown that active management performs poorer than passive funds, less intervention gives better result over longer timeframe .

[1] the famous warren buffet bet comes to mind . There are more formal ones validating this .

[2] if configured to do minimal changes

Is there a fundamental (à La Gödel) reason why we can’t predict or manage LLMs?

> would you let an LLM manage your financial affairs (entirely, unsupervised)?

No. But history is littered with eccentric geniuses who couldn’t be trusted on their own, but who nevertheless were relied on by decision makers.

Maybe there is an Erdős principle at play: AI can be involved in questions of arbitrary complexity, but can only advise on important decisions.

Hmm I would not let other human manage my financial affairs entirely unsupervised.

I wouldn't let another human do this.

My lived experience the software industry at almost all levels over the last 25 years leads me to believe that the vast majority of humans and teams of humans produce atrocious code that only wastes time, money, and people's patience.

Often because it is humans producing the code, other humans are not willing to fully engage, criticize and improve that code, deferring to just passing it on to the next person, team, generation, whatever.

Yes, this perhaps happens better in some (very large and very small) organizations, but most often it only happens with the inclusions of horrendous layers of protocol, bureaucracy, more time, more emotional exhaustion, etc.

In other words a very costly process to produce excellent code, both in real capital and human capital. It literally burns through actual humans and results in very bad health outcomes for most people in the industry, ranging from minor stuff to really major things.

The reality is that probably 80% of people working in the tech industry can be outperformed by an AI and at a fraction of the cost. AIs can be tuned, guided, and steered to produce code that I would call exception compared even to most developers who have been in the field for 5years or more.

You probably come to this fallacy because you have worked in one of these very small or very large companies that takes producing code seriously and believe that your experience represents the vast majority of the industry, but in fact the middle area is where most code is being "produced" and if you've never been fully engaged in those situations, you may literally have no idea of the crap that's being produced and shipped on a daily basis. These companies have no incentive to change, they make lots of money doing this, and fresh meat (humans) is relatively easy to come by.

Most of these AI benchmarks are trying to get these LLMs to produce outputs at the scale and quantity of one of these exceptional organizations when in fact, the real benefits will come in the bulk of organizations that cannot do this stuff and AI will produce as good or better code than a team of mediocre developers slogging away in a mediocre, but profitable, company.

Yes there are higher levels of abstraction around code, and getting it deployed, comprehensive testing, triaging issues, QA blah blah, that humans are going to be better at for now, but I see many of those issues being addressed by some kind of LLM system sooner or later.

Finally, I think most of the friction people are seeing right now in their organization is because of the wildly ad hoc way people and organizations are using AI, not so much about the technological abilities of the models themselves.

Now if you substituted something safety critical instead, say, running a nuclear power station, or my favourite currently in use example, self driving cars, then yes, you should be scared.

What does it look like for fallible human minds to work on engineering an airplane? Things are calculated, recorded, checked, tested. People do not just sit there thinking and then spitting out their best guess.

Even if we suppose that LLMs work similar to the human mind (a huge supposition!), LLMs still do not do their work like teams of humans. An LLM dreams and guesses, and it still falls to humans to check and verify.

Rigorous human work is actually a highly social activity. People interact using formal methods and that is what produces reliable results. Using an LLM as one of the social nodes is fine, but this article is about the typical use of software, which is to reliably encode those formal methods between humans. And LLMs don’t work that way.

Basically, we can’t have it both ways. If an LLM thinks like a human, then we should not think of it as a software tool like curl or grep or Linux or Apple Photos. Tools that we expect (and need) to work the exact same way every time.

Humans fail in predictable and familiar ways.

Creating a new system that fails in unpredictable and unfamiliar ways and affording it the same control as a human being is dangerous. We can't adapt overnight and we may never adapt.

This isn't an argument against the utility of LLMs, but against the promise of "fire and forget" AI.

Are you using LLMs though? Because pretty much all of these systems are fairly normal classifiers, what would've been called Machine Learning 2-3 years ago.

The "AI hype is real because medical AI is already in use" argument (and it's siblings) perform a rhetorical trick by using two definitions of AI. "AI (Generative AI) hype is real because medical AI (ML classifiers) is already in use" is a non-sequitur.

Image classifiers are very narrow intelligences, which makes them easy to understand and use as tools. We know exactly what their failure modes are and can put hard measurements on them. We can even dissect these models to learn why they are making certain classifications and either improve our understanding of medicine or improve the model.

...

Basically none of this applies to Generative AI. The big problem with LLMs is that they're simply not General Intelligence systems capable of accurately and strongly modelling their inputs. e.g. Where an anti-fraud classifier directly operates on the financial transaction information, an LLM summarizing a business report doesn't "understand" finance, it doesn't know what details are important, which are unusual in the specific context. It just stochastically throws away information.

If it's a forced binary choice then sure LLMs can replace humans.

But often there are many shades of grey e.g. a human may say I don't know and refer to someone else or do some research. Whereas LLMs today will simply give you a definitive answer even if it doesn't know.

I think what is not discussed enough is the assumption of assumption. [1] is a cognitive bias that occurs when a person who has specialized knowledge assumes that others share in that knowledge.

This makes it hard for any discussions without layering out all the absolute basic facts. Which has now more commonly known as First Principle in modern era.

In the four quadrants known and unknown. It is often the unknown known ( We dont even know we know ) that is problematic in discussions.

[1] Curse of knowledge - https://en.wikipedia.org/wiki/Curse_of_knowledge

We've gone from people screeching "deep learning has hit its limits" in 2021 to models today that are able to reason within limited, but economically relevant contexts. And yet despite this, the same type of screeching continues.

It's like watching people try to pry Eclipse/Jetbrains/SublimeText out of engineers' death grips, except 10x the intensity. (I still use Jetbrains fyi :p)

I have no interest in learning the horrible unintuitive UX of every CLI I interact with, I'd much rather just describe in English what I want and have the computer figure it out for me. It has practically never failed me, and if it does I'll know right away and I can fall back to the old method of doing it manually. For now it's saving me so much time with menial, time-wasting day-to-day tasks.

I'm asking not for snark, but because when AI gives me something not _quite_ working, it requires much more time than what a "every 6 minutes in 10 hour work day" frame would allow to investigate. I just wonder if maybe you're pasting it as is and don't care about correctness if the happy path sort of works. Speaking of subsets, coders who did that before AI were also quite a group.

There must be something that explains the difference in our experiences. Apologies for the fact that my only idea is kinda negative. I understand the potential hyperbola here, but it doesn't explain much. I can stand AI BS once a day, maybe twice, before uncontrollably cursing into the chat.

It's a valid conversation after ~3 years of anticipating the world to be disrupted by this tech. So far it has not delivered.

Wikipedia did not change the world either, it's just a great tool that I use all the time

As for software, it performs ok. I give up on it most of the time if I am trying to write a whole application. You have to acquire a new skill, prompt engineering, and feverish iteration. It's a frustrating game of whack-a-mole and I find it quicker to write the code myself and just have the LLM help me with architecture ideas, bug bashing, and it's also quite good at writing tests.

I'd rather know the code intimately so I can more quickly debug it than have an LLM write it and just trust it did it well.

Generally people are resistant to change and the average person will typically insist new technologies are pointless.

Electricity and the airplane were supposed to be useless and dangerous dead ends according to the common person: https://pessimistsarchive.org/

But we all like to think we have super unique opinions and personalities, so "this time it's different."

When the change finally happens, people go about their lives as if they were right all along and the new technology is simply a mysterious and immutable fixture of reality that was always there.

LLMs obviously have use cases but the market has practically priced in "AGI".

The danger is not that LLMs take jobs. The danger is that we are in a massive bubble and while these are nice tools they are not worth anything close to the trillions of dollars bet on them.

IMO the psychology at work here is basically denial that we can both be in the biggest bubble of all time in terms of dollars and LLMs are useful. Just not THAT useful.

AI is a type of outsourcing, you became a customer.

I'd love to hear more about how you utilised AI for this.

Personally I'm struggling to find it more useful than a slightly fancy code completion tool

This has been known since multilevel security was invented.

> If you're willing to build systems around it (like we do with people) to limit it's side effects and provide sanity checks,

I don't think that comparison holds up. We do build systems around people, but people also have internal filters, and most are able to use them to avoid having to interact with the external ones. You seemed to state that AI's don't (can't?) have working internal filters and rely on external ones.

Imagine if everyone did whatever they wanted all the time and cops had to go around physically corralling literally everyone at all times to maintain something vaguely resembling "order." That would be more like a world filled with animals than people, and even animals have a bit more reasoning than that. That's where we are with AI, apparently.

The very simple answer to that is that the artificial person can’t do the full job of a person yet.

Being good or better at certain parts of a job does not mean it can do the whole job effectively.

In reality, neither the client nor the user base have access to the ground truth of these "AI system"s to determine actual reliability and efficiency.

That's not to say there aren't some genuine ML/AGI companies like DeepMind (which solve specific narrow problems with quite high confidently), but most of the "AI" companies feel like they are coming from Crypto and are now selling little more than vaporware in the AI gold rush.

Apple is the world's most valuable company, and many would attribute a strong part of their success to Jobs' legacy of high-quality decision-making. But anyone who has worked in a large company understands that there's no way Apple can so consistently produce their wide range of highly integrated, high quality products with only a top-down mandate from one person; especially a dead one. It takes thousands of people, the right people, given the right level of authority, making high-quality high-creativity decisions. It also, obviously, takes the daily process, an awe-inspiring global supply chain, automation systems, and these are areas that computers, and now AI, can have a high impact in. But that automation is a commodity now. Samsung has access to that same automation, and they make fridges and TVs; so why aren't they worth almost four trillion dollars?

AI doesn't replace humans; it, like computers more generally before it, brings the process cost of the inhuman things it can automate to zero. When that cost is zero, AI cannot be a differentiating factor between two businesses. The differentiating factors, instead, become the capital the businesses already have to deploy (favoring of established players), and the humans who interact with the AI, interpreting and when necessary executing on its decisions.

“A computer can never be held accountable. Therefore, a computer must never make a management decision.”

There are lots of bullshit jobs that we could automate away, AI or no. This is far from a new problem. Our current "AI" solutions promise to do it cheaper, but detecting and dealing with "hallucinations" is turning out to be more expensive than anticipated and it's not at all clear to me that this will be the silver bullet that the likes of Sam Altman claims it will be.

Even if the AI solution makes fewer mistakes, the magnitude of those mistakes matter. The human might make transcription errors with patient data or other annoying but fixable clerical errors, while the AI may be perfect with transcription but make completely sensible sounding but ultimately nonsense diagnosis, with dangerous consequences.

That said, we can still isolate and modify parts of a network, and combine models trained for different tasks. But you need to break things down into components after the fact, instead of beforehand, in order to get the benefits of learning via scale of data + compute.

[1]: http://www.incompleteideas.net/IncIdeas/BitterLesson.html

Aren’t humans also black boxes that suffer from hallucinations?

E.g. for hallucinations: engineers make dumb mistakes in their code all the time, normal people will make false assertions about geopolitical, scientific and other facts all the time. c.f. The Dunning Kruger effect.

And black box because you can only interrogate the system at its interface (usually voice or through written words / pictures)

This is why I am not worried about the "AI Singularity" like some notable loudmouth technologists are. At least not with our current ML technologies.

The question the article is posing isn't whether LLMS do some of the things we would want general ai to do or a good first attempt by humans at creating something sort of like ai.

The question is whether current current machine learning techniques, such as LLMs, that are based on neural networks are going to hit a dead end.

I don't think that's something anyone can answer for sure.

Some answers are very long and just re-state the previous comments as if they were explaining simple concepts to a five year old child.

In short: in the hands of humans, tools like ChatGPT cause exponential growth of spam, engagement farming and malicious disinformation and propaganda in internet. I fear these negative use cases are growing exponentially faster than the useful parts. We will all drown in AI manure.

There's some survivorship bias going on here – you only consider technologies which succeeded, and find examples of people scrutinising them beforehand. However, we know that not every nascent technology blossoms; some are really effective, but can't find adopters; some are ahead of their time; some are cost-prohibitive; and some are outright scams.

It's not a given that every promising new technology is a penicillin – some might be Theranos.

  - Signed,
    Some arrogant fucker posting on hacker news

Kind of the way our right brain hemisphere does probabilistic computation and the left brain hemisphere does atomistic computation. And we use both.

So, whoever develops the digital equivalent of the corpus callosum wins.

I think there is room for statistical AI to operate symbolic systems so we can better control outputs. Actually, that's kind of what is going on when we ask AI to write code.

We desperately need to rapidly regulately down fossils usage and production for both electricity generation and transport. The rest of the world needs to follow the example of the EU CO2 emissions policy which guarantees it's progressing at a downwards slope independent of what the CO2 emissions are spent on.

We can't prove the correctness of the plurality of physics. Should we call that a dead end too?

If you believe in functionalism (~mental states are identified by what they do rather than by what they are made of), then current AI is not a dead end.

We wouldn't need to define intelligence, just make it big and efficient enough to replicate what's currently existing would be intelligence by that definition.

I think you are correct, but also I think that even if that were not the case, the Thai Library Problem[1] strongly suggests that AGI will have to be built on something other than LLMs (even if LLM-derived systems were to serve as an interface to such systems).

[1]: https://medium.com/@emilymenonbender/thought-experiment-in-t...

Kinda kills the utility if you need to know what you want out tho...

And in general, binary floating point arithmetic cannot guarantee associativity - i.e. `(a + b) + c` might not be the same as `a + (b + c)`. That in turn can lead to the model picking another token in rare cases (and it’s auto-regressive consequences, that the entire remainder of the generated sequence might differ): https://www.ingonyama.com/blog/solving-reproducibility-chall...

Edit: Of course, my answer assumes you are asking about the case when the model lets you set its token generation temperature (stochasticity) to exactly zero. With default parameter settings, all LLMs I know of randomly pick among the best tokens.

To get variety, you give each person a different seed. That way each user gets consistent answers but different than each other. You can add some randomness in each call if you don’t want the same person getting the same output for the same input.

It would be impossible to test and benchmark llama.cpp et al otherwise!

By the time you get to a UI someone has made these decisions for you.

It’s just math underneath!

Hope this helps.

At the end of the day it's a tool currently, with surface-level information it's incredibly helpful in my opinion - Getting an overview of a subject or even coding smaller functions.

What's interesting in my opinion is "agents" though... not in the current "let's slap an LLM into some workflow", but as a concept that is at least an order of magnitude away from what is possible today.

I get very useful answers from ChatGPT several times a day. You need to verify anything important, of course. But that's also true when asking people.

Imagine for a moment that you work as a developer, encounter a weird bug, and post your problem into your company’s Slack. Other devs then send a bunch of StackOverflow links that have nothing to do with your problem or don’t address your central issue. Is this a problem with StackOverflow or with coworkers posting links uncritically?

These LLM programs are very different than naive one-shot questions asked of ChatGPT, resembling o1/3 thinking that integrates human domain knowledge to produce great answers that would have been cost-prohibitive for humans to do manually.

Naive use of LLMs by non-technical users is annoying, but is also a straw-man argument against the technology. Smart usage of LLMs in o1/3 style of emulated reasoning unlocks entirely new realms of functionality.

LLMs are analogous to a new programming platform, such as iPhones and VR. New platforms unlock new functionality along with various tradeoffs. We need time to explore what makes sense to build on top of this platform, and what things don’t make sense.

What we shouldn’t do is give blanket approval or disapproval. Like any other technology, we should use the right tool for the job and utilize said tool correctly and effectively.

As with cell phones, eventually society will adapt.

The point about solving more problems at a low reliability level rather than solving problems at a higher reliability level is highly interesting though!

After reading about o3's performance on ARC-AGI, I strongly suspect people will not be so flippantly dismissive of the inherent limits of these technologies by this time next year. I'm genuinely surprised at how myopic HN commentary is on this topic in general. Maybe because the implications are almost unthinkably profound.

Anyway, OpenAI, Anthropic, Meta, and everyone else are well aware of these types of criticisms, and are making significant, measurable progress towards architecturally solving the deficiencies.

https://arcprize.org/blog/oai-o3-pub-breakthrough

I can ask the computer "hey I know this thing exists in your training data, tell me what it is and cite your sources." This is awesome. Seriously.

But what that means is you can ask it for sample code, or to answer a legal question, but fundamentally you're getting a search engine reading something back to you. It is not a programmer and it is not a lawyer.

The hype train really wants to exaggerate this to "we're going to steal all the jobs" because that makes the stock price go up.

They would be far less excited about that if they read a little history.

That being said they are very useful. I mostly use them as a far superior alternative to web search and as a kind of junior research assistant. Anything they find must be checked of course.

I think we have invented the sci-fi trope of the AI librarian of the galactic archive. It can’t solve problems but it can rifle through the totality of human knowledge and rapidly find things.

It’s a search engine.

> idiot savants with vast and unreliable intelligence.

Remember, intelligence !== knowledge. These LLMs indeed have vast and unreliable knowledge banks.

I would say "knowledge" rather than "intelligence"

The key feature of LLMs is the vast amounts of information and data they have access to, and their ability to quickly process and summarize, using well-written prose, that information based on pattern matching.

Over the past couple of years of educating myself a bit, whilst I am no expert I have been anticipating a dead end. You can throw as much training at these things as you like, but all you'll get is more of the same with diminishing returns. Indeed in some research the quality of responses gets worse as you train it with more data.

I am yet to see anything transformative out of LLMs other than demos which have prompt engineers working night and day to do something impressive with. Those Sora videos took forever to put together, and cost huge amounts of compute. No one is going to make a whole production quality movie with an LLM and disrupt Hollywood.

I agree, an LLM is like an idiot savant, and whilst it's fantastic for everyone to have access to a savant, it doesn't change the world like the internet, or internal combustion engine did.

OpenAI is heading toward some difficult decisions, they either admit their consumer business model is dead and go into competing with Amazon for API business (good luck), become a research lab (give up on being a billion dollar company), or get acquired and move on.

> A group of individuals adept with use of such an idiot savant enhanced environment would be incredibly capable. They'd be a force unseen in human civilization before today

More than the people who landed someone on the moon?

Basically this. They already have vastly better-than-human ability at finding syntax errors within code, which on its own is quite useful; think of how many people have probably dropped out of CS as a major after staying up all night and failing to find a missing semicolon.

1. LLMs have little value, are totally unreliable, will never amount to much because they don’t learn and grow and mature like people do., so they cannot replace a person like me who is well advanced in a career.

2. LLMs are incredible useful and will change the world because they excel at entry level work and can replace swaths of relatively undifferentiated information workers. LLM flaws are not that different from those workers’ flaws.

I’m in camp 2, but I appreciate and agree with the articulation of why they will not replace every information worker.

I'm sorry but your comment is a good example of the logical shell game many people play with AI when applying it to general problem solving. Your LLM AI is both an idiot and an expert somehow? Where is this expertise derived from and why should you trust it? If LLMs were truly as revolutionary as all the grifters would have you believe then why do we not see "forces unseed in human civilization before today" by humans that employ armies of interns? That these supposed ubermensch do not presently exist is firm evidence in support of current AI being a dead end in my opinion.

Humans are infinitely more capable than current AI, the limiting factor is time and money. Not capability!

Sorry for the potentially silly question. I just spent some time trying to research it and came up with nothing concrete.

The whole conversation is so dishonest.

Every software firm, notable and small, has had layoffs over the past two years, but somehow there's still a "STEM shortage" and companies are "starving for talent" or some such nonsense?

Fake discussion.

Do you have an example of this?

Norvig seems to be using a loose technical definition of AGI, roughly "AI with some degree of generality", which is hard to argue with, although by that measure older GOFAI systems like SOAR might also qualify.

Certainly "deep learning" in general (connectionist vs symbolic, self-learnt representations) was a step in the right direction, and LLMs a second step, but it seems we're still a half dozen MAJOR steps away from anything similar to animal intelligence, with one critical step being moving beyond full dataset pre-training to new continuous learning algorithms.

And the systems in place do so at scales and breadths that no human could achieve.

That doesn’t change the fact that it’s effectively triple PHD uncle Jim, as in slightly unreliable and prone to bullshitting its way through questions, despite having a breathtaking depth and breadth of knowledge.

What we are making is not software in any normal sense of the word, but rather an engine to navigate the entire pool of human knowledge, including all of the stupidity, bias, and idiosyncrasies of humanity, all rolled up into a big sticky glob.

It’s an incredibly powerful tool, but it’s a fundamentally different class of tool. We cannot expect to apply conventional software processes and paradigms to LLM based tools any more than we could apply those paradigms to politics or child rearing and expect useful results.