HNHacker News
TopNewBestAskShowJobs

lsy

3,257 karma · joined April 28, 2015

submissionscomments
lsy··on There is no hard takeoff
Most arguments for "hard takeoff" involve some form of inductive proof: if we have "intelligence level" n, and we know that it can be improved n + 1, then there is a guarantee of eventual performance n + k where k is arbitrarily large. Aside from not actually having any reliable way to quantify n or the concept of "intelligence" in the first place, this proof falls apart if improvement is n + ε, where ε changes at each step, and there is no reason to assume ε will escape the pattern of every other process in the known universe of trending to zero in the face of natural limiting factors.
lsy··on The Myth of AI Omniscience: AI's Epistemological Limits
An LLM can't generate new training data for itself. Mathematically speaking, the output of the LLM is governed by the same statistics embedded in its parameters, and a large enough training set of LLM output should have zero impact on those parameters. The difference is that training on human output contains the implicit assumption that the human output is "correct" in some way. So if the LLM gets more human-generated training data, it can come closer to "correctness". An LLM training on its own output is merely reinforcing its weights.

Maybe to put it more simply, there is no LLM in the world that can determine for itself whether a proposition or piece of training data is sound without a person saying so at some point, either implicitly or explicitly. Even efforts to give LLMs some non-human-mediated reward function are merely people attempting to encode our own principles. While people are no doubt physical, we have a biological imperative to survival and well-being that governs our sense-making faculties, and to which nobody has yet proposed an analogue for LLMs. Our "transformations" are not only governed by other people's "training data" but by our urge to be able to skillfully cope with a world in which we are biologically invested.

lsy··on GPT-4 can't reason
We are not going to prove or disprove "reasoning" through giving the LLM word problems. LLMs subvert the entire foundation of word problems, which is that words correlate to internal representations and are an indicator of thought processes. Word problems don't have construct validity for testing reasoning in LLMs.

On top of this, there is an almost-certainty that OpenAI has teams of contractors reading as many conversations as possible and hand-fixing bad responses, which makes non-reproducibility a difficult concept when the object of inquiry can change from moment-to-moment.

What the field needs is not more people thinking up word problems but rigorous analysis of the internal behavior of these models and maybe more importantly a functional definition of terms like "reasoning" that everyone can agree on.

lsy··on The Myth of AI Omniscience: AI's Epistemological Limits
What does it take to "hook up" an LLM to a camera and get it to output something cogent? We have a set of human-selected discrete photographs and a set of human-written text labels for those photographs, then a model that is trained to assign those text labels to other photographs. Then a new photo is put into the model, a set of labels is emitted, and those labels are processed by the LLM as text (or more accurately as token integers). The same principle holds for other sensors and their data.

All of these inputs are either human-generated or based directly on the human-generated input. So the argument is merely extended to say that "all human-generated text and photographs are sufficient to fully describe the world" which again I think would be hard to argue for. The point of the article is that an LLM is simply a mechanical transformer from its input to its output, and the specifics of that transformation are fully determined by its training data, which is all human-generated.

lsy··on Remote workers are increasingly less productive, anxious, depressed and lonely
So a bunch of pull quotes from CEOs and one non-peer-reviewed survey study? One would expect that out of all the companies pushing for an end to remote work, one would have solid data they were willing to share.
lsy··on The Myth of AI Omniscience: AI's Epistemological Limits
The problem is that new knowledge comes not only from combination but discovery, and discovery fundamentally requires access to and interaction with the actual world. And as of yet we have not created a way that computers can directly access the world, all I/O is mediated, etc. The more pedantic argument would be that what an LLM can write about the world is fundamentally limited to what has thus far been captured in human writing, and any structural properties thereof.

To your point, prediction may not be bound by what the data explicitly shows, but it is necessarily bound by the data and its implicit and explicit structure. But to argue that the LLM can make any discovery a person can, or interact fluidly with the world as it exists, would require one to believe that every property of the world is either explicitly captured in human-written text or in the structure of that text, which I think would be pretty difficult to argue.

lsy··on HackerOne lays off 12% of its workforce
Can you imagine sitting down to write an email announcing a layoff and deciding that opening with "H1" in place of "Hi" and calling people "Hackeronies" would be a great, culture-preserving idea? It's an extremely serious email and for the CEO to have not thought twice about whether this time he should just write "Hi" and refer to people as "our coworkers" says a lot about his state of mind.
lsy··on Patterns for building LLM-based systems and products
For those who don't have 65 minutes, if you write software you are probably familiar with the concepts of evals, caching, guardrails, defensive UX, and collecting user feedback, none of which are really unique to LLMs. The other two items are "fine-tuning" which just means nudging the LLM to be better at responding a certain way, and "RAG" which is a new acronym that just means using the input to look things up in a database first and concatenate them into the prompt so the LLM uses it as part of the context for token generation.
lsy··on Damaging results of mandated return to the office: it’s worse than we thought
Looking at the graph, it seems like productivity skyrocketed in 2020 (when everyone was WFH), returned to the 2019 baseline in 2021, and dived in 2022. To me this indicates that it's somehow hurting productivity for people to have returned to offices.
lsy··on The rule says, “No vehicles in the park”
The thing that really proved this game's point for me is the comments here from people giving slightly different versions of "well it's obvious what 'vehicle' means".
lsy··on Why isn't Titan classed? (2019)
What strikes me is that this all sounds like bluster and prevarication around safety, rather than a situation where the safety has actually been thought through. This blog post also doesn't have any kind of links or references to any more in-depth discussion and ultimately seems even on the surface as though it would fall apart under scrutiny. For instance it's quite obvious that the reason the majority of accidents are the result of operator error is because mechanical failures are rare precisely due to classing and other forms of regulation. It does very little good to have "high-level operational safety" if the vessel you are in catastrophically implodes due to mechanical failure.
lsy··on Moog sells its business to InMusic
Not really. It's an interesting sidebar how companies use the phrase "employee-owned" to confuse people about who is in charge. Moog had an ESOP which just means the employees held 49% of the shares of stock in the company, as far as I understand without any right to influence operations, which remained fully with the CEO. That's "employee-owned" in the same sense that you "own" my house if you hold securities that are invested in my mortgage, i.e. not at all.
lsy··on Generative AI is also a revolution for computer interfaces
Similar to self-driving cars, I think the last 1% of the way is going to be the near-insurmountable barrier that makes widespread implementation difficult or potentially impossible. For tasks where you can plausibly check the results with some kind of external software harness, or defer double-checking to a person, these models will make development quick (if resource-intensive). But any kind of fully-automated complex process where the consequences are important will be tough to hand over to black-box generators.
lsy··on Why are LLMs general learners?
Two points in response to this:

I think it's a category error to call word embedding in a vector space "semantic" representation when discussing concepts like understanding. Semantics deals with the referents of words, but in this case there are no referents, merely a list of representational tokens which are defined as being "close in meaning" to the original due to proximity in text or some other structural characteristic. We call the embedding "semantic" because it is useful for human semantic purposes as we can mechanize some translations from one vector to another and receive a useful response that we then assign meaning to, but that usefulness doesn't indicate that the machine itself has any access to the referents of the tokens it's processing or semantic understanding. Put more simply, "semantics" does not merely mean the relationship between several ungrounded tokens, but that is all a vector embedding can accomplish.

Secondly, I think in the chess thread, the prompt being "engineered" in the example is extremely complex and constrains the output space sufficiently to produce high-quality results, but you start to wonder at what point the LLM is not doing most of the work. Meanwhile deeper in the thread we learn that even this prompting is not reliable and occasionally requires giving feedback that the move was bad(!) and repetition to achieve good results "the majority of the time in less than 3 tries". You can see where the practical problem arises, if we want to rely on LLMs for answers we don't already know. Claiming that we have a "general" function that "just" requires arbitrarily varying the input over an uncountably large space until you achieve the desired result is akin to saying f(x) = rand() * x is a universal computer as long as you find the right x. The ad absurdum version of the chess example is running Stockfish, sending a prompt that contains the Stockfish move and a request to repeat it, and then claiming that the LLM draws against Stockfish. However as we have seen with tokens like "SolidGoldMagikarp", LLMs are not even yet capable of reliably implementing the identity function, so I am not sure we can even say this.

lsy··on Dating, education, and sex ratios
This article points out what is probably real friction from shifting mores but seems mainly preoccupied with implying that a return to more conservative values around women's role in society would be called for.

What I feel is needed is to understand what the fundamentals are and how those things could be achieved in a society that is also compatible with human freedom. Most of the problems described in this article stem from a totally artificial "bundling" of companionship, sexual satisfaction, child-rearing, economic support, and intellectual stimulation into a singular person who not only is expected to satisfy these needs for you (and sometimes fails), but often prevents you from satisfying those needs with other people.

So of course finding "Mr. or Mrs. Right" becomes existential, and of course we turn to dependable sources of entertainment as the anxiety of picking the "right" life mate prevents any of us from actually committing to anything. But with some creativity we can think of actual escape routes from this dynamic that don't require a return to treating women as property, like supporting single child-rearing, allowing people to form mutual support structures with friends or family, detaching self-worth from relationship status, a further liberalization of sexual mores, expanding economic safety nets, and so on and so forth.

lsy··on Modern language models refute Chomsky’s approach to language
None of this makes sense.

1."lè" with a low tone means fine, but "nì síkòní" means "I'm fine", while "là síkòní" means "he/she/it is fine"?

2. The example of "noun prefixes" doesn't include different noun prefixes?

3. The example of "verb ending changes" doesn't include different verb endings?

4. Since the second clause of the second phrase apparently means "I'm fine", the first word must mean "you", but then the second word is repeated in the next sentence which has nothing to do with thanking someone? And "síkòní" gets sprinkled into three sentences with no common concepts?

This just looks, predictably, like gibberish spit out by an LLM.

lsy··on Modern language models refute Chomsky’s approach to language
For people who don't understand this, the reason humans refer to "Alex" much later in a story is not because they are statistically recalling that they said "Alex" dozens or hundreds of words earlier (as the LLM is described doing in the paper), but because they have a world-model they are actively describing, where "Alex" refers to an entity in that world-model. We know that the LLM is only saying "Alex" because it appeared earlier, but we also know humans don't work like that, so how can the LLM's generation of language say anything about how humans acquire and use it?
lsy··on Westfield Gives Up Downtown San Francisco Mall
My sense is that surviving and thriving malls in urban areas today are really destination experiences for dining. Valley Fair has extremely high-quality dining options which brings the foot traffic that props up the whole enterprise. The SF Westfield never felt like they cracked that. It's only been open since '88 so it was sort of a latecomer to the mall era and got stuck in the department store, shopping-as-destination model. It required a detour down from Union Square's shopping district and didn't have much to do inside besides shopping, so I'm surprised it hung on this long.
lsy··on A Mechanistic Interpretability Analysis of Grokking
To me the odd takeaway here is that the model eventually converges to a close approximation but never really lands at the core of the problem. Loss never reaches zero even for the memorized set, and the "grokked" loss is usually higher. Wouldn't you expect something totally bounded like "addition mod 113" to be trainable to the point of zero loss? I think the field is looking for that final phase change that takes us from statistical methods to actual understanding.
lsy··on OpenAI’s Altman and others warning of advanced AI as ‘extinction’ risk
I think on top of the well-covered critique around regulatory capture, Altman and other GPT hangers-on benefit when the media gets into a tizzy about "AI" and needs sound bites and consultants and talking heads, when investors wildly overestimate a product's capability when deciding where to invest money, and when government becomes too distracted by pie-in-the-sky sci-fi scenarios to look at the obvious immediate harms coming out of the current LLM project. Nobody in this list has advanced a plausible way to make an LLM grok the concept of the number 10, let alone posited a real path to human extinction that comes out of next-word generators.
lsy··on Large language models do not recognize identifier swaps in Python
I agree and I think the reason it's sort of an indictment of LLMs is because there's no real way to guarantee performance over a range of input, which makes it unsuitable for many automation tasks.

If you develop an LLM prompt that says "Add <user input> to 5" and prove that it works for a handful of cases, there's no guarantee it works for any other case. Whereas "(lambda (x) (+ x 5))" has some mechanical guarantee based on the implementation. And for the LLM, the negative result becomes more possible the higher the complexity of the prompt. This reduces LLMs to being suitable for tasks where reliability is not particularly important, or a supervisory person or system is checking for consistency and somehow can re-run the LLM to generate new results (which itself is not always reliable).

lsy··on What if we set GPT-4 free in Minecraft?
I can only imagine the desperation of the researcher pleading with a computer to do its best to solve the halting problem.
lsy··on What if we set GPT-4 free in Minecraft?
It's not playing in-context of minecraft, it's playing in-context of an API to minecraft. You can see one of the limitations in its error condition when it tries to craft an "acacia axe" out of acacia planks and sticks, fails, and then replaces all the references to "wooden axe". Of course in the real world it doesn't matter what you call the axe you made, and it's pretty clear what an acacia axe is. Even if it did matter, you could also easily keep the function name and output message, and just make an "wooden axe" behind the scenes. The fact that the GPT is so tightly bound to the formalism of the API is an indication that this is a task the GPT can likely do quite well as this API is well-used and documented.
lsy··on Voyager: An Open-Ended Embodied Agent with LLMs
I'm not sure how the authors arrive at the idea that this agent is embodied or open-ended. It is sending API calls to minecraft, there's no "body" involved except as a symbolic concept in a game engine, and the fact that minecraft is a video game with a limited variety of behaviors (and the authors give the GPT an "overarching goal" of novelty) precludes open-endedness. To me this feels like an example of the ludic fallacy. Spitting out "bot.equip('sword')" requires a lot of non-LLM work to be done on the back end of that call to actually translate to game mechanics, and it doesn't indicate that the LLM understands anything about what it "really" means to equip a sword, or that it would be able to navigate a real-world environment with swords etc.
lsy··on Why are so many giants of AI getting GPTs so badly wrong?
I think there is a pretty sizeable difference between "human perception of the world is necessarily mediated by sense organs" and "all human textual output exhaustively covers the world and can be used to comprehensively describe and navigate it". Unless we are solipsists, we tend to agree that the world maintains a presence whether we are sensing it or not, and that even our senses are merely a limited reflection of what the world is and not the ground truth; that when our ideas or senses don't match the world, the world wins. For an LLM however there is nothing outside of text.
lsy··on Why are so many giants of AI getting GPTs so badly wrong?
This article is of a piece with the "Sparks of AGI" paper in that it doesn't really provide any formal justification for its tests, and is mainly just a dude saying "well, certainly this seems like it would be outside the training set and the LLM seems to handle it OK so it must be reasoning". Meanwhile, we have no way of inspecting the training set, and even if we did the data and model are so massive it would be impossible to tell whether it was doing simple substitution or whatever.

Ultimately e.g. Brooks must be right that the model has no connection to the world, because to the model there is no world, only text. If the training text includes things that are untrue in the world, the model cannot access the world to make a final determination that its internal statistics are faulty. If there is an aspect of the world that isn't covered by the training text, the model cannot divine it. It is just correlating text to other text.

lsy··on Governance of Superintelligence
Nothing makes me think of Altman as a grifter more than his trying to spook uneducated lawmakers with sci-fi notions like "superintelligence" for which there are no plausible mechanisms or natural analogues, and for which the solution is to lobby government build a moat around his business and limit his competitors. We do not even have a consensus around a working definition of "intelligence", let alone any evidence that it is a linear or unbounded phenomenon, and even if it were, there is no evidence ChatGPT is a route to even human-level intelligence. The sum total of research into this "field" is a series of long chains of philosophical leaps that rapidly escape any connection to reality, which is no basis for a wide-ranging government intervention.
lsy··on Lyft CEO tells employees they must return to the office
At this point you can really just give a recipe for this type of article.

- business-brain CEO is flailing

- no better justification than "whiteboards" and "personal connection"

- leadership actually lied when they promised to support remote work

- trying to sell a hybrid setup as "balance" rather than half-assed and distracting

And the totally expected outcome which is of course that everyone ignores it, and a couple months later the CEO has to start counting to 3 in a parental tone for the umpteenth time.

lsy··on Agents are the next AI killer app after ChatGPT
Simply holding temp to 0 or making the selection deterministic isn't an adequate solution unless the process is always run with the same set of inputs (at which point why not run the model once on all inputs and create a map?).

Ultimately with LLMs it's not possible (or desirable) to keep inputs separate from the rest of the prompt, so changing "give me the top X" to "give me the top Y" has the potential for a wildly different result. With traditional code we can achieve reliability because we sanitize and apply bounds to inputs, which then transit through the logic in a way we can reason about. The strength and weakness of an LLM is that it mashes up the input with the rest of its data and prompts, meaning we cannot predict the output for an arbitrary set of inputs.

lsy··on Agents are the next AI killer app after ChatGPT
In practice, I suspect it will be quite tough for these harnesses to get traction. Repeatedly calling an LLM is expensive and also not deterministic. Depending on the implementation, it may also be vulnerable to prompt injection. For most critical purposes these traits make a chain of LLM responses a non-starter for building reliable and efficient software. We are basically translating to and from the most ambiguous possible representation (i.e. natural language) of a task at every step.

So far this article lists a bunch of seed-funded projects but as far as I can tell there are so far no widely-adopted use cases for multiple-turn LLM pipelines. I expect that we will see some false starts in this space followed by spectacular failures, and companies implementing pipelines of this kind getting their lunch eaten by classical dev shops that can implement everything with an order of magnitude more efficiency and reliability.

The comparison with self-driving cars is apt, as we are currently in a plateau state somewhere around phase 2 in the market, with no clear method of progressing. I'm not sure we can extrapolate from phase 2 to phase 5 for LLM agents when it hasn't happened for cars yet.

← PreviousPage 6 of 9Next →