agentcoops

I can’t see that proposed division as anything but a straw-man. You would be hard-pressed to find anyone who genuinely thinks of LLMs as an “all-knowing, all-solving oracle” and yet, even in specialist fields, their utility is certainly more than a mere “heuristic”, which of course isn’t to say they don’t have limits. See only Terrance Tao’s reports on his ongoing experiments.

Do you genuinely think it’s worse that someone makes a decision, whether good or bad, after consulting with GPT versus making it in solitude? I spoke with a handyman the other day who unprompted told me he was building a side-business and found GPT a great aid — of course they might make some terrible decisions together, but it’s unimaginable to me that increasing agency isn’t a good thing. The interesting question at this stage isn’t just about “elder parents having nice conversations”, but about computers actually becoming useful for the general population through an intuitive natural language interface. I think that’s a pretty sober assessment of where we’re at today not hyperbole. Even as an experienced engineer and researcher myself, LLMs continue to transform how I interact with computers.

It's wild -- I've never seen such a persistent split in the Hacker News audience like this one. The skeptics read one set of AI articles, everyone else the others; a similar comment will be praised in one thread and down-voted to oblivion in another.

At this point, it seems to be more engineering throughput that will decide short to medium term outcomes. I’ve yet to see a case where an IC who took a position only because of X in fact outrageous compensation package (especially if not directly tied to longterm company performance through equity) was ever productive again. Meta certainly doesn’t strike me as a company that attracts talent for their “mission.”

TLDR Zuck’s recent actions definitely smell like a predictable failure driven by desperation to me.

Indeed — however it’s interesting that unlike the internet, computers or smartphones the older generation, like the younger, immediately found the use of GPT. This is reflected in the latest Mary Meeker report where it’s apparent that the /organic/ growth of AI use is unparalleled in the history of technology [1]. In my experience with my own parents’ use, GPT is the first time the older generation has found an intuitive interface to digital computers.

I’m still stunned to wander into threads like this where all the same talking points of AI being “pushed” on people are parroted. Marcus et al can keep screaming into their echo chamber and it won’t change a thing.

[1] https://www.bondcap.com/report/pdf/Trends_Artificial_Intelli...

I'm not so sure it's desperation. As an alternative hypothesis, we might simply view it as an attempt from a temporary position of strength to secure their tremendous lead as the primary consumer access point to intelligence. I don't think it's much of an exaggeration to suggest that this is one of the most important open questions at the moment -- one which will likely be relatively winner-takes-all (in contrast to the more commoditized B2B/API side) and where the winner likely won't be decided based on the intelligence side alone. The questions also aren't entirely separate since the winner, here, will have such incomparably valuable usage data...

Unlike most successful startups, OpenAI is not faced with the possibility that the giants (Apple, Google, Microsoft) decide to look their way, but the reality that these are their real competitors and that the stakes are existential for many of them (trends indicating a shift away from search etc). The most likely outcome remains that one if not all of the giants eventually manage to produce a halfway-decent product experience that reduces OpenAI to a B2B player.

1000%. It's really hard to express this to non-engineers who never wasted years of their life trying to work with n-grams and NLTK (even topic models) to make sense of textual data... Projects I dreamed of circa 2012 are now completely trivial. If you do have that comparison ready-at-hand, the problem of understanding what this mind-blowing leap means, to which end I find writing like the OP helpful, is so fascinating and something completely different than complaining that it's a "black box."

I've expressed this on here before, but it feels like the everyday reception of LLMs has been so damaged by the general public having just gotten a basic grasp on the existence of machine learning.

Certainly. I agree of course as to the problem of hype and I'm aware of how many people use LLMs today. I tried to emphasize in my earlier post that I can understand why someone like Sabine has the opinion she does -- I'm more confused how there's still similar positions to be found among software developers, evidenced often within Hacker News threads like the one we're in. I don't intend that to refer to you, who clearly has more than a passing knowledge of LLM internals, but more to the original commenter I was responding to.

More than marketing, I think from my experience it's chat with little control over context as the primary interface of most non-engineers with LLMs that leads to (mis)expectations of the tool in front of them. Having so little control over what is actually being input to the model makes it difficult to learn to treat a prompt as something more like a program.

The whole point is that an LLM is not a search engine and obviously anyone who treats it as one is going to be unsatisfied. It's just not a sensible comparison. You should compare working with an LLM to working with an old "state of the art" language tool like Python NLTK -- or, indeed, specifying a problem in Python versus specifying it in the form of a prompt -- to understand the unbridgeable gap between what we have today and what seemed to be the best even a few years ago. I understand when a popular science author or my relatives haven't understood this several years after mass access to LLMs, but I admit to being surprised when software developers have not.

Hosted and free or subscription-based DeepResearch like tools that integrate LLMs with search functionality (the whole domain of "RAG" or "Retrieval Augmented Generation") will be elementary for a long time yet simply because the cost of the average query starts to go up exponentially and there isn't that much money in it yet. Many people have and will continue to build their own research tools where they can determine how much compute time and API access cost they're willing to spend on a given query. OCR remains a hard problem, let alone appropriately chunking potentially hundreds of long documents into context length and synthesizing the outputs of potentially thousands of LLM outputs into a single response.

Yes, I find the biggest issue in discussing the present state of AI with people outside the field, whether technical or not, is that "machine learning" had only just entered popular understanding: i.e. everyone seems ready today to talk about the limits of training a machine learning model on X limited data set, unable to extrapolate beyond it. The difference between "learning the best binary classifier on a labelled training set" and "exploring the set of all possible programs representable by a deep neural network of whatever architecture to find that which best generates all digitally recorded traces of human beings throughout history" is very far from intuitive to even specialists. I think Ilya's old public discussions of this question are the most insightful for a popular audience, explaining how and why a world model and not simply a Markov chain is necessary to solve the seemingly trivial problem of "predicting the next word in a sequence."

I don't mean to be disparaging to the original author, but I genuinely think a good litmus test today for the calibre of a human researcher's intelligence is what they are able to get out of the current state of the art AI with all of its faults. Contrasting any of Terrance Tao's comments over the last year to the comments above are telling. Perhaps it seems unfair to contrast such a celebrated mathematician to a popular science author, but in fact one would a priori expect that AI ought to be less helpful for the most talented mind in a field. Yet we seem to find exactly the opposite: this cottage industry of "academics" who, now more than a year since LLMs entered popular consciousness, seem to do nothing but decry "AI hype" - while both everyday people and seemingly the most talented researchers continue to pursue what interests them at a higher level.

If I were to be cynical, I think we've seen over the last decade the descent of most of academia, humanities as much as natural sciences, to a rather poor state, drawing entirely on a self-contained loop of references without much use or interest. Especially in the natural sciences, one can today with little effort obtain an infinitely more insightful and, yes, accurate synthesis of the present state of a field from an LLM than 99% of popular science authors.

A common misconception by design [1]. The research that was "hidden" behind this innocent-sounding name is very closely tied to the origins of backpropagation in the work of Paul Werbos where it will be further developed as "approximate dynamic programming." Bellman's research is more general and is close to a lot of thinking in physics after Lagrange, indeed quite close to the very notion of a "Lagrangian." Over-simplifying, the question has to do with optimization and guarantees of convergence inside various increasingly less-deterministic processes: how can we provably find true maxima of a given function step-by-step.

[1] In the article, an extract from Bellman's very interesting biography, we read:

"The 1950s were not good years for mathematical research. We had a very interesting gentleman in Washington named Wilson. He was Secretary of Defense, and he actually had a pathological fear and hatred of the word, research... What title, what name, could I choose? In the first place I was interested in planning, in decision making, in thinking. But planning, is not a good word for various reasons. I decided therefore to use the word, ‘programming.’ I wanted to get across the idea that this was dynamic, this was multistage, this was time-varying... Let’s take a word that has an absolutely precise meaning, namely dynamic, in the classical physical sense. It also has a very interesting property as an adjective, and that is it’s impossible to use the word, dynamic, in a pejorative sense. Try thinking of some com- bination that will possibly give it a pejorative meaning. It’s impossible. Thus, I thought dynamic programming was a good name. It was something not even a Congressman could object to. So I used it as an umbrella for my activities.”

For those interested in the history, this is in fact the Neural Network research path that predated LLMs. Not just in the sense that Hinton et al and the core of the "Parallel Distributed Processing"/Connectionist school were always opposed to Chomsky's identification of brain-thought-language, but that the original early 2000s NSF grant awarded to Werbos, Ng, LeCun et al was for "Deep Learning in the Mammalian Visual Cortex." In their research program, mouse intelligence was posited as the first major challenge.

It’s never just about the company’s numbers, but the financial environment. When interest rates come down enough to tempt investors out of the money market, you’re likely to see a new wave of IPOs.

It's funny, my first job out of college was machine learning for a company processing medical bills to detect fraud on the part of hospitals. It was a very eye-opening process for me as to how American health care is perhaps the worst designed system conceivable by man -- as in, it's genuinely difficult to imagine how the various regulations, incentives etc could be made worse from any perspective (left or right) or measure (human care, cost, equality, whatever). Medical diagnoses are systematized in a rigorous manner through a series of codes (ICD9s, CPTs, DRGs etc), but there is no real set or expected pricing for each code apart from Medicare and these strange agreements between particular insurance providers and particular hospitals (i.e. what defines "in-network" or "out-of-network", where pricing is more or less random). Bills also have an explicit location type (Inpatient, Outpatient, Emergency room, Ambulance etc) and Urgent/Emergency Room care is significantly and consistently the most expensive to the insurer (most American insurance is, after all, high deductible emergency insurance) and thus one of the major vectors of fraud was passing bills off as such. One of the real aspects of madness in the whole system is the pricing disconnect and lack of transparency between all participants: hospitals ask an agreed upon price to insurers who pay it with few questions asked, as hospital pricing drifts up insurers raise premium prices, employers keep paying premiums, and individuals either pay something contractually determined by their insurance company or an essentially random price if uninsured.

TLDR; it isn't at all surprising to me that yet another vulture on American health care expenditure has discovered how profitable emergency rooms can be and that we see such a mad proliferation of urgent care facilities on every corner.

I think for anyone who has been intimately involved with this last generation of "unicorns", it really does not look like an "exit" if that means bankruptcy, end of the company, betrayal of goals, or what have you: for a high-growth startup of approximately ten years, finding liquidity for employees is perhaps counter-intuitively a sign of deep maturity and that the company is concerned for the long haul (and that investors/the market believe this is likely).

At about the ten year mark, there has to be a changing of the guards from the foot soldiers who give their all that an unlikely institution could come to exist in the world at scale to people concerned more with stabilizing that institution and ensuring its continuity. In almost every company that has reached such scale in the last decade, this has often meant a transition from an executive team formed of early employees to a more senior C-team from elsewhere with a different skillset. In a world context where the largest companies are more likely to stay private than IPO, it's a profoundly important move to allow some liquidity for longterm employees, who otherwise might be forced to stay working at the company long past physical burnout.

A really good question! I've been out of the field for a long time, but it's important to stress the difference between "interactive proof assistants" and "automated theorem provers." The latter consist in various software tools that, given a well-formed logical formula, attempt to automatically derive a proof thereof in a given logic. There are of course well-understood limits to such tools given the foundational results of Gödel, Turing etc in the early 20th century.

On the other hand, interactive proof assistants such as Lean, mentioned in the title article, or Isabelle are really something more like functional programming languages in which a human mathematician can specify and prove a given theorem, relying on the work of other mathematicians as equivalent to a third-party library. Again like with programming languages, the different tools will offer various trade-offs between expressibility and the kinds of guarantees that can be made at compile time about the validity of a proof. It has traditionally been viewed as far too time consuming to actually specify even intermediate mathematics in such tools, but LLMs really change the situation, as Terrance Tao in particular has demonstrated.

For anyone interested in the topic, I highly recommend the work of the CMU logician Jeremy Avigad. See, for example: * Formally Verified Mathematics- https://www.andrew.cmu.edu/user/avigad/Papers/cacm.pdf * Automated Reasoning in Mathematics- https://link.springer.com/chapter/10.1007/978-3-031-63498-7_... * The Mechanization of Mathematics- https://www.ams.org/journals/notices/201806/rnoti-p681.pdf

Burnout, which doesn't need scare quotes, very much still applies for the humans involved in building AGI -- in fact, the burnout potential in this case is probably an order of magnitude higher than the already elevated chances when working through the exponential growth phase of a startup at such scale ("delivery apps" etc) since you'd have an additional scientific or societal motivation to ignore bodily limits.

That said, I don't doubt that this particular departure was more the result of company politics, whether a product of the earlier board upheaval, performance related or simply the decision to bring in a new CTO with a different skill set.

I haven't looked at the code in ages, but it's probably the only scaled consumer web application written in Scala and moreover running on Scala 3 that you can see the end-to-end source for. You have all the Twitter open source Scala projects, of course, but that's just infrastructure for running a web application, rather than an actual production quality app -- and my sense is that in 2024 there aren't many product teams outside of Twitter using their application tooling (as opposed to some of their data infrastructure, certainly the area where Scala sees the most use today with Spark etc).

TLDR if you want to see production-quality Scala code that this very second is serving 40k chess games -- and mostly bullet/blitz where ms latency is of course crucial -- definitely take a look.

Not as much hype for the language at the moment over Rust or Kotlin, say, but it remains my language of choice for web backends by far.

I worked with him once on a job -- incredibly nice guy and obviously talented developer who used to work for the French agency responsible for the Scala Play Framework. https://github.com/lichess-org/lila and https://github.com/lichess-org/scalachess are great resources for anyone ever curious to see a production quality Scala3 web application using Cats and all the properly functional properties of the language.

Yes, this is the most important point and I think somehow least present in even discussions here: the technical question of who produces the best/cheapest LLM/future architecture is considerably less important than who, if anyone, creates a fundamentally new and dominant consumer experience built on AI. Most of the existing players (Google, Meta) would of course prefer that nobody produces such a newly dominant paradigm of computation for end-users since it would greatly reduce their relevance and subsequently revenues. Right now, ChatGPT is the only real contender in this space. However, I think you’re correct that it’s actually Apple who is most likely to be the next who attempts such a paradigm shift. Far too early to bet, but let’s say I wouldn’t be surprised if in five years we end up in a world in which Apple has the consumer monopoly and Microsoft the business monopoly, with Google and Meta falling into irrelevance.

Have you tried Cursor out of curiosity? No ties to the company and long-time dev (Scala mostly), just genuinely found it to be transformative to my development practice like no tool before.

I just mean people who have actually used the API directly or through task-specific applications like Cursor that are meant to maximize use of AI for their needs know how much of a breakthrough we’ve had this year. People who doubt or downplay the already existing capabilities of this technology tend to have just played with ChatGPT a little bit (or have whatever ideological or other reason to do so).

Because 99.9% of people who say things like this are just using ChatGPT itself and not any of the various awe-inspiring tools with full access to your codebase dynamically inserted into context via RAG. I have yet to talk to anyone who has actually worked for any amount of time against the GPT4 API or through Cursor, say, who underestimates their capabilities. Sincerely hoping this 'coup' doesn't mean the beginning of the end of that experience for most...

I think part of the misunderstanding of WeWork even within Silicon Valley/tech is what a diminishing part of their business the "coworking spaces" actually were. WeWork gets a lot of hate both inside and outside Silicon Valley, but I worked a lot with their team before the failed IPO and my strong sense is most of the blame for the process that lead to today rests entirely with senior management and not even the core business.

I'd be curious to know how this trajectory played out past the IPO blunder, but at the time they were already focused on the "enterprise" model of being the best place for a large firm to go to source/spin-up offices around the world (where in some sense you could understand the coworking spaces as dogfooding their own systems of vendor/space management for quickly spinning up and operating office space around the world). I.e. rather than either having to maintain profitability at a global network of coworking spaces or whatever franchise model you become the outsourced office team for the world's largest/high growth firms, who want to focus on their core competence not sourcing furniture vendors in China.

Rather than partaking in the joy of mocking WeWork's obvious hubris leading to their downfall, I think the interesting question, especially in this time of high interest rates, is what this bankruptcy signals for the future of "tech-accelerated" firms that want to primarily focus on the real world not software (ie without marginal cost of 0) -- who want to achieve the exponential scale of software yet who have to operate with the margin/cost structure constraint of a traditional firm. Tesla/SpaceX are of course the big success here, but I doubt we'll see anything equivalent until interest rates are back to 0...

Certainly the undisputed winners will be the very few firms with enough engineering resources and GPUs to train their own models (not just fine-tune) where the models in question increase the productivity of workers in their non-ai-related profit centers. After that we have the real question of what the future will be of open source LLMs, on the one hand, and the question most relevant to this article of what sort and whether profitable “AI businesses” can be sustained over time. As Stratechery has analyzed, it is very possible that OpenAI turns out to be a very profitable B2C company with ad revenue in ChatGPT not concerned with their B2B sales or even the objective quality of their AI. Right now is an incredible time for AI: cheap Uber rides never qualitatively changed my life, but the current consumer access to AI models is truly incredible and I hope that only improves. However, even ignoring whatever happens on the regulatory front, I don’t think that is guaranteed at all.

It’s a very important question. There are of course two obvious readings/real concerns: (1) the untested IP issues concerning models trained on code from outside the firm, (2) the possibility of unreliable code. From my experience working with LLMs heavily since January, I would articulate my philosophy as “LLMs cannot be authors” — this is just to say, “authorship” is less about having actually created or having done the work, but being legally liable which is to say having skin in the game. Assuming for a second IP issues are not a concern, my rule for developers (or content creators) post-GPT: “you now have more responsibility for errors in your code than before, use whatever tools allow you to be most certain.” I hope that this is the world we are entering, not of generalized irresponsibility or irrelevance of humans in an automated world, but that we finally focus on what we decide really needs doing (thus of course that deciding itself) and taking responsibility that it be done as best as possible, however that may be.

When I use ChatGPT, I find anything but GPT4 unusable; when I'm programming against the API, I actually tend to find myself using GPT3.5. I still haven't had a chance to experiment with the opensource LLMs, but that's my project for next month.

I'm sure in your career you've seen countless attempts by 1000+ employee firms to spinup an ops team that must construct a giant spreadsheet of product priorities between customers, PMs and engineering teams -- versus having all of Salesforce, Slack, emails, meeting transcripts, user support messages, vectorised and fed into an LLM that any employee can converse with. The latter is not a trivial problem by any means and many, many implementation details, but I'm sure it's such an exciting time to be at Microsoft. Stratechery had a fantastic article about this.

If I had to make a bet, I suspect Google and Meta will royally lose this generation: Apple will dominate personal access to LLMs, Microsoft will dominate business access to LLMs -- and honestly Palantir for the first time terrifies me. The current generation of LLMs are not about automation as such, but helping individuals accomplish self-directed goals (as possible in language and code).

I think you’re still anthropomorphizing LLMs too much by saying the problem is a lack of curiosity. It’s an engineering problem: you haven’t figured out the correct “context” from which business requirements would follow. (And why Microsoft is so incredibly well positioned for the future)

Certainly, but GPT3.5 is effectively free via the API, especially considering you have to hit $5 or whatever to even be as expensive as ChatGPT (and it's even relatively difficult to rack up a high bill on GPT4 now that they have token rate limits). The problem of general and global access to unrestricted LLMs is a very, very serious question, but here it's largely a question of engineers and, since it's relatively difficult to spend more than a Netflix subscription via the API, I'm assuming this is conceivably in budget.