fergal_reid

I think you are misunderstanding my point.

The technology to prevent plague was beyond the population at the time.

"Just use basic hygiene" is useless to a population who has no understanding of what that means.

It was a real virus that had very high mortality.

This makes it a serious counterpoint for people who argue such things can't be created in future.

"The AI designed killer virus using new science turned out to be not that easily transmitted, basic dynamically reprogrammable mrna implants easily prevents it, shame about the 95% who died"

I am not a biology person, but, candidly, this is a bad article, because the author is generalizing from the properties of viruses that tend to arise naturally, and using this to argue about the properties of deliberately designed viruses.

Firstly, it's not even clear that naturally occuring viruses can't be extremely lethal. There are examples (eg black death).

Even the existence of rare examples should be concerning, if attempting to argue against the possibility.

But even if there were no natural examples, it could just be because the ecosystem is in a balance, which them unlikely. That doesn't preclude someone from designing one.

To use an analogy. Natural predator prey ecosystems tend to be in balance. You could look around the world, before modern humans, and conclude that superpredators aren't a real threat, because predators that eat their prey too rapidly typically then die out. Thus most ecosystems you observe are in some balanced state.

But this doesn't mean modern humans can't come along, using their superior intelligence, and act as a superpredator. Ask the wolf population of western Europe.

If something is stopping us from doing this more often, it's alignment rather than capability.

You can't generalized from the typical behaviour that you see in a balanced natural ecosystem, and use that to say it proves there are no much more dangerous possibilities.

If the author made first principles argument about the inherent difficulty of doing something bad, even with abundant intelligence, I'd be much more reassured; or about defense offense asymmetry favouring defence.

What's here seems very naive.

I have no specific context here, but in case it's not obvious, I think you have to read the exchange in the context of British, and I guess RAF, indirectness and understatement.

I'm guessing the air marshal means "what the hell were you thinking"; Holden is agreeing, indicating that he has learned his lesson and acknowledges the error; and the air marshal is then graciously moving on.

Yes - but this will be the year of Linux on the desktop.

Well, you could conclude the world is now more dangerous for any kid let wander - even if fewer do, and even if the observed average risk to any kid is lower (given they are allowed wander a lot less).

I don't want to be on the 'overprotect kids' side of the argument, but I'm not sure the numbers argur cleanly in one direction or the other.

I also often think of selection bias whenever anyone says "I was allowed do a lot more and we were fine" in the context of child safeguarding; because it also sounds like a lot of kids were abused in the past, who don't speak up in that conversation.

I don't know. I worry I overprotect my kids, but I also am not sure how to price in small risks of massively negative events. I think that's the crux of it for parents - trying to weigh hard tradeoffs.

My story of kids growing up in a post AI world:

There's a gag in Star Trek 4 where Scotty goes back in time, and tries talk to a computer.

The gag is funny because he is from the future where you talk to computers normally. When the computer doesn't respond, someone hands him the mouse, and he tries use it as a microphone.

I watched that scene with my kids recently (9 and 6).

They didn't get the gag. They thought Scotty was completely reasonable to try and talk to the computer.

It took a while to explain.

I think the most direct answer is that at scale, inference can be batched, so that processing many queries together in a parallel batch is more efficient than interactively dedicating a single GPU per user (like your home setup).

If you want a survey of intermediate level engineering tricks, this post we wrote on the Fin AI blog might be interesting. (There's probably a level of proprietary techniques OpenAI etc have again beyond these): https://fin.ai/research/think-fast-reasoning-at-3ms-a-token/

Yes we're familiar with the terminology and framing of gofai. Fwiw I read (most of) the 3rd edition of Russell and Norvig in my undergrad days.

However, the point we're trying to make here is at a higher level of abstraction.

Basically most demos of agents you see these days don't prioritize reliability. Even a Copilot use case is quite a bit less demanding than a really frustrated user trying to get a refund or locate a missing order.

I'm not sure putting that in the language of pomdps is going to improve things for the reader, rather than just make us look more well read.

But your feedback is noted!

At Intercom we've also a lot of experience here.

I disagree, basically. In our experience actual real world processes are not compactly defined, and don't have sharp edges.

When you actually go to pull them out of a customer they have messy probabilistic edges, where you can sometimes make progress a lot faster, and end up with a much more compact and manageable representation of the process, by leveraging an LLM.

We've a strong opinion this is the future of the space and that purely deterministic workflows will get left behind! I guess we'll see.

You are correct that variable results could be a symptom of a failure to generalise well beyond the training set.

Such failure could happen if the models were overfit, or for other reasons. I don't think 'overfit', which is pretty well defined, is exactly the word you mean to use here.

However, I respectfully disagree with your claim. I think they are generalising well beyond the training dataset (though not as far beyond as say a good programmer would - at least not yet). I further think they are learning semantically.

Can't prove it in a comment except to say that there's simply no way they'd be able to successfully manipulate such large pieces of code, using English language instructions, it they weren't great at generalisation and ok at understanding semantics.

Strongly agree.

This seems to be very hard for people to accept, per the other comments here.

Until recently I was willing to accept an argument that perhaps LLMs had mostly learned the patterns; e.g. to maybe believe 'well there aren't that many really different leetcode questions'.

But with recent models (eg sonnet-3.7-thinking) they are operating well on such large and novel chunks of code that the idea they've seen everything in the training set, or even, like, a close structural match, is becoming ridiculous.

As an Irish person when I saw the article title, I was immediately sceptical.

I personally believe most articles about the famine shy away from the horror of it, and also from a frank discussion.

Going to give some subjective opinion here: people generally downplay the role of the British government and ruling class in it.

Why? One personal theory - growing up in the 80s in Ireland there was a lot of violence in the north. (Most) Irish people who were educated or middle class were worried about basically their kids joining the IRA, and so kind of downplayed the historical beef with the British. That's come through in the culture.

There's also kind of a fight over the historical narrative with the British, maybe including the history establishment, who yes care a lot about historical accuracy, but, also, very subjectively, see the world through a different lens, and often come up through British institutions that view the British empire positively.

It's often easier to say the famine was the blight, rather than political. (They do teach the political angle in schools in Ireland; but I think it's fair to say it's contested or downplayed in the popular understanding, especially in Britain.)

However that article is written by a famous Irish journalist and doesn't shy away from going beyond that.

Perhaps a note of caution - even by Irish standards he'd be left leaning, so would be very politically left by American standards; he's maybe prone to emphasize the angle that the root cause was lassiez-faire economic and political policies. (I'm not saying it wasn't.)

I personally would emphasize more the fact that the government did not care much about the Irish people specifically. The Irish were looked down on as a people; and also viewed as troublesome in the empire.

Some government folks did sympathize, of course, and did try to help.

But I personally do not think the famine would have happened in England, no matter how lassiez-faire the economic policies of the government. A major dimension must be a lack of care for the Irish people, over whom they were governing; and there are instances of people in power being glad to see the Irish being brought low:

"Public works projects achieved little, while Sir Charles Trevelyan, who was in charge of the relief effort, limited government aid on the basis of laissez-faire principles and an evangelical belief that “the judgement of God sent the calamity to teach the Irish a lesson”." per the UK parliament website!

It's not an easy thing to come to terms with even today. I recently recorded a video talking about how fast the build out of rail infrastructure was, in the UK, as an analogy for how fast the AI infra build out could be; and I got a little quesy realizing that during the Irish potato famine the UK was spending double digit GDP percent on rail build out. Far sighted, yes, and powering the industrial revolution, but wow, doing that while mass exporting food from the starving country next door, yikes.

Similar arguments to LeCun.

People are going to keep saying this about autoregressive models, how small errors accumulate and can't be corrected, while we literally watch reasoning models say things like "oh that's not right, let me try a different approach".

To me, this is like people saying "well NAND gates clearly can't sort things so I don't see how a computer could".

Large transformers can clearly learn very complex behavior, and the limits of that are not obvious from their low level building blocks or training paradigms.

I think the authors misunderstand what's actually going on.

I think this is the crux:

>They are vastly more powerful than what you get on an iPhone, but the principle is similar.

This analogy is bad.

It is true that the _training objective_ of LLMs during pretraining might be next token prediction, but that doesn't mean that 'your phone's autocomplete' is a good analogy, because systems can develop far beyond what their training objective might suggest.

Literally humans, optimized to spread their genes, have developed much higher level faculties than you might naively guess from the simplicity of the optimisation objective.

If the behavior of top LLMs didn't convince you of this, they clearly develop much more powerful internal representations than an autocomplete does, are much more capable etc.

I would point to papers like Othello-gpt, or lines of work on mechanistic interpretability, by Anthropic, and others, as very compelling evidence.

I think that, contrary to the authors, using words like 'understand' and 'think' for these systems is much more helpful than to conceptualise them as autocomplete.

The irony is that many people are autocompleting from the training objective to the limits of the system; or from generally being right by calling BS on AI, to concluding it's right to call BS here.

Yes, in parameter space.

However, TobyTheCamel's point is valid in that there are some parameter spaces where the MLE is going to be much less useful than others.

Even without having to go to high dimensions, if you've got a posterior that looks like a normal distribution, the MLE is going to the you a lot, whereas if it's a multimodal distribution with a lot of mass scattered around, knowing the MLE much less informative.

But this is a complex topic to address in general, so I'm trying to stick to what I see as the intuition behind the original question!

Right - I think this is what's at the heart of the original question.

I know they asked with a continuous example, but I don't interpret their question as limited to continuous cases, and I think it's easier to address using a discrete example, as we avoid the issue of each exact parameter having infinitesimal mass which occurs in a continuous setting.

Let's imagine the parameter we're trying to estimate is discrete and has, say, 500 different possible values.

Let's say the parameter can have the value of the integers between 1 and 500 and most of the mass is clustered in the middle between 230 and 270.

Given some data, it would actually be possible that MLE would come up with the exact value, say 250.

But maybe given the data, a range of values between 240 and 260 are also very plausible, so the likelihood of exactly 250 has a fairly low probability.

The original poster is confused, because they are basically saying, well, if the actual probability is so low, why is this MLE stuff useful?

You are pointing out they should really frame things in terms of a range and not a point estimate. You are right; but I think their question is still legitimate, because often in practice we do not give a range, and just give the maximum likelihood estimate of the parameter. (And also, separately, in a discrete parameter setting, specific parameter value could have substantial mass.)

So why is the MLE useful?

My answer would be, well, that's because for many posterior distributions, a lot of the probability mass will be near the MLE, if not exactly at it - so knowing the MLE is often useful, even if the probability of that exact value of the parameter is low.

I think most of the replies, here and on stack exchange, are answering slightly the wrong question.

It is fair to ask why the likelihoods are useful if they are so small, and it's not a good answer to talk about how they could be expressed as logs, or even to talk about the properties of continuous distributions.

I think the answer is:

Yes, individual likelihoods are so small, that yes even a MLE solution is extremely unlikely to be correct.

However, the idea is that often a lot of the probability mass - an amount that is not small - will be concentrated around the maximum likelihood estimate, and so that's why it makes a good estimate, and worth using.

Much like how the average is unlikely to be the exact value of a new sample from the distribution, but it's a good way of describing what to expect. (And gets better if you augment it with some measure of dispersion, and so on). (If the distribution is very dispersed, then while the average is less useful as an idea of what to expect, it still minimises prediction error in some loss; but that's a different thing and I think less relevant here).

> Lizzie was joking

Well, obviously we shouldn't get too hung up on what a fictional character thought - but I stand by my recollection.

Just googling it, and finding this page: https://www.sparknotes.com/lit/pride/quotes/symbol/pemberley...

I think you can say the last quote on that page is the character joking (although I'm not sure I read it that way); but the second last quote was the one I was referring to, and is in the narrator's voice.

But, look, while reading that did change my perspective on the story, I also don't want to interpret things too cynically; I'm not saying the character of Elizabeth should be read as purely seeking advantage; just that they were clearly evaluating marriage on a combination of advantage, and 'love', with a lot of weight on the former; and all of Austen made a lot more sense when I realised that.

The Austin novels made a lot more sense to me when I started to think of them as closer to tales of corporate Mergers & Acquisitions, rather than love stories. The rich familes then were like large corporations are now, and a marriage was a very financial merger.

I think I realised this when I first read Pride and Prejudice and the main character started talking about basically falling in love with the Pemberley estate.

Thereafter, any time I visit an English country house with extensive gardens, the massive wealth expenditure to create them makes a lot more sense when you view them as M&A marketing budget.

This is hopefully too cynical, and the truth is somewhere in between - but it's equally naive to read Austen as straight love stories with a modern perspective - there's a lot of clear focus on the incomes and social situations in the text.

There's a big gap between training an algorithm on a toy problem, vs building a useful product.

Software engineers often are missing key skills. They can learn them, but won't automatically get them in their traditional training.

First, measuring success. Actually telling how well a production system is doing is tricky. There's an art to developing metrics that tell you if an ML system is delivering value, and a lot for engineers don't have the metric design skills. Often to productionize an ML system, you need a bunch of proxy metrics and a pretty good backtesting setup. This will often depend on the specific problem, and the skill of it is something you won't get in a standard software setting.

Engineers - and especially designers - also struggle with edge cases when things go off the happy path. It's often easy to make an ML prototype that works in 90% of cases, and get a project started - but a nightmare to solve enough the edge cases for a production grade system. Finding and papering over and designing around all those edge cases effectively can require a deep bag of tricks a pure software engineer won't have.

Finally there's a struggle with tactics and culture. A lot of the bread and butter tactics of high performing software delivery are the opposite of what you need for ML projects. E.g. In high velocity frontend work you want to lock a design early, and your designer can probably do a lot of iteration before engineering starts. In ML projects you want to keep the design floating and low fidelity, as you prototype, and lock it late in the project.

So many development tactics, and cultural patterns, that lead to high performing software teams, in a SaaS setting, say, are anathema to ML projects.

>The retrieval part is way more important.

I don't agree with this - at Intercom we've put a lot of work into our Fin chatbot, which uses a RAG architecture, and we're still using GPT-4 for the generation part.

GPT-4 is a really powerful and expensive model but we find we need this power to 1) reduce hallucinations acceptably, and 2) keep the quality of inferences made using the retrieved text high.

Now, our bot is answering customer support questions unsupervised - maybe it'd be different for a human in the loop system - but at least in our case, we feel we need a very powerful generation model to reduce errors, even after having benchmarked this thoroughly.

We've also done work on the retrieval end of things, including a customised model, but found the generation side is where we need the most capable models.

I used to play Total Annihilation online back when I was in secondary school in the late 90s

We'd hang out on IRC, and people would host a game on their IP.

Eventually I got into creating new units for the game. You had to animate them with a C-like language, and I was trying to teach myself C at the time so that was part of my programming journey.

The 'Swedish Yankspankers' were a clan that had a channel on the IRC server, and my recollection, which may be wrong, is that they started building an app, 'TAreplay' that we could use to record and rewatch tournament games. People used to pass the exe around - not very secure, but those were the times!

It's amazing to me that those players hacking little apps together for themselves, have spawned a lineage of RTS engines and games that are still going all this time later.

A small but incredible example of hacker culture.

>Today's AI models are missing the ability to reason abstractly, including asking and answering questions of "Why?" and "How?"

This claim seems over general, because you can ask gpt-4 'Why' and 'How' questions and it seems to do a pretty good job.

The author doesn't provide a lot of contrary evidence.

There's so many articles saying "LLMs can't do X" that leave me wondering whether the author has even tried. Maybe they've tried and have some more sophisticated argument, but I often don't see it.

If I was going to knock LLMs for being unable to do basic science, in particular, I'd make sure to do some experiments first!

Spend some time around a three year old. Human, human intelligence, language skills.

Then try explain quicksort to them. Obvious waste of time.

They wouldn't be much threat in a zero sum strategic interaction either.

LLMs like GPT-4 'natively' speak certain syntaxes very well - e.g. Python, JSON. I'd suggest you want to take advantage of that, if at all possible, rather than embark on training or fine tuning your own LLM.

If you have a particular data structure you want to have the LLM generate or manipulate, which there aren't large quantities of in the training set, you might want to consider writing a translator that will translate it into a format the LLM natively 'speaks', using the LLM on that, and then translating back into your DSL.

Going this direction and also adding examples in some sort of vector store, as others have suggested, could be a good direction.

I personally received a letter to: "Fergal Reid, Longford, Ireland", in the late 90s. Ireland had no postcodes at the time.

I lived in a small village on the edge of a county called Longford; my name isn't that unique in Ireland.

The funny thing about the story, is that the letter was posted in the Philippines, and was physical spam mail, trying to hook me into some sort of scam. They had pulled the partial address from a domain name record. No idea how the economics of that worked!

Is this actually true, or BS? If true, can someone explain?

My mental model is that vsync will lock the framerate to the same as the monitor refresh.

So if monitor is 60hz, my games graphical update rate runs at 60fps.

Let's say without vsync, my graphical update rate would generously go up to 120hz.

Worst case we're talking about an additional input latency due to vsync of maybe 1/60s? 16ms?

I don't believe 16ms is significant for non pro level eSports players and even skeptical it's a big factor at that level.

What am I missing here?

I could understand if the get clock speed was tied to the graphical update rate but I presume that's not the case for CS, server side game etc; or even if it was it's still not going to be that material.

I'm just skeptical - what am I missing here?

Is the mouse input code tied to graphical clock rate or something in some surprising bad way?

Well, the stall he sets out is about 'generative models' generally:

"In the future, what role will these generative models play in assisting a programmer or mathematician?"

I think the fact that we've other generative models that eg. can write descriptive variable names is a fair rebuttal to his criticism of alphacodes poor variable names.

Deepmind fine tuned it on competition code, not on well documented production code, so I think it's shallow to get hung up on it's lack of good variable names (for example).

Huge respect for Norvig, but I think this is a shallow analysis.

For example, I just took Norvig's 'backspacer alpha' function and asked ChatGPT about it. It gave me an ok English language description. It names the variables more descriptively on command.

I'm sure it'll hallucinate and make errors, but I think we're all still learning about how to get the capabilities we want out of these models. I wouldn't rush to judgement about what they can and can't do based on what they did; shallow analysis can mislead both optimistically and pessimistically at the moment!

I feel sympathy for the artists, and who knows what jobs AI will leave!

But I don't think the artists I'm seeing on Twitter understand that, even in their best case, where training on a copyright image is considered total infringement, this genie still isn't going back in the bottle.

Models are getting better and training more efficient all the time.

Soon you'll be able to train a model entirely on public domain content and have great generative art output.

It might be harder to express the style you want than just sharing a name of an artist in that style, but that won't hold things back.

With this worldview the discussion about whether training on something is infringing its copyright fast becomes irrelevant, for better or worse.