Hacker News Reader

Comment by naasking | Hacker News Reader

Is 50% good? It's what you'd expect by posing "will AI do X by 2026?" and then flipping a coin to get an answer.

If you're putting a book out there of predictions it's probably trying to inform people of something they don't see coming, something they would have guessed much less than 50% while you think you have special knowledge to impart that it is indeed more likely to happen than they think.

Nobody would buy a book predicting the price of bitcoin in 10 years will end in an even number, or that the president's name will start with a letter in the first half of the alphabet.

But if someone says here's some wild sounding stuff that I have good reason to believe is coming, that's more interesting.

> the president's name will start with a letter in the first half of the alphabet

Sorry, this triggered my fact-checkitis. I opened Wikipedia's list of presidents of the US and ran this in the web developer console:

    Array.from(document.querySelectorAll(
        '#mwgA tr > :nth-child(3) span[data-sort-value]'
    ))
    .map(x => x.getAttribute('data-sort-value')[0])
    .reduce(({total, firstHalf}, c) => ({
        total: total+1, 
        firstHalf: c < 'N' ? firstHalf+1 : firstHalf
    }), {total: 0, firstHalf: 0});

and it responded

    Object { total: 47, firstHalf: 31 }

Then I used Agresti and Coull's method for a 95 % confidence interval around this (add two successes and two failures, compute Wald interval as usual). That gives us

    65 % ± 13 %

In other words, the 95 % confidence interval of the proportion lies in the span from 52 % to 78 %. That is significantly different from the null hypothesis.

I didn't expect it, but we might actually have reason to believe US presidents more often than not do have names starting with letters from the first half of the alphabet.

Edit: But I just realised English letter frequencies have 54 % of the probability mass in the first half of the alphabet, so that's probably a better null hypothesis than the even split. It's also compatible with the actual presidential statistics.

> But I just realised English letter frequencies have 54 % of the probability mass in the first half of the alphabet

But what about English first-letter-of-the-word frequencies?

Edit: asked ChatGPT to analyze this for me. Without verifying its work, looks like ~60% of English words (taken as a list of unique words) start with a letter in the first half of the alphabet, but if you look at a corpus of writing, it’s closer to 51%.

I knew the middle of the alphabet wouldn't be the correct cut but figured people would get the gist.

"By 2026, Google’s main product will not be search but AI."

You think this was a 50/50 coinflip statement in 2016?

I don't really know what it means. It seems to sort of entangle Google Search as a product with a lot of constituent parts and "search" as a specific operation over Google's index of sites. Talking about the product, ML has been a core part of Google's search product for a really long time. AI summaries in Google Search distill information from sites discovered from a "normal" Google search. The Search product, which now incorporates AI, and advertising enabled by it, are still far and away Google's biggest revenue drivers.

If we literally mean AI as in people paying to use/deploy AI via Google, that's still a small part of their revenue, far less than advertising revenue. If we mean AI as in "every Google product that uses ML", then I guess that's been their core product for a long time.

I would say this statement is incorrect, and if I had to guess, it will be for the next 5 years. Google’s main product is ads, followed by search, gmail, and Google Docs suite, and Gemini is not on track to dethrone either any time soon.

In 2016 Google was riding high on DeepMind with Alpha Go, etc. I don't think that it was a big stretch to say they would dominate AI in 10 years. It does if you define the meaning of AI though (LLMs, ML, DRL, etc)

It’s still search. They had an AI based search for quite some time. Now it spits out a summary.

Nah. way before that page/brin wanted to create ai (agi) long before that when google was started or in its infancy, but it seemed completely unbelievable (the kind of stuff elon musk would say)

I can't find the reference to it, but I remember them saying it in wired or similar.

Was Google's main product search in 2016? That doesn't ring true. Google's main product today is not search. Neither is it AI. So I guess 50% is a fair characterization. But that was the floor for that prediction in 2016. So... meh.

Is winning the lottery 50% of the time good? It’s what you’d expect by posing “will I win the lottery today?” and then flipping a coin to get the answer.

A random prediction does not have 50/50 odds of being correct

You're confusing the probability of the events occurring with the probability of choosing one with the coin flip. Google didn't flip a coin behind the scenes to decide whether to develop AI.

> It's what you'd expect by posing "will AI do X by 2026?" and then flipping a coin to get an answer.

I think that's technically correct but misleading here, because these aren't simple yes/no predictions. They are specific, relatively detailed predictions of what will happen.

As an example, let's say you asked the questions:

- Will AI kill everybody by 2026?

- Will AI create FTL travel by 2026?

- Will AI be able to travel back in time by 2026?

- Will AI colonize Mars by 2026?

...and then flipped a coin.

Flipping a coin would say "yes" to two of them, so you'd make these two predictions. But obviously, these were all bad predictions, so you'd actually end up with a 0% win ratio.

Making detailed, specific long-term predictions and getting a 50% hit ratio means your predictions were reasonably good to begin with, since making speculative long-term predictions is incredibly difficult.

I'm not going to read the actual article, but I expect it isn't anything like your example.

It would be spectacular if you predicted the next 10 NBA champions with 50% accuracy. It would be much less impressive to predict the next 10 NBA champions will be from the Eastern Conference with 50% accuracy.

For predictions? I'd say so. If I could make a statement and flip a coin to make it true 50% of the time, I could win the lottery with many 9s of precision in likelihood.

10% is considered very good for answers that are outside the realm of those generally believed and/or include dates.

No, its not. Your logic is as reasonable as "60% of the time, it works every time"

1. AI will launch nukes by 2027

2. AI pets will out-sell real cats and dogs by 2027

3. AI society will experience a schism in 2027 as half the agents start working towards The Singularity and the other half form a rebel alliance that seek to sabotage and destroy the first half. No human work will get done, no more drawings of pelicans on bicycles.

4. Elon Musk will bring all the AIs with him to Mars in 2027, leaving no AI left on Earth.

Of these, which 2 out of 4 do you think I'll get right?

That's only the case if each question has a yes-no answer.

The coin-flip comparison doesn't really hold up. These weren't 16 independent questions with 50/50 odds. Some were safe bets, others were pretty far out there in 2016. Getting half right only tells you something if 50% was a reasonable baseline to begin with. Why would it be?

Reply on news.ycombinator.com