HNHacker News
TopNewBestAskShowJobs

muskmusk

189 karma · joined October 30, 2021

submissionscomments
muskmusk··on LLMs are not greater than the sum of their parts: researchers
I generally tend to agree with you. However, to give the opposing camp some credit: causation is not always so obvious. For example I don't think today's LLM capabilities followed obviously from the state of affairs in let's say 2019.

Do we call that emergence or do we use another word? I don't care, but it would certainly be nice if we could agree to use the same word :)

To stay with the cookie analogy: i guess it depends on point of view as well. Everyone who can see that i am growing can predict that I will eventually reach the jar. But if I am a cookie inside the jar I can't see that and will only notice once the jar is finally reached.

muskmusk··on Giving GPT “Infinite” Knowledge
I agree with your general premise, but I think you left a couple of points of your list at the end:

it is obscenely expensive to keep training + there are other more low hanging fruit + you expect hardware to get better over time.

I don't think Altman is trying to fool anyone. Even if he were it wouldn't work. The competition is not that stupid and he knows that :)

It's just that hardware tends to get better at a rate that resembles Moore's law so in 18 months the cost of training a 100 mill dollar model is 50 mill dollar. You certainly can just throw money at the problem, but it's expensive and there are other options that are just as effective for now. Why spend money on things that are half as valuable in 18 months when you can spend money on things that don't devalue as fast like producing more/better data?

All that being said you can bet your ass there will be a gpt5 :)

muskmusk··on LLMs are not greater than the sum of their parts: researchers
> I don’t think this rebuts the significance of emergence, since metrics like exact match are what we ultimately want to optimize for many tasks. Consider asking ChatGPT what 15 + 23 is—you want the answer to be 38, and nothing else. Maybe 37 is closer to 38 than -2.591, but assigning some partial credit to that answer seems unhelpful for testing ability to do that task, and how to assign it would be arbitrary.

Not sure I can agree with this. Let's for the sake of the argument say that normal calculators don't exist. I can choose to run my calculation thought ChatGPT or do it myself. It is true that I would vastly prefer the answer to always be correct, but it's not true that there is never value in bounding the error.

Put another way: would you rather want a calculator that is at most 5% off 100% of the time. Or a calculator that is 100% correct 95% of the time, but may output garbage the remaining 5%?

Upon further reflection: it seems mighty ambitious to expect a statistical model to be always correct. If always correct was possible you probably didn't need a statistical model to begin with. Given that the model will fall from time to time it seems especially useful that you can bound the error somehow. +1 for smooth metrics i guess.

I guess all of this is slightly off a tangent as Wei seems to be just arguing that "there might be things LLM's can do or will learn to do in the future that we didn't train them for. And we won't necessarily be able to predict it from capabilities of smaller models". I agree with this, just not the way he arrives at the conclusion.

muskmusk··on LLMs are not greater than the sum of their parts: researchers
There seems to be an element of vague definitions causing problems here.

Humans grow linearly. We don't suddenly wake up 20 cm taller.

Linear as that may be it still shows emergent behaviour when I am suddenly able to reach the cookie jar on the shelf.

muskmusk··on ChatGPT has no inner monologue or meta-cognition
Ilya Sutskever says that one of his early insights is that neural networks can do most things that humans can do quickly. I think that applies here as well.

Some people can do math quickly in their head, but most people cant. They need pen and paper and if you give them pen and paper you get vastly better answers.

I would conjecture that there is a prompt where the LLM is better able to explain its process, but it will require more "pen and paper thinking".

It is true that humans generally introspect better than LLM's, but a lot of the time you will also see people producing garbage (they post rationalize). Human tendency to produce garbage introspection is usually tied to feelings about themselves, which the LLM doesn't have. Either way I find a lot of similarities between the two :)

muskmusk··on Are pull requests bad because they originate from open-source development?
Pr's are not bad. But using them to solve things that should be solved other things like catching bugs and enforcing style guides is bad.

If doing a pr takes longer than 5 minutes in general then consider if you are relying too much on pr's.

muskmusk··on Choose Your Weapon: Survival Strategies for Depressed AI Academics
> If we have learned one thing from deep learning, it is that scaling works. From the ImageNet [15] competitions and their various winners to ChatGPT, Gato [13], and most recently to GPT-4 [1], we have seen that more data and more compute yield quantitatively and often even qualitatively better results. (By the time you are reading this, that list of very recent AI milestones might very well be outdated.). Of course there are improvements to learning algorithms and network architectures as well, but these improvements are only really useful in the context of the massive scale of experiments. (Sutton talks about the “Bitter Pill”, referring to that simple methods that scale well always win the day when more compute becomes available [18].) A scale that is not achievable by academic researchers nowadays. As far as we can tell, the gap between the amount of compute available to ordinary researchers and the amount available to stay competitive is growing every year.

>This goes a long way to explain the resentment that many AI researchers in academia feel towards these companies. Healthy competition from your peers is one thing, but competition from someone that has so much resources that they can easily do things you could never, no matter how good your ideas are, is another thing.

This sounds like teenagers whining they can't all be popular. Science isn't a competition to be won so that you can get praise and attention. Science exists to discover things that then hopefully are useful to people. If someone that isn't you discovers something useful that isn't bad because then you can't discover it. It is good because that is a problem solved.

The resources requirements FOR SOME SPECIFIC PROBLEMS have gone up to a ridiculous degree, but there are plenty of problems left to solve. In fact a new one that is at least as important has been created: Replicate current results with less hardware/parameters/whatever.

The difference between having GPT4 as a slow and costly service that requires network calls vs having it locally with almost no cost will be a huge achievement. Stop sulking and get to work!

muskmusk··on Running Python micro-benchmarks using the ChatGPT Code Interpreter alpha
Out of curiosity, why not?
muskmusk··on Running Python micro-benchmarks using the ChatGPT Code Interpreter alpha
Rather than feeling scared I think you can use this to productively guide you.

Look through the prompts he is using. Do they strike you as something a random person can produce? Not really. They show that Simon has excellent understanding of sqlite and how to do benchmarks. All ChatGPT does for him is speed up the typing. My experience has been that the quality of output is heavily correlated with the quality of input. Good clear prompts give good output. I postulate that if Simon had worse understanding of benchmarks and sqlite he would not have gotten as good output.

If you as a developer make your money by doing what ChatGPT is doing (turning clear instructions into working code) then you are going to be automated away. If you as a developer make your money by having a good understanding of the tools you use and by communicating that understanding clearly, then you have to type less in the future.

The silver lining is that even without GPT good understanding of tools and clear communication were always the more important skills to learn. ChatGPT just solidifies existing structure.

muskmusk··on Building a Database in the 2020s (2022)
Assuming no-one is using the database yes. Also not trivial to set up and use plus vm startup is somewhat slow.

If your want a full featured product (in the postgres space) that does it then look at neon.tech (no affiliation).

muskmusk··on Pause Giant AI Experiments: An Open Letter
Let's say we in fact do agree to pause AI research for 6 months as proposed.

It could happen that the world comes together in perfect unison, figures out the regulatory details and then resumes research. It could also not happen. There are no signatures from any of the world's authoritarian countries. I predict that the next 6 months will then be used for quibbling over stupid details without any real progress, climate-change-style. We need regulatory guidance, but I don't see why stopping the research makes these regulations better or easier to make.

which is more likely:

- the world's authoritarian leaders will abide by this decree

- the world's authoritarian leaders will try and exploit the pause for their own gain

Go to work on fixing the fucking regulation rather than wasting time on petty petitions.

muskmusk··on ChatGPT Plugins and Integration
Sure :)
muskmusk··on I lost everything that made me love my job through Midjourney
Your job, no matter who you are, is always to please the customer. If you focus on that then you will be fine. But if your focus is a more selfish one then you could be exposed.
muskmusk··on Cheating is All You Need
Thank you. Glad to hear i am not the only one who sees this :)
muskmusk··on Adobe Firefly: AI Art Generator
Not really. You just have checkbox somewhere that says "don't copy" and then it won't.

Leave the decision to the user instead of baking it into the technology.

muskmusk··on Google releases Bard to a limited number of users in the US and UK
In the olden days before Satya Nadella. Microsoft was a bloodsucking vampire of a company that couldn't execute anything of significance, but they had a pretty solid research department.

Google could ofcourse sit on a much better version that isn't safe or economic, but they could also just be unable to do better. Good research department, but unable to execute anything significant. Time will tell I guess :)

muskmusk··on Meta Layoffs
I don't think the problem is one type of management vs another type of management. But about a completely disproportional application of one type of management.

Heavy coding and no management have a place in the world. But no coding and heavy management also has a place in the world. Coding "heats up" the environment and management "cools it down". Sometimes you need heat, sometimes you need to temper the heat.

Companies start out with all coding and no management because they need to just produce. Then when they have something valuable, they must reign in the cowboy culture with management. Unfortunately, this often ends up overdone and there are very few leaders who have the guts to prune the overmanaged companies. Let's see how it turns out for the Zuck.

muskmusk··on Meta Layoffs
This is almost always the prevailing idea. "Management are doing this on purpose".

Dont think that is the case. Its more like we hired a somewhat poor manager, that manager hired yet more poor managers who in turn hired even more poor managers. All of these poor managers finds ways to make themselves appear useful using what they know how to do: endlessly discuss things.

The above is a caricature, but I think it has more than one grain of truth to it.

muskmusk··on ChatGPT generated a puzzle game
>The "it's been done before" one is pretty relevant.

But is it? 99.999% of software development has been done before. Even if you do something that is legitimately new (like creating a chatbot that can generate code on demand). Then your solution will still contain more than 99% code that is just a repeat of things that have already been done.

muskmusk··on OpenAI is now everything it promised not to be: closed-source and for-profit
As another new thing it is also useful in the real world.
muskmusk··on TikTok just dropped a new realistic beauty filter
Not sure I understand the commotion. The same thing can be done in the real world with enough time and money. But somehow doing it digitally for free is a problem?

If teenage girls don't know how to handle this then maybe we should teach them?

muskmusk··on ChatGPT Plus
TAKE MY MONEY!
muskmusk··on OpenAI and Microsoft extend partnership
I think that if you could conclusively answer that question you would be sipping drinks on a beach somewhere. The people who are investing seems to think so. Also the applications of this tech is broader than search, but still includes it. A company that had a serious chance of eating at Google's search revenue while also generating new revenue streams. What is that worth? What if you already have 1000 products that would benefit from the new capabilities? This is probably an easy investment decision even if Microsoft gains nothing from the actual investment itself.
muskmusk··on Unbundling Tools for Thought
I use a single folder of files along with a script that can fuzzy search said folder. It works perfectly.

Knowledge is messy and doesn't really conform to a database schema. Certainly not one you would want to write :) I think a better approach would be to find a way to insert all these notes into something like ChatGPT.

muskmusk··on Ask HN: What field in computer science will be AI proof
It's not man Vs machine. It's man with machine. Learn to work effectively with "ai" and you will have no problem.
muskmusk··on FTX’s collapse was a crime, not an accident
I cant hold it in: Yeah it would be really convenient if there was some sort of global database that could hold all this data.
muskmusk··on Ask HN: How would you design an alternative Twitter
This question was relevant 15 years ago when the tech for building something like this did not exist. Today the tech exists so it really doesn't matter.

A still unsolved problem is how to moderate effectively. Spend your time with that problem instead.

muskmusk··on Ask HN: Are we all burned out?
https://hubermanlab.com/controlling-your-dopamine-for-motiva...
muskmusk··on Ask HN: What's the best source code you've read?
The one that I didn't have to read.

A mistake i see many engineers commit is obsess over their code and how "pretty" it is. What they seem to forget is that code is only read when it isn't doing what it's supposed to do. Granted when you do have to read it there are good and bad examples, but the question you should ask yourself is not "how can I make this code easier to read?" But "how can I make sure noone has to read this code". The code that performs this job is not always pretty. If it is impossible to make the code bulletproof then the best bet is usually to just make it really simple. Good code is nonexistent and if it has to exist it is boring.

muskmusk··on Ask HN: Can I see your scripts?
I have my notes in Dendron which is basically a directory of yaml files. I often need to search through the notes so I made the below

search_notes() { input=$(rg -v '(\-\-)|(^\s*$)' --line-number /home/user/some-dir | fzf --ansi --delimiter : --preview 'batcat --color=always {1} --highlight-line {2}' --preview-window 'up,60%,border-bottom,+{2}+3/3,~3' | choose -f : 0) if [[$input = ""]]; then else less $input fi }

It uses various linux utilities including fzf and batcat(https://github.com/sharkdp/bat) to open a terminal with all the places where my query comes up (supporting fuzzy search). Since the workhorses are fzf and ripgrep its is quite fast even for very large directories.

So i will do `search_notes postgres authentication`. I can select a line and it will open the file in less. Works like a charm!

← PreviousPage 2 of 3Next →