saiojd

Three.js works just fine with functions like `.add`, it sure is ugly though. It kind of blows the mind that javascript has had so many syntactic additions over the years but still has no operator overloading.

Another thing that TS does nicely is object handling in general: dot access for objects attributes, object destructuring, typed objects for function options. In most ML projects I see a bunch of functions that look like:

    def my_fn(x, **kwargs):
       ...
       return y_1, y_2, y_3

Which is a pain because kwargs could be anything really + now every call site has to expect 3 return values exactly while knowing their order; there's no way of adding an extra return value without changing everyone. In typescript the same function could look like:

    function myFn(x, options = { someOption: 1 }) {
       ...
       return { y_1, y_2, y_3 };
    }

Which is so much nicer because everything is typed with all types inferred automatically! And you don't burden the call sites with values they don't need:

    const { y_1 } = myFn(x, { someOption: 1 });

In Python, everyone mostly passes unbundled arguments through every function, and changing anything involves threading these untyped arguments through a bunch of untyped call sites, its not the end of the world but we can do better...

I feel like you are being unfairly dismissive. Just imagine the stress of being targeted by the police because you match some criminals statistically.

Since you seem knowledgable on this topic, what is it that TPUs do differently than GPUs? Why are they better at inference?

We've all been there :D

That's a good point. The more I think about it the more what I said was a way of rationalizing not agreeing with the parent comment. Even if there were no ressource constrains it still doesn't feel selfish to me not to have children.

I have to ask: when posting a question like this, what kind of response are you hoping to get?

Not really selfish at all when you consider that the Earth's population cannot grow infinitely. Some people have kids, others don't, not a big deal.

Would you mind defining 'glut' in the context of your post? What you're saying seems very interesting but I can't quite wrap my head around it.

Thanks for sharing, that's interesting angle that I did not consider. So essentially the argument goes that more "surface area" == more vulnerabilities? While that makes perfect sense, don't you feel like the pressure for more feature comes from somewhere? By analogy, dynamic languages like JS/Python make the job of people writing the compiler much, much harder. But, on the whole and despite all their flaws, it is true that they increased productivity by a lot.

Why does it bother you that browsers have many features? At this point browsers are essentially a cross-platform VM and it really seems like this trend will only accentuate in the future.

I feel very sad reading this. You are absolutely correct that I view HN as watercooler where I make hyperbolic statements, but incorrect when you say that that I did not take you seriously. I was frustrated because I felt like you were latching on to a figure of speech just to be argumentative instead of trying to understand what I meant (I was not trying to say that neural networks are literally new, I was trying to say that we have no proof that the parameter to flop ratio is even close to optimal because architectures are still changing a lot).

You are right that neural networks are not in their infancy. Best wishes.

Your tone comes off as very adversarial/angry. Intentional or not? The claim is about the likelihood of more efficient approaches existing, not about the subjective qualifier preceding it (misunderstanding?). Yes, NNs have existed for a while now, but they will also exist for a long time forward in the future (hence the perhaps poor choice of word: infancy).

"if you want to know whether neural nets are in their 'infancy' you shouldn't make 'observations'" -> I think you have understood this as "neural networks did not exist before" when what I meant is "neural networks will still change a lot in the future"

I was interested in discussing, specifically: will NNs continue scaling up in size in the near/long term. In answer to the statement "the training costs will exceed all the compute capacity in existence", I added "it is very likely that more efficient approaches exist.". You replied that NNs have existed for a long time, and that because of this, it is a complete fantasy that more efficient approaches exist. Do you feel like this is a fair assessment or no?

Interesting, I had not considered these points outside of file size! Do you think it is possible they will be overcome or is the chance 0?

I wouldn't be so confident. Flash attention is a recent, significant improvement to training times and sure looks low-hanging now.

I was not familiar with Edge AI, interesting concept. I feel like improving the efficiency of very large models is much more likely.

The recent successes will lead to an even larger influx of $ in the short term --- this is an existential threat to Google, after all. We will see where things go!

> Where does it originate from, I wonder? I suspect someone said something like that on social media, their post went viral, and now all of the internet is reverberating with this thing. It's a meme, yes?

I don't have social media outside of HN.

It comes from a few observations:

1) Large models are still improving with increased parameter counts (we do not know where the ceiling is yet; it could be low but it could also be high).

2) Most current architectures train by using all model parameters to produce an output, which is vastly inefficient. While it is not clear how to improve on this in the general case yet, in the simpler problem of NERFs, sidestepping this issue has led to a ~100x improvement in training time.

3) https://mingukkang.github.io/GigaGAN/ very recently increased the parameter count of StyleGAN by selecting parameters dynamically at runtime. They improved on previous results by a very, very large margin, at somewhat comparable training times.

I stand by my claim: "neural network architectures are still in their infancy, it is very likely that more efficient approaches exist". I am not claiming that AI will become sentient or anything crazy and do not understand why you are associating my point of view with other people. I just said that it is likely that a novel technology will continue to improve (has this it ever NOT been the case for any new technology?).

Interesting observation. Why do you think it's like this?

I would not be so sure about compute capacity? Neural network architectures are still in their infancy, it is very likely that more efficient approaches exist.

I really wish compiling cuda extensions worked better out of the box. Is there a reason they can't bundle nvcc alongside pytorch outside of complexity/expense?

I don't understand what you meant by "nobody knows what color a combination of HSV values" (but you quote HEX strings)? What makes the quoted code hard to maintain?

Very cool, I'd love to see a result dump of random samples

At this point yes. But the goal of "automatic programming" against a formal specification has a long history. It does not seem impossible to me that, in the future, programming becomes closer to prompting + formal specification. Then the prompt can be compiled and validated against the spec.

I'm not sure I agree with the assertion "you can be confident that a compiler will express your program in the way you want". This is only if you program perfectly, which is never the case in practice (after all, bugs exist). What is different is how we can verify correctness (i.e. we read code to check correctness, but reading a text prompt carefully does not help if the LLM is sloppy). For code that is easy to test (i.e. passing a test suite convinces you that the code works, without even reading the code), I can see how the argument that LLMs are compilers makes sense.

What frustrated me with your post was not the argument itself but the sentence "it's fascinating to see the support that this grift has gotten" which feels like flamebait (sorry, I can see how that was not clear).

Hahaha gotta love your terrible acronyms :)

I'm calling you out: you're being edgy for the sake of being edgy by saying it's grift.

It's ugly but you can fix this with `dataclass_transform`

I agree that in the general case it does not work, but if we were to introduce an alternate dataclass/class definition syntax, I feel like something lightweight like this would be nice. (Local functions could just be prefixed with underscores, no need to distinguish them from private methods.)

Probably a controversial opinion, but I would like `self` removed, and have the class body be an init function. Something like:

```

dataclass NormalizedPoint(x: float, y: float):

    _norm = math.sqrt(x\*2 + y\*2)
    x /= _norm    
    y /= _norm

    def __add__(other: NormalizedPoint):
        return NormalizedPoint(x + other.x, y + other.y)

```

In other words, I would really like some construct which sticks as close as possible to a normal function, but that also returns a (typed) object with everything in its scope.

I personally just use `dataclass(kw_only=True)`, which should really be the default in most cases anyways.