Hacker News Reader

Comment by colinsane | Hacker News Reader

> What we really need is a model that you can run on your own hardware on site.

we won’t have that until we come up with a better way to fund these things. “””Open””” AI was founded on that idea, had the most likely chance of anyone in reaching it: even going into things with that intent they failed and switched to lock down the distribution of their models, somehow managed to be bought by MS despite the original non-profit-like structure. you just won’t see what you’re asking for for however long this field is dominated by the profit motive.

View on HN

Nah, it's already being done for GPT-3's competitors and will likely be done soon for GPT-4's competitors

https://arstechnica.com/information-technology/2023/03/you-c...

Curious why even companies at the very edge of innovation are unable to build moats?

I know nothing about AI, but when DALLE was released, I was under the impression that the leap of tech here is so crazy that no one is going to beat OpenAI at it. We have a bunch now: Stable Diffusion, MidJourney, lots of parallel projects that are similar.

Is it because OpenAI was sharing their secret sauce? Or is it that the sauce isn’t that special?

I think it's because everyone's swimming in the same bath. People move around between companies, things are whispered, papers are published, techniques are mentioned and details filled in, products are backwards-engineered. Progress is incremental.

Google got a patent on transfomers but didn't enforce it.

If it wasn't for patents you'd never get a moat from technology. Google, Facebook, Apple and all have a moat because of two sided markets: advertisers go where the audience is, app makers go where the users are.

(There's another kind of "tech" company that is wrongly lumped in with the others, this is an overcapitalized company that looks like it has a moat because it is overcapitalized and able to lose money to win market share. This includes Amazon, Uber and Netflix.)

I don't think this is strictly true, though it's rare. The easiest example is the semiconductor industry. ASML's high end lithography machines are basically alien and cannot be reproduced by anyone else. China has spent billions trying. I don't even think there's a way to make the IP public because of how much of it is in people's heads and in the processes in place. I wonder how much money, time and ASML resources it would take to stand up a completely separate company that can do what ASML does assuming that ASML could dedicate 100% of their time in assisting in training the personnel at said company.

Yeah, this is probably also true for TSMC, Intel and ARM. Look how slow progress is on RISC-V on the high end despite RISC-V having the best academic talent.

I would also add Samsung semi to that list. As I understand, for the small nodes, everyone is using ASML. That's a bit scary to me.

About RISC-V: What does you think is different about RISC-V vs ARM? I can only think that ARM has been used in the wild for longer, so there is a meaningful feedback loop. Designers can incorporate this feedback into future designs. Don't give up hope on RISC-V too soon! It might have a place in IoT which needs more diverse compute.

>despite RISC-V having the best academic talent.

academic performance is a bad predictor for real world performance

It's a decent predictor of real world performance just not a perfect one.

Unfortunately, RISC-V, despite the "open source" marketing, is still basically dominated by one company (SiFive) that designs all the commercial cores. They also employ everyone who writes the spec, so the current "compiled" spec document is about 5 years behind the actual production ISA. Intel and others are trying to break this monopoly right now.

Compare this to the AI ecosystem and you get a huge difference. The architecture of these AI systems is pretty well-known despite not being "open," and there is a tremendous amount of competition.

> the current "compiled" spec document is about 5 years behind the actual production ISA

How could I verify this information?

That was my question, too. What instructions have been undocumented for five years? What non-standardized extensions exist in SiFive cores?

Read the RISC-V foundation website. There are numerous "ratified" parts of the RISC-V instruction set that are not in the latest "compiled" spec document.

Saying a "compiled" spec is out of date may be technically accurate (or not, I don't have any idea) but if open, published documentation of the ratified extensions is on the web site, it's misleading to cite it as evidence that the spec is not open. And I know that the draft specifications are open for public comment prior to being ratified, so it's not a secret what's under development, either.

I never said that it wasn't actually open source. I just said that the openness hasn't actually created meaningful competition, because there is a single company in control of the specs that abuses that control to create a moat.

For a concrete example, the bitmanip extensions (which provide significant increases in MIPS/MHz) were used by SiFive in commercial cores before ratification and finalization. No other company could do that because SiFive employees could just change the spec if they did. They're doing the same thing with vector/SIMD instructions now to support their machine learning ambitions.

It's kind of hilarious how complex some "reduced" instruction sets have become.

The semiconductor industry is only tangentially or partially a tech company. They're producing physical goods that require complex physical manufacturing processes. The means of production are expensive, complex, and require significant expertise to operate once set up. The whole thing involves multiple levels of complex engineering challenges. Even if you wanted to make a small handful of chips, you'd still have to go through all that.

Most modern tech companies are software companies. To them, the means of production are a commodity server in a rack. It might be an expensive server, but that's actually dependent on scale. It might even be a personal computer on a desk, or a smartphone in a pocket. Further, while creating software is highly technical, duplicating it is probably the most trivial computing operation that exists. Not that distribution is trivial (although it certainly can be) just that if you have one copy of software or data, you have enough software or data for 8 billion people.

So you mean "Software" not "tech".

That's the comment you should have responded with instead of the one that you did.

Upthread used the term "tech" when the thread is very clearly talking about AI. AI is software, but because they used the term "tech" you cherry-picked non-software tech as a counter example. It doesn't fit because the type of tech that GPT-4 represents doesn't have the manufacturing cost like a chip fab does. It's totally different in kind regardless of the fact that they're both termed "tech".

That is literally technology. It just isn’t as software heavy as you like?

No, I think it's very clear that upthread is talking about how software is difficult to build a moat around.

Chip fabs are literally one of the most expensive facilities ever created. Saying that because they don't need a special moat so therefore nothing in tech ever needs a special moat is so willfully blind that it borders on disingenuity.

I don't think it's at all clear that upthread is exclusively talking about software.

The first use of "moat" upthread:

> Curious why even companies at the very edge of innovation are unable to build moats?

> Google got a patent on transfomers but didn't enforce it.

Google's Transformer patent isn't relevant to GPT at all. https://patents.google.com/patent/US10452978B2/en

They patented the original Transformer encoder-decoder architecture. But most modern models are built either only out of encoders (the BERT family) or only out of decoders (the GPT family).

Even if they wanted to enforce their patent, they couldn't. It's a classic problem with patenting things that every lawyer warns you about "what if someone could make a change to circumvent your patent".

You can't tell unless you read the claims thoroughly. Degenerate use cases can be covered by general claims.

Indeed. I read the claims. You can too. They're short.

Are you kidding? There are 30 claims, it's an hours' work to make complete sense of how these work together and what they possibly do/do not cover. I've filed my own patents so have read thru enough of prior art and am not doing it for a pointless internet argument.

IANAL. I looked through the patent, not just the Claims. I certainly didn't read all of it. But while it leaves open many possible variations, it's a patent for sequence transduction and it's quite explicit everywhere that the system comprises a decoder and an encoder (see Claim 1, the most vague) and nowhere did I see any hint that you could leave out one or the other or that you could leave out the encoder-decoder attention submodule (the "degenerate use-case" you suggested). The patent is only about sequence transduction (e.g. in translation).

Now an encoder+decoder is very similar to a decoder-only transformer, but it's certainly an inventive step to make that modification and I'm pretty sure the patent doesn't contain it. It does describe all the other pieces of a decoder/encoder-only transformer though, despite not being covered by any of the claims, and I have no idea what a court would think about that since IANAL.

Wait until Google goes down inevitably, then they will apply all their legal force just to save their sinking ship.

Or, Amazon, Uber, and Netflix have access to so much capital based on investors' judgment that they will be able to win and protect market share by effective execution, thereby creating a defensible moat.

I think his point was that If that moat doesn't exist without the ongoing context of more money being thrown at it then it isn't a moat.

Isn't MidJourney a fork of Stable Diffusion?

Not originally, MidJourney came out before Stable Diffusion

One of the middle version models was, but the first and latest model versions are homegrown.

It's because moving forward is hard, but moving backward when you know what the space of answers is, is much easier.

Once you know that OpenAI gets a certain set of results with roughly technology X, it's much easier to recreate that work than to do it in the first place.

This is true of most technology. Inventing the telephone is something, but if you told a competent engineer the basic idea, they'd be able to do it 50 years earlier no problem.

Same with flight. There are some really tricky problems with counter-intuitive answers (like how stalls work and how turning should work; which still mess up new pilots today). The space of possible answers is huge, and even the questions themselves are very unclear. It took the Wright brothers years of experiments to understand that they were stalling their wing. But once you have the basic questions and their rough answers, any amateur can build a plane today in their shed.

I agree with your overall point, but I don't think that we'd be able to get the telephone 50 years earlier because of how many other industries had to align to allow for its invention. Insulated wire didn't readily or cheaply come in spools until after the telegraph in the 1840's. The telephone was in 1876 so 50 years earlier was 1826.

You didn't mention it explicitly but I think the morale factor is also huge. Once you know it's possible, it does away with all those fears of wasted nights/weekends/resources/etc for something that might not actually be possible.

I'm not sure how "keep the secret sauce secret and only offer it as a service" isn't a moat? Here the 'secret sauce' is the training data and the trained network, not the methodology, but the way they're going, it's only a matter of time before they start withholding key details of the methodology too.

Luckily ML isn't that complicated. People will find out stuff without the cool kids at OpenAI telling them.

>Or is it that the sauce isn’t that special?

Most likely this.

OpenAI can't build a moat because OpenAI isn't a new vertical, or even a complete product.

Right now the magical demo is being paraded around, exploiting the same "worse is better" that toppled previous ivory towers of computing. It's helpful while the real product development happens elsewhere, since it keeps investors hyped about something.

The new verticals seem smaller than all of AI/ML. One company dominating ML is about as likely as a single source owning the living room or the smartphones or the web. That's a platitude for companies to woo their shareholders and for regulators to point at while doing their job. ML dominating the living room or smartphones or the web or education or professional work is equally unrealistic.

I also expect a high moat, especially regarding training data.

But the counter for the high moat would be the atomic bomb -- the soviets were able to build it for a fraction of what it cost the US because the hard parts were leaked to them.

GPT-3 afik is an easier picking because they used a bigger model than necessary, but afterwards there appeared guidelines about model size vs. training data, so GPT-4 probably won't be as easily trimmed down.

The sauce really doesn't seem all that special.

Because we are headed to a world of semi-automated luxury socialism. Having a genius at your service for less than $1000 per year is just an insane break to the system we live in. We all need to think hard about how to design the world we want to live in.

You can have the most special sauce in the world but if you're hiding it in the closet because you fear that it will hurt sales of your classic sauce then don't be surprised with what will happen (also known as Innovators Dilemma)

> Or is it that the sauce isn’t that special?

The sauce is special, but the recipe is already known. Most of the stuff things like LLMs are based on comes from published research, so in principle coming up with the architecture that can do something very close, is doable to everyone with the skills to understand the research material.

The problems start with a) taking the architecture to a finished and fine tuned model and b) running that model. Because now we are talking about non-trivial amounts of compute, storage and bandwidth, so quite simple resources suddenly become a very real problem.

> we won’t have that until we come up with a better way to fund these things.

Isn't this already happening with LLaMA and Dalai etc.? Already now you can run Whisper yourself. And you can run a model almost as powerful as gpt-3.5-turbo. So I can't see why it's out of bounds that we'll be able to host a model as powerful as gpt4.0 on our own (highly specced) Mac Studio M3s, or whatever it may be.

Reply on news.ycombinator.com