For a more informed opinion than folks on the internet, here's some work from Microsoft with early/internal access to gpt4: https://arxiv.org/abs/2303.12712 . I don't think people close by these systems share the same dumb parrot sentiment at all.
181 karma · joined September 14, 2010
From my understanding of the issue, mmap'ing the file is showing that inference is only accessing a fraction of the weight data.
Doesn't the forward pass necessitate accessing all the weights and not a fraction of them?
The point isn't that these things haven't existed in some form forever, it's that they're scaling to points where these effects are becoming increasingly dangerous, proportional to the growth of these systems.
The problem is that in 2022, google personalizes (aka re-ranks) the SERPs on enough dimensions (device type, location, user, etc) that a single mapping of search query -> ranking fails to capture any of the nuance that google is applying to their user understanding. I'm not sure how or when the SEO space will actually reckon with this, or if they'll keep just pushing poor 'visibility' metrics that can't actually be reduced to a single dimension.
For a random business that wants to advertise online, without the infrastructure and data capability to back it, they will struggle to compete unless they exist in a segment full of similar peers. When the former happens, we see articles about how PPC doesn't actually work, etc.
Reality is that it takes engineering work and infrastructure, coupled with some data capabilities to unlock real value in the online advertising space.
As noted, online advertising brought all sorts of insight and visibility over traditional 'offline' marketing channels, but with that comes more savvy competitors that will do all the data things you're not.
It's a strange intersection of needs where one wants or needs something like CUDA, but doesn't care to ensure their computation is actually running optimally. If you don't want to be bothered with control and granularity, why are you trying to write high performance CUDA code in the first place?
Would you mind elaborating as to what your hobby project was?
The harder part here will be getting access to the compute required, but again, the folks involved in this project have access to lots of resources (they've already trained models of this size). We'll likely see some trained checkpoints as soon as they're done converging.
According to his CV he's been active in the field for quite some time. The default assumption that he's an idiot and going to kill people just seems too cynical here.
Grandparent - you specifically mention having noted methodology problems, would you mind sharing where in the methodology you think he's gone wrong?
If a given piece of silicon is hosing up a GEMM (matrix multiply), in graphics scenarios this may be invisible to the human eye as it could potentially just introduce artifacts in a scene rendering that could be entirely ephemeral to the frame.
In the case of crypto mining though, it's completely possible (probable?) that there are GPUs that can't possibly ever calculate a proper SHA3 hash (see the paper on AES instructions that fail in symmetric ways).
Given their conclusion that this is the result of manufacturing tiny process transistors, it follows that this can occur in GPUs as well.
If their data is correct, it should follow that these exact issues will happen on the small transistor process GPUs as well.
The difference in data here is obviously scale, google has -way- more CPUs than GPUs so the absolute counts of failures will be different.
Enabling the general software community access to SOTA language models will absolutely unlock an order of magnitude more money (than 100mm) over time. At least for now their obvious strategy for capturing this value is providing these APIs to enable it, and I suspect they'll gladly host such versions for the orgs that don't have the capacity to fine tune / host their own LLMs.
Huggingface is releasing APIs and model checkpoints that allow any random internet user to execute (almost) SOTA language models in production. FYI - that's an amazing leap forward and a strong piece of kit for MLEs to have access to.
So let me rephrase your question: Is general access to SOTA language models worth 100mm to the software market?
I suspect the answer is a resounding yes.
> I guess you could interpret that to mean all these people building these products made a bad choice that succeeded despite using Python but I’d interpret it as another instance of Worse is Better. Just like Linus was told monolithic kernels were the wrong tool for the job but we’re all running Linux anyway.
This isn't the correct perspective or take away. The 'tool' for the job when you're talking about building/scaling a website changes over time as the business requirements shift. When you're trying to find market fit, iterating quickly using 'RAD' style tools is what you need to be doing. Once you've found that fit and you need to scale, those tools will need to be replaced by things that are capable of scaling accordingly.
Evaluating this binary right choice / wrong choice only makes sense when qualified with a point in time and or scale.
This is meant to be a technical blog post and the only technical description of the system/model pipeline involves telling us its a GBDT. That's unfortunately probably the least interesting thing you could pick out to say about such a recommender.
The content briefly touches on real time features and building a feature store. Tell us about that. Tell us about how you measured the improvement of adding near real time features. Explain how this stateless microservice scales horizontally. Tell us about tail latencies while ranking hundreds of rooms for a user. Just something.