GPT Neo: open-source GPT model, with pretrained 1.3B & 2.7B weight models

This is not meant as a goad to you, but more just as more info for everyone, my understanding is it is an open source community of like minded people type project (as opposed to a bigco) and actively solicits contributions (by which I mean code and data) so anyone seeing room for improvement is welcome to step in from what I can tell.

I did find your comment helpful and informative; just adding another angle here.

Isn't GPT-3 the architecture? Are they doing something different or why would it not scale?

Enterprise customers that have no idea about the technical details will just hear about OpenAI's success in this fancy new model and assume that Eleuther can deliver.

I mean, most use cases for "big data" projects that are tiny in comparison with Alphabet's datasets will just work with GPT2 fine, probably.

And Enterprise customers that hear those claims and see some code, maybe some demo, is enough for them to start the consultancy process.

In my opinion that's a policy problem that OpenAI introduced by not requiring the absolute reproducability of both the code and model, and both training and dataset of their models upon release.

Stakes are pretty high in the AI industry, and OpenAI actively influences it. My dream was in the beginning that they are a source of verification, audits and "proof" that models are legit...yet I have the feeling lately that they just buzz around like everyone else.

To this date I haven't seen anyone replicate any of the DNC results, for example.

Anyways, just my two cents on this one.

I don't think any single thing you've claimed is factually wrong, and I don't speak for Eleuther nor am I attempting to justify their claims.

But.

As I understand it (mostly from lurking on their discord and reading publicly available materials) this is a group of volunteer academic types trying to replicate something great and awesome, with the only goal of giving it to the world. You could cut them some slack.

I can't speak for you, but as a "for free, weekend project" what they've done certainly makes me feel I need to up my game.

I suspect they have run into training issues since they are moving to a new repo[2]

[1] https://twitter.com/arankomatsuzaki/status/13737326468119674...

[2] https://github.com/EleutherAI/gpt-neox/

Right on the front they write:

    > Hey there fans! We are currently looking for help funding large storage upgrades, 
    > if you want to help us serve more data see our donation options (crypto, etc) 
    > Thanks for reading, happy downloading!

"In a shocking finding, scientists discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.

Bebek Uranzoglu, another member of the research team from the University of British Columbia, was working on a project the Latino-Canadian rodeo competition equipos to document a rare and remarkable ecosystem in the Andes Mountains.

His curiosity was piqued when he spotted an adolescent herd of about 10 unicorns foraging in a forest near the valley of the Jumbo Flu Group. The unicorns — whose numbers once swelled to 46,000 — were perched on the forest floor and watched the researchers work.

Urizoglu grew excited when he spotted another group that seemed to be thriving in an area below the herd. The team hoped the apparent population growth would indicate a human presence.

But when a team of researchers set up a camera trap, they were surprised to find the unicorns in the first place, and in a forest near a lake — in fact the forest was almost entirely made up of the animals. Despite their own magical presence, the team could not see the herd was populated by humans.

“The whole place almost smelled like animals,” says Bebek. “We were never able to find human footprints at any of the points we stood at. The trees were so large, you wouldn’t have been able to walk 40 meters through them. We assumed that the truth of the matter was, ‘Well the deer didn’t like this forest at all.’”

https://huggingface.co/transformers/task_summary.html#extrac... more precisely

I've been building a library that makes solving tasks like conversational Q&A easy.

https://github.com/backprop-ai/backprop

If you need some pointers, we have example notebooks and I'm happy to chat about it as well. Would love to know if it helps!

It seems really good at generating code too

prompt:

class ConvBlock(nn.Module):

    def __init__(self, channels, stride=1):

        super(ConvBlock, self).__init__()
        self.convolution = nn.Conv2d(channels//stride, channels, 3, stride=stride, padding=1)
        self.normalization = nn.BatchNorm2d(channels)
        #self.activation = nn.LeakyReLU()

    def forward(self, x):

        x =
        #  x = self.convolution(x)
        #  x = self.normalization(x)
        #  x = self.activation(x)

        x = self.convolution(x)
        x = self.normalization(x)
        x = self.activation(x)

        return x

    def __init__(self, stride=1):

        super(Resblock, self).__init__()
        self.res = nn.Sequential(
            nn.Conv2d(9, 9, 3, stride=stride, padding=1),
            nn.BatchNorm2d(9),
            nn.LeakyReLU(0.2)
        )
        self.output = nn.MaxPool2d(2)

    def forward(self, x):

        return self.res(x)

Big ugly stick.

Bowl full of chicken peas.

New York is universally inadvisable.

Long-eared seagull.

This morning I have a tumor in me.

A dog bark.

Invisible dog hides.

A crow blind.

A cat can’t see if it had a billion eyes.

Bumblebee.

Sheep herder, sheep herder.

A fawns abducts.

Two black birds are trapped.

Bottle on her finger.

Elephant sees another elephant.

Bull.

Mice in a box in the library.

A church swelter.

The door of a hotel opens.

Bosnian honey melons.

Grapes in excess.

Cat is on the loose.

Soil is shoveled into a glass jar

tl;dr we are doing pretty much exactly as well as we expected on LAMBADA and WikiText. Results on more sophisticated tasks will take some time, but HuggingFace is currently working on implementing our model in the transformers library and when they do so we can easily run a lot of analyses very quickly.

We actually built an evaluation suite that integrates with HF, but interfacing with the MTF code that GPT-Neo was written in was too much of a pain in the ass because Mesh TensorFlow is the worst. https://github.com/EleutherAI/lm-evaluation-harness

https://huggingface.co/pricing

https://backprop.co

Anyone can access it and our pricing is usage based. If you use less than 1000 seconds of inference a month then it's completely free.

You're better off either: 1- Defining common "intents" that a lot of customer queries are categorized into, and having a model map the incoming message to the appropriate canned response. Look at Rasa, for an example of this.

2- if you insist on generating the text, have it be a recommendation to a human agent that either chooses to send it or writes their own response.

If anything, you could build writing assistance that almost automates responses.

I've been co-authoring a library that lets you finetune such models in a single line of code.

https://github.com/backprop-ai/backprop

In specific the text generation finetuning example should be what you are looking for: https://github.com/backprop-ai/backprop/blob/main/examples/F...

Hope this helps, happy to chat more about it. Pretty curious about the results.

https://github.com/EleutherAI/gpt-neox/

[1]: https://colab.research.google.com/github/EleutherAI/GPTNeo/b...