stephenroller

with hf transformers, you still need to manage GPUs

Why did the share of childless women 35-44 suddenly drop 5 points around 2012? Such a massive difference must be a change in measurement methodology?

The llama1 team did not have a validation set. I don’t know what the Llama2 team did - I left before seeing any of the details.

My guess is Llama2 upsamples Wikipedia a good bit, but given they didn’t report any information about training data, it’s hard to say.

One noteworthy thing is that no one is posting validation curves, only training curves. All these models will happily bring training loss eventually to near zero with infinite compute, as the model overfits to the dataset -- there are no regularizers in any modern LLMs. The validation curves would be considerably more convincing.

The counter argument to above is that none of these models were really trained for multiple-epochs: it's hard to overfit data you've only seen once. But to go to 70T tokens, you'd inevitably have to start using many epochs.

Updating the model happens offline to ensure protections of this. As part of the research on BlenderBot 3 we also have improved detections for safety and can learn from "trolls." https://parl.ai/projects/trollhunting/

(Disclaimer: co-author)

We trained on Reddit comments and HackerNews comments.

I run a fairly large open source project (https://github.com/facebookresearch/ParlAI/) and we use mypy. Our experience has been that it can be quite difficult to placate, so we usually treat it only as a warning. However, having our code annotated with types in many places has significantly improved developer productivity, just from having less ambiguity with what you're dealing with.

https://github.com/pytorch/fairseq/pull/3331 and https://github.com/pytorch/fairseq/pull/3327

Support for this was also added to [Fairscale](https://fairscale.readthedocs.io/en/latest/) and [Fairseq](https://github.com/pytorch/fairseq) last week. In particular, the Fairscale implementation can be used in any pyotrch project without requiring the use of the Deepspeed trainer.

They don't. But half the people got a placebo and half got the vaccine. 185 in the placebo group got covid, with 30 having severe covid, but 11 with covid and 0 severe in the vaccine group got it.

So do you really think that totally by chance, one random group of 15,000 people got a ton of exposure, and another group of 15,000 had little-to-no exposure? The probability of that is quite low.

Streets would be much more useful than a true north compass.

Perhaps you are outside the United States? PBS is the public television channel, so perhaps they geolock :(

The dataset can be obtained around the web. It's mostly CommonCrawl, Reddit, Toronto Book Corpus, and Wikipedia.

You can find a very comparable corpus open sourced and easy to use on the [T5 repo](https://github.com/google-research/text-to-text-transfer-tra...)

No, they aren't releasing the weights. They are releasing it as ML as a service. Right now it's in free beta, but it will open up for commercial usage in the future.

On another note:

At 175B parameters, with float16 representations, the in memory footprint is about 350GB plus activations would take it to another 400GB. You would need 12 or 13 V100GB GPUs to hold it in memory, or three p3.8xlarge. Meaning loading it on AWS would cost around $35-40/hr.

Though if you didn't care about speed, you could load up the weights from disk one at a time and forward through it a few layers at a time on a single GPU.

I would not encourage using the model for anything other than AI research -- we're still in the early days of dialogue, and there are a lot of unexplored avenues. There are still nuances around safety, controlling generation, consistency, and knowledge involvement. For instance, the bot cannot remember what you said even a few turns ago, due to limitations in memory size.

In the paper, we did explore what happens when you do NOT fine tune it on the specialized tasks (knowledge, empathy and personality). The non-finetuned bot was both less engaging and more toxic. The special finetuning is really important to getting this bot to be as high quality as it is.

(BlenderBot original developer here) You can read the paper here, which contains much more technical details of how we built it: https://arxiv.org/abs/2004.13637

Hi there, thanks for asking! It's very cool to make research more accessible. Would you be able to rename your experience to "Modified Small BlenderBot without Safety Layer"? And I think it’d be helpful to call out what changes you made when you say it’s adapted from our research.

Hi there. One of the original authors of the BlenderBot paper here. This cocohub website is not what we released. It appears that cocohub has removed our safety layer and provided other modifications to the bot. We released our model open source, along with specific instructions on interacting with it safely: https://parl.ai/projects/recipes/

It's important to note that dialogue safety is a very important and nuanced topic, and we did our best to release a safety system attached to the model. Our system is not perfect though, and that is why BlenderBot was released as a research project for furthering the state of Artificial Intelligence, and is not meant for production purposes.

I would also mention that the blender small model significantly underperforms compared to the larger models released with the paper, and encourage everyone to try our best models, not our small one.

Embeddings tables aren't hard on the GPU (being only a lookup table), and the output softmax still requires you do the full matrix-multiply. The label may be sparse, but the computation is far from sparse.

We limited the scope of our study to the PersonaChat dataset, which is a standard benchmark in dialogue. In PersonaChat, both participants are presented with Personas and asked to roleplay as that person. (e.g. "I love the beach; Horses are my favorite animal; I am on a diet now").

So we restrict ourselves to meeting someone for the first time and trying to learn about them and share a bit about ourselves.

The personas that were assigned to the bot and the human were both randomized (from a large, fixed set) and unseen in training data.

Yes, that's a great response for one of the models to have. :)

We originally set out to see whether we could use conditional generation to sort of adjust the behavior of these neural generative dialogue systems. Much of the original idea was about sort of "planning" our response to improve the quality and diversity of neural generation.

As we studied though, we became more aware of how much nuance there was in each of these methods, especially when we started having humans chat with the models. In the end, the paper ended up being more of a thorough analysis into what behaviors are more preferred in dialogue agents. For example, we found that you should spend about ~60-70% of your turns asking questions to maximize engagement: enough to convey interest in your partner without coming off as super nosy.

Co-author of the original paper, if you have any questions.