benanne

Glad to hear it! My stubbornness about this makes me feel like a luddite sometimes, but the additional effort required is probably still worth it, for the time being.

I suppose what I was trying to say is that in the research community, people were still a lot more willing to entertain alternative modelling paradigms for language at that point, there was much less of a monoculture than there is today. It was really only after ChatGPT that non-autoregressive language modelling came to be seen as a fringe pursuit. Of course that is a highly subjective assessment, and my perspective is inevitably coloured by my own research interests at the time, and those of the people around me.

For what it's worth, I don't believe post-training diffusion models (of language or otherwise) is uniquely difficult, just a lot less thoroughly explored so far.

Thanks for pointing this out! I'm not sure why, the files are still on my Dropbox, they must have changed the link format at some point? I've gone ahead and fixed them.

It kind of does! In the modern era of generative modelling, it seems like we rely on pre-training to capture the data distribution, and then on post-training (and various other tricks) to carve out a sliver of that distribution that we actually care about (i.e. what we want our model to generate).

To be able to specify that subset with relatively few examples, a good high-level understanding of the data distribution is necessary. The way I see this, is that training a diffusion model gets you to that point, and then once you've selected the part of the distribution you actually care about, you can distill it down quite aggressively, because you no longer need all of that computation to model a much simpler distribution (sometimes all the way to one step, but usually it's a few steps in practice).

I briefly covered that connection in an earlier blog post: https://sander.ai/2023/07/20/perspectives.html#flow ... but it's definitely something that might deserve a longer-form treatment at some point :)

I actually wrote down some thoughts about audio phase in a previous blog post: https://sander.ai/2020/03/24/audio-generation.html#motivatio...

I have an example audio clip in there where the phase information has been replaced with random noise, so you can perceive the effect. It certainly does matter perceptually, but it is tricky to model, and small "vocoder" models do a decent job of filling it in post-hoc.

Sorry to hear that. My blog posts are intended to build intuition. I also write academic papers, which of course involves a different standard of rigour. Perhaps you'd prefer those, only one of those is about diffusion models though.

I'm not sure if frequency decomposition makes sense for anything that's not grid-structured, but there is certainly evidence that there is positive "transfer" between generative modelling tasks in vastly different domains, implying that there are some underlying universal statistics which occur in almost all data modalities that we care about.

That said, the gap between perceptual modalities (image, video, sound) and language is quite large in this regard, and probably also partially explains why we currently use different modelling paradigms for them.

Oof, you're not going to like this other blog post I wrote then :D https://sander.ai/2023/07/20/perspectives.html

Thanks for reading! Absolutely, I included a few references that explore that approach at the bottom of section 4 (last two paragraphs).

Thanks for reading! The paper that directly inspired this blog post actually investigates the latter (blurring as the corruption process): https://arxiv.org/abs/2206.13397

Thanks for reading! Check out subspace diffusion: https://arxiv.org/abs/2205.01490

I've since moved on to work primarily on diffusion models, so I have a series of blog posts about that topic as well!

- https://sander.ai/2022/01/31/diffusion.html is about the link between diffusion models and denoising autoencoders, IMO the easiest to understand out of all interpretations; - https://sander.ai/2023/07/20/perspectives.html covers a slew of different perspectives on diffusion models (including the "autoencoder" one).

In a nutshell, diffusion models break up the difficult task of generating natural signals (such as images or sound) into many smaller partial denoising tasks. This is done by defining a corruption process that gradually adds noise to an input until all of the signal is drowned out (this is the "diffusion"), and then learning how to invert that process step-by-step.

This is not dissimilar to how modern language models work: they break up the task of generating text into a series of easier next-word-prediction tasks. In both cases, the model only solves a small part of the problem at a time, and you apply it repeatedly to generate a signal.

In Georgian, mother is "deda" and father is "mama", which lends further evidence to this :)

A nice property of the model is that it is easy to compute exact log-likelihoods for both training data and unseen data, so one can actually measure the degree of overfitting (which is not true for many other types of generative models). Another nice property of the model is that it seems to be extremely resilient to overfitting, based on these measurements.

Agreed. Theano has trouble dealing efficiently with very deeply nested graphs.

Why wouldn't this work in Theano?

    >>> import theano
    >>> import theano.tensor as T
    >>> state = theano.shared(1.0)
    >>> states = []
    >>> for step in range(10):
    >>>     state = state + state
    >>>     states.append(state)
    >>> 
    >>> f = theano.function([], states)
    >>> f()
    [array(2.0),
     array(4.0),
     array(8.0),
     array(16.0),
     array(32.0),
     array(64.0),
     array(128.0),
     array(256.0),
     array(512.0),
     array(1024.0)]

Nobody forced us to open-source Lasagne, so I think that remark was a bit unfair. If we really didn't care about anything but graduating, why would we bother going through the trouble of sharing the code in the first place?

But I do see your point. Google obviously has a lot more manpower to spend on this, so it might be a better bet in the long run.

It's also worth comparing this to a few similar projects that have been announced recently: MXNet (http://mxnet.readthedocs.org/en/latest/), Chainer (http://chainer.org/) and CGT (http://rll.berkeley.edu/cgt/). And Theano of course, which has been the flag-bearer of the computational graph approach in deep learning for many years.

I have some doubts about this. Deep learning moves fast and DBNs are pretty much outdated models, even for unsupervised pre-training. It doesn't make much sense to me that unsupervised pre-training would help for this problem to begin with, seeing as their dataset totals around 65TB.

The paper is worth checking out: http://arxiv.org/abs/1509.03602 I haven't read it in full, but based on a quick skim, the convnet architectures they evaluated seem laughably tiny and shallow (at most three convolutional layers) by today's standards -- although I appreciate that there may be other constraints at play here (limits on training time etc.).

But to claim that DBNs are better suited for this problem than convnets based on these results is quite far-fetched. I'm confident that a convnet could crush these results, given enough effort and time spent on hyperparameter tuning.

I find this part particularly misleading (section 6, page 13): "shape/edge based features which are predominantly learned by various Deep architectures are not very useful in learning data representations for satellite imagery. This explains the fact why traditional Deep architectures are not able to converge to the global optima even for reasonably large as well as Deep architectures."

The whole point of learning features is so that they are better suited for the task at hand. If "shape/edge based features" are not suitable to perform a particular task, then a properly trained convnet should not learn them. I think the conclusions drawn from this work would have been very different if the chosen network architectures were more sensible.

I feel like this guide comes about 5 years too late - RBMs as density models have been shown to be relatively weak, except in the case of binary data. For continuous data, you can often do better even with a simple Gaussian mixture model. Other than that they are cumbersome to train (the gradient needs to be approximated), and the for continuous variants training can be unstable unless you use very low learning rates.

They were very popular for unsupervised pre-training a while ago, but the utility of pre-training has greatly diminished. Unless you have a ton of unlabeled data and very few labels, it's not worth the effort. And if it is, you are better off using autoencoders for pre-training anyway. They are conceptually much simpler and easier to understand, and you'll get roughly the same results.

If you want to get started with deep learning, focus on feed-forward and recurrent neural nets instead, you'll get much more useful knowledge out of that. For most of the common deep learning use cases there is no need to bother with RBMs anymore.

Not to my knowledge. For speech it should be sort of doable to get enough data to train a large net and then do inceptionism-style stuff with it. For music, gathering enough training data is a huge challenge due to licensing issues, among other things. My main research topic is deep learning for music, so I'd love to try it sometime.

That has been there since the start :) it randomly picks from the chat suggestions every minute and acknowledges the one it picked. I'm one of Jonas's co authors, feel free to ask me anything as well!

Not yet, I start in about a week! :)

We sort of reverse-engineered this last week and set up a stream with live interactive "hallucinations": http://www.twitch.tv/317070

You can suggest what objects the network should dream about (combinations of two are also possible).

Our code will be published on GitHub later today!

One of the co-authors here, we are planning to release the source code on GitHub in a few days.

Indeed, 64 bits of accuracy is overkill for a lot of ML algorithms, where there is so much noise that the additional quantization noise due to low precision is negligible. Most deep neural nets are already being trained in single precision because that is what NVIDIA GPUs can do the fastest. There has been quite some research on reducing the bit depth further the last couple of years, see e.g. http://arxiv.org/abs/1502.02551 and http://arxiv.org/abs/1412.7024.

The current generation of GPUs already has limited support for half-precision operations, and the tools for using these operations in neural networks (dot product, convolution implementation) are starting to become available as well, which is awesome (see e.g. https://github.com/NervanaSystems/nervanagpu).

NVIDIA themselves have also noticed and better hardware support for low-precision arithmetic is coming in Pascal: http://techreport.com/news/27978/nvidia-pascal-to-feature-mi...

I guess... but that's so boring! ;)

I think so.

Unlike Tolkien, George R.R. Martin never created full-fledged languages for his books, but he did make up a bunch of words. I guess they didn't want to replace those with different words from other real-world languages.

One of the challenges for David Peterson was to 'retrofit' the languages he created to what Martin had already made up. He coined a bunch of words and phrases for the books, and those needed to make sense and be grammatically correct in the newly created languages.

The phrase "valar morghulis", meaning "all men must die" in High Valyrian is a nice example. Peterson made the -ar suffix indicate the collective number. This is expressed by 'all' in English, but in many languages this is a separate number from singular and plural. He also added a paucal number (meaning "a few"). vala (man), vali (men), valun (a few men), valar (all men).

This is great, thanks for sharing!