great points; yes, the challenge becomes considerably more challenging with MCMC!
13 karma · joined April 3, 2015
give it a shot at let us know!
edward does not implement completely implement advi yet. the piece that is missing is the automated transformation of constrained latent variable spaces to the real coordinate space. however, edward offers much more flexibility in specifying the dependency structure of the posterior approximation. diagonal is, just like in stan, the fastest and easiest. however introducing structure (e.g. assigning a dense normal to some of the latent variables while assigning a diagonal to others) is much easier in edward.
our first approach is the simplest: stochastic variational inference. consider a likelihood that factorizes over datapoints. stochastic variational inference then computes stochastic gradients of the variational objective function at each iteration by subsampling a "minibatch" of data at random.
i reckon the techniques you suggest would work as we move forward!
please take a look here: http://edwardlib.org/troubleshooting
edward should answer some of your prayers :) there's still some time until stan goes parallel/gpu, though there's lots of interest there.
riemannin hmc is likely just around the corner!
1. you touch upon the right strengths of TF; that was certainly one consideration. edward is designed to address two goals that complement stan. the first is to be a platform for inference research: as such, edward is primarily a tool for machine learning researchers. the second is to support a wider class of models than stan (at the cost of not offering a "works out of the box" solution).
our recent whitepaper explains these goals in a bit more detail:
https://arxiv.org/pdf/1610.09787.pdf
2) no immediate plans. but we have HMC and are looking for volunteers :)
3) same answer as above :) should be relatively easy to implement tempering.
4) this is already in the works! stay tuned!
(shakir's blog posts are amazing; i recommend them all.)
the key disadvantages of VI (particularly ADVI) are:
1. mean-field variational inference cannot model posterior correlations. so if you expect your model + dataset to give a "skewed" posterior, then mean-field variational inference will have a difficult time describing such a posterior. (it will under estimate marginal variances.)
2. full-rank variational inference can model posterior correlations. but it can become too expensive for big models. there is a lot of great research coming up in this vein, such as [1,2].
3. in either case, the version of variational inference we have in Stan (ADVI) uses a normal approximation in a transformed parameter space. thus, there is an additional mismatch of the shape of the variational posterior to the full MCMC posterior.
in terms of advantages:
1. variational inference is (in general) a non-convex optimization problem. so it's easy to know when we've converged to a local optimum. convergence in MCMC is a bit more tricky to assess.
2. if your model has a multi-modal posterior, then variational inference will focus on just one of the modes. this is sometimes desirable as MCMC techniques might end up jumping around all of the modes and producing poor samples.
this is just the tip of the iceberg. but i hope it helps!
[1] http://arxiv.org/pdf/1506.03159.pdf [2] http://arxiv.org/pdf/1502.07685.pdf
ADVI [1] is a variant of BBVI [2] where we fully leverage all of the amazing things that Stan has to offer (like automatic differentiation and automatic transformations of constrained parameters).
you can use ADVI to get an approximation to the Bayesian posterior. the advantage of using ADVI over sampling is that ADVI is typically faster for large models (both in terms of # of parameters and # of data observations). ADVI also a bit better at handling models with multi-modal posteriors, such as mixture models.
ADVI is currently in cmdStan (but not in RStan or PyStan). we're continuing to make the algorithm more robust.
[1] http://arxiv.org/abs/1506.03431 [2] http://www.jmlr.org/proceedings/papers/v33/ranganath14.pdf
we're also working on making it more robust to initialization and step-sizes. stay tuned.
happy to answer any questions here.