Hacker News Reader

Comment by shiandow | Hacker News Reader

It's perhaps useful to note that their notation is not accidental.

When they state:

    P = 1/Z exp -li Hi

They're directly referring to the Gibbs distribution that pops up in thermodynamics. The Hi are the different terms of the Hamiltonian and the Z is the corresponding partition function.

From that point of view they're taking a couple of 'response functions' treating them as if they are part of some kind of energy and find the statistical distribution that such a system would have at different temperatures depending on the average value of each term. The next part is then to find the value for each term that best predicts the observed distribution.

It's a bit like observing how a system behaves for different volumes and then using that information to determine the temerature, pressure and volume of an observed system. (perhaps volume isn't the best example, but something like molarity and chemical potential is hard to visualize)

View on HN

Yes, but...

> some kind of energy

The negenergy (negative energy) here is just the (unnormalised, hence the 1/Z term) log-probability mass or density. Like the logit outputs of classifier.

You shouldn't read anything more into the 'energy'. There is no temperature parameter in these minimax models (although adding one to smooth the model during sampling or learning can be useful, to interpolate between pure noise and the learnt model).

I've found that reading more into these 'accidental' similarities is a lot more useful than not reading into it at all.

For what it's worth there is a parameter in front that functions as a kind of temperature, except there's more than one parameter so it's not an exact translation. You can still reason about it somewhat (or split it up into a single temperature times a convex combination of hamiltonians).

Sure, there are fruitful parallels. But actually, energy is much 'realer' or more fundamental here than temperature. BTW, those λ_j Lagrangian multipliers can be negative, so don't map onto inverse temperature well. They're more like interaction weights in a Hopfield network.

But there's one setting in which the energy E(x) = Σ_j λ_j H_j(x) really does become a physical (physics-esque) energy: if you sample from the distribution using Hamiltonian Monte Carlo (an MCMC algorithm). Then E(x) is the potential energy of a sample/particle at position x, which is also given a kinetic energy K with the sume E+K held constant as the particle moves about to explore the distribution. It's a cool algorithm but after having used it I'm still not sure how useful it is.

HMC allows you to jump quite a large distance by obeying the 'equations of motion' of the 'Hamiltonian'. This helps give more diverse samples, making them less correlated thereby increasing number the effective number of samples.

It is indeed quite cool how it works.

Right, but I mean that HMC has parameters to tune and multiples the computation needed per sample produced so it's not clear to me for which problems it's significantly beneficial and with how much tuning. I'd guess it's best where you otherwise don't have a good proposal distribution for Metropolis-Hastings leading to either many rejects or very slow mixing.

The NUTS sampler limits how much tuning you need. As far as I know it's currently one of the best algorithms for sampling the kind of posterior probability distributions you get with Bayesian methods.

Reply on news.ycombinator.com