597 karma · joined August 23, 2008
In a sense, that's what this is though, right? It's a training algorithm for the simpler class of model. It just has to go via the deep model to get there.
The author might be interested in the theano library used to compute gradients for neural models in pylearn2. It supports performing some algebraic simplifications on the compute graph before running AD. I'm not sure if it supports exactly what's being proposed here, but it does seem to enable some interesting hybrid symbolic / automatic approaches.
2. Not systematically. I think a lot of the fun of it is seeing what you can make with the ingredients you have lying around and an art to supplementing the set of ingredients with things which will open up possibilities. Also fun to see what's marked down in the supermarket and figure out something you can do with it. That said I'll usually have one or two meals in mind for a week's shop (although might change my mind mid-shop depending on what's on offer). And it certainly helps to keep certain staples around that you know you can rely on for a lazy dinner if you don't have the energy for something more involved.
3. I like to change it up although certain things I keep coming back to in mild variations. Various indian curries, stews (particularly keen on persian and turkish), pasta or polenta with something homemade (fresh pesto, some kind of ragu, simple tomato sauce, putanesca, garlic prawns...), a classic english roast on a weekend, chinese stir-fries, attempts at thai and vietnamese, chilli, pulled pork, something vaguely spanish with chorizo and beans, or vaguely north african, salad, fish, stuffed veggies...
4. Not really. An occasional short-lived attempt to cut down on carbs and meat.
5. Don't rely too much on recipes, use them to bootstrap your intuitions about ingredients, the principles behind different cuisines and formats of dish. Try things, fail. Even if you fail most of the time it'll still be edible and perhaps even tasty. Taste things (obvious but bears saying). Learn about individual ingredients and how to use them, herbs spices and seasonings in particular. Seek out new exciting ones. Visit ethnic shops. Make use of the freezer: leftovers, stock, sauces, bargains which you snap up close to expiry. Know what freezes well and what doesn't. Get a rice cooker, they're great. Don't bother with a slow cooker, they create mush and dry meat. Stews are a great way to practise -- hard to screw up too badly, one pot, chance to try out various seasonings and cuisines and get a feel for what works together. Get the flavour thesaurus, great little book.
I'm not a neurobiologist (if you are please correct/clarify!), but here are just a few of doubtless many significant differences as I understand them:
* Real neurons don't update in a bunch of coordinated discrete timesteps, as an ANN learning algorithm does. They can fire independently and in continuous time.
* Real neurons' activation behaviour is closer to sudden delta-function-like spikes, rather than a smooth activation function or something which allows them to stay in a firing state for longer than a short pulse.
* The structure of the connection graph of neurons in the brain is incredibly more complex than that of artificial neural net models, can change over time, isn't split into obvious layers from input to output, there isn't a clearly-identifiable error signal which is used to train it at the outputs, ...
* A real neuron is an incredibly complex biological system whose behaviour can be modulated in all sorts of ways and which I expect would take a nightmarishly complicated set of PDEs to model with any degree of realism even as an isolated unit. To think we've captured all aspects of their behaviour relevant to human cognition with a simple weighted sum and a sigmoid (say) seems pretty naive.
Details aside though, the idea that you can start drawing deep philosophical conclusions about the nature of human thought, our ability to conceive of our own limitations etc, based on an analogy between a complex biological system and a simple mathematical formalism which is at best loosely influenced by certain limited aspects of it -- it's just silly, one of those "not even wrong" sort of statements.
What matters is not only whether they can learn it but how much data they need to learn it to a given degree of accuracy. This is the kind of question addressed by nonparametric statistics and statistical learning theory.
Bayesian methods work by averaging over a bunch of different models / different values of the parameters.
What it means to compute a mean depends on the parameterisation in which you do it: simplest example being that an arithmetic mean is not in general the same as a geometric mean, or a harmonic mean.
There's no "neutral" / parameterisation-independent way to specify how this averaging is done, so if you care about the average case, you're going to have commit to doing it some particular favoured parameterisation. Choosing that parameterisation is equivalent to choosing the prior.
Frequentist methods avoid the need for this decision; the price they pay is that without a prior they're unable to condition on the observed data. They must consider every parameter value and its resulting sampling distribution separately and can't average over them.
It's not possible for a prior to be "flat" with respect to all re-parameterisations of a continuous parameter in a model. E.g. a flat prior for the variance isn't flat for its inverse (precision) or its square root (the std. dev.), and the choice of which of these alternative parameterisations you use to express the unknown quantity in the model is arbitrary. In the frequentist case it doesn't affect the result of the inference; in the Bayesian case it matters which of the parameterisations you choose your prior to be flat with respect to.
Bayesian analysis seeks an estimator which minimises posterior expected loss, conditioning on the data and with the expectation taken over the parameters under a particular prior.
A frequentist analysis might seek an estimator for which uniform bounds on the worst-case expected loss are available, which hold in expectation over the data, given any value of the parameters.
Both approaches fit into a decision theoretic framework and there are good reasons why you might care about frequentist properties when making decisions. I agree that this isn't only about average case vs worst case -- as you point out it's also about whether you take expectations over data given params or over the params given data, and that's important too. But I think the average case vs worst case aspect of this is an important part of what this is all about and gets to the heart of what the trade-offs are when choosing between these methods.
I disagree that the sampling distribution is "irrelevant for making decisions", that's quite an extreme view which I don't think many applied Bayesian statisticians would take. Frequentist properties are something people often validly care about when deciding on a statistical procedure to use in an experimental design context, i.e. before collecting the data -- and especially if you're choosing an estimator which you intend to use many times for many experiments, even if they're not all exact replicates of each other.
Also, worth a reminder that continuous estimation problems can be viewed as decision problems too, it's not just about discrete decision-making.
I don't think it's great to take the view that, because I'm not making a decision based on this estimate myself, I don't need to worry about which loss function I'm implicitly optimising for when choosing an estimator. Someone else may need to.
Sure, provided that the reported p-values for positive findings have been corrected appropriately for any multiple comparisons. Abstracts should summarise but not mislead...
Maybe they're non-stationary then? Doesn't sound like a failure of probability, just a failure of a particular probabilistic model.
For those interested in this sort of thing, a couple of other libraries with a similar range of uses:
https://github.com/hyperopt/hyperopt https://github.com/JasperSnoek/spearmint
Would be interested if anyone has any thoughts on how these compare (I've been quite impressed so far by hyperopt but haven't tried the others yet).
Generally to make inference for these kinds of models fast, you have to make it approximate.
MCMC has the nice property that, while it's an approximate method, it becomes exact in the limit of infinite samples. Meaning that when applying it in a general setting, the compiler doesn't have to make hard and final decisions about where and how to introduce approximations -- it produces MCMC code and you decide how long you can be bothered to wait for it to converge and how accurate you want the answer to be, based on various diagnostics.
Most other non-MCMC-based approximate inference methods (MAP, EM, Variational Bayes, EP, various hybrids of these and MCMC...) don't converge to the exact answer, they converge to an approximate answer which remains inexact no matter how long you run the algorithm for.
Different approximations have different strengths and weaknesses, and the best choice may depend on the model, the data, what you actually want to do with the posterior (a mode, a mean, an expected loss, predictive means, extreme quantiles?), and what frequentist properties you want for the results of the inference, especially given that this is no longer pure Bayesian inference but some messy approximation to it. Often the only practical way to decide will be to try a bunch of different things and see which does best on the application (or best predicts held-out data), even armed with full human intuition. A compiler doesn't really have a chance here.
In short, it's not going to be easily to automate fully because it's not something you can decide on formal grounds alone. You have to decide how and where you're willing to approximate, and you have to understand the approximations used and check the resulting approximated posterior against data in order to know whether to trust the results and how to interpret them.
I'd prefer to decouple those things though, so I can benefit from the first and the second without waiting for a "sufficiently smart compiler" to arrive which can automate the whole shebang well enough to rely on in a fully general programming language setting. That was what I was expressing skepticism about, although I'd be very happy to be proven wrong.
I think there's a lot of value (and a lot of buy-in) for probabilistic modelling languages, which impose some constraints in order to get efficient inference, which help you meet those constraints and don't provide a false promise of generality. And tonnes of value in research which looks for nice unifying formalisms to enable this sort of thing. Also in nice formalisms for inference in general, so perhaps that will be a useful side-effect of this work.
It's the idea of full-blown probabilistic programming, where you have unconstrained turing-complete non-deterministic programming language and inference just works, where I've seen a bit more in the way of healthy skepticism. Of the "this will be nice if it ever arrives, but in the meantime I'll be getting on with doing statistics, which it doesn't obviate the need for".
Similar to a computer scientist and the ultimate "sufficiently smart" compiler I suppose.
* Problems with few parameters and/or few data (I was going to say toy problems, but there are sometimes important and interesting problems of this nature)
* Problems where the generative model is so complicated that you have no hope of doing any better than this and turn to it as a last resort. A bit like in combinatorial optimisation where you just say "gah, let's throw it at a SAT solver!".
(Perhaps that's not a bad analogy actually. If they can get to the point where SAT solvers are now, that would actually not be a bad proposition.)
* In particular, problems where the generative model is complicated but the complicated part of it is largely deterministic -- perhaps some kind of non-linear inverse problem where there's some simple additive observation noise tacked onto the end, for example.
What I do fear about, is the suggestion that people can just start building fiendishly complicated hierarchical Bayesian models using these things and get valid, useful, robust, interpretable inferences from them without much in the way of statistical training. I suspect even a lot of statisticians would be a bit scared of this sort of thing. Make sure you really read up on things like model checking and sensitivity analysis, that you know something about the trade-offs of different model structures and priors etc. And that's before you start to worry about the frequentist properties and failure cases of any approximate inference procedure which is magically derived for you.
Statisticians tend to favour simpler parsimonious models, not only for computational convenience but because it's easier to reason about them, understand and check their assumptions, understand their failure cases and so on.
I wish these guys lots of luck though, it is a really interesting area and the computer scientist in me really wants them to succeed!
Very true. Furthermore, if you start to think seriously about making differential privacy guarantees across a whole company, it gets really hard and impractical.
It's not enough, as you might naively hope, to certify individual data analysis jobs separately as being "differentially private". Taken cumulatively they can amount to something which isn't [1]
It's not even sufficient to certify whole planned and delimited programmes of use for whole datasets as differentially private, if the company also holds (or wants to hold in future) other data on some of the same users and there's some chance of new actions being taken which directly or indirectly depend on both sources of data.
For the theory to truly apply, one needs to track (in a particular formal sense) how much "privacy budget" is used up by pretty much every user-data-dependent action taken across a company and across all datasets you hold pertaining to any overlapping set of users. Once your preallocated budget is used up it's pretty much game over in terms of taking any further actions based on any data (acquired now or in future) on any of those users, unless these actions can be based on inferences which were already obtained within the original budget.
These aren't necessarily insurmountable problems, and I'm sure there are new tools being developed to help which I'm not up to date with. (One which I am aware of is Microsoft's PINQ project [2]).
Still, it seems the organisations with the biggest chance of actually making this work, are those whose relationships with sets of users are inherently transitory and/or firewalled off from eachother. For example a B2B company who operate one-off surveys for clients, throwing away the raw data afterwards.
The fundamental problem is that differential privacy guarantees a degree of resistance to an incredibly strong adversary -- one with unlimited prior knowledge about your users, who is able to bring unlimited intelligence and computational resources to bear on drawing inferences about them based on every action you ever take conditional on user data. It's impressive that one can obtain any guarantees at all under this model, but perhaps not surprising that it can be hard to scale up and compose the guarantees without things blowing up.
That's not to say that there's isn't value in trying to obtain differential privacy guarantees for smaller scale pieces of work. It's still considerably better than other more naive pseudo-anonymisation methods.
[1] at least not to the same strength -- see http://en.wikipedia.org/wiki/Differential_privacy#Composabil... for the less handwavey version [2] http://research.microsoft.com/en-us/projects/PINQ/
As an example application, when sampling from the posterior of a Bayesian model it can help to take this natural geometry of the parameter space into account, e.g. via Riemannian Manifold Hamiltonian Monte Carlo [1]
Then again, from what I've seen most Manifold Learning only uses the word manifold in a loose handwavey sense to motivate what they're doing -- the added abstraction level of differential geometry doesn't always add very much you're just interested in learning smooth functions from R^m -> R^n.
[1] http://www.dcs.gla.ac.uk/publications/PAPERS/9149/RMHMC_MG_B...
True, although I suspect there's probably a certain amount of historical accident to the set of functions included in math.h. There's a strong argument that some of these common 'special' functions are at least as fundamental to a variety of mathematical fields as (say) arc hyperbolic cosine, although admittedly it'd probably be hard to get clear consensus on exact which.
I expect if you asked a variety of stats/ML geeks they might have other priorities to those given here -- e.g. I might request the digamma and trigamma functions (first and second derivatives of log gamma, useful for computing gradients in various Bayesian models) before asking for Bessel functions, say. It might also be sensible to prefer log-domain versions of some of the functions he suggests, since much practical computation with probabilities needs to happen in the log domain.
The multivariate Gaussian distribution is a great example of this. It's probably the most fundamental and important distribution in statistics, and working with these distributions is pretty much pure linear algebra -- quadratic forms over vectors of parameters, eigendecompositions of covariance matrices etc.
Even for non-statistically-motivated data mining: any time you're optimising over a lot of parameters, it's likely that linear algebra (and, as before, multivariate calculus) will help. Linear algebra is as important to calculus over multiple variables, as plain old high-scool algebra is to plain old univariate calculus.
Perhaps the useful information is more in the correlations between different related metrics, than in drawing up ranked lists (surprise surprise, Java > Haskell!). This graph helps visualise the correlation between these two, which seems significant but far from perfect. Outliers like SQL can then be identified, which point to problems with the metrics (e.g. with SQL, presumably github fails to spot lines of SQL embedded in other langauges).
And of course basing the github measure on lines of code gives quite an unfair advantage to verbose languages like Java.
Perhaps they could look at compression ratios when (say) gzipping a decent sample of each language, and use these to weight the LOC metric?
Perhaps even just the same commits replicated in lots of people's forks of (say) scipy -- do they control for this? what about rebases of the same commit?
"For all natural numbers x and y, if 3x = 3y, then x = y"
An equally trivial but rather more abstract way to phrase this is in the category of finite sets:
"For all finite sets x and y, whenever there's a bijection between the product x cross 3 and the product y cross 3, then there's a bijection between x and y". (Here "3" represents any set of cardinality 3, but typically chosen to be the set {0,1,2} where 0 = {}, 1 = {0}, 2 = {0,1}).
What's (apparently!) not so trivial, is when you remove the "finite" restriction from the above, and you want to show it holds for all sets, including those of various infinite cardinalities. Actually constructing the bijection seems like it would require an infinite number of arbitrary choices, which is why it's impressive they don't rely on the axiom of choice.
There's something you don't know (the parameters). Are you looking to make statements which bound how bad things could be under the worst-case setting of those parameters? Or do you have some idea upfront about how likely different parameter settings are, and want to make statements about them in the "average" case?
Rather like with worst-case vs average-case analysis of algorithms, which is more appropriate depends what you're trying to do, and sometimes both are interesting.
Anyway a few "next thing to try" suggestions from a machine learning perspective:
The model selection process used here is by its own admission quite ad-hoc, based on a gut feel about diminishing returns. There are various more principled methods you can use to find the sweet spot between over- and under-fitting with these kind of models, a lot of them based on held-out validation data.
One way to do this would be leave-one-out cross validation (LOO-CV): hold out one whisky, fit the model, and see how 'surprised' the model is by the held-out whisky, repeat for the next whisky and average over all the folds. Because the dataset is tiny this should be quite feasible.
To measure 'surprisal' you could e.g. look at the distance from the held-out data point to the nearest cluster, although something better motivated would be if you switched to a probabilistic model and used likelihood of the held-out data. Probably the simplest next thing you could try in that direction would be a Gaussian mixture model (GMM) trained using EM. K-means is actually a degenerate limiting case of this.
A probabilistic model would also allow you to use Bayesian model selection criteria, which can get quite interesting (and might lead you eventually to things like Dirichlet process mixture models).
It would make it easier to compare the model's explanatory power with other unsupervised probabilistic models. For example some kind of latent factor model like Factor analysis or pPCA would be quite interesting to investigate too, whether taken alone or in combination with clustering as a dimensionality reduction step as tlarkworthy is suggesting.
Also concur that doing multiple runs with different randomised initialization is generally a good idea for k-means or EM, since they can get stuck in poor local minima. Perhaps more common practise to pick the best of multiple runs than to average them though.
Model averaging can be quite useful when you're averaging over versions of the same model with different hyperparameters, e.g. the number of clusters in a mixture model.
You still need a good hyper-prior over the hyperparameters to avoid overfitting in these cases though, as an example IIRC dirichlet process mixture models can often overfit the number of clusters.
Agreed that model averaging could be harder to justify as a scientist comparing models which are qualitatively quite different.
I quite like "Bayesian reasoning and machine learning" too: http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/090310.pdf