In summary: interesting idea but I'm not sure it applies outside a very narrow domain.
The beta distribution is uniquely determined by two parameters called alpha and beta. Typically, these represent the sum of observations we've seen of some events. (On the same website is an example with heads and tails from coin flips.) The beta distribution forms a monoid simply by adding these parameters from each distribution. This is very similar to the Categorical distribution, and also generalizes to the Dirichlet distribution.
Things get really interesting when you stop talking about distributions, and start using them in the context of actual machine learning algorithms. If you know how the Naive Bayes algorithm works, for example, it should be plausible that it forms a monoid in the same way that the Gaussian forms a monoid. The future posts will cover how to apply this to some of the "important" learning algorithms like this.
suppose we have:
x ~ N(0,1)
y|x ~ N(x, 1)
y ~ N(0, 2)
however, i believe gaussians are not the only distribution with this property either: i think this property corresponds to the stable family of distributions: https://en.wikipedia.org/wiki/Stable_distributions
Gaussian distributions belong to the class of stable distributions, though, because of another of their properties: independent Gaussians, when added, are again Gaussian.
infinite divisibility is (yet another) property of gaussians!
Closure under marginalization is something else.
It so happens that the functional form of the gaussian satisfies both, but the two properties are not at all the same.
P1: X, Y gaussian => Z = X+Y gaussian
P2: X, Y gaussian => X | Y gaussian