How a HyperLogLog works
opensourceconnections.com
opensourceconnections.com
"The basic idea of HyperLogLog (and its predecessors PCSA, LogLog, and others) is to apply a good hash function to each value observed in the stream and record the longest run of zeros seen as a prefix of any hashed value. If the hash function is good, the bits in any hashed value should be close to statistically independent, so seeing a value that starts with exactly X zeros should happen with probability close to 2 -(X + 1). So, if you've seen a run of 5 zeros in one of your hash values, you're likely to have around 2 6 = 64 values in the underlying set. The actual implementation and analysis are much more advanced than this, but that's the idea."
Of course that's just an artifact of what's really happening, which is pseudo-random values + laws of probability.
And the problem posed was not to count the number of visitors, but to count the number of different people who visited... the fancy word for that is the "cardinality of unique visitors" but "number of different people" is just as accurate.
Counting the number of people could NOT be done with the same amount of space. This example required two simple counters on the blackboard... call it 30 bits of memory, assuming you didn't expect either counter to get more than 8. That's just about exactly enough space to store ONE social security number... and nowhere near enough to count the unique people.
They may not specifically use the word "distinct" (opting for phrases like "unique visitor" instead), but they certainly make the concept clear.
:)
My concerns are not really about the contrived usage of only 2 buckets and decimal numbers. This may give a picture to start with.
But, this is a pity to elude how the final cardinal estimate is computed. And this is shame to publish a gist so sloppy : all key points are buggy (bucket assignation, count of leading zeros, final outcome).
I started my day with "Oh fine, a post witch may help me to understand quickly how hyperloglog works" but I was so confused after that reading, that I forced me to dive into the paper and a high quality code (redis implementation) to leave with a better understanding and no more false assumptions.
This may be ok to simplify an algorithm to help people to grasp it. But at least, as an author, we have to give a true insight.
[0]: http://blog.notdot.net/2012/09/Dam-Cool-Algorithms-Cardinali...
The short, simplified story is a list of hlls (a 'counterstack' in the paper, which is an unfortunate bit of historically motivated terminology) are used to count the number of distinct blocks requested. Each hll is fed every block request. A new hll is created after a certain number of requests, so the counts your hlls report correspond to estimations of the number of unique blocks accessed in tails of the trace. The oldest hll has seen every block in the trace, while the most recently created has seen a tail of the trace of length <= your creation interval parameter.
It turns out that you can use this structure to compute stack distances, which are the number of unique blocks accessed between two requests to the same block. The details are a little hairy, so see the paper for the full story.