This is actually fairly trivial to do right.
This is actually fairly trivial to do right.
2. We also wanted to provide real-time stats for pages, sites, and hosts. This requires IDing the user and page in a reversible way. Even if it's not reversible, you could easily use a rainbow table to reverse, especially with the limited IPv4 namespace.
However, it seems that such URL collection isn't too big of a concern among the general population.
You also have to maintain plugins or extensions across all the major browsers, which is a pain.
This was a project I was working on before completing a CS degree and/or getting exposure to some of the solutions that could be used. I'd be interested in exploring it again, but I have a few more interesting projects on the stack for now.
Additionally, there are many other large databases (Hadoop, etc) that will scale linearly. While it may not be ideal for this use case, but a proper multi-master setup with many read slaves in either PostgreSQL or MySQL could be described as being able to scale linearly to some unspecified extremely large upper limit (see work done previously at Yahoo, Google, Amazon, et al). Much of this also depends on what you define as scaling linearly (are you looking for linear scaling for memory, reads, writes, storage, latency, costs, etc?).
Real time stats for pages, sites, hosts for a shared annotation system across browsers doesn't mesh well with your initial condition of "privacy concerns."
[0] http://techblog.netflix.com/2011/11/benchmarking-cassandra-s...
The "privacy concerns" weren't necessarily my own, I just anticipated that others might have them and therefore not adopt the system. For me, the point was to share what sites you were visiting because you'd be annotating (well, chat/message-boarding) those sites with others. Additionally, I thought it would be cool to see who was where at a given time, what sites were popular, or which pages people found particularly interesting (interesting enough to visit anyway) at any given time.
partition_id = well_distributed_hash(normalize_domain_name(<domain>))
really don't think this would be an issue, or even a concern, as 1 billion writes per second are "Google-levels" of writes. One would likely run out of financial resources supporting an expansion of 86 terrabytes / day (assuming 1kb minimum new data per write), far faster than you would ever max out the capabilities of the technology.
It's more a finance/business exercise to determine if revenue inflows would actually meet, cover, or exceed revenue outflows on a project like this. You can always make the technology scale and meet the demand.