Edit: It was Office 2000 that enabled this, called "Web Discussions". You had a "Discussion" button in IE[1]. Here's a paper talking about how useful it was: http://research.microsoft.com/en-us/um/redmond/groups/coet/a...
It used the discussion feature of Microsoft Office Server aka SharePoint Portal Server. The history of SharePoint is very complex but interesting:
https://web.archive.org/web/20120419073951/http://blogs.msdn... , https://web.archive.org/web/20080303180618/http://blogs.msdn... , https://web.archive.org/web/20120409232321/http://www.joinin..., http://en.wikipedia.org/wiki/SharePoint , https://web.archive.org/web/20120519235139/http://www.seocon... , http://en.wikipedia.org/wiki/Microsoft_FrontPage
I remember an SP2003 installation with thousands of sub portal sites, hundreds of GB (blobs) stored in SQL server. What I really don't understand is the almost non-existing upgrade paths (between versions '01, '03, '07, '10, '13) and the URL-hell (IIS URL alias would solve that).
Instead of Cairo-OS, WinFS, an improved Explorer shell & improved file server, nowadays SharePoint fulfill various enterprise needs. http://en.wikipedia.org/wiki/Cairo_(operating_system) , http://en.wikipedia.org/wiki/WinFS
It also sounds like the kind of thing some would want to use to overlay their own advertisements onto other people's websites...
I also didn't know how to scale the solution up to allow it to work on every web page for every users.
Also, I anticipated that there would be lots of privacy concerns as you'd basically be collecting every web page a user went to.
This is actually fairly trivial to do right.
2. We also wanted to provide real-time stats for pages, sites, and hosts. This requires IDing the user and page in a reversible way. Even if it's not reversible, you could easily use a rainbow table to reverse, especially with the limited IPv4 namespace.
However, it seems that such URL collection isn't too big of a concern among the general population.
You also have to maintain plugins or extensions across all the major browsers, which is a pain.
This was a project I was working on before completing a CS degree and/or getting exposure to some of the solutions that could be used. I'd be interested in exploring it again, but I have a few more interesting projects on the stack for now.
Additionally, there are many other large databases (Hadoop, etc) that will scale linearly. While it may not be ideal for this use case, but a proper multi-master setup with many read slaves in either PostgreSQL or MySQL could be described as being able to scale linearly to some unspecified extremely large upper limit (see work done previously at Yahoo, Google, Amazon, et al). Much of this also depends on what you define as scaling linearly (are you looking for linear scaling for memory, reads, writes, storage, latency, costs, etc?).
Real time stats for pages, sites, hosts for a shared annotation system across browsers doesn't mesh well with your initial condition of "privacy concerns."
[0] http://techblog.netflix.com/2011/11/benchmarking-cassandra-s...
The "privacy concerns" weren't necessarily my own, I just anticipated that others might have them and therefore not adopt the system. For me, the point was to share what sites you were visiting because you'd be annotating (well, chat/message-boarding) those sites with others. Additionally, I thought it would be cool to see who was where at a given time, what sites were popular, or which pages people found particularly interesting (interesting enough to visit anyway) at any given time.
partition_id = well_distributed_hash(normalize_domain_name(<domain>))
really don't think this would be an issue, or even a concern, as 1 billion writes per second are "Google-levels" of writes. One would likely run out of financial resources supporting an expansion of 86 terrabytes / day (assuming 1kb minimum new data per write), far faster than you would ever max out the capabilities of the technology.
It's more a finance/business exercise to determine if revenue inflows would actually meet, cover, or exceed revenue outflows on a project like this. You can always make the technology scale and meet the demand.
hereby super limited in reality.
and if it's people then you just invented comments. which are toxic most of the time. also invented (rap)genius. congrats, millions of funding for you.