codeslinger

Spammers have employed this technique since roughly about the time they started sending images. See here for some examples:

http://www.jgc.org/tsc.html

Also, real OCR is too expensive to do at the volumes of mail being sent today. Gmail is likely using an OCR-esque technique that has lower precision but is much faster to process a given image.

"It can be really stressful especially if there is a bug in the program. That pushes us back by almost a week."

I'd love to only ever have these kinds of bugs.

Ok, Facebook is not worth $15 billion. Its not even worth $500 million. I want this meme to end now.

Its called "challenge-response" or C/R, for short. Its been around almost as long as email has and its not new or innovative. Nick would have had some trouble productizing that even if he wanted to, in that its a terrible "solution" that almost nobody wants.

I'm totally with sbh here. This title is ridiculous and overblown. What he was dealing with were spammers, not marketers.

They don't appear to be relevant to Google because they don't help Google better determine ways to leverage AdWords and AdSense. GA's JavaScript snippet in fact returns to Google more and more pertinent information than is resident in the average web server log file. Plus it removes the spiders and non-page view data just by being a JavaScript-based tracking solution so that's less work and cleaner data for them (and anyone else who does this e.g. all of the other Web analytics vendors).

I think it would be less bandwidth. Considering the overhead of all the individual HTTP headers both in request and response, you could reduce that significantly by batching up log files and uploading them at regular intervals, perhaps once a day? GA seems to perform batch updates at that interval in any case so it might not even put a big cramp in their style from that perspective.

However, there are a bunch more issues with this. To allow this, they'd have to be able to parse all of the most popular Web server log formats. This, in and of itself, is not that hard until you account for people customizing their logging format via measures like the LogFormat directive in Apache. As well, customizing the log format is a virtual necessity, as GA requires more information than is in the standard log formats of most web servers. Another issue is that the use of a JavaScript snippet on the page prevents non-browser user agents from appearing, clearing GA's logs of spiders and other such cruft that detracts from determining human user behavior (what they are apparently most interested in for the purposes of optimizing AdWords and AdSense).

But the biggest problem for them, I think, is that with this method they are no longer able to set first-party cookies on your user's browsers and thus (more) reliably track individual (read: human) users. So, in effect, they would waste time with this, in that they'd still have to make you put a JS snippet on the page to set cookies that you'd then have to modify your web server logging format to include in every hit. Urchin worked like this back when it was still alive and you could run it on your own servers. So, all in all, it seems like something Google would just be wasting their time in doing given what they want to get out of your data.

I think both Werner Vogels and the paper itself were pretty clear that this was NOT the basis for S3. This line from section 1 is pretty telling even had they not spoken up on this:

"To meet the reliability and scaling needs, Amazon has developed a number of storage technologies, of which the Amazon Simple Storage Service (also available outside of Amazon and known as Amazon S3), is probably the best known. This paper presents the design and implementation of Dynamo, another highly available and scalable distributed data store built for Amazon's platform."

More supporting evidence from section 2.1:

"Dynamo targets applications that need to store objects that are (usually less than 1 MB)."

S3 allows individual objects up to 5GB in size. This in and of itself does not rule out Dynamo being the underlying store for S3 but does certainly lend weight to what Werner and the paper's authors have claimed regarding S3's origin.

KFS won't really be interesting until they do a MapReduce-clone. You need something like that in order to be able to do anything with the data stored.

It sounds to me like the closest thing to what you want is a library for Object Prevalence <http://en.wikipedia.org/wiki/Object_prevalence>. There is one that I know of for Python called Pypersyst <http://pypersyst.org/>. There is also an IBM DeveloperWorks article about this library by one of the authors, as well <http://www.ibm.com/developerworks/library/l-pypers.html>. Hope that helps :)

I tried to get it from Amazon again. Lets see if they can handle it this time ;-) I'm not going to buy it from Oreilly b/c I get free (amortized) shipping from Amazon and its $13 cheaper.

Its not out yet. I bought it on Amazon and then they told me a couple days later that it would take until November to actually receive it. I cancelled the order.

+1. All of Google's AJAX apps start blowing up the VSZ/RSS of the process when I leave them open for even a medium amount of time (i.e. longer than 15 minutes). This is using Firefox on Mac OS X. Most of the Google AJAX apps are not supported in Safari as yet, so I can't really compare and contrast ;-) I can't say that FF is great on RAM to begin with, however, but this could be b/c I always have at least 3 tabs going, sometimes up to 30.

epi0Bauqu and Riley, I'm in Plymouth Meeting, PA. Hit me at codeslinger _AT_ gmail _DOT_ com to rap about stuff.

You should not use any, if you can get away with it. Start with Hadoop (http://lucene.apache.org/hadoop/) and see if you can wrap your head around that first before tying yourself to the RDBMS lifestyle.

That article oscillates between being tautological and stereotyping. How the hell did it make it on this forum?

Yes.

There is AllegroCache for storing objects, which is supposed to be nice (though, I've never used it):

http://www.franz.com/products/allegrocache/

I don't know how much it costs, either.