pork

Thanks for this. The more I look, the more it seems like the answer really is "cool, so go make it". :)

Fantastic, thanks so much! It looks like it's almost there, and definitely something I will check out in more detail. Admittedly, my use case is limited, but as services like Mailgun and Sendgrid are demonstrating -- it is not an insignificant use case. Lamson looks like it's trying to achieve a bit more than that single case.

So a couple of things I noticed that might be missing from Lamson:

(1) Spam blocking auto-updates -- does this tap into Spamhaus, etc.? (2) UTF-8 transcoding? (3) Signature, quote stripping (like Mailgun). (4) DKIM signing? (5) Simple client libraries for sending mail (i.e., I don't want to build a "Lamson application", I just want to talk to a Lamson server on a privileged port using JSON).

Of course, all these concerns might have answers that I'm missing.

A bit more about my (fairly common, I'm guessing) use case: you have a VPS and a MongoDB instance. I want my MTA to take a minimal set of config parameters, say:

-- MongoDB database name -- domain -> MongoDB table mapping -- privileged API sending port

Now I want the MTA to receive mail for me, perform spam blocking, transcode to UTF-8, parse out signature and quotes in a separate field, and dump the whole object to a MongoDB document with the index field = the recipient's email address. I want it to auto-update it's spam blocking rules using whatever external services and internal analysis necessary. I want it to periodically dump some usage statistics into a separate MongoDB table (maybe a circular connection).

Now for sending mail, all I should have to do is connect to the privileged port from a process on the same machine (it's firewalled to the outside world), and submit an HTTP POST request that specifies recipients, message body, attachments, etc. The MTA should accept the request, perform whatever queuing, rate-limiting, and retrying is necessary to send the message.

All failures and diagnostic information are dumped to separate MongoDB tables. The applications deal with the database/storage system directly, to keep the MTA simple.

I'm glossing over a lot, naturally, but I hope that helps.

Minor nit: the distribution of Republican, Democrat, and Independent in the population at large isn't 1/3rd each, so your sample is a bit off from representative.

While we all love services like Postmark and Mailgun on HN (and they are great services), does anyone know of any easy-to-setup open source projects that offer similar functionality? Traditional MTAs are a pain to setup and outdated to boot for webapps (mail parsing, signature and quote removal, UTF-8 transcoding, automatically updating spam detection, linkage to arbitrary storage handlers like MongoDB, and an HTTP+SSL JSON API should be a minimum).

Don't waste your time on space partitioning trees if you have big data, jump straight to Locality Sensitive Hashing or approximate nearest neighbor graph construction (WWW 2011)

Redis now uses the same journaling for durability that Mongo does. What you get with Redis is complex atomic ops and transactions, but at the cost of not having sharding.

...in UTF8.

Hey, while we're on USPS feature requests, this one is a bit overdue: SPAM FILTERING.

Allow me to blacklist certain senders, that's all.

Now see, your comment rubs me the wrong way. After I got my first job, I splurged on a used 911 Turbo. It got terrible mileage, but made me smile every single day I drove it (which wasn't everyday).

Attitudes to money are varied, and the accumulation of wealth, like squirrel nuts, is not everyone's goal.

Modern FPS games have detailed stories, considerable original acting and dialog (motion and voice capture), and incredibly detailed artwork. So your analogy doesn't really hold.

The problem you mention is solved by evented sockets.

I agree with the sentiment, but in this case, the police raided a private business with probable cause, and will hold all assets of that business as possible evidence in a criminal investigation. If your car is found full of drugs, they will also impound it and inconvenience the other possibly innocent passengers. If they find a body in a storage facility, they will probably deny everyone else access to the facility until they have gathered all the evidence they need.

This will most likely continue to happen in the form of tighter integrations, continually boosting the engagement number, until an antitrust lawsuit causes G+ to split off into it's own company. I suppose the bet is that by then, there will be enough momentum for network effects to take care of the rest. Like Google+ or not, it's a pretty nifty, calculated strategy.

What's wrong with long polling and keep-alive connections, handled with evented socket code on the server side? Is it mainly because of the HTTP request overhead?

Not at all. He feels that he has the right, as a lawful paying customer, not to be threatened and preached to. Imagine if you sat down at a restaurant, and the server lectured you about dining-and-dashing before serving you. It's his right to be pissed off about sanctimonious bullshit from the MPAA

In response to your suggestion on data being made available, there are often issues of proprietary datasets that cannot be made public (for legal or ownership reasons). It is a frustrating situation, but one that is not likely to change anytime soon, so I would be happy with just the open-access publication requirement.

I'm 45.

A bit off, but CPAN really hasn't changed all that much. I tried installing a module the other day, something simple like a word stemmer, and got so disgusted that I quit Perl.

> It's a bit disingenuous to equate some Google employees from doing this with Google as a company doing it.

Um...no, it's not disingenuous at all.

As I said, I appreciate the initiative. At the very least, consider posting benchmarks comparing your scheme to the output of gzip.

I respect the initiative you're showing, but I'm a little skeptical of your choice of presenting your own eponymous compression algorithm as part of a portfolio, especially without any theoretical guarantees or rigorous experimental data on its performance. It gives the impression that you're either not aware of the rich history of data compression algorithms, and the heavy burden of proof on any new algorithm, or you like re-inventing the wheel. Please take this as constructive feedback.

What really stood out for me is the realization that children are imprisoned at Guantanamo. And we're not talking infants, but 13+ kids.

If they're anything like Google, multiple employees have probably already read this and a few have probably already started the escalation process.

> "only one theoretically-exploitable memory corruption vulnerability in the release history",

I presume you mean qmail [1]

[1] http://cr.yp.to/qmail/guarantee.html

Thanks for clarifying; I find this much more insightful.

I read your comment several times, but I still don't understand why you think Hadoop is the key technology for data interchange between organizations. I don't mean to be harsh, but your comment is a bit like buzzword soup (hadoop, etl, cloud, bring the computation to the data).

> [Hadoop] provides the integration point for both slurping the data out of internal databases, and transforming it into consumable form

Hadoop does no such thing. It doesn't "slurp data out of internal databases". It's just a DFS coupled with a MapReduce implementation. Perhaps you're thinking of Hive?

> Currently there are no solutions for transferring data between different organizations’ hadoop installations.

All data isn't "big data". By being myopically hadoop-focused, you're ignoring the real problem, which is data interchange. XML was supposed to be the golden standard; it's debatable how far it's achieved its initial goal.

> So some publishing technology that would connect hadoop’s HDFS to the .data domain

So basically, forsake all internal business logic, access control, and just pipe your database to the net? When you have a hammer...

> Transferring terabytes of data is non-trivial. But if the data is published to a cloud provider, others can access it without having to create their own copy, and it can be computed upon within the high-speed internal network of the provider

See AWS public datasets for exactly this, but it's still a long shot. It also ignores the problem of data freshness (i.e., once a provider uploads a dataset, they also need to keep updating it). http://aws.amazon.com/publicdatasets/

Numb3rs, to me, is a parody of math. You can argue whether it's a disservice to math or not when it's presented as a magical tool obfuscated by senseless jargon and technical term soup, which is what I found Numb3rs to be (and I'm a mathematician). It's sort of like the "zoom in on B3 and enhance" of math.

Thomas, this is totally irrelevant, but would you consider yourself one of those "older" developers? I'm intimately familiar with your comments, but I have no idea how old you are.

No, that too, but the downloading makes the hammer come down heavier.

Try not to be so literal. My implication was that any service that lets a user download copyrighted material for free is bound to be terminated by the labels, and that is largely what is happening.