We just open sourced Flanker, our Python email address and Mime parsing library | Hacker News Reader

We just open sourced Flanker, our Python email address and Mime parsing library

blog.mailgun.com

160 pointstwakefield26 comments

Note that MIME is also used for multipart uploads. I'm currently still using the builtin-parser.

Looking at a 100mb multipart upload I can see that email module takes 3.7s and allocates 265 extra MB to parse it, while flanker.mime takes 0.22s and allocates 0.2mb extra memory.

I would love confirmation on this statement.

I'm a corelibs nazi and refuse to add dependencies to the very last when I'm writing Python.

But if what you're saying is true then I can see a very good argument for replacing email with flanker.

There seem to be a bunch of people who have the same opinion -- stick to the corelibs until it hurts. Do you want to articulate why?

Less dependencies make for happy users.

Python has already simplified programming so much coming from languages like C, so I guess I view it as a tolerance matter.

I'm willing to put up with the longer way of using corelibs in some cases, to keep my dependencies low. And I'm willing to tolerate this longer way because I feel that Python in it self is such a shortcut from other languages that I'm already reaping huge benefits.

I know there's usually a huge community testing and supporting all dependencies but having installed some Ruby and Node apps I've seen massive lists of dependencies and I've had situations where it breaks down on some obscure library used to do some really mundane task.

I could not have solved many of those cases without the years of Linux experience I already have. I don't think users should have to go through that.

I'm working on a new email platform,[1] and we've been testing Flanker quite a bit. It's awesome!

Dealing with MIME has always been an issue in Python, and the last move was Zed Shaw's Lamson project a few years ago. The native email.parser module is crazy slow once you start doing heavy processing.

Flanker's addresslib also is a no-brainer to use. These are the sorts of things which should be bundled with the Python email module.

Thanks for open sourcing them!

[1] This isn't announced yet, and will deserve a much longer post. You can sign-up for news here: https://www.inboxapp.com.

> These are the sorts of things which should be bundled with the Python email module.

But only after they don't need to be changed any more...

Agreed.

Hi All!

Mailgunners will be hanging around in comments, ready to answer any questions, should you have any.

This looks really useful - thanks very much for releasing it!

Thanks, this looks great!

I am in two minds - eight years ago and more we were all desperate for a good mail handling library. Now, I hardly seem to have a use for automated mailings that is not handled by Mailchimp (sorry!) or I just want to fire off HTTP calls to some API to mail custom mailings for me (which is I think what you guys do)

In short email is probably more important to my business (even with my paltry list numbers) but I no longer run a mail server, anywhere, I do not even have a local mail client unless you count JavaScript.

I'm wondering if I am trading off vulnerability for ease at the wrong places.

... hmmm a little unfocused I am afraid but I really have not made any explicit strategic decisions over email but even so, my whole mail handling has altered.

I just am surprised I have not noticed how big a step it has been.

By making email processing easier, we hope that we can help others to innovate in the email space, not waste time on the basic processing problems we've solved a while ago

I think my point is that its still companies / providers "innovating", not individuals, because many indviduals have given up running their own mail.

So, its a good thing, yes, but I think the unit of abstraction is not the individual developer but the "organisation". WHich is oddly different from most OSS

Github repo for the lazies: https://github.com/mailgun/flanker

BTW, it took us a while to make sure we deploy right from pip on production, that means that community will get all the latest hotfixes and the github repo won't be stale as it's not just the mirror or one time copy.

Jeffrey Steadfast discusses the edge cases of MIME parsing and performance optimizations in his MIT/X11 C# library here: http://jeffreystedfast.blogspot.com/2013/10/optimization-tip...

I appreciate how the Flanker release includes benchmarks without thrashing on alternative implementations.

Wonderful, thank you for releasing this! I hope you guys continue to release good bits for handling email, it really is a bear. The ongoing joke in Zapier is a small library that I made called simple_imap. Well, as the story goes, it quickly became anything but simple...

Where is the python-requests for email? I will seriously buy someone as much beer they can consume in an evening if such a thing exists!

As an aside, I use MailGun with the Linux nullmailer package to send notification emails from my home (and every other, really) server.

It's a solid service, very easy to configure, with generous free limits (for the five emails a month my server sends) and with a very nice UI. Props to the team.

Thanks! Makes our day to hear (I'm a mailgunner).

Well, you guys deserve it. I've had so many hassles sending email that nowadays I just use you to do it. The centralized, one-page instructions on setting up are especially helpful. "Do these things and your emails won't bounce or get sent to spam".

This is great guys. It's surprising that you're able to get an 8x-20x speedup over email.parser with another pure python parser. (What does this say about email.parser, eh?)

Did you consider using a C extension and tools like lex or ragel for the rfc822 scanning phase? Not worth it?

I think it says that email.parser reads the whole message then parses. Flanker can just read the headers and ignore the message, so I guess with the right configuration and test data you get the 20x speed up.

It's pretty unlikely that many well-used and years old OSS language libraries have orders of magnitude speed ups lying on the table, without there being a design decision in there that can be argued many different ways.

But Flanker's full-message parser is still 8x faster than email.parser's full-message parser, according to their benchmark.

It's pretty unlikely that many well-used and years old OSS language libraries have orders of magnitude speed ups lying on the table...

Normally I'd agree with you, but this is the python stdlib we're talking about.

I missed that. But 20 down to 8 is still a headline vs a benchmark, and benchmarks can be ... swayed based on who is benchmarking.

But I think my point elsewhere stands - I hardly want to parse email these days, even though I consume more of it. Most of my mail is outsourced and it seems to be a trend.

Maybe not a good one. I am not sure. I used to hate spamassassin though

I've been using Guardpost in production for the past few months, and it works really well. I'm curious to see the internals. Thanks for this!

Great work mailgunners!

Where does the project name (Flanker) come from?

Reply on news.ycombinator.com