Comment by dmbaggett | Hacker News Reader

Comment by dmbaggett | Hacker News Reader

Parent Full thread

There are at least 75 nontrivial problems, each of which is worth of its own startup-amount-of-effort. A few random examples:

- sanitizing HTML email: you must eliminate XSS while preserving the appearance; this requires real HTML and CSS parsing; we do this on the end user's phone at MB/sec rates, which is not easy

- conversation threading: any email in a conversation that's been sent with Outlook will have stripped the standard threading headers and added the proprietary Thread-Index header, so in practice you must rely on heuristics

- properly mapping the IMAP/Exchange/POP "database" onto UI views is immensely complex in a real client, and requires exactly the right set of abstractions (which are very subtle)

- it is a vast understatement to stay that the IMAP protocol is a mess; it is also plagued by some really bad design choices (such as relying on message sequence numbers which vary by connection rather than exclusively using UIDs, which don't).

- instant full text search over a million message inbox is hard, but this is a real-world use case

- converting arbitrary MIME emails to HTML for rendering purposes is a huge rathole

- making offline mode really work (supporting search and modification of cached mails) is complicated, and most clients punt on it (because they are implemented server-side and therefore require a network connection for anything to work)

- implementing everything server side is (relatively) easier, but requires you to scale bandwidth per user and mailbox, which is expensive; on the other hand, doing everything client-side like Inky does is much harder because you've got tremendous variation among hardware platforms and operating systems, and it's a really heavyweight app

In general, an email client subsumes many other highly nontrivial subproblems, somewhat like a browser does. Just doing the security-related stuff alone properly -- TLS, certs, revocation, encrypted email -- is really labor-intensive.

You've managed to miss a few of the biggies I'd count:

- the charset parameter is nowhere near as correct as you'd like it to be - many RFCs are not followed in practice. In particular, RFC 2047 is literally more often incorrectly implemented than correctly implemented.

I also once recall trying an admittedly weird message (a message whose body is of type message/rfc822) on 4 different IMAP server implementations and getting 4 different answers for part numbering.

I'd say building an email client is harder than building a web browser because at least the people who write the specifications for the latter have been trying to thoroughly document all the idiosyncrasies and how to handle them.

Also, screw CFWS.

Oh God, IMAP part numbering. I'd blocked that out until now. And regarding RFC 2047, I still remember when I first learned about RFC 2231. It really reads like an April Fools joke. Actual text from this RFC:

Character set and language information may be combined with the parameter continuation mechanism. For example:

   Content-Type: application/x-stuff
    title*0*=us-ascii'en'This%20is%20even%20more%20
    title*1*=%2A%2A%2Afun%2A%2A%2A%20
    title*2="isn't it!"

You (well, the spec writer) forgot the semicolons (it's errata'd). :-)

And now I'm stuck trying to remember off the top of my head whether or not the first few lines are permitted to use quoted-strings in the first place.

With regards to RFC 2047, the worst header I've seen to date is this: Re: [Kitchen Nightmares] Meow! Gordon Ramsay Is =?ISO-8859-1?B?UEgR lqZ VuIEhlYWQgVH rbGeOIFNob BJc RP2JzZXNzZW?= With My =?ISO-8859-1?B?SHVzYmFuZ JzX0JhbGxzL JfU2F5c19BbXiScw==?= Baking Company Owner

I've been unable to find any possible decoding of those encoded-words that make sense.

... I can't wait until everyone decides to just use UTF-8 for all the headers. Except then I get to yell at everyone who uses Windows-1252 headers right now. >:(

I think sanitizing email is a solved problem with this:

https://github.com/cure53/DOMPurify

I'm not the author but I've used it in the past. According to the readme it is what Fastmail uses.

Probably not fast enough for us, but it is a very nice library. (We use libxml2.)

The reference we test against, by the way, is Mike Cardwell's excellent Email Privacy Tester:

  https://emailprivacytester.com/

Try it on your own mail client!

We have no speed problems using this in production at FastMail (we're kind of known for our speed in fact…). We use it at render time, both on desktop and mobile.

Also, Google has a tool called Caja that is open source:

https://github.com/google/caja

You mention that the client is a really heavyweight app. A quick look at the packages for Windows, iOS, and Android suggests that you're using Python, specifically the CPython interpreter. Would you choose something else if you were starting today? In particular, I assume that stuffing Python into the mobile apps was challenging. These days I'd use .NET/Xamarin or the Elements compiler (www.elementscompiler.com) when starting on a new multi-platform client-heavy application like this.

If starting today, I'd probably try to use Rust.

However, using Python actually makes it much lighter weight than it would be running all native code, because Python bytecode is quite compact, and we use small C extensions for things that need to be fast. Being able to link with C code is hugely important for us, and I don't know if .NET/Xamarin would allow that..?

Getting Python to run on a POSIX environment is actually pretty trivial. Windows RT is harder, which is one reason we've never bothered with a Windows RT port. And on iOS you are limited to using a single core under Python, because you aren't allowed by the O/S to fork.

The biggest pain I've had with Python is memory management: it's almost impossible not to leak memory in a Python program because the garbage collector is quite primitive and reference counting is just so error prone. For an email app you need to be able to run for days/weeks without crashing or blowing up memory usage; you can't just restart the process every 3am like you can server-side.

That's one reason Rust is appealing: memory management is helped by the type system.

Re: invoking C/C++ from .NET

https://msdn.microsoft.com/en-us/library/aa288468(v=vs.71).a...

I seriously doubt that using Python bytecode rather than native code makes the application more lightweight. Here are the problems I see:

1. Code size: Bytecode instructions may be more compact than their native code counterparts, since the bytecode instructions are dealing with higher-level abstractions. But the dynamism of Python prevents some major optimizations. First, the bytecode has to include identifiers, since everything is late-bound. More importantly, a packaged Python application tends to include a lot of dead code, because the packaging tool can't accurately tell what modules, let alone individual functions, are actually used. And that's assuming you're using something like py2exe which at least tries to prune the set of distributed modules.

2. Startup time: Of course, even native code can vary here. If you have an executable that's statically linked except for OS libraries, the OS can just read that executable into memory and jump to the entry pooint; that's fast even for a cold start. But if your application consists of an executable plus several DLLs or shared libraries, the OS has to do a lot more work to get it loaded. Python modules tend to be more fine-grained than typical DLLs, and for each module, the interpreter has to find it on the search path, then open the file and read it.

3. Memory footprint: Native code is mapped into memory as shared read-only pages. The OS doesn't actually have to add any of those pages to the working set until they're executed, and it can page them out at any time without having to write to a swap file, which mobile platforms don't have. A few bytecode interpreters, most notably Dalvik, have managed to exploit this, but not Python. So all the Python bytecode has to be in memory all of the time. This is exacerbated by the limited ability to trim the code size as described above.

More generally, my thinking on performance, especially perceived performance, in end-user applications has been heavily influenced by this blog post:

http://chadaustin.me/2009/02/logic-vs-array-processing/

Here's the money quote:

> With disk and memory speeds improving so much more slowly than CPU speeds, the difference between a snappy desktop application and a sluggish application is a handful of page faults. When choosing a technology platform for a project, it’s worth considering the impact to overall responsiveness down the road. And I’m pretty sure I just recommended writing your entire application in C++, which sounds insane, even to me. I’ll leave it at that.

That post was several years old. These days I'd expect him to recommend Rust over C++.

Of course, you've been developing software for a long time, and you've actually used Python in a mobile app; I haven't. So maybe the issues I've raised just don't matter. I'm particularly curious if startup time has been an issue for Inky at all, especially on the mobile platforms.

Hard to say, really. Of course if you write something entirely in C or assembly language, you can make it really compact. In the article you cited, for example, Chad Austin mentions GOAL, which was what we used for the Crash Bandicoot games. That worked well because it was domain-specific and because the rest of the code was all highly-optimized C and hand-coded assembly (the latter mostly by me, in fact). And it allowed us to pack the (tiny) code for the critters in data pages that were very, very precious (the PS1 only had 2MB of RAM).

However, x86 assembly, at least, is pretty big, and -- as you point out -- Python instructions are quite high level so you need far fewer of them to do anything. In practice, .pyo files really are generally pretty compact. I'm not sure how it compares to ARM, which has a less crazy instruction set. But I'm very skeptical on the code size claims. It "feels" like Python code is much smaller, but that said I haven't done a real experiment.

Regarding startup times, you do have to be careful about imports and regexes: it's easy to write code where importing a module results in a ton of additional imports you don't really need yet, and regex compiles that are better deferred. But in any case, the big driver of startup times now isn't Python -- it's paging in and decrypting the database pages required to construct the first inbox view. That's all done by SQLite, so Python's largely irrelevant to the startup time these days.

I'm not sure how things come down with respect to memory footprint, but we fit comfortably even on a lowly iPhone 4, so I think it's becoming less of a practical issue.

The huge win for me personally using Python has been that it is extremely easy to come back to code I wrote several years ago (or that someone else wrote) and immediately understand it. For such a nasty domain as email, this really helps a lot. Anecdotally, I've been coding in C dialects since 1979 and I still haven't got the same ability to understand old C code as I do old Python code. For context, virtually all the code I wrote at ITA was C and C++, and Inky is the first "real" thing I've used Python for. It really is a nice language, despite having (in CPython, at least) some frustrating warts (and the whole Py2 vs Py3 issue).

Finally, it's worth pointing out that we do use C for the most heavyweight things (embedded database, TLS/crypto, parsing, image format conversion, etc.) So we're probably not typical of either a C or a Python app, really.

FWIW, you've been coding in C dialects since just before I was born. So your experience and judgment have a lot of weight for me. Thanks for taking the time to discuss these things.

It's typical for Python applications to depend on C libraries for databases, crypto/TLS, and image format conversion. For parsing, it depends; lxml is popular for XML and HTML parsing, but I don't know of any off-the-shelf equivalent for RFC 822 and MIME.

Supporting POP3 seems like an anti-feature in 2015.

It seems to me, that muddling its lack of features into the middle of an application that properly supports IMAP would make your internal api much more complex?

Actually POP is easy except for efficiently determining what's changed in the mailbox since the last time you checked. And some providers still only offer POP access -- for example, Verizon.

From the outside, it looks like you have great engineering talent, but on the product management side you seem to lack some vision other than to cover backwards compatibility. This is frustrating to witness, even at a distance.

Perhaps I should clarify that I worked for a founder where we had this problem, and after a year almost of the engineers I started with had left. It was maddening, and I finally had to leave as well. The company was eventually sold for pennies on the dollar.

Reply on news.ycombinator.com