JMAP: A better way to email (2014)
blog.fastmail.com
blog.fastmail.com
A quick look at the protocol looks like in order to get a list of mailboxes, you have to ask for them all. In IMAP, you can say, show me top level mailboxes, show me direct child mailboxes of mailbox x, etc. In a previous role, I was responsible for working on a mail archiving solution which had a large hierarchy of folders (into the thousands IIRC) in a single account. Access to the archive was via IMAP. Looks to me that JMAP here would require the client to potentially fetch megabytes of data just to start showing a list of folders, where the solution we had with IMAP probably took a couple of kilobytes at most to get started.
It does make me think that the protocol may have been designed with less thought for uncommon use cases than IMAP was, but I've only spent a few minutes looking at the protocol, so I could be wrong.
Of course nothing stops you storing thousands of mailboxes in a non-heirarchical layout and making IMAP have large LIST fetches too.
(I do wonder - did your solution via IMAP still fetch everything each time, because in that case all you're doing is spreading that megabyte out over multiple round trips)
I'm not invested enough to sign up to a mailing list and try to convince people to change a protocol. I'm only invested enough to fire off a comment on a forum I am already signed up to. I explained my use case above. Whether or not it is compelling is up to the reader to decide. Maybe I'm the only person in the World who has dealt with large hierarchies of mailboxes.
> Of course nothing stops you storing thousands of mailboxes in a non-heirarchical layout and making IMAP have large LIST fetches too.
Yes. I would issue the LIST command, and as the IMAP server started streaming the results to the client, I would immediately process each mailbox as soon as it was down, rather than waiting for them all to be retrieved. I fear that with a JSON based protocol, most server and client implementations wouldn't deal with streaming (although they technically could), and would just deal with completed blobs.
Does the Cyrus implementation construct the entire blob of JSON before beginning sending it to the client, or does it start constructing and streaming the JSON as soon as possible, before it has even fetched the entire list of mailboxes from whatever the backend data store is? The JMAP streaming solution seems like a more complicated solution for developers to implement than the IMAP streaming solution.
> (I do wonder - did your solution via IMAP still fetch everything each time, because in that case all you're doing is spreading that megabyte out over multiple round trips)
People were accessing this mailbox with many different clients. How each of those clients worked under the hood, I do not know. I did build a web frontend, and that web frontend worked efficiently, fetching only the folders that were being displayed to the user, and their immediate children. A user clicked a folder to expand it, and they would immediately see the pre-fetched child folders, and a new request to fetch those folders children was then triggered.
I'll pass a link to this comment on to the list and see if others chime in. The downside of allowing filtered getMailboxes is more protocol complexity, but maybe it's worth it.
On to the next topic:
Cyrus creates the entire blob. Even when it's streaming a LIST reply it still batches a lot in memory. You'll discover pretty fast if you don't do that, you might wind up holding a lock for a LONG time if you're trying to build consistent replies and the remote end is on a slow network link or just consuming slowly. So we batch all replies right now.
If I was building a brand new product I might use an engine which supported MVCC (like Postgres database for example) and hence have flexibility. Having said that - even then you might find that a server talking to slow clients would want to buffer to temp file and free up memory and locks while it streamed the data.
With HTTP we get around this by having nginx handle all the mess, so I guess that's what will happen with JMAP too, nginx buffering large bodies to disk.
Yes. And like everybody, I have my own priorities, and will not apologise for JMAP not being one of them. Somebody like yourself who is actually invested in making JMAP a success should welcome people pointing out real life use cases where the protocol you're trying to improve upon actually works better than the one you've come up with. I would hope you're spending significant effort trying to get people to look over your spec and point out flaws.
This is a very high burden to place on casual feedback from somebody trying to be nice.
Having said that, I'm not sure I agree that it's an unreasonable burden to ask somebody to join a mailing list and defend an opinion if they want to influence a standard which will hopefully be useful for tens of years and implemented by hundreds or thousands of people. Any additional complexity for server and client implementers becomes quite a lot of work multiplied by all those people, and the additional burden of implementing a bigger standard may even be enough to discourage people from using it at all.
And it's a fact, the decisions are made by those who show up and put the work in. Wishing it was otherwise doesn't change the reality that we need to make tradeoffs and that the IETF favours rough consensus and working code. Both of which take work.
I will forward this discussion to the mailing list, that's about all I can do as chair, I won't proxy all the IETF work over to comments on HN!
No big deal either way!
HTTP is not that bad, anyway. You don't have to use every quirk.
> many organizations don't allow anything but HTTP, so it's easier to just use it.
Remember when we allowed for newer protocols to have their own ports and didn't blacklist everything but HTTP? Why are we doing this again, now? Please don't say security/firewalls or else I'm going to ram an ice pick in my eye.
If your organisation is blocking ports, they're probably filtering HTTP. Companies that care about filtering deny access to most personal e-mail on work machines. So basically you're saying "let's abuse HTTP so people can violate company policy."
You may or may not also being saying, "Let's build a brand new awesome shiny latest and greatest protocol...and permanently tie it to the technical debt wreckage of HTTP/2.
oh and Google could just fix their terrible broken IMAP implementation.
HTTPS (when combined with JSON) provides requests and responses, without harsh size limits, with nice addressing. TCP provides a byte stream. A lot of people are finding it easier to express their upper-level protocols in terms of requests and responses than in terms of a byte stream.
Having to cope with NAT middleboxes breaking the byte stream doesn't help, of course.
As for existing solutions that have been around for a while, http keepalives implemented in a smart manner, and exposing apis through a cdn are huge difference makers.
This is still the most often cited reason, but I don't think it's a good one anymore (as said before, nothing prevents you from using TLS).
If you have a stateless protocol is dealing with autonomous request, HTTP is quite frankly the best general-purpose transport layer for you. The only other well-supported transport layers are the message-based UDP and the stream-based TCP, TLS (which offers encryption), SSH (which offers encryption and multiplexing).
If you've got a stateless request-based protocol, you want a message-response based transport, but UDP doesn't quite offer the same abstraction and lacks reliability guarantees and large message support. With a stream-based protocol on the other hand, you're bound to redo an ad-hoc, informally-specified, bug-ridden, slow implementation of half of HTTP[1].
Yeah, you probably don't need all of the features of HTTP for your protocol, but being able to use existing HTTP infrastructure solutions (such as proxies, load balancers, caches, testing tools, endpoint monitoring tools) is a great boon that you would never get with your own homegrown protocol.
BOSH uses long-polling, and as someone from the XMPP community I'll admit it's a bit of a hack around HTTP, mainly because it's stateful.
However, it's entirely possible to have RESTful/stateless long-polling, for example in Dropbox's API: https://www.dropbox.com/developers/documentation/http/docume...
I think the choice of HTTP vs WebSockets comes down to architecture style. WebSockets are optimized for bidirectional communication with a specific remote server. In other words, something about the service makes it better if client I/O is sticky, maybe due to an in-memory session.
HTTP can make more sense in a stateless architecture, where GETs and POSTs could potentially go to different servers in the cluster. Of course, one could use WebSockets statelessly too (have no messages depend on past messages for context).
"Our servers are in New York, and our developers are in Australia. Even on a good day, the ping times are over 230 milliseconds. Anything more than the absolute minimum number of round trips is felt very keenly."
I wonder if some web apps might be built better if developers were bandwidth throttled and latency hindered artificially on their own machines, for say, one day a week.
I think moving towards progressive web apps (PWA) should make the web much faster because good performance over poor bandwidths is one of the cornerstones of PWA.
PWA should help somewhat, but there is still quite a bit of javascript bloat and "too many assets" to combat. When my phone runs out of the included 4G and drops to 3G, many sites are unbearably slow.
[1] https://blogs.wsj.com/digits/2015/10/27/facebook-slows-the-i...
Sitting in a bus with unreliable phone connection on my way back home trying to reply to everything now. Cheers.
Next time, try dropping down to 2G and you should make it through fine.
The biggest expense is all the index.c stuff to support IMAP protocol requirements. There's a lot of cost in there which will go away with JMAP, and which will go away even more when we replace the separate cyrus.index files with a structured database per user. That's still got a ton of work to do though... one of the goals for long term Cyrus improvements.
That's how I would expect a JSON based protocol to work for pretty much all types of requests that provide potentially large lists in a response. I would expect getMailboxes to work this way too to be honest.
One of our slack messages is something that's become a bit of a cliche! "JMAP will fix it" - in this case, a lot of the cost is generating the message counts by iterating every match, and that's not going to be cheaper in JMAP. We don't have data structures that allow sorted partial search responses, so we can't generate the first page until we've resolved the search for every message :(
I've not seen it deployed anywhere else yet as a production thing, and I don't know how much progress has been made on standardisation of it, but it does seem to offer vast improvements over IMAP.
The ietf working group and the Cyrus IMAP source code are the most up to date right now. I'll be updating the proxy (perl, open source) soon.
I'm still unsure of the viability at this point, but time will tell.