Comment by sergiosgc | Hacker News Reader

Comment by sergiosgc | Hacker News Reader

Parent Full thread

> Basically, all messages in a mailbox are numbered from 1 to N, and you can reference a message by number, and when a message is deleted all message numbers above it shift down by one. But since this could confuse clients that are sending inflight requests, you can't start translating from the new shifted-down-by-one numbers until you've told your clients about it, which could take a theoretically unbounded amount of time.

Message ids are monotonically increasing, but not necessarily consecutive. You may need to renumber when you hit the 32-bit ceiling, but that is much less frequent than what you describe. Worst case scenario, you can disconnect unresponsive clients when renumbering, clients behave well on disconnects.

This invalidates your conclusions.

> (1) You can't have concurrent mailbox access because it might mess up MSNs reified to disk

You can. Renumbering is not frequent.

> (3) you can't have connections live too long because they passively eat up RAM with queued updates and a stored MSN list

My dovecot IMAP processes use an average of 5MB reserved memory, of which 4MB is shared memory, so a server with 100GB memory can hold close to 100k connections.

> (8) because opening a new session can be heavy.

Monitoring tells me that on loaded production servers, session open + select + list + logout occurs comfortably in less than 100ms. That's not heavy.

Dovecot starts to fail on mailboxes with about a million messages. It's a limit I can live with. Users have learned to archive messages by date.

IMAP has two ways of numbering messages. It is important to be clear whether you are discussing message sequence numbers (which are dense, and compacted on EXPUNGE) or unique IDs (which are not compacted). GP was complaining about MSNs, your reply is about UIDs.

As fanf2 said, MSNs != UIDs. Also, I didn't mean to imply that MSNs are an insurmountable limitation for performance, they just complicate things a lot.

I'm not familiar with Dovecot's code or very good at reading C, but I think part of how they handle it is here: https://github.com/dovecot/core/blob/81b5b188c478ec36bea8bda...

Basically, building a mail index for UID<->MSN translation that can serialize to disk, which would explain how it doesn't necessarily need absurd amounts of RAM and can select fast, probably at heavy cost when EXPUNGeing old messages. I'm sure there were worse implementations that did just maintain it all in RAM way back in the day, and still some that do (like the one I ran into).

Dovecot puts those index files in the maildir too, so one can read those directly if wanted.

The cost of deletions is heavy, and I wouldn't have done it this way, but note that where it really matters is if you have to do this repeatedly in a loop. But you don't have to do it repeatedly -- you can just delete a ton of messages at once, at the same cost as deleting a single once. As long as user perceptions go, the network latency is probably going to dominate shifting down even a million integers (< 2 ms on my machine), which I suspect most people don't even have.

> Basically, building a mail index for UID<->MSN translation that can serialize to disk, which would explain how it doesn't necessarily need absurd amounts of RAM and can select fast

That does explain why indexes can get corrupted and need to be cleared out, and why load balancing IMAP can get tricky.

Reply on news.ycombinator.com