Hacker News Reader

The locking thing has not turned out to be a really serious problem for me, and I have several gigabytes of email. mu index is really fast. Also, using the Emacs sendmail hasn't been a real problem. It turns out I don't need to be productive during the ~1s it takes to send the email and I have not run into any edge cases or bugs in the ~5 years I've been rocking this setup.

It doesn't matter how fast the index is, only one process can hold the Xapian lock, if I'm reading my E-Mail in Emacs with mu4e (which invokes "mu server") and try to index on the command-line I get:

    $ mu index [...]
    mu: mu_store_new_writable: xapian
    error 'Unable to get write lock on
    <Maildir>/xapian: already locked' (11)

How do you send E-Mail with no network connection? Last I checked this was some ad-hoc thing where you neede dto queue them up in some queue managed by Emacs, and remember to turn that mode off when you were back on WiFi (or hack up your own hooks to automate this), v.s. sending it to a local smtpd which already has retry logic built-in.

Why on earth would I try and index it twice concurrently?

I never have a scenario where I need to send email without network.

No, you wouldn't index from two processes concurrently, but for Emacs to read your mail in mu4e and for you to concurrently issue "mu index" on the command-line two processes ("mu server" and "mu index") would need to poen the same Xapian database read-write at the same time, and just like SQLite Xapian doesn't support that.

... I still don't see why I would do that. Emacs runs offlineimap and then Emacs runs mu index. Why would I do it again myself from the command line?

Yeah I use notmuch with Emacs and it's the same. If you try to search, refresh search results, or read email while the index process is running you get an error. It's not a big deal for me, because I download and index my mail from the command line, so I'll never be reading or sending email at the same time.

This used to be true but notmuch has added support for Xapian's DB_RETRY_LOCK feature a couple of years ago so if you have non-ancient notmuch (0.23+) and Xapian (1.3.2+) then the second process will wait for the lock instead of erroring out. The feature can be disabled at compile time if you don't want it for some reason.

> You need to do the indexing with a timer in Emacs, and because Emacs is single-threaded this blocks things / slows down the UI.

It's worth noting that multithreading recently landed in Emacs master, so it's now possible to avoid this sort of problem.

Now possible as in mu4e would need to use some API that's not in a release version of Emacs, or is it more transparent than that?

The former. I wasn't suggesting that mu4e should be converted right away to use this feature, but more concerned that an unqualified statement that long-running IO must block in Emacs would create the wrong impression.

The locking is only a problem because he's doing it wrong, I think. I'm doing this:

    http://notes.secretsauce.net/notes/2014/12/12_mu4e-and-desktop-notifications.html

I.e. there's an external script that invokes IMAP IDLE. And when IMAP tells it there's new mail, the script tells EMACS there's new mail. Then the reindexing happens within the emacs, and it uses the lock it already has. Makes this particular issue go away completely.

That must be it. I also trigger mu4e indexing using emacsclient, after IMAP sync. BTW you do not need to have mu4e open to index mail (and single-window mu4e mode will not have a * mu4e-main * buffer), it is enough to (progn (require 'mu4e) (mu4e-update-mail-and-index)).

I must also be missing something because I never explicitly index my mail. I fetch it with mbsync every two minutes and then use mu4e to read it and new mail shows up as one would expect.

¯\_(ツ)_/¯

How many tens of GB of mail do you have? I think this problem only tends to show up on big mailboxes.

10-25GB. But a fast computer, so?

Good for you. To save power and use smaller machines should always be an option. Indexing saves much resources here.

Agreed 100%. I'm just stating my case, not denying others'.

I recommend notmuch in place of mu/mu4e and msmtp in place of a full-blown smtpd implementation.

I also recommend isync's mbsync over offlineimap

Agree with using Isync. Notmuch lacks the backward synchronization to the IMAP server and to another mail client then notmuch.

I primarily use mu4e but want to see changes on my mobile as well as on any remote computer using a web client in a browser.

As far as I know notmuch can just be synchronized to another notmuch instance.

If you use https://github.com/gauteh/gmailieer you can have bidirectional sync between gmail tags and notmuch tags.

Personally, I don't bother with it. When I have to do mail on my phone, it's usually so limited in scope that I don't need to retain state when I sit down at my main computer. I can imagine that's not for everyone, though.

> Maybe I'm missing something obvious here, but calling "mu index" from the offlineimap postsynchook is going to suck, because mu4e invokes "mu server"[1], that aquires a Xapian lock on the index, which means you can't run the "mu index" while reading your E-Mail.

> This is the worst thing about mu/mu4e. You need to do the indexing with a timer in Emacs, and because Emacs is single-threaded this blocks things / slows down the UI. After trying to get it to work in some automated way I gave up. I manually issue a key-combo to re-index and then read my E-Mail.

I tried both approaches (actually, I had mu4e reindex on `g` query refresh, no need for separate keybindings), and prefer having indexing run after IMAP sync. I don't notice any problems (and I mostly use very slow Atom hardware).

> Another comment: I recommend not using Emacs to send E-Mail, but to outsource this to a local smtpd such as Exim (which I use with mu4e on Debian). Then if you don't have WiFi or whatever the smtpd just queues those up for later sending, and has queue inspection tools etc. All of the solutions for doing this in Emacs itself are dirty hacks full of edge cases and bugs smtpds have solved decades ago.

FUD. smtpmail.el can do this. smtpmail.el can queue mail, and you can run it in a separate Emacs process periodically or on demand, to attempt to deliver those emails. The emails are queued as files in a directory, so you don't need separate queue inspection tools. I run smtpmail.el this way with mu4e, and think it is a far superior solution to having to install and configure yet another external program.

I use imapnotify to connect to my imap server and respond only when a mail is received by running offlineimap to sync the mail and telling mu4e via emacsclient to sync. Since mu is never ever called except by emacs and only exactly when an email has come in there is never contention for the lock.

You can configure mu4e to queue mail to send later:

https://www.djcbsoftware.nl/code/mu/mu4e/Queuing-mail.html

Yeah, that's the dirty hack I was referring to. I don't need to tell an smtpd "now's when I expect not to have an Internet connection", which this method relies on, and could happen as I walk into another room and lose WiFi. I just give it mail to deliver, and if it doesn't work it keeps retrying until it works (with configurable backoff and a setting saying when to give up for good).

Yup, I see where you're coming from. I've actually gotten used to mu4e's queuing behavior, and just leave it on all the time. For one thing, I like having a chance to edit/delete mail after hitting send (sort of like gmail's "undo"), and for another, I like the satisfaction of having a dozen or so messages sent all at once at the end of an email session.

My mail fetching script kills mu before calling "mu index", like so :

$ pgrep mu && kill `pgrep mu`

$ mu index

It works perfectly. Mu4e then restarts mu when it needs it.

I think I saw that as the recommended setup somewhere in the mu docs.

> $ pgrep mu && kill `pgrep mu`

This could be rewritten as:

$ pkill mu

With `pkill $(pgrep mu)` or even with `pkill mu` you can kill mutt and any mu$ process as well.

`pkill -x mu` matches the whole name without substrings.