1. I am curious if the author tried setting the autovacuum more appropriately for a table that is in continuous use at 50 new rows/second.
This article (http://okigiveup.net/postgresql-vacuuming-an-introduction-fo...) suggests:
ALTER TABLE person SET (autovacuum_vacuum_scale_factor = 0.0); ALTER TABLE person SET (autovacuum_vacuum_threshold = 4000);
This should result in entries being removed from the index and table regularly.
2. If I understand correctly, it seems the other part of the problem is that there are lots of rows to be individually tested to see if they can be locked, based on the way the author has implemented their queue. While lock attempts are being made on a set of rows, they aren't being deleted, since they're 'in use'. This non-deletion causes things to get out of control.
As far as I know, 9.5's SKIP LOCKED is meant to help address that (https://blog.2ndquadrant.com/what-is-select-skip-locked-for-...). Though it appears to be doing something kinda similar under the hood.
If the author of the original article (@ brandur.org) is following this discussion I would be grateful if they could evaluate these two options in the context of their test implementation.