Thanks. You can find some of that code here:
https://github.com/twosigma/postgresql-contribI need to finish the open sourcing of tailfhttpd though. It's... an open-coded HTTP server, written in C, specifically tailored for tailing files over HTTP. It uses epoll and inotify, and is C10K, and blazingly fast. You have to front it with nginx or envoy to get TLS support, naturally. Because it's so simple, open-coding HTTP/1.1 seemed reasonable at the time, though nowadays writing this in Rust with some reasonable framework would be better.
The key to tailing files over HTTP is to have a server that supports:
- weak ETags (st_dev, st_ino, generation number)
- If-Match and If-None-Match conditional request
support
- Range requests
- when the right end of the requested range is
left unspecified, use chunked transfer-encoding
and don't send the terminator chunk until
a) EOF is reached, and b) the file is renamed
out of the way or unlinked (which the server can
detect using inotify or similar)
With this clients can tail a file. Heartbeats can only be supported by the application itself -- HTTP/1.1 does not have the ability to send empty non-terminating chunks (HTTP/2.0 doesn't have that problem).
If client loses its tail, it can simply resume by using If-Match and Range starting at the next byte offset after the last byte received before. So recovery is trivial.
And presto, super cheap, simple, and reliable pub/sub.