HNHacker News
TopNewBestAskShowJobs

nbari

5 karma · joined October 29, 2016

[ my public key: https://keybase.io/nbari; my proof: https://keybase.io/nbari/sigs/6Ms0CtE2OkNkH6_qmXq_NSjQUcWa86pnOOe1KTi17FA ]
submissionscomments
nbari··on pg_exporter: A modular Prometheus exporter for PostgreSQL metrics
Repo: https://github.com/nbari/pg_exporter/
nbari··on Immortal – A cross-platform, OS-agnostic supervisor for Unix-like platforms
Hi, so assuming immortal was created using the best in the class programming language for its purpose, and let's say was following all the existing best practices, I still have one question that never really found a proper solution, which is:

For cases in where is required to monitor services like unicorn/gunicorn/Nginx that demonize them self and have the capability to fork how to make aware the supervisor of does changes and to continue monitoring the new process without entering into a race condition, to be more clear here is an example of this problem https://asciinema.org/a/80371, and work around's here http://serverfault.com/a/587065/94862.

Immortal and Systemd seems to fix this problem, but based on JdeB comments, they are employing the "old broken idea - Witness https://github.com/immortal/immortal/blob/9ed9087a8ef1e64d64...

Therefore wondering, what is the best approach, solutions for this cases?

Or could it be than Immortal is not that wrong, at the end, is solving a problem, maybe broken or not but is working and could be improved for sure:

https://asciinema.org/a/80360

As a reference, from systemd man page for options

Type= If set to forking, it is expected that the process configured with ExecStart= will call fork() as part of its start-up. The parent process is expected to exit when start-up is complete and all communication channels are set up. The child continues to run as the main daemon process. This is the behavior of traditional UNIX daemons. If this setting is used, it is recommended to also use the PIDFile= option, so that systemd can identify the main process of the daemon. systemd will proceed with starting follow-up units as soon as the parent process exits.

PIDFile= Takes an absolute file name pointing to the PID file of this daemon. Use of this option is recommended for services where Type= is set to forking. systemd will read the PID of the main process of the daemon after start-up of the service. systemd will not write to the file configured here, although it will remove the file after the service has shut down if it still exists.

nbari··on Immortal – A cross-platform, OS-agnostic supervisor for Unix-like platforms
For avoiding having issues, try to compile from source on your target platform, this is due the https://golang.org/cmd/cgo/

the readme file has some instructions about how to do it: https://github.com/immortal/immortal/

Regarding if is mature enough, strictly talking still not a stable version, current beeing `0.11.0` and not jet a `1.x.x` but the more you could test and evaluate could be nice.

Please feel to raise any issue if needed here: https://github.com/immortal/immortal/issues

nbari··on Immortal – A cross-platform, OS-agnostic supervisor for Unix-like platforms
Many thanks for your time on this extended feedback, is very nice to read and know more about other supervisors and what is expected to accomplish with this kind of tools.

I apologize in advance because unfortunately, the current documentation of the project don't cover all cases and because of that, some assumptions can be made until going deep into the code or testing, I will try to clarify some point hoping I could transmit a better idea of what the goal was and how is working, so pease bear with me.

For example, there is a "down" option, same as the "once" options https://github.com/immortal/immortal/blob/9ed9087a8ef1e64d64...

Same with the enable/disable that in immortal is "start/stop" or also -x that is "exit"

You indeed can send a signal STOP, mainly because of this when using [immortalctl](https://immortal.run/post/immortalctl/) the option "stop" is used instead of down:

https://github.com/immortal/immortal/blob/master/cmd/immorta...

When following a pid, you indeed can still send signals to the "adopted PID" if we may call it that way.

Now immortal is totally capable of knowing the process ID of the daemon, but the methods you described as for reading PIDS from a file and checking if process are up and running, does methods are mainly used within the `-f PID` follow PID option in where you want to supervise or know status about a PID that the supervisor didn't create and, for this in BSD systems Kqueue is used instead of long polling https://github.com/immortal/immortal/blob/9ed9087a8ef1e64d64...

I have some animations here http://immortal.run/about/ maybe they are not very clear but hope they could clarify this.

In case you know a better way of knowing if a process exists, something like asked here: http://stackoverflow.com/a/15210305/1135424, I will be more than happy to implement it.

Regarding the login options, I would say it was the main motivation to create `immortal`, http://serverfault.com/questions/752839/how-to-make-svlogd-s..., these days dealing with microservices and try to follow a 12-factor approach force you to innovate, in this case the idea of delegating a logger to a 3rht party is because doesn't limit your options, for example, if you want you could just pipe directly to [filebeat](https://www.elastic.co/products/beats/filebeat) and at the same time continue writing files to disk and rotate them.

The idea of not using link files was to simplify more the process, is true that while writing to the file there could be some issues, I will have to check/test more in detail, but the approach I took, was to keep checksum of the files to guarantee integrity and also based on the atime take actions: https://github.com/immortal/xtime, not perfect probably but is a start.

Regarding the portability, I think currently I am bound to the output and systems in where [xtime](https://github.com/immortal/xtime) could work, so basically, limits to, Darwin, FreeBSD, Linux, Dragonfly, NetBSD, OpenBSD,

Regarding the web server, the idea came by accident and I would say that thanks to the power of [go](https://golang.org/) this became very handy, mainly because daemontools/runit use named pipes for controlling the processes, but since they are not bi-directional you need two pipes, now the idea of using a Unix socket, is because they are bi-directional, therefore you just need one file and by "accident" you just inherit a nice capability that allows you to interact via HTTP with the socket.

http://immortal.run/post/nginx/

And well, for sure more work and improvements need to be done, including the bad documentation that may cause bad impressions about what really is wanted to be solved and cover.

One again thanks for your feedback and time on this, maybe we could use https://github.com/immortal/immortal/issues for special issues.

nbari··on Immortal – A cross-platform, OS-agnostic supervisor for Unix-like platforms
I think that's why they are external "monitoring" tools to check how the status of your services and rather than just restarting a daemon, the ones if required could add more instances, restart them or rebuild everything, maybe something like https://epazote.io/
nbari··on Immortal – A cross-platform, OS-agnostic supervisor for Unix-like platforms
Instead of using named pipes for managing the process, a Unix socket is been used, so if required this opens the possibility to manage via a web JSON API the services an example about how to use Nginx for this is here https://immortal.run/post/nginx/

The idea behind was to avoid a developer to deal with a DevOps/operator to just stop/restart services, something useful on staging environments.

For starting, stopping a process there is no need to unlink/link a file, this simplifies things a little bit, for example, https://immortal.run/post/ansible/ by only updating the file contents or "touching" the file it can be started or re-started.

When following a PID, signals can still be sent to the process, these things become handy while working with a lot of microservices.

The docs need to be improved, but any feedback, comments or even desired feature is more than welcome.

nbari··on Immortal – A cross-platform, OS-agnostic supervisor for Unix-like platforms
Maybe this can help, https://immortal.run/post/ansible/
nbari··on Immortal – A cross-platform, OS-agnostic supervisor for Unix-like platforms
Maybe this https://immortal.run/post/nginx/