Hacker News Reader

Comment by hirundo | Hacker News Reader

Does anyone know an easy way to tail follow two logs at once on a pair of load balancing servers?

  mkdir foo bar
  sshfs foo.server:foo.dir/ foo/
  sshfs bar.server:bar.dir/ bar/
  tail -F foo/foo.log bar/bar.log

A method that doesn't involve sshfs:

  servers=(foo bar)
  pipes=("${servers[@]/%/.pipe}")
  mkfifo "${pipes[@]}"
  tail -F "${pipes[@]}" &
  for p in "${pipes[@]}"; do
    true > "$p"
  done
  for s in "${servers[@]}"; do
    ssh "$s" tail -F file.log > "${s/%/.pipe}" &
  done
  %tail

There's timing issues on this alternative. I just ran each statement separately in a prompt, and luckily it worked. The first `true` statement needs to run after `tail` has started trying to read the first argument, and once that dies, the second run of `true` needs to run after `tail` starts reading the second argument. Last time I did something like this, I just typed the statements without the loops and ran by hand.

Wow, I never used sshfs but this seems like a fantastic convenience.

Sorry to plug lnav again, but if the data volume is not "too much"[1], the latest version of lnav has support for tailing files on remote hosts that are accessible via ssh:

https://lnav.org/2021/05/03/tailing-remote-files.html

So, to tail "/path/to/log" on host1 and host2, you'd start lnav with:

  $ lnav host1:/path/to/log host2:/path/to/log

[1] - This is the initial release of this feature, so it probably has some performance issues. I'd be interested to know if it worked or not.

I tried but my timestamp is of the form %Y%m%d.%H%M%S: Do you have an option to just specify the timestamp format? If not, can you give an example of defining own format? Many thanks

I'll add this timestamp to the set of defaults for the final v0.10.0 version later today.

The documentation for the format files is here:

https://docs.lnav.org/en/latest/formats.html

The "timestamp-format" property in the format definition is an array of the formats to be tried beyond the builtin set.

Here are some example formats written by a user:

https://github.com/aspiers/lnav-formats

I think some of them define their own timestamp format.

You can ssh host1 tail -F log | sed -e s/^/host1:/

But you might eventually want to log everything to a third host. Use syslog to send logs across the network and configure your local software to log to syslog.

When you have 10k hosts then sampling 1 in N log lines is still useful as a canary, while avoiding being a firehouse for the central log sampler. If something heinous occurs you can get the unsampled log from the original machine, if you need it.

Much more sophisticated techniques abound, and others will comment I’m sure, but it’s also nice to have the syslog trail as a fallback. You never know in advance what’s going to fail and syslog will show you everything from your Apache logs to kernel bugs and faulty NICs.

> You can ssh host1 tail -F log | sed -e s/^/host1:/

That's only for one host. If you mean something akin to:

  tail -F <(ssh ...) <(ssh ...)

that wouldn't work, because `tail` would wait for the first `ssh/sed` to close their output (to reach the end of the first pipe) before starting to read the second `ssh/sed`'s output.

You can do:

  $ ssh ... >a.log &
  $ ssh ... >b.log &
  $ tail -F a.log b.log

You're right. That's pretty practical. I guess I try too hard to avoid writing to disk when I can.

Honestly Loki and Promtail are so easy to setup, and jump into not just tailing and searching but incorporating logs into Grafana dashboards (you are using Grafana already to monitor host and service metrics, right?), that they make sense to run for everything but the smallest scale deployment (that is, a single service on a single node).

Use syslog to send all the logs to one place. Either a third server or just have one send to the other if you can handle the load.

Reply on news.ycombinator.com