tstack

Yes, Log Parser was the inspiration for that feature in lnav. That's why the column names are basically same between the two (e.g. c_ip, cs_uri_stem).

Sorry to plug lnav again, but if the data volume is not "too much"[1], the latest version of lnav has support for tailing files on remote hosts that are accessible via ssh:

https://lnav.org/2021/05/03/tailing-remote-files.html

So, to tail "/path/to/log" on host1 and host2, you'd start lnav with:

  $ lnav host1:/path/to/log host2:/path/to/log

[1] - This is the initial release of this feature, so it probably has some performance issues. I'd be interested to know if it worked or not.

If you're tailing logs, can I suggest that you try out the Logfile Navigator (https://lnav.org). It really is possible to do better than tail/less/whatever when you just want to look at a local log file:

https://lnav.org/2013/09/10/competing-with-tail.html

I’m planning to use it in my terminal-based log file viewer (https://lnav.org) to deploy an agent (over ssh) to remote machines to monitor their log files. CosmoC allows me to build a single agent binary that will work on lots of target machines and package that into the main lnav binary. I’ll still be building lnav packages for different platforms, but the agent build and deployment is vastly simpler with CosmoC.

I’ll be using this feature in my day-to-day work because I interact with short-lived VMs running the software I’m developing. Being able to run lnav on my devbox and have it live-tailing logs on the VMs running the changes will be an improvement over the current workflow where I manually copy the files onto the devbox.

Check out the Netflix animated series as well, it’s quite charming.

Note that there are graphic novels as well as chapter books based on the Netflix version of the stories.

I wrote a log file viewer (https://lnav.org) to help me debug the software I was working on at the time (2006). This software generated multiple log files and I got tired of manually collating messages to figure out what went wrong. (The stuff ran in an appliance so we would get dumps from customers.) I’ve used this log viewer practically every day since and it’s been useful as I’ve moved from one job to another.

FYI, the Logfile Navigator (https://lnav.org) can display gzipped/bzipped files on the fly. The top-of-tree has new support for automatically unpacking archives to a temporary directory using libarchive and displaying the contents.

lnav[0] can take care of parsing the log files and displaying them in a TUI. There's also a SQLite interface for doing queries. However, you'll need to build the filters/queries yourself, there aren't any built-in ones at the moment.

[0] - https://lnav.org

> very few places AFAIK actually process their HTTP server logs into a form that can be queried by SQL.

There are at least two tools that support directly querying log files with SQL:

* logparser -- https://en.wikipedia.org/wiki/Logparser

* lnav -- https://lnav.org

lnav uses SQLite and provides quite a few extensions, so quite a lot is possible. The logs are exposed via virtual tables, so there is no separate import step.

> Personally, I find coding challenges fun and even quite enjoyable at times, so I don’t see why it’s a waste of time to get good at them.

You really can't imagine that someone feels differently? Please try to have a more open mind.

I, for one, would much rather spend my free time on my side-project that is actually used by people. I would have a hard time working on something that was a throwaway. (Which is what actually happened when I tried to interview at FB. I spent more time working on the side-project than studying leetcode. FB promptly told me I couldn't code... "OK")

Slightly off topic... but, if you're interested in console logging tools, have you looked into https://lnav.org or https://github.com/rcoh/angle-grinder/ ?

For the Logfile Navigator (https://lnav.org), I implemented a tool[0] to record console input/output and then replay the input while verifying the output. It basically works, but needs some usability improvement. For example, the expected output is the raw escape sequences, so it's hard to look at a diff and see what really happened. I think it can be improved, I just haven't found the time yet.

[0] - https://github.com/tstack/lnav/blob/master/test/scripty.cc

Uh... what? The comment I replied to argued that a random employee might understand why a particular bolt was chosen. The walkway collapse is literally about bolts/nuts being changed around. That design change by a third party resulted in a disaster that killed over 100 people... Seems like appropriate evidence that a line worker would not actually understand design details. That's not to say they couldn't make a suggestion and feedback their experience. Of course they should do that.

> But you still assume that the employee isn't smart enough to know why bolts are different sizes and whether a bigger bolt could work in the place of the smaller bolts.

It sounds like you are not familiar with this disaster:

https://en.wikipedia.org/wiki/Hyatt_Regency_walkway_collapse

There may be a very good reason things are the way they are.

For reference, a 435MB web access log with about 3 million lines takes:

* lnav about 4 seconds to index[1] and display

* vim about a second to display

* emacs about 3 seconds to display

This is on a late 2013 27-inch iMac with a 3.5Ghz i7 and the file cached.

[1] - Indexing in lnav involves parsing all log lines to extract their timestamp and log level (status codes in this case)

Try the Log File Navigator (http://lnav.org). It works directly with log files. Can understand many log formats by default and be customized. The log messages from separate files are collated into a single view where highlighting is also applied. You can even do analyses on the messages by executing SQLite queries (without having to insert the messages into a database).

> This is why I love this question as a phone-screener for onsite interviews and stuff.

Same here and I'm confused as to why this comment has been downvoted. It's a perfectly good question for level setting. You get an idea of how much the person knows about various aspects of the system without having to ask pointed questions that they may not know. Instead, they get to show you what they are knowledgeable about.

I built lnav (https://lnav.org) to help dig through the log files of the product I was working on at the time. I grew tired of having to manually interleave log messages from multiple files, so I created lnav to do the work for me. I started it over a decade ago and have used it at every job I've worked at since then.

JSON-formatted log files can be pretty-printed by tools like the Logfile Navigator:

http://lnav.org

That should make the decision a lot easier.

Parsing time is a huge cost when the Logfile Navigator (http://lnav.org) is parsing log files. The libc strptime() is terribly slow, so I wrote a simple compiler[1] that would spit out inlined C++ code for parsing a particular time format (e.g. "%Y-%m-%dT%H:%M:%S.%f%z"). Coupled with the fact that a time format is not going to differ across log lines in a single file, lnav is able to lock onto a date format and parse all the lines in a file fairly quickly.

[1] - https://github.com/tstack/lnav/blob/master/src/ptimec.cc

Give lnav (http://lnav.org) a try, it's a powerful tool for analyzing log files using a terminal.

Where does lnav fall short in this case? As the author of it, I'd like to know so that I can give you some tips or try to fill in the gaps.

Can you elaborate on "extra power"? A c++ constructor/destructor pair is equivalent to __enter__ and __exit__. I fail to see how this grants a significant amount of power. There's certainly differences, but the gap is probably not as large as your weasel words make it out to be.

You complain about having to put initialization into 2 constructors, but there's the flip-side with things like mutexes. In c++, you have to have 2 classes, one for the lock and one for the guard. Whereas with a context-manager, you have one class and the locking code is in the __enter__ and __exit__.

    with self.lock:
        self.do_stuff()

In that case, I would say that the context-manager is nicer.

Python has context managers, which were added for exactly this purpose.

  with alloc_resource() as resource:
      resource.use()

You can pipe into lnav like you can with less/more:

  make |& lnav

The '-t' option will prepend timestamps to the lines that are coming in on stdin so that it will be treated as a log file. The '-w <file>' option will write the stdin data to a file if you want to look at it later:

  make |& lnav -t -w /tmp/make.out

Is that what you're asking about?

There are Debian (https://packages.debian.org/sid/admin/lnav) and Ubuntu (http://packages.ubuntu.com/trusty/admin/lnav) packages available from other maintainers. I use the RPM at my day-job, so I use OBS for that. I've not spent time to setup OBS to build DEBs. I'll look into it, though.

Oops, I disabled this, sorry for the confusion.

Yes, log messages from different files are interleaved based on their timestamps.

Sorry! This seems to be a bug in squarespace. I've disabled the lightboxes for now.

There's also lnav (http://lnav.org), the Logfile Navigator. It can do what 'tail -f' and 'less' do plus: syntax coloring, filtering, loading compressed files, interleaves log messages from multiple files, and more.

Many of the comments I see on here about using '-F' with tail instead of '-f', live searching, and so on are handled in lnav by default. I would really encourage folks to give it a try.