Hacker News Reader

The spiciest file I've ever had to deal with was an 18TB text file with no carriage returns/line feeds (all on one line). It was a log generated by an older Nokia network appliance. I think I ended up 'head'ing the first 2MB into another file and opening that, then I could grok (not the AI) the format and go from there.

View on HN

Oof, that sounds nasty. Did it turn out to be a standard-ish formatting with a separator where you break the line after x number of separators? I really dislike having to parse a log like that before just being able to read the log

From memory there was no dedicated event separator, it just went straight from the last character of the event to the first character of the timestamp of the next event. I think there was an XML payload in the event somewhere too?

Fortunately I didn't have to edit the log in-place as we were ingesting it into Splunk, so I just wrote some parsing configuration and Splunk was able to munch on it without issue.

Reply on news.ycombinator.com