Comment by DenisM | Hacker News Reader

Comment by DenisM | Hacker News Reader

Speaking of logs, I want to put some logging in place for my web server. I log every single request with extensive details, so I can debug things later if needed. It's several gigabytes per day now, so I can no longer just dump it on disk as I did for the last couple of years.

Since I'm on AWS EC2, I want to try this:

  - Write the logs to local SSD, asynchronously 
    so as not hold back the http request.
  - Have a separate cron job that loops through 
    the log directory and scoops up all the files.
  - The job will then stuff those files into a Kinesis Firehose.
    AFAIK, Kinesis Firehose does not require any capacity provisioning, 
    unlike the Kinesis Streams, so I'm set "for life" (up to 5MB/second)
  - The firehose will accumulate the logs and put them into S3. 
    Hurray unlimited storage!
  - S3 will trigger a Lambda.
  - Lambda will parse through the log from S3, pull out 
    interesting properties (IP address, user id, session id, etc) and 
    stuff them into a DynamoDb table.
  - If I need to see data from one user/ip/session I will use DynamoDb 
    to find the right S3 blobs.
  - If I need to reprocess the logs to extract a new piece 
    of data that I did not foresee earlier, I can run a 
    map-reduce task

Except the last piece, this looks like something I can half-ass in a couple of days and forget about it for another couple of years.

Any opinions? I don't really want to use a SaaS log service because gigabytes per day.

I've heard some bad things about Kinesis in general. Why not have your cron job just put the logs onto S3 directly?

I've used Lambda a bit. The debugging process can be a pain, since you're forced the upload a ZIP file, and if your code times out Lambda doesn't give you any traceback to indicate what happened. There's also a maximum run time of each Lambda invocation, which I believe is 5 minutes. Is there a chance your parsing may run longer than that? Also, what will you do if you upload some bad code and the parsing fails? Will it be the end of the world if you lose data while you fix the parsing?

Oh, I see you plan on doing map-reduce to re-parse the logs, so maybe that part isn't as big a deal.

You could also consider doing something like rsyslog -> db-of-choice while also rotating the files off to S3 for long-term map-reduces. This is all to ignore the obvious ELK cluster solution, which will give you good data visualization and investigation options, but may be more of a headache to set up and maintain than you are looking for.

Anyway, those are my thoughts. Hope they help.

Thanks for sharing your thoughts, esp on Lambda.

For Kinesis, I planned to use Firehose, not Streams (the latter have to be provisioned, which I was hoping to avoid). The firehose could put data into S3 for me, and S3 would trigger lambda. However I just realized that S3 will only make 3 attempts to invoke the Lambda, so that pretty much rules out this part of my design - the data will not get lost, but it will not get indexed either. I may run map/reduce later, but I don't want to be dependent on doing that to pick the loose ends.

These are just server logs, they don't affect business continuity. Still I wouldn't want to be sitting there and wondering "is this user having connectivity problems, or did I just lose a pile of logs?".

I could probably put the files directly into S3, got carried away stacking my AWS features together. :) I'd need to be more careful with batching, so as not to create batches too small or too large. Perfectly doable, though Kinesis Firehose already does that for me. Plus in case of the EC2 instance death I will lose the current batch with the hand-rolled solution, but not with Firehose.

So I guess I should just put a batch of data directly to S3 and send an SQS message to make sure that it's indexed properly, then delete the local files.

I really don't want to pick up another thing that I have to understand and manage. Like ELK. Someone who knows ELK will probably have no problem managing it, but my head is full with business domain problems.

We just went through an extensive ecosystem survey and reached an identical design, which we're currently implementing. So far, so good! AWS has aws-kinesis-agent, which you can deploy for the log aggregation bit, which is very easy to use.

I just found out that an s3-triggered lambda is only retried 3 times and then it gives up.

Reply on news.ycombinator.com