The Developer’s Guide to Audit Logs / SIEM

This article suggests that there should be separate data collection systems for: analytics, SIEM logs, and performance metrics.

The article mentions the CEF (Common Event Format) standard but not syslog or GELF or other JSON formats.

[ArcSight] Common Event Format [PDF]: https://kc.mcafee.com/resources/sites/MCAFEE/content/live/CO...

GELF: Graylog Extended Log Format: https://docs.graylog.org/en/latest/pages/gelf.html

Wikipedia > Syslog lists a few limitations of Syslog (no message delivery confirmation, though there is a reliable delivery RFC; and insufficient payload standardization) and also links to the existing Syslog RFCs. https://en.wikipedia.org/wiki/Syslog

Are push-style systems ideal for security logshipping systems? What sort of a message broker is ideal? AMQP has reliable delivery; while, for example, ZeroMQ does not and will drop messages due to resource exhaustion.

Developers simply need an API for their particular framework to non-blockingly queue and then log structs to a remote server. This typically means moving beyond a single-threaded application architecture so that the singular main [green] thread is not blocked when the remote log server is not responding.

SIEM: Security information and event management: https://en.wikipedia.org/wiki/Security_information_and_event...