Back when I started DShield.org, one of the challenges was dealing with variations in log formats. 10+ years laters, this problem hasn't really changed, even though there are some promising solutions (which isn't that different form 10+ years ago).
Firewall logs are a pretty simple example. The basic information captured is pretty similar across different firewalls: Packet header data. Some log formats are more verbose then others, but the idea is the same and it is not too hard to come up with a standard to express these logs. For DShield, we used a smallest common denominator approach. It wasn't our goal to collect all the details offered by different firewalls. For an enterprise log management system however, you may need to preserve this detail, and the simple tab delimited format we came up with for DShield wouldn't be extensible enough.
One of the logging standards that is gaining some steam is "CEE", or "Common Event Expression" . To be successful, a logging standard has to address a number of different problems:
So how does CEE solve these issues?
CEE supports two different formats: XML and JSON. XML is the "primary" standard allowing for the most flexibility, but JSON, due to its simple structure, is easier to parse and sufficient in many applications. It is also not terribly hard to convert JSON to XML.
CEE doesn't really solve all of this problem, but it starts by defining common labels and data types (like "src.ipv4" for the IPv4 address of a source). In part, CEE refers to other standards like CVE to come up with a vocabulary to use to identify events.
I didn't list this problem above, but it is certainly important to consider how logs are transported. In the Unix world, various versions of syslog have become the de-facto standard for log transport. But once you leave Unix based systems, syslog support is no longer a given. CEE addresses various issues like support for compression and protecting log integrity (which plain old syslog doesn't do well at all)
I do think CEE is certainly a standard to watch out for. Right now, the standard is labeled as "beta". The tricky part will be vendor support. The CEE board does include representatives from a number of important vendors, but I don't see a lot (any?) log management vendors on the list. Of course CEE would help the most if devices generating logs would support it.
*/Defending Web Applications Security Essentials - SANS Brussels September 2019
Oct 19th 2012
6 years ago