Put A Little Log In Your Life

Logging for compliance is hard. Sure space is hard, and math is hard and diamonds are hard - but logging for compliance is really hard. Which is why most seem to struggle with it. There is no hard and fast rule for compliance logging but I will do my best to provide a little guidance, as I too have struggled with this and if my solutions can help just one person…

PCI-DSS Hates You

Sure that’s an antagonistic sentiment, but it feels right. For the sake of scope, I am narrowing things down to PCI-DSS because that is what I am currently working on. It’s not the only game in town compliance-wise, but it will suit our needs.

Take a minute to peruse the logging specific section (10.6) of PCI-DSS- you don’t have to go much further than 10.6.1 to get a grasp of the hill you have to climb. I use the term “hill” in the Himalayan sense here.

“Review the follow at least daily”

  • All security events
  • Logs of all system components…
  • Logs of all critical system components…
  • Logs of all servers and systems components…

It’s no wonder many folks just throw up their hands and resort to the log-everything-and-sort-it-out-later (in a time of crisis) solution.

We’ll just conveniently ignore the “review” part of this for now… that’s a whole other can of worms, or Graboids rather.

While the log-everything strategy at first blush seems to be the most direct route to success, or at least to checking the box, it is, in my opinion, a disaster in the making.

  1. Even in the smallest of shops, logging everything quickly becomes untenable.
  2. The length of retention requirement for the volume of logs you are about to ingest will make your project costs and complexity soar.
  3. In a time of crisis when you need to recover artifacts for forensic review, they will be buried under a mountain of noise creating additional stress and putting your investigation in jeopardy.

What do you do?

Precision Guided Logging

In the simplest terms, be smart about your logging. This is not without risk, as with PCI-DSS the details of what you need to log are intentionally vague, so this process requires some diligence on your part, possibly a little guesswork and nerves of steel because at some point you have to draw the line on what to log and what not to and accept the fact you might miss something. However, we are going to do everything we can to minimize that risk.

Let’s break this down a bit, the first part of 10.6.1 is “thou shalt review the following daily”. They don’t even bother to acknowledge this is an impossible task for a human being. In the cosmos of time-sinks, this is a singularity. This is where you and the business need to acknowledge this is a machine-assisted process, you need technology to solve this problem and that will cost money.

ENTER THE LOGGING PLATFORM

Loggly, ELK, Splunk, Sumo Logic, PaperTrail, there is no shortage of options here, however I went with Graylog. Before I go too far, you can find a great intro to Graylog here, Abe does an excellent job of laying things out if you’re unfamiliar with Graylog. Give Big Abe a follow on Twitter while you’re at it.

Costs vary, and with some of these solutions you can definitely get your logging project off the ground with only investments of time and hardware. That is not the same as FREE.

Architecture is important, don’t undersize your logging solution. I built a 3-node Graylog cluster on spare (read “aged”) hardware due to budget contraints. Despite being out-of-warranty, the servers are capable. The cluster has a nominal capacity for logs of ~600GB. While I see over 1000 Events per second on the input side, this is trimmed down significantly and is something I will get into in a future blog. We see bursts up to 3x that and even 10x at one point due to some issues with the systems that were sending logs. Lesson: be very careful when trying to log SQL server stuff.

In the end it amounts to 5-6GB/day of indexed logs which are split into a variety of indices. As such, some logs of lower volume that get their own dedicated index are easy to keep for longer periods, whereas others of higher volume pose greater challenges.

Prepare To Be Assimilated

You must comply. In general the minimum retention period for compliance seems to be 1 year. Should you have a 100TB SAN backing your logging solution, then fine, log it all… I guess. Otherwise, if you are like the rest of us, begging for scraps and thankful for every bit we can get, you need a more refined approach.

If you look at your logs at all, it quickly becomes evident that some, make that many, or even most, are of little use in a compliance sense. Some logs have near zero value and are generated in copious amounts. This is noise.

The very best approach for noise is to tune it out at the source when possible. For many reasons, this is often not an option, but when you can do it, do it. This should be the very first order of business once you get your logs flowing to a central location. Kill the noise.

In our PCI-DSS use case, the next step is to curl up with a cup of {insert beverage here} and read through section 10.6 multiple times to try to get your head around what precisely you need to log and retain to be compliant. PCI-DSS documentation only gets you part way there, as it is intentionally vague, but it is a start.

Love Me, Define Me

So my long-winded diatribe has finally lead us to the meat and potatoes part of the story. Accurate definitions of what represents specific in-scope logs are hard to come by, but scouring the Internet has borne fruit. These are specific to Cisco ASA firewalls and their ilk:

New Net Technolgies has a good post in which they also provide a link to this article from Cisco. As a bonus, the Cisco doc also has some advice for killing the noise.

Overall, logging from the ASAs is like drinking from the firehose and those assets alone have proven to be by far the most prodigious source of logs in our environment. However, the amount of those logs that fall into PCI scope is only a fraction.

I have a Graylog pipeline for filtering these logs and marking them for PCI retention by adding a new field called “PCI” which I set to the value “true”.

Graylog Pipeline

rule "fw_pci"

when

    has_field("cisco_msg_code") AND

    contains(to_string($message.message),"%ASA-6-606006:") ||

    contains(to_string($message.message),"%ASA-6-611101:") ||

    contains(to_string($message.message),"%ASA-6-611102:") ||

    contains(to_string($message.message),"%ASA-6-111009:") ||

    contains(to_string($message.message),"%ASA-6-113008:") ||

    contains(to_string($message.message),"%ASA-6-113012:") ||

    contains(to_string($message.message),"%ASA-6-713228:") ||

    contains(to_string($message.message),"%ASA-3-414002:") ||

    contains(to_string($message.message),"%ASA-3-414008:") ||

    starts_with(to_string($message.cisco_msg_code),"%ASA-2-109") ||

    starts_with(to_string($message.cisco_msg_code),"%ASA-3-109") ||

    starts_with(to_string($message.cisco_msg_code),"%ASA-4-109") ||

    starts_with(to_string($message.cisco_msg_code),"%ASA-5-109") ||

    starts_with(to_string($message.cisco_msg_code),"%ASA-6-109") ||

    starts_with(to_string($message.cisco_msg_code),"%ASA-2-113") ||

    starts_with(to_string($message.cisco_msg_code),"%ASA-3-113") ||

    starts_with(to_string($message.cisco_msg_code),"%ASA-4-113") ||

    starts_with(to_string($message.cisco_msg_code),"%ASA-5-113") ||

    starts_with(to_string($message.cisco_msg_code),"%ASA-6-113") ||

    starts_with(to_string($message.cisco_msg_code),"%ASA-5-5021")

   then

    set_field("PCI", "true");

end

One thing of note, unless you are extracting / normalizing logs the same way I am (you aren’t), you will not have an index field called “cisco_msg_code”, so you can’t use the above pipeline rule exactly as written, but you can adjust according to your needs.

But I Digress

This ended up being far more lengthy that I was expecting, so congratulations if you made it to the end. I will continue this series as well as posting a separate blog on the log trimming pipeline I do to keep the ASA verbosity from over-running the log server like a Mongol horde.