Almost every company we work with struggles with cybersecurity metrics. Personally, I am not surprised. At the end of the day effective security means you didn’t get breached. From that perspective, the thing we are trying to measure is essentially the absence of “bad”.

“How do our security metrics look this month?”

“They look great – Zero breaches.”

“Great job – keep spending millions of dollars to keep it there!”

I know it sounds absurd, but this vignette isn’t terribly far from the truth. The reality is cybersecurity is much more complex than a few numbers can describe. Compounding this complexity is how interrelated everything is.

In previous articles I’ve highlighted that some of these relationships create competing incentives, which in turn create friction. Detection engineers want to increase visibility and add new detections. The existing detections are already generating a high volume of alerts that SOCs struggle keep up with. Too often a new detection is released into production and immediately floods the SOC with added volume of new alerts that end up being false positives.

The natural result is competing incentives between the two groups – increased visibility for one group at the cost of reduced efficiency for another.

These competing incentives can be extrapolated to underlying security metrics themselves. A common frustration we see in Fortune 500 SOCs is a focus on just one or two metrics that are driven by a myriad of contributing factors. MTTR may go up 10% one month, which by itself is bad. However, if that increase is qualified with the fact that a new security data source was added and new detections went into production, the MTTR increase may indicate the SOC’s successful absorption of the increased visibility.

Risk Based Alerting not only changes the game for detection engineering and incident response workflows in a large enterprise, it opens up a new potential for how we create and track associated security metrics. Based on many conversations with SOCs and security managers, we would like to propose a new set of metrics, enabled by Splunk® with Outpost RBA, that tells a complete story around just how effective your security program is, as well as their rates of improvement.

Category #1 – Visibility Metrics

Visibility is the blessing and the curse of security analytics and alerting. On one hand you have ALL the data in system and traffic logs, and Splunk allows you to ingest and search that data in near-real time.

The curse is that there is A LOT of data, and in that data is mostly “business as usual” activity. It is the goal and the challenge of cybersecurity to identify the “bad” amongst all of this “good”.

Security Data

Data is the foundation. If you don’t have the data, you are blind to whatever may be happening. This is our first set of Metrics:

Proposed Metrics:

Data sources – How many and are the number of them growing over time?

Data volume – What is the volume of activity we are monitoring and how is it changing over time?

Security Detections

Security detections are what we use to find things in these terabytes a day of log data. How we use these large volumes of data is our second set of metrics:

Proposed Metrics:

Detections – How many and are they growing (we call these risk rules in RBA)

Data Diversity – What categories of security data are we searching? (e.g. Endpoint, Email, Network)

MITRE Coverage – How do these detections map to MITRE ATT&CK and is that coverage growing?

Threat Intelligence – What volume of Threat Intelligence are we collecting and matching in our security detections and is it changing?

Overall – these metrics give us context around how our security visibility is changing. An increase in any of these numbers I would argue is positive as by simply increasing visibility, you are decreasing risk. However, an increase in visibility has a potential 2nd order effect of increasing Incident response metrics. More visibility, more things to respond to, more work for IR.

A decrease in any of these metrics is, on the surface, bad. Especially if you simply stop seeing data that you were searching before. However, some of the decreases could easily be explained by the 2nd set of metrics we have defined.

Category #2 – Environment Metrics

IT & security data comes from somewhere – your enterprise’s environment. And as Captain Obvious would point out here – large environments are very dynamic. These dynamics directly contribute to the volumes of security data that we are measuring in the visibility metrics. Naturally for added context, we need to measure these as well.

IT Environment Changes

Users – how many and are they growing over time?

Business units – where do these users come from? (e.g. adding a newly acquired company)

Infrastructure – what infrastructure changes have taken place since the last measurement period? (e.g. did we add a new security tool? Go live with a significant cloud migration?)

These are few direct root causes for changes in visibility metrics. While a few of them may be qualitative, they at least can be observed as influencers and make some sense of more dramatic changes in visibility, for the positive or the negative.

Category #3 – Incident Response Metrics

SOC metrics seem to be the most popular reports up to the CISO and the board. I’m not going to argue against them, but I do believe these are the easiest to calculate, which is why they’ve become mainstay. Herein lies the source of the frustration of the incident response directors and managers. These metrics taken on the surface don’t tell the whole story. In addition to MTTO, MTTR, and any other performance metrics, we propose adding top of funnel metrics to quantify the true production of your incident response program.

Detection Data/IR Metrics

Risk Events – the volume of risk events produced by the security detections – in essence the output of all your security detections. Essentially a raw volume of “potential IOCs”.

Notables – the volume of security alerts generated by the analysis of the risk events of which an analyst needs to review and make a decision on.

Historically, we have seen these metrics have a natural ebb & flow, based on a number of contributing factors (many of which we are measuring in earlier categories). Simply put, user activity varies from day to day, week to week, as does attacker activity. One example is an observable increase in phishing campaigns on Tuesdays and Wednesdays between 8-10am.

Time and resources are limited. When you report the productivity metrics of an incident response program, you also need to include contextual volume metrics to understand if the overall performance is improving, staying the same, or decreasing.

Bonus Category – Outcome Metrics

The metrics presented so far are meant to be very straightforward and easily measured or reported. (We are happy to show you our Beta dashboard in Splunk – just ask). However, they all fail to capture the most important measurement of all – and that is EFFECTIVENESS.

How effective is our security and is it improving?

Great question. Trust that Highland Defense is committed to help your company answer this question. In the meantime, let me introduce you to some of our thoughts.

If we want to measure security effectiveness, then we need to measure the outcomes of security operations activities and investments. Fortunately, the SIEM with Outpost RBA has the foundational elements to do so. Let’s use this simple rubric to frame the universe of security alert outcomes.

Alert Outcomes and Desired IR

Note: These outcomes are represented in “hind-sight” – it’s very difficult to classify each alert with this information at the time it is generated.

We already measure speed in relation to alerts or notables. Notables are closed with a “status” field, so we have some indication of their outcome. If we could add “accuracy” and “impact” to these measurements, we could EASILY tell if a security program is becoming more effective. Couple these metrics with the visibility metrics above and imagine the confidence your CISO could present to the board in demonstrating the robustness of your security posture quarter after quarter.

New Security Metrics Made Possible by Outpost RBA

We hope that after reading this it has given you a fresh perspective on creating and presenting security metrics that tell the whole story. We see security teams kicking-butt every day in some of the world’s largest companies. Trying to capture and report the magnitude of their contributions, and put numbers to just how much risk is being reduced or eliminated by the work of detection engineers and security analysts, is the challenge.

At Highland Defense we are working to solve this challenge and believe the approach summarized in this article is a step on the path forward. Please reach out to us to discuss in depth or see some of the dashboards and reports we’ve built to tell these important stories. The stories of your security teams working hard in the trenches to level up the security of your organization every day. Let’s put some metrics behind them and let the CISO beam with pride at the work you are doing.

Finally, here is a just sample of the metrics we can pull automatically from you Splunk data.

Cross posted on Linked-In