Are Cybersecurity Performance Measures Realistic?

A government watchdog urged the White House to establish metrics that would help determine the effectiveness of federal cybersecurity initiatives, but it’s a lot easier to recommend developing outcome-oriented performance measures for cybersecurity than it is to actually develop them.

See Also: Live Webinar | Securing the Cloud: Mitigating Vulnerabilities for Government

The Government Accountability Office in a report Thursday called on Office of the National Cyber Director to establish performance measures and estimate implementation costs. GAO said the Department of Treasury collects data on the dollar value of annual ransomware-related incidents, which the watchdog said “demonstrates that developing such measures is feasible and can be used for measuring effectiveness.”

Office of the National Cyber Director staff pushed back on GAO’s recommendations, according to the report, stating “it was not realistic to develop outcome-oriented measures at this point” and that “estimating the cost to implement the entire strategy was unrealistic.”

Experts say it can be particularly challenging to develop performance measures in cybersecurity due in part to the constant, rapidly evolving and complex threat landscape, as well as the complexity of cyber systems and emerging threats.

“Even if you pen test a cyber system today, you don’t know whether attackers will discover a new zero day tomorrow,” said Jim Dempsey, a cybersecurity expert and lecturer for UC Berkeley Law. Dempsey previously criticized the Transportation Security Administration in 2022 over a set of directives issued in response to the Colonial Pipeline ransomware attack that he said lacked measurable outcomes.

“Instead of developing faux performance metrics, we should focus on controls,” Dempsey recommended, saying technical controls like multifactor authentication and resetting default passwords “have large ROI” and “adherence to them can be measured.”

“There are actually very few – if any – metrics for cybersecurity that are truly performance- or outcomes-based,” he noted.

GAO warned that ONCD “will be limited in its ability to demonstrate the effectiveness” of the national cybersecurity strategy until the office establishes adequate, outcome-oriented performance measures.

In its response to the report, ONCD partly agreed with GAO and said that “developing outcome-based performance measures for cybersecurity is a challenging topic.”

The office said it will assess initiatives included in the cybersecurity strategy that lend themselves to outcome-oriented performance measures and apply those measures to initiatives “to the extent that validated measures exist.”

Developing outcome-oriented performance measures can be difficult, but there are benchmarks and indicators that can and should be used to gauge the effectiveness of national level cybersecurity initiatives, said Austin Berglas, a former FBI cyber division special agent, now an executive at cyber defense platform BlueVoyant.

Those indicators include incident response times, the percentage of systems patched, total unresolved vulnerabilities, the number of employees trained on up-to-date cybersecurity practices and the cost of a cybersecurity incident, Berglas said, adding that a “lower overall cost to recover and respond to an incident may be an indicator of effective measures.”