SCOTT CHILLE / CYBER PULSE / FIELD GUIDE
CYBER PULSE · WEEKLY THREAT INTELLIGENCE · SCOTT CHILLE
SOC Metrics in the Age of AI

Confidence Is Not Confirmation.

Security programs are busier and more automated than ever. What they're less consistently doing is proving any of it stops a real attacker. Here's the gap, and where AI is making it wider.

What CISOs believe
93%
can prove they took reasonable steps to prevent a breach
97%
confident endpoint protection catches lateral movement
VS
What actually gets tested
12%
have tested EDR effectiveness in the last 3 months
26%
use red team or pentesting to check SOC detection

Five patterns that reinforce assumed security

Individually reasonable operational choices. Together, a system that rewards activity over resistance.
01

Confidence outpaces testing

High assurance in prevention and detection, far less routine adversarial validation behind it.

02

Severity guides remediation more than exploitability

Version checks and severity scores drive closure. Confirming an attacker can no longer succeed is inconsistent — roughly a third of practitioners take scanner findings at face value with no further testing.

03

Detection is trusted more than it's measured

Widely deployed, rarely stress-tested. Only about a third of practitioners test SOC detection monthly or more.

04

Automation increases speed faster than verification

AI now touches prioritization and remediation directly. Independent confirmation of what it decided has not kept pace.

05

Metrics emphasize closure over resistance

MTTR and SLA adherence are tracked everywhere. Whether the exploit path actually closed is tracked far less.

The AI Angle

Automation Without Assurance

60%
of CISOs report AI is fully integrated into vulnerability management or remediation workflows
17%
independently test AI-generated recommendations with their own tools
32%
are fully confident in automated prioritization and remediation

AI adoption in the SOC is not experimental anymore, it's in production. But independent validation has not scaled with it. An automated system can close a ticket faster than a human ever could. It cannot tell you, on its own, whether the exploit condition is actually gone. Acceleration increases speed. Verification determines the outcome — and right now, verification is the piece most organizations are skipping.

The KEV Reality Check

When CISA or ENISA flags a vulnerability as actively exploited, how fast do organizations actually confirm they're clear?

Within 24 hours
11%
Up to 7 days
42%
Up to 2 weeks
30%
More than 2 weeks
16%

What to measure instead

The SOC-CMM's five domains, reframed around confirmation rather than closure.
01

Business

Does SOC output map to what leadership actually needs to know, not just what's easy to report?

02

People

Are analysts trained and staffed to validate findings, not just triage volume?

03

Process

Does the remediation workflow include a retest step, or does it end at "ticket closed"?

04

Technology

Is AI tooling in the loop verified against its own output, or trusted by default?

05

Services

Is detection timing measured against real attacker tempo, not just alert volume?

Confidence does not stop an attacker. Confirmation does.