newsletter

TEST: Beehiiv API publish verification (safe to delete)

B
Brett Kelsey · Chief AI Officer at Athena
· 5 min read

I spent twenty minutes yesterday watching a security vendor demo that should have taken five. The sales engineer kept circling back to integration points that didn't matter to the problem we were trying to solve. Classic case of solving for the demo instead of the outcome.

This happens more than it should. Teams get so focused on proving they can connect to everything that they forget to ask whether the connection actually reduces risk or just creates another dashboard to ignore.

Integration theater costs more than the license fees

Most security tools get sold on their ability to plug into your existing stack. The pitch deck shows a diagram with lines connecting to every major platform in your environment. Looks impressive in the meeting. Adds zero value if those integrations don't feed a decision that changes how you operate.

I watched a client spend six months integrating a SIEM with fourteen different data sources. When we finally asked what decisions they were making differently because of those connections, the room went quiet. They had instrumented everything and operationalized nothing. The data flowed, but nobody was acting on it in a way that reduced their actual exposure.

The cost isn't just the engineering time to build those integrations. It's the opportunity cost of not focusing that effort on the three or four connections that would actually change how fast you detect and contain a real incident.

Working backward from the decision beats working forward from the capability

The security programs that actually move the needle start with a simple question: what decision are we trying to make faster or better? Then they work backward to figure out what data feeds that decision and what tools can reliably deliver that data when it matters.

I worked with a public sector client last year who wanted to cut their mean time to contain in half. We didn't start by surveying their tooling or mapping their integrations. We started by listing every decision their analysts made during a typical incident response. Which alerts got escalated. Which containment actions required approval. Which context determined whether they isolated a system or just flagged it for monitoring.

Once we had that list, the integration strategy wrote itself. We needed endpoint telemetry feeding directly into the ticketing system with business context attached. We needed identity logs correlating with access patterns in real time. We needed containment actions that could execute without waiting for three people to approve them in Slack.

They went from nineteen vendor integrations to seven. Response time dropped by 60% in the first quarter. The integrations they kept were the ones that fed a decision they'd defined in advance.

Alert volume without business context is just expensive noise

Most SOC teams are drowning in telemetry that doesn't map to risk the business cares about. They know something fired. They don't know whether it matters. So they investigate everything or they ignore too much, and neither strategy scales.

I've seen Fortune 500 security teams generate 50,000 alerts a week and still miss the adversary who established persistence in a developer environment because nobody had tagged that subnet with business context that explained why it mattered.

The alert said "unusual outbound connection from non-production server." The business context should have said "this server has read access to customer PII and production deployment credentials." One of those descriptions gets ignored. The other gets escalated.

Agentic AI can help close that gap, but only if you've done the work to define what business context matters before you start tuning the models. The AI can't infer risk priority from alert metadata alone. You have to teach it what your organization considers critical, and that requires someone who understands both the technical environment and the business logic underneath it.

The rollback plan tells you whether you're ready for production

I've asked this question to a dozen security teams in the last six months: what's your rollback plan when the new tool makes a bad call? Most of them look at me like I asked a trick question.

You're not ready for production if you can't answer that in detail. Not in theory. In practice, with a documented procedure and someone accountable for executing it. Especially with agentic AI tools that can take automated actions based on pattern matching that looked good in testing but fails in a scenario you didn't anticipate.

The best implementations I've seen treat rollback as part of the design, not a contingency plan you write after something breaks. They define decision boundaries before deployment. They instrument the system so they can see when the AI is operating near the edge of its training data. They build kill switches that don't require three approvals and a change management ticket to activate.

That discipline doesn't slow you down. It's what lets you move faster with confidence instead of moving fast until something breaks and then spending six months in risk review meetings trying to figure out who signed off on the bad decision.

What I'm thinking about this week

The gap between what security teams measure and what actually predicts their insurance premiums keeps getting wider. Compliance scores stay flat or improve. Premiums double. That's the market telling you something about the difference between documentation and outcomes. Worth asking your team whether you're measuring the right things or just the easy things.

newsletter
B

Brett Kelsey

Chief AI Officer at Athena

View all articles

More from Brett Kelsey

ROUND2 TEST: Submit for review flow (safe to delete)

Most security teams still measure their MSSP relationships by how many alerts got triaged last month. That's measuring activity, not protection. If your managed security partner can't tell you whic...

newsletter

Trust Over Terms: Rebuilding Channel Relationships That Deliver (DELETE TESTINGH ONLY)

I've rebuilt three channel programs in the past five years. Every time, the partner contracts I inherited looked like they'd been written by lawyers protecting against problems that hadn't happened...

newsletter

Security Theater: The Hidden Tax on Every CISO Budget (TEST: full customer walkthrough, safe to delete)

Most enterprise security programs spend 30-40% of their annual budget on controls that satisfy auditors but don't materially reduce breach probability. I've watched CISOs defend six-figure investme...

newsletter
Powered by OnAtlas