Detection Rule Staleness: How AI Can Solve This Now

Your detection rule staleness is leaving you vulnerable, and you don’t know it.

Here’s a scenario I’ve watched play out more times than I’d like. Six months ago, we wrote a Sigma rule for T1059.001 – PowerShell. We tagged it to the technique, PySigma parsed it clean, CI went green, and we shipped it. And it’s been running in the pipeline ever since. No errors. No false-positive flood. Dashboard is still green.

In those six months, MITRE modified T1059.001 more than once (new procedure examples, revised data components, shifted language around what the technique actually encompasses). Our rule is still technically accurate; it catches the thing we wrote it to catch, but the technique has grown around it. What used to be 80% coverage is now 40%. Nothing failed. Nothing told me. The gap just opened underneath the rule.

That’s detection staleness, and the deeper you look, the more you realize it’s really two problems pretending to be one. One is solvable with deterministic checks that anyone can write in a weekend. The other has been solved using a novel machine learning approach.

This post is about both problems, the bridge between them, and the solution that addresses both. There’s a reveal at the end about a tool that handles all this shipping in May.

The Two Problems Hiding Behind “Staleness”

When most engineers first hear “detection rule staleness,” they reach for a timestamp check: when was the rule last modified, when was ATT&CK last updated, what’s the gap? Call this timestamp drift. It’s real, and it’s tractable. You query the ATT&CK STIX API, pull the technique’s modified timestamp, compare it to your rule’s last touched date, and flag anything past a threshold you set. A weekend project. Maybe a hundred lines of Python.

Then there’s the harder problem. A rule whose timestamps look fine; modified six weeks ago, ATT&CK technique modified seven weeks ago, drift looks acceptable. But that rule no longer matches what the technique now describes. MITRE rewrote the technique’s text and added new procedure examples that your rule doesn’t cover. Not to mention restructuring data components in a way that makes your detection logic narrower than the technique is now. The rule is timestamp fresh and semantically obsolete at the same time. We can call this semantic drift (or “stealth stale”). It’s invisible to any check that only looks at metadata.

Here’s the same picture in table form:

The left column is what every modern detection-as-code pipeline already covers. The middle column is solvable with a deterministic check against the public STIX API. The right column requires something else entirely.

Why CI Misses Both

Three structural reasons your pipeline misses staleness today. They’re all fixable, but none of them get fixed by the tools most teams run today.

ATT&CK is a living standard, not a fixed reference. The framework moves under you. Techniques get added, split, merged, deprecated, and rewritten multiple times per year. MITRE publishes updates at least quarterly. Major releases can restructure entire tactics; sub-techniques didn’t exist before v7. Most teams pin to whichever version of ATT&CK existed when the rule was written and never recheck. The rule’s tag becomes a fossil pointing at a technique that no longer looks like it did the day you tagged it.

CI validates syntax, not semantics. Green CI is a dangerous green. Your detection-as-code pipeline is almost certainly running pySigma, a KQL linter, or an EQL parser on every pull request. Those tools verify your rule parses. They do not verify that your rule still matches current reality. The pipeline will cheerfully ship you a semantically obsolete rule, and the PR check will stay green the whole way.

No tooling exists at the practitioner layer. The enterprise space has detection-adjacent platforms, SOCPrime for content marketplaces, CardinalOps for detection posture management, Spectrum Security (just launched with $19M seed) for automated detection maintenance, AttackIQ/Cymulate for adversary-emulation validation. Each solves a real problem.

But none of them address this problem.

None of them publicly claims to track the specific gap where an ATT&CK technique description gets rewritten, and your rule quietly falls behind, or our “semantic drift” problem. That’s the gap. The problem isn’t skill; it’s that the practitioner layer has never been built.

The Scale of It

The Anvilogic 2025 State of Detection Engineering report found that 86% of security professionals take a week or more just to deploy a new rule. If shipping a rule is that expensive, what’s the realistic budget for auditing the hundreds of rules already in production? In my experience, it rounds to a big ol’ zero. Detection libraries accumulate. But they don’t get pruned. And they certainly don’t get revalidated against a standard that has moved three or four versions forward since the rule was written.

Staleness isn’t a failure of diligence. It’s the predictable output of a workflow with no step for it. Here’s what I see when I look at a typical detection-as-code repo in the wild:

• Rule last-modified dates from 18+ months ago
• ATT&CK technique tags locked to whichever version was current at write time
• A pull request template that asks about log sources but never about technique drift
• Zero tooling that would tell anyone if a tag no longer means what it did

The timestamp problem is real but solvable. The semantic problem is where things get interesting because the solution doesn’t come from inside detection engineering. It comes from a completely different industry that’s been solving an almost identical problem for over a decade.

The Solution: Three-Dimensional Scoring

Here’s a solution that handles both problems together — plus a third dimension that turns the audit into something actionable instead of just a list of red flags.

Dimension 1: Timestamp drift. Deterministic. Query the ATT&CK STIX API. Pull modified timestamps for every technique your library tags. Compare to your rule’s last-touched date. Flag anything past a configurable threshold. Cost: nearly free. Catches: every fossilized rule pointing at a technique that’s been rewritten.

Dimension 2: Semantic drift. Embedding-based detection logic. Embed each technique’s current description from the STIX bundle the same way. Compute cosine similarity. Then, flag rules whose similarity to their tagged technique falls below a threshold you set. Cost: ~2 seconds per technique on first run, milliseconds with caching. Catches: every stealth stale rule that timestamps would miss.

Dimension 3: LLM diff proposals. BYOLLM here. For rules flagged by either of the first two dimensions, send the rule + the current technique description to an LLM (OpenAI, Anthropic, your local model) and ask it to propose specific edits. Human reviews the proposal in a pull request. The LLM never auto-applies. You bring your own API key, so you control cost and data. Cost: pennies per flagged rule. Catches: gives you a starting point for the fix instead of just a red flag.

The three dimensions stack. Timestamp catches the obvious. Semantic catches the hidden. LLM proposes the fix. Each layer has a clear input, a clear output, a named threshold, and a reason. None of them is a black box.

Why Explainability Matters Here

A pattern worth flagging about AI-powered detection tooling: most of it is opaque by necessity. Enterprise platforms need IP moats. Their scoring math lives behind an API. Along with their LLM prompts are internal. Their “fleet of agents” makes decisions you can’t audit. This is a rational business strategy at a venture scale.

It’s also exactly why detection engineers are skeptical of AI in their pipeline. In security, false confidence is worse than slow. A rule you can’t reason about is a rule you can’t trust.

The three-dimensional design is built on the opposite premise: explainability is a feature, not a constraint.

• Every score has a reason. Timestamp drift = a specific date delta you can verify. Semantic drift = a cosine similarity number with a named threshold you can change. LLM diff = a proposal with a rationale, not an opaque output.
• Every layer is inspectable. The LLM prompts live in the repo as Jinja2 templates you can read and modify. The scoring math is plain Python.
• Every AI output is human-gated. The LLM dimension is opt-in and proposes and it never auto-applies. You review, you reject, you apply via PR. Same workflow you already use for every other code change.
• Multi-signal by design. Three independent dimensions mean no single layer can produce a false positive that slips through. If the timestamp says drift, but semantic similarity is still 0.95, you can downgrade the alert. The signals are designed to disagree usefully, not to amplify each other into noise.

What’s Shipping

I’m building this. It’s called TTP-Staleness. Here’s what’s in v1:

• CLI and GitHub Action that scores your Sigma rules on all three dimensions
• Sigma at launch; KQL and EQL coming in v0.2 (June 2026)
• Local embedding model – no data leaves your environment by default
• BYOLLM for the LLM dimension – opt-in, your API key, cost-capped by default
• Rich terminal output, HTML report, JSON output for downstream tooling
• Embedding and proposal caching – most reruns are <2 seconds
• MIT license, free forever

Public launch: May 23, 2026

A SaaS dashboard for teams comes later in Q3 2026: historical staleness tracking across multiple repos, multi-repo aggregation, Jira and Slack alerts, hosted LLM proposals so you don’t have to bring your own key, and team-level drift dashboards. The CLI stays free regardless.

Get Early Access

Subscribers get the GitHub link 24 hours before the public May 23 launch. No other spam. I send when there’s a tool to ship, not on a schedule.

If you want to go deeper on the math behind the three-dimensional design, like the actual embedding code, the prompt templates, the eval harness for LLM proposal quality, etc., I’ll be publishing a technical follow-up on April 30th. Subscribers get that first, too.

If you have a Sigma rule library you’d like to test against on launch day, send me a note on X at @jamesbower. I’m looking for a handful of publicly available repos to benchmark against in late May, and I’m interested in what stealth staleness looks like in real-world repositories at scale. Thanks!

Additional reading

• MITRE ATT&CK STIX 2.1 data — the public source of truth this whole post depends on
• Sigma HQ — the community detection rule repository most readers will recognize
• Anvilogic — 2025 State of Detection Engineering — source of the 86% figure
• pySigma — the syntactic validator that (correctly) stays silent on semantic drift