Tools

I build open-source detection engineering tools at Detect-Forge (github.com/jamesbower/Detect-Forge). They validate your rules: not just that they fire, but that they still catch what they claim. AI-native, CI-native, and inspectable. Everything runs locally or in GitHub Actions; no data leaves your environment. Opt-in LLM features are BYOLLM (your API key, your model choice).

pip install detect-forge

detect-forge stale

Your detection rules are probably stale, and you don’t know it. stale audits Sigma and Elastic rule repositories for freshness across three dimensions: timestamp drift against MITRE ATT&CK (deterministic), semantic drift between rule logic and current technique descriptions (local embeddings), and optional LLM-generated diff proposals to close the gap. Green CI tells you your rule parses. stale tells you whether it still matches reality.

→ GitHub · The blog post that started it

detect-forge backtest

Catch noisy rules before they fire in production. backtest runs your rules against labeled attack telemetry (the OTRF Security-Datasets corpus, SHA-verified) and scores each one on precision, recall, and F1. Empirical evidence for what a rule catches, and what it doesn’t, before it ships. Every run emits a reproducibility manifest so the result can be re-checked later.

→ GitHub

detect-forge coverage

Which techniques do your rules actually cover? coverage maps your rule set against MITRE ATT&CK and reports the gaps, weighted by a priority technique list you control. Coverage counts are only meaningful if the rules work, which is why this runs alongside backtest rather than instead of it.

→ GitHub

detect-forge audit

All of the above in one report. audit runs staleness, backtesting, and coverage across a rule estate and produces a scored, explainable report with a fix list. Exit codes gate CI: 0 clean, 2 if a priority rule fails. Every score has a reason.

→ GitHub

What Comes Next

There is no hosted version yet, and there won’t be one until the numbers say there should be: the build gate is two teams paying for the audit. I publish those numbers every week. If you want to know when it happens, the build log is where it gets announced first.

Get the build log →

New tools ship with an announcement to the newsletter 24 hours before the public launch, and get discussed first in the Machine Learning in Security Discord.