News Date: 2026-09-29
Security teams are increasingly interested in compact AI models that can classify alerts, assign severity and recommend actions without generating lengthy responses. Sophos has examined this trend through Jev, a decision-focused model from TypeSafe.ai that returns probabilities across a predefined set of answers.
Efficiency Can Magnify Small Errors
The appeal is straightforward. A specialized model can make decisions faster and more cheaply than a general-purpose large language model. That may allow a security operations center to analyze more alerts, investigate more events and automate routine judgments that previously consumed analyst time.
The danger is that lower costs encourage organizations to use the model everywhere. Even a small error rate can produce a large number of incorrect decisions when millions of classifications are performed. A system that is reasonably accurate in a laboratory could therefore create more total mistakes after deployment simply because it is being asked to decide far more often.
Sophos also separates accuracy from calibration. A model may provide the right answer most of the time while remaining unreliable about its own confidence. That distinction matters when confidence scores determine whether an alert is closed automatically or escalated to a human analyst.
Security Teams Need Local Evidence
Organizations should test decision models against their own alerts rather than relying solely on public benchmarks. Useful measurements include missed intrusions, false positives, analyst workload, response cost and the model's ability to recognize uncertain cases.
- Validate each proposed use case with representative security data.
- Set separate confidence thresholds for different systems and risk levels.
- Require human approval for destructive or difficult-to-reverse actions.
- Retest models as infrastructure, users and attacker techniques change.
- Include adversarial text in evaluations because attackers may control parts of the evidence an AI system reads.
Automation Must Earn Its Authority
In my view, the important lesson is not that inexpensive AI decisions should be rejected. It is that automation should receive authority only after its behavior has been measured under realistic conditions. A model that saves a few seconds per alert can provide substantial value, but not if confident mistakes allow a genuine intrusion to be dismissed.
The economics of AI will make automated judgment increasingly common. Security leaders should focus less on the price of each decision and more on the operational cost of the decisions that turn out to be wrong.