Unconstrained CTI-REALM with Open-Weights
Key points
- Study evaluated 26 models, 19 open-weight and 7 closed, on the CTI-REALM security benchmark.
- An open-weight model reportedly matched the strongest closed models at under a tenth of their cost, given enough turns.
- A published limit on agent messages reportedly zeroed out tasks, so relaxing it changed rankings and favored open-weight models.
- Different models lead different workflow stages, suggesting per-stage routing instead of choosing one model by overall score.
Why it matters for marketers: Teams building security agents can weigh routing workflow stages to different models, and should not judge models by price or a single overall score.