Open-weight models match closed frontier models on CTI-REALM at lower cost

Key points

• Study evaluated 26 models, 19 open-weight and 7 closed, on the CTI-REALM security benchmark.
• An open-weight model reportedly matched the strongest closed models at under a tenth of their cost, given enough turns.
• A published limit on agent messages reportedly zeroed out tasks, so relaxing it changed rankings and favored open-weight models.
• Different models lead different workflow stages, suggesting per-stage routing instead of choosing one model by overall score.

Why it matters for marketers: Teams building security agents can weigh routing workflow stages to different models, and should not judge models by price or a single overall score.