What changed in AI today

• Benchmark pairs LLM-written scenarios with hidden probabilistic graphical models that supply exact ground-truth answers.
• Main leaderboard ACED-CORE has 180 tasks; ACED-QUAL adds a 150-question yes/no qualification check.
• Seven contemporary LLMs and two earlier models were tested across three active agent designs.
• Reported failures include testing the wrong intervention, losing relevant evidence, and continuing after sufficient evidence appeared.

Why it matters for marketers: Content teams relying on AI for research summaries should note that agents may gather evidence poorly or stop too late, so outputs need human checks.

• Study evaluated 26 models, 19 open-weight and 7 closed, on the CTI-REALM security benchmark.
• An open-weight model reportedly matched the strongest closed models at under a tenth of their cost, given enough turns.
• A published limit on agent messages reportedly zeroed out tasks, so relaxing it changed rankings and favored open-weight models.
• Different models lead different workflow stages, suggesting per-stage routing instead of choosing one model by overall score.

Why it matters for marketers: Teams building security agents can weigh routing workflow stages to different models, and should not judge models by price or a single overall score.