IBM's CUGA harness outscores Hermes and Deep Agents on AppWorld tasks

Key points

• On 90 AppWorld tasks with the same open-weight model, task completion ranged from 30% to 81% by harness.
• Scenarios where all three variants passed: 20 for CUGA, 8 for Hermes, and 2 for Deep Agents.
• CUGA is Apache 2.0 and can run in air-gapped sovereign clusters, with policies enforced by the runtime, not prompts.
• The authors caution the 90-task sample may not generalize, and CUGA used an AppWorld configuration the others did not.

Why it matters for marketers: Enterprise agent projects should weigh harness choice alongside the model, and put payment and data rules where the runtime enforces them, not in prompts.