What Counts as the Same Task? Executable Contracts for Robustness Evaluation of Tool-Using Agents

arXiv:2610.14477v1 Announce Type: cross Abstract: Robustness tests for tool-using agents often change the agent's environment, for example by hiding a hostile note in an invoice, and then judge the agent by what it does. The result tells us about the original task only if the changed test case is still a valid test of that task. Many evaluations decide this with weak rules, such as accepting any well-formed edit or any edit that stays close to the original in embedding space. We propose executable task contracts. A contract states which facts define a task, which edits are allowed, and what fi

This content is for informational purposes only and is not professional advice. Specifications, prices, plan tiers, and features change frequently and may differ from what is shown here; verify current details on the manufacturer's or company's official page before purchasing. Ratings are based on analysis of published documentation, not independent lab testing.