OSWorld

Unlike benchmarks using simulated or simplified interfaces, OSWorld places agents inside actual Linux, Windows, and macOS virtual machine environmentscomplete with real applications like web browsers, office suites, code editors, and file managersto assess whether AI systems can automate computer tasks the way a human user would.

The benchmark includes over 360 computer tasks across domains including web browsing, document editing, spreadsheet manipulation, file management, and multi-application workflows that require coordinating actions across several programs.

Agents interact with the environment through standard computer interfacesscreenshots, keyboard input, mouse actionsand are evaluated on task completion success rather than intermediate step accuracy, forcing agents to handle the full variability of real application behavior rather than idealized API responses.

OSWorld was developed to address a gap in AI evaluation: existing benchmarks for computer use agents relied on constrained environments that didn't reflect the complexity of actual desktop computing.

AI researchers developing computer use agents (like those from Anthropic, Google, and Microsoft) use OSWorld as an external evaluation standard.

Practitioners building automation systems that control desktop softwarefor robotic process automation, accessibility, or automated testinguse OSWorld results to benchmark agent capability before deploying to real-world workflows.