OSWorld

If you wish to run the baseline agent used in our paper, you can execute the following command as an example under the GPT-4o pure-screenshot setting:

Parallel execution (example showing switching provider to docker)

The results, which include screenshots, actions, and video recordings of the agent's task completion, will be saved in the ./results (or other resultdir you specified) directory in this case.

For manual verification and examination of specific benchmark tasks, you can use the manual examination tool:

Please start by reading through the agent interface and the environment interface.

Correctly implement the agent interface and import your customized version in the run.py (for single-threaded execution) or scripts/python/runmultienv.py / scripts/python/runmultienvxxx.py (for parallel execution) file.

Afterward, you can execute a command similar to the one in the previous section to run the benchmark on your agent.

title={OSWorld: Benchmarking Multimodal Agents...