Agent S: An Open Agentic Framework that Uses Computers Like a Human

February 27, 2025
Saaket Agashe*, Jiuzhou Han*, Shuyu Gan, Jiachen Yang, Ang Li, Xin Eric Wang
Paper Code Product

Hey! A few months ago, I gave a talk at Princeton University on my thoughts about agents and Simular. Figured I should put together a summary and turned it into a blog post.

State-of-the-Art Performance

My first job was as a research scientist at Google DeepMind, where a key part of my role involved collaborating with various Google product teams to identify opportunities for applying our cutting-edge AI technology. However, one Googler asked me a totally unrelated question that may have ultimately sparked my decision to leave DeepMind and start Simular.

Agent S Open Source First Launch on October 2024 - YouTube

Agent S Open Source First Launch on October 2024

Simular185 subscribers

Agent S is a new agentic framework designed to enable computers to be used as intuitively as a human would

We introduce an Experience-Augmented Hierarchical Planning method. This method utilizes Online Web Knowledge for up-to-date information on frequently changing software and websites, along with Narrative Memory to leverage high-level experiences from past interactions. By breaking complex tasks into manageable subtasks and using Episodic Memory for step-by-step guidance, Agent S continuously refines its actions and learns from experience, achieving adaptable and effective task planning.

Abstract

We present Agent S, an open agentic framework that enables autonomous interaction with computers through Graphical User Interface (GUI), aimed at transforming human-computer interaction by automating complex, multi-step tasks. To this end, Agent S introduces experience-augmented hierarchical planning, which learns from external knowledge search and internal experience retrieval at multiple levels, facilitating efficient task planning and subtask execution. In addition, it employs an Agent-Computer Interface to better elicit the reasoning and control capabilities of GUI agents based on Multimodal Large Language Models. Evaluation on the OSWorld benchmark shows that Agent S outperforms the baseline by 9.37% on success rate (an 83.6% relative improvement) and achieves a new state-of-the-art. Comprehensive analysis highlights the effectiveness of individual components and provides insights for future improvements.

Agent S addresses three key challenges in automating computer tasks:

Task Instruction

Help me to remove the account “anonym-x2024@outlook.com”

  1. Open Account Settings:
    agent.click (41,1, “left”)
  2. Switch Applications:
    agent.switch_applications (“Thunderbird”)
  3. Open Account Settings:
    agent.click (95,1, “left”)
  4. Remove the Account:
    agent.click (86, 1, “left”)
  5. Remove the Account:
    agent.click (93, 1, “left”)
  6. Remove the Account:
    agent.click (149, 1, “left”)

Pipeline of Memory Construction and Update

The pipeline of memory construction and update, which contains two phases: Self-supervised Exploration and Continual Memory Update. The initial Narrative & Episodic Memory is constructed through some randomly curated tasks during the exploration phase, and then it is updated based on the inference tasks continually.

Main Result

This table shows the performance comparison between Agent S and the baseline models, evaluated across the whole OSWorld test set. For the GPT-4o model, Agent S achieves an overall success rate of 20.58%, nearly doubling the performance of the best corresponding baseline (GPT-4o with 11.21%).

Agent S consistently outperforms the baselines in the “Daily” and “Professional” tasks, where it reaches 27.06% and 36.73% success rates, respectively, compared to the best baseline results of 12.33% and 14.29%. These tasks are commonly used in daily life or involved with knowledge-intensive professional applications, which benefit more from the retrieval augmentation of Agent S.

Analysis

To demonstrate the effectiveness of individual modules of Agent S, we stratified sampled a subset of 65 instances from the full test set for the ablation study.

Learning from experience enhances the domain knowledge of GUI agents

Learning from universal experience available as web knowledge allows Agent S to make informed plans across a wide range of tasks and has the most significant impact. The results demonstrate that each component plays a critical role in enhancing the agent’s domain knowledge. Removing all three components (w/o All) degrades the performance significantly, revealing the importance of learning from experience in the design.

ACI elicits better reasoning abilities of LLMs and supports better agentic learning

Comparing the baseline with Agent S (ACI-only) highlights the enhanced reasoning abilities achieved by incorporating ACI.

Hierarchical Planning supports long-horizon workflows

The observed performance drop underscores the importance of Hierarchical Planning in modeling long-horizon workflows.

Exploration, Continual Memory Update and Self-Evaluator are indispensable for memory construction

The results reveal that ablating both the continual memory update and self-supervised exploration phases results in a performance drop.

Generalization to Different Operating Systems

We test the Agent S framework with no modification on WindowsAgentArena, comparing Agent S with the similar configuration with GPT-4o as the MLLM backbone.

BibTex

@misc{AgentS, title={Agent S: An Open Agentic Framework that Uses Computers Like a Human}, author={Saaket Agashe*, Jiuzhou Han*, Shuyu Gan, Jiachen Yang, Ang Li, Xin Eric Wang}, year={2024}, archivePrefix={arXiv}, primaryClass={cs.AI} }

Understanding the AI Agentic Framework

The AI agentic framework combines artificial intelligence (AI) with agent-based modeling for improved decision-making processes.

Key Concepts of the Agentic Framework

  • Agent-Based Framework: Individual agents work together, boosting efficiency.
  • Agentic Approach: Agents act independently, highlighting their ability to learn and adapt.
  • Workflows: Planned paths that agents follow to enhance processes.

Applications of AI Agentic Framework

  • AI Framework Variations: Adjusted to meet specific industry needs.
  • AI Solutions: From virtual assistants to intricate management systems.

Benefits of Using an Agentic Framework

  • Efficiency: Increases productivity by reducing manual work.
  • Quality Management: Ensures consistent quality through structured processes.

Challenges in Implementing Agentic Frameworks

  • Data Privacy: Protecting sensitive data is critical.
  • AI Governance: Setting regulations for proper use and oversight.

Conclusion

The AI agentic framework is instrumental in utilizing intelligent systems effectively, fostering innovation, and enhancing efficiency.