How did you hear about us?

We are glad that you have chosen Simular, before installation we would like to know how you heard about Simular?

Others

Product hunt

.svg)TikTok

Instagram

.svg)X

Google search

.svg)LinkedIn

.svg)YouTube

Word of mouth or friend

Download Simular Pro

Thank you! Your submission has been received!

Oops! Something went wrong while submitting the form.

Featured

Agent S2: An Open, Modular, and Scalable Framework for Computer Use Agents

March 12, 2025

Code Product Agent S2 Technical Review

Computer-use agents are autonomous AI agents that observe, reason, and perform tasks on behalf of human users, by directly interacting with graphical user interfaces (GUI), including desktops, mobile devices, browsers, and various softwares. They function as intelligent intermediaries between human users and their digital tools in the most intuitive way—mouse and keyboard control, just like a human. This human-like ability to navigate and control software marks a foundational leap in AI, setting the stage for the next era of technological progress powered by autonomous computer-use agents.

Today, we're excited to announce our next leap forward in computer-use agents: Agent S2, the second generation of our agentic framework. Building on our initial successes, Agent S2 offers even greater performance and modularity by leveraging both frontier foundation models and specialized models. Agent S2 achieves new state-of-the-art results, scales well with more steps, and most importantly, it is fully open!

Agent S2 - An Open, Modular, and Scalable Framework for Computer Use Agents - YouTube

Tap to unmute

Agent S2 - An Open, Modular, and Scalable Framework for Computer Use Agents Simular

Simular185 subscribers

State-of-the-Art Performance

Agent S2 w. Claude 3.7 + UI-Tars on the OSWorld Benchmark

(Agent S2 w. Claude 3.7 + UI-Tars on the OSWorld Benchmark)

Agent S2 demonstrates superior computer and phone use, seen by significant advancements across key benchmark challenges.

‍

For computer use, Agent S2 delivers State-of-the-art results on OSWorld on both 15-step and 50-step evaluations (two most practical settings for real-world usage), proving that our agentic framework takes more precise actions and generates the best plan for a task, while being able to correct itself and improve over a long horizon. Notably, Agent S2 achieves 34.5% accuracy on 50-step evaluation, surpassing the previous SOTA (OpenAI CUA/Operator at 32.6%), demonstrating how agentic frameworks can scale beyond a single trained model.

For smartphone use, Agent S2 achieves 50% accuracy on AndroidWorld, surpassing previous SOTA (UI-TARS at 46.8%), demonstrating the generalization of agentic frameworks across different visual UI environments.

Subsequent to this blog post, we achieved stronger results on AndroidWorld while preparing our paper. We have updated this table to reflect the latest performance. Please refer to the paper for comprehensive details.

Why Modular Frameworks Matter: Inspiration from Human Brain

The human brain is a remarkable example of modular design—a network of specialized components working in unison. Different regions excel at distinct tasks: the left hemisphere drives analytical thinking, the right fuels creativity, while motor and sensory areas manage physical coordination. This modular structure, optimized for collaboration, inspires how we approach AI agent design for computer use.

At Simular, we believe the most effective AI agents should follow a similar principle—modular frameworks that seamlessly orchestrate diverse models, rather than relying on a single monolithic system. Our initial agent framework, Agent S, launched on October 11, 2024, embodies this vision. With experience-augmented hierarchical planning as the core, Agent S achieved better overall performance than any models and frameworks at the time.

‍

Our latest research further shows that a well-designed modular framework, even with suboptimal individual models, can outperform the best standalone model. Why? Because different models excel in different areas, each possessing unique strengths and weaknesses. A robust framework optimizes the orchestration among these modules, ensuring that each model contributes where it performs best, leading to superior overall outcomes. In the rapidly evolving landscape of foundation models, modularity is key. Our next generation agentic framework, Agent S2, achieves significantly better perception, planning and fine-grained control by virtue of its improved modularity and flexibility.

Agent S2: How it works

Agent S2 is built to handle complex digital tasks through a modular and scalable approach. Its framework emphasizes four key design principles:

Proactive Hierarchical Planning

‍

Agent S2 follows a natural task hierarchy, combining specialized models for low-level execution with generalized models for high-level planning. Low-level tasks, such as UI element selection or text highlighting, demand high precision and domain-specific expertise, while high-level tasks require broader adaptability and strategic oversight. In addition, a key advancement in Agent S2 is its shift from reactive to proactive planning. Instead of replanning only after encountering errors, which would require more steps to backtrack and may accrue more error, Agent S2 dynamically updates its plans after each subtask. This proactive approach improves adaptability to real-time changes, continuity from one subtask to the next, and optimality of future steps.

Visual Grounding for Precise Interaction

‍

Agent S2 achieves high-precision interaction with graphical user interfaces (GUIs) through specialized visual grounding models. Unlike its predecessor, which relied on accessibility trees for UI understanding, Agent S2 operates solely on raw screenshots as input, eliminating the need for structured accessibility data. By delegating visual understanding to dedicated models, Agent S2 can accurately locate and manipulate UI elements like buttons, text, images, and cells—enabling fine-grained control that was previously limited by accessibility constraints.

Agent-Computer Interface with Expert Modules

‍

Agent S2 improves its Agent-Computer Interface (ACI) by offloading complex, low-level tasks such as text highlighting to specialized expert modules. This reduces the cognitive load on the foundation models, allowing them to focus solely on high-level planning and strategic decision-making.

Agentic Memory Mechanism

‍

Agent S2 uses a continual learning memory mechanism that allows it to evolve with experience, improving efficiency over time. The experience from previously completed tasks are retained, allowing Agent S2 to recall prior actions and refine future strategies based on historical successes and failures. This adaptive learning capability enables Agent S2 to become more proficient with each application, creating a foundation for long-term adaptive intelligence and personalized automation.

This modular architecture also makes scaling and adaptation effortless. New modules powered by foundation or expert models can be easily integrated, removed, or swapped, allowing Agent S2 to rapidly adapt to new task domains with ease.

Agent S2 in Action

Computer Use

Download an image from Google Drive and use GIMP to compress it

Subtask 1: Launch GIMP\ \ 01. Click the "Activities" button. 02. Click the "Cancel" button. 03. Click the "X" button to close. 04. Click the "Activities" button. 05. Type "GIMP" in the search bar. 06. Click the GIMP application icon Subtask 2: Download image\ \ 07. Press Ctrl + Alt + T. 08. Type wget in the terminal Subtask 3: Open image in GIMP\ \ 09. Press Ctrl + O to open a file. 10. Click the File menu in GIMP. 11. Click the main GIMP window. 12. Click the File menu 13. Click Open... 14. Click the Downloads folder 15. Click the user folder 16. Click the file named 'uc?exp... 17. Click the Open button. Subtask 4: Resize image\ \ 18. Click the Image menu 19. Click Scale Image... 20. Type 2000 in the width input 21. Click the Scale button. Subtask 5: Export compressed image\ \ 22. Press Alt + F, then E to open... 23. Click Export As... 24. Click the Desktop folder 25. Type compressed.jpeg 26. Click the Export button. 27. Type 60 in the Quality field. 28. Click the Export button 29. Done.

Copy Image Into Doc ‍

Copy an image from GIMP to a LibreOffice Writer doc, and then export the doc

Subtask 1: Open GIMP\ \ 01. Open GIMP. Subtask 2: Open XCF file\ \ 02. Click the File menu in GIMP. 03. Press Ctrl + O to open a file. 04. Click Open... in the File menu. 05. Click the Desktop folder. 06. Click the Open button. Subtask 3: Open LibreOffice Writer\ \ 07. Open LibreOffice Writer. Subtask 4: Copy image in GIMP\ \ 08. Switch to GIMP. 09. Press Ctrl + A to select all. 10. Press Ctrl + C to copy the image. Subtask 5: Paste image in Writer\ \ 11. Switch to LibreOffice Writer. 12. Click the empty document area. 13. Press Ctrl + V to paste the image. Subtask 6: Save document as DOCX\ \ 14. Press Ctrl + Shift + S to open Save As. 15. Click the Desktop option in the sidebar. 16. Type image.docx in the Name field. 17. Click the file type dropdown menu. 18. Click Word 2007-365 (.docx) 19. Type Word 2007-365 (.docx) in the file type field. 20. Click the dropdown arrow 21. Scroll the file format list up. 22. Click Word 2007-365 (.docx) in the list. 23. Click the Save button. 24. Done.

Setup Web Extension ‍

Set up a web extension

Subtask 1: Click Create Button\ \ 01. Click the "Let's create" button Subtask 2: Configure Extension Settings\ \ 02. Type "happy-extension" in the and press Enter. 03. Type nothing in the input field and press Enter. 04. Type "v0.0.1" in the version input field and press Enter. 05. Type "0.0.1" in the version with the incorrect format and press Enter. 06. Click the "Yes" button to enable the background script. 07. Click the "No" button to disable the content script. 08. Click the "Yes" button to enable the browser action. 09. Click the "No" button to disable the page action. 10. Click the "No" button to disable extension options. Subtask 3: Navigate to Projects Directory\ \ 11. Click the Projects folder Subtask 4: Extract Extension Files\ \ 12. Click the Downloads folder. 13. Wait 2 seconds. 14. Switch to Chromium. 15. Click the download button in the browser’s toolbar. 16. Press F5 to refresh the page. 17. Switch back to Nautilus file manager. 18. Press Ctrl + L to focus on the address bar. 19. Type "/home/user/Downloads" and press Enter. 20. Click the happy-extension.zip file in the Downloads folder. 21. Press Ctrl + C to copy the file. 22. Click the Projects folder. 23. Click the Home button in the navigation bar. 24. Double-click the Projects folder in the main window. 25. Press Ctrl + V to paste the file. 26. Right-click the happy-extension.zip file 27. Click the "Extract Here" option. 28. Done.

Remove Video Subtitles

Remove subtitles from a video and export the new video

Subtask 1: Disable Subtitles\ \ 01. Click the Subtitle menu option in VLC's top menu bar. 02. Click Sub Track from the Subtitle dropdown menu. 03. Click Disable in the Sub Track submenu to turn off subtitles. Subtask 2: Open Codec Information\ \ 04. Click the Tools menu in VLC's top menu bar. 05. Press Ctrl + J to open Codec Information. 06. Click Codec Information from the Tools dropdown menu. 07. Click the Codec tab in the Current Media Information window. Subtask 3: Access Codec Tab\ \ 08. Click the Codec tab in the Current Media Information window. 09.Click the Codec tab 10. Press Tab three times, then Enter to navigate the tab. 11. Click Cancel on the Software Updater dialog. 12. Open the Tools menu in VLC. 13. Press Ctrl + J again to return to Codec Information. Subtask 4: Open Media Information\ \ 14. Press Ctrl + I to open the Media Information window. 15. Click Media Information in the Tools menu. Subtask 5: Access Metadata Tab\ \ 16. Click the Metadata tab at the top of the Current window. Subtask 6: Access Codec Tab\ \ 17. Click the Codec tab again in the Current window. Subtask 7: Close Media Info Window\ \ 18.Click the Close button at the bottom right Subtask 8: Open Terminal\ \ 19. Press Ctrl + Alt + T to open the terminal. Subtask 9: Extract subtitles using FFmpeg\ \ 20. Type the command ffmpeg -i video.mp4 -map 0:s:0 subtitles.srt in the terminal and press Enter. 21. Done.

Calculate Profit

Calculate the profit in a LibreOffice Calc sheet

Subtask 1: Insert New Column\ \ 01. Click the column header D 02. Click "Insert Columns Before" Subtask 2: Add Profit Column Header\ \ 03. Set cell D1 to "Profit". Subtask 3: Enter Profit Formula\ \ 04. Set cell D2 to "=B2-C2" Subtask 4: Copy Profit Formula to All Rows\ \ 05. Drag the fill handle from cell D2 down to D11. 06. Drag the fill handle from the bottom right of D2 to D11 07. Click cell D2 08. Drag the small fill handle to D11 09. Copy the profit formula in D2 to D11. 10. Click cell D2 to confirm 11. Press Ctrl + C to copy 12. Set cell values from D2 through D11 13. Done.

.webp)

Strikethrough Paragraph ‍

Strikethrough the last paragraph in a LibreOffice Writer doc

Subtask 1: Select Last Paragraph\ \ 01. Highlight text from "Each student will be responsible" to "comprises 30% of final grade". Subtask 2: Apply Strikethrough Formatting\ \ 02. Click the strikethrough button in the formatting toolbar. 03. Done.

.webp)

Agent S2 on your Smartphone

Fill forms

Task: Go to the new contact screen and enter the following details: First Name: Grace, Last Name: Taylor, Phone: 799-802-1530, Phone Label:

Work. Do NOT hit save.

01. Click the Phone app icon in the bottom dock. 02. Click the Contacts tab at the bottom right of the screen. 03. Click Create new contact at the top of the contacts list. 04. Type “Grace” in the First Name field. 05. Type “Taylor” in the Last Name field. 06. Click the Phone field below the Company field. 07. Type “799-802-1530” in the Phone field. 08. Click the Label field showing “Mobile” with a dropdown indicator. 09. Click “Work” in the dropdown menu to change the phone label. 10. Mark the task as done (without clicking Save).

Organize files

Task: Move the file holiday_photos.jpg from Podcasts within the sdk_gphone_x86_64 storage area to the DCIM within the same sdk_gphone_x86_64 storage area in the Android filesystem.

01. Swipe up in the middle of the screen to open the app drawer. 02. Click the Files app (blue folder icon in the fifth row). 03. Click the hamburger menu icon in the top left corner. 04. Click sdk_gphone64_arm64 to open main storage. 05. Click the Podcasts folder to access its contents. 06. Click the Images filter at the top of the screen. 07. Long-press on holiday_photos.jpg to select it. 08. Click the three-dot menu icon in the top right corner. 09. Click Move to… from the dropdown menu. 10. Click the hamburger menu icon in the top left corner. 11. Click sdk_gphone64_arm64 to access main storage. 12. Click the DCIM folder to set it as the destination. 13. Click the MOVE button to move the file. 14. Click sdk_gphone64_arm64 to return to main storage. 15. Click DCIM folder to verify the file is moved. 16. Confirm that holiday_photos.jpg is present in DCIM. 17. Mark the task as done.

What are the Key Features of a Computer Use Agent?

What functionalities does a computer use agent offer beyond basic automation?

A computer use agent goes beyond just automating tasks. It uses artificial intelligence to handle complex jobs and boost productivity. By integrating processes smoothly, users can automate workflows efficiently. AI capabilities help these agents analyze data, predict results, and adjust strategies, enhancing productivity.

How does modularity enhance the capabilities of a computer use agent?

Modularity improves a computer use agent by making it flexible and scalable. This setup allows for ongoing updates and customization to fit specific needs. Each module has its function, so users can upgrade parts without affecting the whole system. This adaptability helps agents keep up with changing technology, supporting growth and efficiency.

Can a computer use agent adapt to different operating systems and software applications?

Adaptability is vital for a computer use agent, ensuring it works well with varying operating systems and software. These agents integrate easily, maintaining performance on any platform, whether Windows, macOS, or Linux. Their broad compatibility means organizations can use them across diverse IT systems without facing issues.

How Does a Computer Use Agent Learn and Improve?

What learning mechanisms enable a computer use agent to adapt to user needs?

Computer use agents use machine learning and artificial intelligence to learn and adjust to user needs. They process large data sets to find patterns and make predictions. This helps them provide tailored solutions based on user behavior and preferences. By using feedback loops, they refine their operations for better accuracy over time. These systems effectively meet changing user demands with these methods.

How does a computer use agent handle unexpected situations or errors?

To manage unexpected situations or errors, computer use agents use strong error handling and automation. AI boosts their reliability by spotting and addressing anomalies quickly. These agents stay robust, preventing small issues from becoming serious problems. Automation in troubleshooting allows them to fix errors swiftly and keeps operations running smoothly even in unforeseen circumstances. This enhances user trust and system reliability.

What data privacy measures are implemented in a computer use agent?

Data privacy is crucial for computer use agents. They use strict security protocols to protect sensitive data and comply with regulations. Privacy measures include encrypting data during transmission and storage to guard against unauthorized access. Regular monitoring and updates of security systems help maintain data integrity. By focusing on data privacy, users can trust that their information is secure and handled properly.

What are the Practical Applications of Computer Use Agents?

Computer use agents, powered by artificial intelligence and automation, have a significant role in managing complex workflows. These advanced systems do more than just automate simple tasks. With their sophisticated reasoning abilities, they can handle intricate processes. Autonomous AI agents can organize multiple tasks, use resources efficiently, and keep workflows running smoothly. This technology is changing industries, helping businesses operate more efficiently and foster innovation with browser automation and AI workplace assistant.

Beyond automating simple tasks, what complex workflows can a computer use agent manage?

Computer use agents are deployed to handle complex workflows that need coordination and decision-making. These agents use artificial intelligence and automation to improve operations. Their advanced reasoning lets them assess situations, predict outcomes, and make decisions. This is useful in fields like logistics, finance, and healthcare that require quick adaptation to changing conditions.

How can a computer use agent improve productivity in specific professional contexts?

Using computer use agents in daily operations can greatly increase productivity and efficiency. In professional areas like project management or customer service, these agents automate regular tasks. This allows employees to focus on more strategic work. The innovative use of computer technology improves workflow and speeds up innovation, giving organizations a competitive edge.

What emerging technologies are integrated with advanced computer use agents?

The development of computer use agents is linked with advances in emerging technologies. Machine learning and foundation models, such as those from OpenAI, are crucial for these agents. These technologies allow agents to learn from data, adapt to new information, and improve over time. Continuous integration of new technology ensures that computer use agents remain highly effective across different domains.

Simular AI excels in this field by providing advanced solutions that use these technological developments. By keeping up with emerging technologies, Simular AI ensures its computer use agents are optimized for varied applications, offering great value to clients.

What are the Potential Limitations of Computer Use Agents?

What are the challenges in building truly reliable and trustworthy computer use agents?

Building reliable and trustworthy computer use agents involves several challenges. Integrating AI into these systems can create unexpected issues. This makes solid development protocols essential. It's important to ensure that automation aligns with human values and safety standards. To build trustworthy AI, we need to address biases, improve transparency, and perform thorough tests. Ongoing development and learning from real-world applications help enhance reliability.

How can the risk of errors and malfunctions be minimized in computer use agents?

Minimizing errors and malfunctions in computer use agents requires careful engineering practices and protocols. Advanced computing techniques should be used to detect and correct errors. Rigorous testing and simulation before deployment can help identify potential risks. Continuous monitoring after deployment allows for quick fixes. Effective risk management includes having fallback systems and regular software updates to fix vulnerabilities and boost operational stability.

What are the ethical considerations regarding the development and deployment of computer use agents?

When developing and deploying computer use agents, ethics are a major consideration. Protecting user data privacy and security is crucial. Developers must promote responsible AI use by setting transparent guidelines to prevent misuse or bias. Ethical considerations include assessing AI's impact on society and addressing job displacement concerns. Continuous dialogue among stakeholders is important for responsible development that aligns with societal values and legal standards.

How Can I Get Started with a Computer Use Agent?

What are the available options for accessing and utilizing computer use agent technology?

You have several options for accessing and using a computer use agent. Some popular software platforms include open-source solutions, which offer customization, and products from companies like Microsoft and Google. These platforms provide APIs, allowing seamless integration into your existing systems. Choosing the right one depends on your needs and how well it fits into your current technology setup.

What factors should be considered when choosing a computer use agent solution?

When picking a computer use agent, consider the following:

Functionality: Make sure the agent fulfills your operational needs.
Integration: Verify compatibility with your systems, such as OpenAI API.
Cost: Look at both initial and ongoing costs.
Support and Security: Check the availability of support and the strength of security features.
User Interface: Ensure it's easy to use and intuitive for users.

By assessing these points, you can find a solution that meets your organization's goals and technical needs.

Where can I find resources and support for learning more about and using computer use agents?

To learn more about using computer use agents, consider these resources:

Tutorials and Documentation: You can find guides on GitHub and the official websites of platforms like Microsoft Azure and OpenAI.
Community Forums: Join forums to gain insights and practical knowledge from other users.
Training Programs: Participate in training sessions by providers or external educators for hands-on experience.
Learning Resources: Many online platforms offer courses and materials focusing on different aspects of computer use agents.

These resources will help you fully utilize your chosen computer use agent technology.

What are the benefits of using an autonomous agent for computer use?

Autonomous agents can automate and optimize computer use, improving efficiency. They help manage complex tasks, reduce human errors, and increase productivity by adapting to specific user needs.

How can a computer agent improve cyberattack management?

A computer agent analyzes suspicious behavior in real time to reinforce security against cyberattacks. It monitors networks, detects anomalies, and initiates quick responses to minimize risks.

What is the role of artificial intelligence in computer foundation models?

Artificial intelligence enhances foundation models by improving natural language understanding, which facilitates processing and analyzing complex data. It supports the development of innovative and user-friendly solutions.

Why is it important to consider user interfaces when implementing computer agents?

User interfaces ensure smooth interaction between computer agents and users. They make functionalities accessible and understandable, thereby improving user experience and operational efficiency.

How do navigation agents affect computer software usage?

Navigation agents simplify software interaction by guiding users through complex processes. They facilitate task customization and automation, optimizing software use.

What role do usage agents play in improving organizational workflows?

Usage agents automate redundant tasks and integrate various tools to create efficient workflows. They enable smooth operation management, reduce response times, and boost productivity in organizations.

How can computer agent usage strategies adapt to current work environments?

Computer agent strategies adapt by integrating emerging technologies like federated learning and multimodal models. These approaches make systems more flexible and responsive to changing work environment dynamics.

What are advanced language models, and how do they influence interaction with computer agents?

Advanced language models, such as large language models, enhance computer agents' ability to understand and generate human text. This leads to more natural and effective interactions, enriching the overall user experience.

Key Insights

We offer advanced solutions for computer use agents in regions such as California, Canada, Florida, New York, Texas, the United Kingdom, and Washington.
Our platform utilizes state-of-the-art computer use software to enhance computer utilization, positioning it as a leading intelligent system.
Experience seamless interaction with user agents in GUI and UI design, which provides innovative navigation and autonomous agents.
Work alongside industry leaders like Bill Gates and Sam Altman on groundbreaking projects in AI technology and sandbox environments.
Engage with foundation models and large language models ( LLMs) for top performance, backed by notable researchers like Ilya Sutskever.
Gain insights into AI with resources from IEEE, MIT Technology Review, and publications on trustworthy AI practices.
Prioritize secure computing environments to shield against cyber attacks, using tools and protocols for strong data integrity.
Innovate with AI-driven solutions through platforms like OpenAI's ChatGPT and DALL-E, supported by pioneers such as Kai-Fu Lee.
Access our suite of tools, including apps, plugins, and AI technologies that transform user interaction.
Improve efficiency with AIOps for incident management and federated learning, optimizing organizational strategies and workflows.
Enhance emotional well-being and decision-making through effective AI applications in business scenarios.
Use advanced reasoning and adaptive systems for solving complex problems and strategic initiatives.
Stay updated with the latest developments in AI, focusing on practical applications and emerging technologies.
Apply machine learning across sectors, ensuring compliance and ethical standards in deployment.
Explore uses of GPT and AGI in real-world situations through demonstrations, evaluations, and scholarly citations.
Navigate the AI landscape with expertise, using resources like stargate datasets, deepfake prevention, and open-source contributions.
Integrate AI seamlessly into existing agentic frameworks with reference implementations and best practice guidelines.
Encourage AI innovation in computing, fostering community engagement and collaborative growth in tech.
Enhance potential with AI-powered solutions for data analysis, automation, and process optimization across industries.
Connect with professionals on platforms like LinkedIn to explore trends, job opportunities, and advancements in AI.
Optimize your digital experience with tools designed for efficient browsing, communication, and information management.
Maintain robust cybersecurity measures and safe data practices in all AI deployments and integrations.

Ready to use your computer in a Simular way?

Shares and organize your memory, and personalize your tasks.

Try Sai