Web Autonomous AI Agents

The next AI revolution

Author

Alexandre HOTTON
November 24, 2024

We're testing a new "deep dive" format—don't hesitate to share your feedback or leave a comment! We didn’t invent anything here but aimed to highlight the key points from this great video: https://www.youtube.com/watch?v=j4bdkWYNvIY.

As with any technology, we strongly encourage you to explore and experiment with it yourself. If you're curious about autonomous web agents, we highly recommend trying out this tool: https://github.com/corbt/agent.exe?ref=futuretools.io.

Now, let’s dive into the subject! 😊

Introduction

Large Language Models (LLMs) have revolutionized AI with groundbreaking capabilities like zero-shot reasoning, in-context learning and many more. These successes naturally lead to the question: what's next?

One exciting frontier is Autonomous AI Agents, systems capable of performing tasks we currently handle manually, such as navigating the web, setting up technical infrastructure, or organizing information. The potential is immense—these agents could streamline and automate routine tasks, saving significant time and effort.

Today, we'll focus on web agents, specialized autonomous systems designed to crawl the web and perform specific tasks. These agents require a deep understanding of HTML structures, text, and even visual elements like images.

For example, using GPT-4 and a specialized approach, a web agent could handle tasks like:

  1. Find a good Thai restaurant in Pittsburgh: Navigate to a page for Thai restaurants with at least 200 reviews and an average rating of 4.3 stars or higher. Select the one with the highest rating.

  2. Make a reservation: Book a table at Pusadee’s Garden for two people on the earliest available dinner date, using the name "JY Koh" and phone number "650-555-5555."

Here is a video from Anthropic showcasing how Claude is used as an autonomous web agent :

Core Concepts: How Autonomous Web AI Agents Work

Autonomous Web AI Agents function using a Partially Observable Markov Decision Process (POMDP) framework. Let’s break this down:

1. Environment Representation

The environment is represented by four components: State (S), Actions (A), Observations (O), and Transitions (T).

  • State (S):
    This represents the current condition of the web environment, including:

    • The structure and content of the webpage, such as its HTML and DOM (Document Object Model).

    • The visual layout of the page as seen by users.

      DOM (Document Object Model): A tree-like structure representing all elements of a webpage (e.g., buttons, images, and text). It allows the agent to interact with specific parts of the webpage programmatically.

  • Actions (A):
    Actions are the possible operations the agent can perform on the webpage. Examples include:

    • click[elem]: Clicking a specific element, like a button.

    • type[elem][text]: Typing text into an input field.

    • scroll[up/down]: Scrolling up or down on a page.

    • goto[url]: Navigating to a new webpage.

  • Observations (O):
    Since the agent cannot "see" the entire state of the webpage directly, it relies on observations such as:

    • Visual Information: Screenshots of the webpage.

    • HTML/DOM Structure: The webpage's underlying structure.

    • Set-of-Marks (SoM): Visual overlays that highlight important parts of the webpage (e.g., numbering buttons or text fields) to help the agent ground its actions.

  • Transition (T):
    This describes how the environment changes when an action is performed. For example:

    • Clicking a "Next Page" button might load a new webpage.

    • Formally, this is written as:
      "The next state depends on the current state and the action taken."
      If an agent is on a login page (current state) and enters credentials (action), the next state might be the user dashboard.

2. Reward Mechanism

The agent learns to optimize its actions using a reward system:

  • Reward for Correct Actions: The agent earns a reward (e.g., a value of 1) when it performs the correct action and completes the task.

  • No Reward for Incorrect Actions: The agent receives no reward (a value of 0) for incorrect actions or failure to complete the task.

For example:

  • If the task is "log in to the website," the agent is rewarded for correctly entering the username and password.

  • If it clicks the wrong button or leaves the page, it gets no reward.

3. Observations: Visual Language Models as Web Agents

The agent uses Visual Language Models (VLMs) to process observations from the webpage. These models are enhanced with Set-of-Marks (SoM):

  • Set-of-Marks (SoM): SoM overlays highlight or label actionable elements, such as clickable buttons, input fields, or menu items. For example:

    • A button might be labeled "1" to indicate it is a clickable element.

    • This helps the agent visually align actions with specific parts of the webpage.

The observations provided to the agent include:

  • HTML/DOM Data: The structure of the webpage.

  • Screenshots with SoM Markers: Screenshots where key elements are highlighted for easier interaction.

4. Decision-Making: Multimodal LLM with Chain-of-Thought Reasoning

The agent decides its actions using a Multimodal Large Language Model (LLM) (like GPT-4V). This model processes both visual and text-based inputs to generate logical action sequences. The agent employs Chain-of-Thought (CoT) reasoning to break tasks into smaller steps.

Example of Chain-of-Thought Reasoning:

  1. Locate a Thai restaurant on the webpage.

  2. Check for one with more than 200 reviews and a rating of at least 4.3.

  3. Select the highest-rated restaurant.

  4. Navigate to the reservation page.

  5. Enter the reservation details.

This step-by-step reasoning ensures the agent can handle complex, multi-step tasks efficiently.

5. Action Execution

After deciding the next step, the agent performs the corresponding action from its predefined set of possible operations.

Benchmark and Accuracy of Autonomous Web AI Agents

Benchmarks are critical for evaluating the performance of autonomous web AI agents. They provide controlled environments to test, compare, and iterate on models, ensuring progress toward real-world usability. A robust benchmark highlights the challenges agents face and sets clear targets for improvement. WebArena and its successor, VisualWebArena, exemplify this evolution, pushing the boundaries of agent capabilities from text-only to multimodal understanding.

1. From WebArena to VisualWebArena: Key Differences

WebArena:

  • Environment: A realistic web simulation with open-source re-implementations of popular websites (e.g., shopping platforms, Reddit forums).

  • Challenges:

    • Messy Input: HTML pages are highly complex, often exceeding 100k tokens, including compressed JavaScript and nested structures.

    • Text-only Tasks: Agents must rely solely on HTML and text, which limits their ability to interpret layouts or visual cues.

  • Performance:

    • Human Success Rate: ~78%.

    • LLM-based Agent Success Rate: ~14%, highlighting significant limitations when working with messy, text-heavy inputs.

  • Limitations: This benchmark assumes that all tasks can be achieved through text parsing alone, which fails to mimic realistic web navigation scenarios involving visual elements.

VisualWebArena:

  • Improvement: Incorporates multimodal agents capable of processing both textual and visual inputs.

    • Leverages HTML/DOM representations and visual screenshots with Set-of-Marks (SoM) overlays to improve grounding.

    • Tasks now require agents to interpret page layouts visually, making them more aligned with real-world web interactions.

  • Benefits: By combining textual and visual understanding, agents in VisualWebArena can tackle tasks that are more complex and realistic compared to WebArena.

2. Results

Challenges and Limitations in Autonomous Web AI Agents

While significant progress has been made in developing autonomous web AI agents, several challenges remain that explain why their performance lags behind human-level capabilities. These challenges stem from limitations in reasoning, planning, visual processing, and compounding errors over multiple steps. Below is a breakdown of the key issues:

1. Failures in Long-Horizon Reasoning and Planning

One of the fundamental difficulties lies in managing tasks that require multiple steps, especially when those steps depend on each other.

  • Looping Behavior: Agents often oscillate between two webpages, getting stuck in a loop rather than progressing toward task completion.

  • Undoing Correct Actions: In some cases, agents perform the correct steps but then undo their progress due to a lack of persistent task memory.

  • Premature Stopping: Agents tend to halt exploration too early, failing to gather all necessary information or complete the full task.

These issues arise because agents struggle to maintain a coherent plan across the entire sequence of actions, particularly in complex environments where new information continually updates the task requirements.

2. Failures in Visual Processing

Despite improvements with multimodal inputs, visual reasoning and processing remain significant hurdles for agents.

  • Clicking the Wrong Item: Agents frequently misinterpret visual elements, clicking on incorrect buttons or links due to misaligned spatial reasoning.

  • Difficulty Identifying Specific Items: On visually complex webpages, agents struggle to locate specific elements, such as products or fields, among cluttered interfaces.

  • Spatial Reasoning Challenges: Agents often fail tasks requiring spatial understanding, such as determining “the prices of products in the first row” or differentiating between rows and columns.

These failures highlight the need for stronger integration of visual and textual information, particularly when dealing with spatially demanding tasks.

3. What’s Missing?

Several capabilities are still lacking, which prevents agents from achieving robust performance:

  1. Long-Horizon Reasoning and Planning:

    • Agents need better mechanisms for reasoning over multiple steps and coordinating actions over long horizons.

    • This includes the ability to plan, execute, and adapt to new information dynamically.

  2. Parallel Execution and Confirmation:

    • Agents should be able to explore multiple pathways in parallel, make decisions across multiple instances, and clarify ambiguous instructions with users.

  3. Improved Vision-Language-Code Models:

    • Existing models lack strong integration between vision, language, and code understanding, which is essential for navigating and interacting with complex webpages.

  4. Level of Abstraction:

    • Determining the optimal level of abstraction for agent inputs is critical. Whether agents should focus on raw HTML, screenshots, APIs, or a combination of these remains an open question.

4. Why Is This Hard?

The challenges described above are amplified by the exponential compounding of errors during multi-step tasks. Here’s why this problem persists:

  • Error Compounding:

  • For example, if an agent has a 90% accuracy per step and a task requires 5 steps, the probability of completing the task correctly drops to just 59%:

  • Implications of Error Propagation:

    • Small mistakes in earlier steps often lead to catastrophic failures later in the task. For instance, clicking the wrong item on a webpage can redirect the agent to an entirely irrelevant context, making recovery difficult.

    • Unlike humans, agents lack intuition to backtrack effectively or recognize when they’ve gone off track.

5. Summary of Challenges

  • Reasoning and Planning: Agents lack robust long-horizon reasoning, leading to premature stopping, looping behaviors, and undone progress.

  • Visual Processing: Significant gaps remain in visual comprehension, particularly in spatial reasoning and identifying specific elements on complex pages.

  • Error Propagation: The compounding effect of errors over multi-step tasks drastically reduces success rates.

  • Missing Capabilities: Better integration of reasoning, visual understanding, and abstraction (HTML/screenshots/APIs) is required to improve agent reliability.

Real-World Applications of Autonomous Web Agents

Autonomous web agents are transforming various industries by automating complex tasks and enhancing operational efficiency. Notable applications include:

1. Automated Website Testing

In software development, autonomous web agents streamline the testing process by navigating websites, executing user interactions, and identifying issues without human intervention. Tools like Kaptivate exemplify this approach, ensuring that web applications function correctly and efficiently.

2. Robotic Process Automation (RPA)

Autonomous web agents are integral to RPA, automating repetitive tasks such as data entry, form submissions, and information retrieval across web platforms. By mimicking human interactions with web interfaces, these agents enhance productivity and reduce the potential for errors.

3. Intelligent Web Scraping and Data Extraction

These agents can autonomously browse websites to collect and analyze data, facilitating tasks like market research, price monitoring, and content aggregation. Their ability to adapt to dynamic web environments makes them valuable tools for businesses seeking to leverage web-based information.

Future Directions

To address these challenges, research should focus on:

  1. Developing stronger multimodal models that seamlessly integrate vision, language, and code understanding.

  2. Creating frameworks for multi-step planning and execution that mitigate error propagation through dynamic correction mechanisms.

  3. Exploring the optimal abstraction level for web interactions to balance complexity and efficiency.

  4. Designing agents capable of asking for clarifications or confirmations to reduce errors and ambiguity during tasks.

By addressing these gaps, we can improve the accuracy and reliability of autonomous web AI agents, enabling them to perform complex tasks on par with human users.

If you want to go further, feel free to study the followings :

Reply

or to participate.