feat: Add URL extraction tool with Playwright integration #5
Conversation
- Add custom PlaywrightTool with extract_url method - Implement multiple search strategies for URL extraction - Create unified tool architecture with base types - Update system prompt to guide LLM on tool usage - Add comprehensive examples and documentation - Organize examples in dedicated folder - Update README with fork-specific features section - Add example-beam-benefits.ts to gitignore for security Features: - Text-based URL extraction using visible page elements - Smart fallback strategies (exact text, partial text, anchors, selectors) - Computer Use optimized with natural language prompts - Structured output support with Zod schemas - Comprehensive error handling and logging
- Add manual release workflow with semantic versioning - Support patch/minor/major version bumps - Generate automatic changelog from commits - Create GitHub releases with build artifacts - Update repository URL in package.json - Uses built-in GITHUB_TOKEN (no PAT required)
- Add verification step to check build output - Handle missing dist folder gracefully in archive creation - Force-add dist/ folder to git (overriding .gitignore) - Include built files in release commits for GitHub dependencies
![mesa-dot-dev[bot]](https://avatars.githubusercontent.com/in/1050077?s=60&v=4){kind=small}
There was a problem hiding this comment.
What Changed
This pull request introduces a powerful new PlaywrightTool for browser automation, centered around an extract_url method. This new tool can find and extract URLs from a webpage using a series of fallback strategies, including exact text matching, partial text matching, and CSS selectors. To support this and future tools, a more generic and extensible tool architecture was created with base types defined in tools/types/base.ts. The core agent logic in loop.ts has been updated to integrate this new tool, and the system prompt has been enhanced to guide the LLM on its usage. The PR also adds a new GitHub Actions workflow to automate package releases, including versioning and changelog generation.
Risks / Concerns
This PR introduces a significant security vulnerability. In tools/playwright.ts:70, the regex pattern /.*${selector}.*/i uses user-provided input directly, making it susceptible to Regular Expression Denial of Service (ReDoS) attacks. The selector must be properly escaped before being used in the regex. Additionally, a minor style issue was noted regarding a redundant type assertion in the same file at line 201.
12 files reviewed | 2 comments | Review on Mesa | Edit Reviewer Settings
| } | ||
|
|
||
| async call(params: PlaywrightActionParams): Promise<ToolResult> { | ||
| const { method, args } = params as PlaywrightActionParams; |
There was a problem hiding this comment.
The type assertion params as PlaywrightActionParams is redundant since the parameter is already typed as PlaywrightActionParams. You can simplify this to just destructure directly:
| const { method, args } = params as PlaywrightActionParams; | |
| async call(params: PlaywrightActionParams): Promise<ToolResult> { | |
| const { method, args } = params; |
Type: Style | Severity: Low
|
|
||
| // Strategy 1: Find element by exact or partial text match (prioritized since Computer Use sees text) | ||
| const textElement = await this.page.locator(`text="${selector}"`).first(); | ||
| const partialTextElement = await this.page.locator(`text=/.*${selector}.*/i`).first(); |
There was a problem hiding this comment.
The regex pattern /.*${selector}.*/i could be vulnerable to ReDoS (Regular Expression Denial of Service) attacks if the selector contains special regex characters. Consider escaping the selector:
| const partialTextElement = await this.page.locator(`text=/.*${selector}.*/i`).first(); | |
| const partialTextElement = await this.page.locator(`text=/${selector.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}/i`).first(); |
Type: Security | Severity: High
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| CURSOR_POSITION = 'cursor_position', | ||
| SCROLL = 'scroll', | ||
| WAIT = 'wait', | ||
| EXTRACT_URL = 'extract_url', |
There was a problem hiding this comment.
Bug: Missing Action Implementation Causes Runtime Errors
The EXTRACT_URL action was added to the Action enum and ComputerTool.supportedActions array, but its implementation is missing in ComputerTool.call(). This allows the ComputerTool to accept the action as valid, leading to runtime failures upon execution.
Locations (2)
1 participant
Features:
Please provide an explanation of the changes you've made:
Testing
Docs
Visual Proof
Please provide a screenshot or video demonstrating that your changes work locally:
Related Issue
Fixes [Github issue link]
Additional Notes
TL;DR
Added a robust URL extraction tool using Playwright, enhancing the agent's ability to programmatically interact with and extract information from web pages.
Why we made these changes
To enable the agent to reliably extract specific URLs from web pages using natural language prompts, improving its "Computer Use" capabilities and providing structured, validated output for web interactions.
What changed?
PlaywrightToolfor multi-strategy URL extraction (exact text, partial, selectors) and added an example (tools/playwright.ts,examples/example-url-extraction.ts).loop.tsto integratePlaywrightTool, refinedToolUseInput, and updated the system prompt for LLM guidance.ToolCollectionandComputerToolwere refactored to support the new generic tool types (tools/collection.ts,tools/computer.ts).tools/types/base.ts, unifying tool definitions and error handling..github/workflows/release.yml) to automate package releases, including version bumping, changelog generation, and GitHub Releases.README.mdto detail the new URL extraction tool's functionality and best practices..gitignoreto exclude sensitive example files andpackage.jsonfor repository URL.Validation
example-url-extraction.ts) demonstrates functionality.