feat(agui): add support for frontend tool calls and update the examples #762
Conversation
|
|
Summary of ChangesHello @TornadoDragon, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the AgentScope AG-UI by enabling agents to interact with the client-side browser environment through a new frontend tool calling mechanism. This expands the capabilities of agents, allowing them to perform actions like retrieving browser information, accessing local storage, or reading the clipboard, directly from the user's interface. The changes include new client-side JavaScript for tool management and execution, UI updates to display these tools, and backend logic to integrate these frontend tools seamlessly with the agent's existing toolkit. Highlights
Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces frontend tool call support, a significant feature. The implementation is mostly solid, but there are a few critical bugs in the JavaScript logic for handling tool results that would prevent the feature from working correctly. Additionally, there's significant code duplication in the main HTML file that should be refactored to improve maintainability. I've also noted some smaller issues regarding API clarity in both the Java and JavaScript code. Addressing these points will make the new feature more robust and the codebase easier to maintain.
| requestMessages.push({ | ||
| id: 'tool-result-' + Date.now() + '-' + Math.random(), | ||
| role: 'tool', | ||
| content: result.content, |
There was a problem hiding this comment.
There is a bug here when constructing the tool result message. The result object from pendingToolResults has an output property containing the tool's return value, not a content property. This will result in content being undefined in the request. It should be JSON.stringify(result.output) to match the implementation in sendToolResultMessage.
| content: result.content, | |
| content: JSON.stringify(result.output), |
| id: 'tool-result-' + Date.now() + '-' + Math.random(), | ||
| role: 'tool', | ||
| content: JSON.stringify(result.output), | ||
| toolCallId: result.toolCallId |
There was a problem hiding this comment.
The property name for the tool call ID should be tool_call_id to match the snake_case convention expected by the backend and defined in the AguiMessage model. Using toolCallId will likely cause deserialization issues on the server.
| toolCallId: result.toolCallId | |
| tool_call_id: result.toolCallId |
| onToolCallEnd: (toolCallId) => { | ||
| // Tool call completed | ||
| // Tool call completed - if it's a frontend tool, execute it | ||
| const pendingCall = frontendTools.getPendingCall(toolCallId); | ||
| if (pendingCall) { | ||
| executeFrontendTool(pendingCall.toolCallId, pendingCall.toolName, pendingCall.parsedArgs); | ||
| } | ||
| }, |
There was a problem hiding this comment.
The onToolCallEnd callback now executes the frontend tool. However, the sendToolResultMessage function, which is called after tool execution, has significant code duplication with the sendMessage function. Both functions set up and execute a client.run call with nearly identical callbacks. This duplication makes the code harder to maintain.
Consider refactoring the client.run logic into a shared function that can be called by both sendMessage and sendToolResultMessage to avoid repeating the same setup and event handling logic.
| case FRONTEND_ONLY -> { | ||
|
|
||
| // remove all tool | ||
| toolkit.removeToolGroups(toolkit.getActiveGroups()); |
There was a problem hiding this comment.
In FRONTEND_ONLY mode, the goal is to replace all of the agent's tools with those from the frontend. However, toolkit.removeToolGroups(toolkit.getActiveGroups()) only removes active tool groups. Any tools in inactive groups will remain in the toolkit's registry (though their groups will be empty after the next line). This could lead to unexpected behavior and doesn't fully enforce the FRONTEND_ONLY policy.
To correctly clear all tools, you should ensure all groups are removed, not just the active ones. The Toolkit API may need to be extended to support this cleanly (e.g., with a clear() or getAllGroupNames() method).
| * @param groupName Group name | ||
| * @param toolName Tool name |
There was a problem hiding this comment.
The order of @param tags in the Javadoc does not match the method signature's parameter order. For better readability and consistency, the Javadoc should reflect the (String toolName, String groupName) order.
| * @param groupName Group name | |
| * @param toolName Tool name | |
| * @param toolName Tool name | |
| * @param groupName Group name |
| setTimeout(() => { | ||
| if (!isRunning) { | ||
| sendToolResultMessage(); | ||
| } | ||
| }, 500); |
There was a problem hiding this comment.
Using setTimeout with a fixed delay of 500ms to trigger sending tool results is a code smell. It introduces an arbitrary delay and can be unreliable. If the goal is to batch results from multiple tool calls, there might be a more deterministic approach. For example, you could wait for all tool calls in a given turn to complete before sending the results back. The current implementation might be sensitive to timing issues.
| let requestMessages = [...messageHistory]; | ||
| pendingResults.forEach(result => { | ||
| requestMessages.push({ | ||
| id: 'tool-result-' + Date.now() + '-' + Math.random(), |
There was a problem hiding this comment.
Using Math.random() to generate part of a message ID is not a robust way to ensure uniqueness, as collisions are possible, although unlikely in this specific context. For a more production-ready example, consider using a library that generates UUIDs (e.g., crypto.randomUUID() which is available in modern browsers/Node.js) or a simple incrementing counter to guarantee unique IDs within the session.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
| * @param toolName Tool name | ||
| * @param groupName Group name | ||
| */ | ||
| public void addToolToGroup(String toolName, String groupName) { |
2 participants
AgentScope-Java Version
1.0.9-SNAPSHOT
Description
Add support for frontend tool calls in agui and update the examples.
Checklist
Please check the following items before code is ready to be reviewed.
mvn spotless:applymvn test)