Skip to content

new features for click by rect box;semantic wait #41

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rcholic merged 3 commits into from
Dec 22, 2025
Merged

new features for click by rect box;semantic wait #41

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
38c465f
new features for click by rect box;semantic wait
rcholic Dec 22, 2025
fad9fb4
flexible wait
rcholic Dec 22, 2025
a3895dc
updated readme
rcholic Dec 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 40 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -195,6 +195,7 @@ const firstRow = query(snap, 'bbox.y<600');

### Actions - Interact with Elements
- **`click(browser, elementId)`** - Click element by ID
- **`clickRect(browser, rect)`** - Click at center of rectangle (coordinate-based)
- **`typeText(browser, elementId, text)`** - Type into input fields
- **`press(browser, key)`** - Press keyboard keys (Enter, Escape, Tab, etc.)

Expand All @@ -209,18 +210,55 @@ console.log(`Duration: ${result.duration_ms}ms`);
console.log(`URL changed: ${result.url_changed}`);
```

**Coordinate-based clicking:**
```typescript
import { clickRect } from './src';

// Click at center of rectangle (x, y, width, height)
await clickRect(browser, { x: 100, y: 200, w: 50, h: 30 });

// With visual highlight (default: red border for 2 seconds)
await clickRect(browser, { x: 100, y: 200, w: 50, h: 30 }, true, 2.0);

// Using element's bounding box
const snap = await snapshot(browser);
const element = find(snap, 'role=button');
if (element) {
await clickRect(browser, {
x: element.bbox.x,
y: element.bbox.y,
w: element.bbox.width,
h: element.bbox.height
});
}
```

### Wait & Assertions
- **`waitFor(browser, selector, timeout?)`** - Wait for element to appear
- **`waitFor(browser, selector, timeout?, interval?, useApi?)`** - Wait for element to appear
- **`expect(browser, selector)`** - Assertion helper with fluent API

**Examples:**
```typescript
// Wait for element
// Wait for element (auto-detects optimal interval based on API usage)
const result = await waitFor(browser, 'role=button text="Submit"', 10000);
if (result.found) {
console.log(`Found after ${result.duration_ms}ms`);
}

// Use local extension with fast polling (250ms interval)
const result = await waitFor(browser, 'role=button', 5000, undefined, false);

// Use remote API with network-friendly polling (1500ms interval)
const result = await waitFor(browser, 'role=button', 5000, undefined, true);

// Custom interval override
const result = await waitFor(browser, 'role=button', 5000, 500, false);

// Semantic wait conditions
await waitFor(browser, 'clickable=true', 5000); // Wait for clickable element
await waitFor(browser, 'importance>100', 5000); // Wait for important element
await waitFor(browser, 'role=link visible=true', 5000); // Wait for visible link

// Assertions
await expect(browser, 'role=button text="Submit"').toExist(5000);
await expect(browser, 'role=heading').toBeVisible();
Expand Down
99 changes: 99 additions & 0 deletions examples/click-rect-demo.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
/**
* Example: Using clickRect for coordinate-based clicking with visual feedback
*/

import { SentienceBrowser, snapshot, find, clickRect, BBox } from '../src/index';

async function main() {
// Get API key from environment variable (optional - uses free tier if not set)
const apiKey = process.env.SENTIENCE_API_KEY as string | undefined;

const browser = new SentienceBrowser(apiKey, undefined, false);

try {
await browser.start();

// Navigate to example.com
await browser.getPage().goto('https://example.com');
await browser.getPage().waitForLoadState('networkidle');

console.log('=== clickRect Demo ===\n');

// Example 1: Click using rect object
console.log('1. Clicking at specific coordinates (100, 100) with size 50x30');
console.log(' (You should see a red border highlight for 2 seconds)');
let result = await clickRect(browser, { x: 100, y: 100, w: 50, h: 30 });
console.log(` Result: success=${result.success}, outcome=${result.outcome}`);
console.log(` Duration: ${result.duration_ms}ms\n`);

// Wait a bit
await browser.getPage().waitForTimeout(1000);

// Example 2: Click using element's bbox
console.log('2. Clicking using element\'s bounding box');
const snap = await snapshot(browser);
const link = find(snap, 'role=link');

if (link) {
console.log(` Found link: '${link.text}' at (${link.bbox.x}, ${link.bbox.y})`);
console.log(' Clicking at center of element\'s bbox...');
result = await clickRect(browser, {
x: link.bbox.x,
y: link.bbox.y,
w: link.bbox.width,
h: link.bbox.height
});
console.log(` Result: success=${result.success}, outcome=${result.outcome}`);
console.log(` URL changed: ${result.url_changed}\n`);

// Navigate back if needed
if (result.url_changed) {
await browser.getPage().goto('https://example.com');
await browser.getPage().waitForLoadState('networkidle');
}
}

// Example 3: Click without highlight (for headless/CI)
console.log('3. Clicking without visual highlight');
result = await clickRect(browser, { x: 200, y: 200, w: 40, h: 20 }, false);
console.log(` Result: success=${result.success}\n`);

// Example 4: Custom highlight duration
console.log('4. Clicking with custom highlight duration (3 seconds)');
result = await clickRect(browser, { x: 300, y: 300, w: 60, h: 40 }, true, 3.0);
console.log(` Result: success=${result.success}`);
console.log(' (Red border should stay visible for 3 seconds)\n');

// Example 5: Click with snapshot capture
console.log('5. Clicking and capturing snapshot after action');
result = await clickRect(
browser,
{ x: 150, y: 150, w: 50, h: 30 },
true,
2.0,
true
);
if (result.snapshot_after) {
console.log(` Snapshot captured: ${result.snapshot_after.elements.length} elements found`);
console.log(` URL: ${result.snapshot_after.url}\n`);
}

// Example 6: Using BBox object
console.log('6. Clicking using BBox object');
const bbox: BBox = { x: 250, y: 250, width: 45, height: 25 };
result = await clickRect(browser, bbox);
console.log(` Result: success=${result.success}\n`);

console.log('✅ clickRect demo complete!');
console.log('\nNote: clickRect uses Playwright\'s native mouse.click() for realistic');
console.log('event simulation, triggering hover, focus, mousedown, mouseup sequences.');
} catch (e: any) {
console.error(`❌ Error: ${e.message}`);
console.error(e.stack);
} finally {
await browser.close();
}
}

main().catch(console.error);

141 changes: 141 additions & 0 deletions examples/semantic-wait-demo.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
/**
* Example: Semantic waitFor using query DSL
* Demonstrates waiting for elements using semantic selectors
*/

import { SentienceBrowser, snapshot, find, waitFor, click } from '../src/index';

async function main() {
// Get API key from environment variable (optional - uses free tier if not set)
const apiKey = process.env.SENTIENCE_API_KEY as string | undefined;

const browser = new SentienceBrowser(apiKey, undefined, false);

try {
await browser.start();

// Navigate to example.com
await browser.getPage().goto('https://example.com', { waitUntil: 'domcontentloaded' });

console.log('=== Semantic waitFor Demo ===\n');

// Example 1: Wait for element by role
console.log('1. Waiting for link element (role=link)');
let waitResult = await waitFor(browser, 'role=link', 5000);
if (waitResult.found) {
console.log(` ✅ Found after ${waitResult.duration_ms}ms`);
console.log(` Element: '${waitResult.element?.text}' (id: ${waitResult.element?.id})`);
} else {
console.log(` ❌ Not found (timeout: ${waitResult.timeout})`);
}
console.log();

// Example 2: Wait for element by role and text
console.log('2. Waiting for link with specific text');
waitResult = await waitFor(browser, 'role=link text~"Example"', 5000);
if (waitResult.found) {
console.log(` ✅ Found after ${waitResult.duration_ms}ms`);
console.log(` Element text: '${waitResult.element?.text}'`);
} else {
console.log(' ❌ Not found');
}
console.log();

// Example 3: Wait for clickable element
console.log('3. Waiting for clickable element');
waitResult = await waitFor(browser, 'clickable=true', 5000);
if (waitResult.found) {
console.log(` ✅ Found clickable element after ${waitResult.duration_ms}ms`);
console.log(` Role: ${waitResult.element?.role}`);
console.log(` Text: '${waitResult.element?.text}'`);
console.log(` Is clickable: ${waitResult.element?.visual_cues.is_clickable}`);
} else {
console.log(' ❌ Not found');
}
console.log();

// Example 4: Wait for element with importance threshold
console.log('4. Waiting for important element (importance > 100)');
waitResult = await waitFor(browser, 'importance>100', 5000);
if (waitResult.found) {
console.log(` ✅ Found important element after ${waitResult.duration_ms}ms`);
console.log(` Importance: ${waitResult.element?.importance}`);
console.log(` Role: ${waitResult.element?.role}`);
} else {
console.log(' ❌ Not found');
}
console.log();

// Example 5: Wait and then click
console.log('5. Wait for element, then click it');
waitResult = await waitFor(browser, 'role=link', 5000);
if (waitResult.found && waitResult.element) {
console.log(' ✅ Found element, clicking...');
const clickResult = await click(browser, waitResult.element.id);
console.log(` Click result: success=${clickResult.success}, outcome=${clickResult.outcome}`);
if (clickResult.url_changed) {
console.log(` ✅ Navigation occurred: ${browser.getPage().url()}`);
}
} else {
console.log(' ❌ Element not found, cannot click');
}
console.log();

// Example 6: Using local extension (fast polling)
console.log('6. Using local extension with auto-optimized interval');
console.log(' When useApi=false, interval auto-adjusts to 250ms (fast)');
waitResult = await waitFor(browser, 'role=link', 5000, undefined, false);
if (waitResult.found) {
console.log(` ✅ Found after ${waitResult.duration_ms}ms`);
console.log(' (Used local extension, polled every 250ms)');
}
console.log();

// Example 7: Using remote API (slower polling)
console.log('7. Using remote API with auto-optimized interval');
console.log(' When useApi=true, interval auto-adjusts to 1500ms (network-friendly)');
if (apiKey) {
waitResult = await waitFor(browser, 'role=link', 5000, undefined, true);
if (waitResult.found) {
console.log(` ✅ Found after ${waitResult.duration_ms}ms`);
console.log(' (Used remote API, polled every 1500ms)');
}
} else {
console.log(' ⚠️ Skipped (no API key set)');
}
console.log();

// Example 8: Custom interval override
console.log('8. Custom interval override (manual control)');
console.log(' You can still specify custom interval if needed');
waitResult = await waitFor(browser, 'role=link', 5000, 500, false);
if (waitResult.found) {
console.log(` ✅ Found after ${waitResult.duration_ms}ms`);
console.log(' (Custom interval: 500ms)');
}
console.log();

// Example 9: Wait for visible element (not occluded)
console.log('9. Waiting for visible element (not occluded)');
waitResult = await waitFor(browser, 'role=link visible=true', 5000);
if (waitResult.found) {
console.log(` ✅ Found visible element after ${waitResult.duration_ms}ms`);
console.log(` Is occluded: ${waitResult.element?.is_occluded}`);
console.log(` In viewport: ${waitResult.element?.in_viewport}`);
}
console.log();

console.log('✅ Semantic waitFor demo complete!');
console.log('\nNote: waitFor uses the semantic query DSL to find elements.');
console.log('This is more robust than CSS selectors because it understands');
console.log('the semantic meaning of elements (role, text, clickability, etc.).');
} catch (e: any) {
console.error(`❌ Error: ${e.message}`);
console.error(e.stack);
} finally {
await browser.close();
}
}

main().catch(console.error);

Loading