fix: enhance snapshot file handling and browser management

[Cyberlink404]

关联 Issue

主要改动

1. 增加一个 ignoreRobots 参数，跳过 robots 检测
2. 完善无效文件清理逻辑
3. 完善浏览器实例管理逻辑

界面截图

测试计划 (暂无)

检查清单

• /api/crawl success
• /api/snap success

[github-actions]

AIGNE Framework

[github-actions]

AIGNE Framework

Commits

Files that changed from the base of the PR and between 674bc5f and 2b2ac21 commits.

Files selected (5)

• blocklets/snap-kit/api/src/routes/index.ts (2)
• packages/crawler/src/crawler.ts (4)
• packages/crawler/src/metrics.ts (1)
• packages/crawler/src/puppeteer.ts (4)
• packages/crawler/src/services/snapshot.ts (2)

Files ignored due to filter (5)

• CHANGELOG.md
• blocklets/snap-kit/blocklet.yml
• package.json
• packages/crawler/package.json
• packages/middleware/package.json

Files not summarized due to errors (5)

• blocklets/snap-kit/api/src/routes/index.ts (nothing obtained from openai)
• packages/crawler/src/services/snapshot.ts (nothing obtained from openai)
• packages/crawler/src/puppeteer.ts (nothing obtained from openai)
• packages/crawler/src/crawler.ts (nothing obtained from openai)
• packages/crawler/src/metrics.ts (nothing obtained from openai)

Files not reviewed due to errors (5)

• packages/crawler/src/services/snapshot.ts (no response)
• packages/crawler/src/crawler.ts (no response)
• blocklets/snap-kit/api/src/routes/index.ts (no response)
• packages/crawler/src/metrics.ts (no response)
• packages/crawler/src/puppeteer.ts (no response)

Review comments generated (0)

• Review: 0
• LGTM: 0

---

提示

与 AIGNE Reviewer (@aignebot) 聊天

• 回复此机器人留下的审查评论以提出后续问题。审查评论是对差异或文件的评论。
• 通过在回复中标记 @aignebot 邀请机器人加入审查评论链。

代码建议

• 机器人可能会提出代码建议，但在提交前请仔细审查它们，因为行号范围可能会不对齐。
• 你可以编辑机器人做出的评论，并在建议稍有偏差时手动调整。

暂停增量审查

• 在 PR 描述中的任何位置添加 @aignebot: ignore 以暂停机器人的进一步审查。