Release of initial benchmark#52

Merged

harumiWeb merged 38 commits intomainfrom

Jan 29, 2026

Commits on Jan 24, 2026

feat: Initialize benchmark project with extraction and evaluation pipelines
harumiWeb
committed
feat: Add manifest and truth data for application forms, flowcharts, and Gantt charts
harumiWeb
committed
fix
harumiWeb
committed
feat: Update Makefile and README for exstruct installation; enhance paths and extraction logic
harumiWeb
committed
fix: Correct flowchart ID and file paths in manifest.json
harumiWeb
committed
feat: Add taskipy as a development dependency and update task definitions in pyproject.toml
harumiWeb
committed
fix
harumiWeb
committed
feat: Update LLM client and CLI to support temperature parameter for model queries
harumiWeb
committed
feat: Update manifest and truth files for improved data extraction; add certificate of employment JSON structure
harumiWeb
committed
feat: Add tax report case to manifest and corresponding truth data
harumiWeb
committed
feat: Enhance scoring functions with normalization and support for nested structures
harumiWeb
committed
feat: Add SmartArt organization chart case to manifest with corresponding truth data
harumiWeb
committed
feat: Refactor extraction process to use ExStructEngine for improved functionality
harumiWeb
committed
feat: Add basic document case to manifest with corresponding truth data
harumiWeb
committed
feat: Add total cost and call count tracking to ask function
harumiWeb
committed
feat: Update tax report question and truth data structure for improved clarity and consistency
harumiWeb
committed
feat: Add normalization rules and scoring enhancements for improved evaluation accuracy
harumiWeb
committed
feat: Add alias rules for certificate of employment to normalization rules
harumiWeb
committed
feat: Enhance benchmark report with interpretation guidelines for accuracy evaluation
harumiWeb
committed
feat: Move summary output to the end of the report function for better clarity
harumiWeb
committed
feat: Add evaluation protocol to README and report function for reproducibility
harumiWeb
committed
feat: Add reproducibility scripts for Windows PowerShell and macOS/Linux
harumiWeb
committed
feat: Add normalization rules and truth data for heatstroke and workflow cases
harumiWeb
committed
feat: Add raw evaluation metrics and update README for new evaluation tracks
harumiWeb
committed
fix: Format JSON structure for better readability and consistency
harumiWeb
committed

Commits on Jan 25, 2026

Commits on Jan 26, 2026

Commits on Jan 27, 2026