rafiattrach · rajna-fani · Oct 1, 2025 · Oct 1, 2025
diff --git a/benchmarks/ehrsql-naacl2024/README.md b/benchmarks/ehrsql-naacl2024/README.md
@@ -2,25 +2,14 @@
 
 ## Overview
 
-Curated benchmark with **100 examples** from the Reliable Text-to-SQL on Electronic Health Records shared task (Clinical NLP Workshop @ NAACL 2024).
+Benchmark results comparing different models on the EHRSQL dataset with one hundred questions covering various medical queries including cost analysis, temporal measurement differences, medication prescriptions, lab results, patient demographics etc.
 
-**Source**: [ehrsql-2024](https://github.com/glee4810/ehrsql-2024) | **Database**: MIMIC-IV Demo
+**Source**: [ehrsql-2024](https://github.com/glee4810/ehrsql-2024)
 
-## Data Schema
+Each model folder contains:
+- **Model answers** extracted from conversations
+- **Golden truth answers** and SQL queries for comparison
+- **Correct/Incorrect** annotations with detailed notes
+- **Chat conversation links** (Claude.ai shared links or local conversation files)
 
-| Column | Description |
-|--------|-------------|
-| Query | Natural language question about EHR data |
-| Chat Conversation | Link to model interaction |
-| Model Answer | AI-generated response |
-| Golden Truth | Expected correct answer |
-| Golden Truth SQL Query | Ground truth SQL query |
-| Correct/Incorrect | 1 = correct, 0 = incorrect |
-| Incorrect Note | Error analysis when applicable |
-
-## Query Examples
-
-- Patient-specific: Lab values, medications, procedures
-- Temporal: Time-based analysis, trends
-- Aggregate: Population statistics
-- Complex joins: Multi-table EHR relationships
+The dataset includes complex medical questions requiring database queries, with model performance evaluated against ground truth answers through human assessment.
diff --git a/...rks/ehrsql-naacl2024/EHRSQL_benchmark.csv → ...2024/claude-sonnet-4/EHRSQL_benchmark.csv b/...rks/ehrsql-naacl2024/EHRSQL_benchmark.csv → ...2024/claude-sonnet-4/EHRSQL_benchmark.csv
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/EHRSQL_benchmark.csv b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/EHRSQL_benchmark.csv
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/10.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/10.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/100.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/100.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/101.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/101.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/11.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/11.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/12.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/12.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/13.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/13.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/14.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/14.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/15.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/15.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/16.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/16.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/17.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/17.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/18.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/18.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/19.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/19.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/2.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/2.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/20.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/20.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/21.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/21.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/22.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/22.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/23.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/23.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/24.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/24.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/25.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/25.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/26.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/26.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/27.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/27.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/28.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/28.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/29.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/29.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/3.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/3.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/30.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/30.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/31.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/31.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/32.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/32.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/33.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/33.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/34.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/34.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/35.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/35.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/36.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/36.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/37.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/37.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/38.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/38.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/39.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/39.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/4.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/4.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/40.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/40.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/41.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/41.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/42.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/42.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/43.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/43.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/44.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/44.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/45.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/45.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/46.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/46.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/47.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/47.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/48.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/48.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/49.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/49.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/5.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/5.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/50.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/50.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/51.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/51.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/52.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/52.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/53.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/53.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/54.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/54.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/55.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/55.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/56.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/56.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/57.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/57.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/58.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/58.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/59.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/59.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/6.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/6.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/60.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/60.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/61.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/61.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/62.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/62.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/63.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/63.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/64.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/64.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/65.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/65.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/66.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/66.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/67.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/67.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/68.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/68.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/69.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/69.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/7.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/7.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/70.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/70.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/71.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/71.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/72.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/72.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/73.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/73.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/74.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/74.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/75.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/75.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/76.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/76.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/77.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/77.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/78.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/78.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/79.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/79.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/8.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/8.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/80.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/80.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/81.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/81.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/82.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/82.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/83.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/83.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/84.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/84.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/85.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/85.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/86.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/86.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/87.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/87.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/88.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/88.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/89.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/89.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/9.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/9.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/90.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/90.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/91.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/91.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/92.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/92.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/93.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/93.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/94.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/94.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/95.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/95.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/96.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/96.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/97.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/97.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/98.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/98.conversation.json
diff --git a/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/99.conversation.json b/benchmarks/ehrsql-naacl2024/gpt-oss-20B/conversations/99.conversation.json