Enhancement: Improve Data Synthesiser Question Variety #5
Description
Is your feature request related to a problem? Please describe.
The current synthesise_test_cases function is excellent but tends to generate simple, factual-retrieval questions. A more robust evaluation requires testing against a wider variety of question types.
Describe the solution you'd like
Enhance the synthesis prompt and logic to generate different types of questions, such as:
- Comparative questions: "What is the difference between X and Y?"
- Multi-hop questions: Questions that require information from multiple parts of the context.
- Negative questions: "What is not mentioned about X?"
Describe alternatives you've considered
Users can write these complex questions by hand, but automating their generation would be a powerful feature.
Additional context
This would make the generated test sets much more challenging and comprehensive.