feat: add new ranking tasks for melo #37
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Addresses #30
Description
This PR introduces the MELO Benchmark (Multilingual Entity Linking of Occupations) as a new ranking task for job title normalization into ESCO. MELO provides 42 evaluation datasets spanning 21 languages, built from crosswalks between national occupation taxonomies and ESCO published by official labor-related organizations across EU member states.
Additionally, we include MELS (Multilingual Entity Linking of Skills), a sibling benchmark following the same methodology but targeting skill normalization into ESCO Skills rather than occupations. MELS currently covers 5 languages with 8 datasets, providing complementary evaluation coverage for the skill normalization task group.
This PR is built on top of #34, which introduces a refactor with the generalized dataset indexing infrastructure required for this implementation. As such, this PR is contingent on #34 being merged. If the maintainers prefer a different approach for the refactor, I would be happy to adapt the implementation accordingly.
Changes:
MELORankingtask class with 42 datasets across 21 languages for job normalizationMELSRankingtask class with 8 datasets across 5 languages for skill normalizationRankingDatasetconstructor to supportallow_duplicate_targetsparameter (required by MELO)Checklist