Skip to content

Conversation

@federetyk
Copy link
Contributor

Addresses #30

Description

This PR introduces the MELO Benchmark (Multilingual Entity Linking of Occupations) as a new ranking task for job title normalization into ESCO. MELO provides 42 evaluation datasets spanning 21 languages, built from crosswalks between national occupation taxonomies and ESCO published by official labor-related organizations across EU member states.

Additionally, we include MELS (Multilingual Entity Linking of Skills), a sibling benchmark following the same methodology but targeting skill normalization into ESCO Skills rather than occupations. MELS currently covers 5 languages with 8 datasets, providing complementary evaluation coverage for the skill normalization task group.

This PR is built on top of #34, which introduces a refactor with the generalized dataset indexing infrastructure required for this implementation. As such, this PR is contingent on #34 being merged. If the maintainers prefer a different approach for the refactor, I would be happy to adapt the implementation accordingly.

Changes:

  • Add MELORanking task class with 42 datasets across 21 languages for job normalization
  • Add MELSRanking task class with 8 datasets across 5 languages for skill normalization
  • Extend RankingDataset constructor to support allow_duplicate_targets parameter (required by MELO)
  • Add unit tests for dataset ID filtering logic with various language combinations
  • Add defensive check in e2e test to skip tasks with no datasets for the requested language set
  • Update README with new task entries

Checklist

  • Added new tests for new functionality
  • Tested locally with example tasks
  • Code follows project style guidelines
  • Documentation updated
  • No new warnings introduced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant